Optoelectronic computing system
By designing an optical matrix multiplication unit and an optical modulator in the photoelectric computing system, combined with a digital analog conversion unit, the problem of difficulty in efficient matrix multiplication in the existing technology is solved, and the calculation efficiency and flexibility are improved.
Patent Information
- Application Number
- CN202110243631.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-19
- Filing Date
- 2019-06-04
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2039-06-04
AI Technical Summary
When performing artificial neural network computing, it is difficult to efficiently combine electrical signals and optical signals for matrix multiplication operations, which has problems such as low signal conversion efficiency and high system complexity.
A system including an optical matrix multiplication unit, an optical modulator, a laser unit, a digital analog conversion unit and a controller is designed. The optical input vector is converted into an optical output vector through the optical matrix multiplication unit, and the optical modulator and a laser unit are used. Matrix multiplication operation is realized, signal conversion is performed in combination with digital and analog conversion units, and the controller finally performs nonlinear conversion and storage operations.
It improves the efficiency and flexibility of artificial neural network computing, reduces system complexity, realizes the efficient combination of optical signals and electrical signals, and reduces calculation delay and power consumption.
Smart Images

Figure CN113159305B_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application with application date of June 4, 2019, application number 201980030464.8, and invention name “Optoelectronic Computing System”.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims priority to U.S. Provisional Application No. 62 / 680,944 filed on June 5, 2018, U.S. Provisional Application No. 62 / 744,706 filed on October 12, 2018, U.S. Provisional Application No. 62 / 792,144 filed on January 14, 2019, and U.S. Provisional Application No. 62 / 820,562 filed on March 19, 2019. The entire disclosures of the applications are incorporated herein by reference. Technical Field
[0004] The present disclosure relates to an optoelectronic computing system. Background Art
[0005] Neuromorphic computing is an approach in electronics that approximates the operation of the brain. A prominent approach in neuromorphic computing is the artificial neural network (ANN), which is a collection of artificial neurons that are interconnected in a specific way to process information in a manner similar to the functioning of the brain. ANNs have found use in a variety of applications, including artificial intelligence, speech recognition, text recognition, natural language processing, and various forms of pattern recognition.
[0006] ANN has an input layer, one or more hidden layers, and an output layer. Each layer has nodes or artificial neurons, and the nodes are interconnected between the layers. Each node of the hidden layer performs a weighted sum of the signals received from the nodes of the previous layer, and performs a nonlinear transformation ("activation") of the weighted sum to produce an output. The weighted sum can be calculated by performing a matrix multiplication step. Therefore, calculating an ANN usually involves multiple matrix multiplication steps, which are usually performed using electronic integrated circuits.
[0007] Computations performed on electronic data encoded in analog or digital form on electronic signals (e.g., voltages or currents) are typically implemented using electronic computing hardware, such as analog or digital electronics implemented in integrated circuits (e.g., processors, application-specific integrated circuits (ASICs), or systems on a chip (SoCs)), electronic circuit boards, or other electronic circuits. Optical signals have been used to transmit data over long distances and shorter distances (e.g., within data centers). Operations performed on such optical signals are typically performed in the context of optical data transmission, such as within devices used to switch or filter optical signals in a network. The use of optical signals in computing platforms has been more limited. Various components and systems for all-optical computing have been proposed. Such systems may include conversions from and to electrical signals at input and output, respectively, but both types of signals (electrical and optical) may not be used for important operations performed in computing. Summary of the invention
[0008] In general, in a first aspect, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; a digital-to-analog conversion (DAC) unit configured to generate a plurality of modulator control signals and to generate a plurality of weight control signals; an optical processor including a laser unit configured to generate a plurality of optical outputs; a plurality of optical modulators coupled to the laser unit and the DAC unit, the plurality of optical modulators configured to generate an optical input vector by modulating the plurality of optical outputs generated by the laser unit based on the plurality of modulator control signals; an optical matrix multiplication unit coupled to the plurality of optical modulators and the DAC unit, the optical matrix multiplication unit configured to convert the optical input vector into an optical output vector based on the plurality of weight control signals; and a photodetector. A detection unit, coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vector; an analog-to-digital conversion (ADC) unit, coupled to the photodetection unit and configured to convert the plurality of output voltages into a plurality of digital optical outputs; a controller, including an integrated circuit, configured to perform the following operations: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector; storing the input data set and the first plurality of neural network weights in a storage unit; and generating, through a DAC unit, a first plurality of modulator control signals based on the first digital input vector, and generating a first plurality of weight control signals based on the first plurality of neural network weights.
[0009] Embodiments of the system may include one or more of the following features. For example, the operation may further include: obtaining a first plurality of digital light outputs corresponding to the light output vector of the light matrix multiplication unit from the ADC unit, the first plurality of digital light outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.
[0010] The system may have a first cycle period defined as the time elapsed between the step of storing the input data set and the first plurality of neural network weights in the memory unit and the step of storing the first transformed digital output vector in the memory unit. The first cycle period may be less than or equal to 1 ns.
[0011] In some embodiments, the operations may further include: outputting an artificial neural network output generated based on the first converted digital output vector.
[0012] In some embodiments, the operations may further include generating, by a DAC unit, a second plurality of modulator control signals based on the first converted digital output vector.
[0013] In some embodiments, the artificial neural network calculation request may further include a second plurality of neural network weights, and the operation may further include: based on the acquisition of the first plurality of digital light outputs, generating a second plurality of weight control signals based on the second plurality of neural network weights by a DAC unit. The first plurality of neural network weights and the second plurality of neural network weights may correspond to different layers of the artificial neural network.
[0014] In some embodiments, the input data set may further include a second digital input vector, and the operation may further include: generating a second plurality of modulator control signals based on the second digital input vector by a DAC unit; obtaining a second plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit from the ADC unit, the second plurality of digital optical outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; storing the second transformed digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first transformed digital output vector and the second transformed digital output vector. The optical output vector of the optical matrix multiplication unit is generated by the second optical input vector generated based on the second plurality of modulator control signals, and the second optical input vector is transformed by the optical matrix multiplication unit based on the first mentioned plurality of weight control signals.
[0015] In some embodiments, the system may further include: an analog nonlinear unit, which is arranged between the photodetection unit and the ADC unit, the analog nonlinear unit is configured to receive multiple output voltages from the photodetection unit, apply a nonlinear transfer function, and output multiple conversion output voltages to the ADC unit, and the operation further includes: obtaining a first plurality of conversion digital output voltages corresponding to the multiple conversion output voltages from the ADC unit, the first plurality of conversion digital output voltages forming a first conversion digital output vector; and storing the first conversion digital output vector in a storage unit.
[0016] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.
[0017] In some embodiments, the system may further include: an analog storage unit disposed between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store analog voltages and output the stored analog voltages; and an analog nonlinear unit disposed between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output voltages from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output voltages. The analog storage unit may include a plurality of capacitors.
[0018] In some embodiments, the analog storage unit may be configured to receive and store multiple conversion output voltages of the analog nonlinear unit, and output the stored multiple conversion output voltages to multiple optical modulators, and the operation may further include: storing multiple conversion output voltages of the analog nonlinear unit in the analog storage unit based on generating a first plurality of modulator control signals and a first plurality of weight control signals; outputting the stored conversion output voltages through the analog storage unit; obtaining a second plurality of conversion digital output voltages from the ADC unit, the second plurality of conversion digital output voltages forming a second conversion digital output vector; and storing the second conversion digital output vector in the storage unit.
[0019] In some embodiments, the input data set of the artificial neural network calculation request may include multiple digital input vectors. The laser unit may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator bank configured to generate multiple optical input vectors, each optical modulator bank corresponds to one of the multiple wavelengths, and generates a corresponding optical input vector with a corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including multiple wavelengths. The photodetection unit may be further configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output voltages. The operation may include: obtaining multiple digital demultiplexed optical outputs from the ADC unit, the multiple digital demultiplexed optical outputs forming multiple first digital output vectors, wherein each of the multiple first digital output vectors corresponds to one of the multiple wavelengths; performing nonlinear conversion on each of the multiple first digital output vectors to generate multiple converted first digital output vectors; and storing the multiple converted first digital output vectors in a storage unit. Each of the multiple digital input vectors may correspond to one of the multiple optical input vectors.
[0020] In some embodiments, the artificial neural network calculation request may include multiple digital input vectors. The laser unit may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponds to one of the multiple wavelengths, and generates a corresponding optical input vector with a corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including multiple wavelengths. The operation may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vector from the ADC unit, the optical output vector including multiple wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear conversion on the first digital output vector to generate a first converted digital output vector; and storing the first converted digital output vector in a storage unit.
[0021] In some embodiments, the DAC unit may include: a 1-bit DAC subunit configured to generate a plurality of 1-bit modulator control signals. The resolution of the ADC unit may be 1 bit. The resolution of the first digital input vector may be N bits. The operation may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors by the 1-bit DAC subunit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear conversion on the constructed N-bit digital output vector to generate a converted N-bit digital output vector; and storing the converted N-bit digital output vector in a storage unit.
[0022] In some embodiments, the storage unit may include: a digital input vector memory configured to store a digital input vector and including at least one SRAM; and a neural network weight memory configured to store a plurality of neural network weights and including at least one DRAM.
[0023] In some embodiments, the DAC unit may include: a first DAC subunit configured to generate a plurality of modulator control signals; and a second DAC subunit configured to generate a plurality of weight control signals, wherein the first DAC subunit and the second DAC subunit are different.
[0024] In some embodiments, the laser unit may include: a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.
[0025] In some embodiments, the plurality of optical modulators may include one of an MZI modulator, a ring resonator modulator, or an electro-absorption modulator.
[0026] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output voltages.
[0027] In some embodiments, the integrated circuit may be an application specific integrated circuit.
[0028] In some embodiments, the optical matrix multiplication unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation of the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.
[0029] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change a separation ratio of the MZI; and a second phase shifter configured to shift a phase of an output of the MZI, wherein the first phase shifter and the second phase shifter are coupled to a plurality of weight control signals.
[0030] In another aspect, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; a driver unit configured to generate a plurality of modulator control signals and to generate a plurality of weight control signals; an optical processor including: a laser unit configured to generate a plurality of optical outputs; a plurality of optical modulators coupled to the laser unit and the driver unit, the plurality of optical modulators configured to generate an optical input vector by modulating the plurality of optical outputs generated by the laser unit based on the plurality of modulator control signals; an optical matrix multiplication unit coupled to the plurality of optical modulators and the driver unit, the optical matrix multiplication unit configured to convert the optical input vector into an optical output vector based on the plurality of weight control signals; and a photodetection unit coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors; a comparator unit coupled to the photodetection unit and configured to convert the plurality of output voltages into a plurality of digital 1-bit optical outputs; and A controller, including an integrated circuit, is configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector with an N-bit resolution; store the input data set and the first plurality of neural network weights in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through a driver unit; obtain a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from a comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.
[0031] In another aspect, a method is provided for performing artificial neural network calculations in a system having an optical matrix multiplication unit, the optical matrix multiplication unit being configured to convert an optical input vector into an optical output vector based on a plurality of weight control signals, the method comprising: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights from a computer, wherein the input data set includes a first digital input vector; storing the input data set and the first plurality of neural network weights in a storage unit; generating a first plurality of modulator control signals based on the first digital input vector and a first plurality of weight control signals based on the first plurality of neural network weights through a digital-to-analog conversion (DAC) unit; obtaining a first plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit from an analog-to-digital conversion (ADC) unit, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear conversion on the first digital output vector by a controller to generate a first converted digital output vector; storing the first converted digital output vector in the storage unit; and outputting, by the controller, an artificial neural network output generated based on the first converted digital output vector.
[0032] In another aspect, a method includes providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optically converting the optical input vector into an optical output vector based on optical matrix multiplication; converting the optical output vector into an electronic format; and electronically applying a nonlinear transformation to the electronically converted optical output vector to provide output information in an electronic format.
[0033] Embodiments of the method may include one or more of the following features. For example, the method may further include: repeating electronic-to-optical converting, optical transforming, optical-to-electronic converting, and electrically applied nonlinear conversion for new electronic input information corresponding to output information provided in an electronic format.
[0034] In some embodiments, the optical matrix multiplication for the initial optical conversion and the optical matrix multiplication for the repeated optical conversion may be the same and may correspond to the same layer of the artificial neural network.
[0035] In some embodiments, the optical matrix multiplication for the initial optical conversion and the optical matrix multiplication for the repeated optical conversion may be different and may correspond to different layers of the artificial neural network.
[0036] In some embodiments, the method may further include: for different parts of the electronic input information, repeating electro-optical conversion, optical conversion, optoelectronic conversion and electrically applied nonlinear conversion, wherein the optical matrix multiplication used for the initial optical conversion and the optical matrix multiplication for the repeated optical conversion are the same and correspond to the first layer of the artificial neural network.
[0037] In some embodiments, the method may further include: providing intermediate information in an electronic format based on electronic output information for multiple parts of the electronic input information generated by the first layer of the artificial neural network; and for each different part of the electronic intermediate information, repeating electro-optical conversion, optical conversion, optoelectronic conversion and electrically applied nonlinear conversion, wherein the optical matrix multiplication used for the initial optical conversion and the optical matrix multiplication of the repeated optical conversion associated with the different parts of the electronic intermediate information are the same and correspond to the second layer of the artificial neural network.
[0038] In another aspect, a system includes: an optical processor comprising a passive diffractive optical element, wherein the passive diffractive optical element is configured to convert a light input vector or matrix into a light output vector or matrix representing the result of matrix processing applied to the light input vector or matrix and a predetermined vector defined by the arrangement of the diffractive optical element.
[0039] Embodiments of the system may include one or more of the following features.For example, matrix processing may include matrix multiplication between a light input vector or matrix and a predetermined vector defined by the arrangement of the diffractive optical components.
[0040] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide array for receiving an optical input vector; an optical interference unit including a passive diffractive optical component, wherein the optical interference unit is in optical communication with the input waveguide array and is configured to perform a linear transformation of the optical input vector into a second optical signal array; and an output waveguide array, in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide of the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.
[0041] In some embodiments, the light interference unit may include a substrate having at least one of a hole or a stripe, the hole has a size in a range of 100 nm to 10 μm, and the stripe has a width in a range of 100 nm to 10 μm.
[0042] In some embodiments, the light interference unit may include a substrate having passive diffractive optical components arranged in a two-dimensional configuration, and the substrate includes at least one of a planar substrate or a curved substrate.
[0043] In some embodiments, the substrate may include a planar substrate that is parallel to the direction of light propagation from the input waveguide array to the output waveguide array.
[0044] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide matrix for receiving an optical input matrix; an optical interference unit including a passive diffractive optical component, wherein the optical interference unit is in optical communication with the input waveguide matrix and is configured to perform a linear conversion of the optical input matrix into a second optical signal matrix; and an output waveguide matrix, which is in optical communication with the optical interference unit and is used to guide the second optical signal matrix, wherein at least one input waveguide of the input waveguide matrix is in optical communication with each output waveguide in the output waveguide matrix through the optical interference unit.
[0045] In some embodiments, the light interference unit may include a substrate having at least one of holes or stripes, the size of the holes is in the range of 100 nm to 10 μm, and the width of the stripes is in the range of 100 nm to 10 μm.
[0046] In some embodiments, a light interference unit may include a substrate having passive diffractive optical components arranged in a three-dimensional configuration.
[0047] In some embodiments, the substrate may have a shape of at least one of a cube, a column, a prism, or an irregular volume.
[0048] In some embodiments, the optical processor may include an optical interference unit including a hologram having a passive diffractive optical component, the optical processor being configured to receive modulated light representing a light input matrix and to continuously convert the light as the light passes through the hologram until the light is emitted from the hologram as a light output matrix.
[0049] In some embodiments, the light interference unit may include a substrate having a passive diffractive optical component, and the substrate includes at least one of silicon, silicon oxide, silicon nitride, quartz, lithium niobate, a phase change material, or a polymer.
[0050] In some embodiments, the light interference unit may include a substrate having a passive diffractive optical component, and the substrate includes at least one of a glass substrate or an acrylic substrate.
[0051] In some embodiments, a passive diffractive optical component may be formed in part by a dopant.
[0052] In some embodiments, the matrix processing may represent processing of input data by a neural network, where the input data is represented by an optical input vector.
[0053] In some embodiments, the optical processor may include: a laser unit configured to generate multiple light outputs; a plurality of optical modulators coupled to the laser unit and configured to generate a light input vector by modulating the multiple light outputs generated by the laser unit based on a plurality of modulator control signals; an optical matrix processing unit coupled to the plurality of optical modulators, the optical matrix processing unit including a passive diffractive optical component configured to convert the light input vector into a light output vector based on a plurality of weights defined by the passive diffractive optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate a plurality of output electrical signals corresponding to the light output vectors.
[0054] In some embodiments, the passive diffractive optical component may be arranged in a three-dimensional configuration, the plurality of light modulators comprises a two-dimensional light modulator array, and the photodetection unit comprises a two-dimensional photodetector array.
[0055] In some embodiments, the optical matrix processing unit may include a housing module to support and protect the input waveguide array, the optical interference unit and the output waveguide array, the optical processor includes a receiving module, the receiving module is configured to receive the optical matrix processing unit, the receiving module includes a first interface, enabling the optical matrix processing unit to receive optical input vectors from multiple optical modulators, and a second interface, enabling the optical matrix processing unit to transmit the optical output vector to the photoelectric detection unit.
[0056] In some embodiments, the plurality of output electrical signals may include at least one of a plurality of voltage signals or a plurality of current signals.
[0057] In some embodiments, the system may include: a storage unit; a digital-to-analog conversion (DAC) unit configured to generate a plurality of modulator control signals; an analog-to-digital conversion (ADC) unit coupled to the photodetection unit and configured to convert a plurality of output electrical signals into a plurality of digital outputs; and a controller including an integrated circuit configured to perform the following operations: receive an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector; store the input data set in the storage unit; and generate a first plurality of modulator control signals based on the first digital input vector through the DAC unit.
[0058] In another aspect, a method includes 3D printing a light matrix processing unit including a passive diffractive optical component, wherein the passive diffractive optical component is configured to convert a light input vector or matrix into a light output vector or matrix representing a result of matrix processing applied to the light input vector or matrix and a predetermined vector defined by an arrangement of the diffractive optical component.
[0059] In another aspect, a method includes: using one or more laser beams to produce a hologram including a passive diffractive optical component, wherein the passive diffractive optical component is configured to convert a light input vector or matrix into a light output vector or matrix representing the result of matrix processing applied to the light input vector or matrix and a predetermined vector defined by the arrangement of the diffractive optical component.
[0060] In another aspect, a system includes: an optical processor comprising passive diffractive optical components arranged in a one-dimensional manner, wherein the passive diffractive optical components are configured to convert an optical input into an optical output representing a result of matrix processing applied to the optical input and a predetermined vector defined by the arrangement of the diffractive optical components.
[0061] Embodiments of the system may include one or more of the following features.For example, the matrix processing may include a matrix multiplication between the light input and a predetermined vector defined by the arrangement of the diffractive optical components.
[0062] In some embodiments, the optical processor may include an optical matrix processing unit, which includes: an input waveguide for receiving an optical input; an optical interference unit including a passive diffractive optical component, wherein the optical interference unit is in optical communication with the input waveguide and is configured to perform a linear transformation of the optical input; and an output waveguide, in optical communication with the optical interference unit, for guiding the optical output.
[0063] In some embodiments, the light interference unit may include a substrate having at least one of a hole or a grating, and the hole or the grating component may have a size in the range of 100 nm to 10 μm.
[0064] In another aspect, a system includes: a storage unit; a digital-to-analog conversion (DAC) unit configured to generate multiple modulator control signals; and an optical processor including: a laser unit configured to generate multiple light outputs; multiple light modulators coupled to the laser unit and the DAC unit, the multiple light modulators configured to generate a light input vector by modulating the multiple light outputs generated by the laser unit based on the multiple modulator control signals; an optical matrix processing unit coupled to the multiple light modulators, the optical matrix processing unit including a passive diffractive optical component configured to convert the light input vector into a light output vector based on multiple weights defined by the passive diffractive optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate multiple output electrical signals corresponding to the light output vectors. The system further includes: an analog-to-digital conversion (ADC) unit coupled to the photodetection unit and configured to convert the plurality of output electrical signals into a plurality of digital light outputs; and a controller including an integrated circuit configured to perform the following operations: receiving an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector; storing the input data set in a storage unit; and generating, through a DAC unit, a first plurality of modulator control signals based on the first digital input vector.
[0065] Embodiments of the system may include one or more of the following features. For example, the matrix processing unit may include a passive diffractive optical component configured to convert a light input vector into a light output vector, the light output vector representing the product of a matrix multiplication between the light input vector and a predetermined vector defined by the passive diffractive optical component.
[0066] In some embodiments, the operation further includes: obtaining a first plurality of digital light outputs corresponding to the light output vector of the light matrix processing unit from the ADC unit, the first plurality of digital light outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first converted digital output vector; and storing the first converted digital output vector in a storage unit.
[0067] In some embodiments, the system may have a first cycle period defined as the time elapsed between storing an input data set in a memory cell and storing a first converted digital output vector in the memory cell, and wherein the first cycle period may be less than or equal to 1 ns.
[0068] In some embodiments, the operations may further include: outputting an artificial neural network output generated based on the first converted digital output vector.
[0069] In some embodiments, the operations may further include generating, by a DAC unit, a second plurality of modulator control signals based on the first converted digital output vector.
[0070] In some embodiments, the input data set may further include a second digital input vector, and wherein the operation may further include: generating a second plurality of modulator control signals based on the second digital input vector through a DAC unit; obtaining a second plurality of digital light outputs corresponding to the light output vector of the light matrix processing unit from the ADC unit, the second plurality of digital light outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second converted digital output vector; storing the second converted digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first converted digital output vector and the second converted digital output vector, wherein the light output vector of the light matrix processing unit is generated by the second light input vector generated based on the second plurality of modulator control signals, and the second light input vector is converted by the light matrix processing unit based on a plurality of weights defined by the passive diffractive optical component.
[0071] In some embodiments, the system may further include: an analog nonlinear unit, which is arranged between the photodetection unit and the ADC unit, and the analog nonlinear unit is configured to receive multiple output electrical signals from the photodetection unit, apply a nonlinear transfer function, and output multiple conversion output electrical signals to the ADC unit, wherein the operation may further include: obtaining a first plurality of conversion digital output electrical signals corresponding to the multiple conversion output electrical signals from the ADC unit, the first plurality of conversion digital output electrical signals forming a first conversion digital output vector; and storing the first conversion digital output vector in a storage unit.
[0072] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.
[0073] In some embodiments, the system may further include: an analog storage unit, disposed between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store an analog voltage and output the stored analog voltage; and an analog nonlinear unit, disposed between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output electrical signals from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output electrical signals.
[0074] In some embodiments, the analog memory cell may include multiple capacitors.
[0075] In some embodiments, the analog storage unit may be configured to receive and store multiple conversion output electrical signals of the analog nonlinear unit, and output the stored multiple conversion output electrical signals to multiple optical modulators, and wherein the operation may further include: based on generating a first plurality of modulator control signals, storing the multiple conversion output electrical signals of the analog nonlinear unit in the analog storage unit; outputting the stored conversion output electrical signals through the analog storage unit; obtaining a second plurality of conversion digital output electrical signals from the ADC unit, the second plurality of conversion digital output electrical signals forming a second conversion digital output vector; and storing the second conversion digital output vector in the storage unit.
[0076] In some embodiments, the input data set of the artificial neural network calculation request may include multiple digital input vectors, wherein the laser unit may be configured to generate multiple wavelengths, and wherein the multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponds to one of the multiple wavelengths, and generates a corresponding optical input vector with a corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including the multiple wavelengths. The photodetection unit may be further configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output electrical signals, and the operation may include: obtaining multiple digital demultiplexed optical outputs from the ADC unit, the multiple digital demultiplexed optical outputs forming multiple first digital output vectors, wherein each of the multiple first digital output vectors corresponds to one of the multiple wavelengths; performing nonlinear conversion on each of the multiple first digital output vectors to generate multiple converted first digital output vectors; and storing the multiple converted first digital output vectors in a storage unit, wherein each of the multiple digital input vectors corresponds to one of the multiple optical input vectors.
[0077] In some embodiments, the artificial neural network calculation request may include a plurality of digital input vectors, wherein the laser unit is configured to generate a plurality of wavelengths, and wherein the plurality of optical modulators may include: an optical modulator group configured to generate a plurality of optical input vectors, each optical modulator group corresponding to one of the plurality of wavelengths, and generating a corresponding optical input vector having a corresponding wavelength; and an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector including the plurality of wavelengths. The operation may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vector from the ADC unit, the optical output vector including the plurality of wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in a storage unit.
[0078] In some embodiments, the DAC unit may include: a 1-bit DAC unit configured to generate a plurality of 1-bit modulator control signals, wherein the resolution of the ADC unit may be 1 bit, and wherein the resolution of the first digital input vector may be N bits. The operation may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through the 1-bit DAC unit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of the N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of the N digital 1-bit optical outputs; performing a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and storing the transformed N-bit digital output vector in a storage unit.
[0079] In some embodiments, the storage unit may include: a digital input vector memory configured to store the digital input vector and including at least one SRAM.
[0080] In some embodiments, the laser unit may include: a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.
[0081] In some embodiments, the plurality of optical modulators include one of an MZI modulator, a ring resonance modulator, or an electro-absorption modulator.
[0082] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output electrical signals.
[0083] In some embodiments, the integrated circuit may include an application specific integrated circuit.
[0084] In some embodiments, the optical matrix processing unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation of the optical input vector into a second optical signal array, wherein the optical interference unit includes a passive diffraction optical component; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array optically communicates with each output waveguide in the output waveguide array through the optical interference unit.
[0085] In another aspect, a system includes: a storage unit; a driver unit configured to generate multiple modulator control signals; an optical processor including: a laser unit configured to generate multiple light outputs; multiple light modulators coupled to the laser unit and the driver unit, the multiple light modulators configured to generate a light input vector by modulating the multiple light outputs generated by the laser unit based on the multiple modulator control signals; an optical matrix processing unit coupled to the multiple light modulators and the driver unit, the optical matrix processing unit including a passive diffractive optical component, configured to convert the light input vector into a light output vector based on multiple weight control signals defined by the passive diffractive optical component; and a photodetection unit coupled to the optical matrix processing unit and configured to generate multiple output electrical signals corresponding to the light output vectors. The system also includes a comparator unit, coupled to the photodetection unit and configured to convert multiple output electrical signals into multiple digital 1-bit light outputs; and a controller, including an integrated circuit, configured to perform the following operations: receive an artificial neural network calculation request including an input data set from a computer, wherein the input data set includes a first digital input vector with an N-bit resolution; store the input data set in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through a driver unit; obtain a sequence of N digital 1-bit light outputs corresponding to the sequence of N 1-bit modulator control signals from the comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit light outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.
[0086] Embodiments of the system may include one or more of the following features. For example, the optical matrix processing unit may include an optical matrix multiplication unit configured to convert an optical input vector into an optical output vector, the optical output vector representing the product of a matrix multiplication between an input vector represented by the optical input vector and a predetermined vector defined by the passive diffractive optical component.
[0087] In another aspect, a method for performing artificial neural network calculations in a system having an optical matrix processing unit comprises: receiving an artificial neural network calculation request including an input data set from a computer, the input data set including a first digital input vector; storing the input data set in a storage unit; generating a first plurality of modulator control signals based on the first digital input vector through a digital-to-analog conversion (DAC) unit; converting the optical input vector into an optical output vector by using an optical matrix processing unit including an arrangement of passive diffractive optical components, wherein the optical output vector represents a result of matrix processing applied to the optical input vector and a predetermined vector defined by the arrangement of the diffractive optical components; obtaining a first plurality of digital optical outputs corresponding to the optical output vector of the optical matrix processing unit from an analog-to-digital conversion (ADC) unit, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear conversion on the first digital output vector by a controller to generate a first converted digital output vector; storing the first converted digital output vector in the storage unit; and outputting an artificial neural network output generated based on the first converted digital output vector by the controller.
[0088] Embodiments of the method may include one or more of the following features. For example, converting the light input vector to the light output vector may include converting the light input vector to the light output vector representing the product of a matrix multiplication between the digital input vector and a predetermined vector defined by the arrangement of the diffractive optical components.
[0089] In another aspect, a method includes: providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optically converting the optical input vector into an optical output vector based on optical matrix processing by an optical processor including a passive diffractive optical component; converting the optical output vector into an electronic format; and electronically applying a nonlinear transformation to the electronically converted optical output vector to provide output information in an electronic format.
[0090] Embodiments of the method may include one or more of the following features. For example, optically converting the light input vector to the light output vector may include optically converting the light input vector to the light output vector based on a light matrix multiplication between a digital input vector represented by the light input vector and a predetermined vector defined by the passive diffractive optical component.
[0091] In some embodiments, the method may further include repeating the electrical-to-optical conversion, the optical conversion, the optoelectronic conversion, and the electrically applied non-linear conversion for new electronic input information corresponding to the output information provided in the electronic format.
[0092] In some embodiments, the light matrix processing for the initial light conversion and the light matrix processing for the repeated light conversion may be the same and may correspond to the same layer of the artificial neural network.
[0093] In some embodiments, the method may further include: for different parts of the electronic input information, repeating electro-optical conversion, optical conversion, optoelectronic conversion and electrically applied nonlinear conversion, wherein the optical matrix processing for the initial optical conversion and the optical matrix processing for the repeated optical conversion may be the same and correspond to a layer of the artificial neural network.
[0094] In another aspect, a system includes an optical matrix processing unit configured to process an input vector of length N, wherein the optical matrix processing unit includes N+2 layers of directional couplers and N layers of phase shifters, and N is a positive integer.
[0095] Embodiments of the system may include one or more of the following features: For example, the optical matrix processing unit may include no more than N+2 layers of directional couplers.
[0096] In some embodiments, the optical matrix processing unit may include an optical matrix multiplication unit.
[0097] In some embodiments, the optical matrix processing unit may include a substrate and interconnected interferometers disposed on the substrate, wherein each interferometer includes an optical waveguide disposed on the substrate, and the directional coupler and the phase shifter are part of the interconnected interferometer.
[0098] In some embodiments, the optical matrix processing unit may include a layer of attenuators after the last layer of directional couplers.
[0099] In some embodiments, a layer of attenuators may include N attenuators.
[0100] In some embodiments, the system may include one or more homodyne detectors for detecting the output from the attenuator.
[0101] In some embodiments, N=3, and the optical matrix processing unit may include: an input terminal configured to receive an input vector; a first layer of directional couplers coupled to the input terminal; a first layer of phase shifters coupled to the first layer of directional couplers; a second layer of directional couplers coupled to the first layer of phase shifters; a second layer of phase shifters coupled to the second layer of directional couplers; a third layer of directional couplers coupled to the second layer of phase shifters; a third layer of phase shifters coupled to the third layer of directional couplers; a fourth layer of directional couplers coupled to the third layer of phase shifters; and a fifth layer of directional couplers coupled to the fourth layer of directional couplers.
[0102] In some embodiments, N=4, and the optical matrix processing unit may include: an input end configured to receive an input vector; first, second, third, and fourth layers of directional couplers, each layer of directional couplers is followed by a layer of phase shifters, wherein the first layer of directional couplers is coupled to the input end; the second-to-last layer of directional couplers is coupled to the fourth layer of phase shifters; and a final layer of directional couplers is coupled to the second-to-last layer of directional couplers.
[0103] In some embodiments, N=8, and the optical matrix processing unit may include: an input end configured to receive an input vector; eight layers of directional couplers, each layer of directional couplers being followed by a layer of phase shifters, wherein a first layer of directional couplers is coupled to the input end; a second-to-last layer of directional couplers is coupled to an eighth layer of phase shifters; and a final layer of directional couplers is coupled to the second-to-last layer of directional couplers.
[0104] In some embodiments, the optical matrix multiplication unit may include: an input end configured to receive an input vector; N layers of directional couplers, each layer of directional couplers is followed by a layer of phase shifters, wherein a first layer of directional couplers is coupled to the input end; a second-to-last layer of directional couplers is coupled to the Nth layer of directional couplers; and a final layer of directional couplers is coupled to the second-to-last layer of directional couplers.
[0105] In some embodiments, N is an even number.
[0106] In some embodiments, each i-th layer of directional couplers includes N / 2 directional couplers, where i is an odd number, and each j-th layer of directional couplers includes N / 2-1 directional couplers, where j is an even number.
[0107] In some embodiments, for each i-th layer of directional couplers where i is an odd number, the k-th directional coupler may be coupled to the (2k-1)-th and 2k-th outputs of the previous layer, where k is an integer from 1 to N / 2.
[0108] In some embodiments, for each jth layer of directional couplers where j is an even number, the mth directional coupler may be coupled to the (2m)th and (2m+1)th outputs of the previous layer, where m is an integer from 1 to N / 2-1.
[0109] In some embodiments, each i-th layer of phase shifters may include N phase shifters, where i is an odd number, and each j-th layer of phase shifters may include N-2 phase shifters, where j is an even number.
[0110] In some embodiments, N may be an odd number.
[0111] In some embodiments, each layer of directional couplers may include (N-1) / 2 directional couplers.
[0112] In some embodiments, each layer of phase shifters may include N-1 phase shifters.
[0113] In another aspect, a system includes: a generator configured to generate a first data set, wherein the generator includes an optical matrix processing unit; and a discriminator configured to receive a second data set including data from the first data set and data from a third data set, the data in the first data set having similar characteristics to the data in the third data set, and classify the data in the second data set as data from the first data set or data from the third data set.
[0114] Embodiments of the method may include one or more of the following features. For example, the optical matrix processing unit may include at least one of the following: (i) the optical matrix multiplication unit described above, (ii) the passive diffractive optical component described above, or (iii) the optical matrix processing unit described above.
[0115] In some embodiments, the third data set may include real data, the generator is configured to generate synthesized data similar to the real data, and the discriminator is configured to classify the data as real data or synthesized data.
[0116] In some embodiments, the generator may be configured to generate a data set for training at least one of an autonomous vehicle, a medical diagnostic system, a fraud detection system, a weather forecasting system, a financial forecasting system, a facial recognition system, a speech recognition system, or a product defect detection system.
[0117] In some embodiments, the generator may be configured to generate an image that resembles an image of at least one of a real object or a real scene, and the discriminator is configured to classify a received image as (i) an image of a real object or a real scene, or (ii) a synthetic image generated by the generator.
[0118] In some embodiments, the real object may include at least one of a person, an animal, a cell, a tissue, or a product, and the real scene includes a scene encountered by a vehicle.
[0119] In some embodiments, the discriminator may be configured to classify a received image as whether it is (i) an image of a real person, a real animal, a real cell, a real tissue, a real product, or a real scene encountered by a vehicle, or (ii) a synthetic image produced by the generator.
[0120] In some embodiments, the vehicle may include at least one of a motorcycle, a car, a truck, a train, a helicopter, an airplane, a submarine, a ship, or a drone.
[0121] In some embodiments, the generator may be configured to generate images of tissue or cells associated with at least one of a human disease, an animal disease, or a plant disease.
[0122] In some embodiments, the generator may be configured to generate images of tissue or cells associated with a human disease, and the disease includes at least one of cancer, Parkinson's disease, sickle cell anemia, heart disease, cardiovascular disease, diabetes, chest disease, or skin disease.
[0123] In some embodiments, the generator may be configured to generate images of tissue or cells associated with cancer, and the cancer may include at least one of skin cancer, breast cancer, lung cancer, liver cancer, prostate cancer, or brain cancer.
[0124] In some embodiments, the system may further include a random noise generator configured to generate random noise that is input to the generator, and the generator is configured to generate the first data set based on the random noise.
[0125] In another aspect, a system includes: a random noise generator configured to generate random noise; and a generator configured to generate data based on the random noise, wherein the generator includes a light matrix processing unit.
[0126] Embodiments of the system may include one or more of the following features. For example, the optical matrix processing unit may include at least one of (i) the optical matrix multiplication unit described above, (ii) the passive diffractive optical component described above, or (iii) the optical matrix processing unit described above.
[0127] In another aspect, a system includes: an optical circuit configured to perform a logic function on two input signals, the optical circuit comprising: a first directional coupler having two input ends and two output ends, the two input ends being configured to receive the two input signals; a first pair of phase shifters configured to modify the phases of the signals at the two output ends of the first directional coupler; a second directional coupler having two input ends and two output ends, the two input ends being configured to receive signals from the first pair of phase shifters; and a second pair of phase shifters configured to modify the phases of the signals at the two output ends of the second directional coupler.
[0128] Embodiments of the method may include one or more of the following features. For example, the phase shifter may be configured to cause the optical circuit to perform a rotation:
[0129]
[0130] In some embodiments, when input signals x1 and x2 are provided to two input terminals of the first directional coupler, the phase shifter may be configured to cause the optical circuit to perform the operation:
[0131]
[0132] In some embodiments, the optical circuit may include a first photodetector configured to generate an absolute value of a signal from the second pair of phase shifters to cause the optical circuit to perform the operation:
[0133]
[0134] In some embodiments, the optical circuit may include a comparator configured to compare the output signal of the first photodetector with a threshold value to generate a binary value to cause the optical circuit to generate an output:
[0135]
[0136] In some embodiments, the optical circuit may include a feedback mechanism configured such that an output signal of the photodetector is fed back to an input terminal of the first directional coupler, passes through the first directional coupler, the first pair of phase shifters, the second directional coupler, and the second pair of phase shifters, and is detected by the photodetector to cause the optical circuit to perform the operation:
[0137]
[0138] It produces the outputs AND(x1,x2) and OR(x1,x2).
[0139] In some embodiments, the optical circuit may include: a third directional coupler having two inputs and two outputs, the two inputs configured to receive signals from the second pair of phase shifters; a third pair of phase shifters configured to modify the phases of signals at the two outputs of the third directional coupler; a fourth directional coupler having two inputs and two outputs, the two inputs configured to receive signals from the third pair of phase shifters; a fourth pair of phase shifters configured to modify the phases of signals at the two outputs of the fourth directional coupler; and a second photodetector configured to generate an absolute value of the signal from the fourth pair of phase shifters, so as to cause the optical circuit to perform the operation:
[0140]
[0141] It produces the outputs AND(x1,x2) and OR(x1,x2).
[0142] In some embodiments, a system may include a bitonic sorter configured to perform a sorting function of the bitonic sorter using an optical circuit.
[0143] In some embodiments, a system may include an apparatus configured to perform a hashing function using an optical circuit.
[0144] In some embodiments, the hash function may include secure hash algorithm 2 (SHA-2).
[0145] Generally speaking, systems for performing computations use different types of operations to produce computational results, each of which is performed on a signal (e.g., an electrical or optical signal) that best suits the underlying physical properties of the operation (e.g., in terms of energy consumption and / or speed). By way of example, three such operations are: copying, summation, and multiplication. Copying can be performed using optical power splitting, summation can be performed using electrical current-based summation, and multiplication can be performed using optical amplitude modulation, as described in more detail below. An example of a computation that can be performed using these three types of operations is multiplying a vector by a matrix (e.g., as employed by artificial neural network computations). Various other computations can be performed using these operations, which represent a set of general linear operations that can perform various computations, including but not limited to: vector-vector dot product, vector-vector element-wise multiplication, vector-scalar element wise multiplication, or matrix-matrix element wise multiplication. Some of the examples described herein show techniques and configurations for vector-matrix multiplication, but the corresponding techniques and configurations can be used for any of these types of computations.
[0146] Aspects may have one or more of the following advantages.
[0147] The optoelectronic computing systems described herein that use electrical and optical signals can promote increased flexibility and / or efficiency. In the past, there may be potential challenges associated with combining optical (or photonic) integrated devices with electrical (or electronic) integrated devices on a common platform (e.g., a common semiconductor die, or multiple semiconductor dies combined in a controlled collapsed chip connection or a "flip-chip" arrangement). For example, such potential challenges may include input / output (I / O) packaging or temperature control. For those systems described herein, potential challenges may be increased when used with a relatively large number of optical input ports and a relatively large number of electrical output ports (e.g., 4 or more optical input / output ports, 200 or more electrical input / output ports). These potential challenges can be mitigated using appropriate system design. For example, the system may use a high-density packaging arrangement that uses temperature control (e.g., thermoelectric cooling) to control thermal expansion between different material types (e.g., semiconductor materials (e.g., silicon), glass materials (silicon dioxide or "silica"), ceramic materials, etc.), and / or uses an enclosing housing as a heat sink and provides a degree of sealing. Using such temperature stabilization techniques, different coefficients of thermal expansion (CTE) and the resulting misalignment between the system ports and the ports of the packaged high-density optical fiber array can be limited.
[0148] For the copy operation, since the optical power splitting is passive, no power is consumed to perform the operation. In addition, the frequency bandwidth of the electrical splitter has a limit related to the RC time constant. In contrast, the frequency bandwidth of the optical splitter is essentially unlimited. Different types of optical power splitters can be used, including waveguide optical splitters or free-space beam splitters, as described in more detail below.
[0149] For multiplication operations, one value may be encoded as an optical signal and the other value may be encoded as an amplitude scaling coefficient (e.g., multiplication by a value in the range of 0 to 1). After the scaling coefficient is set, multiplication operations in the optical domain require reduced (or no) conditioning of the electrical signal and thus reduce constraints due to electrical noise, power consumption, and bandwidth limitations. By appropriately selecting the detection scheme, a signed result (e.g., multiplication by a value between -1 and +1) may be obtained, as described in more detail below.
[0150] For the summation operation, different techniques can be used to achieve a result in which the magnitude of the current in the conductor is determined based on the sum of different contributions. In the case of input current signals, when two or more conductors carrying those input current signals are combined at a junction, the single conductor carrying the output current signal represents the sum of those input current signals. In the case of input optical signals, when two or more light waves of different wavelengths are irradiated on the detector, the current signal carried on the photocurrent generated by the detector represents the sum of the powers in the input optical signals. Both produce an electrical signal (e.g., current) as an output representing the sum, but one uses current as input (current-input-based summation, also known as "electrical summation" performed in the "electrical domain"), while the other uses light waves as input (optical-input-based summation, also known as "optoelectronic summation" performed in the "optoelectronic domain"). However, in some embodiments, current input based summing is used rather than light input based summing, which enables a single optical wavelength to be used in the system, avoiding potentially complex components of the system that may need to be provided and maintain multiple wavelengths.
[0151] Combinations of these basic operations performed by these modules can be arranged to provide a device that performs linear operations (e.g., vector-matrix multiplication with arbitrary matrix element magnitudes). Other implementations of matrix multiplication using optical signals and interferometers for combining signals using optical interference have been limited to providing vector-matrix multiplication with certain restrictions (e.g., unitary matrices or diagonal matrices). In addition, some other embodiments may rely on large-scale phase alignment of multiple optical signals as they propagate through a relatively large number of optical components (e.g., optical modulators). Alternatively, the embodiments described herein may relax such phase alignment constraints by converting the optical signal into an electrical signal after propagating through fewer optical components (e.g., after propagating through no more than a single optical amplitude modulator), which allows the use of optical signals with reduced coherence, or even the use of incoherent optical signals that do not rely on constructive / destructive interference optical modulators.
[0152] For time domain coding of optical and electrical signals, described in more detail below, analog electronic circuits may be optimized for operation at specific power levels, which may be helpful if the circuits are operating at high speeds. Such time domain coding is useful in reducing any challenges that may be associated with accurately controlling a relatively large number of clearly distinguishable intensity levels for each symbol. In contrast, when accurate control of the duty cycle is applied in the time domain over multiple time slots within a single symbol duration, a relatively constant amplitude may be used (with zero or near zero amplitude for an "on" level and an "off" level).
[0153] By integrating photonic devices and electronic devices on a common substrate (e.g., a silicon chip), modules can be easily manufactured on a large scale. Routing signals on the substrate are optical rather than electrical, and grouping photodetectors in a portion of the substrate can help avoid long electronic wiring and its associated challenges (e.g., parasitic capacitance, inductance, and crosstalk).
[0154] For embodiments of the system using sub-matrix multiplication, different devices (e.g., different cores, different processors, different computers, different servers) can be used to simultaneously compute each element of the output vector, helping to alleviate certain potential limitations (e.g., memory walls) and helping the overall system scale to very large matrices. In some embodiments, a different device can be used to multiply each sub-matrix by the corresponding sub-vector. The sum can then be calculated by collecting or accumulating the summands from the different devices. Intermediate results in the form of optical signals can be conveniently transmitted between devices, even if the devices are separated by relatively large distances.
[0155] Other aspects include other combinations of the above features and other features expressed as methods, apparatus, systems, program products, and other forms.
[0156] Particular embodiments of the subject matter described in this specification may be implemented to achieve one or more of the following advantages: ANN computation throughput, latency, or both may be improved. ANN computation power efficiency may be improved.
[0157] In another aspect, an apparatus includes: a plurality of optical waveguides, wherein a set of a plurality of input values are encoded on respective optical signals carried by the optical waveguides; a plurality of replicating modules, and for each of at least two subsets of the one or more optical signals, a respective set of the one or more replicating modules is configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplying modules, and for each of at least two copies of a first subset of the one or more optical signals, a respective multiplying module is configured to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation, wherein at least one of the multiplying modules includes an optical amplitude modulator, the optical amplitude modulator includes one input port and two output ports, and provides a pair of correlated optical signals from the two output ports such that a difference between amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value; and one or more summing modules, and for the results of two or more multiplication modules, a respective one of the summing modules is configured to generate an electrical signal, the electrical signal representing a sum of the results of the two or more multiplication modules.
[0158] Embodiments of the apparatus may include one or more of the following features.For example, an input value of a set of multiple input values encoded on a corresponding optical signal may represent an element of an input vector multiplied by a matrix including one or more matrix element values.
[0159] In some embodiments, a set of multiple output values may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values in the set of multiple output values may represent elements of an output vector generated by multiplying the input vector by a matrix.
[0160] In some embodiments, each optical signal carried by the optical waveguide may include a light wave having a common wavelength, the common wavelength being substantially the same for all optical signals.
[0161] In some embodiments, the replication module may include at least one replication module having an optical splitter that sends a predetermined proportion of the power of the light wave at the input port to the first output port and sends the remaining proportion of the power of the light wave at the input port to the second output port.
[0162] In some embodiments, the optical splitter may include a waveguide optical splitter that sends a predetermined proportion of the power of the optical wave guided by the input optical waveguide to the first output optical waveguide and sends the remaining proportion of the power of the optical wave guided by the input optical waveguide to the second output optical waveguide.
[0163] In some embodiments, a guided mode of the input optical waveguide can be adiabatically coupled to a guided mode of each of the first and second output optical waveguides.
[0164] In some embodiments, the light splitter may include a beam splitter including at least one surface that transmits a predetermined proportion of the power of the light wave at the input port and reflects a remaining proportion of the power of the light wave at the input port.
[0165] In some embodiments, at least one of the plurality of optical waveguides may include an optical fiber coupled to an optical coupler that couples a guided mode of the optical fiber to a free-space propagation mode.
[0166] In some embodiments, the multiplication module may include at least one coherence-sensitive multiplication module, which is configured to multiply one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation based on interference between optical waves, and the optical waves have a coherence length, and the coherence length is at least as long as a propagation distance through the coherence-sensitive multiplication module.
[0167] In some embodiments, the coherence-sensitive multiplication module may include a Mach-Zehnder interferometer (MZI), which separates the light wave guided by the input optical waveguide into a first optical waveguide arm of the MZI and a second optical waveguide arm of the MZI, the first optical waveguide arm includes a phase shifter, the phase shifter produces a relative phase shift relative to the phase delay of the second optical waveguide arm, and the MZI combines the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.
[0168] In some embodiments, the MZI may combine light waves from the first optical waveguide arm and the second optical waveguide arm into each of the first output optical waveguide and the second output optical waveguide, the first photodetector may receive the light wave from the first output optical waveguide to generate a first photocurrent, the second photodetector may receive the light wave from the second output optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.
[0169] In some embodiments, the coherence-sensitive multiplication module may include one or more ring resonators, including at least one ring resonator coupled to the first optical waveguide and at least one ring resonator coupled to the second optical waveguide.
[0170] In some embodiments, the first photodetector may receive a light wave from a first optical waveguide to generate a first photocurrent, the second photodetector may receive a light wave from a second optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.
[0171] In some embodiments, the multiplication module may include at least one coherence-insensitive multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation based on energy absorption within the optical wave.
[0172] In some embodiments, the coherent insensitive multiplication module may include an electro-absorption modulator.
[0173] In some embodiments, the one or more summing modules may include at least one summing module having the following components: (1) two or more input conductors, each input conductor carrying an electrical signal in the form of an input current, the magnitude of the input current representing a corresponding result of a corresponding one of the multiplication modules, and (2) at least one output conductor, the output conductor carrying an electrical signal representing the sum of the corresponding results in the form of an output current, the output current being proportional to the sum of the input currents.
[0174] In some embodiments, the two or more input conductors and the output conductor may include a plurality of wires that meet at one or more nodes between the wires, and the output current is substantially equal to the sum of the input currents.
[0175] In some embodiments, at least a first one of the input currents may be provided in the form of at least one photocurrent generated by at least one photodetector that receives an optical signal generated by a first one of the multiplication modules.
[0176] In some embodiments, the first input current may be provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding light signals generated by the first multiplication module.
[0177] In some embodiments, one of the copies of the first subset of the one or more optical signals may consist of a single optical signal, wherein one of the input values is encoded on the single optical signal.
[0178] In some embodiments, the multiplication module corresponding to the replica of the first subset may multiply the encoded input value by a single matrix element value.
[0179] In some embodiments, one of the copies of the first subset of the one or more optical signals may include more than one, and less than the number of all, optical signals on which the plurality of input values are encoded.
[0180] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input values by different corresponding matrix element values.
[0181] In some embodiments, different multiplication modules corresponding to different respective copies of the first subset of one or more optical signals may be included in different devices that are in optical communication to transmit one of the copies of the first subset of one or more optical signals between the different devices.
[0182] In some embodiments, two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules may be disposed on a substrate of a common device.
[0183] In some embodiments, the apparatus performs a vector-matrix multiplication, where an input vector may be provided as a set of optical signals and an output vector may be provided as a set of electrical signals.
[0184] In some embodiments, the device may further include an accumulator that combines input electrical signals corresponding to the outputs of the multiplication module or the summation module, wherein the input electrical signal may be encoded using time domain encoding, the time domain encoding using on-off amplitude modulation within each of multiple time slots, and the accumulator may generate an output electrical signal, the output electrical signal being encoded with more than two amplitude levels, the amplitude levels corresponding to different duty cycles of the time domain encoding over multiple time slots.
[0185] In some embodiments, each of the two or more of the multiplication modules corresponds to a different subset of the one or more optical signals.
[0186] In some embodiments, for each copy of a second subset of the one or more optical signals, different from the optical signals in the first subset of the one or more optical signals, the apparatus may further include a multiplication module configured to multiply the one or more optical signals of the second subset by one or more matrix element values using optical amplitude modulation.
[0187] In another aspect, a method includes encoding a set of multiple input values on corresponding optical signals; for each of at least two subsets of the one or more optical signals, using a corresponding set of one or more replication modules to split the subset of the one or more optical signals into two or more copies of the optical signals; for each of the at least two copies of a first subset of the one or more optical signals, using a corresponding multiplication module to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation, wherein at least one of the multiplication modules includes an optical amplitude modulator, the optical amplitude modulator includes an input port and two output ports, and provides a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by the signed matrix element value; and for the results of two or more multiplication modules, using a summation module to generate an electrical signal, the electrical signal representing a sum of the results of the two or more multiplication modules.
[0188] In another aspect, a method includes: encoding a set of input values representing elements of an input vector on corresponding optical signals; encoding a set of coefficients representing matrix elements as amplitude modulation levels of a set of optical amplitude modulators coupled to the optical signal, wherein at least one optical amplitude modulator including one input port and two output ports provides a pair of correlated optical signals from the two output ports such that a difference between the amplitudes of the correlated optical signals corresponds to a result of multiplying the input value by a signed matrix element value; and encoding a set of output values representing elements of an output vector on corresponding electrical signals, wherein at least one electrical signal is in the form of a current whose amplitude corresponds to the sum of the corresponding element of the input vector multiplied by the corresponding element of a row of the matrix.
[0189] Embodiments of the method may include one or more of the following features. For example, at least one optical signal may be provided by a first optical waveguide, and the first optical waveguide may be coupled to an optical splitter that sends a predetermined proportion of the power of the optical wave guided by the first optical waveguide to a second output optical waveguide and sends a remaining predetermined proportion of the power of the optical wave guided by the first optical waveguide to a third optical waveguide.
[0190] In another aspect, an apparatus includes: a plurality of optical waveguides encoding a set of input values representing elements of an input vector on corresponding optical signals carried by the optical waveguides; a set of optical amplitude modulators coupled to the optical signals encoding a set of coefficients representing matrix elements as amplitude modulation levels, wherein at least one optical amplitude modulator including one input port and two output ports provides a pair of related optical signals from the two output ports such that a difference between the amplitudes of the related optical signals corresponds to a result of multiplying the input value by a signed matrix element value; and a plurality of summing modules encoding a set of output values representing elements of an output vector on corresponding electrical signals, wherein at least one electrical signal is in the form of a current whose amplitude corresponds to the sum of the corresponding element of the input vector multiplied by the corresponding element of a row of the matrix.
[0191] In another aspect, a method for multiplying an input vector by a given matrix includes: encoding a set of input values representing elements of the input vector on corresponding optical signals of a set of optical signals; coupling a first set of one or more devices to a first set of one or more waveguides providing a first subset of the set of optical signals and generating a result of multiplying a first submatrix of the given matrix by the values encoded on the first subset of the set of optical signals; coupling a second set of one or more devices to a second set of one or more waveguides providing a second subset of the set of optical signals and generating a result of multiplying a second submatrix of the given matrix by the values encoded on the second subset of the set of optical signals; coupling a third set of one or more devices to a first set of one or more waveguides providing a copy of the first subset of the set of optical signals generated by the first optical splitter; a third group of one or more waveguides providing a copy of the second subset of the group of optical signals generated by the second optical separator, and producing a result of a third sub-matrix of the given matrix multiplied by the values encoded on the first subset of the group of optical signals; coupling a fourth group of one or more devices to a fourth group of one or more waveguides providing a copy of the second subset of the group of optical signals generated by the second optical separator, and producing a result of a fourth sub-matrix of the given matrix multiplied by the values encoded on the second subset of the group of optical signals; wherein the first, second, third and fourth sub-matrices connected together form the given matrix; and wherein at least one output value representing an element of an output vector is encoded on an electrical signal, the output vector corresponding to the input vector multiplied by the given matrix, the electrical signal being generated by a device in communication with the first group of one or more devices and the second group of one or more devices.
[0192] Embodiments of the method may include one or more of the following features: For example, each pair of the first group of one or more devices, the second group of one or more devices, the third group of one or more devices, and the fourth group of one or more devices may be mutually exclusive.
[0193] In another aspect, an apparatus includes: a first set of one or more apparatuses configured to receive a first set of optical signals and produce a result of multiplying a first matrix by a value encoded on the first set of optical signals; a second set of one or more apparatuses configured to receive a second set of optical signals and produce a result of multiplying the second matrix by a value encoded on the second set of optical signals; a third set of one or more apparatuses configured to receive a third set of optical signals and produce a result of multiplying the third matrix by the value encoded on the third set of optical signals; a fourth set of one or more apparatuses configured to receive a fourth set of optical signals and produce a result of multiplying a fourth matrix by the value encoded on the fourth set of optical signals; and A connection path is configured between two or more of a first group of one or more devices, a second group of one or more devices, a third group of one or more devices, or a fourth group of one or more devices, wherein a first configuration of the configurable connection path is configured to (1) provide a copy of the first group of optical signals as at least one of the second group of optical signals, the third group of optical signals, or the fourth group of optical signals, and (2) provide one or more signals from the first group of one or more devices and one or more signals from the second group of one or more devices to a summing module, the summing module being configured to generate an electrical signal representing a sum of values encoded on the signals received by the summing module.
[0194] In another aspect, an apparatus includes: a first set of one or more apparatuses configured to receive a first set of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the first set of optical signals; a second set of one or more apparatuses configured to receive a second set of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the second set of optical signals; a third set of one or more apparatuses configured to receive a third set of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the third set of optical signals; a fourth set of one or more apparatuses configured to receive a fourth set of optical signals and generate a result based on optical amplitude modulation of one or more optical signals of the fourth set of optical signals; and a configurable connection path between two or more of the first set of one or more apparatuses, the second set of one or more apparatuses, the third set of one or more apparatuses, or the fourth set of one or more apparatuses, wherein a first configuration of the configurable connection path is configured to (1) provide a copy of the first set of optical signals as a third set of optical signals, or (2) provide one or more signals from the first set of one or more apparatuses and one or more signals from the second set of one or more apparatuses to a summing module, the summing module configured to generate an electrical signal representing a sum of values encoded on the signals received by the summing module.
[0195] Embodiments of the apparatus may include one or more of the following features. For example, each pair of the first group of one or more apparatuses, the second group of one or more apparatuses, the third group of one or more apparatuses, and the fourth group of one or more apparatuses may be mutually exclusive.
[0196] In some embodiments, the first configuration of the configurable connection path is configured to (1) provide a copy of the first set of optical signals as a third set of optical signals, and (2) provide one or more signals from the first set of one or more devices and one or more signals from the second set of one or more devices to a summing module, the summing module being configured to generate an electrical signal representing a sum of values encoded on at least two different signals received by the summing module.
[0197] In some embodiments, a first configuration of the configurable connection paths may be configured to provide a copy of the first set of optical signals as a third set of optical signals, and a second configuration of the configurable connection paths may be configured to provide one or more signals from the first set of one or more devices and one or more signals from the second set of one or more devices to a summing module configured to generate an electrical signal representing a sum of values encoded on the signals received by the summing module.
[0198] In another aspect, an apparatus includes: a plurality of optical waveguides, wherein a set of a plurality of input values are encoded on respective optical signals carried by the optical waveguides; a plurality of replication modules, for each of at least two subsets of the one or more optical signals, the plurality of replication modules including a respective set of one or more replication modules configured to split the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, for each of at least two copies of a first subset of the one or more optical signals, the plurality of multiplication modules including a respective multiplication module configured to multiply the one or more optical signals of the first subset by one or more values using optical amplitude modulation; and one or more summation modules, for results of two or more of the multiplication modules, the one or more summation modules including a summation module configured to generate an electrical signal, the electrical signal representing a sum of the results of the two or more multiplication modules, wherein the result includes at least one result encoded on the electrical signal and the result is derived from a copy of the optical signal that propagated through no more than a single optical amplitude modulator before being converted to the electrical signal.
[0199] In another aspect, a system includes: a first unit configured to generate a plurality of modulator control signals; and a processor including: a light source configured to provide a plurality of light outputs; a plurality of light modulators coupled to the light source and the first unit, the plurality of light modulators configured to generate a light input vector by modulating the plurality of light outputs provided by the light source based on the plurality of modulator control signals, the light input vector including a plurality of light signals; and a matrix multiplication unit coupled to the plurality of light modulators and the first unit, the matrix multiplication unit configured to convert the light input vector into an analog output vector based on a plurality of weight control signals. The computing system also includes a second unit coupled to the matrix multiplication unit, and the second unit is configured to convert the analog output vector into a digital output vector; and a controller including an integrated circuit configured to perform the following operations: receiving an artificial neural network computing request, the artificial neural network computing request including an input data set, the input data set including a first digital input vector; receiving a first plurality of neural network weights; and generating, by the first unit, a first plurality of modulator control signals based on the first digital input vector, and generating a first plurality of weight control signals based on the first plurality of neural network weights.
[0200] Embodiments of the system may include one or more of the following features. For example, the first unit may include a digital-to-analog converter (DAC).
[0201] In some embodiments, the second unit may include an analog-to-digital converter (ADC).
[0202] In some embodiments, the system may include a storage unit configured to store a data set and a plurality of neural network weights.
[0203] In some embodiments, the integrated circuit of the controller may be further configured to perform operations including storing the input data set and the first plurality of neural network weights in the memory unit.
[0204] In some embodiments, the first unit may be configured to generate a plurality of weight control signals.
[0205] In some embodiments, the controller may include an application specific integrated circuit (ASIC), and receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a general purpose data processor.
[0206] In some embodiments, the first unit, the processing unit, the second unit, and the controller may be disposed on at least one of a multi-chip module or an integrated circuit. Receiving the artificial neural network computing request may include receiving the artificial neural network computing request from a second data processor, wherein the second data processor may be external to the multi-chip module or the integrated circuit, the second data processor may be coupled to the multi-chip module or the integrated circuit via a communication channel, and the processing unit may process data at a data rate at least one order of magnitude greater than a data rate of the communication channel.
[0207] In some embodiments, the first unit, the processing unit, the second unit, and the controller can be used in a photoelectric processing cycle that is repeated in multiple iterations, and the photoelectric processing cycle includes: (1) at least a first optical modulation operation based on at least one of a plurality of modulator control signals, and at least a second optical modulation operation based on at least one of the weight control signals, and (2) at least one of (a) an electrical summation operation or (b) an electrical storage operation.
[0208] In some embodiments, the photoelectric processing cycle may include an electrical storage operation, and the electrical storage operation is performed using a storage unit coupled to the controller, wherein the operation performed by the controller may further include storing an input data set and a first plurality of neural network weights in the storage unit.
[0209] In some embodiments, the photoelectric processing cycle may include an electrical summation operation, and the electrical summation operation may be performed using an electrical summation module within a matrix multiplication unit, wherein the electrical summation module may be configured to generate currents corresponding to elements of an analog output vector, wherein the elements of the analog output vector represent the sum of the corresponding elements of the optical input vector multiplied by the corresponding neural network weights.
[0210] In some embodiments, the optoelectronic processing cycle may include at least one signal path on which no more than one first optical modulation operation is performed in a single loop iteration based on at least one of a plurality of modulator control signals, and no more than one second optical modulation operation is performed in a single loop iteration based on at least one of the weight control signals.
[0211] In some embodiments, the first light modulation operation may be performed by one of a plurality of light modulators coupled to the light source of the light output and the matrix multiplication unit, and the second light modulation operation may be performed by a light modulator included in the matrix multiplication unit.
[0212] In some embodiments, an optoelectronic processing cycle may include at least one signal path on which no more than one electrical storage operation is performed in a single cycle iteration.
[0213] In some embodiments, the light source may include a laser unit configured to generate a plurality of light outputs.
[0214] In some embodiments, the matrix multiplication unit may include: an input waveguide array for receiving an optical input vector, and the optical input vector includes a first optical signal array; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation of the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array optically communicates with each output waveguide in the output waveguide array through the optical interference unit.
[0215] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change a separation ratio of the MZI; and a second phase shifter configured to shift a phase of an output of the MZI, wherein the first phase shifter and the second phase shifter are coupled to a plurality of weight control signals.
[0216] In some embodiments, the matrix multiplication unit may include: a plurality of replication modules, each of which corresponds to a subset of one or more optical signals of an optical input vector and is configured to divide the subset of one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, each of which corresponds to a subset of one or more optical signals and is configured to multiply the one or more optical signals of the subset by one or more matrix element values using optical amplitude modulation; and one or more summation modules, each of which is configured to generate an electrical signal representing the sum of the results of two or more of the multiplication modules.
[0217] In some embodiments, at least one multiplication module includes an optical amplitude modulator, which includes one input port and two output ports, and can provide a pair of correlated optical signals from the two output ports, such that the difference between the amplitudes of the correlated optical signals corresponds to the result of multiplying the input value by the signed matrix element value.
[0218] In some embodiments, the matrix multiplication unit may be configured to multiply the input vector by a matrix comprising one or more matrix element values.
[0219] In some embodiments, a set of multiple output values may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values in the set of multiple output values may represent elements of an output vector generated by multiplying the input vector by a matrix.
[0220] In some embodiments, the computing system may include a storage unit configured to store an input data set and a neural network weight, the second unit may include an analog-to-digital conversion (ADC) unit, and the operation may further include: obtaining a first plurality of digital outputs of an analog output vector corresponding to a matrix multiplication unit from the ADC unit, the first plurality of digital outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector to generate a first transformed digital output vector; and storing the first transformed digital output vector in the storage unit.
[0221] In some embodiments, the system has a first cycle period, the first cycle period is defined as the time elapsed between the step of storing an input data set and a first plurality of neural network weights in a memory unit and the step of storing a first converted digital output vector in the memory unit, and wherein the first cycle period is less than or equal to 1ns.
[0222] In some embodiments, the operations may further include: outputting an artificial neural network output generated based on the first converted digital output vector.
[0223] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, and the operations may further include generating, by the DAC unit, a second plurality of modulator control signals based on the first converted digital output vector.
[0224] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the artificial neural network calculation request may further include a second plurality of neural network weights, and wherein the operation may further include: based on obtaining the first plurality of digital outputs, generating a second plurality of weight control signals based on the second plurality of neural network weights through the DAC unit.
[0225] In some embodiments, the first plurality of neural network weights and the second plurality of neural network weights may correspond to different layers of the artificial neural network.
[0226] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, and the input data set may further include a second digital input vector. The operation may further include: generating a second plurality of modulator control signals based on the second digital input vector by the DAC unit; obtaining a second plurality of digital outputs corresponding to the output vector of the matrix multiplication unit from the ADC unit, the second plurality of digital outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; storing the second transformed digital output vector in a storage unit; and outputting an artificial neural network output generated based on the first transformed digital output vector and the second transformed digital output vector. The output vector of the matrix multiplication unit may be generated by a second optical input vector generated based on the second plurality of modulator control signals, the second optical input vector being transformed by the matrix multiplication unit based on the first-mentioned plurality of weight control signals.
[0227] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight, and the second unit may include an analog-to-digital conversion (ADC) unit. The system may further include: an analog nonlinear unit disposed between the matrix multiplication unit and the ADC unit, the analog nonlinear unit may be configured to receive a plurality of output voltages from the matrix multiplication unit, apply a nonlinear transfer function, and output a plurality of conversion output voltages to the ADC unit. The operations performed by the integrated circuit of the controller may further include: obtaining a first plurality of conversion digital output voltages corresponding to the plurality of conversion output voltages from the ADC unit, the first plurality of conversion digital output voltages forming a first conversion digital output vector; and storing the first conversion digital output vector in the storage unit.
[0228] In some embodiments, the integrated circuit of the controller may be configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.
[0229] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, and the second unit may include an analog-to-digital conversion (ADC) unit. The matrix multiplication unit may include: an optical matrix multiplication unit coupled to the plurality of optical modulators and the DAC unit, the optical matrix multiplication unit being configured to convert the optical input vector into the optical output vector based on the plurality of weight control signals; and a photodetection unit coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors.
[0230] In some embodiments, the system may further include: an analog storage unit, disposed between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store an analog voltage and output the stored analog voltage; and an analog nonlinear unit, disposed between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output voltages from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output voltages.
[0231] In some embodiments, the analog memory cell may include multiple capacitors.
[0232] In some embodiments, the analog storage unit may be configured to receive and store a plurality of conversion output voltages of the analog nonlinear unit, and output the stored plurality of conversion output voltages to a plurality of optical modulators. The operation may further include: storing a plurality of conversion output voltages of the analog nonlinear unit in the analog storage unit based on generating a first plurality of modulator control signals and a first plurality of weight control signals; outputting the stored conversion output voltages through the analog storage unit; obtaining a second plurality of conversion digital output voltages from the ADC unit, the second plurality of conversion digital output voltages forming a second conversion digital output vector; and storing the second conversion digital output vector in the storage unit.
[0233] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight, and the input data set requested by the artificial neural network calculation may include multiple digital input vectors. The light source may be configured to generate multiple wavelengths. The multiple optical modulators may include: an optical modulator group configured to generate multiple optical input vectors, each optical modulator group corresponds to one of the multiple wavelengths, and generates a corresponding optical input vector with a corresponding wavelength; and an optical multiplexer configured to combine the multiple optical input vectors into a combined optical input vector including multiple wavelengths. The photodetection unit may be further configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output voltages. The operation may include: obtaining multiple digital demultiplexed optical outputs from the ADC unit, the multiple digital demultiplexed optical outputs forming multiple first digital output vectors, wherein each of the multiple first digital output vectors corresponds to one of the multiple wavelengths; performing nonlinear conversion on each of the multiple first digital output vectors to generate multiple converted first digital output vectors; and storing the multiple converted first digital output vectors in the storage unit. Each of the multiple digital input vectors corresponds to one of the multiple optical input vectors.
[0234] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight, the second unit may include an analog-to-digital conversion (ADC) unit, and the artificial neural network calculation request may include a plurality of digital input vectors. The light source may be configured to generate a plurality of wavelengths. The plurality of optical modulators may include: an optical modulator group configured to generate a plurality of optical input vectors, each optical modulator group corresponding to one of the plurality of wavelengths and generating a corresponding optical input vector having a corresponding wavelength; and an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector including a plurality of wavelengths. The operation may include: obtaining a first plurality of digital optical outputs corresponding to the optical output vector from the ADC unit, the optical output vector including a plurality of wavelengths, the first plurality of digital optical outputs forming a first digital output vector; performing a nonlinear conversion on the first digital output vector to generate a first converted digital output vector; and storing the first converted digital output vector in the storage unit.
[0235] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the second unit may include an analog-to-digital conversion (ADC) unit, and the DAC unit may include: a 1-bit DAC subunit configured to generate a plurality of 1-bit modulator control signals. The resolution of the ADC unit may be 1 bit, and the resolution of the first digital input vector may be N bits. The operation may include: decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors by the 1-bit DAC subunit; obtaining a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals from the ADC unit; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear conversion on the constructed N-bit digital output vector to generate a converted N-bit digital output vector; and storing the converted N-bit digital output vector in a storage unit.
[0236] In some embodiments, the system may include a storage unit configured to store an input data set and a neural network weight. The storage unit may include: a digital input vector memory configured to store a digital input vector and including at least one SRAM; and a neural network weight memory configured to store a plurality of neural network weights and including at least one DRAM.
[0237] In some embodiments, the first unit may include a digital-to-analog conversion (DAC) unit, the digital-to-analog conversion unit including: a first DAC subunit configured to generate a plurality of modulator control signals; and a second DAC subunit configured to generate a plurality of weight control signals, wherein the first DAC subunit and the second DAC subunit are different.
[0238] In some embodiments, the light source may include: a laser source configured to generate light; and an optical power splitter configured to split the light generated by the laser source into a plurality of light outputs, wherein each of the plurality of light outputs has substantially the same power.
[0239] In some embodiments, the plurality of optical modulators include one of an MZI modulator, a ring resonance modulator, or an electro-absorption modulator.
[0240] In some embodiments, the photodetection unit may include: a plurality of photodetectors; and a plurality of amplifiers configured to convert photocurrents generated by the photodetectors into a plurality of output voltages.
[0241] In some embodiments, the integrated circuit may be an application specific integrated circuit.
[0242] In some embodiments, a system may include a plurality of optical waveguides coupled between an optical modulator and a matrix multiplication unit, wherein an optical input vector may include a set of multiple input values encoded on a corresponding optical signal carried by the optical waveguide, and each optical signal carried by one of the optical waveguides may include an optical wave having a common wavelength, the common wavelength being substantially the same for all optical signals.
[0243] In some embodiments, the replication module may include at least one replication module having an optical splitter that sends a predetermined proportion of the power of the light wave at the input port to the first output port and sends the remaining proportion of the power of the light wave at the input port to the second output port.
[0244] In some embodiments, the optical splitter may include a waveguide optical splitter that sends a predetermined proportion of the power of the optical wave guided by the input optical waveguide to the first output optical waveguide and sends the remaining proportion of the power of the optical wave guided by the input optical waveguide to the second output optical waveguide.
[0245] In some embodiments, a guided mode of the input optical waveguide can be adiabatically coupled to a guided mode of each of the first and second output optical waveguides.
[0246] In some embodiments, the light splitter may include a beam splitter including at least one surface that transmits a predetermined proportion of the power of the light wave at the input port and reflects a remaining proportion of the power of the light wave at the input port.
[0247] In some embodiments, at least one of the plurality of optical waveguides may include an optical fiber coupled to an optical coupler that couples a guided mode of the optical fiber to a free space propagating mode.
[0248] In some embodiments, the multiplication module may include at least one coherence-sensitive multiplication module configured to multiply one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation based on interference between optical waves, the optical waves having a coherence length that is at least as long as a propagation distance through the coherence-sensitive multiplication module.
[0249] In some embodiments, the coherence-sensitive multiplication module may include a Mach-Zehnder interferometer (MZI), which separates the light wave guided by the input optical waveguide into a first optical waveguide arm of the MZI and a second optical waveguide arm of the MZI, the first optical waveguide arm includes a phase shifter, the phase shifter produces a relative phase shift relative to the phase delay of the second optical waveguide arm, and the MZI can combine the light waves from the first optical waveguide arm and the second optical waveguide arm into at least one output optical waveguide.
[0250] In some embodiments, the MZI may combine light waves from the first optical waveguide arm and the second optical waveguide arm into each of the first output optical waveguide and the second output optical waveguide, the first photodetector may receive the light wave from the first output optical waveguide to generate a first photocurrent, the second photodetector may receive the light wave from the second output optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.
[0251] In some embodiments, the coherence-sensitive multiplication module may include one or more ring resonators including at least one ring resonator coupled to the first optical waveguide and at least one ring resonator coupled to the second optical waveguide.
[0252] In some embodiments, the first photodetector may receive a light wave from a first optical waveguide to generate a first photocurrent, the second photodetector may receive a light wave from a second optical waveguide to generate a second photocurrent, and the result of the coherence-sensitive multiplication module may include a difference between the first photocurrent and the second photocurrent.
[0253] In some embodiments, the multiplication module may include at least one coherence insensitive multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation based on energy absorption within the optical wave.
[0254] In some embodiments, the coherent insensitive multiplication module may include an electro-absorption modulator.
[0255] In some embodiments, the one or more summing modules may include at least one summing module having the following components: (1) two or more input conductors, each input conductor carrying an electrical signal in the form of an input current, the magnitude of the input current representing a corresponding result of a corresponding one of the multiplication modules, and (2) at least one output conductor, the output conductor carrying an electrical signal representing the sum of the corresponding results in the form of an output current, the output current being proportional to the sum of the input currents.
[0256] In some embodiments, the two or more input conductors and the output conductor may include wires that meet at one or more nodes between the wires, and the output current is substantially equal to the sum of the input currents.
[0257] In some embodiments, at least a first one of the input currents may be provided in the form of at least one photocurrent generated by at least one photodetector that receives an optical signal generated by a first one of the multiplication modules.
[0258] In some embodiments, the first input current may be provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding light signals generated by the first multiplication module.
[0259] In some embodiments, one of the copies of the first subset of the one or more optical signals may consist of a single optical signal, wherein one of the input values is encoded on the single optical signal.
[0260] In some embodiments, the multiplication module corresponding to the replica of the first subset may multiply the encoded input value by a single matrix element value.
[0261] In some embodiments, one of the copies of the first subset of the one or more optical signals may include more than one, and less than the number of all, optical signals on which the plurality of input values are encoded.
[0262] In some embodiments, the multiplication modules corresponding to the replicas of the first subset may multiply the encoded input values by different corresponding matrix element values.
[0263] In some embodiments, different multiplication modules corresponding to different respective copies of the first subset of one or more optical signals may be included in different devices that are in optical communication to transmit one of the copies of the first subset of one or more optical signals between the different devices.
[0264] In some embodiments, two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules may be disposed on a substrate of a common device.
[0265] In some embodiments, the apparatus performs a vector-matrix multiplication, where an input vector may be provided as a set of optical signals and an output vector may be provided as a set of electrical signals.
[0266] In some embodiments, the device may further include an accumulator that combines input electrical signals corresponding to the outputs of the multiplication module or the summation module, wherein the input electrical signal may be encoded using time domain coding, the time domain coding using switching amplitude modulation within each of multiple time slots, and the accumulator may generate an output electrical signal, the output electrical signal is encoded with more than two amplitude levels, the amplitude levels corresponding to different duty cycles of the time domain coding over multiple time slots.
[0267] In some embodiments, each of the two or more of the multiplication modules corresponds to a different subset of the one or more optical signals.
[0268] In some embodiments, for each copy of a second subset of the one or more optical signals, different from the optical signals in the first subset of the one or more optical signals, the apparatus may further include a multiplication module configured to multiply the one or more optical signals of the second subset by one or more matrix element values using optical amplitude modulation.
[0269] In another aspect, a system includes: a storage unit configured to store a data set and a plurality of neural network weights; and a driver unit configured to generate a plurality of modulator control signals. The system includes an optoelectronic processor, the optoelectronic processor including: a light source configured to provide a plurality of light outputs; a plurality of light modulators coupled to the light source and the driver unit, the plurality of light modulators configured to generate a light input vector by modulating the plurality of light outputs generated by the light source based on the plurality of modulator control signals; a matrix multiplication unit coupled to the plurality of light modulators and the driver unit, the matrix multiplication unit configured to convert the light input vector into an analog output vector based on the plurality of weight control signals; and a comparator unit coupled to the matrix multiplication unit and configured to convert the analog output vector into a plurality of digital 1-bit outputs. The system includes a controller, which includes an integrated circuit and is configured to perform the following operations: receive an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector with an N-bit resolution; store the input data set and the first plurality of neural network weights in a storage unit; decompose the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generate a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors through a driver unit; obtain a sequence of N digital 1-bit outputs corresponding to the sequence of N 1-bit modulator control signals from a comparator unit; construct an N-bit digital output vector from the sequence of N digital 1-bit outputs; perform a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and store the transformed N-bit digital output vector in the storage unit.
[0270] Embodiments of the system may include one or more of the following features. For example, receiving an artificial neural network computation request may include receiving an artificial neural network computation request from a general purpose computer.
[0271] In some embodiments, the driver unit may be configured to generate a plurality of weight control signals.
[0272] In some embodiments, the matrix multiplication unit may include: an optical matrix multiplication unit coupled to a plurality of optical modulators and a driver unit, the optical matrix multiplication unit being configured to convert an optical input vector into an optical output vector based on a plurality of weight control signals; and a photodetection unit coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vectors.
[0273] In some embodiments, the matrix multiplication unit may include: an input waveguide array for receiving an optical input vector; an optical interference unit in optical communication with the input waveguide array, for performing a linear transformation of the optical input vector into a second optical signal array; and an output waveguide array in optical communication with the optical interference unit, for guiding the second optical signal array, wherein at least one input waveguide in the input waveguide array is in optical communication with each output waveguide in the output waveguide array through the optical interference unit.
[0274] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs including: a first phase shifter configured to change a separation ratio of the MZI; and a second phase shifter configured to shift a phase of an output of the MZI, wherein the first phase shifter and the second phase shifter may be coupled to a plurality of weight control signals.
[0275] In some embodiments, the matrix multiplication unit may include: a plurality of replication modules, for each of at least two subsets of one or more optical signals of an optical input vector, the plurality of replication modules include a corresponding group of one or more replication modules configured to divide the subset of the one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, for each of at least two copies of a first subset of the one or more optical signals, the plurality of multiplication modules include a corresponding multiplication module configured to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation; and one or more summation modules, for the results of two or more multiplication modules, the one or more summation modules include a summation module configured to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules.
[0276] In some embodiments, at least one multiplication module may include an optical amplitude modulator, which includes one input port and two output ports, and can provide a pair of correlated optical signals from the two output ports such that the difference between the amplitudes of the correlated optical signals corresponds to the result of multiplying the input value by the signed matrix element value.
[0277] In some embodiments, the matrix multiplication unit may be configured to multiply the input vector by a matrix comprising one or more matrix element values.
[0278] In some embodiments, a set of multiple output values may be encoded on corresponding electrical signals generated by one or more summing modules, and the output values in the set of multiple output values may represent elements of an output vector generated by multiplying the input vector by a matrix.
[0279] In another aspect, a method is provided for performing artificial neural network calculations in a system having a matrix multiplication unit, the matrix multiplication unit being configured to convert an optical input vector into an analog output vector based on a plurality of weight control signals. The method includes: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector; storing the input data set and the first plurality of neural network weights in a storage unit; generating a first plurality of modulator control signals based on the first digital input vector, and generating a first plurality of weight control signals based on the first plurality of neural network weights; obtaining a first plurality of digital outputs corresponding to an output vector of the matrix multiplication unit, the first plurality of digital outputs forming a first digital output vector; performing a nonlinear transformation on the first digital output vector by a controller to generate a first transformed digital output vector; storing the first transformed digital output vector in the storage unit; and outputting the artificial neural network output generated based on the first transformed digital output vector by the controller.
[0280] Embodiments of the method may include one or more of the following features.For example, receiving the artificial neural network computation request may include receiving the artificial neural network computation request from a computer via a communication channel.
[0281] In some embodiments, generating the first plurality of modulator control signals may include generating the first plurality of modulator control signals via a digital to analog conversion (DAC) unit.
[0282] In some embodiments, obtaining the first plurality of digital outputs may include obtaining the first plurality of digital outputs from an analog-to-digital conversion (ADC) unit.
[0283] In some embodiments, the method may include: applying a first plurality of modulator control signals to a plurality of optical modulators coupled to a light source and a DAC unit; and generating a light input vector using the plurality of optical modulators by modulating a plurality of light outputs generated by a laser unit based on the plurality of modulator control signals.
[0284] In some embodiments, a matrix multiplication unit may be coupled to the plurality of optical modulators and the DAC unit, and the method may include converting the optical input vector into an analog output vector based on the plurality of weight control signals using the matrix multiplication unit.
[0285] In some embodiments, the ADC unit may be coupled to the matrix multiplication unit, and the method may include converting the analog output vector into a first plurality of digital outputs using the ADC unit.
[0286] In some embodiments, the matrix multiplication unit may include an optical matrix multiplication unit coupled to a plurality of optical modulators and a DAC unit. Converting the optical input vector into an analog output vector may include using the optical matrix multiplication unit to convert the optical input vector into an optical output vector based on a plurality of weight control signals. The method may include: using a photodetection unit coupled to the optical matrix multiplication unit to generate a plurality of output voltages corresponding to the optical output vectors.
[0287] In some embodiments, the method may include: receiving an optical input vector at an input waveguide array; performing a linear transformation of the optical input vector into a second optical signal array using an optical interference unit in optical communication with the input waveguide array; and guiding the second optical signal array using an output waveguide array in optical communication with the optical interference unit, wherein at least one input waveguide in the input waveguide array optically communicates with each output waveguide in the output waveguide array through the optical interference unit.
[0288] In some embodiments, the optical interference unit may include: a plurality of interconnected Mach-Zehnder interferometers (MZIs), each of the plurality of interconnected MZIs may include a first phase shifter and a second phase shifter, and the first phase shifter and the second phase shifter may be coupled to a plurality of weight control signals. The method may include: using the first phase shifter to change the separation ratio of the MZI, and using the second phase shifter to shift the phase of one output of the MZI.
[0289] In some embodiments, the method may include: for each of at least two subsets of one or more optical signals of an optical input vector, using a corresponding set of one or more replication modules to divide the subset of the one or more optical signals into two or more copies of the optical signals; for each of the at least two copies of a first subset of the one or more optical signals, using a corresponding multiplication module to multiply the one or more optical signals of the first subset by one or more matrix element values using optical amplitude modulation; and for the results of the two or more multiplication modules, using a summation module to generate an electrical signal, the electrical signal representing the sum of the results of the two or more multiplication modules.
[0290] In some embodiments, at least one multiplication module may include an optical amplitude modulator, which includes one input port and two output ports, and can provide a pair of correlated optical signals from the two output ports such that the difference between the amplitudes of the correlated optical signals corresponds to the result of multiplying the input value by the signed matrix element value.
[0291] In some embodiments, the method may include multiplying the input vector by a matrix including one or more matrix element values using a matrix multiplication unit.
[0292] In some embodiments, the method may include encoding a set of multiple output values on corresponding electrical signals generated by one or more summing modules, and using output values from the set of multiple output values to represent elements of an output vector, the output vector being generated by multiplying the input vector by a matrix.
[0293] In another aspect, a method includes providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; optoelectronically converting the optical input vector into an analog output vector based on matrix multiplication; and electronically applying a nonlinear transformation to the analog output vector to provide output information in an electronic format.
[0294] Embodiments of the method may include one or more of the following features. For example, the method may further include repeating the electro-optical conversion, the opto-electrical conversion, and the electrically applied non-linear conversion for new electronic input information corresponding to the output information provided in the electronic format.
[0295] In some embodiments, the matrix multiplication for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion may be the same and may correspond to the same layer of the artificial neural network.
[0296] In some embodiments, the matrix multiplication for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion may be different and may correspond to different layers of the artificial neural network.
[0297] In some embodiments, the method may further include: for different parts of the electronic input information, repeating electro-optical conversion, photoelectric conversion and electrically applied nonlinear conversion, wherein the matrix multiplication used for the initial photoelectric conversion and the matrix multiplication for the repeated photoelectric conversion are the same and correspond to the first layer of the artificial neural network.
[0298] In some embodiments, the method may further include: providing intermediate information in an electronic format based on electronic output information for multiple parts of the electronic input information generated by the first layer of the artificial neural network; and repeating electro-optical conversion, optoelectronic conversion, and electrically applied nonlinear conversion for each of the different parts of the electronic intermediate information, wherein the matrix multiplication used for the initial optoelectronic conversion and the matrix multiplication of the repeated optoelectronic conversion associated with the different parts of the electronic intermediate information are the same and correspond to the second layer of the artificial neural network.
[0299] In another aspect, a system is provided for performing artificial neural network calculations. The system includes: a first unit configured to generate a plurality of vector control signals and to generate a plurality of weight control signals; a second unit configured to provide an optical input vector based on the plurality of vector control signals; and a matrix multiplication unit coupled to the second unit and the first unit, the matrix multiplication unit configured to convert the optical input vector into an output vector based on the plurality of weight control signals. The system includes a controller, the controller including an integrated circuit, configured to perform the following operations: receiving an artificial neural network calculation request including an input data set and a first plurality of neural network weights, wherein the input data set includes a first digital input vector; and generating, by the first unit, a first plurality of vector control signals based on the first digital input vector, and a first plurality of weight control signals based on the first plurality of neural network weights; wherein the first unit, the second unit, the matrix multiplication unit, and the controller are used for a photoelectric processing cycle repeated in a plurality of iterations, and the photoelectric processing cycle includes: (1) at least two optical modulation operations, and (2) at least one of (a) an electrical summation operation or (b) an electrical storage operation.
[0300] In another aspect, a method is provided for performing artificial neural network computations. The method includes: providing input information in an electronic format; converting at least a portion of the electronic input information into an optical input vector; and converting the optical input vector into an output vector based on a matrix multiplication using a set of neural network weights. The providing and converting are performed in an optical processing cycle, the optical processing cycle is repeated in multiple iterations using different corresponding sets of neural network weights and different corresponding input information, and the optical processing cycle includes: (1) at least two optical modulation operations, and (2) at least one of (a) an electrical summing operation or (b) an electrical storage operation.
[0301] In another aspect, an optoelectronic computing system is provided. The optoelectronic computing system includes: a first unit configured to generate a plurality of modulator control signals; and a processing unit. The processing unit includes: a light source configured to provide a plurality of light outputs; a plurality of light modulators coupled to the light source and the first unit, the plurality of light modulators configured to generate a light input vector by modulating the plurality of light outputs provided by the light source based on the plurality of modulator control signals, the light input vector including a plurality of light signals; and a matrix multiplication unit coupled to the plurality of light modulators and the first unit, the matrix multiplication unit configured to convert the light input vector into an analog output vector based on a plurality of weight control signals. The optoelectronic computing system includes: a second unit coupled to the matrix multiplication unit, and the second unit configured to convert the analog output vector into a digital output vector; and a controller including an integrated circuit configured to perform operations. The operation includes: receiving an artificial neural network calculation request, the artificial neural network calculation request including an input data set, the input data set including a first digital input vector; receiving a first plurality of neural network weights; and generating, by the first unit, a first plurality of modulator control signals based on the first digital input vector and a first plurality of weight control signals based on the first plurality of neural network weights. The matrix multiplication unit includes: a plurality of replication modules, wherein each of the replication modules corresponds to a subset of one or more optical signals of the optical input vector and is configured to divide the subset of one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, wherein each of the multiplication modules corresponds to a subset of one or more optical signals and is configured to multiply the one or more optical signals of the subset by one or more matrix element values using optical amplitude modulation; and one or more summation modules, wherein each summation module is configured to generate an electrical signal, the electrical signal representing the sum of the results of two or more of the multiplication modules.
[0302] The details of one or more embodiments of the subject matter described in this disclosure are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the disclosure will become apparent from the description and drawings.
[0303] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. In the event of a conflict with a patent application or patent application publication incorporated by reference, the present disclosure, including definitions, controls. BRIEF DESCRIPTION OF THE DRAWINGS
[0304] The present disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to scale. On the contrary, the sizes of the various features are arbitrarily expanded or reduced for clarity.
[0305] Figure 1A is a schematic diagram of an example of an artificial neural network (ANN) system.
[0306] Figure 1B is a schematic diagram of an example of an optical matrix multiplication unit.
[0307] Figure 1C and Figure 1D is a schematic diagram of an example configuration of interconnected Mach Zehnder interferometers (MZIs).
[0308] Figure 1E is a schematic diagram of an example of an MZI.
[0309] Figure 1F is a schematic diagram of an example of a wavelength division multiplexed ANN (WDM) computing system.
[0310] Figure 2A is a flow chart showing an example of a method for performing ANN calculations.
[0311] Figure 2B Yes Display Figure 2A FIG. 1 is a diagram of one aspect of a method.
[0312] Figure 3A and Figure 3B is a schematic diagram of an example of an ANN computing system.
[0313] Figure 4A is a schematic diagram of an example of an ANN computing system with 1-bit internal resolution.
[0314] Figure 4B yes Figure 4A A mathematical representation of the operation of an ANN computing system.
[0315] Figure 5 is a schematic diagram of an example of an artificial neural network (ANN) computing system.
[0316] Figure 6 is a schematic diagram of an example of an optical matrix multiplication unit.
[0317] Figure 7 is a schematic diagram of an example of an artificial neural network (ANN) computing system.
[0318] Figure 8is a diagram of an example of an optical matrix multiplication unit.
[0319] Fig. 9 is a schematic diagram of an example of an artificial neural network (ANN) computing system.
[0320] Fig.10 is a diagram of an example of an optical matrix multiplication unit.
[0321] Fig.11 is a diagram of an example of a compact matrix multiplier unit.
[0322] Fig. 12A A diagram comparing photonic matrix multiplier units is shown.
[0323] Fig. 12B is a diagram of a compact interconnect interferometer.
[0324] Fig.13 is a diagram of a compact matrix multiplier unit.
[0325] Fig.14 This is a graph of an optical generative adversarial network.
[0326] Fig.15 is a diagram of a Mach-Zehnder interferometer.
[0327] Fig.16 , Fig.17A as well as Fig. 17B is a diagram of a photonic circuit.
[0328] Fig.18 is a schematic diagram of an example of an optoelectronic computing system.
[0329] Fig.19A and Fig.19B is a schematic diagram of an example system configuration.
[0330] Fig. 20A is a schematic diagram of an example of a symmetric differential configuration.
[0331] Fig. 20B and Fig. 20C is a circuit diagram of an example of a system module.
[0332] Fig.21A is a schematic diagram of an example of a symmetric differential configuration.
[0333] Fig.21B is a schematic diagram of an example of a system configuration.
[0334] Fig.22A is a schematic diagram of an example optical amplitude modulator.
[0335] FIG. 22B to FIG. 22D is a schematic diagram of an example of an optical amplitude modulator using optical detection in a symmetric differential configuration.
[0336] FIG. 23A to FIG. 23C is an optoelectronic circuit diagram of an example system configuration.
[0337] FIG. 24A to FIG. 24E is a schematic diagram of an example computing system using multiple optoelectronic systems.
[0338] Fig.25 is a flow chart showing an example of a method for performing ANN calculations.
[0339] Fig.26 and Fig. 27 is a schematic diagram of an example of an ANN computing system.
[0340] Fig.28 is a schematic diagram of an example of a neural network computing system using a passive 2D optical matrix multiplication unit.
[0341] Fig.29 is a schematic diagram of an example of a neural network computing system using a passive 3D optical matrix multiplication unit.
[0342] Fig.30 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses a passive 2D optical matrix multiplication unit.
[0343] Fig.31 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses a passive 3D optical matrix multiplication unit.
[0344] Fig.32A is a schematic diagram of an example of an artificial neural network (ANN) computing system.
[0345] Fig.32B is a schematic diagram of an example of an optoelectronic matrix multiplication unit.
[0346] Fig.33 is a flow chart showing an example of a method for performing ANN calculations using an optoelectronic processor.
[0347] Fig.34 Yes Display Fig.33 FIG. 1 is a diagram of one aspect of a method.
[0348] Fig.35Ais a schematic diagram of an example of a wavelength division multiplexing ANN computing system using an optoelectronic processor.
[0349] Fig.35B and Fig.35C is a schematic diagram of an example of a wavelength division multiplexing optoelectronic matrix multiplication unit.
[0350] Fig.36 and Fig.37 is a schematic diagram of an example of an ANN computing system using an optoelectronic matrix multiplication unit.
[0351] Fig.38 is a schematic diagram of an example of an artificial neural network computing system with 1-bit internal resolution, where the system uses an optoelectronic matrix multiplication unit.
[0352] Fig.39A is a schematic diagram of an example of a Mach-Zehnder modulator.
[0353] Fig.39B Yes Display Fig.39A A graph of the intensity-voltage curve of a Mach-Zehnder modulator.
[0354] Fig.40 is a schematic diagram of a homodyne detector.
[0355] Fig.41 is a schematic diagram of a computing system including optical fibers, each of which carries signals having multiple wavelengths.
[0356] Like reference numbers and names in the various drawings refer to like components. DETAILED DESCRIPTION
[0357] Figure 1A A schematic diagram of an example of an artificial neural network (ANN) computing system 100 is shown. The system 100 includes a controller 110, a storage unit 120, a digital-to-analog conversion (DAC) unit 130, an optical processor 140, and an analog-to-digital conversion (ADC) unit 160. The controller 110 is coupled to a computer 102, the storage unit 120, the DAC unit 130, and the ADC unit 160. The controller 110 includes an integrated circuit that is configured to control the operation of the ANN computing system 100 to perform ANN computing.
[0358] The integrated circuit of controller 110 may be an application specific integrated circuit that is specifically configured to perform the steps of the ANN computation process. For example, the integrated circuit may implement microcode or firmware specific to performing the ANN computation process. As such, controller 110 may have a reduced instruction set relative to a general purpose processor used in a conventional computer (e.g., computer 102). In some embodiments, the integrated circuit of controller 110 may include two or more circuits that are configured to perform different steps of the ANN computation process.
[0359] In an example operation of the ANN computing system 100, the computer 102 may issue an artificial neural network computing request to the ANN computing system 100. The ANN computing request may include a neural network weight defining the ANN, and an input data set processed by the provided ANN. The controller 110 receives the ANN computing request and stores the input data set and the neural network weight in the storage unit 120.
[0360] The input data set may correspond to various digital information to be processed by the ANN. Examples of input data sets include image files, audio files, LiDAR point clouds, and GPS coordinate sequences, and the operation of the ANN computing system 100 will be described based on receiving image files as input data sets. Generally speaking, the size of the input data set can vary greatly, from hundreds of data points to millions of data points or more. For example, a digital image file with a resolution of 1 megapixel has about one million pixels, and each of the one million pixels can be a data point processed by the ANN. Due to the large number of data points in a typical input data set, the input data set is usually divided into multiple digital input vectors of smaller size to be processed separately by the optical processor 140. As an example, for a grayscale digital image, the elements of the digital input vector may be 8-bit values representing image intensity, and the digital input vector may have a length ranging from tens of elements (e.g., 32 elements, 64 elements) to hundreds of elements (e.g., 256 elements, 512 elements). In general, an input data set of arbitrary size can be divided into digital input vectors of a size suitable for processing by the optical processor 140. In the case where the number of elements of the input data set is not divisible by the length of the digital input vector, zero padding can be used to fill the data set so that it is divisible by the length of the digital input vector. The processed outputs of the respective digital input vectors can be processed to reconstruct the complete output, which is the result of processing the input data set by the ANN. In some embodiments, block matrix multiplication techniques can be used to implement the division of the input data set into multiple input vectors and subsequent vector-level processing.
[0361] Neural network weights are a set of values that define the connectivity of the artificial neurons of the ANN, including the relative importance or weights of those connections. The ANN may include one or more hidden layers with corresponding sets of nodes. In the case of an ANN with a single hidden layer, the ANN may be defined by two sets of neural network weights, one set corresponding to the connectivity between the input nodes and the nodes of the hidden layer, and the second set corresponding to the connectivity between the hidden layer and the output nodes. Each set of neural network weights describing the connectivity corresponds to a matrix implemented by the optical processor 140. For an ANN with two or more hidden layers, additional sets of neural network weights are required to define the connectivity between the additional hidden layers. As such, in general, the neural network weights included in the ANN calculation request may include multiple sets of neural network weights representing the connectivity between the various layers of the ANN.
[0362] Since the input data set to be processed is usually divided into multiple smaller digital input vectors for separate processing, the input data set is usually stored in digital storage. However, the speed of storage operations between the storage and processor of the computer 102 is significantly slower than the rate at which the ANN computing system 100 can perform ANN calculations. For example, the ANN computing system 100 can perform tens to hundreds of ANN calculations during a typical memory read period of the computer 102. As a result, during the process of processing an ANN calculation request, if the ANN calculation of the ANN computing system 100 involves multiple data transfers between the system 100 and the computer 102, the rate of ANN calculations that can be performed by the ANN computing system 100 may be limited to below its full processing rate. For example, if the computer 102 is to access the input data set from its own storage and provide the digital input vector to the controller 110 when requested, the operation of the ANN computing system 100 may be greatly slowed down by the time required for the series of data transfers required between the computer 102 and the controller 110. It is worth noting that the memory access latency of the computer 102 is generally non-deterministic, which further complicates and reduces the speed at which the digital input vector can be provided to the ANN computing system 100. In addition, processor cycles of the computer 102 may be wasted in managing data transfers between the computer 102 and the ANN computing system 100.
[0363] In contrast, in some embodiments, the ANN computing system 100 stores the entire input data set in a storage unit 120 that is part of and dedicated to the ANN computing system 100. The dedicated storage unit 120 allows transactions between the storage unit 120 and the controller 110 and is particularly suitable for allowing smooth and uninterrupted data flow between the storage unit 120 and the controller 110. This uninterrupted data flow can significantly improve the overall throughput of the ANN computing system 100 by allowing the optical processor 140 to perform matrix multiplications at its full processing rate without being limited by the slow memory operations of traditional computers (such as computer 102). In addition, because all of the data required in performing the ANN calculations is provided to the ANN computing system 100 by the computer 102 in a single transaction, the ANN computing system 100 can perform its ANN calculations in a unique manner independent of the computer 102. This unique operation of the ANN computing system 100 reduces the computational burden on the computer 102 and eliminates external dependencies in the operation of the ANN computing system 100 , improving the performance of the system 100 and the computer 102 .
[0364] The internal operation of the ANN computing system 100 will now be described. The optical processor 140 includes a laser unit 142, a modulator array 144, a detection unit 146, and an optical matrix multiplication (OMM) unit 150. The optical processor 140 operates by encoding a digital input vector of length N onto an optical input vector of length N and propagating the optical input vector through the OMM unit 150. The OMM unit 150 receives an optical input vector of length N and performs an N×N matrix multiplication on the received optical input vector in the optical domain. The N×N matrix multiplication performed by the OMM unit 150 is determined by the internal configuration of the OMM unit 150. The internal configuration of the OMM unit 150 can be controlled by an electrical signal, such as the electrical signal generated by the DAC unit 130.
[0365] The OMM unit 150 may be implemented in various ways. Figure 1B A schematic diagram of an example of an OMM cell 150 is shown. The OMM cell 150 may include an array of input waveguides 152 to receive an optical input vector; an optical interference cell 154 in optical communication with the array of input waveguides 152; and an array of output waveguides 156 in optical communication with the optical interference cell 154. The optical interference cell 154 linearly converts the optical input vector into a second optical signal array. The array of output waveguides 156 guides the second optical signal array output by the optical interference cell 154. At least one input waveguide in the array of input waveguides 152 is in optical communication with each output waveguide in the array of output waveguides 156 through the optical interference cell 154. For example, for an optical input vector of length N, the OMM cell 150 may include N input waveguides 152 and N output waveguides 156.
[0366] The optical interference unit may include a plurality of interconnected Mach Zehnder interferometers (MZIs). Figure 1C and Figure 1D Schematic diagrams of example configurations 157 and 158 of interconnected MZIs are shown. The MZIs may be interconnected in various ways (eg, in configurations 157 or 158) to achieve linear transformation of an optical input vector received through the array of input waveguides 152.
[0367] Figure 1EA schematic diagram of an example of an MZI 170 is shown. The MZI 170 includes a first input waveguide 171, a second input waveguide 172, a first output waveguide 178, and a second output waveguide 179. In addition, each MZI 170 in a plurality of interconnected MZIs includes a first phase shifter 174 configured to change a splitting ratio of the MZI 170; and a second phase shifter 176 configured to shift a phase of an output of the MZI 170, such as light exiting the MZI 170 through the second output waveguide 179. The first phase shifter 174 and the second phase shifter 176 of the MZI 170 are coupled to a plurality of weight control signals generated by the DAC unit 130. The first phase shifter 174 and the second phase shifter 176 are examples of reconfigurable components of the OMM unit 150. Examples of reconfigurable components include thermo-optic phase shifters or electro-optic phase shifters. Thermo-optic phase shifters operate by heating the waveguide to change the refractive index of the waveguide and cladding materials, which translates into a change in phase. Electro-optic phase shifters operate by applying an electric field (e.g., lithium niobate (LiNbO3), reverse biased PN junction) or current (e.g., forward biased PIN junction), which changes the refractive index of the waveguide material. By changing the weight control signal, the phase delay of the first phase shifter 174 and the second phase shifter 176 of each interconnected MZI 170 can be changed, which reconfigures the optical interference unit 154 of the OMM unit 150 to achieve a specific matrix multiplication determined by the phase delay set across the optical interference unit 154. Additional embodiments of the OMM unit 150 and the optical interference unit are disclosed in U.S. Patent Publication No. US 2017 / 0351293A1, entitled “APPARATUS AND METHODS FOR OPTICAL NEURAL NETWORK,” which is incorporated herein by reference in its entirety.
[0368] An optical input vector is generated by the laser unit 142 and the modulator array 144. The optical input vector of length N has N independent optical signals, each of which has an intensity corresponding to the value of a corresponding element of the digital input vector of length N. As an example, the laser unit 142 can generate N optical outputs. The N optical outputs have the same wavelength and are optically coherent. The optical coherence of the optical outputs allows the optical outputs to optically interfere with each other, which is a property utilized by the OMM unit 150 (e.g., in the operation of the MZI). In addition, the optical outputs of the laser unit 142 can be substantially identical to each other. For example, the N optical outputs can be substantially uniform in their intensities (e.g., within 5%, within 3%, within 1%, within 0.5%, within 0.1%, or within 0.01%) and their relative phases (e.g., within 10 degrees, within 5 degrees, within 3 degrees, within 1 degree, within 0.1 degrees). The uniformity of the light output can improve the faithfulness of the light input vector to the digital input vector, thereby improving the overall accuracy of the light processor 140. In some embodiments, the light output of the laser unit 142 can have an optical power of 0.1 mW to 50 mW per output, a wavelength in the near infrared range (e.g., between 900 nm and 1600 nm), and a line width of less than 1 nm. The light output of the laser unit 142 can be a single transverse-mode light output.
[0369] In some embodiments, the laser unit 142 includes a single laser source and an optical powersplitter. The single laser source is configured to generate laser light. The optical powersplitter is configured to split the light generated by the laser source into N light outputs having substantially the same intensity and phase. By splitting the single laser output into multiple outputs, optical coherence of the multiple light outputs can be achieved. For example, the single laser source can be a semiconductor laser diode, a vertical-cavity surface-emitting laser (VCSEL), a distributed feedback (DFB) laser, or a distributed Bragg reflector (DBR) laser. For example, the optical powersplitter can be a 1:N multimode interference (MMI) splitter, a multi-stage splitter including multiple 1:2 MMI splitters or directional couplers, or a star coupler. In some other embodiments, a master-slave laser configuration may be used, in which a slave laser is injection locked by a master laser to have a stable phase relationship to the master laser.
[0370] The optical output of the laser unit 142 is coupled to the modulator array 144. The modulator array 144 is configured to receive the optical input from the laser unit 142 and modulate the intensity of the received optical input based on the modulator control signal (which is an electrical signal). Examples of modulators include Mach-Zehnder interferometer (MZI) modulators, ring resonator modulators, and electro-absorption modulators. The modulator array 144 has N modulators, each modulator receiving one of the N optical outputs of the laser unit 142. The modulator receives a control signal corresponding to an element of a digital input vector and modulates the intensity of the light. The control signal may be generated by the DAC unit 130.
[0371] The DAC unit 130 is configured to generate a plurality of modulator control signals and generate a plurality of weight control signals under the control of the controller 110. For example, the DAC unit 130 receives a first DAC control signal from the controller 110, the first DAC control signal corresponding to a digital input vector to be processed by the optical processor 140. The DAC unit 130 generates a modulator control signal based on the first DAC control signal, the modulator control signal being an analog signal suitable for driving the modulator array 144. For example, the analog signal may be a voltage or a current, depending on the technology and design of the modulators of the array 144. The voltage may have an amplitude ranging from ±0.1V to ±10V, and the current may have an amplitude ranging from 100μA to 100mA. In some embodiments, the DAC unit 130 may include a modulator driver configured to buffer, amplify or adjust the analog signal so that the modulators of the array 144 can be fully driven. For example, some types of modulators can be driven with differential control signals. In this case, the modulator driver may be a differential driver that generates a differential electrical output based on a single-ended input signal. As another example, certain types of modulators may have a 3 dB bandwidth that is less than the desired processing rate of the optical processor 140. In this case, the modulator driver may include a pre-emphasis circuit or other bandwidth enhancement circuit designed to extend the operating bandwidth of the modulator.
[0372] In some cases, the modulators of array 144 may have a nonlinear transfer function. For example, an MZI optical modulator may have a nonlinear relationship (e.g., a sinusoidal dependence) between an applied control voltage and its transmission. In this case, the first DAC control signal may be adjusted or compensated based on the nonlinear transfer function of the modulator so that a linear relationship between the digital input vector and the generated optical input vector may be maintained. Maintaining this linearity is generally important for ensuring that the input to the OMM unit 150 is an accurate representation of the digital input vector. In some embodiments, compensation of the first DAC control signal may be performed by the controller 110 through a lookup table that maps the value of the digital input vector to the value to be output by the DAC unit 130 so that the resulting modulated optical signal is linearly proportional to the elements of the digital input vector. The lookup table may be generated by characterizing the nonlinear transfer function of the modulator and calculating the inverse function of the nonlinear transfer function.
[0373] In some embodiments, nonlinearities of the modulator and the resulting nonlinearities in the generated optical input vector can be compensated by an ANN computation algorithm.
[0374] The optical input vector generated by the modulator array 144 is input to the OMM unit 150. The optical input vector may be N spatially separated optical signals, each of which has an optical power corresponding to an element of the digital input vector. For example, the optical power of the optical signal is typically in the range of 1 μW to 10 mW. The OMM unit 150 receives the optical input vector and performs an N×N matrix multiplication based on its internal configuration. The internal configuration is controlled by an electrical signal generated by the DAC unit 130. For example, the DAC unit 130 receives a second DAC control signal from the controller 110, the second DAC control signal corresponding to the neural network weight to be implemented by the OMM unit 150. The DAC unit 130 generates a weight control signal based on the second DAC control signal, the weight control signal being an analog signal suitable for controlling a reconfigurable component within the OMM unit 150. For example, the analog signal may be a voltage or a current, depending on the type of reconfigurable component of the OMM unit 150. The voltage may have an amplitude ranging from 0.1 V to 10 V, and the current may have an amplitude ranging from 100 μA to 10 mA.
[0375] The modulator array 144 can operate at a modulation rate that is different from the reconfiguration rate of the reconfigurable OMM cell 150. The optical input vector generated by the modulator array 144 propagates through the OMM cell at a rough proportion of the speed of light (e.g., 80%, 50%, or 25% of the speed of light), depending on the optical properties of the OMM cell 150 (e.g., effective index). For a typical OMM cell 150, the propagation time of the optical input vector is in the range of 1 to tens of picoseconds, which corresponds to a processing rate of tens to hundreds of GHz. As such, the rate at which the optical processor 140 can perform matrix multiplication operations is partially limited by the rate at which the optical input vector can be generated. Modulators with bandwidths of tens of GHz are readily available, and modulators with bandwidths in excess of 100 GHz are being developed. As such, for example, the modulation rate of the modulator array 144 can be in the range of 5 GHz, 8 GHz, or tens to hundreds of GHz. To maintain operation of the modulator array 144 at such a modulation rate, the integrated circuit of the controller 110 may be configured to output control signals for the DAC unit 130 at a rate greater than or equal to, for example, 5 GHz, 8 GHz, 10 GHz, 20 GHz, 25 GHz, 50 GHz, or 100 GHz.
[0376] Depending on the type of reconfigurable component implemented by the OMM cell 150, the reconfiguration rate of the OMM cell 150 can be significantly slower than the modulation rate. For example, the reconfigurable component of the OMM cell 150 can be a thermo-optical type, which uses a micro-heater to adjust the temperature of the optical waveguide of the OMM cell 150, which in turn affects the phase of the optical signal within the OMM cell 150 and causes a matrix multiplication. Due to the thermal time constant associated with the heating and cooling of the structure, the reconfiguration rate can be limited to a few 100 kHz to a few 10 MHz. As such, the modulator control signals used to control the modulator array 144 and the weight control signals used to reconfigure the OMM cell 150 may have significantly different speed requirements. In addition, the electrical characteristics of the modulator array 144 can be significantly different from the electrical characteristics of the reconfigurable components of the OMM cell 150.
[0377] In order to accommodate the different characteristics of the modulator control signal and the weight control signal, in some embodiments, the DAC unit 130 may include a first DAC subunit 132 and a second DAC subunit 134. The first DAC subunit 132 may be specifically configured to generate the modulator control signal, and the second DAC subunit 134 may be specifically configured to generate the weight control signal. For example, the modulation rate of the modulator array 144 may be 25 GHz, and the first DAC subunit 132 may have a per-channel output update rate of 25 giga-samples per second (GSPS) and a resolution of 8 bits or higher. The reconfiguration rate of the OMM unit 150 may be 1 MHz, and the second DAC subunit 134 may have an output update rate of 1 mega-samples per second (MSPS) and a resolution of 10 bits. Implementing separate first and second DAC subunits 132 and 134 allows the DAC subunits to be independently optimized for the respective signals, which can reduce the overall power consumption, complexity, cost, or a combination thereof of the DAC unit 130. It is worth noting that although the first DAC subunit 132 and the second DAC subunit 134 are described as subcomponents of the DAC unit 130, generally speaking, the first DAC subunit 132 and the second DAC subunit 134 can be integrated on a common chip or can be implemented as separate chips.
[0378] Based on the different characteristics of the first DAC subunit 132 and the second DAC subunit 134, in some embodiments, the storage unit 120 may include a first storage subunit and a second storage subunit. The first storage subunit may be a memory dedicated to storing input data sets and digital input vectors, and may have an operating speed sufficient to support the modulation rate. The second storage subunit may be a memory dedicated to storing neural network weights, and may have an operating speed sufficient to support the reconfiguration rate of the OMM unit 150. In some embodiments, the first storage subunit may be implemented using SRAM, and the second storage subunit may be implemented using DRAM. In some embodiments, the first storage subunit and the second storage subunit may be implemented using DRAM. In some embodiments, the first storage unit may be implemented as part of the controller 110 or as a cache of the controller 110. In some embodiments, the first and second storage subunits may be implemented as different address spaces by a single physical storage device.
[0379] The OMM unit 150 outputs a light output vector of length N, which corresponds to the result of the N×N matrix multiplication of the light input vector and the neural network weights. The OMM unit 150 is coupled to a detection unit 146, which is configured to generate N output voltages corresponding to the N light signals of the light output vector. For example, the detection unit 146 may include an array of N photodetectors (which are configured to absorb light signals and generate photocurrents) and an array of N transimpedance amplifiers (which are configured to convert the photocurrents into output voltages). The bandwidth of the photodetector and the transimpedance amplifier can be set based on the modulation rate of the modulator array 144. The photodetector can be formed of various materials based on the wavelength of the detected light output vector. Examples of materials for the photodetector include germanium, silicon-germanium alloys, and indium gallium arsenide (InGaAs).
[0380] The detection unit 146 is coupled to the ADC unit 160. The ADC unit 160 is configured to convert the N output voltages into N digital light outputs, which are quantized digital representations of the output voltages. For example, the ADC unit 160 may be an N-channel ADC. The controller 110 may obtain the N digital light outputs corresponding to the light output vector of the light matrix multiplication unit 150 from the ADC unit 160. The controller 110 may form a digital output vector of length N from the N digital light outputs, which corresponds to the result of the N×N matrix multiplication of the input digital vector of length N.
[0381] The various electronic components of the ANN computing system 100 can be integrated in various ways. For example, the controller 110 can be a dedicated integrated circuit manufactured on a semiconductor die. Other electronic components (such as a memory unit 120, a DAC unit 130, an ADC unit 160, or a combination thereof) can be monolithically integrated on a semiconductor die on which the controller 110 is manufactured. As another example, two or more electronic components can be integrated into a system-on-chip (SoC). In an embodiment of the SoC, the controller 110, the memory unit 120, the DAC unit 130, and the ADC unit 160 can be manufactured on the corresponding die, and the corresponding die can be integrated on a common platform (e.g., an interposer) that provides electrical connections between the integrated components. Compared to the method of separately arranging and routing components on a printed circuit board (PCB), this SoC method can allow faster data transmission between the electronic components of the ANN computing system 100, thereby improving the operating speed of the ANN computing system 100. In addition, the SoC approach can allow the use of different manufacturing technologies optimized for different electronic components, which can improve the performance of the different components and reduce the overall cost of the monolithic integration approach. Although the integration of the controller 110, the storage unit 120, the DAC unit 130, and the ADC unit 160 has been described, in general, a subset of the components can be integrated, while other components are implemented as separate components for various reasons (e.g., performance or cost). For example, in some embodiments, the storage unit 120 can be integrated with the controller 110 as a functional block within the controller 110.
[0382] Various optical components of the ANN computing system 100 can also be integrated in various ways. Examples of optical components of the ANN computing system 100 include a laser unit 142, a modulator array 144, an OMM unit 150, and a photodetector of a detection unit 146. These optical components can be integrated in various ways to improve performance and / or reduce costs. For example, the laser unit 142, the modulator array 144, the OMM unit 150, and the photodetector can be monolithically integrated on a common semiconductor substrate as a photonic integrated circuit (PIC). On a photonic integrated circuit formed based on a compound semiconductor material system (e.g., III-V compound semiconductors (e.g., indium phosphide (InP))), lasers, modulators (e.g., electroabsorption modulators), waveguides, and photodetectors can be monolithically integrated on a single die. This monolithic integration approach can reduce the complexity of aligning the inputs and outputs of various separate optical components, which may require alignment accuracy ranging from sub-micron to several microns. As another example, the laser source of the laser unit 142 can be manufactured on a compound semiconductor die, while the optical power splitter of the laser unit 142, the modulator array 144, the OMM unit 150, and the photodetector of the detection unit 146 can be manufactured on a silicon die. PICs manufactured on silicon wafers (which may be referred to as silicon photonics technology) generally have greater integration density, higher lithographic resolution, and lower cost relative to III-V-based PICs. This greater integration density may be beneficial in the manufacture of the OMM unit 150, because the OMM unit 150 generally includes tens to hundreds of optical components, such as power splitters and phase shifters. In addition, the higher lithographic resolution of silicon photonics technology can reduce the manufacturing variation of the OMM unit 150, thereby improving the accuracy of the OMM unit 150.
[0383] The ANN computing system 100 can be implemented in various form factors. For example, the ANN computing system 100 can be implemented as a co-processor that is inserted into a host computer. Such an ANN computing system 100 may have a form factor such as a PCI Express card and communicate with the host computer via a PCIe bus. The host computer can host multiple co-processor type ANN computing systems 100 and connect to the computer 102 via a network. This type of embodiment may be applicable to cloud data centers, where server racks may be dedicated to processing ANN computing requests received from other computers or servers. As another embodiment, a co-processor type ANN computing system 100 may be directly inserted into the computer 102 that issues the ANN computing request.
[0384] In some embodiments, the ANN computing system 100 can be integrated into a physical system that requires real-time ANN computing capabilities. For example, systems that rely heavily on real-time artificial intelligence tasks (such as self-driving vehicles, autonomous drones, object or facial recognition security cameras, and various Internet-of-Things (IoT) devices) can benefit from having the ANN computing system 100 directly integrated with other subsystems of such systems. The ANN computing system 100 with direct integration can implement real-time artificial intelligence in devices with poor or no network connectivity and enhance the reliability and availability of mission-critical artificial intelligence systems.
[0385] Although the DAC unit 130 and the ADC unit 160 are shown coupled to the controller 110, in some embodiments, the DAC unit 130, the ADC unit 160, or both may be alternatively or additionally coupled to the memory unit 120. For example, direct memory access (DMA) operations of the DAC unit 130 or the ADC unit 160 may reduce the computational burden on the controller 110 and reduce the latency of reading and writing to the memory unit 120, thereby further increasing the operating speed of the ANN computing unit 100.
[0386] Figure 2A A flow chart of an example of a method 200 for performing ANN calculations is shown. The steps of the method 200 may be performed by the controller 110. In some embodiments, the various steps of the method 200 may be run in parallel, combined, looped, or in any order.
[0387] At step 210, an artificial neural network (ANN) computation request including an input data set and a first plurality of neural network weights is received. The input data set includes a first digital input vector. The first digital input vector is a subset of the input data set. For example, it can be a sub-region of an image. The ANN computation request can be generated by various entities, such as computer 102. The computer can include one or more of various types of computing devices, such as a personal computer, a server computer, a vehicle computer, and a flight computer. The ANN computation request generally refers to an electrical signal that notifies or informs the ANN computing system 100 of the ANN computation to be performed. In some embodiments, the ANN computation request can be divided into two or more signals. For example, the first signal can query the ANN computing system 100 to check whether the system 100 is ready to receive the input data set and the first plurality of neural network weights. In response to an affirmative response from the system 100, the computer can send a second signal including the input data set and the first plurality of neural network weights.
[0388] In step 220, the input data set and the first plurality of neural network weights are stored. The controller 110 may store the input data set and the first plurality of neural network weights in the storage unit 120. Storing the input data set and the first plurality of neural network weights in the storage unit 120 may allow flexibility in the operation of the ANN computing system 100, which may, for example, improve the overall performance of the system. For example, the input data set may be divided into digital input vectors of a set size and format by retrieving a desired portion of the input data set from the storage unit 120. Different portions of the input data set may be processed in various orders, or shuffled, to allow various types of ANN computations to be performed. For example, where the input and output matrices are of different sizes, shuffling may allow matrix multiplication to be performed by block matrix multiplication techniques. As another example, storing the input data set and the first plurality of neural network weights in the storage unit 120 may allow multiple ANN computation requests to be queued by the ANN computing system 100, which may allow the ANN computing system 100 to maintain operation at its full speed without periods of inactivity.
[0389] In some embodiments, the input data set may be stored in a first storage subunit and the first plurality of neural network weights may be stored in a second storage subunit.
[0390] In step 230, a first plurality of modulator control signals are generated based on the first digital input vector, and a first plurality of weight control signals are generated based on the first plurality of neural network weights. The controller 110 may send a first DAC control signal to the DAC unit 130 to generate the first plurality of modulator control signals. The DAC unit 130 generates the first plurality of modulator control signals based on the first DAC control signal, and the modulator array 144 generates an optical input vector representing the first digital input vector.
[0391] The first DAC control signal may include a plurality of digital values to be converted by the DAC unit 130 into a first plurality of modulator control signals. The plurality of digital values generally correspond to a first digital input vector and may be associated by various mathematical relationships or lookup tables. For example, the plurality of digital values may be linearly proportional to the values of the elements of the first digital input vector. As another example, the plurality of digital values may be associated with the elements of the first digital input vector by a lookup table configured to maintain a linear relationship between the digital input vector and the light input vector generated by the modulator array 144.
[0392] The controller 110 may send the second DAC control signal to the DAC unit 130 to generate a first plurality of weight control signals. The DAC unit 130 generates a first plurality of weight control signals based on the second DAC control signal, and reconfigures the OMM unit 150 according to the first plurality of weight control signals to implement a matrix corresponding to the first plurality of neural network weights.
[0393] The second DAC control signal may include a plurality of digital values to be converted by DAC unit 130 into a first plurality of weight control signals. The plurality of digital values generally correspond to the first plurality of neural network weights and may be related by various mathematical relationships or lookup tables. For example, the plurality of digital values may be linearly proportional to the first plurality of neural network weights. As another example, the plurality of digital values may be calculated by performing various mathematical operations on the first plurality of neural network weights to generate weight control signals that may configure OMM unit 150 to perform matrix multiplication corresponding to the first plurality of neural network weights.
[0394] In some embodiments, the first plurality of neural network weights representing the matrix M may be decomposed into M=USV* by a singular value decomposition (SVD) method, where U is an M×M unitary matrix, S is an M×N diagonal matrix with non-negative real numbers on the diagonal, and V* is the complex conjugate of the N×N unitary matrix V. In this case, the first plurality of weight control signals may include a first plurality of OMM unit control signals corresponding to the matrix V, and a second plurality of OMM unit control signals corresponding to the matrix S. Further, the OMM unit 150 may be configured to have a first OMM subunit configured to implement the matrix V, a second OMM subunit configured to implement the matrix S, and a third OMM subunit configured to implement the matrix U, such that the OMM unit 150 as a whole implements the matrix M. The SVD method is further described in U.S. Patent Publication No. US2017 / 0351293A1, entitled “APPARATUS AND METHODS FOR OPTICAL NEURAL NETWORK,” which is fully incorporated herein by reference.
[0395] In step 240, a first plurality of digital optical outputs corresponding to the optical output vectors of the optical matrix multiplication unit are obtained. The optical input vector generated by the modulator array 144 is processed by the OMM unit 150 and converted into an optical output vector. The optical output vector is detected by the detection unit 146 and converted into an electrical signal, which can be converted into a digital value by the ADC unit 160. The controller 110 can, for example, send a conversion request to the ADC unit 160 to start converting the voltage output by the detection unit 146 into a digital optical output. Once the conversion is completed, the ADC unit 160 can send the conversion result to the controller 110. Alternatively, the controller 110 can obtain the conversion result from the ADC unit 160. The controller 110 can form a digital output vector from the digital optical output, which corresponds to the result of the matrix multiplication of the input digital vector. For example, the digital optical output can be organized or concatenated to have a vector format.
[0396] In some embodiments, the ADC unit 160 may be set or controlled to perform ADC conversion based on a DAC control signal issued by the controller 110 to the DAC unit 130. For example, the ADC conversion may be set to start at a preset time after the DAC unit 130 generates the modulation control signal. Such control of the ADC conversion may simplify the operation of the controller 110 and reduce the number of necessary control operations.
[0397] In step 250, a nonlinear transformation is performed on the first digital output vector to produce a first transformed digital output vector. The nodes or artificial neurons of the ANN operate by first performing a weighted sum of the signals received from the nodes of the previous layer, and then performing a nonlinear transformation ("activation") of the weighted sum to produce an output. Various types of ANNs can implement various types of differentiable nonlinear transformations. Examples of nonlinear transformation functions include a rectified linear unit (RELU) function, a sigmoid function, a hyperbolic tangent function, an X^2 function, and an |X| function. This nonlinear transformation is performed by the controller 110 on the first digital output to produce a first transformed digital output vector. In some embodiments, the nonlinear transformation may be performed by a dedicated digital integrated circuit within the controller 110. For example, the controller 110 may include one or more modules or circuit blocks that are particularly suitable for accelerating the calculation of one or more types of nonlinear transformations.
[0398] In step 260, the first converted digital output vector is stored. The controller 110 may store the first converted digital output vector in the storage unit 120. In the case where the input data set is divided into multiple digital input vectors, the first converted digital output vector corresponds to the ANN calculation result of a portion of the input data set, such as the first digital input vector. In this way, storing the first converted digital output vector allows the ANN computing system 100 to perform and store additional calculations on other digital input vectors of the input data set to be aggregated into a single ANN output later.
[0399] In step 270, an artificial neural network output generated based on the first transformed digital output vector is output. The controller 110 generates an ANN output, which is the result of processing the input data set by the ANN defined by the first plurality of neural network weights. In the case where the input data set is divided into multiple digital input vectors, the generated ANN output is an aggregate output including the first transformed digital output, but may further include additional transformed digital outputs corresponding to other parts of the input data set. Once the ANN output is generated, the generated output is sent to the computer (e.g., computer 102) that initiated the ANN calculation request.
[0400] Various performance metrics may be defined for the ANN computing system 100 implementing the method 200. Defining performance metrics may allow the performance of the ANN computing system 100 implementing the optical processor 140 to be compared with the performance of other systems for alternative ANN computing that implement an electronic matrix multiplication unit. In one aspect, the rate at which the ANN computing can be performed may be indicated in part by a first cycle period, which is defined as the time elapsed between the step 220 of storing the input data set and the first plurality of neural network weights in the memory unit and the step 260 of storing the first converted digital output vector in the memory unit. Thus, the first cycle period includes the time taken to convert the electrical signal into an optical signal (e.g., step 230), perform the matrix multiplication in the optical domain, and convert the result back to the electrical domain (e.g., step 240). Both steps 220 and 260 involve storing data in the memory unit 120, which is a step shared between the ANN computing system 100 and a conventional ANN computing system without the optical processor 140. As such, measuring the first cycle period of memory-to-memory transaction time may allow a realistic or fair comparison of ANN computation throughput between the ANN computing system 100 and an ANN computing system without the optical processor 140 (e.g., a system implementing an electrical matrix multiplication unit).
[0401] Due to the rate at which the modulator array 144 can generate optical input vectors (e.g., at 25 GHz) and the processing rate of the OMM unit 150 (e.g., >100 GHz), the first cycle period of the ANN computing system 100 for performing a single ANN computation of a single digital input vector can be close to the inverse of the speed of the modulator array 144 (e.g., 40 ps). After considering delays associated with signal generation by the DAC unit 130 and ADC conversion by the ADC unit 160, the first cycle period can be, for example, less than or equal to 100 ps, less than or equal to 200 ps, less than or equal to 500 ps, less than or equal to 1 ns, less than or equal to 2 ns, less than or equal to 5 ns, or less than or equal to 10 ns.
[0402] In comparison, the multiplication runtime of an M×1 vector and an M×M matrix of an electrical matrix multiplication unit is typically proportional to M^2-1 processor clock cycles. For M=32, such a multiplication will take approximately 1024 cycles, which results in a runtime of over 300ns at a 3GHz clock speed, which is several orders of magnitude slower than the first cycle period of the ANN computing system 100.
[0403] In some embodiments, method 200 further includes the step of generating a second plurality of modulator control signals based on the first converted digital output vector. In some types of ANN calculations, a single digital input vector may be repeatedly propagated through the same ANN or processed by the same ANN. An ANN that implements multi-pass processing may be referred to as a recurrent neural network (RNN). An RNN is a neural network in which the output of the network is recycled back to the input of the neural network during the (k)th pass through the neural network and is used as an input during the (k+1)th pass. RNNs may have various applications in pattern recognition tasks, such as speech or handwriting recognition. Once the second plurality of modulator control signals are generated, method 200 may proceed from step 240 to step 260 to complete the second pass of the first digital input vector through the ANN. Generally speaking, depending on the characteristics of the RNN received in the ANN calculation request, the converted digital output may be repeatedly recycled into a digital input vector for a predetermined number of cycles.
[0404] In some embodiments, the method 200 further includes the step of generating a second plurality of weight control signals based on a second plurality of neural network weights. In some cases, the artificial neural network calculation request further includes a second plurality of neural network weights. Generally speaking, in addition to the input layer and the output layer, the ANN also has one or more hidden layers. For an ANN having two hidden layers, the second plurality of neural network weights may correspond to the connectivity between the first layer of the ANN and the second layer of the ANN. In order to process the first digital input vector through the two hidden layers of the ANN, the first digital input vector may first be processed according to the method 200 until step 260, wherein the result of processing the first digital input vector through the first hidden layer of the ANN at step 260 is stored in the storage unit 120. The controller 110 then reconfigures the OMM unit 150 to perform a matrix multiplication corresponding to the second plurality of neural network weights associated with the second hidden layer of the ANN. Once the OMM unit 150 is reconfigured, the method 200 may generate a plurality of modulator control signals based on the first converted digital output vector, which generates an updated optical input vector corresponding to the output of the first hidden layer. The updated optical input vector is then processed by the reconfigured OMM unit 150, which corresponds to the second hidden layer of the ANN. Generally speaking, the steps described may be repeated until the digital input vector has been processed through all hidden layers of the ANN.
[0405] As described above, in some embodiments of the OMM unit 150, the reconfiguration rate of the OMM unit 150 may be significantly slower than the modulation rate of the modulator array 144. In such cases, the throughput of the ANN computing system 100 may be adversely affected by the amount of time spent reconfiguring the OMM unit 150 during periods during which ANN computations cannot be performed. In order to mitigate the effects of the relatively slow reconfiguration time of the OMM unit 150, batch processing techniques may be utilized in which two or more digital input vectors are propagated through the OMM unit 150 without configuration changes to amortize the reconfiguration time over a larger number of digital input vectors.
[0406] Figure 2B Show description Figure 2A Schematic diagram 290 of aspects of the method 200 of FIG. For an ANN having two hidden layers, instead of processing the first digital input vector through the first hidden layer, reconfiguring the OMM unit 150 for the second hidden layer, processing the first digital input vector through the reconfigured OMM unit 150, and repeating the same operation for the remaining digital input vectors, all digital input vectors of the input data set can be first processed by the OMM unit 150 configured for the first hidden layer (configuration #1), as shown in the upper part of the schematic diagram 290. Once all digital input vectors have been processed by the OMM unit 150 with configuration #1, the OMM unit 150 is reconfigured to configuration #2, which corresponds to the second hidden layer of the ANN. This reconfiguration can be significantly slower than the rate at which the OMM unit 150 can process input vectors. Once the OMM unit 150 is reconfigured for the second hidden layer, the output vectors from the previous hidden layer can be processed by the OMM unit 150 in batches. For large input data sets with tens or hundreds of thousands of numeric input vectors, the impact of reconfiguration time can be reduced by approximately the same factor, which can significantly reduce the portion of time the ANN computing system 100 spends in reconfiguration.
[0407] To achieve batch processing, in some embodiments, method 200 further includes the steps of generating a second plurality of modulator control signals based on a second digital input vector by a DAC unit; obtaining a second plurality of digital optical outputs corresponding to the optical output vector of the optical matrix multiplication unit from an ADC unit, the second plurality of digital optical outputs forming a second digital output vector; performing a nonlinear transformation on the second digital output vector to generate a second transformed digital output vector; and storing the second transformed digital output vector in a storage unit. For example, generating the second plurality of modulator control signals may be after step 260. In addition, the ANN output of step 270 in this case is now based on the first transformed digital output vector and the second transformed digital output vector. The obtaining, executing and storing steps are similar to steps 240 to 260.
[0408] Batch processing technology is one of the various technologies used to improve the throughput of the ANN computing system 100. Another technology for improving the throughput of the ANN computing system 100 is to process multiple digital input vectors in parallel by utilizing wavelength division multiplexing (WDM). WDM is a technology that simultaneously propagates multiple optical signals of different wavelengths through a common propagation channel (e.g., a waveguide of the OMM unit 150). Unlike electrical signals, optical signals of different wavelengths can be propagated through a common channel without affecting other optical signals of different wavelengths on the same channel. In addition, optical signals can be added (multiplexed) or discarded (demultiplexed) from a common propagation channel using well-known structures such as optical multiplexers and demultiplexers.
[0409] In the context of the ANN computing system 100, multiple optical input vectors of different wavelengths can be independently generated, simultaneously propagated through the OMM unit 150, and independently detected to enhance the throughput of the ANN computing system 100. Figure 1F , a schematic diagram of an example of a wavelength division multiplexing (WDM) artificial neural network (ANN) computing system 104 is shown. Unless otherwise described, the WDM ANN computing system 104 is similar to the ANN computing system 100. To implement WDM technology, in some embodiments of the ANN computing system 104, the laser unit 142 is configured to generate multiple wavelengths, such as λ 1 , 2 and 3. The multiple wavelengths may preferably be separated by a wavelength spacing that is large enough to allow easy multiplexing and demultiplexing onto a common propagation channel. For example, a wavelength spacing greater than 0.5 nm, 1.0 nm, 2.0 nm, 3.0 nm, or 5.0 nm may allow simple multiplexing and demultiplexing. On the other hand, the range between the shortest wavelength and the longest wavelength of the multiple wavelengths ("WDM bandwidth") may preferably be small enough so that the characteristics or performance of the OMM unit 150 remain substantially the same across the multiple wavelengths. Optical components are typically dispersive, meaning that their optical characteristics vary with wavelength. For example, the power separation ratio of the MZI may vary with wavelength. However, by designing the OMM unit 150 to have a sufficiently large operating wavelength window, and by limiting the wavelengths to within the operating wavelength window, the optical output vector output by the OMM unit 150 at each wavelength may be a sufficiently accurate result of the matrix multiplication performed by the OMM unit 150. The operating wavelength window may be, for example, 1 nm, 2 nm, 3 nm, 4 nm, 5 nm, 10 nm or 20 nm.
[0410] Fig.39A A diagram of an example of a Mach-Zehnder modulator 3900 that can be used to modulate the amplitude of an optical signal is shown. The Mach-Zehnder modulator 3900 includes two 1×2 port multimode interference couplers (MMI_1x2) 3902a and 3902b, two balanced arms 3904a and 3904b, and a phase shifter 3906 in one arm (or one phase shifter in each arm). When a voltage is applied to the phase shifter in one arm through a signal line 3908, there will be a phase difference between the two arms 3904a and 3904b that will be converted into amplitude modulation. The 1×2 port multimode interference couplers 3902a and 3902b and the phase shifter 3906 are configured as broadband photonic components, and the optical path lengths of the two arms 3904a and 3904b are configured to be equal. This enables the Mach-Zehnder modulator 3900 to operate over a wide wavelength range.
[0411] Fig.39B is a graph 3910 showing the wavelengths of 1530 nm, 1550 nm, and 1570 nm using Fig.39A Intensity-voltage curves of a Mach-Zehnder modulator 3900 of the configuration shown in . Graph 3910 shows that the Mach-Zehnder modulator 3900 has similar intensity-voltage characteristics for different wavelengths in the range of 1530 nm to 1570 nm.
[0412] Return to reference Figure 1F, the modulator array 144 of the WDM ANN computing system 104 includes banks of optical modulators configured to generate a plurality of optical input vectors, each of the optical modulator banks corresponding to one of a plurality of wavelengths and generating a corresponding optical input vector having a corresponding wavelength. For example, for a vector having a length of 32 and 3 wavelengths (e.g., λ 1 , 2 and 3 ), the modulator array 144 can have 3 groups of 32 modulators in each group. In addition, the modulator array 144 also includes an optical multiplexer that is configured to combine multiple optical input vectors into a combined optical input vector including multiple wavelengths. For example, the optical multiplexer can combine the outputs of three modulator groups at three different wavelengths into a single propagation channel (e.g., waveguide) for each element of the optical input vector. As a result, returning to the example above, the combined optical input vector will have 32 optical signals, each signal including 3 wavelengths.
[0413] In addition, the detection unit 146 of the WDM ANN computing system 104 is further configured to demultiplex the multiple wavelengths and generate multiple demultiplexed output voltages. For example, the detection unit 146 may include a demultiplexer configured to demultiplex the three wavelengths in each of the 32 signals included in the multi-wavelength optical output vectors and route the three single wavelength optical output vectors to three sets of photodetectors coupled to three sets of transimpedance amplifiers.
[0414] In addition, the ADC unit 160 of the WDM ANN computing system 104 includes an ADC group, which is configured to convert multiple demultiplexed output voltages of the detection unit 146. Each group in the ADC group corresponds to one of the multiple wavelengths and generates a corresponding digital demultiplexed optical output. For example, the ADC group can be coupled to the transimpedance amplifier group of the detection unit 146.
[0415] The controller 110 may implement a method similar to the method 200, but expanded to support multi-wavelength operation. For example, the method may include the steps of obtaining a plurality of digital demultiplexed optical outputs from the ADC unit 160, the plurality of digital demultiplexed optical outputs forming a plurality of first digital output vectors, wherein each of the plurality of first digital output vectors corresponds to one of the plurality of wavelengths; performing a nonlinear conversion on each of the plurality of first digital output vectors to generate a plurality of converted first digital output vectors; and storing the plurality of converted first digital output vectors in a storage unit.
[0416] In some cases, the ANN can be specifically designed and the digital input vector can be specifically formed so that the multi-wavelength light output vector can be detected without demultiplexing. In this case, the detection unit 146 can be a wavelength-insensitive detection unit that does not demultiplex the multiple wavelengths of the multi-wavelength light output vector. In this way, each photodetector of the detection unit 146 effectively adds multiple wavelengths of the optical signal into a single photocurrent, and each voltage output by the detection unit 146 corresponds to the element-by-element sum of the matrix multiplication results of the multiple digital input vectors.
[0417] To date, nonlinear transformations of weighted sums performed as part of ANN computations have been performed in the digital domain by the controller 110. In some cases, the nonlinear transformations may be computationally intensive or power consuming, significantly increasing the complexity of the controller 110, or limiting the performance of the ANN computing system 100 in terms of throughput or power efficiency. As such, in some embodiments of the ANN computing system, the nonlinear transformations may be performed in the analog domain by analog electronics.
[0418] Figure 3A A schematic diagram of an example of an ANN computing system 300 is shown. The ANN computing system 300 is similar to the ANN computing system 100, except that an analog nonlinear unit 310 is added. The analog nonlinear unit 310 is disposed between the detection unit 146 and the ADC unit 160. The analog nonlinear unit 310 is configured to receive an output voltage from the detection unit 146, apply a nonlinear transfer function, and output a converted output voltage to the ADC unit 160.
[0419] When the ADC unit 160 receives the voltage that has been nonlinearly converted by the analog nonlinear unit 310, the controller 110 can obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 160. Because the digital output voltage obtained from the ADC unit 160 has been nonlinearly converted ("activated"), the nonlinear conversion step of the controller 110 can be omitted, thereby reducing the calculation burden of the controller 110. Then, the first converted voltage obtained directly from the ADC unit 160 can be stored in the storage unit 120 as a first converted digital output vector.
[0420] The analog nonlinear unit 310 may be implemented in various ways. For example, a high gain amplifier in a feedback configuration, a comparator with an adjustable reference voltage, a nonlinear IV characteristic of a diode, a breakdown behavior of a diode, a nonlinear CV characteristic of a variable capacitor, or a nonlinear IV characteristic of a variable resistor may be used to implement the analog nonlinear unit 310.
[0421] The use of analog nonlinear units 310 can improve the performance of the ANN computing system 300, such as throughput or power efficiency, by reducing the steps performed in the digital domain. Moving the nonlinear conversion steps out of the digital domain can allow additional flexibility and improvements in the operation of the ANN computing system. For example, in a recurrent neural network, the output of the OMM unit 150 is activated and recycled back to the input of the OMM unit 150. The activation step is performed by the controller 110 in the ANN computing system 100, which requires digitizing the output voltage of the detection unit 146 each time it passes through the OMM unit 150. However, because the activation step is now performed before digitization by the ADC unit 160, the number of ADC conversions required in performing recurrent neural network calculations can be reduced.
[0422] In some embodiments, the analog nonlinear unit 310 can be integrated into the ADC unit 160 as a nonlinear ADC unit. For example, the nonlinear ADC unit can be a linear ADC unit with a nonlinear lookup table that maps the linear digital output of the linear ADC unit to the desired nonlinear converted digital output.
[0423] Figure 3B A schematic diagram of an example of an ANN computing system 302 is shown. The ANN computing system 302 is similar to Figure 3A The analog storage unit 320 is a system 300 of the present invention, except that it further includes an analog storage unit 320. The analog storage unit 320 is coupled to the DAC unit 130 (e.g., through the first DAC subunit 132), the modulator array 144, and the analog non-linear unit 310. The analog storage unit 320 includes a multiplexer having a first input coupled to the DAC unit 130 and a second input coupled to the analog non-linear unit 310. This allows the analog storage unit 320 to receive a signal from the DAC unit 130 or the analog non-linear unit 310. The analog storage unit 320 is configured to store an analog voltage and output the stored analog voltage.
[0424] The analog storage unit 320 can be implemented in various ways. For example, a capacitor array can be used as an analog voltage storage component. The capacitor of the analog storage unit 320 can be charged to the input voltage by a charging circuit. The storage of the input voltage can be controlled based on a control signal received from the controller 110. The capacitor can be electrically isolated from the surrounding environment to reduce charge leakage that causes undesirable capacitor discharge. Additionally (or alternatively), a feedback amplifier can be used to maintain the voltage stored on the capacitor. The stored voltage of the capacitor can be read out by a buffer amplifier, which allows the charge stored by the capacitor to be maintained while outputting the stored voltage. These aspects of the analog storage unit 320 can be similar to the operation of a sample and hold circuit. The buffer amplifier can implement the function of a modulator driver for driving the modulator array 144.
[0425] The operation of the ANN computing system 302 will now be described. The first plurality of modulator control signals output by the DAC unit 130 (e.g., by the first DAC subunit 132) are first input to the modulator array 144 through the analog storage unit 320. In this step, the analog storage unit 320 may simply pass or buffer the first plurality of modulator control signals. The modulator array 144 generates a light input vector based on the first plurality of modulator control signals, which is propagated through the OMM unit 150 and detected by the detection unit 146. The output voltage of the detection unit 146 is nonlinearly converted by the analog nonlinear unit 310. At this time, instead of being digitized by the ADC unit 160, the output voltage of the detection unit 146 is stored by the analog storage unit 320, which is then output to the modulator array 144 to be converted into the next light input vector to be propagated through the OMM unit 150. Under the control of the controller 110, this recurrent processing may be performed at a preset amount of time or a preset number of cycles. Once the recursive process is completed for a given digital input vector, the converted output voltage of the analog non-linear unit 310 is converted by the ADC unit 160 .
[0426] The use of analog storage unit 320 can significantly reduce the number of ADC conversions during recurrent neural network calculations, for example, down to a single ADC conversion for each RNN calculation of a given digital input vector. Each ADC conversion takes a while and consumes a certain amount of energy. As a result, the throughput of RNN calculations of ANN computing system 302 can be higher than the throughput of RNN calculations of ANN computing system 100.
[0427] The execution of the recurrent neural network calculation can be controlled by controlling the analog storage unit 320. For example, the controller can control the analog storage unit 320 to store a voltage at a specific time and output the stored voltage at a different time. In this way, the signal cycle from the analog storage unit 320 to the modulator array 144 through the analog nonlinear unit 310 and back to the analog storage unit 320 can be controlled by the controller 110 controlling the storage and reading of the analog storage unit 320.
[0428] Thus, in some embodiments, the controller 110 of the ANN computing system 302 may perform the following steps: based on generating a first plurality of modulator control signals and a first plurality of weight control signals, storing a plurality of conversion output voltages of the analog nonlinear unit through an analog storage unit; outputting the stored conversion output voltages through the analog storage unit; obtaining a second plurality of conversion digital output voltages from the ADC unit, the second plurality of conversion digital output voltages forming a second conversion digital output vector; and storing the second conversion digital output vector in the storage unit.
[0429] The input data set processed by the ANN computing system typically includes data with a resolution greater than 1 bit. For example, a typical pixel of a grayscale digital image may have a resolution of 8 bits, i.e., 256 different levels. One way to represent and process this data in the optical domain is to encode the pixels at 256 different intensity levels as 256 different power levels of the optical signal input to the OMM unit 150. Optical signals are analog in nature and are therefore susceptible to noise and detection errors. Return to Reference Figure 1A In order to maintain 8-bit resolution of the digital input vector throughout the ANN computing system 100 and produce a true 8-bit digital light output at the output of the ADC unit 160, each part of the signal chain can preferably be designed to reproduce and maintain 8-bit resolution.
[0430] For example, the DAC unit 130 may preferably be designed to support conversion of an 8-bit digital input vector into a modulator control signal with at least 8-bit resolution, so that the modulator array 144 can generate an 8-bit optical input vector that faithfully represents the digital input vector. Generally speaking, the modulator control signal may need to have an additional resolution of more than 8 bits of the digital input vector to compensate for the nonlinear response of the modulator array 144. In addition, the internal configuration of the OMM unit 150 may preferably be stable enough to ensure that the value of the optical output vector is not corrupted by any fluctuations in the configuration of the OMM unit 150. For example, the temperature of the OMM unit 150 may need to be stable within 5 degrees, 2 degrees, 1 degree, or 0.1 degrees. In addition, the detection unit 146 may preferably have low enough noise to not corrupt the 8-bit resolution of the optical output vector, and the ADC unit 160 may preferably be designed to support digitization of analog voltages with at least 8-bit resolution.
[0431] The power consumption and design complexity of various electronic components generally increase with bit resolution, operating speed, and bandwidth. For example, as a first-order approximation, the power consumption of the ADC unit 160 can be linearly scaled with the sampling rate, and the scaling factor is 2^N, where N is the bit resolution of the conversion result. In addition, the design considerations of the DAC unit 130 and the ADC unit 160 generally result in a trade-off between sampling rate and bit resolution. As such, in some cases, it may be desirable for the ANN computing system to operate internally at a bit resolution lower than the resolution of the input data set while maintaining the resolution of the ANN computing output.
[0432] Reference Figure 4A , a schematic diagram of an example of an artificial neural network (ANN) computing system 400 having a 1-bit internal resolution is shown. The ANN computing system 400 is similar to the ANN computing system 100, except that the DAC unit 130 is now replaced by a driver unit 430, and the ADC unit 160 is now replaced by a comparator unit 460.
[0433] The driver unit 430 is configured to generate a 1-bit modulator control signal and a multi-bit weight control signal. For example, the driver circuit of the driver unit 430 can directly receive a binary digital output from the controller 110 and adjust the binary signal to a two-level voltage or current output suitable for driving the modulator array 144.
[0434] The comparator unit 460 is configured to convert the output voltage of the detection unit 146 into a digital 1-bit light output. For example, the comparison circuit of the comparator unit 460 can receive a voltage from the detection unit 146, compare the voltage with a preset threshold voltage, and output a digital 0 or 1 when the received voltage is less than or greater than the preset threshold voltage, respectively.
[0435] Reference Figure 4B , which shows a mathematical representation of the operation of the ANN computing system 400. Figure 4B The operation of the ANN computing system 400 is described. For a given ANN computing to be performed by the ANN computing system 400, there is a corresponding numeric input vector V and a neural network weight matrix U. In this embodiment, the input vector V is a vector having elements V 0 To V 3 , and the matrix U is a vector of length 4 with weights U 00 To U 33 Each element of vector V has a 4-bit resolution. Each 4-bit vector element has bit 0 corresponding to positions 2^0 to 2^3. 0 ) to bit 3 3 ). In this way, by 2^0*bit 0 +2^1*bit 1 +2^2*bit 2 +2^3*bit 3 The decimal (base 10) value of the 4-bit vector element is calculated by summing the sum of . Therefore, as shown in the figure, the input vector V can be similarly decomposed into V by the controller 110 simulation. bit0 To V bit3 .
[0436] Next, a particular ANN calculation may be performed by performing a series of matrix multiplications of 1-bit vectors and then summing the matrix multiplication results. For example, the decomposed input vector V may be converted to V by generating a sequence of four 1-bit modulator control signals corresponding to four 1-bit input vectors through the driver unit 430. bit0 To V bit3 Each of is multiplied by the matrix U. This in turn generates a sequence of four 1-bit optical input vectors, which are propagated through the OMM unit 150, which is configured to implement the matrix multiplication of the matrix U through the driver unit 430. Then, the controller 110 can obtain a sequence of four digital 1-bit optical outputs corresponding to the sequence of four 1-bit modulator control signals from the comparator unit 460.
[0437] In the case of decomposing a 4-bit vector into four 1-bit vectors, each vector should be processed by the ANN computing system 400 at four times the speed at which other ANN computing systems (e.g., system 100) can process a single 4-bit vector to maintain the same effective ANN computing throughput. This increased internal processing speed can be viewed as time-division multiplexing of four 1-bit vectors into a single timeslot for processing a 4-bit vector. The required increase in processing speed can be achieved at least in part by an increased operating speed of the driver unit 430 and the comparator unit 460 relative to the DAC unit 130 and the ADC unit 160, because a reduction in the resolution of the signal conversion process generally results in an increase in the achievable signal conversion rate.
[0438] Although the signal conversion rate in 1-bit operation is increased by a factor of four, the power consumption generated can be significantly reduced relative to 4-bit operation. As described above, the power consumption of signal conversion processing usually scales exponentially with the bit resolution and linearly with the conversion rate. As a result, a 16-fold reduction in power per conversion may be due to a 4-fold reduction in bit resolution, followed by a 4-fold increase in power generated due to the increase in conversion rate. In summary, a 4-fold reduction in operating power can be achieved over, for example, the ANN computing system 100 through the ANN computing system 400, while maintaining the same effective ANN computing throughput.
[0439] Then, the controller 110 may construct a 4-bit digital output vector from the 4 digital 1-bit light outputs by multiplying each digital 1-bit light output by a corresponding weight 2^0 to 2^3. Once the 4-bit digital output vector is constructed, an ANN calculation may be performed by performing a nonlinear transformation on the constructed 4-bit digital output vector to generate a transformed 4-bit digital output vector; and the transformed 4-bit digital output vector is stored in the storage unit 120.
[0440] Alternatively (or additionally), in some embodiments, each of the four digital 1-bit optical outputs may be nonlinearly transformed. For example, a step-function nonlinear function may be used for the nonlinear transformation. A transformed 4-bit digital output vector may then be constructed from the nonlinearly transformed digital 1-bit optical outputs.
[0441] Although a single ANN computing system 400 has been shown and described, in general, Figure 1AThe ANN computing system 100 can be designed to implement functions similar to the ANN computing system 400. For example, the DAC unit 130 may include a 1-bit DAC subunit configured to generate a 1-bit modulator control signal, and the ADC unit 160 may be designed to have a resolution of 1 bit. Such a 1-bit ADC may be similar to or effectively equivalent to a comparator.
[0442] Furthermore, although the operation of the ANN computing system with a 1-bit internal resolution has been described, in general, the internal resolution of the ANN computing system can be reduced to an intermediate level below the N-bit resolution of the input data set. For example, the internal resolution can be reduced to 2^Y bits, where Y is an integer greater than or equal to 0.
[0443] The embodiments of the subject matter and functional operations described in the present disclosure can be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in the present disclosure and their structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described in the present disclosure can be implemented using one or more computer program instruction modules encoded on a computer-readable medium to be executed or control the operation of the data processing device by a data processing device. The computer-readable medium can be a manufactured product (such as a hard drive in a computer system or an optical disk sold through a retail channel) or an embedded system. The computer-readable medium can be separately obtained and then encoded using one or more modules of computer program instructions, such as transmitting one or more modules of computer program instructions via a wired or wireless network. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a storage device, or a combination of one or more of them.
[0444] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.
[0445] The processes and logic flows described in the present disclosure may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may be implemented as, special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC).
[0446] Although the present disclosure includes many implementation details, these should not be interpreted as limitations on the scope of the present disclosure or the claims, but as descriptions of specific features of specific embodiments of the present disclosure. Certain features described in the present disclosure in the context of the respective embodiments may also be implemented in combination in a single embodiment. Relatively speaking, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination, respectively. In addition, although the above features are described as acting on certain combinations and even initially requested for protection, in some cases, one or more features may be removed from the requested combination, and the requested combination may point to a sub-combination or a change in a sub-combination.
[0447] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring the operations to be performed in the particular order shown or in sequence, or to perform all displayed operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated in a single software product or packaged in multiple software products.
[0448] Thus, certain embodiments of the present disclosure have been described. Other embodiments are within the scope of the following claims. Additionally, the actions recited in the claims can be performed in a different order and still achieve the desired results. For example, Figure 1A The optical matrix multiplication unit 150 in the embodiment includes an optical interference unit 154, which includes a plurality of interconnected Mach-Zehnder interferometers. In some embodiments, the optical interference unit can be implemented using a one-dimensional, two-dimensional, or three-dimensional passive diffractive optical element that consumes almost no power. Compared to an optical interference unit including a Mach-Zehnder interferometer, if the number of inputs / outputs remains unchanged, the optical interference unit using the passive diffractive optical element can have a smaller size, or can handle a larger number of inputs / outputs for the same chip size. Compared to the Mach-Zehnder interferometer, the passive diffractive optical element can be manufactured at a lower cost.
[0449] Reference Figure 5 In some embodiments, the artificial neural network computing system 500 includes a controller 110, a storage unit 120, a DAC unit 506, an optical processor 504, and an ADC unit 160. The storage unit 120 and the ADC unit 160 are similar to Figure 1A The optical processor 504 is configured to perform matrix calculations using optical components. In the system 500, the weights of the two-dimensional optical matrix multiplication unit 502 are fixed. The DAC unit 506 is similar to Figure 1A The first DAC subunit 132 of the system 100 .
[0450] In an example operation of the ANN computing system 500, the computer 102 may issue an artificial neural network computing request to the ANN computing system 500. The ANN computing request may include an input data set to be processed by the provided ANN. The controller 110 receives the ANN computing request and stores the input data set in the storage unit 120.
[0451] In some embodiments, a hybrid approach is used, where a portion of the optical matrix multiplication unit 150 includes a Mach-Zehnder interferometer and another portion of the optical matrix multiplication unit 150 includes a passive diffractive component.
[0452] The internal operation of the ANN computing system 500 will now be described. The optical processor 504 includes a laser unit 142, a modulator array 144, a detection unit 146, and a two-dimensional optical matrix multiplication (OMM) unit 502. The laser unit 142, the modulator array 144, and the detection unit 146 are similar to Figure 1A . In this example, the two-dimensional OMM unit 502 includes a two-dimensional diffractive optical component and can be implemented as a passive integrated silicon photonic chip. The two-dimensional optical matrix multiplication unit 502 can be configured to implement a diffractive neural network and can perform matrix multiplication at almost zero power consumption.
[0453] The optical processor 504 operates by encoding a digital input vector of length N onto an optical input vector of length N and propagating the optical input vector through the two-dimensional OMM unit 502. The two-dimensional OMM unit 502 receives the optical input vector of length N and performs an N×N matrix multiplication on the received optical input vector in the optical domain. The N×N matrix multiplication performed by the two-dimensional OMM unit 502 is determined by the internal configuration of the two-dimensional OMM unit 502. The internal configuration of the two-dimensional OMM unit 502 includes the size, position and geometry of the diffractive optical components, and the doping of impurities (if any).
[0454] The two-dimensional OMM cell 502 may be implemented in various ways. Figure 6 A schematic diagram of an example of a two-dimensional OMM cell 502 using a two-dimensional array of diffractive components is shown. The two-dimensional OMM cell 502 may include an array of input waveguides 602 to receive an optical input vector, a two-dimensional optical interference cell 600 in optical communication with the array of input waveguides 602, and an array of output waveguides 604 in optical communication with the optical interference cell 600. The optical interference cell 600 includes a plurality of diffractive optical components and performs a conversion (e.g., linear conversion) of the optical input vector to a second optical signal array. The array of output waveguides 604 guides the second optical signal array output by the optical interference cell 600. At least one input waveguide in the array of input waveguides 602 is in optical communication with each output waveguide in the array of output waveguides 604 through the optical interference cell 600. For example, for an optical input vector of length N, the two-dimensional OMM cell 502 may include N input waveguides 602 and N output waveguides 604.
[0455] In some embodiments, the optical interference unit 600 includes a substrate having a diffraction component, and the diffraction component is arranged in two dimensions (for example, in a 2D array). For example, a plurality of circular holes can be drilled or etched in the substrate. The size of these holes can be comparable to the size of the wavelength of the input light in order of magnitude, so that the light is diffracted by the holes (or the structure defining the holes). For example, the size of the holes can be in the range of 100nm to 2μm. The holes can have the same or different sizes. The holes can also have other cross-sectional shapes, such as triangles, squares, rectangles, hexagons, or irregular shapes. The substrate can be made of a material that is transparent or translucent to the input light, for example, having a transmittance of 1% to 99% relative to the input light. For example, the substrate can be made of silicon, silicon oxide, silicon nitride, quartz, crystals (for example, lithium niobate (LiNbO3)), III-V materials (for example, gallium arsenide or indium phosphide), erbium modified semiconductors (erbiummodified semiconductors) or polymers.
[0456] In some embodiments, a holographic method can be used to form a two-dimensional diffractive optical component in a substrate. The substrate can be made of glass, crystal, or photorefractive material.
[0457] When designing a two-dimensional OMM unit 502, the size and position of the diffractive components are considered in two dimensions (e.g., X and Y directions), without considering the relative position of the diffractive components in the third dimension (e.g., Z direction). Each diffractive component can be a three-dimensional structure formed in the substrate, such as a hole, column, or stripe with a certain depth.
[0458] exist Figure 6 In the figure, the diffractive optical components are represented by circles. The diffractive optical components can also have other shapes, such as triangles, squares, rectangles or irregular shapes. The diffractive optical components can have various sizes. The diffractive optical components do not have to be located on the grid points, and their positions can be changed. Figure 6 The figures in are for illustration purposes only. Actual diffractive optical components may differ from those shown in the figures. Different arrangements of diffractive optical components may be used to implement different matrix calculations, such as different matrix multiplication functions.
[0459] An optimization process may be used to determine the configuration of the diffractive optical element. For example, the substrate may be divided into an array of pixels, and each pixel may be filled with the substrate material (no holes) or filled with air (holes). The configuration of the pixels may be modified iteratively, and for each configuration of the pixels, a simulation may be performed by passing light through the diffractive optical element and evaluating the output. After performing simulations of all possible configurations of the pixels, the configuration that provides the closest result to the desired matrix process is selected as the diffractive optical element configuration of the two-dimensional OMM unit 502.
[0460] As another example, the diffractive component is initially configured as an array of holes. The location, size, and shape of the holes may be slightly different from their initial configuration. The parameters of each hole may be iteratively adjusted, and simulations may be performed to find an optimized configuration of the holes.
[0461] In some embodiments, a machine learning process is used to design a diffractive optical component. An analytical function is determined for how pixels affect input light to produce output light, and an optimization process (eg, a gradient descent method) is used to determine the best configuration of the pixels.
[0462] In some embodiments, the two-dimensional OMM unit 502 may be implemented as a user-changeable component, and different two-dimensional OMM units 502 with different optical interference units 600 may be installed for different applications. For example, the system 500 may be configured as an optical character recognition system, and the optical interference unit 600 may be configured to implement a neural network for performing optical character recognition. For example, a first OMM unit may have a first optical interference unit, the first optical interference unit including a passive diffractive optical component, which is configured to implement a first neural network for an optical character recognition engine for a first set of written languages and fonts. The second OMM unit may have a second optical interference unit, the second optical interference unit including a passive diffractive optical component, which is configured to implement a second neural network for an optical character recognition engine for a second set of written languages and fonts, etc. When a user wants to use the system 500 to apply optical character recognition to the first set of written languages and fonts, the user can insert the first OMM unit into the system. When a user wants to use the system 500 to apply optical character recognition to the second set of written languages and fonts, the user can swap out the first OMM unit and insert the second OMM unit into the system.
[0463] For example, system 500 may be configured as a speech recognition system, and optical interference unit 600 may be configured to implement a neural network for performing speech recognition. For example, a first OMM unit may have a first optical interference unit, the first optical interference unit including a passive diffractive optical component, which is configured to implement a first neural network for a speech recognition engine for a first spoken language. A second OMM unit may have a second optical interference unit, the second optical interference unit including a passive diffractive optical component, which is configured to implement a second neural network for a speech recognition engine for a second spoken language, etc. When a user wants to use system 500 to recognize speech in a first spoken language, the user can insert the first OMM unit into the system. When a user wants to use system 500 to recognize speech in a second spoken language, the user can swap out the first OMM unit and insert the second OMM unit into the system.
[0464] For example, the system 500 may be part of a control unit of an autonomous vehicle, and the optical interference unit 600 may be configured to implement a neural network for performing road condition recognition. For example, the first OMM unit may have a first optical interference unit, the first optical interference unit including a passive diffractive optical component, which is configured to implement a first neural network for identifying road conditions (including road signs) in the United States. The second OMM unit may have a second optical interference unit, the second optical interference unit including a passive diffractive optical component, which is configured to implement a second neural network for identifying road conditions (including road signs) in Canada. The third OMM unit may have a third optical interference unit, the third optical interference unit including a passive diffractive optical component, which is configured to implement a third neural network for identifying road conditions (including road signs) in Mexico, and so on. When the autonomous vehicle is used in the United States, the first OMM unit is inserted into the system. When the autonomous vehicle crosses the border and enters Canada, the first OMM unit is swapped out and the second OMM unit is inserted into the system. On the other hand, when the autonomous vehicle crosses the border and enters Mexico, the first OMM unit is swapped out and the third OMM unit is inserted into the system.
[0465] For example, the system 500 can be used for genetic sequencing. DNA sequences can be classified using a convolutional neural network implemented using the system 500 including a passive diffractive optical component. For example, the system 500 can implement a neural network for distinguishing tumor types, predicting tumor grade, and predicting patient survival from gene expression patterns. For example, the system 500 can implement a neural network for identifying a subset of genes or features that are most predictive of the characteristics analyzed. For example, the system 500 can implement a neural network for predicting or inferring the expression level of all genes from a profile of a subset of genes. For example, the system 500 can implement a neural network for epigenomic analysis, such as predicting transcription factor binding sites, enhancer regions, and chromatin accessibility from a gene sequence. For example, the system 500 can implement a neural network for capturing structures within a gene sequence.
[0466] For example, the system 500 can be configured as a medical diagnostic system, and the two-dimensional OMM unit 502 can be configured to implement a neural network for analyzing physiological parameters to perform screening for diseases. For example, the system 500 can be configured as a bacterial detection system, and the two-dimensional OMM unit 502 can be configured to implement a multiplication function for analyzing DNA sequences to detect certain bacterial strains.
[0467] In some embodiments, the two-dimensional OMM unit 502 includes a housing (e.g., a cartridge) that protects a substrate having a diffractive optical component. The housing supports an input interface coupled to an input waveguide 602, and an output interface coupled to an output waveguide 604. The input interface is configured to receive an output from the modulator array 144, and the output interface is configured to send the output of the two-dimensional OMM unit 502 to the detection unit 146. The two-dimensional OMM unit 502 can be designed as a module suitable for handling by an ordinary consumer, allowing a user to easily switch from one two-dimensional OMM unit 502 to another two-dimensional OMM unit 502. Machine learning technology improves over time. A user can upgrade the system 500 by swapping out an old two-dimensional OMM unit 502 and inserting a new upgraded version.
[0468] Similar to how an optical compact disc can store digital information that can be retrieved by a CD player, an OMM unit can store a neural network configuration that can be used in an optical processor. Just as an optical compact disc is a low-cost medium for distributing digital information (including audio, video, and software programs) to consumers, an OMM unit can be a low-cost medium for distributing pre-configured neural networks or matrix processing functions (e.g., multiplication, convolution, or any other linear operation) to consumers.
[0469] In some embodiments, system 500 is an optical computing platform that is configured to operate with OMM units provided by different companies. This allows different companies to develop different passive optical neural networks for various applications. The passive optical neural network is sold to end users in standardized packaging, which can be installed in the optical computing platform to allow system 500 to perform various intelligent functions.
[0470] In some embodiments, the system may have a holder mechanism for supporting multiple two-dimensional OMM units 502, and a mechanical handling mechanism may be provided for automatically swapping out two-dimensional OMM units 502. The system determines which two-dimensional OMM unit 502 is needed for the current application, and automatically takes the appropriate OMM unit from the holder mechanism and inserts it into the optical processor 504 using the mechanical handling mechanism.
[0471] For an optical chip of a given size, more passive diffractive components can be assembled on a substrate than using an active interferometer (e.g., a Mach-Zehnder interferometer). Figure 1B The optical interference unit 154 in can be configured to process 200×200 matrix multiplications, while the optical interference unit 600 having the same overall size and using passive diffraction components (each having a size of approximately 100nm×100nm) can be configured to process 5000×5000 matrix multiplications.
[0472] Passive diffractive optical components consume almost no power, so the two-dimensional OMM unit 502 can be used in low-power devices, such as battery-operated devices. The two-dimensional OMM unit 502 is suitable for edge computing. For example, the two-dimensional OMM unit 502 can be used in smart sensors, where the raw data from the sensor is processed using an optical processor using the two-dimensional OMM unit 502. The smart sensor can be configured to send the processed data to a central computer server, thereby reducing the amount of raw data sent to the central computer server. By placing intelligent processing functions on the smart sensor, faults and anomalies can be detected earlier and handled more efficiently. The two-dimensional OMM unit 502 is suitable for applications that need to process large matrix multiplications. The two-dimensional OMM unit 502 is suitable for applications where a neural network has been trained and the weights have been determined and do not need to be modified.
[0473] The substrate on which the diffractive optical element is formed may be planar or curved. Figure 6 In the example of FIG. 6 , input light enters the light interference unit 600 from the left side, and output light leaves the light interference unit 600 from the right side (the terms "left", "right", "upper" and "lower" refer to the directions shown in the drawings). In some embodiments, the passive diffractive optical component can be configured so that some output light leaves the light interference unit from the top or bottom, or any combination of the left, right, upper and lower sides of the light interference unit 600. The substrate for the light interference unit 600 can have various shapes, such as square, rectangular, triangular, circular or elliptical. The light interference unit 600 may include a reflective component or a reflector to redirect the light propagation direction.
[0474] In some embodiments, the artificial neural network computing system 500 may be modified by adding an analog nonlinear unit 310 between the detection unit 146 and the ADC unit 160. The analog nonlinear unit 310 is configured to receive the output voltage from the detection unit 146, apply a nonlinear transfer function, and output the converted output voltage to the ADC unit 160. The controller 110 may obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 160. Because the digital output voltage obtained from the ADC unit 160 has been nonlinearly converted ("activated"), the nonlinear conversion step of the controller 110 may be omitted, thereby reducing the computational burden of the controller 110. Then, the first converted voltage directly obtained from the ADC unit 160 may be stored in the storage unit 120 as a first converted digital output vector.
[0475] The optical interference unit can be realized using passive diffractive optical components arranged in three dimensions. Figure 7In some embodiments, the artificial neural network computing system 700 has an optical processor 702, which includes a three-dimensional OMM unit 708. The system 700 includes a storage unit 120 and an ADC unit 160, which are similar to Figure 5 The optical processor 702 is configured to perform matrix calculations using diffractive optical components arranged in three dimensions.
[0476] The optical processor 702 includes a laser unit 704 configured to output a two-dimensional beam array 714, and a two-dimensional modulator array 706 configured to modulate the two-dimensional beam array 714 to generate a modulated two-dimensional beam array 716. The optical processor 702 includes a three-dimensional optical matrix multiplication (OMM) unit 708, which has a three-dimensionally arranged diffractive optical component and is configured to process the modulated two-dimensional beam array 716 and generate a two-dimensional output beam array 718. The optical processor 702 includes a detection unit 710 having a two-dimensional light sensor array to detect the two-dimensional output beam array 718. The ADC unit 160 converts the output of the detection unit 710 into a digital signal.
[0477] For example, the 3D OMM unit 708 may be implemented as a passive integrated silicon photonic rod or cube. The three-dimensional optical matrix multiplication unit 708 may be configured to implement a diffractive neuron network and may perform matrix multiplication at almost zero power consumption.
[0478] There are many ways to encode input data for use by the optical processor 702. For example, a digital input vector of length N×N can be encoded onto an optical input matrix of size N×N, which is propagated through the three-dimensional OMM unit 708. The three-dimensional OMM unit 708 performs a (N×N)×(N×N) matrix multiplication on the received optical input matrix in the optical domain. The (N×N)×(N×N) matrix multiplication performed by the three-dimensional OMM unit 708 is determined by the internal configuration of the three-dimensional OMM unit 708, which includes the size, position and geometry of the diffractive optical components arranged in three dimensions, as well as the doping of impurities (if any).
[0479] The three-dimensional OMM unit 708 may be implemented in various ways. Figure 8A schematic diagram of an example of a three-dimensional OMM cell 708 using a three-dimensional arrangement of diffractive components is shown. The three-dimensional OMM cell 708 may include an input waveguide matrix for receiving an optical input matrix 802, a three-dimensional optical interference cell 804 in optical communication with the input waveguide matrix, and an output waveguide matrix in optical communication with the optical interference cell 804 for providing an optical output matrix 806. The optical interference cell 804 includes a plurality of diffractive optical components and performs a transformation (e.g., a linear transformation) of an optical input (e.g., an N×N vector or matrix) to an optical output (e.g., an N×N vector or matrix). The output waveguide matrix guides the optical signal output by the optical interference cell 804. At least one input waveguide in the input waveguide matrix is in optical communication with each output waveguide in the output waveguide matrix through the optical interference cell 804. For example, for an optical input vector of length N×N, the three-dimensional OMM cell 708 may include N×N input waveguides and N×N output waveguides.
[0480] In some embodiments, the optical interference unit 804 includes a substrate block having a diffractive component, which is arranged in three dimensions (e.g., in a 3D matrix). For example, multiple holes can be drilled or etched in each of a plurality of substrate slices, and a plurality of substrate slices can be combined to form a substrate block. The size of these holes can be comparable in magnitude to the size of the wavelength of the input light, so that the light is diffracted by the holes (or the structure defining the holes). The holes can have the same or different sizes. The holes can also have other cross-sectional shapes, such as triangles, squares, rectangles, hexagons, or irregular shapes. In some embodiments, a holographic method can be used to form a three-dimensional diffractive optical component throughout the substrate block. The substrate can be made of a material that is transparent or translucent to the input light, for example, having a transmittance of 1% to 99% relative to the input light.
[0481] When designing a three-dimensional OMM unit 708, the size and position of the diffractive component in the x, y and z directions are considered. An optimization process can be used to determine the configuration of the diffractive optical component. For example, the substrate block can be divided into a three-dimensional pixel matrix, and each pixel can be filled with substrate material (no holes) or filled with air (holes). The configuration of the pixels can be modified iteratively, and for each configuration of the pixels, simulations can be performed by passing light through the diffractive optical component and evaluating the output. After performing simulations of all possible configurations of the pixels, the configuration that provides the result closest to the desired matrix process is selected as the diffractive optical component configuration of the three-dimensional OMM unit 708.
[0482] As another example, the diffractive component is initially configured as a three-dimensional matrix of holes. The positions, sizes, and shapes of the holes may vary slightly from their initial configuration. The parameters of each hole may be iteratively adjusted, and simulations may be performed to find an optimized configuration of the holes.
[0483] In some embodiments, a machine learning process is used to design a three-dimensional diffractive optical component. An analytical function of how pixels affect input light is determined, and a gradient descent method is used to determine the optimal configuration of the pixels.
[0484] In some embodiments, the three-dimensional OMM unit 708 may be implemented as a user-variable component, and different three-dimensional OMM units 708 with different optical interference units 804 may be installed for different applications. For example, the system 700 may be configured as a medical diagnostic system, and the optical interference unit 804 may be configured to implement a neural network for analyzing physiological parameters to perform screening for diseases. For example, the first OMM unit may have a first optical interference unit, which includes a 3D passive diffractive optical component, which is configured to implement a first neural network for screening a first group of diseases. The second OMM unit may have a second optical interference unit, which includes a 3D passive diffractive optical component, which is configured to implement a second neural network for screening a second group of diseases, and so on. The first and second OMM units may be developed by different companies that specialize in developing technologies for screening different diseases. When a user wants to use the system 700 to screen for a first group of diseases, the user can insert the first OMM unit into the system. When the user wants to use the system 700 to screen for a second group of diseases, the user can swap out the first OMM unit and insert the second OMM unit into the system.
[0485] For example, the system 700 may be configured as an optical character recognition system, and the optical interference unit 804 may be configured to implement a neural network for performing optical character recognition. For example, the system 700 may be configured as a speech recognition system, and the optical interference unit 804 may be configured to implement a neural network for performing speech recognition. For example, the system 700 may be part of a control unit of an autonomous vehicle, and the optical interference unit 804 may be configured to implement a neural network for performing road condition recognition.
[0486] For example, system 700 can be used for gene sequencing. DNA sequences can be classified using convolutional neural networks, which are implemented using system 700 including passive diffractive optical components. For example, system 700 can implement a neural network for distinguishing tumor types, predicting tumor grades, and predicting patient survival from gene expression patterns. For example, system 700 can implement a neural network for identifying a subset of genes or features that are most predictive of the analyzed characteristics. For example, system 700 can implement a neural network for predicting or inferring the expression levels of all genes from a data map of a subset of genes. For example, system 700 can implement a neural network for epigenetic analysis, such as predicting transcription factor binding sites, enhancer regions, and chromatin accessibility from gene sequences. For example, system 700 can implement a neural network for capturing structures within gene sequences. For example, system 700 can be configured as a bacterial detection system, and optical interference unit 804 can be configured to implement a multiplication function for analyzing DNA sequences to detect certain bacterial strains.
[0487] In some embodiments, the three-dimensional OMM unit 708 includes a housing (e.g., a cassette) that protects a substrate having a 3D diffractive optical component. The housing supports an input interface coupled to an input waveguide, and an output interface coupled to an output waveguide. The input interface is configured to receive an output from the modulator array 706, and the output interface is configured to send the output of the three-dimensional OMM unit 708 to a detection unit 710. The three-dimensional OMM unit 708 can be designed as a module suitable for handling by an ordinary consumer, allowing a user to easily switch from one three-dimensional OMM unit 708 to another three-dimensional OMM unit 708. Machine learning technology improves over time. A user can upgrade the system 700 by swapping out an old three-dimensional OMM unit 708 and inserting a new upgraded version.
[0488] In some embodiments, system 700 is an optical computing platform that is configured to operate with OMM units provided by different companies. This allows different companies to develop different 3D passive optical neural networks for various applications. The 3D passive optical neural network is sold to end users in standardized packaging, which can be installed in the optical computing platform to allow system 700 to perform various intelligent functions.
[0489] In some embodiments, the system may have a holder mechanism for supporting multiple three-dimensional OMM units 708, and a mechanical handling mechanism may be provided for automatically swapping out three-dimensional OMM units 708. The system determines which three-dimensional OMM unit 708 is required for the current application, and automatically obtains the appropriate three-dimensional OMM unit 708 from the holder mechanism and inserts it into the optical processor 702 using the mechanical handling mechanism.
[0490] In some embodiments, the artificial neural network computing system 700 may be modified by adding an analog nonlinear unit between the detection unit 710 and the ADC unit 160. The analog nonlinear unit is configured to receive the output voltage from the detection unit 710, apply a nonlinear transfer function, and output the converted output voltage to the ADC unit 160. The controller 110 may obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 160. Because the digital output voltage obtained from the ADC unit 160 has been nonlinearly converted ("activated"), the nonlinear conversion step of the controller 110 may be omitted, thereby reducing the computational burden of the controller 110. Then, the first converted voltage directly obtained from the ADC unit 160 may be stored in the storage unit 120 as a first converted digital output vector.
[0491] The optical interference unit can be realized using passive diffractive optical components arranged in one dimension. Fig. 9 In some embodiments, the artificial neural network computing system 900 has an optical processor 906 that includes a one-dimensional optical multiplication unit 916. The system 900 includes a storage unit 120 that is similar to Figure 1A The optical processor 906 is configured to perform the multiplication calculation using diffractive optical components arranged in one dimension (along the light propagation axis).
[0492] The optical processor 906 includes a laser unit 908 configured to output a laser beam 910 and a modulator 912 configured to modulate the laser beam 910 to generate a modulated light beam 914. The optical processor 906 includes a one-dimensional optical multiplication unit 916 having a one-dimensionally arranged diffractive optical component and configured to process the modulated light beam 914 and generate an output light beam 918. The optical processor 906 includes a detection unit 920 having a light sensor for detecting the output light beam 918. The output of the detection unit 920 is converted into a digital signal by an ADC unit 930.
[0493] For example, the one-dimensional optical multiplication unit 916 may be implemented as a passive integrated silicon photonic waveguide with diffractive optical elements (eg, gratings or holes). The one-dimensional optical multiplication unit 916 may be configured to perform multiplication operations with almost zero power consumption.
[0494] There are many ways to encode input data for use by the optical processor 906. For example, a digital input vector can be encoded as an optical input that propagates through a one-dimensional optical multiplication unit 916. The one-dimensional optical multiplication unit 916 performs multiplication on the received optical input in the optical domain. The multiplication performed by the one-dimensional optical multiplication unit 916 is determined by the internal configuration of the one-dimensional optical multiplication unit 916, including, for example, the size, position, and geometry of the diffractive optical components arranged in one dimension along the optical propagation path, as well as the doping of impurities (if any).
[0495] The one-dimensional optical multiplication unit 916 may be implemented in various ways. Fig.10 A schematic diagram of an example of a one-dimensional optical multiplication unit 916 using a one-dimensional arrangement of diffractive components is shown. The one-dimensional optical multiplication unit 916 may include an input waveguide for receiving an optical input 1002, a one-dimensional optical interference unit 1004 in optical communication with the input waveguide, and an output waveguide in optical communication with the optical interference unit 1004 for providing an optical output 1006. The optical interference unit 1004 includes a plurality of diffractive optical components and performs a conversion (e.g., linear conversion) of the optical input to the optical output. The output waveguide guides the optical signal output by the optical interference unit 1004.
[0496] In some embodiments, the optical interference unit 1004 includes an elongated substrate having a diffractive component arranged in one dimension along the light propagation path. For example, a plurality of holes can be drilled or etched in the substrate. The size of these holes can be comparable to the size of the wavelength of the input light in order of magnitude, so that the light is diffracted by the holes (or the structure defining the holes). The holes can have the same or different sizes. The substrate can be made of a material that is transparent or translucent to the input light, for example, having a transmittance of 1% to 99% relative to the input light. In some embodiments, a holographic method can also be used to form a diffractive optical component in a substrate.
[0497] When designing the light interference unit 1004, the size and position of the diffractive component along the propagation path of the light beam are considered. The configuration of the diffractive optical component can be determined using an optimization process. For example, the substrate can be divided into a series of pixels, and each pixel can be filled with a substrate material (no holes) or filled with air (holes). The configuration of the pixel can be modified iteratively, and for each configuration of the pixel, a simulation can be performed by passing light through the diffractive optical component and evaluating the output. After performing the simulation of all possible configurations of the pixel, the configuration that provides the result closest to the desired multiplication process is selected as the diffractive optical component configuration of the light interference unit 1004.
[0498] As another example, the diffractive component is initially configured as a series of holes. The positions and sizes of the holes may vary slightly from their initial configuration. The parameters of each hole may be iteratively adjusted, and simulations may be performed to find an optimal configuration of the holes.
[0499] In some embodiments, a machine learning process is used to design a one-dimensional diffractive optical component. An analytical function is determined for how pixels affect input light, and a gradient descent method is used to determine the optimal configuration of the pixels.
[0500] In some embodiments, the one-dimensional optical multiplication unit 916 may be implemented as a user-variable component, and different one-dimensional optical multiplication units 916 with different optical interference units 1004 may be installed for different applications. For example, the system 900 may be configured as a bacteria detection system, and the optical interference unit 1004 may be configured to implement a multiplication function for analyzing DNA sequences to detect certain bacterial strains. For example, the first optical multiplication unit may have a first optical interference unit, which includes a 1D passive diffraction optical component, which is configured to implement a first multiplication function for detecting a first group of bacteria. The second optical multiplication unit may have a second optical interference unit, which includes a 1D passive diffraction optical component, which is configured to implement a second multiplication function for detecting a second group of bacteria, and so on. The first and second optical multiplication units may be developed by different companies that specialize in developing technologies for detecting different bacteria. When a user wants to use the system 900 to detect a first group of bacteria, the user can insert the first optical multiplication unit into the system. When the user wants to use system 900 to detect a second group of bacteria, the user can swap out the first light multiplication unit and insert the second light multiplication unit into the system. By using one-dimensional diffractive optical components, laser unit 908, modulator 912, detection unit 920, and ADC unit 930 can be manufactured at low cost.
[0501] In some embodiments, the one-dimensional optical multiplication unit 916 includes a housing (e.g., a cassette) that protects a substrate having a 1D diffractive optical component. The housing supports an input interface coupled to an input waveguide, and an output interface coupled to an output waveguide. The input interface is configured to receive an output from the modulator 912, and the output interface is configured to send the output of the one-dimensional optical multiplication unit 916 to a detection unit 920. The one-dimensional optical multiplication unit 916 can be designed as a module suitable for handling by an ordinary consumer, allowing a user to easily switch from one one-dimensional optical multiplication unit 916 to another one-dimensional optical multiplication unit 916. Machine learning technology improves over time. A user can upgrade the system 900 by swapping out an old one-dimensional optical multiplication unit 916 and inserting a new upgraded version.
[0502] In some embodiments, system 900 is an optical computing platform that is configured to operate with optical multiplication units provided by different companies. This allows different companies to develop different 1D passive optical multiplication functions for various applications. The 1D passive optical multiplication functions are sold to end users in standardized packages that can be installed in an optical computing platform to allow system 900 to perform various intelligent functions.
[0503] In some embodiments, the system may have a holder mechanism for supporting multiple one-dimensional light multiplication units 916, and a mechanical handling mechanism may be provided for automatically swapping out the one-dimensional light multiplication unit 916. The system determines which one-dimensional light multiplication unit 916 is required for the current application, and automatically takes the appropriate one-dimensional light multiplication unit 916 from the holder mechanism and inserts it into the light processor 906 using the mechanical handling mechanism.
[0504] In some embodiments, the artificial neural network computing system 900 can be modified by adding an analog nonlinear unit between the detection unit 920 and the ADC unit 930. The analog nonlinear unit is configured to receive the output voltage from the detection unit 920, apply a nonlinear transfer function, and output the converted output voltage to the ADC unit 930. The controller 902 can obtain a converted digital output voltage corresponding to the converted output voltage from the ADC unit 930. Because the digital output voltage obtained from the ADC unit 930 has been nonlinearly converted ("activated"), the nonlinear conversion step of the controller 902 can be omitted, thereby reducing the computational burden of the controller 902. Then, the first converted voltage directly obtained from the ADC unit 930 can be stored in the storage unit 120 as a first converted digital output vector.
[0505] Passive chips with passive diffractive optical components have many advantages. First, because the active components (usually the bulkiest parts) have been eliminated, a chip of any given size can contain a larger neural network. Typically useful neural networks may include millions of weights, which are challenging to implement on active chips and may require multiple data runs through the chip and reprogramming of the chip. In contrast, a single passive chip may be able to support an entire neural network. Second, the very low power consumption of passive chips is important for "edge" applications because such applications may require a small footprint and low power consumption. Third, passive chips can be manufactured at a very low cost because they do not contain active components.
[0506] The optical matrix multiplication unit with passive diffractive optical components can also be used in a wavelength division multiplexing artificial neural network computing system. For example, Figure 1FThe OMM unit 150 of the system 104 in FIG. 1 may be replaced by an OMM unit using a passive diffractive optical component. In this example, the second DAC subunit 134 may be removed.
[0507] In some embodiments, the optical processors (eg, 504, 702) may perform matrix processing other than matrix multiplication. The optical matrix multiplication units 502 and 708 may be replaced by optical matrix processing units that perform other types of matrix processing.
[0508] Fig.25 A flow chart is shown of an example of a method 2500 for performing ANN computations using an ANN computing system 500, 700, or 900 that includes one or more optical matrix multiplication units or optical multiplication units with passive diffractive components, such as a 2D OMM unit 502, a 3D OMM unit 708, or a 1D OM unit 916. The steps of the process 2500 may be performed at least in part by a controller 110 or 902. In some embodiments, the various steps of the method 2500 may be run in parallel, combined, looped, or in any order.
[0509] In step 2510, an artificial neural network (ANN) calculation request including an input data set is received. The input data set includes a first digital input vector. The first digital input vector is a subset of the input data set. For example, it can be a sub-region of an image. The ANN calculation request can be generated by various entities (e.g., computer 102). The computer can include one or more of various types of computing devices, such as a personal computer, a server computer, a vehicle computer, and a flight computer. The ANN calculation request generally refers to an electrical signal that notifies or informs the ANN computing system 500, 700, or 900 that the ANN calculation is to be performed. In some embodiments, the ANN calculation request can be divided into two or more signals. For example, the first signal can query the ANN computing system 500, 700, or 900 to check whether the system 500, 700, or 900 is ready to receive the input data set. In response to the affirmative response of the system 500, 700, or 900, the computer can send a second signal including the input data set.
[0510] In step 2520, the input data set is stored. The controller 110 may store the input data set in the storage unit 120. Storing the input data set in the storage unit 120 may allow flexibility in the operation of the ANN computing system 500, 700, or 900, for example, may improve the overall performance of the system. For example, by obtaining the desired portion of the input data set from the storage unit 120, the input data set may be divided into digital input vectors of a set size and format. Different portions of the input data set may be processed in various orders, or shuffled, to allow various types of ANN calculations to be performed. For example, in the case where the input and output matrices are of different sizes, shuffling may allow matrix multiplication to be performed by block matrix multiplication techniques. As another example, storing the input data set in the storage unit 120 may allow multiple ANN computing requests to be queued by the ANN computing system 500, 700, or 900, which may allow the system 500, 700, or 900 to maintain operation at its full speed without periods of inactivity.
[0511] In step 2530, a first plurality of modulator control signals are generated based on the first digital input vector. The controller 110 may send a first DAC control signal to the DAC unit 506, 712, or 904 to generate the first plurality of modulator control signals. The DAC unit 506, 712, or 904 generates the first plurality of modulator control signals based on the first DAC control signal, and the modulator array 144, 706, or the modulator 912 generates a light input vector representing the first digital input vector.
[0512] The first DAC control signal may include a plurality of digital values to be converted by the DAC unit 506, 712, or 904 into a first plurality of modulator control signals. The plurality of digital values generally correspond to a first digital input vector and may be associated by various mathematical relationships or lookup tables. For example, the plurality of digital values may be linearly proportional to the values of the elements of the first digital input vector. As another example, the plurality of digital values may be associated with the elements of the first digital input vector by a lookup table that is configured to maintain a linear relationship between the digital input vector and the light input vector generated by the modulator array 144, 706, or the modulator 912.
[0513] In some embodiments, the 2D OMM unit 502, the 3D OMM unit 708, or the 1D OM unit 916 is configured to perform optical matrix processing or optical multiplication based on the optical input vector and the plurality of neural network weights implemented using the passive diffractive component. The plurality of neural network weights representing the matrix M can be decomposed into M=USV* by a singular value decomposition (SVD) method, where U is an M×M unitary matrix, S is an M×N diagonal matrix with non-negative real numbers on the diagonal, and V* is the complex conjugate of the N×N unitary matrix V. In this case, the passive diffractive component can be configured to implement the matrix V, the matrix S, and the matrix U, so that the OMM unit 502 or 708 implements the matrix M as a whole.
[0514] In step 2540, a first plurality of digital optical outputs corresponding to the optical output vectors of the optical matrix multiplication unit or optical multiplication is obtained. The optical input vector generated by the modulator array 144, 706 or the modulator 912 is processed by the 2D OMM unit 502, the 3D OMM unit 708 or the 1D OM unit 916 and converted into an optical output vector. The optical output vector is detected by the detection unit 146, 710 or 920 and converted into an electrical signal, which can be converted into a digital value by the ADC unit 160 or 930. The controller 110 or 902 can, for example, send a conversion request to the ADC unit 160 or 930 to start converting the voltage output by the detection unit 146, 710 or 920 into a digital optical output. Once the conversion is completed, the ADC unit 160 or 930 can send the conversion result to the controller 110 or 902. Alternatively, the controller 110 or 902 can obtain the conversion result from the ADC unit 160 or 930. The controller 110 or 902 may form a digital output vector from the digital light output, the digital output vector corresponding to the result of a matrix multiplication or a vector multiplication of the input digital vector. For example, the digital light output may be organized or connected to have a vector format.
[0515] In some embodiments, the ADC unit 160 or 930 may be set or controlled to perform ADC conversion based on a DAC control signal issued by the controller 110 or 902 to the DAC unit 506, 712, or 904. For example, the ADC conversion may be set to start at a preset time after the DAC unit 506, 712, or 904 generates the modulation control signal. Such control of the ADC conversion may simplify the operation of the controller 110 or 902 and reduce the number of necessary control operations.
[0516] In step 2550, a nonlinear transformation is performed on the first digital output vector to produce a first transformed digital output vector. The nodes or artificial neurons of the ANN operate by first performing a weighted sum of the signals received from the nodes of the previous layer, and then performing a nonlinear transformation ("activation") of the weighted sum to produce an output. Various types of ANNs can implement various types of differentiable nonlinear transformations. Examples of nonlinear transformation functions include a rectified linear unit (RELU) function, a sigmoid function, a hyperbolic tangent function, an X^2 function, and an |X| function. This nonlinear transformation is performed on the first digital output by the controller 110 or 902 to produce a first transformed digital output vector. In some embodiments, the nonlinear transformation may be performed by a dedicated digital integrated circuit within the controller 110 or 902. For example, the controller 110 or 902 may include one or more modules or circuit blocks that are particularly suitable for accelerating the calculation of one or more types of nonlinear transformations.
[0517] In step 2560, the first converted digital output vector is stored. The controller 110 or 902 may store the first converted digital output vector in the storage unit 120. In the case where the input data set is divided into multiple digital input vectors, the first converted digital output vector corresponds to the ANN calculation result of a portion of the input data set, such as the first digital input vector. In this way, storing the first converted digital output vector allows the ANN computing system 500, 700, or 900 to perform and store additional calculations on other digital input vectors of the input data set to be aggregated into a single ANN output at a later time.
[0518] In step 2570, an artificial neural network output generated based on the first transformed digital output vector is output. The controller 110 or 902 generates an ANN output, which is the result of processing the input data set by the ANN defined by the first plurality of neural network weights. In the case where the input data set is divided into multiple digital input vectors, the generated ANN output is an aggregate output including the first transformed digital output, but may further include additional transformed digital outputs corresponding to other parts of the input data set. Once the ANN output is generated, the generated output is sent to the computer (e.g., computer 102) that initiated the ANN calculation request.
[0519] The 2D OMM cell 502, 3D OMM cell 708 or 1D OM cell 916 may represent the weight coefficients of one hidden layer of the neural network. If the neural network has multiple hidden layers, additional 2D OMM cells 502, 3D OMM cells 708 or 1D OM cells 916 may be coupled in series. Fig.26An example of an ANN computing system 2600 for implementing a neural network with two hidden layers is shown. A first 2D optical matrix multiplication unit 2604 represents a weight coefficient of the first hidden layer, and a second 2D optical matrix multiplication unit 2606 represents a weight coefficient of the second hidden layer. The ANN computing system 2600 includes a controller 110, a storage unit 120, a DAC unit 506, and an optoelectronic processor 2602. The storage unit 120 and the DAC unit 506 are similar to Figure 5 The optoelectronic processor 2602 is configured to perform matrix calculations using optical and electronic components.
[0520] The optoelectronic processor 2602 includes a first laser unit 142a, a first modulator array 144a, a first 2D optical matrix multiplication unit 2604, a first detection unit 146a, a first analog nonlinear unit 310a, an analog storage unit 320, a second laser unit 142b, a second modulator array 144b, a second 2D optical matrix multiplication unit 2606, a second detection unit 146b, a second analog nonlinear unit 310b, and an ADC unit 160. The operation of the first laser unit 142a, the first modulator array 144a, the first detection unit 146a, the first analog nonlinear unit 310a, and the analog storage unit 320 is similar to that of the first laser unit 142a. Figure 3B The first 2D OMM unit 2604 is similar to Figure 5 2D OMM 502. The output of the analog storage unit 320 drives the second modulator array 144b, which modulates the laser light from the second laser unit 142b to generate a light vector. The light vector from the second modulator array 144b is processed by the second 2D OMM unit 2606, which performs matrix multiplication and generates a light output vector, which is detected by the second detection unit 246b. The second detection unit 246b is configured to generate an output voltage corresponding to the light signal of the light output vector from the second 2D OMM unit 2606. The ADC unit 160 is configured to convert the output voltage into a digital output voltage. The controller 110 can obtain a digital output corresponding to the light output vector of the second 2D OMM unit 2606 from the ADC unit 160. The controller 110 can form a digital output vector from the digital output, and the digital output vector corresponds to the result of the second matrix multiplication of the nonlinear transformation of the result of the first matrix multiplication of the input digital vector. The second laser unit 142b is combined with the first laser unit 142a by using a light splitter to redirect some of the light from the first laser unit 142a to the second modulator array 144b.
[0521] The above principles can be applied to implement a neural network with three or more hidden layers, where the weight coefficients of each hidden layer are represented by the corresponding 2D OMM unit.
[0522] Fig. 27 An example of an ANN computing system 2700 for implementing a neural network with two hidden layers is shown. A first 3D optical matrix multiplication unit 2704 represents a weight coefficient of the first hidden layer, and a second 3D optical matrix multiplication unit 2706 represents a weight coefficient of the second hidden layer. The ANN computing system 2700 includes a controller 110, a storage unit 120, a DAC unit 712, and an optoelectronic processor 2702. The storage unit 120 and the DAC unit 712 are similar to Figure 7 The optoelectronic processor 2702 is configured to perform matrix calculations using optical and electronic components.
[0523] The optoelectronic processor 2702 includes a first laser unit 704a, a first modulator array 706a, a first 3D light matrix multiplication unit 2704, a first detection unit 710a, a first analog nonlinear unit 310a, an analog storage unit 320, a second laser unit 704b, a second modulator array 706b, a second 3D light matrix multiplication unit 2706, a second detection unit 710b, a second analog nonlinear unit 310b, and an ADC unit 160. The operations of the first laser unit 704a, the first modulator array 706a, the first detection unit 710a, the first analog nonlinear unit 310a, and the analog storage unit 320 are similar to those of FIG. Figure 3B The first 3D OMM unit 2704 is similar to Figure 73D OMM 708. The output of the analog storage unit 320 drives the second modulator array 706b, which modulates the laser light from the second laser unit 704b to generate a light vector. The light vector from the second modulator array 706b is processed by the second 3D OMM unit 2706, which performs matrix multiplication and generates a light output vector, which is detected by the second detection unit 710b. The second detection unit 710b is configured to generate an output voltage corresponding to the light signal of the light output vector from the second 3D OMM unit 2706. The ADC unit 160 is configured to convert the output voltage into a digital output voltage. The controller 110 can obtain a digital output corresponding to the light output vector of the second 3D OMM unit 2406 from the ADC unit 160. The controller 110 can form a digital output vector from the digital output, and the digital output vector corresponds to the result of the second matrix multiplication of the nonlinear transformation of the result of the first matrix multiplication of the input digital vector. The second laser unit 704b may be combined with the first laser unit 704a by redirecting some of the light from the first laser unit 704a to the second modulator array 706b using a light splitter.
[0524] The above principles can be applied to implement a neural network with three or more hidden layers, where the weight coefficient of each hidden layer is represented by a corresponding 3D OMM unit.
[0525] The 2D OMM unit 502 and the 3D OMM unit 708 with passive diffractive optical components are suitable for use in a recursive neural network (RNN), wherein the output of the network during the (k)th pass through the neural network is recycled back to the input of the neural network and used as input during the (k+1)th pass, so that the weight coefficients of the neural network remain the same during multiple passes.
[0526] Fig.28 An example of a neural network computing system 2800 is shown, which can be used to implement a recurrent neural network. The system 2800 includes an optical processor 2802, which is similar to Figure 3B The optical processor 140 operates in the same manner as the optical processor 140, except that the OMM unit 150 is replaced by a 2D OMM unit 2804. The 2D OMM unit 2804 can be similar to Figure 6 The neural network weights of the 2D OMM unit 2804 are fixed, so the system 2800 does not need to be Figure 3B The second DAC subunit 134 used in the system 302.
[0527] Fig.29 An example of a neural network computing system 2900 is shown, which can be used to implement a recurrent neural network. The system 2900 includes an optical processor 2902, which is similar to Figure 3B The optical processor 140 is operated in the same manner as the optical processor 140, except that the laser unit 142, the modulator array 144, the OMM unit 150 and the detection unit 146 are respectively Figure 7 The laser unit 704, modulator array 706, 3D OMM unit 2904 and detection unit 710 are replaced. The neural network weights of the 3D OMM unit 2904 are fixed, so the system 2900 does not need to be Figure 3B The second DAC subunit 134 used in the system 302.
[0528] Fig.30 A schematic diagram of an example of an artificial neural network computing system 3000 having an internal resolution of 1 bit is shown. The ANN computing system 3000 is similar to Figure 4A The ANN computing system 400 is a 2D OMM unit 3004 (which is similar to Figure 5 The ANN computing system 3000 is replaced by the 2D OMM unit 502 of FIG. 4 , and the second driver subunit 434 is omitted. The ANN computing system 3000 operates in a manner similar to the ANN computing system 400, where the input vector is decomposed into multiple 1-bit vectors, and then certain ANN calculations can be performed by summing the matrix multiplication results after performing a series of matrix multiplications of the 1-bit vectors.
[0529] Fig.31 A schematic diagram of an example of an artificial neural network computing system 3100 having an internal resolution of 1 bit is shown. The ANN computing system 3100 is similar to Figure 4A The ANN computing system 400 includes the OMM unit 150 and the 3D OMM unit 3104 (which is similar to Figure 7 The 3D OMM unit 708 is replaced, and the second driver subunit 434 is omitted. Fig.31 In the example, Figure 4A The laser unit 142, the modulator array 144 and the detection unit 146 are respectively composed of Figure 7 The laser unit 704, the modulator array 706, and the detection unit 710 are replaced by the laser unit 704, the modulator array 706, and the detection unit 710. The ANN computing system 3100 operates in a manner similar to the ANN computing system 400, where the input vector is decomposed into multiple 1-bit vectors, and then certain ANN calculations can be performed by summing the matrix multiplication results after performing a series of matrix multiplications of the 1-bit vectors.
[0530] The principle of optical diffraction neural network is described below. The optical diffraction neural network can be implemented as several layers of diffraction or transmission optical media. Based on the Huygens-Fresnel principle, each point in the diffraction medium can be considered as a secondary light source. For each light source, the farfield diffraction can be described by the following equation:
[0531]
[0532] Here, the indices l and i refer to the i-th neuron in the l-th layer of the neural network, λ is the wavelength of light, and r is the distance, where:
[0533]
[0534] The output from each secondary light source can be written as the input multiplied by the phase...
Claims
1. An optoelectronic computing system, comprising: A first unit configured to generate a plurality of modulator control signals; Processing unit, comprising: a light source configured to provide a plurality of light outputs; a plurality of optical modulators coupled to the light source and the first unit, the plurality of optical modulators being configured to generate a light input vector by modulating the plurality of light outputs provided by the light source based on the plurality of modulator control signals, the light input vector comprising a plurality of light signals; and a matrix multiplication unit coupled to the plurality of optical modulators and the first unit, the matrix multiplication unit being configured to convert the optical input vector into an analog output vector based on a plurality of weight control signals; a second unit coupled to the matrix multiplication unit and configured to convert the analog output vector into a digital output vector; and The controller, including an integrated circuit, is configured to perform the following operations: receiving an artificial neural network computation request, the artificial neural network computation request comprising an input data set, the input data set comprising a first digital input vector; receiving a first plurality of neural network weights; and generating, by the first unit, a first plurality of modulator control signals based on the first digital input vector, and generating a first plurality of weight control signals based on the first plurality of neural network weights; Wherein the matrix multiplication unit comprises: a plurality of replication modules, wherein each of the replication modules corresponds to a subset of one or more optical signals of the optical input vector and is configured to split the subset of one or more optical signals into two or more copies of the optical signals; a plurality of multiplication modules, wherein each of the multiplication modules corresponds to a subset of one or more optical signals and is configured to multiply the one or more optical signals of the subset by one or more matrix element values using optical amplitude modulation; and One or more summing modules, wherein each summing module is configured to generate an electrical signal representing a sum of the results of two or more of the multiplication modules.
2. The optoelectronic computing system according to claim 1, wherein the first unit, the processing unit, the second unit, and the controller are provided on at least one of a multi-chip module or an integrated circuit, and Receiving an artificial neural network computation request includes receiving an artificial neural network computation request from a second data processor, wherein the second data processor is external to the multi-chip module or the integrated circuit, the second data processor is coupled to the multi-chip module or the integrated circuit via a communication channel, and the processing unit is configured to process data at a data rate at least one order of magnitude greater than a data rate of the communication channel.
3. The electro-optical computing system of claim 1 , wherein the matrix multiplication unit comprises an electrical summation module configured to perform an electrical summation operation, The electrical summation module is configured to generate currents corresponding to elements of the analog output vector, wherein the analog output vector represents the sum of corresponding elements of the light input vector multiplied by corresponding neural network weights.
4. An optoelectronic computing system according to any one of claims 1 to 3, wherein at least one of the multiplication modules comprises an optical amplitude modulator, the optical amplitude modulator comprising an input port and two output ports, and providing a pair of correlated optical signals from the two output ports such that the difference between the amplitudes of the correlated optical signals corresponds to the result of multiplying the input value by the signed matrix element value.
5. An electro-optical computing system according to any one of claims 1 to 3, wherein the matrix multiplication unit is configured to multiply the input vector by a matrix comprising the one or more matrix element values, A set of multiple output values is encoded on corresponding electrical signals generated by the one or more summing modules, and the output values in the set of multiple output values represent elements of an output vector, which is generated by multiplying the input vector by the matrix.
6. An electro-optical computing system according to any one of claims 1 to 3, wherein the replication module comprises at least one replication module having an optical splitter, the optical splitter being configured to send a predetermined proportion of the power of the light wave at an input port to a first output port, and to send a remaining proportion of the power of the light wave at the input port to a second output port.
7. The optoelectronic computing system according to any one of claims 1 to 3, comprising a storage unit configured to store the input data set and the neural network weights, wherein the second unit comprises an analog-to-digital conversion ADC unit, and the operations further comprise: obtaining a first plurality of digital outputs corresponding to the analog output vector of the matrix multiplication unit from the ADC unit, the first plurality of digital outputs forming a first digital output vector; performing a non-linear transformation on the first digital output vector to produce a first transformed digital output vector; as well as storing the first converted digital output vector in the storage unit; wherein the system has a first cycle period, the first cycle period being defined as the time elapsed between the step of storing the input data set and the first plurality of neural network weights in the memory unit and the step of storing the first transformed digital output vector in the memory unit, and The first cycle period is less than or equal to 1 ns.
8. The optoelectronic computing system of any one of claims 1 to 3, comprising a storage unit configured to store the input data set and the neural network weights, wherein the second unit comprises an analog-to-digital conversion ADC unit, and the operations further comprise: obtaining a first plurality of digital outputs corresponding to the analog output vector of the matrix multiplication unit from the ADC unit, the first plurality of digital outputs forming a first digital output vector; performing a non-linear transformation on the first digital output vector to produce a first transformed digital output vector; as well as storing the first converted digital output vector in the storage unit; Wherein the first unit comprises a digital-to-analog converter (DAC) unit, the artificial neural network calculation request further comprises a second plurality of neural network weights, and the operation further comprises: generating, by the DAC unit, a second plurality of modulator control signals based on the first converted digital output vector, generating a second plurality of weight control signals by the DAC unit based on the first plurality of digital outputs and the second plurality of neural network weights, The first plurality of neural network weights and the second plurality of neural network weights correspond to different layers of an artificial neural network.
9. An optoelectronic computing system according to any one of claims 1 to 3, wherein the system comprises a storage unit configured to store the input data set and the neural network weights, the second unit comprises an analog-to-digital conversion ADC unit, and the system further comprises: an analog nonlinear unit disposed between the matrix multiplication unit and the ADC unit, the analog nonlinear unit being configured to receive a plurality of output voltages from the matrix multiplication unit, apply a nonlinear transfer function, and output a plurality of converted output voltages to the ADC unit, The operations performed by the integrated circuit of the controller further include: Obtaining a first plurality of converted digital output voltages corresponding to the plurality of converted output voltages from the ADC unit, the first plurality of converted digital output voltages forming a first converted digital output vector; and The first converted digital output vector is stored in the storage unit.
10. The electro-optical computing system of any one of claims 1 to 3, wherein the integrated circuit of the controller is configured to generate the first plurality of modulator control signals at a rate greater than or equal to 8 GHz.
11. The optoelectronic computing system according to any one of claims 1 to 3, wherein the first unit comprises a digital-to-analog conversion DAC unit, the second unit comprises an analog-to-digital conversion ADC unit, and the matrix multiplication unit comprises: an optical matrix multiplication unit coupled to the plurality of optical modulators and the DAC unit, the optical matrix multiplication unit being configured to convert the optical input vector into an optical output vector based on the plurality of weight control signals; as well as a photoelectric detection unit, coupled to the optical matrix multiplication unit and configured to generate a plurality of output voltages corresponding to the optical output vector, an analog storage unit disposed between the DAC unit and the plurality of optical modulators, the analog storage unit being configured to store an analog voltage and output the stored analog voltage; as well as An analog nonlinear unit is disposed between the photodetection unit and the ADC unit, the analog nonlinear unit being configured to receive the plurality of output voltages from the photodetection unit, apply a nonlinear transfer function, and output a plurality of converted output voltages.
12. The optoelectronic computing system according to claim 11, wherein the analog storage unit is configured to receive and store a plurality of conversion output voltages of the analog nonlinear unit, and output the stored plurality of conversion output voltages to the plurality of optical modulators, and The operations further include: storing a plurality of converted output voltages of the analog nonlinear unit in the analog storage unit based on generating the first plurality of modulator control signals and the first plurality of weight control signals; outputting the stored converted output voltage via the analog storage unit; Obtaining a second plurality of converted digital output voltages from the ADC unit, the second plurality of converted digital output voltages forming a second converted digital output vector; as well as The second converted digital output vector is stored in the storage unit.
13. The electro-optical computing system of claim 11, wherein the system comprises a storage unit configured to store the input data set and the neural network weights, the input data set of the artificial neural network computation request comprising a plurality of digital input vectors, wherein the light source is configured to generate a plurality of wavelengths, The plurality of optical modulators include: A light modulator group configured to generate a plurality of light input vectors, each light modulator group corresponding to one of the plurality of wavelengths and generating a corresponding light input vector having a corresponding wavelength; as well as an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector comprising the plurality of wavelengths, wherein the photodetection unit is further configured to demultiplex the plurality of wavelengths and generate a plurality of demultiplexed output voltages, and The operations include: obtaining a plurality of digital demultiplexed optical outputs from the ADC unit, the plurality of digital demultiplexed optical outputs forming a plurality of first digital output vectors, wherein each of the plurality of first digital output vectors corresponds to one of the plurality of wavelengths; performing a non-linear transformation on each of the plurality of first digital output vectors to generate a plurality of transformed first digital output vectors; and storing the plurality of converted first digital output vectors in the storage unit, Each of the plurality of digital input vectors corresponds to one of the plurality of light input vectors.
14. An optoelectronic computing system according to any one of claims 1 to 3, wherein the system comprises a storage unit configured to store the input data set and the neural network weights, the second unit comprises an analog-to-digital conversion ADC unit, and the artificial neural network calculation request comprises a plurality of digital input vectors, wherein the light source is configured to generate a plurality of wavelengths, The plurality of optical modulators include: A light modulator group configured to generate a plurality of light input vectors, each light modulator group corresponding to one of the plurality of wavelengths and generating a corresponding light input vector having a corresponding wavelength; as well as an optical multiplexer configured to combine the plurality of optical input vectors into a combined optical input vector comprising the plurality of wavelengths, The operations include: Obtaining a first plurality of digital light outputs corresponding to the light output vector from the ADC unit, the light output vector comprising a plurality of wavelengths, the first plurality of digital light outputs forming a first digital output vector; performing a non-linear transformation on the first digital output vector to produce a first transformed digital output vector; and The first converted digital output vector is stored in the storage unit.
15. The optoelectronic computing system according to any one of claims 1 to 3, wherein the first unit comprises a digital-to-analog conversion (DAC) unit, the second unit comprises an analog-to-digital conversion (ADC) unit, and the DAC unit comprises: A 1-bit DAC subunit configured to generate a plurality of 1-bit modulator control signals, The resolution of the ADC unit is 1 bit. wherein the resolution of the first digital input vector is N bits, and The operations include: Decomposing the first digital input vector into N 1-bit input vectors, each of the N 1-bit input vectors corresponding to one of the N bits of the first digital input vector; generating, by the 1-bit DAC subunit, a sequence of N 1-bit modulator control signals corresponding to the N 1-bit input vectors; Obtaining from the ADC unit a sequence of N digital 1-bit optical outputs corresponding to the sequence of N 1-bit modulator control signals; constructing an N-bit digital output vector from the sequence of N digital 1-bit optical outputs; performing a nonlinear transformation on the constructed N-bit digital output vector to generate a transformed N-bit digital output vector; and The converted N-bit digital output vector is stored in a storage unit.
16. An optoelectronic computing system according to any one of claims 1 to 3, wherein the multiplication module includes at least one coherence insensitive multiplication module, the coherence insensitive multiplication module being configured to multiply one or more optical signals of the first subset of the one or more optical signals by one or more matrix element values using optical amplitude modulation based on energy absorption within the optical wave.
17. The electro-optical computing system of claim 16, wherein the coherent insensitive multiplication module comprises an electro-absorption modulator.
18. An electro-optical computing system according to any one of claims 1 to 3, wherein the one or more summing modules include at least one summing module having the following components: two or more input conductors, each of the two or more input conductors carrying an electrical signal in the form of an input current, the magnitude of the input current representing a respective result of a respective one of the multiplication modules, and At least one output conductor carries an electrical signal representative of the sum of the respective results in the form of an output current, the output current being proportional to the sum of the input currents.
19. The electro-optical computing system of claim 18, wherein the two or more input conductors and the output conductor comprise wires that meet at one or more nodes between the wires, and the output current is substantially equal to the sum of the input currents.
20. The optoelectronic computing system of claim 18, wherein at least a first one of the input currents is provided in the form of at least one photocurrent generated by at least one photodetector that receives an optical signal generated by a first one of the multiplication modules.
21. The optoelectronic computing system of claim 20, wherein the first input current is provided in the form of a difference between two photocurrents, the two photocurrents being generated by different corresponding photodetectors that receive different corresponding optical signals generated by the first multiplication module.
22. An electro-optical computing system according to any one of claims 1 to 3, wherein one of the copies of the first subset of one or more optical signals consists of a single optical signal on which one of the input values is encoded.
23. The electro-optical computing system of claim 22, wherein the multiplication modules corresponding to the copies of the first subset are configured to multiply the encoded input value by a single matrix element value.
24. An electro-optical computing system according to any one of claims 1 to 3, wherein one of the copies of the first subset of one or more optical signals comprises more than one, and less than the number of all, optical signals on which a plurality of input values are encoded.
25. The electro-optical computing system of claim 24, wherein the multiplication modules corresponding to the copies of the first subset are configured to multiply the encoded input values by different corresponding matrix element values.
26. An optoelectronic computing system according to claim 25, wherein different multiplication modules corresponding to different respective copies of the first subset of one or more optical signals are included in different devices, and the different devices are in optical communication to transmit one of the copies of the first subset of one or more optical signals between the different devices.
27. An optoelectronic computing system according to any one of claims 1 to 3, wherein two or more of the plurality of optical waveguides, two or more of the plurality of replica modules, two or more of the plurality of multiplication modules, and at least one of the one or more summation modules are disposed on a substrate of a common device.
28. An electro-optical computing system according to claim 27, wherein the apparatus is configured to perform vector-matrix multiplication, wherein an input vector is provided as a set of optical signals and an output vector is provided as a set of electrical signals.
29. The electro-optical computing system of claim 27, wherein each of two or more of the multiplication modules corresponds to a different subset of the one or more optical signals.
30. The electro-optical computing system according to any one of claims 1 to 3, further comprising an accumulator, the accumulator combining input electrical signals corresponding to the outputs of the multiplication module or the summation module, The input electrical signal is encoded using time domain coding, the time domain coding uses switching amplitude modulation within each of multiple time slots, and the accumulator generates an output electrical signal, which is encoded with more than two amplitude levels, and the amplitude levels correspond to different duty cycles of the time domain coding over the multiple time slots.
