Photonic systems and methods

Abstract

To provide a photonic processing device and a method of performing matrix-vector multiplication.SOLUTION: One aspect relates to a photonic processing device and a method of performing matrix-vector multiplication. An optical encoder may encode an input vector into a first plurality of optical signals. A photonic processor may receive the first plurality of optical signals; perform a plurality of operations on the first plurality of optical signals, the plurality of operations implementing a matrix multiplication of the input vector by a matrix; and output a second plurality of optical signals representing an output vector. An optical receiver may detect the second plurality of optical signals and output an electrical digital representation of the output vector.SELECTED DRAWING: Figure 1-1

Description

[Background technology]

[0001] Traditional computing uses processors that contain circuits of millions of transistors and perform logic gates on bits of information represented by electrical signals. Traditional central processing unit (CPU) architectures are designed for general-purpose computing, but are not optimized for specific types of algorithms. Graphics processing, artificial intelligence, neural networks, and deep learning are a few examples of types of algorithms that are computationally intensive and do not run efficiently using CPUs.

[0002] As a result, specialized processors have been developed with architectures that are better suited for specific algorithms. Graphical processing units (GPUs), for example, have a highly parallel architecture that makes them more efficient than CPUs for performing image processing and graphical operations. After the development of GPUs for graphics processing, GPUs were found to be more efficient than GPUs for other memory-intensive algorithms as well, such as neural networks and deep learning. This realization, as well as the increasing popularity of artificial intelligence and deep learning, has led to further research into new electrical circuit architectures that can further enhance the speed of these algorithms. Summary of the Invention [Means for solving the problem]

[0003] In some embodiments, a photonic processor is provided that may include a first array of interconnected variable beam splitters (VBS) including a first plurality of optical inputs and a first plurality of optical outputs, a second array of interconnected VBS including a second plurality of optical inputs and a second plurality of optical outputs, and a plurality of controllable optical elements, each of a plurality of the controllable optical elements coupling a single one of the first plurality of optical outputs of the first array to a respective single one of the second plurality of optical inputs of the second array.

[0004] In some embodiments, a photonic processing system is provided. The photonic processing system may include an optical encoder configured to encode an input vector into a first plurality of optical signals. The photonic processing system may also include a photonic processor configured to receive the first plurality of optical signals, each of the first plurality of signals being received by a respective input spatial mode of a plurality of input spatial modes of the photonic processor, perform a plurality of operations on the first plurality of optical signals, the plurality of operations implementing a matrix multiplication of the input vector by a matrix, and output a second plurality of optical signals representing an output vector, each of the second plurality of signals being transmitted by a respective output spatial mode of a plurality of output spatial modes of the photonic processor. The photonic processing system may also include an optical receiver configured to detect the second plurality of optical signals and output an electrical digital representation of the output vector.

[0005] In some embodiments, a method for optically performing matrix-vector multiplication is provided, which may include receiving a digital representation of an input vector, encoding the input vector into a first plurality of optical signals using an optical encoder, performing singular value decomposition (SVD) of the matrix using a processor to determine a first SVD matrix, a second SVD matrix, and a third SVD matrix, controlling a photonic processor including a plurality of variable beam splitters (VBS) to optically perform the first SVD matrix, the second SVD matrix, and the third SVD matrix, propagating the first plurality of optical signals through the photonic processor, detecting the second plurality of optical signals received from the photonic processor, and determining an output vector based on the detected second plurality of optical signals, the output vector representing a result of the matrix-vector multiplication.

[0006] The above apparatus and method embodiments may be implemented in any suitable combination of the aspects, features, and operations described above or further below. These and other aspects, embodiments, and features of the present teachings may be more fully understood from the following description taken in conjunction with the accompanying drawings.

[0007] Various aspects and embodiments of the present application are described with reference to the following figures. It should be understood that the figures are not necessarily drawn to scale. Items that appear in more than one figure are designated with the same reference numeral in all figures in which they appear. [Brief description of the drawings]

[0008] [Figure 1-1] 1 is a schematic diagram of a photonic processing system according to some non-limiting embodiments. [Figure 1-2] 1 is a schematic diagram of an optical encoder according to some non-limiting embodiments. [Figure 1-3] 1 is a schematic diagram of a photonic processor according to some non-limiting embodiments. [Figure 1-4]FIG. 1 is a schematic diagram of an interconnected variable beamsplitter array according to some non-limiting embodiments. [Figure 1-5] 1 is a schematic diagram of a variable beam splitter according to some non-limiting embodiments. [Figure 1-6] FIG. 1 is a schematic diagram of a diagonal attenuation and phase shift implementation according to some non-limiting embodiments. [Figure 1-7] 1 is a schematic diagram of an attenuator according to some non-limiting embodiments. [Figure 1-8] FIG. 2 is a schematic diagram of a power tree according to some non-limiting embodiments. [Figure 1-9] 1 is a schematic diagram of an optical receiver according to some non-limiting embodiments. [Figure 1-10] FIG. 1 is a schematic diagram of a homodyne detector according to some non-limiting embodiments. [Figure 1-11] 1 is a schematic diagram of a folded photonic processing system according to some non-limiting embodiments. [Figure 1-12A] 1 is a schematic diagram of a wavelength-division-multiplexed (WDM) photonic processing system according to some non-limiting embodiments. [Fig.1-12B] FIG. 1-12B is a schematic diagram of a front-end of the wavelength division multiplexing (WDM) photonic processing system of FIGS. 1-12A according to some non-limiting embodiments. [Fig.1-12C] FIG. 1-12B is a schematic diagram of a back end of the wavelength division multiplexed (WDM) photonic processing system of FIGS. 1-12A according to some non-limiting embodiments. [Figure 1-13] FIG. 2 is a schematic diagram of a circuit that performs analog summation of optical signals, according to some non-limiting embodiments. [Figure 1-14] 1 is a schematic diagram of a photonic processing system showing a column-global topology according to some non-limiting embodiments. [Figure 1-15] 1 is a plot illustrating the effect of an uncorrected global phase shift on a homodyne detector, according to some non-limiting embodiments. [Figure 1-16]1 is a plot illustrating quadrature uncertainty of light in a coherent state according to some non-limiting embodiments. [Figure 1-17] FIG. 2 is an illustration of matrix multiplication in accordance with some non-limiting embodiments. [Figure 1-18] FIG. 13 is a diagram of performing matrix multiplication by splitting a matrix into sub-matrices, according to some non-limiting embodiments. [Figure 1-19] 1 is a flowchart of a method of manufacturing a photonic processing system according to some non-limiting embodiments. [Figure 1-20] 1 is a flow chart of a method of manufacturing a photonic processor according to some non-limiting embodiments. [Figure 1-21] 1 is a flowchart of a method for performing optical computation, according to some non-limiting embodiments. [Figure 2-1] 1 is a flowchart of a process for training a latent variable model, according to some non-limiting embodiments. [Figure 2-2] 1 is a flowchart of a process for configuring a photonic processing system to implement a unitary transfer matrix, according to some non-limiting embodiments. [Figure 2-3] 1 is a flowchart of a process for calculating an error vector using a photonic processing system in accordance with some non-limiting embodiments. [Figure 2-4] 1 is a flowchart of a process for identifying updated parameters of a unitary transfer matrix according to some non-limiting embodiments. [Figure 2-5] 1 is a flowchart of a process for updating parameters of a unitary transfer matrix according to some non-limiting embodiments. [Figure 3-1] 1 is a flowchart of a method for computing a forward pass through a convolutional layer, according to some non-limiting embodiments. [Figure 3-2] 1 is a flowchart of a method for computing a forward pass through a convolutional layer, according to some non-limiting embodiments. [Figure 3-3A]1 is a flowchart of a method suitable for computing a two-dimensional convolution, according to some non-limiting embodiments. [Figure 3-3B] 1 is a flowchart of a method suitable for constructing a circulant matrix, according to some non-limiting embodiments. [Figure 3-4A] 1 illustrates a pre-processing step of constructing a filter matrix from an input filter matrix that includes multiple output channels, according to some non-limiting embodiments. [Figure 3-4B] 1 illustrates construction of a circulant matrix from an input matrix containing multiple input channels, according to some non-limiting embodiments. [Figure 3-4C] 1 illustrates a two-dimensional matrix multiplication operation according to some non-limiting embodiments. [Figure 3-4D] 13 illustrates a post-processing step of rotating vector rows, according to some non-limiting embodiments. [Figure 3-4E] 4 illustrates post-processing steps of vector row addition according to some non-limiting embodiments. [Figure 3-4F] 1 illustrates reshaping of an output matrix into multiple output channels according to some non-limiting embodiments. [Figure 3-5] 1 is a flowchart of a method for performing a one-dimensional Fourier transform, according to some non-limiting embodiments. [Diagram 3-6] 1 is a flowchart of a method for performing a two-dimensional Fourier transform, according to some non-limiting embodiments. [Diagram 3-7] 1 is a flowchart of a method for performing a two-dimensional Fourier transform, according to some non-limiting embodiments. [Diagram 3-8] 1 is a flowchart of a method for performing a convolution using a Fourier transform, according to some non-limiting embodiments. [Figure 4-1] 1 is a block diagram illustrating an optical modulator according to some non-limiting embodiments. [Figure 4-2A] FIG. 1 is a block diagram illustrating an example of a photonic system in accordance with some non-limiting embodiments. [Figure 4-2B]1 is a flow chart illustrating an example of a method for processing signed real numbers in the optical domain, according to some non-limiting embodiments. [Figure 4-3A] FIG. 4-2B is a schematic diagram illustrating an example of a modulator that may be used in connection with the photonic system of FIG. 4-2A, according to some non-limiting embodiments. [Figure 4-3B] 4-3B is a plot showing the intensity and phase spectral response of the modulator of FIG. 4-3A when no voltage is applied, according to some non-limiting embodiments. [Figure 4-3C] 4-3B is a plot showing the intensity and phase spectral response of the modulator of FIG. 4-3A when driven at specific voltages according to some non-limiting embodiments. [Fig.4-3D] 4-3B is an example of a coding table associated with the modulator of FIG. 4-3A according to some non-limiting embodiments. [Figure 4-3E] 4-3B is a visual representation in the complex plane of the spectral response of the modulator of FIG. 4-3A according to some non-limiting embodiments. [Figure 4-4A] 1 is a visual representation in the complex plane of different spectral responses at the output of an optical transformation unit according to some non-limiting embodiments. [Figure 4-4B] 1 is a visual representation in the complex plane of different spectral responses at the output of an optical transformation unit according to some non-limiting embodiments. [Figure 4-4C] 1 is a visual representation in the complex plane of different spectral responses at the output of an optical transformation unit according to some non-limiting embodiments. [Figure 4-5] 1 is a visual representation in the complex plane of how detection of an optical signal may be performed according to some non-limiting embodiments. [Figure 4-6] 1 is an example of a decoding table according to some non-limiting embodiments. [Diagram 4-7] 1 is a block diagram of an optical communication system according to some non-limiting embodiments. [Figure 4-8]8 is a plot illustrating an example of a power spectral density output by the modulator of FIGS. 4-7 according to some non-limiting embodiments. [Figure 4-9] 1 is a flow chart illustrating an example method for fabricating a photonic system, according to some non-limiting embodiments. [Figure 5-1] FIG. 1 is a circuit diagram illustrating an example of a differential optical receiver in accordance with some non-limiting embodiments. [Figure 5-2] FIG. 5-2 is a schematic diagram illustrating a photonic circuit that can be combined with the differential optical receiver of FIG. 5-1 according to some non-limiting embodiments. [Figure 5-3A] FIG. 2 is a schematic diagram illustrating a substrate including a photonic circuit, a photodetector, and a differential op-amp, according to some non-limiting embodiments. [Figure 5-3B] FIG. 1 is a schematic diagram illustrating a first substrate including a photonic circuit and a photodetector and a second substrate including a differential op-amp, the first and second substrates being flip-chip bonded together according to some non-limiting embodiments. [Figure 5-3C] FIG. 1 is a schematic diagram illustrating a first substrate including a photonic circuit and a photodetector and a second substrate including a differential op-amp, the first and second substrates being wire-bonded together according to some non-limiting embodiments. [Figure 5-4] 1 is a flow chart illustrating an example method for assembling an optical receiver, according to some non-limiting embodiments. [Figure 5-4A] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-4B] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-4C] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-4D] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-4E] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-4F] 1 illustrates an example assembly sequence for an optical receiver according to some non-limiting embodiments. [Figure 5-5] 1 is a flow chart illustrating an example method of receiving an optical signal, according to some non-limiting embodiments. [Figure 6-1A] FIG. 1 is a schematic top view of a Nano-Opto-Electromechanical System (NOEMS) phase modulator according to some non-limiting embodiments. [Figure 6-1B] FIG. 6-1B is a top view schematic illustrating a suspended multi-slot optical structure of the NOEMS phase modulator of FIG. 6-1A according to some non-limiting embodiments. [Figure 6-1C] 6-1C is a plot illustrating an example of optical modes occurring in the suspended multi-slot optical structure of FIG. 6-1B, according to some non-limiting embodiments. [Figure 6-1D] FIG. 6-1B is a top view diagram that illustrates a schematic of the mechanical structure of the NOEMS phase modulator of FIG. 6-1A, according to some non-limiting embodiments. [Figure 6-1E] 6-1B is a schematic top view of a transition region of the NOEMS phase modulator of FIG. 6-1A according to some non-limiting embodiments. [Figure 6-2] FIG. 6-1B is a cross-sectional view of the NOEMS phase modulator of FIG. 6-1A showing a suspended waveguide in the yz plane according to some non-limiting embodiments. [Figure 6-3] FIG. 6-1B is a cross-sectional view of the NOEMS phase modulator of FIG. 6-1A showing a portion of a suspended multi-slot optical structure in the xy plane according to some non-limiting embodiments. [Figure 6-4A] 1A-1C are cross-sectional views illustrating how a suspended multi-slot optical structure can be mechanically actuated to vary the width of the slots between waveguides, according to some non-limiting embodiments. [Figure 6-4B] 1A-1C are cross-sectional views illustrating how a suspended multi-slot optical structure can be mechanically actuated to vary the width of the slots between waveguides, according to some non-limiting embodiments. [Figure 6-4C]1A-1C are cross-sectional views illustrating how a suspended multi-slot optical structure can be mechanically actuated to vary the width of the slots between waveguides, according to some non-limiting embodiments. [Figure 6-5] 1 is a plot showing how the effective refractive index of a suspended multi-slot optical structure can be modified as a function of slot width, according to some non-limiting embodiments. [Figure 6-6] 1 is a flow chart illustrating an example method for fabricating a NOEMS phase modulator, according to some non-limiting embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] I. Photocore A. Overview of Photonics-Based Processing The inventors have recognized and understood that there are limitations to the speed and efficiency of conventional processors based on electrical circuits. Every wire and transistor in the circuit of an electrical processor has resistance, inductance and capacitance that cause propagation delays and power dissipation in any electrical signal. For example, the connection of multiple processor cores and / or the connection of processor cores to memory uses conductive traces with non-zero impedance. A large value of impedance limits the maximum rate at which data can be transferred through the traces with negligible bit error rates. In applications where time delays are crucial, such as high frequency stock trading, delays of even a few hundredths of a second can render an algorithm unusable. In processes that require billions of operations by billions of transistors, these delays ultimately result in a large loss of time. In addition to the inefficiency of electrical circuits in speed, the heat caused by the energy dissipation caused by the impedance of the circuit is also a barrier in the development of electrical processors. The inventors have further recognized and understood that the use of optical signals instead of electrical signals eliminates many of the above problems with electrical computing. Optical signals travel at the speed of light in the medium in which light travels, and therefore the latency of photonic signals is a much smaller limitation than the electrical propagation delay. Furthermore, by increasing the distance that optical signals travel, power is not dissipated, opening up new topologies and processor layouts that would not be feasible using electrical signals. Thus, optical-based processors, such as photonics-based processors, may have better speed and efficiency performance than traditional electrical processors.

[0010] Furthermore, the present inventors have recognized and appreciated that light-based processors, such as photonics-based processors, may be suitable for certain types of algorithms. For example, many machine learning algorithms, such as support vector machines, artificial neural networks, and probabilistic graphical model learning, rely heavily on linear transformations on multidimensional arrays / tensors. The simplest example is multiplication of a vector by a matrix, which takes approximately O(n 2), where n is the dimension of the square matrix being multiplied. The inventors have recognized and appreciated that the photonic-based processor may be a highly parallel linear processor in some embodiments, and may perform linear transformations such as matrix multiplication in a highly parallelized manner by propagating a particular set of input optical signals through a configurable array of beam splitters. Using such an implementation, a matrix multiplication of a matrix having dimension n=512 may be completed in hundreds of picoseconds, as opposed to tens to hundreds of nanoseconds using conventional processing. Using some embodiments, it is estimated that matrix multiplication is accelerated by two orders of magnitude compared to conventional techniques. For example, a multiplication that may be performed by a state-of-the-art graphics processing unit (GPU) and may be performed in about 10 ns may be performed in about 200 ps by a photonic processing system according to some embodiments.

[0011] To implement a photonics-based processor, the inventors have recognized and understood that multiplication of an input vector by a matrix may be accomplished by propagating a coherent optical signal, e.g., a laser pulse, through a first array of interconnected variable beam splitters (VBS), a second array of interconnected variable beam splitters, and a number of controllable optical elements (e.g., electro-optical or opto-mechanical elements) between the two arrays, connecting one output of the first array to one input of the second array. Details of a photonic processing system including a photonic processor are described below.

[0012] B. Photonic Processing Systems Overview Referring to FIG. 1-1, a photonic processing system 1-100 according to some embodiments includes an optical encoder 1-101, a photonic processor 1-103, an optical receiver 1-105, and a controller 1-107. The photonic processing system 1-100 receives as input an input vector represented by a set of input bit strings from an external processor (e.g., a CPU) and generates an output vector represented by a set of output bit strings. For example, if the input vector is an n-dimensional vector, the input vector may be represented by n separate bit strings, each bit string representing each component of the vector. The input bit string may be received from the external processor as an electrical or optical signal, and the output bit string may be transmitted to the external processor as an electrical or optical signal. In some embodiments, the controller 1-107 does not necessarily output output bits after every process iteration. Instead, the controller 1-107 may use one or more output bit strings to determine a new input bit stream to feed through the components of the photonic processing system 1-100. In some embodiments, the output bit string itself may be used as an input bit string in a subsequent iteration of a process performed by photonic processing system 1-100. In other embodiments, multiple output bit streams are combined in various ways to determine a subsequent input bit string. For example, one or more output bit strings may be added together as part of determining a subsequent input bit string.

[0013] The optical encoder 1-101 is configured to convert the input bit string into optically encoded information that is processed by the photonic processor 1-103. In some embodiments, each input bit string is sent to the optical encoder 1-101 by the controller 1-107 in the form of an electrical signal. The optical encoder 1-101 converts each component of the input vector from a digital bit string into an optical signal. In some embodiments, the optical signal represents the value and sign of the associated bit string as the amplitude and phase of an optical pulse. In some embodiments, the phase may be limited to a binary selection of either a zero phase shift or a π phase shift, representing positive and negative values, respectively. The embodiments are not limited to real input vector values. Complex vector components may be represented, for example, by using three or more phase values ​​when encoding optical signals. In some embodiments, the bit string is received by the optical encoder 1-101 from the controller 1-107 as an optical signal (e.g., a digital optical signal). In these embodiments, the optical encoder 1-101 converts the digital optical signal into an analog optical signal of the type described above.

[0014] The optical encoder 1-101 outputs n separate optical pulses that are sent to the photonic processor 1-103. Each output of the optical encoder 1-101 is coupled one-to-one to one input of the photonic processor 1-103. In some embodiments, the optical encoder 1-101 may be located on the same substrate as the photonic processor 1-103 (e.g., the optical encoder 1-101 and the photonic processor 1-103 are on the same chip). In such embodiments, the optical signal may be transmitted from the optical encoder 1-101 to the photonic processor 1-103 in a waveguide, such as a silicon photonic waveguide. In other embodiments, the optical encoder 1-101 may be located on a separate substrate from the photonic processor 1-103. In such embodiments, the optical signal may be transmitted from the optical encoder 1-101 to the photonic processor 1-103 in an optical fiber.

[0015] The photonic processor 1-103 performs multiplication of the input vector by the matrix M. As will be described in more detail below, the matrix M is decomposed into three matrices using a combination of singular value decomposition (SVD) and unitary matrix decomposition. In some embodiments, the unitary matrix decomposition is performed using an operation similar to the Givens rotation in QR decomposition. For example, SVD in combination with Householder decomposition may be used. The decomposition of the matrix M into three components may be performed by the controller 1-107, and each component may be implemented by a portion of the photonic processor 1-103. In some embodiments, the photonic processor 1-103 includes three parts: a first array of variable beam splitters (VBS) configured to perform a transformation equivalent to a first matrix multiplication on an array of input optical pulses (see, e.g., the first matrix implementation 1-301 in Figures 1-3), a group of controllable optical elements configured to adjust the intensity and / or phase of each optical pulse received from the first array, where the adjustment is equivalent to a second matrix multiplication by a diagonal matrix (see, e.g., the second matrix implementation 1-303 in Figures 1-3), and a second array of VBS configured to perform a transformation equivalent to a third matrix multiplication on optical pulses received from the group of controllable electro-optical elements (see, e.g., the third matrix implementation 1-305 in Figure 3).

[0016] The photonic processor 1-103 outputs n separate optical pulses that are transmitted to the optical receiver 1-105. Each output of the photonic processor 1-103 is coupled one-to-one to one input of the optical receiver 1-105. In some embodiments, the photonic processor 1-103 may be located on the same substrate as the optical receiver 1-105 (e.g., the photonic processor 1-103 and the optical receiver 1-105 are on the same chip). In such embodiments, the optical signal may be transmitted from the photonic processor 1-103 to the optical receiver 1-105 in a silicon photonic waveguide. In other embodiments, the photonic processor 1-103 may be located on a separate substrate from the optical receiver 1-105. In such embodiments, the optical signal may be transmitted from the photonic processor 103 to the optical receiver 1-105 in an optical fiber.

[0017] The optical receiver 1-105 receives the n optical pulses from the photonic processor 1-103. Each optical pulse is then converted to an electrical signal. In some embodiments, the intensity and phase of each optical pulse is measured by an optical detector within the optical receiver. Electrical signals representing the measured values ​​are then output to the controller 1-107.

[0018] The controller 1-107 includes a memory 1-109 and a processor 1-111 that controls the optical encoder 1-101, the photonic processor 1-103, and the optical receiver 1-105. The memory 1-109 can be used to store input and output bit streams and measurements from the optical receiver 1-105. The memory 1-109 also stores executable instructions that, when executed by the processor 1-111, control the optical encoder 1-101, perform a matrix decomposition algorithm, control the VBS of the photonic processor 1-103, and control the optical receiver 1-105. The memory 1-109 can also include executable instructions that cause the processor 1-111 to determine a new input vector to send to the optical encoder based on a set of one or more output vectors determined by measurements performed by the optical receiver 1-105. In this manner, the controller 1-107 can control the iterative process in which an input vector is multiplied by multiple matrices by adjusting the settings of the photonic processor 1-103 and feeding the detected information from the optical receiver 1-105 back to the optical encoder 1-101. Thus, the output vector transmitted by the photonic processing system 1-100 to an external processor may be the result of multiple matrix multiplications rather than just one.

[0019] In some embodiments, a matrix may be too large to be encoded in the photonic processor using a single pass. In such a situation, a portion of the large matrix may be encoded in the photonic processor and a multiplication process may be performed on that one portion of the large matrix. The result of that first operation may be stored in memory 1-109. Subsequently, a second portion of the large matrix may be encoded in the photonic processor and a second multiplication process may be performed. This "chunking" of the large matrix may continue until a multiplication process has been performed on all portions of the large matrix. The results of the multiple multiplication processes may be stored in memory 1-109 and then combined to form the final result of the multiplication of the input vector by the large matrix.

[0020] In other embodiments, only the collective behavior of the output vectors is used by the external processor. In such embodiments, only the collective result, such as the average or max / min of multiple output vectors, is sent to the external processor.

[0021] C. Optical Encoder Referring to FIG. 1-2, according to some embodiments, an optical encoder includes at least one light source 1-201, a power tree 1-203, an amplitude modulator 1-205, a phase modulator 1-207, a digital to analog converter (DAC) 1-209 associated with the amplitude modulator 1-205, and a 1-DAC 211 associated with the phase modulator 1-207. Although the amplitude modulator 1-205 and the phase modulator 1-207 are shown in FIG. 1-2 as one block with n inputs and n outputs (each of the inputs and outputs is, for example, a waveguide), in some embodiments, each waveguide may include a respective amplitude modulator and a respective phase modulator such that the optical encoder includes n amplitude modulators and n phase modulators. Furthermore, there may be individual DACs for each amplitude and phase modulator. In some embodiments, rather than having an amplitude modulator and a separate phase modulator associated with each waveguide, one modulator may be used to encode both amplitude and phase information. Using a single modulator to perform such encoding limits the ability to precisely adjust both the amplitude and phase of each optical pulse, but there are some encoding schemes, described later in this specification, that do not require precise adjustment of both the amplitude and phase of the optical pulses.

[0022] The light source 1-201 may be any suitable source of coherent light. In some embodiments, the light source 1-201 may be a diode laser or a vertical-cavity surface emitting laser (VCSEL). In some embodiments, the light source 1-201 is configured to have an output power of more than 10 mW, more than 25 mW, more than 50 mW, or more than 75 mW. In some embodiments, the light source 1-201 is configured to have an output power of less than 100 mW. The light source 1-201 may be configured to emit a continuous wave of light or a pulse of light ("optical pulse") at one or more wavelengths (e.g., C-band or O-band). The duration of the optical pulse may be, for example, about 100 ps.

[0023] Although the light source 1-201 is shown in Figure 1-2 as being on the same semiconductor substrate as other components of the optical encoder, the embodiment is not so limited. For example, the light source 1-201 could be a separate laser package that is edge-bonded or surface-bonded to the optical encoder chip. Alternatively, the light source 1-201 could be completely off-chip and optical pulses could be coupled to the waveguide 1-202 of the optical encoder 1-101 via optical fiber and / or grating couplers.

[0024] Although the light source 1-201 is shown as two light sources 1-201a and 1-201b, the embodiments are not so limited. Some embodiments may include one light source. Including multiple light sources 201a-b, which may include three or more light sources, can provide redundancy if one of the light sources fails. Including multiple light sources can extend the useful life of the photonic processing system 1-100. The multiple light sources 1-201a-b can each be coupled to a waveguide of the optical encoder 1-101 and then combined at a waveguide combiner configured to direct optical pulses from each light source to the power tree 1-203. In such an embodiment, only one light source is used at any one time.

[0025] Some embodiments can use two or more phase-locked light sources of the same wavelength simultaneously to increase the optical power going into the optical encoder system. A small portion of the light from each of the two or more light sources (e.g., obtained via a waveguide tap) can be directed to a homodyne detector where a beat error signal can be measured. The beat error signal can be used to identify possible phase drift between the two light sources. The beat error signal can be fed, for example, to a feedback circuit that controls a phase modulator that phase-locks the output of one light source to the phase of the other light source. Phase locking can be generalized in a master-slave manner, where N≧1 slave light sources are phase-locked to one master light source. As a result, the optical encoder system can utilize a total of N+1 phase-locked light sources.

[0026] In other embodiments, each separate light source may be associated with a different wavelength of light. The use of multiple wavelengths of light allows some embodiments to be multiplexed such that multiple calculations can be performed simultaneously using the same optical hardware.

[0027] The power tree 1-203 is configured to split one optical pulse from the light source 1-201 into an array of spatially separated optical pulses. Thus, the power tree 1-203 has one optical input and n optical outputs. In some embodiments, the optical power from the light source 1-201 is split evenly across n optical modes associated with n waveguides. In some embodiments, the power tree 1-203 is an array 1-801 of 50:50 beam splitters as shown in FIG. 1-8. The numerical "depth" of the power tree 1-203 depends on the number of waveguides at the output. For a power tree with n output modes, the depth of the power tree 1-203 is ceil(log2(n)). The power tree 1-203 in FIG. 1-8 shows only a tree depth of 3 (each layer of the tree is marked across the bottom of the power tree 1-203). Each layer has 2 m-1beam splitters, where m is the number of layers, such that the first layer has one beam splitter 1-801a, the second layer has two beam splitters 1-801b through 1-801c, and the third layer has four beam splitters 1-801d through 1-801g.

[0028] Although power tree 1-203 is shown as an array of cascaded beam splitters, which may be implemented as an evanescent waveguide coupler, the embodiment is not so limited and any optical device that converts one optical pulse into multiple spatially separated optical pulses may be used. For example, power tree 1-203 may be implemented using one or more multimode interferometers (MMIs), in which case the equations governing the layer widths and depths are modified accordingly.

[0029] Regardless of the type of power tree 1-203 used, it is likely difficult, if not impossible, to fabricate a power tree 1-203 such that the split ratio is exactly equal among the n output modes. Thus, adjustments can be made to the amplitude modulation settings to compensate for the unequal intensity of the n optical pulses output by the power tree. For example, the waveguide with the lowest optical power can be set as the maximum power for any given pulse sent to the photonic processor 1-103. Thus, any optical pulse with a power higher than the maximum power can be modulated by the amplitude modulator 1-205 to have a lower power, in addition to the modulation on the amplitude that is performed to encode information into the optical pulse. A phase modulator can also be placed in each of the n output modes and used to adjust the phase of each output mode of the power tree 1-203 so that all output signals have the same phase.

[0030] Alternatively or additionally, power tree 1-203 may be implemented using one or more Mach-Zehnder Interferometers (MZIs) in which the splitting ratio of each beam splitter in the power tree may be adjusted to produce pulses of approximately equal intensity at the output of power tree 1-203.

[0031] The amplitude modulator 1-205 is configured to modify the amplitude of each optical pulse received from the power tree 1-203 based on each input bit sequence. The amplitude modulator 1-205 may be a variable attenuator or any other suitable amplitude modulator controlled by a DAC 1-209, which may be further controlled by the controller 1-107. Several amplitude modulators for telecommunications applications are known and may be used in some embodiments. In some embodiments, a variable beam splitter may be used as the amplitude modulator 1-205, where only one output of the variable beam splitter is kept and the other output is discarded or ignored. Other examples of amplitude modulators that may be used in some embodiments include traveling wave modulators, cavity-based modulators, Franz-Keldysh modulators, plasmon-based modulators, 2D material-based modulators, and nano-optoelectromechanical switches (NOEMS).

[0032] The phase modulator 1-207 is configured to change the phase of each optical pulse received from the power tree 1-203 based on each input bit sequence. The phase modulator may be a thermo-optical phase shifter or any other suitable phase shifter that may be electrically controlled by 1-211, which may be further controlled by the controller 1-107.

[0033] Although FIG. 1-2 shows the amplitude modulator 1-205 and the phase modulator 1-207 as two separate components, these may be combined into one element that controls both the amplitude and phase of the optical pulse. However, there are advantages to separately controlling the amplitude and phase of the optical pulse; that is, there is a phase shift associated with any amplitude shift due to the coupling between the amplitude shift and the phase shift via the Kramers-Kronig relation. To precisely control the phase of the optical pulse, the phase shift produced by the amplitude modulator 1-205 should be compensated for due to the use of the phase modulator 1-207. As an example, the total amplitude of the optical pulses leaving the optical encoder 1-101 is A=α0α1α2, and the total phase of the optical pulses leaving the optical encoder is θ=Δθ+Δψ+ψ, where α0 is the input intensity of the input optical pulse (assuming zero phase at the input of the modulator), α1 is the amplitude attenuation of the amplitude modulator 1-205, Δθ is the phase shift imparted by the amplitude modulator 1-205 while modulating the amplitude, Δψ is the phase shift imparted by the phase modulator 1-207, α2 is the attenuation associated with the optical pulse passing through the phase modulator 1-209, and ψ is the phase imparted to the optical signal due to the propagation of the optical signal. Thus, the settings of the amplitude and phase of the optical pulses are not two independent specifications. Rather, to accurately encode a particular amplitude and phase into the optical pulses output from the optical encoder 1-101, the settings of both the amplitude modulator 1-205 and the phase modulator 1-207 should be taken into account during both settings.

[0034] In some embodiments, the amplitude of the optical pulse is directly related to the bit sequence value. For example, a high amplitude pulse corresponds to a high bit sequence value and a low amplitude pulse corresponds to a low bit sequence value. The phase of the optical pulse encodes whether the bit sequence value is positive or negative. In some embodiments, the phase of the optical pulse output by the optical encoder 1-101 may be selected from two phases 180 degrees (π radians) apart. For example, a positive bit sequence value may be encoded with a zero degree phase shift and a negative bit sequence value may be encoded with a 180 degree (π radians) phase shift. In some embodiments, the vector is intended to be complex-valued, and thus the phase of the optical pulse is not simply selected from only two values, 0π and 2π.

[0035] In some embodiments, the controller 1-107 determines the amplitude and phase to be applied by both the amplitude modulator 1-205 and the phase modulator 1-207 based on the input bit sequence and the above equations linking the output amplitude and output phase to the amplitude and phase applied by the amplitude modulator 1-204 and the phase modulator 1-207. In some embodiments, the controller 1-107 may store in memory 1-109 a table of digital values ​​that drive the amplitude modulator 1-205 and the phase modulator 1-207. In some embodiments, the memory may be located near the modulators to reduce communication latency and power consumption.

[0036] A digital-to-analog converter (DAC) 1-209 associated with and communicatively coupled to the amplitude modulator 1-205 receives digital drive values ​​from the controller 1-107 and converts the digital drive values ​​to analog voltages that drive the amplitude modulator 1-205. Similarly, a DAC 1-211 associated with and communicatively coupled to the phase modulator 1-207 also receives digital drive values ​​from the controller 1-107 and converts the digital drive values ​​to analog voltages that drive the phase modulator 1-207. In some embodiments, the DAC may include an amplifier that amplifies the analog voltage to a level high enough to achieve a desired extinction ratio in the amplitude modulator (e.g., the highest extinction ratio physically possible to implement using a particular phase modulator) and a desired phase shift range in the phase modulator (e.g., a phase shift range covering the full range of 0 to 2π). Although DAC1-209 and DAC1-211 are shown in FIGS. 1-2 as being located within and / or on the chip of the optical encoder 1-101, in some embodiments, DAC1-209 and 1-211 may be located off-chip while still being communicatively coupled to the amplitude modulator 1-205 and phase modulator 1-207, respectively, using conductive traces and / or wires.

[0037] After modulation by the amplitude modulator 1-205 and the phase modulator 1-207, the n optical pulses are transmitted from the optical encoder 1-101 to the photonic processor 1-103. D. Photonic Processor 1-3, the photonic processor 1-103 performs matrix multiplication on an input vector represented by n optical input pulses and includes three main components: a first matrix implementation 1-301, a second matrix implementation 1-303, and a third matrix implementation 1-305. In some embodiments, as described in more detail below, the first matrix implementation 1-301 and the third matrix implementation 1-305 include interconnected arrays of programmable, reconfigurable variable beam splitters (VBS) configured to convert the n input optical pulses from an input vector to an output vector, where the components of the vector are represented by the amplitude and phase of each optical pulse. In some embodiments, the second matrix implementation 1-303 includes a group of electro-optical elements.

[0038] The matrix by which the input vector is multiplied by passing the input optical pulses through the photonic processor 1-103 is called M. The matrix M is a general m×n matrix known to the controller 1-107 as the matrix to be implemented by the photonic processor 1-103. Thus, the controller 1-107 knows that the matrix M has three constituent matrices: M=V T Singular value decomposition (SVD) is used to decompose the matrix M, as represented by ΣU, where U and V are real diagonal n×n and m×m matrices, respectively (U T U=UU T = 1 and V T V=VV T =1), Σ is an m×n diagonal matrix with real entries. The superscript “T” in all equations denotes the transpose of the associated matrix. Determining the SVD of a matrix is ​​known, and the controller 1-107 may use any suitable technique to determine the SVD of matrix M. In some embodiments, matrix M is a complex matrix, in which case matrix M may be expressed as M=V † ΣU, where V and U are complex unitary n×n and m×m matrices, respectively. † U=UU † = 1 and V † V=VV †=1), Σ is an m×n diagonal matrix with real or complex entries. The values ​​of the diagonal singular values ​​may be further normalized such that the maximum absolute value of the singular values ​​is 1.

[0039] Once the controller 1-107 has identified matrices U, Σ, and V of the matrix M, if the matrices U and V are diagonal real matrices, the controller may further decompose the two diagonal matrices U and V into a series of real-valued Givens rotation matrices. The Givens rotation matrix G(i,j,θ) can be expressed as follows: If k ≠ i, j, g kk =1, For k=i,j, g kk = cos(θ), g ij =-g ji = -sin(θ), In other cases, g kl =0 Each component is defined by: ij represents the element in the i-th row and j-th column of matrix G, and θ is the angle of rotation associated with the matrix. In general, matrix G is any 2×2 unitary matrix with determinant 1 (SU(2) group) and is parameterized by two parameters. In some embodiments, those two parameters are the rotation angle θ and another phase value φ. Nevertheless, matrix G can also be parameterized by other values ​​besides angle or phase, for example reflectance / transmittance or separation distance (in the case of NOEMS).

[0040] An algorithm for expressing any real diagonal matrix in complex space as a product of a pair of Givens rotations is given in M. Reck et al., Experimental realization of any discrete unitary operator. No. 5,993,311 ("Clements"), and W. R. Clements et al., "Optimal design for universal multiport interferometers," Optica, Vol. 3, No. 12 (2016) ("Clements"), both of which are incorporated herein by reference for their discussion of techniques for decomposing real diagonal matrices globally and at least with respect to the Givens rotation. (In the event that any term used herein conflicts with the use of that term in Reck and / or Clements, the term should be given the meaning most consistent with how one of ordinary skill in the art would understand that use herein.) The decomposition that is generated is the following equation:

[0041]

number

[0042] where U is an n×n orthogonal matrix, and S k is the set of indices associated with the kth set of Givens rotations to be applied (as defined by the decomposition algorithm), and θ ij (k) represents the angle applied to the Givens rotation between components i and j in the kth set of Givens rotations, and D is a diagonal matrix with either +1 or -1 entries representing the global sign of each component. k depends on whether n is even or odd. For example, if n is even, and k is odd, then S k ={(1,2),(3,4),...,(n-1,n)}, for even k, S k = {(2,3),(4,5),...,(n-2,n-1)}. If n is odd, and k is odd, then S k={(1,2),(3,4),...,(n-2,n-1)}, for even k, S k ={(2,3),(4,5),...,(n-1,n)}.

[0043] By way of example and not limitation, the decomposition of a 4×4 orthogonal matrix may be:

[0044]

number

[0045] It can be expressed as: An outline of one embodiment of an algorithm for decomposing an n×n matrix U in terms of n sets of real-valued Givens rotations that can be implemented using the controller 1-107 follows.

[0046]

number

[0047] The matrix U′ produced as a result of the above algorithm is lower triangular and has the formula:

[0048]

number

[0049] where the label S L denotes the set of two modes connected by a VBS on the left side of U', and is labeled S R denotes the set of two modes connected by the VBS on the right side of U'. Since U is an orthogonal matrix, U' is a diagonal matrix with {-1,1} entries along the diagonal. This matrix U'=D U is called a "phase screen."

[0050] The next step of the algorithm may be implemented using the controller 1-107 as follows:

[0051]

number

[0052] Using any G T jk (θ1)D U =D U G jk The goal is to iteratively find whether (θ2) is achieved. The above algorithm is based on V and / or V T can also be decomposed and used to identify the m layers of VBS values ​​and associated phase screens.

[0053] The above concept of decomposing an orthogonal matrix into a real-valued Givens rotation matrix can be extended to complex matrices, e.g., unitary matrices, rather than orthogonal matrices. In some embodiments, this can be achieved by including an additional phase in the parameterization of the Givens rotation matrix. Thus, the general form of a Givens matrix with an additional phase term added is T(i,j,θ,φ), where If k ≠ i, j, then t kk =1, t ii =e iφ cos(θ), t jj = cos(θ), t ij = -sin(θ), t ji =e iφ sin(θ), Otherwise, t kl =0 where t ij represents the i-th row and j-th column of a matrix T, θ is the rotation angle associated with the matrix, and φ is an additional phase. Any unitary matrix can be decomposed into a matrix of type T(i,j,θ,φ). By choosing to set the phase φ=0, one obtains the conventional real-valued Givens rotation matrix mentioned above. Alternatively, for phase φ=π, one obtains a set of matrices known as Householder matrices. A Householder matrix H has the form

[0054]

number

[0055] where I is the n×n identity matrix and v is a unit vector;

[0056]

number

[0057] is the cross product. Householder matrices represent reflections about a hyperplane orthogonal to the unit vector ν. In this parameterization, the hyperplane is a two-dimensional subspace, rather than an n-1-dimensional subspace, as is common when defining Householder matrices for the QR decomposition. Thus, decomposing a matrix into Givens rotations is equivalent to decomposing the matrix into Householder matrices.

[0058] Based on the above decomposition of any unitary matrix into a restricted set of Givens rotations, any unitary matrix can be implemented with a specific sequence of rotations and phase shifts. Also, in photonics, rotations can be represented by variable beam splitters (VBS) and phase shifts are easily implemented using phase modulators. Thus, for n optical inputs of the photonic processor 1-103, the first matrix implementation 1-301 and the third matrix implementation 1-305 representing the unitary matrices of the SVD of the matrix M can be implemented by interconnected arrays of VBS and phase shifters. Due to the parallel nature of passing optical pulses through the VBS array, the matrix multiplication can be performed in O(1) time. The second matrix implementation 1-303 is the diagonal matrix of the SVD of the matrix M combined with the diagonal matrix D associated with each orthogonal matrix of the SVD. As mentioned above, each matrix D is called a "phase screen" and can be denoted with a subscript to indicate whether it is a phase screen associated with the matrix U or a phase screen associated with the matrix V. Therefore, the second matrix implementation 303 is a matrix Σ′=D V ΣD U It is.

[0059] In some embodiments, the VBS unit cells of the photonic processor 1-103 associated with the first matrix implementation 1-301 and the third matrix implementation 1-305 can be Mach-Zehnder interferometers (MZIs) with internal phase shifters. In other embodiments, the VBS unit cells can be microelectromechanical systems (MEMS). In some embodiments, an external phase shifter may be used to implement the additional phase required for the Givens rotation.

[0060] Diagonal matrix D V ΣD U The second matrix implementation 1-303 representing may be implemented using amplitude modulators and phase shifters. In some embodiments, the VBS may be used to split off a portion of the light that can be dumped to variably attenuate the optical pulse. Additionally or alternatively, a controllable gain medium may be used to amplify the optical signal. For example, GaAs, InGaAs, GaN or InP may be used as an active gain medium to amplify the optical signal. Other active gain processes such as second harmonic generation in materials with crystal inversion symmetry, e.g., KTP and lithium niobate, as well as four-wave mixing processes in materials without inversion symmetry, e.g., silicon, may also be used. The phase shifters in each optical mode may be used to apply either a 0 phase shift or a π phase shift depending on the phase screen being implemented. In some embodiments, only one phase shifter is used for each optical mode, rather than one phase shifter for each phase screen. This allows each matrix D V , Σ and D U This is possible because is diagonal and therefore commutes. Thus, the value of each phase shifter of the second matrix implementation 1-303 of the photonic processor 1-103 is the result of the product of two phase screens: V D U .

[0061] Referring to FIG. 1-4, the first matrix implementation 1-301 and the third matrix implementation 1-305 are implemented as an array of VBS1-401 according to some embodiments. For simplicity, only n=6 optical pulses (number of rows) are shown, resulting in a "circuit depth" (e.g., number of columns) equal to the number of input optical pulses (e.g., 6). For clarity, only one VBS1-401 is labeled with a reference number. However, the VBSs are labeled with a subscript that identifies which optical mode is being mixed by the particular VBS and a superscript that denotes the associated column. Each VBS1-401 implements a complex Givens rotation T(i,j,θ,φ) as described above, where i and j are equal to the subscript label of the VBS1-401, θ is the rotation angle of the Givens rotation, and φ is an additional phase associated with the generalized rotation.

[0062] Referring to FIG. 1-5, each VBS 1-401 may be implemented using an MZI 1-510 and at least one external phase shifter 1-507. In some embodiments, a second external phase shifter 1-509 may also be included. The MZI 1-510 includes a first evanescent coupler 1-501 and a second evanescent coupler 1-503 to mix two input modes of the MZI 1-510. An internal phase shifter 1-505 modulates the phase θ in one arm of the MZI 1-510 to create a phase difference between the two arms. By adjusting the phase θ, the intensity of the light output by the VBS 1-401 is changed from one output mode of the MZI 1-510 to the other output mode, thereby creating a controllable and variable beam splitter. In some embodiments, a second internal phase shifter may be applied in the second arm. In this case, it is the difference between the two internal phase shifters that causes the output light intensity to change. The average between the two internal phases imparts a global phase to the light entering modes i and j. Thus, the two parameters θ and φ can be controlled by the phase shifters, respectively. In some embodiments, a second external phase shifter 1-509 can be used to correct unwanted differential phase across the output modes of the VBS due to static phase perturbations.

[0063] In some embodiments, the phase shifters 1-505, 1-507, and 1-509 may include thermo-optic, electro-optic, or opto-mechanical phase modulators. In other embodiments, rather than including an internal phase modulator 505 within the MZI 510, a NOEMS modulator may be used.

[0064] In some embodiments, the number of VBSs increases with the size of the matrix. The inventors have recognized and appreciated that controlling a large number of VBSs can be difficult and that there are advantages to sharing one control circuit among multiple VBSs. One example of a parallel control circuit that may be used to control multiple VBSs is a digital-to-analog converter, which receives as an input a digital string that encodes an analog signal to be applied to a particular VBS. In some embodiments, the circuit also receives as a second input the address of the VBS to be controlled. The circuit may then apply the analog signal to the addressed VBS. In other embodiments, the control circuit may automatically scan through several VBSs and apply analog signals to multiple VBSs without being actively addressed. In this case, the addressing sequence is predefined such that the VBS array is traversed in a known order.

[0065] Referring to FIG. 1-6, the second matrix implementation 1-303 implements a diagonal matrix Σ′=D V ΣD U This may be achieved using two phase shifters 1-601 and 1-605 to implement two phase screens, and an amplitude modulator 1-603 to adjust the intensity of the associated optical pulse by an amount η. As noted above, in some embodiments, only one phase modulator 1-601 may be used, since the two phase screens can be combined together because the three component matrices that form Σ′ are diagonal and therefore commute.

[0066] In some embodiments, the amplitude modulator 1-603 may be implemented using an attenuator and / or an amplifier. If the value of the amplitude modulation η is greater than 1, the optical pulse is amplified. If the value of the amplitude modulation η is less than 1, the optical pulse is attenuated. In some embodiments, only attenuation is used. In some embodiments, the attenuation may be implemented by a bank of integrated attenuators. In other embodiments, as shown in FIG. 1-7, the attenuation 1-603 may be implemented using an MZI that includes two evanescent couplers 1-701 and 1-703 and a controllable internal phase shifter 1-705 to adjust the amount of input light transmitted from the input of the MZI to a first output port 1-709 of the MZI. The second output port 1-707 of the MZI may be ignored, blocked, or dumped.

[0067] In some embodiments, the controller 1-107 controls the value of each phase shifter in the photonic processor 1-103. Each phase shifter described above may include a DAC similar to the DAC discussed in connection with the phase modulator 1-207 of the optical encoder 1-101.

[0068] The photonic processor 1-103 can include any number of input nodes, but the size and complexity of the interconnected VBS arrays 1-301 and 1-305 increases as the number of input modes increases. For example, if there are n input optical modes, the photonic processor 1-103 has a circuit depth of 2n+1, where the first matrix implementation 1-301 and the second matrix implementation 1-305 each have a circuit depth of n, and the second matrix implementation 1-303 has a circuit depth of 1. Importantly, the complexity in time to perform one matrix multiplication is not even linear with the number of input optical pulses (it is always O(1)). In some embodiments, this low order complexity provided by parallelization results in energy and time efficiencies that cannot be obtained using conventional electrical processors.

[0069] It should be noted that although the embodiments described herein show the photonic processor 1-103 as having n inputs and n outputs, in some embodiments the matrix M implemented by the photonic processor 1-103 may not be a square matrix. In such embodiments, the photonic processor 1-103 may have a different number of outputs and inputs.

[0070] Furthermore, due to the topology of the interconnections of the VBSs in the first and second matrix implementations 1-301 and 1-305, it is possible to divide the photonic processors 1-103 into non-interacting subsets of rows so that more than one matrix multiplication can be performed simultaneously. For example, in the VBS array shown in Figures 1-4, if each VBS 1-401 that couples optical modes 3 and 4 is set such that optical modes 3 and 4 are not coupled at all (e.g., as if the VBS 1-401 with subscript "34" did not exist in Figures 1-4), the upper three optical modes will operate completely independently from the lower three optical modes. Such division can be done on a much larger scale in photonic processors with a large number of input optical modes. For example, a photonic processor with n=64 can simultaneously multiply eight 8-component input vectors by each 8x8 matrix (each of the 8x8 matrices is separately programmable and controllable). Furthermore, the photonic processors 1-103 do not need to be divided evenly. For example, an n=64 photonic processor may be split into 7 different input vectors with 20, 13, 11, 8, 6, 4 and 2 components, respectively, each multiplied by each matrix simultaneously. It should be understood that the above numerical example is for illustrative purposes only and any number of splits are possible.

[0071] Additionally, although the photonic processor 1-103 performs vector-matrix multiplication, where a vector is multiplied by a matrix by passing an optical signal through the VBS array, the photonic processor 1-103 may also be used to perform matrix-matrix multiplication. For example, multiple input vectors may be passed through the photonic processor 1-103 sequentially, one input vector at a time, where each input vector represents a column of the input matrix. After optically computing each of the individual vector-matrix multiplications (each multiplication producing an output vector corresponding to a column of the resulting output column of the matrix), the results may be digitally combined to form an output matrix resulting from the matrix-matrix multiplication.

[0072] E. Optical receiver The photonic processor 1-103 outputs n optical pulses that are transmitted to the optical receiver 1-105. The optical receiver 1-105 receives the optical pulses and generates an electrical signal based on the received optical signal. In some embodiments, the amplitude and phase of each optical pulse is determined. In some embodiments, this is accomplished using a received homodyne or heterodyne detection scheme. In other embodiments, simple phase-insensitive optical detection can be performed using a conventional photodiode.

[0073] Referring to FIG. 1-9, the optical receiver 1-105 includes, according to some embodiments, a homodyne detector 1-901, a transimpedance amplifier 1-903, and an analog-to-digital converter (ADC) 1-905. Although the components are shown as one element for all optical modes in FIG. 1-9, this is for the sake of simplicity. Each optical mode may have a dedicated homodyne detector 1-901, a dedicated transimpedance amplifier 1-903, and a dedicated ADC 1-905. In some embodiments, the transimpedance amplifier 1-903 may not be used. Instead, any other suitable electronic circuit that converts a current to a voltage may be used.

[0074] 1-10, the homodyne detector 1-903, according to some embodiments, includes a local oscillator (LO) 1-1001, a quadrature controller 1-1003, a beam splitter 1-1005, and two detectors 1-1007 and 1-1009. The homodyne detector 1-903 outputs a current based on the difference between a current output by a first detector 1-1007 and a current output by a second detector 1-1009.

[0075] The local oscillator 1-1001 is coupled with the input optical pulse at the beam splitter 1-1005. In some embodiments, a portion of the light source 1-201 is transmitted to the homodyne detector 1-901 via an optical waveguide and / or optical fiber. The light from the light source 1-201 may itself be used as the local oscillator 1-1001, or in other embodiments, the local oscillator 1-1001 may be a separate light source that uses light from the light source 1-201 to generate a phase-matched optical pulse. In some embodiments, an MZI may replace the beam splitter 1-1005 to allow for adjustment between the signal and the local oscillator.

[0076] The quadrature controller 1-1003 controls the cross-section angle in phase space at which measurements are made. In some embodiments, the quadrature controller 1-1003 may be a phase shifter that controls the relative phase between the input optical pulse and a local oscillator. The quadrature controller 1-1003 is shown as a phase shifter in input optical mode. However, in some embodiments, the quadrature controller 1-1003 may be in local oscillator mode.

[0077] A first detector 1-1007 detects the light output by a first output of the beam splitter 1-1005, and a second detector 1-1009 detects the light output by a second output of the beam splitter 1-1005. Detectors 1-1007 and 1-1009 can be photodiodes operated with zero bias.

[0078] The subtraction circuit 1-1011 subtracts the current from the first detector 1-1007 from the current from the second detector 1-1009. The resulting current thus has an amplitude and a sign (positive or negative). The transimpedance amplifier 1-903 converts this difference in currents into a voltage, which can be positive or negative. Finally, the ADC 1-905 converts the analog signal into a digital bit string. This output bit string represents the output vector result of the matrix multiplication and is an electrical digital version of the optical output representation of the output vector output by the photonic processor 1-103. In some embodiments, the output bit string may be sent to the controller 1-107 for further processing, which may include determining a next input bit string based on one or more output bit strings and / or sending the output bit string to an external processor as described above.

[0079] The inventors have further recognized and understood that the components of the photonic processing system 1-100 described above need not be serially coupled together such that the first matrix implementation 1-301 is connected to the second matrix implementation 1-303, which is connected to the third matrix implementation 1-305. In some embodiments, the photonic processing system 1-103 may include only one unitary circuit that performs one or more multiplications. The output of the one unitary circuit may be directly connected to the optical receiver 1-105, where the result of the multiplication is determined by detecting the output optical signal. In such an embodiment, the one unitary circuit may implement, for example, the first matrix implementation 1-301. The result detected by the optical receiver 1-105 may then be digitally transmitted to a conventional processor (e.g., processor 1-111), and the second diagonal matrix implementation 1-303 may be performed in the digital domain using the conventional processor (e.g., processor 1-111). The controller 1-107 may then execute a third matrix implementation 1-305, determine an input bit sequence based on the result of the digital implementation of the second matrix implementation, and reprogram the one unitary circuit to control the optical encoder to transmit an optical signal encoded based on the new input bit sequence through the one unitary circuit with the reprogrammed settings. The resulting output optical signal is detected by the optical receiver 105 and then used to determine the result of the matrix multiplication.

[0080] The inventors have also recognized and understood that it may be advantageous to concatenate multiple photonic processors 1-103 in series. For example, to perform a matrix multiplication M=M1M2, where M1 and M2 are arbitrary matrices, but M2 changes more frequently than M1 based on a changing input workload, a first photonic processor may be controlled to perform M2, and a second photonic processor optically coupled to the first photonic processor may perform M1, which is kept constant. In this way, only the first photonic processing system needs to be updated frequently based on a changing input workload. Such a configuration not only accelerates the calculations, but also reduces the number of data bits moving between the controller 1-107 and the photonic processors.

[0081] F. Folded Photonic Processing Systems In FIG. 1-1, in such a configuration, the optical encoder 1-101 and the optical receiver 1-105 are located on opposite sides of the photonic processing system 1-100. In applications where feedback from the optical receiver 1-105 is used to determine the input to the optical encoder 1-101 for future iterations of the process, the data is transferred electronically from the optical receiver 1-105 to the controller 1-107 and the optical encoder 1-101. The inventors have recognized and appreciated that reducing the distance these electrical signals must travel (e.g., by reducing the length of the electrical traces and / or wires) results in power savings and lower latency. Furthermore, it is not necessary for the optical encoder 1-101 and the optical receiver 1-105 to be located on opposite ends of the photonic processing system.

[0082] Thus, in some embodiments, the optical encoder 1-101 and the optical receiver 1-105 are located near each other (e.g., on the same side of the photonics processor 1-103) such that the distance that the electrical signal needs to travel between the optical encoder 1-101 and the optical receiver 1-105 is less than the width of the photonics processor 1-103. This can be achieved by physically interleaving the components of the first matrix implementation 1-301 and the third matrix implementation 1-305 to be physically in the same part of the chip. This configuration is called a "folded" photonic processing system because the light first propagates in a first direction through the first matrix implementation 1-301 until it reaches a physical part of the chip far from the optical encoder 1-101 and the optical receiver 1-105, and then when implementing the third matrix implementation 1-305, the waveguides fold back to turn the light to propagate in the opposite direction to the first direction. In some embodiments, the second matrix implementation 1-303 is physically located adjacent to the fold in the waveguide. Such a configuration reduces the complexity of the electrical traces connecting the optical encoder 1-101, the optical receiver 1-105, and the controller 1-107, and reduces the total chip area used to implement the photonic processing system 1-100. For example, some embodiments using the fold configuration use only 65% ​​of the total chip area required if the continuous photonic configuration of FIG. 1-1 is used. This may reduce the cost and complexity of the photonic processing system.

[0083] The inventors have recognized and understood that the folded configuration not only has electrical advantages, but also optical advantages. For example, by reducing the distance that an optical signal must travel from the light source to be used as a local oscillator in homodyne detection, the time-dependent phase variation of the optical signal can be reduced, resulting in a higher quality detection result. In particular, by locating the light source and the homodyne on the same side of the photonics processor, the distance traveled by the optical signal used for the chest oscillator is no longer dependent on the size of the matrix. For example, in the continuous configuration of FIG. 1-1, the distance traveled by the optical signal towards the local oscillator increases and decreases linearly with the size of the matrix, whereas the travel distance in the folded configuration is constant regardless of the matrix size.

[0084] FIG. 1-11 is a schematic diagram of a folded photonic processing system 1-1100 according to some embodiments. The folded photonic processing system 1-1100 includes a power tree 1-1101, a plurality of optical encoders 1-1103a to 1-1103d, a plurality of homodyne detectors 1-1105a to 1-1105d, a plurality of selector switches 1-1107a to 1-1107d, a plurality of U matrix elements 1-1109a to 1-1109j, a plurality of diagonal matrix elements 1-1111a to 1-1111d, and a plurality of V matrix elements 1-1113a to 1-1113j. For clarity, not all of the components of the folded photonic processing system are shown in the figure. It should be understood that the folded photonic processing system 1-1100 may include similar components to the continuous photonic processing system 1-100.

[0085] The power tree 1-1101 is similar to the power tree 1-203 of FIG. 2 and is configured to send light from a light source (not shown) to an optical encoder 1-1103. However, the difference between the power tree 1-1101 and the power tree 1-203 is that the power tree sends the optical signal directly to the homodyne detector 1-1105a. In FIG. 2, the light source 201 sends a local oscillator signal to the homodyne detector on the other side of the photonic processor by taking a portion of the optical signal from the light source and guiding the optical signal using a waveguide. In FIG. 1-11, the power tree 1-1101 includes a number of outputs equal to twice the number of spatial modes. For example, FIG. 1-11 shows only four spatial modes of the photonic processor, and eight output modes are generated from the power tree 1-1101. One output directs light to each optical encoder 1-1103 and one output directs light to each homodyne detector 1-1105. The power tree may be implemented, for example, using cascaded beam splitters or multimode interferometers (MMIs).

[0086] The optical encoder 1-1103 is similar to the power tree optical encoder 1-101 of Figure 1 and is configured to encode information in the amplitude and / or phase of the optical signal received from the power tree 1-1101. This may be accomplished, for example, as described in connection with the optical encoder 1-101 of Figure 2.

[0087] The homodyne detector 1-1105 is disposed between the power tree 1-1101 and the U matrix element 1-1109. In some embodiments, the homodyne detector 1-1105 is physically positioned in a row with the optical encoder 1-1103. In some embodiments, the optical encoder 1-1103 and the homodyne detector 1-1105 may be interleaved in a row. In this manner, the optical encoder 1-1103 and the homodyne detector 1-1105 are in close proximity to each other, reducing the distance of electrical traces (not shown) used to connect the optical encoder 1-1103 and the homodyne detector 1-1105 and a controller (not shown) that may be physically located adjacent to the row of the optical encoder 1-1103 and the homodyne detector 1-1105.

[0088] Each optical encoder 1-1103 is associated with a respective homodyne detector 1-1105. Both the optical encoder 1-1103 and the homodyne detector 1-1105 receive an optical signal from the power tree 1-1101. The optical encoder 1-1103 encodes an input vector using the optical signal as described above. The homodyne detector 1-1105 uses the received optical signal received from the power tree as a local oscillator as described above.

[0089] Each pair of optical encoder 1-1103 and homodyne detector 1-1105 is associated with a selector switch 1-1107 and connected to the selector switch 1-1107 by a waveguide. The selector switches 1-1107a to 1-1107d can be implemented using, for example, conventional 2×2 optical switches. In some embodiments, the 2×2 optical switch is an MZI with an internal phase shifter that controls the behavior of the MZI from cross to bar. The switch 1-1107 is connected to a controller (not shown) to control whether the optical signal received from the optical encoder 1-1103 is guided towards the U matrix element 1-1109 or towards the V matrix element 1-1113. The optical switch is also controlled to guide the light received from the U matrix element 1-1109 and / or the V matrix element 1-1113 towards the homodyne detector 1-1105 for detection.

[0090] The technique for implementing the matrix multiplication is similar to that in the photonically folded photonic processing system 1-1100 as described above in connection with the continuous system, as described in FIG. 1-3. The difference between the two systems is the physical arrangement of the matrix elements and the implementation of the fold 1-1120, where the optical signal is changed from generally left-to-right propagation in FIG. 1-11 to generally right-to-left propagation. In FIG. 1-11, the connections between the components may represent waveguides. The solid line connections represent, in some embodiments, portions of the waveguide where the optical signal is propagating from left to right, and the dashed line connections represent, in some embodiments, portions of the waveguide where the optical signal is propagating from right to left. Notably, given this notation, the embodiment shown in FIG. 1-11 is one in which the selector switch 1-1107 first guides the optical signal to the U matrix element 1-1109. In other embodiments, the selector switch 1-1107 may guide the optical signal first to V matrix element 1-1113, where the dashed lines represent the portions of the waveguide where the optical signal is propagating from left to right and the solid line connections represent the portions of the waveguide where the optical signal is propagating from right to left.

[0091] The U matrix of the SVD of matrix M is implemented in the photonic processing system 1-1100 using U matrix elements 1-1109 interleaved with V matrix elements 1-1113. Thus, unlike the serial configuration embodiment shown in FIG. 1-3, the U matrix elements 1-1109 and V matrix elements 1-1113 are not all physically located in each self-contained array within one physical area. Thus, in some embodiments, the photonic processing system 1-1100 includes multiple columns of matrix elements, with at least one of the columns including both U matrix elements 1-1109 and V matrix elements 1-1113. In some embodiments, as shown in FIG. 1-11, the first column may only have U matrix elements 1-1109. The U matrix elements 1-1109 are implemented similarly to the first matrix implementation 1-301 of FIG. 3.

[0092] Due to the interleaved structure of the U matrix elements 1-1109 and the V matrix elements 1-1113, the folded photonic processing system 1-1100 includes waveguide crossovers 1-1110 at various locations between columns of matrix elements. In some embodiments, the waveguide crossovers can be constructed using adiabatic evanescent elevators between two or more layers in an integrated photonics chip. In other embodiments, the U matrix and the V matrix can be located on different layers of the same chip, and waveguide crossovers are not used.

[0093] After the optical signal propagates through all of the U matrix elements 1-1109, it propagates to the diagonal matrix element 1-1111, which is implemented similarly to the second matrix implementation 1-303 of FIG. 1-3.

[0094] After the optical signal propagates through all of the diagonal matrix elements 1-1111, it propagates to the V matrix elements 1-1113, which are implemented similarly to the third matrix implementation 1-305 of FIG. 1-3. The V matrix of the SVD of matrix M is implemented in the photonic processing system 1-1100 using the V matrix elements 1-1113 interleaved with the U matrix elements 1-1109. Thus, all of the V matrix elements 1-1113 are not physically located in one self-contained array.

[0095] After the optical signal has propagated through all of the V matrix elements 1-1113, it returns to the selector switch 1-1107, which guides the optical signal to the homodyne detector 1-1105 for detection.

[0096] The inventors have further recognized and appreciated that by including a selector switch after the optical encoder and before the matrix elements, the folded photonic processing system 1-1100 allows for efficient bidirectionality of the circuit. Thus, in some embodiments, a controller, such as the controller 1-107 described in connection with FIG. 1-1, may select whether the optical signal is first multiplied by the U matrix, or first multiplied by the V matrix. TFor an array of VBSs configured to implement a unitary matrix U when an optical signal propagates from left to right, propagating an optical signal from right to left may control how the matrix U is multiplied. T Thus, the same configuration of an array of VBSs will produce U and U depending on which direction the optical signal propagates through the array. T Both can be implemented and which direction the optical signal propagates through the array can be controlled using the selector switch at 1-1107. In some applications, such as backpropagation used to train machine learning algorithms, it may be desirable to pass the optical signal in the reverse direction through one or more matrices. In other applications, the bidirectionality can be used to compute the inverse matrix operation on an input vector. For example, for an invertible n×n matrix M, SVD gives M=V T ΣU is generated. The inversion of this matrix is ​​M -1 =U T Σ -1 V, where Σ -1 is the inverse of a diagonal matrix, which can be efficiently computed by inverting each diagonal element. To multiply a vector by a matrix M, the switch multiplies the optical signal by the matrix U, then Σ, then V. T Inverted M -1 To multiply a vector by, the singular values ​​are first multiplied by Σ -1 The VBS is then configured to program the implementation of the matrix V. This constitutes a change in the settings of only one column of the VBS, instead of all 2n+1 columns of the photonic processor, which would be the case in a unidirectional photonic processing system such as that shown in Figures 1-3. An optical signal representing an input vector is then input to the matrix V T , then Σ -1 , then propagates to U in a second direction opposite to the first direction. Using the selector switch 1-1107, the folded photonic processing system 1-1100 first performs the U matrix (or its transpose), so that V T This can easily be modified to perform the matrix (or its transpose) first.

[0097] G. Wavelength division multiplexing The inventors have further recognized and understood that there are applications in which different vectors may be multiplexed with the same matrix. For example, when training or using machine learning algorithms, data sets may be processed using the same matrix multiplication. The inventors have recognized and understood that this may be accomplished using one photonic processor if the components before and after the photonic processor are wavelength division multiplexed (WDM). Thus, some embodiments include multiple front-ends and back-ends, each associated with a different wavelength, while performing the matrix multiplication using only one photonic processor.

[0098] 1-12A illustrates a WDM photonic processing system 1-1200 according to some embodiments. The WDM photonic processing system 1-1200 includes N front ends 1-1203, one photonic processor 1-1201 with N spatial modes, and N back ends 1-1205.

[0099] The photonic processor 1-1201 may be similar to the photonic processor 1-103 and has N input modes and N output modes. Each of the N front ends 1-1203 is connected to a respective input mode of the photonic processor 1-1201. Similarly, each of the N back ends 1-1205 is connected to a respective output mode of the photonic processor 1-1201.

[0100] FIG. 1-12B shows at least one detail of the front end 1-1203. As with the photonic processing systems of the other embodiments, the photonic processing system 1-1200 includes an optical encoder 1-1211. However, in this embodiment, there are M different optical encoders, where M is the number of wavelengths multiplexed by the WDM photonic processing system 1-1200. Each of the M optical encoders 1-1211 receives light from a light source (not shown) that generates M optical signals, each at a different wavelength. The light source can be, for example, an array of lasers, a frequency comb generator, or any other light source that generates coherent light at different wavelengths. Each of the M optical encoders 1-1211 is controlled by a controller (not shown) to perform the appropriate amplitude and phase modulation to encode the data into the optical signal. The M encoded optical signals are then combined into one waveguide using an M:1 WDM 1-1213. One waveguide then connects to one of N waveguides of photonic processor 1-1201.

[0101] FIG. 1-12C shows at least one detail of the backend 1-1205. As with the photonic processing systems of the other embodiments, the photonic processing system 1-1200 includes a detector 1-1223, which may be a phase-sensitive or phase-insensitive detector. However, in this embodiment, there are M different detectors 1-1223, where M is the number of wavelengths multiplexed by the WDM photonic processing system 1-1200. Each of the M detectors 1-1223 receives light from a 1:M WDM 1-1221, which splits one output waveguide from the photonic processor 1-1201 into M different wavelengths, each carrying an optical signal of each wavelength. Each of the M detectors 1-1223 may be controlled by a controller (not shown) to record the measurement results. For example, each of the M detectors 1223 may be a homodyne detector or a phase-insensitive photodetector.

[0102] In some embodiments, the VBS in the photonic processor 1-1201 may be selected to be non-dispersive within the M wavelengths of interest. Thus, all input vectors are multiplied by the same matrix. For example, an MMI may be used instead of a directional coupler. In other embodiments, the VBS may be selected to be dispersive within the M wavelengths of interest. In some applications related to stochastic optimization of parameters of a neural network model, this is equivalent to adding noise when calculating the gradient of the parameters, and increasing the gradient noise may be advantageous for faster optimization convergence and may improve the robustness of the neural network.

[0103] Although FIG. 1-12A shows a continuous photonic processing system, similar WDM techniques can be used to form a WDM folded photonic processor using the techniques described in connection with folded photonic processor 1-1100.

[0104] H. Analog summing of outputs The inventors have recognized and appreciated that there are applications in which it is useful to calculate the sum of the averages of the outputs from the photonic processor 1-103 over time. For example, when the photonic processing system 1-100 is used to calculate a more rigorous matrix-vector multiplication of a single data point, it may be desirable to pass the single data point through the photonic processor multiple times to improve the statistical results of the calculation. Additionally or alternatively, when calculating gradients in a backpropagation machine learning algorithm, one may not want a single data point to determine the gradient, and thus multiple training data points may be passed through the photonic processing system 1-100 and the average result used to calculate the gradient. When the photonic processing system is used to run a batch gradient-based optimization algorithm, this averaging may increase the quality of the gradient estimates, thereby reducing the number of optimization steps required to achieve a high quality solution.

[0105] The inventors have further recognized and appreciated that the output signals may be summed in the analog domain before converting the outputs to digital electrical signals. Thus, in some embodiments, low pass filters are used to sum the outputs from the homodyne detectors. By performing the summation in the analog domain, the homodyne electronics may use slower speed ADCs rather than the more expensive faster ADCs (e.g., ADCs with higher power consumption requirements) required to perform the summation in the digital domain.

[0106] FIG. 1-13 shows a portion of an optical receiver 1-1300 according to some embodiments and how a low pass filter 1-1305 can be used in conjunction with a homodyne detector 1-1301. The homodyne detector 1-1301 performs measurements of the field and phase of the input optical pulse. If k is the label of the different input pulses over time, and there are a total of K inputs, the summation over k can be performed automatically in the analog domain using the low pass filter 1-1305. The main difference between this optical receiver 1-1300 and the optical receiver 1-105 shown in FIG. 1-9 is that the low pass filter is after the transimpedance amplifier 1-1303 after the output of the homodyne detector. A total of K signals (components y i (k) ) has one slow sampling period T s (slow) If the homodyne detector is reached within y i (k) The final output of the low pass filter is

[0107]

number

[0108] which is proportional to the sampling frequency f s (slow) =1 / T s (slow) =f s / K, where fs is the originally required sampling frequency. In an ideal system, the low pass filter should have a 3 dB bandwidth: 3dB =f s (slow) For a low-pass filter using an RC circuit as shown in the embodiment of Figure 1-13, f 3dB = 1 / (2πRC), and the values ​​of R and C are determined by the desired sampling frequency: f s (slow) can be chosen to obtain

[0109] In some embodiments, there may be both high-speed and low-speed ADCs. In this regard, a high-speed ADC is an ADC configured to receive and convert each individual analog signal into a digital signal (e.g., an ADC with a sampling frequency equal to or greater than the frequency at which the analog signals arrive at the ADC), and a low-speed ADC is an ADC configured to receive multiple analog signals and convert the sum or average of the received multiple analog signals into one digital signal (e.g., an ADC with a sampling frequency less than the frequency at which the analog signals arrive at the ADC). An electrical switch may be used to switch from the homodyne detector and possibly the transimpedance amplifier to a low-pass filter or high-speed ADC with a low-speed ADC. In this way, the photonic processing system of some embodiments may switch between performing analog summation using a low-speed ADC and measuring all optical signals using a high-speed ADC.

[0110] I. Phase stabilization The inventors have recognized and appreciated that it is desirable to stabilize the phase of a local oscillator used to perform phase-sensitive measurements (e.g., homodyne detection) to ensure accurate results. The photonic processor of the embodiments described herein performs matrix operations by interfering light between N distinct spatial modes. In some embodiments, the results are measured using a phase-sensitive detector, such as a homodyne or heterodyne detector. Thus, to ensure that the matrix operations are performed accurately, the phases imposed on the various parts of the photonic processor should be as accurate as possible, and the phase of the local oscillator used to perform the phase-sensitive detection should be precisely known.

[0111] The inventors have recognized and understood that not only should parallel interference operations such as those performed within a row of VBSs in a photonic processor impart precise phase using phase modulators that control the relative phase within the MZIs of the VBSs and the phase of the outputs of the MZIs, but that each VBS in the row should impart the same global phase shift across all spatial modes of the photonic processor. In this application, the global phase shift of a row of VBSs in a photonic processor is referred to as the "row-global phase." Row-global phases are imparted due to effects associated with the VBSs that are not related to the programmed phase, such as phase imparted due to propagation through the waveguide or phase due to temperature shifts. These phases do not need to be imparted exactly simultaneously within a row on the VBS, but only as a result of traversing the row of interest. Ensuring that the row-global phase is uniform across the different spatial modes of the row is important because the output optical signal from one row is likely to be interfered at one or more VBSs in the following row. If the row-global phase in the previous row is not uniform, the subsequent interference (and therefore the accuracy of the calculation itself) will be inaccurate.

[0112] FIG. 1-14 shows the column-global phase and total global phase of a photonic processing system 1-1400. As with the above embodiments of a photonic processing system, the photonic processing system 1-1400 includes a U matrix implementation 1-1401, a diagonal matrix implementation 1-1403, a V matrix implementation 1-1405, and multiple detectors 1-1407a through 1-1407d. These implementations are similar to the first, second, and third matrix implementations described above. For simplicity, only four modes of the photonic processing system 1-1400 are shown, but it should be understood that any greater number of modes may be used. Also, for simplicity, only the VBS associated with the U matrix implementation 1-1401 is shown. The arrangement of components of the diagonal matrix implementation 1-1403 and the V matrix implementation 1-1405 are similar to the third and fourth matrix implementations described above.

[0113] U-matrix implementation 1-1401 includes multiple VBS1-1402, but only one VBS1-1402 is labeled for clarity, however, the VBSs are labeled with a subscript that identifies which optical mode is being mixed by that particular VBS and a superscript that denotes the associated column.

[0114] As shown in FIG. 1-14, each column is associated with a column-global phase that is ideally uniform across all elements of the column. For example, column 1 of the U matrix implementation 1-1401 has a column-global phase Φ U1 and column 2 of the U matrix implementation 1-1401 is associated with the column-global phase Φ U2 and column 3 of the U matrix implementation 1-1401 is the column-global phase Φ U3 and column 4 of the U matrix implementation 1-1401 is associated with column-global phase Φ4.

[0115] In some embodiments, the column-global phase can be made uniform, at least in part, by implementing each VBS1-1402 as an MZI in a push-pull configuration. Alternatively or additionally, an external phase shifter can be added to the output of each MZI to correct any phase error imparted from the internal phase elements (e.g., phase shifters) of the MZI.

[0116] The inventors have further recognized and appreciated that even if each column of the photonic processing system 1-1400 provides a uniform column-global phase, phase may develop as the signal propagates from the first column to the last column. U which is equal to the sum of the individual column-global phases. Similarly, the diagonal matrix implementation 1-1403 has a global diagonal matrix phase Φ Σ , and the V matrix implementation 1-1405 is related to the global diagonal matrix phase Φ V† Therefore, the total global phase Φ of the entire photonic processing system 1-1400 is G is given by the sum of the three individual global matrix phases. This total global phase can be set to be uniform among all output modes, but the local oscillators used for phase-sensitive detection did not propagate through the photonic processor and were not subjected to this total global phase. The total global phase Φ G If not accounted for, this can lead to errors in the values ​​read by the homodyne detectors 1-1407a to 1-1407d.

[0117] The inventors have determined that the error in the multiplication operation is proportional to the effective refractive index n effIt has further been recognized that the phase difference between the local oscillator signal and the local oscillator may result from temperature changes that change the phase of the local oscillator signal. Thus, in some embodiments, a stabilization circuit may be placed in each circuit such that the temperature of each string is set uniformly or actively adjusted so that the phase imparted to all modes in a string is uniform. Furthermore, as the local oscillator optical signal propagates through different parts of the system, temperature differences between different parts of the system may introduce errors into phase sensitive measurements. The amount of phase difference between the signal and the local oscillator is:

[0118]

number

[0119] where T S and T LO are the temperatures of the signal waveguide and the local oscillator waveguide in the photonic processor, respectively, and n eff (T) is the effective refractive index as a function of temperature, λ is the mean wavelength of light, and L S and L LO are the propagation lengths through the signal and local oscillator waveguides in the photonic processor, respectively. The temperature difference ΔT = T LO -T S Assuming that is small, the effective refractive index is

[0120]

number

[0121] Therefore, the phase difference between the signal and the LO is

[0122]

number

[0123] This can be well approximated by, which increases linearly with the propagation length L. Thus, for a sufficiently long propagation distance, a small temperature change can produce a large phase shift (on the order of 1 radian). Importantly, L S The value of L LO The maximum difference between the two is the coherence length L of the source. coh For a source with bandwidth Δν, the coherence length is given by

[0124]

number

[0125] can be better approximated by eff is the speed of light in the transmission medium. L S and L LO The difference in length between coh As long as the interference between the signal and the local oscillator is much shorter than 100 MHz, accurate operation of the photonic processing system is possible.

[0126] Based on the above, the inventors have identified at least two sources of phase error that may occur between the output signal of the photonic processor in some embodiments and the local oscillator used for homodyne detection. Thus, an ideal homodyne detector would measure the magnitude and phase of the signal output by subtracting the outputs of two photodetectors, resulting in a phase-sensitive intensity output measurement I out ∝│E s ││E LO │cos(θ s -θ LO +Φ G +Φ T ) occurs, where E S is the electric field magnitude of the optical signal from the output of the photonic processor, E LO is the electric field magnitude of the local oscillator, and θ S is the phase shift imparted by the photonic processor that one wishes to measure, and φ G is the total global phase, and φ Tis the phase shift caused by the temperature difference between the local oscillator and the optical signal. Therefore, if the total global phase and the phase shift due to the temperature difference are not accounted for, the results of homodyne detection may be erroneous. Thus, in some embodiments, the total systematic phase error Δφ=φ G +φ T is measured and the system is calibrated based on that measurement. In some embodiments, the total systematic phase error includes contributions from other sources of error that are not necessarily known or identified.

[0127] According to some embodiments, a homodyne detector may be calibrated by sending a pre-calculated test signal to the detector and using the difference between the pre-calculated test signal and the measured test signal to correct for the total systematic phase error in the system.

[0128] In some embodiments, a total global phase φ as associated with an optical signal propagating through a photonic processor may be G and the phase shift φ caused by the temperature difference T Rather than considering the signal [x,p] as a signal that does not introduce any phase shift and that introduces a LO with a total systematic phase error -Δφ, Figure 1-15 shows the effect on the results of a homodyne measurement in such a situation. T The original (precise) vector of quadrature values ​​of is the inexact quadrature values ​​[x',p'] T φ which produces

[0129] Based on the quadrature rotation due to the total systematic error, in some embodiments, the value of Δφ is obtained as follows:

[0130]

number

[0131] is selected (e.g., a random vector) using, for example, the controller 1-107. The vector is of a type that can be provided by an optical encoder in a photonic processing system.

[0132]

number

[0133] The output value of is calculated, for example, using the controller 1-107 or some other computing device, where M is a matrix implemented by the photonic processor in the ideal case assuming no unknown phase caused by Δφ.

[0134]

number

[0135] Each element of x k +ip k where k denotes each output mode of the photonic processor. In some embodiments, the loss in propagating a random vector through a photonic processor is less than the theoretical prediction x k +ip k For example, for a photonic processor with a transmission efficiency η, x k +ip k The field signal is

[0136]

number

[0137] become. Next, the random vector

[0138]

number

[0139] is prepared by the optical encoder of the actual system and propagates through the photonic processor, and each element of the output vector is measured in both quadratures to obtain x k '+ip k ' is obtained. The phase difference Δφ between the local oscillator and the signal of output mode k k teeth,

[0140]

number

[0141] (In general, the path length of the LO to the detector in mode k may be different from that in mode l, so that for k ≠ l, Δφ k ≠Δφ l (It is.) Finally, the local oscillator phase shifter used to select the measurement quadrature of the homodyne detector is LO,k = Δφ k As a result, the axis (x,p) is aligned with the axis (x',p') as shown in Figure 1-15. At this stage, the calibration is checked and the vector

[0142]

number

[0143] If both quadratures are measured by propagating

[0144]

number

[0145] Correctness can be ensured by checking again that ≠ 0 is equal to ≠ 0. In general, Δφ k The value of is the field amplitude

[0146]

number

[0147] If is as large as possible, then a more precise specification can be obtained. For example, in the field E S,k An optical signal can theoretically be modeled as a coherent state if |E| is considered to be a coherent signal, e.g., from a laser source. An intuitive illustration is given in Figure 1-16, where a signal has an amplitude |E S,k |, where the noise is given by the standard deviation of the Gaussian coherent states. k > is the annihilation operator α k is an eigenstate of α k |α k >=α k |α k > The electric field of mode k with one frequency ω is given by

[0148]

number

[0149] which is also an eigenstate of the coherent state.

[0150]

number

[0151] A homodyne detector with a local oscillator of the same frequency ω has a frequency of θ LO If =0, quadrature measurement

[0152]

number

[0153] Execute θ LO If = π / 2,

[0154]

number

[0155] An ideal homodyne detector would perform these measurements as follows:

[0156]

number

[0157] We find that the eigenvalue has an intrinsic quantum noise of . This noise is related to quantum uncertainty and can be reduced by squeezing the quadrature. The angle Δφ k The accuracy with which we can determine N is directly related to the signal-to-noise ratio (SNR) of these measurements. ph A coherent state signal E with photons S,k (i.e.

[0158]

number

[0159] If x k and p k The SNRs of both are

[0160]

number

[0161] The upper limit is set by (SNR x The limit of θ S If = 0 or π, it saturates and the SNR p The limit to θ S =π / 2 or 3π / 2, it saturates. Therefore, to increase the SNR, Δφ k To more precisely specify the value of , some embodiments use the vector

[0162]

number

[0163] In some embodiments, a small number of different choices of (e.g., multiple different random vectors) of

[0164]

number

[0165] The choice of k is made by selecting the amplitude |E S,k |=N ph is chosen to maximize For example, there may be phase drift during operation of the photonic processing system due to temperature variations over time. Thus, in some embodiments, the calibration procedure may be performed repeatedly during operation of the system. For example, in some embodiments, the calibration procedure is performed periodically on a time scale that is shorter than the natural time scale of phase drift.

[0166] The inventors have further realized and understood that it is possible to perform signed matrix operations without requiring any phase-sensitive measurements. Thus, in applications, each homodyne detector at each output mode can be replaced with a direct photodetector that measures the intensity of light at that output mode. In such a system, since there is no local oscillator, the systematic phase error Δφ is nonexistent and meaningless. Thus, according to some embodiments, phase-sensitive measurements such as homodyne detection can be avoided so that the systematic phase error is insignificant. For example, when calculating matrix operations of signed matrices and vectors, complex matrices and vectors as well as multi-dimensional (quaternary, octanerial and other isomorphic (e.g., elements of unitary algebra)) matrices and vectors using unsigned matrices do not require phase-sensitive measurements.

[0167] To show how phase-sensitive measurements are not needed, we consider the signed matrix M and the signed vector

[0168]

number

[0169] Consider performing a matrix multiplication between a signed output

[0170]

number

[0171] To calculate the value of M, the following procedure may be performed by the controller 1-107, for example. + and M - where M + (M - ) is a matrix that contains all the positive (negative) entries of M. In this case, M=M + -M - Second, a vector is a vector

[0172]

number

[0173] where

[0174]

number

[0175] teeth,

[0176]

number

[0177] is a vector containing all positive (negative) entries of . As a result of the division,

[0178]

number

[0179] Each term in this final equation is a separate operation that can be individually performed by a photonic processing system.

[0180]

number

[0181] The output of each operation is a vector of one (positive) sign and can therefore be measured using direct detection methods without the need for homodyne detection. Photodetector methods measure intensity, but the square root of the intensity can be determined to produce the electric field amplitude. In some embodiments, each operation is performed separately and the results are calculated by performing all the separate operations and combining the results digitally to produce the final result of the multiplication.

[0182]

number

[0183] is stored in a memory (eg, memory 1-109 of controller 1-107) until M + and M - The above scheme works because both are matrices with all positive entries.

[0184]

number

[0185] are both vectors with all positive entries, so the result of multiplication is a vector with all positive entries - regardless of connections. The inventors have further recognized and appreciated that the above partitioning techniques can be extended to complex-valued vectors / matrices, quaternion-valued vectors / matrices, octonion-valued vectors / matrices, and other multi-dimensional representations. Complex numbers utilize two distinct base units {1,i}, quaternions utilize four distinct base units {1,i,j,k}, and octonions utilize eight base units {e0≡1,e1,e2,...,e7}.

[0186] In some embodiments, a complex vector may be multiplied by a complex matrix without requiring phase-sensitive detection by splitting the multiplication into separate operations similar to the procedure described above for signed matrices and vectors. For complex numbers, the multiplication is split into 16 separate multiplications of an all-positive matrix and an all-positive vector. The results of the 16 separate multiplications may then be digitally combined to determine the output vector result.

[0187] In some embodiments, a quaternion-valued vector may be multiplied by a quaternion-valued matrix without the need for phase-sensitive detection by splitting the multiplication into separate operations similar to the procedure described above for signed matrices and vectors. For quaternion-valued numbers, the multiplication is split into 64 separate multiplications of an all-positive matrix and an all-positive vector. The results of the 64 separate multiplications may then be digitally combined to determine the output vector result.

[0188] In some embodiments, an octonary vector may be multiplied by an octonary matrix without requiring phase-sensitive detection by splitting the multiplication into separate operations similar to the procedure described above for signed matrices and vectors. For octonary numbers, the multiplication is split into 256 separate multiplications of an all-positive matrix and an all-positive vector. The results of the 256 separate multiplications may then be digitally combined to determine the output vector result.

[0189] We have determined that the temperature-dependent phase φ T It has further been recognized and understood that the temperature measurement can be compensated for by placing a temperature sensor next to each MZI in the photonic processor. The temperature measurement can then be used as an input to a feedback circuit that controls the external phase of each MZI. The external phase of the MZI is set to cancel the temperature dependent phase occurring in every MZI. A similar temperature feedback loop can be used in the local oscillator propagation path. In this case, the temperature measurement is used to inform the setting of a homodyne detector quadrature selective phase shifter to cancel the phase introduced by the local oscillator due to the detected temperature effects.

[0190] In some embodiments, the temperature sensor may be one that is conventionally used in semiconductor devices, such as a pn junction or bipolar junction transistor, or may be a photonic temperature sensor, for example using a resonator whose resonance changes with temperature. In some embodiments, an external temperature sensor, such as a thermocouple or thermistor, may also be used.

[0191] In some embodiments, the resulting phase can be measured directly, for example by tapping some light at every column and performing homodyne detection with the same global local oscillator. This phase measurement can directly inform the value of the external phase to use at each MZI to correct any phase error. In the case of directly measured phase error, the error does not need to be column global in order to be corrected.

[0192] J. Large Data Intermediate Calculation The inventors have recognized and appreciated that the matrix-vector products performed by photonic processor 1-103 and / or any other photonic processor according to other embodiments described in this disclosure can be generalized to tensor (multi-dimensional array) operations. For example, if M is a matrix,

[0193]

number

[0194] is a vector

[0195]

number

[0196] The core operation of can be generalized to matrix-matrix multiplication: MX, where both M and X are matrices. In this particular example, consider an n-by-m matrix X as a set of m column vectors, each with n elements. That is,

[0197]

number

[0198] The photonic processor can complete the matrix-matrix product MX one column vector at a time, resulting in a total of m matrix-vector products. The computations are preferably parallelizable linear operations and can therefore be distributed across multiple photonic processors, e.g., the output of any one matrix-vector product does not depend on the results of other matrix-vector products. Alternatively, the computations can be performed sequentially by one photonic processor over time, e.g., by performing each matrix-vector product one at a time and then digitally combining the results (e.g., by storing the results in an appropriate memory configuration) to determine the result of the matrix-matrix product.

[0199] The above concept can be generalized to computing the product (e.g., dot product) of two multidimensional tensors. The general algorithm is as follows, and may be executed at least in part by a processor such as processor 1-111: (1) take a matrix slice of the first tensor; (2) take a vector slice of the second tensor; (3) use a photonic processor to perform a matrix-vector product of the matrix slice in step 1 and the vector slice in step 2 to generate an output vector; (4) iterate over the tensor indexes obtained from the matrix slice (from step 1) and the tensor indexes obtained from the vector slice (from step 2). Note that when taking the matrix slice and the vector slice (steps 1 and 2), multiple indices may be combined into one. For example, a matrix may be vectorized by stacking all columns into one column vector, and in general, a tensor may be matrixized by stacking all matrices into one matrix. All operations are fully linear, so again this can be highly parallelized, and each of the multiple photonic processors does not need to know whether the other photonic processors have completed their jobs.

[0200] As a non-limiting example, multiplication between two three-dimensional tensors C ijlm =Σ k A ijk B klm The pseudocode based on the above procedure is as follows: (1) Take a matrix slice. A i ←A[i,:,:], (2) Take a vector slice.

[0201]

number

[0202]

number

[0203] (4) Iterate over indices i, l, and m to find the four-dimensional tensor C ijlm where the values ​​of all j-indexed elements are fully specified using one matrix-vector multiplication.

[0204] The inventors further recognized and understood that the size of the multiplying matrix / vector may be larger than the number of modes supported by the photonic processor. For example, a convolution operation in a convolutional neural network architecture may use only a few parameters to define a filter, but may consist of several matrix-matrix multiplications between the filter and different patches of data. Combining the different matrix-matrix multiplications produces two input matrices that are larger than the size of the original filter matrix or data matrix.

[0205] The inventors have devised a method for performing matrix operations using photonic processors when the matrices to be multiplied are larger than the size of the photonic processor being used to perform the calculations / the number of modes the photonic processor being used to perform the calculations has. In some embodiments, the method includes using a memory to store intermediate information during the calculations. The final calculation result is calculated by processing the intermediate information. For example, as shown in Figure 1-17, consider the multiplication 1-1700 of an I×J matrix A and a J×K matrix B to give a new matrix C=AB having I×K elements, using an n×n photonic processing system, where n≦I, J, K. In Figure 1-17, the shaded elements simply indicate that element 1-1701 of matrix C is calculated using elements in row 1-1703 of matrix A and column 1-1705 of matrix B. The method illustrated in Figures 1-17 and 1-18 is as follows.

[0206] Construct n×n submatrix blocks in matrices A and B. Parenthesized superscript A (ij) and B. (jk) Denote the blocks by , where i ∈ {1, ..., ceil(I / n)}, j ∈ {1, ..., ceil(J / n)}, k ∈ {1, ..., ceil(K / n)}. If the values ​​of I, j, or K are not divisible by n, the matrix may be padded with zeros so that the new matrix has dimensions divisible by n (hence the ceil function at the indices of i, j, and k). In an example multiplication 1-1800 shown in FIG. 1-18, matrix A is split into six n x n sub-matrix blocks 1-1803 and matrix B is split into three n x n sub-matrix blocks 1-1805 to generate a result matrix C composed of two n x n sub-matrix blocks 1-1801.

[0207] n×n submatrix block C in matrix C (ik) To calculate, for example, (1) Submatrix A (ij) (e.g., one of sub-matrices 1-1803); (2) Submatrix B (jk)(e.g., one of submatrices 1-1805), and propagating the signal through a photonic processor; (3) storing the intermediate results of each matrix-vector multiplication in memory; (4) iterating through the values ​​of j, repeating steps (a) through (c); (5) A digital electronic circuit, e.g., a processor, is used to combine the intermediate results to produce the final sub-matrix C (ik) (e.g., calculate one of the sub-matrices 1-1801) This allows multiplication in the photonic processor.

[0208]

number

[0209] Execute. As discussed above and illustrated in Figures 1-17 and 1-18, the method may include expressing matrix multiplication using bracketed index notation and performing the matrix-matrix multiplication operation using bracketed superscript indices instead of subscript indices used in this disclosure to describe matrix elements. These bracketed superscript indices correspond to n x n blocks of sub-matrices. In some embodiments, the method can be generalized to tensor-tensor multiplication, for example, by combining the method with the tensor-tensor multiplication described above to divide the multidimensional array into n x n sub-matrix block slices.

[0210] In some embodiments, an advantage of using a photonic processor with fewer modes to process blocks of sub-matrices is that it provides diversity with respect to the shape of the matrices being multiplied. For example, if I>>J, then performing a singular value decomposition requires a block of sub-matrices of size I 2 The first unitary matrix of size J 2 and a diagonal matrix with J parameters. 2The hardware requirements for storing or processing matrix elements may be too large for the number of optical modes included in some embodiments of the photonic processor. By processing sub-matrices rather than the entire matrix all at once, matrices of any size can be multiplied without imposing limitations based on the number of modes of the photonic processor.

[0211] In some embodiments, the sub-matrices of B are further vectorized. For example, matrix A is first

[0212]

number

[0213] The matrix can then be padded with

[0214]

number

[0215] Divided into grids (each of size [n×n]), A (ij) is the [n × n] submatrix in the i-th row and j-th column of this grid, and B is first

[0216]

number

[0217] The matrix is ​​padded, then the submatrix

[0218]

number

[0219] The grid is divided into n×K grids, each of which has size [n×K]. (j) is the [n × K] submatrix in the j-th row of this grid, and C is first

[0220]

number

[0221] The matrix is ​​padded, then the submatrix

[0222]

number

[0223] It is divided into grids (each of size [n×K]), C (i) is the [n × K] submatrix in the i-th row of this grid. In this vectorized form, the computation is

[0224]

number

[0225] As shown by: Using the vectorization process described above, the photonic processor:

[0226]

number

[0227] Any GEMM can be computed by loading K distinct matrices into the photonic array and, for each loaded matrix, propagating K distinct vectors through the photonic array. This can be done by

[0228]

number

[0229] The input vectors are then summed together to produce the desired [I×K] output matrix, as defined by the equation above.

[0230] K. Accuracy of Calculations The inventors have recognized and appreciated that photonic processors 1-103 and / or any other photonic processors according to other embodiments described in this disclosure are examples of analog computers, and because most data in this information age is stored in digital representations, it is important to quantify the digital precision of the calculations performed by the photonic processor. In some embodiments, a photonic processor according to some embodiments may be configured to perform a matrix-vector product:

[0231]

number

[0232] Execute the formula:

[0233]

number

[0234] is the input vector, M is an n-by-n matrix,

[0235]

number

[0236] is the output vector. In index notation, this multiplication is ij (repeat over j) and n elements of x j (repeat over j) and then add the results together.

[0237]

number

[0238] Since photonic processors are physically analog systems, in some embodiments, the element M ij and x jis expressed using a fixed-point representation. Within this representation, M ij ∈{0,1} m1 is the number of m1 bits, and X j ∈{0,1} m2 is an m-bit number, then a total of m1+m2+log2(n) bits are required for the resulting vector element y i is used to completely represent. Typically, the number of bits used to represent the result of a matrix-vector product is greater than the number of bits required to represent the inputs of the operation. If the analog-to-digital converter (ADC) used cannot read the output vector with full precision, the output vector elements may be rounded to the precision of the ADC.

[0239] The inventors have recognized and appreciated that building an ADC with high bit precision at a bandwidth corresponding to the rate at which an input vector in the form of an optical signal is transmitted through a photonic processing system can be difficult to achieve. Thus, in some embodiments, the bit precision of the ADC is determined by the matrix element M ij and vector element x j One may limit the bit precision with which M is expressed (if full precision computation is desired). Therefore, we have devised a method to obtain the output vector with full precision, which can be arbitrarily high, by computing partial products and sums. For clarity, M ij or x j is assumed to be the same, i.e., m1 = m2 = m. However, this assumption may nevertheless not be necessary in general and does not limit the scope of the embodiments of the present disclosure.

[0240] As a first operation, a method according to some embodiments calculates a matrix element M ij and vector element x j It involves dividing the bit string representation of M into d partitions, each partition having k=m / d bits. (If k is not an integer, zeros may be appended until m is divisible by d.) The resulting matrix elements are ij =M ij [0] 2 k(d-1) +Mij [1] 2 k(d-2) +...+M ij [d-1] 2 0 where M ij [a] is M ij The k-bit value of the a-th most significant k-bit string of M ij =M ij [0] M ij [1] ...M ij [d-1] Similarly, x j =x j [0] 2 k(d-1) +x j [1] 2 k(d-2) +...+x j [d-1] 2 0 where the vector element x j =x j [0] x j [1] ...x j [d-1] Multiplication y i =Σ j M ij x j teeth,

[0241]

number

[0242] where the set S p is the set of all integer values ​​of a and b, where a+b=p. As a second operation, the method ij [a] and controlling the photonic processor to implement j [b]and propagating y through the photonic processor in the form of an encoded optical signal. This matrix-vector multiplication operation results in y i [a,b] =Σ j M ij [a] x j [b] The method performs a linear transformation on the output vector y i [a,b] This includes storing the

[0243] The method is to p repeating the second operation for each of the different values ​​of a and b; and obtaining an intermediate result y i [a,b] and storing the

[0244] As a third operation, the method further comprises using digital electronic circuitry, such as a processor, to sum over the different iterations of a and b to obtain a final result.

[0245]

number

[0246] This includes calculating The precision of the ADC used to capture the highest precision calculations according to some embodiments of this method is only 2k+log2(n) bits, which is less than the 2m+log2(n) bits of precision that would be required if the calculations were performed using only a single pass.

[0247] The inventors have further recognized and appreciated that the above method embodiments can be generalized to operate on tensors. As previously mentioned, the photonic processing system can perform tensor-tensor multiplication by using a matrix slice and a vector slice of two tensors. The above described method can be applied to the matrix slice and the vector slice to obtain an output vector slice of an output tensor with full precision.

[0248] Some embodiments of the method use the linearity of the base representation of the matrix. In the above description, the matrix is ​​expressed in terms of the Euclidean matrix space, and matrix-vector multiplication is linear in this Euclidean space. In some embodiments, the matrix is ​​expressed in terms of the phase of the VBS, and therefore the division can be performed on the bit strings representing the phase, rather than directly on the matrix elements. In some embodiments, if the map between the phase and the matrix elements is a linear map, then the relationship between the input parameters - in this case the phase of the VBS and the input vector elements - and the output vector is linear. If this relationship is linear, then the method is still applicable. However, in general, according to some embodiments, a non-linear map from the base representation of the matrix to the photonic representation can be considered. For example, a bit string division of the Euclidean space matrix elements from the most significant k bit strings to the least significant k bit strings can be decomposed into a topological representation and used to generate a set of different matrices to implement using a photonic processor.

[0249] The division does not have to be performed simultaneously on both the matrix elements and the input vector elements. In some embodiments, the photonic processor may propagate many input vectors of the same matrix. Since the digital-to-analog converters (DACs) for vector preparation may operate at a wide bandwidth, while the DACs for VBS may be quasi-static for multiple vectors, it may be efficient to perform the division only on the input vectors and keep the VBS control at a set precision (e.g., maximum precision). In general, it is more difficult to design a DAC with high bit precision at a high bandwidth than at a low bandwidth. Thus, in some embodiments, the output vector elements may be more precise than the ADC allows, but the ADC automatically performs some rounding of the output vector values ​​to the bit precision allowed by the ADC.

[0250] L. Manufacturing method Embodiments of the photonic processing system may be fabricated using conventional semiconductor fabrication techniques, for example, the waveguides and phase shifters may be formed in a substrate using conventional deposition, masking, etching and doping techniques.

[0251] FIG. 1-19 illustrates an example method 1-1900 of fabricating a photonic processing system according to some embodiments. In operation 1-1901, method 1-1900 includes forming an optical encoder, for example, using conventional techniques. For example, a plurality of waveguides and modulators may be formed in a semiconductor substrate. The optical encoder may include one or more phase and / or amplitude modulators, as described elsewhere herein.

[0252] In operation 1-1903, method 1-1900 includes forming a photonic processor and optically connecting the photonic processor to the optical encoder. In some embodiments, the photonic processor is formed on the same substrate as the optical encoder and the optical connection is made using waveguides formed in the substrate. In other embodiments, the photonic processor is formed on a substrate separate from the substrate of the optical encoder and the optical connection is made using optical fibers.

[0253] In operation 1-1905, method 1-1900 includes forming an optical receiver and optically connecting the optical receiver to the photonic processor. In some embodiments, the optical receiver is formed on the same substrate as the photonic processor and the optical connection is made using a waveguide formed in the substrate. In other embodiments, the optical receiver is formed on a substrate separate from the substrate of the photonic processor and the optical connection is made using an optical fiber.

[0254] FIG. 1-20 illustrates an example method 1-2000 of forming a photonic processor as shown in operation 1-1903 of FIG. 1-19. In operation 1-2001, method 1-2000 includes forming a first optical matrix implementation, for example, in a semiconductor substrate. The first optical matrix implementation may include an array of interconnected VBSs, as described in various embodiments above.

[0255] In operation 1-2003, method 1-2000 includes forming a second optical matrix implementation and connecting the second optical matrix implementation to the first optical matrix implementation. The second optical matrix implementation may include one or more optical components capable of controlling the intensity and phase of each optical signal received from the first optical matrix implementation, as described in various embodiments above. The connection between the first optical matrix implementation and the second optical matrix implementation may include a waveguide formed in the substrate.

[0256] In operation 1-2005, the method 1-2000 includes forming a third optical matrix implementation and connecting the third optical matrix implementation to the second optical matrix implementation. The third optical matrix implementation may include an array of interconnected VBSs as described in various embodiments above. The connection between the second optical matrix implementation and the third optical matrix implementation may include a waveguide formed in the substrate.

[0257] In any of the above operations, the components of the photonic processor may be formed in the same layer of a semiconductor substrate or in different layers of the semiconductor substrate. M.How to use FIG. 1-21 illustrates a method 1-2100 for performing optical processing according to some embodiments. In operation 1-2101, the method 1-2100 includes encoding a bit sequence into an optical signal. In some embodiments, this may be performed using a controller and an optical encoder as described in connection with various embodiments of the present application. For example, a complex number may be encoded into the intensity and phase of the optical signal.

[0258] In operation 1-2103, method 1-2100 includes controlling the photonic processor to implement a first matrix. As described above, this can be accomplished by having the controller perform an SVD on the matrix and split the matrix into three separate matrix components that are implemented using separate parts of the photonic processor. The photonic processor can include multiple interconnected VBSs that control how various modes of the photonic processor mix together to coherently interfere with the optical signal as it propagates through the photonic processor.

[0259] In act 1-2105, method 1-2100 includes propagating optical signals through an optical processor such that the optical signals coherently interfere with one another to implement a desired matrix, as described above.

[0260] In operation 1-2107, method 1-2100 includes detecting the output optical signal from the photonic processor using an optical receiver. As described above, the detection may use a phase-sensitive or phase-insensitive detector. In some embodiments, the detection result is used to determine a new input bit sequence that is encoded and propagated through the system. In this manner, multiple calculations may be performed sequentially, where at least one calculation is based on the result of a previous calculation result.

[0261] II. Training Algorithm The present inventors have recognized and understood that in many matrix-based differentiable program (e.g., neural networks or latent variable graphical models) techniques, a large portion of the computational complexity resides in the matrix-matrix products that are computed as layers of the model are traversed. The complexity of the matrix-matrix products is O(IJK), where the two matrices have dimensions I×J and J×K. Furthermore, these matrix-matrix products are performed during both the training and evaluation phases of the model.

[0262] Deep neural networks (i.e., neural networks with two or more hidden layers) are one example of a type of matrix-based differentiable program that may utilize some of the techniques described herein. However, it should be understood that the techniques described herein for performing parallel processing may be used with other types of matrix-based differentiable programs, including, but not limited to, Bayesian networks, trellis decoders, topic models, and hidden Markov models (HMMs).

[0263] The success of deep learning is due in large part to the development of the backpropagation technique, which allows for the training of the weight matrix of a neural network. In traditional backpropagation techniques, the error from a loss function is propagated backwards through the individual weight matrix elements using the chain rule of calculus. The backpropagation technique calculates the gradient of the elements in the weight matrix, which then determines updates to the weight matrix using an optimization algorithm such as stochastic gradient descent (SGD), AdaGrad, RMSProp, Adam, or any other gradient-based optimization algorithm. A succession of this procedure is used to determine the final weight matrix that minimizes the loss function.

[0264] The inventors have recognized and appreciated that optical processors of the type described herein enable gradient calculations to be performed by reformulating the weighting matrix into an alternative parameter space, referred to herein as the "phase space" or "angle representation." In particular, in some embodiments, the weighting matrix is ​​reparameterized as a composite of unitary transfer matrices, such as the Givens rotation matrix. In such a reparameterization, training the neural network involves adjusting the angular parameters of the unitary transfer matrices. In this reparameterization, the gradients of a single rotation angle are separated from other rotations, allowing for parallel computation of the gradients. This parallelization results in a computational acceleration in terms of the number of computational steps required, compared to conventional serial gradient determination techniques.

[0265] An example photonic processing system that may be used to implement the backpropagation techniques described herein is provided above. The phase space parameters of the reparameterized weighting matrix may be encoded into a phase shifter or variable beam splitter of the photonic processing system to implement the weighting matrix. The encoding of the weighting matrix into a phase shifter or variable beam splitter may be used for both the training and evaluation phases of the neural network. The backpropagation procedure is described with reference to a particular system below, but it should be understood that the embodiments are not limited to the particular details of the photonic processing system described in this disclosure.

[0266] As mentioned above, in some embodiments, photonic processing system 100 may be used to implement aspects of neural networks or other matrix-based differentiable programs that may be trained using backpropagation techniques.

[0267] An example backpropagation technique 2-100 for updating a matrix of values ​​(eg, the weight matrix of a layer of a neural network) in a Euclidean vector space of a differentiable program (eg, a neural network or a latent variable graphical model) is shown in FIG. 2-1.

[0268] In act 2-101, a matrix of values ​​in Euclidean vector space (e.g., a weight matrix of a layer of a neural network) may be represented as an angle representation, for example, by configuring components of the photonics processing system 100 to represent the matrix of values. After the matrix is ​​represented in an angle representation, the process 2-100 proceeds to act 2-102, where the training data (e.g., a set of input training vectors and associated labeled outputs) are processed to calculate an error vector by evaluating a performance measure of the model. The process 2-100 then proceeds to act 2-103, where gradients of at least some of the parameters in the angle representation required for backpropagation are determined in parallel. For example, as described in more detail below, the techniques described herein allow the gradients of an entire column of parameters to be determined simultaneously, significantly accelerating the amount of time required to perform backpropagation as compared to individually evaluating the gradients for each angle rotation. Process 2-100 then proceeds to operation 2-104, where a matrix of values ​​in Euclidean vector space (e.g., weight matrix values ​​of a layer of a neural network) is updated by updating the angle representation using the determined gradient. Further description of each operation shown in process 2-100 of FIG. 2-1 is provided below.

[0269] FIG. 2-2 shows a flow chart of how operation 2-101 shown in FIG. 2-1 may be performed according to some embodiments. In operation 2-201, a controller (e.g., controller 107) may receive weight matrices for a layer of a neural network. In operation 2-202, the weight matrices may be calculated by: W=V T ΣU The weight matrix W may be decomposed into a first unitary matrix V, a second unitary matrix U, and a diagonal matrix Σ of signed singular values, as defined as: where U is an m×m unitary matrix, V is an n×n unitary matrix, and Σ is an n×m diagonal matrix with signed singular values. The superscript "T" denotes the transpose of a matrix. In some embodiments, the weight matrix W is first divided into tiles, and each tile is decomposed into a triple product of such matrices. The weight matrix W may be a conventional weight matrix as known in the field of neural networks.

[0270] In some embodiments, the weighting matrix decomposed in phase space is a pre-specified weighting matrix, such as by using a random initialization procedure or a partially trained weighting matrix. If there is no partially specified weighting matrix to use to initialize the backpropagation routine, the decomposition in act 2-202 may be skipped and instead the parameters of the angular representation (e.g., the singular values ​​and parameters of the unitary or orthogonal decomposition) may be initialized, for example, by random sampling of the phase space parameters from a particular distribution. In other embodiments, a predetermined initial set of singular values ​​and angular parameters may be used.

[0271] In act 2-203, two unitary matrices U and V may be expressed as a composite of a first and second set of unitary transfer matrices, respectively. For example, if the matrices U and V are orthogonal matrices, they may be converted in act 2-203 into a series of real-valued Givens rotation matrices or Householder reflectors, an example of which is described above in Section V.

[0272] In operation 2-204, the photonics-based processor may be configured to implement a unitary transfer matrix based on the decomposed weight matrix. For example, as described above, a first set of components of the photonics-based processor may be configured based on a first set of unitary transfer matrices, a second set of components of the photonics-based processor may be configured based on a diagonal matrix of signed singular values, and a third set of components of the photonics-based processor may be configured based on a second set of unitary transfer matrices. It should be understood that although the process described herein relates to implementing backpropagation techniques using a photonics-based processor, the backpropagation techniques may be implemented using other computational architectures that provide parallel processing capabilities, and the embodiments are not limited in this respect.

[0273] Returning to process 2-100 in FIG. 2-1, operation 2-102 relates to processing the training data to calculate an error vector after the weight matrix has been expressed as an angle representation, for example using a photonics-based processor as described above. FIG. 2-3 shows a flowchart of implementation details for performing operation 2-102 according to some embodiments. Before processing the training data using the techniques described herein, the training data may be divided into batches. The training data may take any form. In some embodiments, the data may be divided into batches in the same manner as is done in some conventional mini-batch stochastic gradient descent (SGD) techniques. The gradients calculated using this procedure can be used for any optimization algorithm, including, but not limited to, SGD, AdaGrad, Adam, RMSProp, or any other gradient-based optimization algorithm.

[0274] In the process shown in FIG. 2-3, each vector in a particular batch of training data may be passed through a photonics processor and a value of a loss function for that vector may be calculated. In operation 2-301, an input training vector from a particular batch of training data is received. In operation 2-302, the input training vector is converted to a photonic signal, for example, by encoding the vector using optical pulses having amplitudes and phases corresponding to the input training vector values ​​as described above. In operation 2-303, the photonic signal corresponding to the input training vector is provided as an input to a photonics processor (e.g., photonics processor 103), which is configured to implement a weighting matrix (e.g., using an array of configurable phase shifters and beam splitters) as described above to generate an output vector of pulses. The optical intensity of the pulses output from the photonics processor may be detected, for example, using homodyne detection as described above in connection with FIG. 9 and FIG. 10 to generate a decoded output vector. In operation 2-304, a value of a loss function (also known as a cost function or error metric) for the input training vector is calculated. The process of acts 2-301 through 2-304 is then repeated until all input training vectors in a particular batch have been processed and corresponding values ​​of the loss function have been identified. In act 2-305, a total loss is calculated by aggregating these losses from each input training vector, which may take the form of an average, for example.

[0275] Returning to process 2-100 of FIG. 2-1, act 2-103 relates to calculating in parallel gradients of parameters of the angle representation (e.g., weight values ​​implemented using components of the photonics processor as Givens rotations). The gradients may be calculated based on the calculated error vector, an input data vector (e.g., from a batch of training data), and a weight matrix implemented using the photonics processor. FIG. 2-4 shows a flowchart of a process for performing act 2-103 according to some embodiments. In act 2-401, the kth set G of Givens rotations in the decomposition is calculated. (k) (e.g., for a column of the decomposed matrix - referred to herein as "derivative column k"), a block diagonal derivative matrix is ​​calculated containing derivatives with respect to each angle in the kth set. In operation 2-402, the products of all unitary transfer matrices between derivative column k and the output are calculated (herein referred to as a "partial backward pass") with the error vector identified in operation 2-102. In operation 2-403, the products of all unitary transfer matrices starting from the input up to and including derivative column k with the input data vector are calculated (herein referred to as a "partial forward pass"). In operation 2-404, dot products between successive pairs of elements output from operations 2-402 and 2-403 are calculated to identify the gradient of derivative column k. The dot products between successive pairs of elements are

[0276]

number

[0277] where the superscript (k) represents the kth column of the photonic element, and i, j are the parameters θ ij (k)where x is the output of the partial forward pass and δ is the output of the partial backward pass. In some embodiments, an offset is applied before successive pairs of outputs (e.g., output pairs could be (1,2), (3,4), etc., rather than (0,1), (2,3). The identified gradients can then be used as appropriate for the particular selected optimization algorithm (e.g., SGD) being used for training.

[0278] Provided below is a pseudocode example for implementing the backpropagation technique in a photonic processor having the left-to-right topology shown in FIGS. 1-4, according to some embodiments.

[0279] ·Initialize two lists x', δ' of intermediate propagation results. ·For each column of MZI, start with the last column and work towards the first column. Rotate the angles in the columns to correspond to the derivative matrix.

[0280] · Propagate the input data vector through the photonics processor. ·Store the result in x'. ·Make the current column transparent. ·Incrementally build the transpose matrix column by column. For each new column, · Propagate the error vector through the photonics processor.

[0281] ·Store the result in δ'. ·For each x'[i], δ'[i], Calculate the dot product between consecutive pairs, the result is the gradient of the angle in the i-th column of MZI. According to some embodiments, rather than adjusting the weights of the weight matrix via gradient descent as is done in some conventional backpropagation techniques, parameters of the angle representation (e.g., the singular values ​​of matrix Σ and the Givens rotation angles of the decomposed orthogonal matrices U and V) are adjusted. To further illustrate how backpropagation in a reparameterized space works according to some embodiments, the following is a comparison of backpropagation within a single layer of a neural network using conventional techniques versus methods according to some embodiments of the present disclosure.

[0282] The loss function E measures the performance of the model on a particular task. In some traditional stochastic gradient descent algorithms, the weight matrix is ​​iteratively adjusted such that the weight matrix at time t+1 is defined as a function of the weight matrix at time t, and the derivative of the loss function with respect to the weights in the weight matrix is

[0283]

number

[0284] where η is the learning rate and (a, b) represent the entries in the a-th row and b-th column of the weight matrix W. If this iterative process is reworked using the decomposed weight matrix, then the weights w ab are the singular values ​​σ of the matrix Σ i and the rotation angle θ of the orthogonal matrices U and V ij Therefore, the iterative adjustment of the backpropagation algorithm is

[0285]

number

[0286] become. To perform iterative adjustments to the singular values ​​and rotation angles, the derivatives of the loss function must be taken. Before describing how this can be accomplished in a system such as photonic processing system 100, we first provide a description of backpropagation based iterative adjustment of the weights in the weight matrix. In this regard, the output results measured by the system for a single layer of a neural network are expressed as an output vector y i =f((Wx) i +b i ), where W is the weight matrix, x is the data vector input to the layer, b is a vector of biases, and f is a nonlinear function. The chain rule of calculus is applied to calculate the gradient of the loss function with respect to any parameter in the weight matrix (here, for convenience of expression, we use the definition z i =(Wx) i +b i is used).

[0287]

number

[0288] w ab Calculating the derivative of z with respect to

[0289]

number

[0290] Using this, the sum representing the gradient of the loss function is

[0291]

number

[0292] The first sum is defined as the error vector e, where x is the input vector, and the following equation:

[0293]

number

[0294] Using the above formula from traditional backpropagation techniques, the formula can be expanded to the case where the weight matrix is ​​decomposed into a singular value matrix and a unitary transfer matrix. Using the fact that the weight matrix is ​​a function of the rotation angle, the chain rule becomes

[0295]

number

[0296] It can be written as: Thus, backpropagation in phase space involves the same components as conventional backpropagation (the error vector and the input data), with the addition of a term that is the derivative of the weight matrix with respect to the rotation angle of the unitary transfer matrix.

[0297] To determine the derivative of the weight matrix with respect to the rotation angle of the unitary transfer matrix, the derivative of one Givens rotation matrix is ​​of the following form:

[0298]

number

[0299] It is noted that As can be seen, the derivative of any entry of the Givens rotation matrix that is not in the ith row and jth column is zero. Therefore, G (k) All derivatives of the interior angles may be grouped into one matrix. Computing the derivatives for all floor(n / 2) angles within a column of a unitary transfer matrix may be accomplished in some embodiments using a two-step process as described above. First, the error vector is propagated through the decomposed matrix from the right (output) to the current set of rotations being differentiated (a partial backward pass). Second, the input vector is propagated from the left (input) to the current set of rotations (a partial forward pass), and then the derivative matrix is ​​applied.

[0300] In some embodiments, the derivatives of the singular values ​​are achieved using a similar process. i The derivative with respect to the element Σ' ii Set to 1 and all other Σ' jj to 0. Thus, all derivatives of the singular values ​​can be computed together. In some embodiments, this can be done by propagating the error vector from the left (a partial forward pass) and the input vector from the right (a partial backward pass), and then computing the Hadamard product from the outputs of the forward and backward passes.

[0301] In the implementation of operation 2-103 described in FIG. 2-4, all angles in column k are rotated by π / 2 to calculate the gradient term for that column. In some embodiments, this rotation is not performed. Consider the matrix of one MZI.

[0302]

number

[0303] Taking the derivative with respect to θ, we get

[0304]

number

[0305] is generated. This matrix corresponds to adding π / 2 to θ, but also to swapping the columns of the original matrix and ignoring one of them. In mathematical notation, this is

[0306]

number

[0307] means... To determine the gradient of a column of the decomposed unitary matrix, instead of rotating the angle of each MZI in the column by π / 2 and then calculating the dot product between successive pairs of elements output from operations 2-402 and 2-403 as described above (e.g., x1δ1+x2δ2), in some embodiments, the angle is not rotated by π / 2, but instead the relationship x1δ2-x2δ1 is calculated to obtain the same gradient. In some embodiments where the size of matrix W(n×m) matches the size of the photonics processor with matrix U of size n×n and matrix V of size m×m, operations 2-401 through 2-404 allow the controller to obtain the floor(n / 2) gradient of the unitary / orthogonal matrix of size n×n. Thus, in hardware such as the photonic processing system 100 described above, where each matrix multiplication can be calculated in O(1) operations, the entire backpropagation procedure can be completed in O(n+m) operations if the photonic processor is of sufficient size to represent the complete matrix. If the photonic processor is not of sufficient size to represent the complete matrix, the matrix can be divided into tiles, as described above. Consider a photonic processor of size N. For the task of multiplying a matrix of size I×J by a vector of size J, one matrix-vector product has complexity O(IJ / N 2 ) (assuming I and J are both divisible by N), because each dimension of the matrix must be split into a matrix of size N, loaded into a processor, and used to compute the partial results. For a batch of K vectors (e.g., a second matrix of size J×K), the complexity of the matrix-vector product is O(IJK / N 2 ).

[0308] The photonic processor embodiments described above having n optical modes naturally compute a matrix-vector product between a matrix of size [n×n] and an n-element vector. This is equivalently expressed as a matrix-matrix product between a matrix of size [n×n] and a matrix of size [n×1]. Furthermore, a series of K matrix-vector product operations with K distinct input vectors and one repeated input matrix can be expressed as computing a matrix-matrix product between matrices of size [n×n] and [n×K]. However, the applications and algorithms described herein often involve computing general matrix-matrix multiplication (GEMM) between matrices of any size, i.e. computing

[0309]

number

[0310] wherein a ij is the element in the i-th row and j-th column of the [I × J] matrix A, and b jk is the jth row and kth column of the [J×K] matrix B, and c ik is the element in the i-th row and k-th column of the [I×K] matrix C=AB. Due to the recursive nature of this calculation, this is

[0311]

number

[0312] where A is first

[0313]

number

[0314] The matrix is ​​padded, then the sub-matrices (each of size [n×n]) are

[0315]

number

[0316] Divided into a grid, A ij is the [n × n] submatrix in the i-th row and j-th column of this grid, and B is first

[0317]

number

[0318] The matrix is ​​padded, then the sub-matrices (each of size [n×K]) are

[0319]

number

[0320] Divided into a grid, B j is the [n × K] submatrix in the j-th row of this grid, and C is first

[0321]

number

[0322] The matrix is ​​padded, then the sub-matrices (each of size [n×K]) are

[0323]

number

[0324] Divided into a grid, C i is the [n × K] submatrix in the i-th row of this grid. Using this process, the photonic processor:

[0325]

number

[0326] Any GEMM can be computed by loading K distinct matrices into the photonic array and, for each loaded matrix, propagating K distinct vectors through the photonic array. This can be done by

[0327]

number

[0328] This produces output vectors (each consisting of n elements), subsets of which may be added together to produce the desired [I x K] output matrix, as defined by the equation above. In the implementation of operation 2-103 described in FIG. 2-4, a left-to-right topology of the photonics processor implementing the matrix was assumed, with the vectors input to the left side of the array of optical components and the output vectors provided to the right side of the array of optical components. In this topology, when propagating the error vector through the photonics processor, the transpose of the angle representation matrix needs to be calculated. In some embodiments, the photonic processor is implemented using a folded topology that places both the input and the output on one side (e.g., the left side) of the array of optical components. In such an architecture, the use of switches can determine which direction the light should propagate - from input to output or from output to input. With on-the-fly configurable direction selection, the propagation of the error vector can be achieved by first switching the direction to use the output as the input and then propagating the error vector through the array, as described above, when the gradient of column k is identified, canceling the phase (e.g., by rotating the angle of each photonic element in column k by π / 2) and eliminating the need to transpose the columns.

[0329] Returning to process 2-100 in FIG. 2-1, operation 2-104 relates to updating the weight matrix by updating the parameters of the angular representation based on the identified gradient. The process described here and shown in FIG. 2-4 is for calculating the gradient of the angular parameters of one set (e.g., column k) of gradients in the decomposed unitary matrix based on one input data example. To update the weight matrix, it is necessary to calculate the gradient of each set (e.g., column) in a batch for each input data vector. FIG. 2-5 shows a flowchart of a process for calculating all gradients required to update the weight matrix according to some embodiments. In operation 2-501, the gradient of one set of multiple sets of unitary transfer matrices (e.g., Givens rotations) is identified (e.g., using the process shown in FIG. 2-4). In operation 2-502, it is then determined whether there are additional sets (e.g., columns) of unitary transfer matrices for which gradients need to be calculated for the current input data vector. If it is determined that there are additional gradients to be calculated, the process returns to operation 2-501, where a new set (e.g., columns) is selected, and the gradients of the newly selected set are calculated. As described below, in some computing architectures, all columns of the array may be read simultaneously such that the determination in operation 2-502 is not necessary. The process continues until it is determined in operation 2-502 that the gradients of all sets (e.g., columns) of the unitary transfer matrix in the decomposed unitary matrix have been identified. The process then proceeds to operation 2-503, where it is determined whether there are any more input data vectors to process in the batch of training data being processed. If it is determined that there are additional input data vectors, the process returns to operation 2-501, where a new input data vector is selected from the batch, and the gradients are calculated based on the newly selected input data vector. The process repeats until it is determined in operation 2-503 that all input data vectors in the batch of training data have been processed. In act 2-504, the gradients identified in acts 2-501 through 2-503 are averaged, and parameters of the angle representation (e.g., angles of the Givens rotation matrix of the decomposition of the weight matrix) are updated based on an update rule that utilizes the averaged gradients.As non-limiting examples, the update rules may include scaling the average gradient by a learning rate, or may include "momentum" or other corrections to the history of the parameter gradients.

[0330] As briefly mentioned above, the above example was applied to a real weight matrix in a single-layer neural network, but the results can be generalized to networks with multiple layers and complex weight matrices. In some embodiments, the neural network consists of multiple layers (e.g., 50 layers or more in a deep neural network). To calculate the gradient of a matrix in a layer L, the input vector to that layer is the output of the previous layer L-1, and the error vector to that layer is the backpropagated error from the following layer L+1. The value of the backpropagated error vector can be calculated using the chain rule of traditional multivariate calculus. Furthermore, in some embodiments, complex U and V matrices (e.g., unitary matrices) can be used by adding an additional complex phase term to the Givens rotation matrix.

[0331] While the above description generally applies regardless of hardware architecture, certain hardware architectures provide greater computational acceleration than others. In particular, implementation of the backpropagation techniques described herein on graphical processing units, systolic matrix multipliers, photonic processors (e.g., photonic processing system 100) or other hardware architectures capable of parallel computation of gradients is preferred for maximum gains compared to conventional approaches.

[0332] As mentioned above, the photonic processing system 100 is configured to perform any unitary transformation. A series of Givens rotations is one example of such a unitary transformation, and therefore the photonic processing system 100 can be programmed to compute the transformations in the above decomposition in O(1) time. As mentioned above, the matrices can be implemented by controlling a regular array of variable beam splitters (VBS). The unitary matrices U and V Tcan be decomposed into a tiled array of VBSs, where each tiled array of VBSs performs a 2×2 orthogonal matrix operation (e.g., a Givens rotation). The diagonal matrix Σ is a diagonal phase screen D U and D. V (Diagonal matrix D U ΣD V form) and can be implemented by photonic processing system 100 by controlling the intensity and phase of the optical pulses as described above.

[0333] Each entry in the diagonal matrix Σ corresponds to amplification or attenuation of each photonic mode. Entries with a magnitude greater than or equal to 1 correspond to amplification, and entries with a magnitude less than or equal to 1 correspond to attenuation, with the combination of the VBS and the gain medium allowing either attenuation or amplification. For an n×n square matrix M, the number of optical modes required for application of the diagonal matrix Σ is n. However, if the matrix M is not square, the number of optical modes required is equal to the smaller dimension.

[0334] As mentioned above, in some embodiments, the size of the photonic processor is the same as the size of the matrix M and the input vector being multiplied. In practice, however, the size of the matrix M and the size of the photonic processor are often different. Consider a photonic processor of size N. For the task of multiplying a matrix of size I×J by a vector of size J, one matrix-vector product can be multiplied with complexity O(IJ / N 2 ) (assuming I and J are both divisible by N), because each dimension of the matrix must be split into a matrix of size N, loaded into a processor, and used to compute the partial results. For a batch of K vectors (e.g., a second matrix of size J×K), the complexity of the matrix-vector product is O(IJK / N 2 ).

[0335] The ability to work with small N×N matrix partitions can be advantageous when the matrix is ​​non-square, especially when either I>>J or J>>I. Given a non-square matrix A, direct SVD of the matrix produces one I×I unitary matrix, one J×J unitary matrix, and one I×J diagonal matrix. When either I>>J or J>>I, the number of parameters required to represent this decomposed matrix is ​​much larger than the original matrix A.

[0336] However, if matrix A is decomposed into multiple N×N square matrices with smaller dimensions, SVD on these N×N matrices produces two N×N unitary matrices and one N×N diagonal matrix. In this case, the number of parameters required to represent the decomposed matrix is ​​still N 2 (equal to the size of the original matrix A), and the total non-square matrix can be decomposed using a number of parameters that total approximates IJ. 2 If it is divisible by , then the approximation reduces to an equality.

[0337] For a photonic processor with 2N+1 columns, partial results can be calculated by backpropagating the error vector for each column. Thus, for a batch of K vectors, the complexity of backpropagation using a photonic processor of size N is O(IJK / N). By comparison, computing the backpropagated error using a matrix multiplication algorithm on a non-parallel processor (e.g., a CPU) is O(IJK).

[0338] The above description has focused on the use of matrices in neural network layers with input vector data and back-propagated error vectors. The inventors have recognized and understood that data in deep neural network computations are not necessarily vectors, but are generally multi-dimensional tensors. Similarly, weight values ​​describing connections between neurons are generally multi-dimensional tensors. In some embodiments, the above-described method can be directly applied when the weight tensors are sliced ​​into matrix slices, each independent of the other. Thus, singular value decomposition and Givens-like rotation decomposition can be performed to obtain a valid representation for the phase of a particular matrix slice. The same method of calculating the gradient of the phase can then be applied, using the appropriately positioned input tensor data and back-propagated error data as well. The gradient of a particular matrix slice should be calculated using the portion of the input data and error data that contributed to that particular matrix slice.

[0339] For concreteness, we use a general n-dimensional weight tensor W a1a2a3…ai…an Consider. i} i=1, ... ,n Select two indices that make up the matrix slice from the b and a c ), ∂E / ∂θ ij (k) Perform the decomposition by calculating the phase θ ij (k) Importantly, b and c can be any value between 1 and n. Here, we obtain a1…ai…an For a valid tensor operation, it must be true that the output of the operation of the weight tensor on this input tensor produces an (nk)-dimensional output tensor. Therefore, we assume that the backpropagated error tensor of the weight tensor of this layer is an (nk)-dimensional tensor: e ak+1…ai…an Therefore, the gradient to be calculated is

[0340]

number

[0341] where, for simplicity (but not a requirement), the indices of the weight tensor are ordered such that the first k indices operate on x, and the last (nk) indices operate on the error e.

[0342] In other embodiments, it may be more convenient to perform a higher order generalization of the singular value decomposition, such as the Tucker decomposition, where any n-dimensional tensor can be expressed as

[0343]

number

[0344] where each U aibi (i) is an orthogonal matrix that can be decomposed into a Givens rotation phase, and g b1…bn is an n-dimensional core tensor. In some cases, the core tensor can be chosen to be superdiagonal, using a special case of Tucker decomposition called the CANDECOMP-PARAFAC (CP) decomposition. The Tucker decomposition can be made similar to the two-dimensional SVD form by multiplying it by several inversions (transposes or conjugate transposes) of unitary matrices. For example, the decomposition is

[0345]

number

[0346] where the first m unitary matrices are pushed to the left of the core tensor. The set of unitary matrices on either side of the core tensor can be decomposed into rotations, and the gradient of each rotation is obtained by calculus' chain rule and contraction of the gradient with the input and error tensors.

[0347] The inventors have recognized and appreciated that the gradient of the phase (e.g., of the decomposed matrices U and V) and the gradient of the signed singular values ​​(e.g., of the matrix Σ) may have different upper bounds. Consider the task of computing the gradient of a scalar loss function L with respect to the neural network parameters. In Euclidean space, the value of the gradient is

[0348]

number

[0349] where W is a matrix. In a topological space, a particular scalar topology θ k For, the chain rule is

[0350]

number

[0351] to provide. By the definition of trace, this means that

[0352]

number

[0353] where:

[0354]

number

[0355] are both matrices. The trace is the Frobenius norm product Tr(AB)≦││A││ F ││AB││ F Separated by ││A││ F =││A T ││ F It is known that:

[0356]

number

[0357] It is. Since differentiation with respect to θ does not change the singular values ​​of W and therefore does not change the Frobenius norm, the following holds:

[0358]

number

[0359] A specific singular value σ k Differentiating with respect to

[0360]

number

[0361] All singular values ​​go to zero except for the singular values ​​in the derivative, which go to 1, meaning therefore,

[0362]

number

[0363] It is. In some embodiments, the phase and singular value gradients are scaled separately during the update of the parameters of the angle representation, for example to account for differences in upper limits. By scaling the gradients separately, either the phase gradient or the singular value gradient can be rescaled to have the same upper limit. In some embodiments, the phase gradient is scaled by the Frobenius norm of the matrix. According to some update rules, scaling the phase and singular value gradients is equivalent to having different learning rates for the phase gradient and the singular value gradient, independently. Thus, in some embodiments, a first learning rate for updating the elements of the set of U and V matrices is different from a second learning rate for updating the elements of the set of Σ matrices.

[0364] The inventors have recognized and understood that when the weight matrix is ​​decomposed into a phase space, it may be that both the phase and the singular values ​​do not need to be updated at every iteration to obtain a good solution. Thus, if only the singular values ​​(and not the phase) are updated during an epoch for some portion of the overall training time, then the training time can be O(n 2 ), only O(n) parameter updates are required, leading to improved overall execution time. Updating only the singular values ​​or only the phase during some iterations may be referred to as "parameter clamping." In some embodiments, parameter clamping may be performed according to one or more of the following clamping techniques:

[0365] Fixed Clamping: Train all parameters for a certain number of iterations and subsequently update only the singular values. · Circular clamping: Train all parameters for some epochs M, then freeze the phase (i.e. update only the singular values) for some epochs N. Resume training all parameters for another M epochs, then freeze the phase again for N epochs. Repeat until the total number of desired epochs is reached.

[0366] Warm-up clamping: Train all parameters for some number of epochs K, then start cyclic clamping for the remaining number of epochs. · Threshold clamping: keep updating the phase or singular values ​​until the updates are smaller than a threshold ε.

[0367] The inventors have recognized and understood that the architecture of the photonic processor may affect the computational complexity. For example, in the architecture shown in FIG. 1, the detector is at only one end of the photonic processor array, resulting in a column-by-column approach to calculating the gradients (e.g., using partial forward and backward passes) as described above. In an alternative architecture, the detector may be located at every photonic element in the photonic array (e.g., at each MZI). In such an architecture, the column-by-column approach may be replaced with one forward pass and one backward pass, and the output at every column is read out simultaneously, thus providing additional computational acceleration. Intermediate solutions are also contemplated when columns of detectors are placed intermittently throughout the array. Any addition of a detector column accordingly reduces the number of partial forward and backward passes required for the gradient calculation.

[0368] The techniques described above illustrate techniques for performing weight matrix parameter updates while keeping all calculations in phase space (e.g., using an angular representation of the matrices). In some embodiments, at least some of the calculations may be performed in Euclidean vector space, while other calculations are performed in phase space. For example, quantities required to perform the updates may be calculated in phase space as described above, but the actual updates of the parameters may occur in Euclidean vector space. The updated matrices calculated in Euclidean vector space may then be re-factored into weight space for the next iteration. In Euclidean vector space, at a given layer, the update rule is:

[0369]

number

[0370] It could be. In this calculation, δ can be calculated using a backward pass through the entire photonic processor in phase space. The above cross product between x and δ can then be calculated separately (e.g., off-chip). Once the updates are applied, the updated matrix can be re-factored and the factorized values ​​can be used to set the phase of the photonic processor as described above.

[0371] III. Convolutional Layer Convolution and cross-correlation are common signal processing operations in many applications such as audio / video coding, probability theory, image processing, and machine learning. The terms convolution and cross-correlation generally refer to mathematical operations that accept two signals as inputs and generate as output a third signal that represents the similarity that exists between the inputs. The inventors have recognized and understood that computing convolution and cross-correlation can be computationally resource intensive. In particular, the inventors have developed techniques that improve the speed and efficiency of computing convolution and cross-correlation. Embodiments of these techniques include computing the convolution and cross-correlation by converting the convolution operation into a matrix-vector product and / or a product of a multidimensional array. Embodiments of these techniques further include computing the convolution according to a discrete transformation.

[0372] The inventors have further recognized and appreciated that the convolution and cross-correlation calculations may be performed in a variety of ways depending on the intended application. The input and output signals may be discrete or continuous. The data values ​​that make up the signals may be defined over a variety of numerical domains, such as the real numbers, the complex plane, or the ring of finite integers. The signals may have any number of dimensions. The signals may also have multiple channels, a technique commonly used in convolutional neural networks (CNNs). The embodiments described herein may be implemented to accommodate these variations in any combination.

[0373] Furthermore, embodiments of these techniques may be implemented in any suitable computing system configured to perform matrix operations. Examples of such computing systems that may benefit from the techniques described herein include central processing units (CPUs), graphic processing units (GPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and photonic processors. Although the embodiments described herein may be described in the context of photonic processors, it should be understood that these techniques are applicable to other computing systems, such as those mentioned above, without limitation.

[0374] The following is a more detailed description of various concepts related to the calculation of convolution and cross-correlation, as well as embodiments and techniques thereof. It should be understood that the various aspects described herein can be implemented in any number of ways. Examples of specific implementations are provided herein for illustrative purposes only. In addition, the various aspects described in the following embodiments can be used alone or in any combination, and are not limited to the combinations expressly described herein.

[0375] The inventors have further realized and understood that using optical signals instead of electrical signals eliminates many of the problems described above with electrical computing. Optical signals travel at the speed of light in the medium in which light travels, and therefore the latency of the photonic signal is not a significant limitation compared to the electrical propagation delay. Furthermore, the increased distance that the optical signal travels does not dissipate power, opening up new topologies and processor layouts that would not have been feasible using electrical signals. Thus, optical-based processors such as photonic processors may have better speed and efficiency performance than some traditional electrical-based processors.

[0376] To implement a photonics-based processor, the inventors have recognized and understood that multiplication of an input vector by a matrix may be accomplished by propagating a coherent optical signal, e.g., a laser pulse, through a first array of interconnected variable beam splitters (VBS), a second array of interconnected variable beam splitters, and a plurality of controllable electro-optical elements between the two arrays, connecting one output of the first array to one input of the second array.

[0377] As described herein, some embodiments of a photonic processor having n optical modes naturally compute a matrix-vector product between a matrix of size [n×n] and an n-element vector. This may be equivalently expressed as a matrix-matrix product between a matrix of size [n×n] and a matrix of size [n×1]. A series of K matrix-vector product operations with K distinct input vectors and one iterative input matrix may be expressed as a matrix-matrix product computation between a matrix of size [n×n] and a matrix of size [n×K]. However, the applications and algorithms described herein often involve the computation of general matrix-matrix multiplication (GEMM) between matrices of any size, i.e., the computation

[0378]

number

[0379] wherein a ij is the element in the i-th row and j-th column of the [I × J] matrix A, and b jk is the jth row and kth column of the [J×K] matrix B, and c ik is the element in the i-th row and k-th column of the [I×K] matrix C=AB. Due to the recursive nature of this calculation, this is

[0380]

number

[0381] where A is first

[0382]

number

[0383] The matrix is ​​padded, then the sub-matrices (each of size [n×n]) are

[0384]

number

[0385] Divided into a grid, A ij is the [n × n] submatrix in the i-th row and j-th column of this grid, and B is first

[0386]

number

[0387] The matrix is ​​padded, then the sub-matrices (each of size [n×K]) are

[0388]

number

[0389] Divided into a grid, B j is the [n × K] submatrix in the j-th row of this grid, and C is first

[0390]

number

[0391] The matrix is ​​padded, then the sub-matrices (each of size [n×K]) are

[0392]

number

[0393] Divided into a grid, C i is the [n × K] submatrix in the i-th row of this grid. According to some embodiments, using this process, the photonic processor:

[0394]

number

[0395] Any GEMM can be computed by loading K distinct matrices into the photonic array and, for each loaded matrix, propagating K distinct vectors through the photonic array. This can be done by

[0396]

number

[0397] This produces output vectors (each consisting of n elements), subsets of which may be added together to produce the desired [I x K] output matrix, as defined by the equation above. The inventors have recognized and appreciated that, although a photonic processor may accelerate the process of computing the convolution and cross-correlation, the convolution and cross-correlation computing embodiments described herein may be implemented in any suitable computing system. The embodiments described herein are discussed in terms of two-dimensional convolution, but may be generalized to any number of dimensions. h ×I w ] input (referred to herein as an “image”, but it should be understood that the input may represent any suitable data) G and [K h ×K w ]For a filter F, the formula for two-dimensional convolution is

[0398]

number

[0399] It is. The two-dimensional cross-correlation is

[0400]

number

[0401] where:

[0402]

number

[0403] is a function of G determined by the boundary conditions,

[0404]

number

[0405] denotes complex conjugation and · denotes scalar multiplication. In some implementations, the convolution and cross-correlation operations may be interchangeable, since the cross-correlation of complex-valued two-dimensional signals G and F is

[0406]

number

[0407] This is because it can be converted to a convolution via Although the embodiments described herein focus on the convolution case, it should be understood that the embodiments described herein may be used for convolution and cross-correlation calculations.

[0408] In both convolution and cross-correlation, there are different variations depending on how the boundary conditions are handled. Two boundary conditions described in some embodiments herein are:

[0409]

number

[0410] and padding:

[0411]

number

[0412] where a%n denotes a modulo n. According to some embodiments, additional boundary condition variables may be used. These boundary conditions include symmetric (also called mirror or reflection) boundary conditions, where the image is reflected across the boundary. Padding boundary conditions may be variously called linear or filling, in some embodiments. Circular boundary conditions are also known as wrapped.

[0413] Additionally, different output modes may be utilized to determine which elements interact with boundary conditions. These output modes include valid output mode, same output mode (or half-padding), and full output mode. In valid output mode, the output must consist of only elements that do not depend on boundary conditions. In same output mode, the output must be the same size as the input. In full output mode, the output consists of all elements that do not depend exclusively on boundary conditions.

[0414] Different output modes control the number of points [x,y] for which the output is defined. Thus, each embodiment described herein can be modified to operate with any given output mode. While the embodiments described herein focus on the case of same-modal convolution, it should be understood that these implementations can be extended to calculate cross-correlations and / or alternative output modes as an alternative or in addition to the embodiments described herein.

[0415] In some implementations, such as CNN, these operations may be applied to and / or generalized to generate multi-channel data. As an example, an RGB image has three color channels. For an input G with two spatial dimensions and a C channel, the multi-channel operations are

[0416]

number

[0417] is defined as:

[0418]

number

[0419] represents either convolution or cross-correlation, M is the number of output channels, and G is the three-dimensional [C×I h ×I w ] tensor, and F is a four-dimensional [M×C×I h ×I w ] tensor,

[0420]

number

[0421] is the three-dimensional [M×I h ×I w ] tensor. In the above case, slice index notation is used and the spatial dimension is F[m,c] is the second dimension of F [K h ×K w ] space slice, G[c] is the two-dimensional [I h ×I w ] is constrained to access the spatial slice.

[0422] In general, techniques for expressing a convolution as a matrix operation may follow the process of FIG. 3-1. In operation 3-102, pre-processing of the image and / or filter matrices may be performed prior to the matrix operation, e.g., to ensure that the matrices are of the correct dimensions and comply with boundary conditions and / or output modes. In operation 3-104, a core matrix or matrix-vector operation may be applied to create the output of the convolution. In operation 3-106, post-processing of the output of the convolution may be performed, e.g., to reshape the output, as discussed in more detail herein.

[0423] Some embodiments may use a photonic processor to compute the convolution as a matrix-vector product. The inventors have recognized and understood that an array of variable beam splitters (VBS), such as may be included in some embodiments of a photonic processor as described herein above, may be used to represent any unitary matrix. As an example, these techniques may be used to compute the augmented image G mat If we express mat =V T It can be decomposed using singular value decomposition as ΣU.

[0424] In some embodiments of the photonic processor, the two unitary matrices U and V can then be decomposed using the algorithm described above. The phase generated from that calculation is programmed into the photonic array along with the singular values. In some embodiments, the processor decomposes the filter rather than the image, so the filter can remain loaded for an entire batch of images.

[0425] An example of a process for computing convolutions in a photonic processor according to some embodiments is shown in FIG. 3-2. In operation 3-202, the process derives a matrix G from an input image matrix G. mat Construct the matrix G mat is not limited to being a doubly block circulant matrix or a doubly block Toeplitz matrix, mat may be constructed in any suitable manner according to the selected boundary conditions, including by constructing

[0426] Next, in operation 3-204, the decomposed matrix G mat may be loaded into the photonic array. For each filter F in the input batch, the loop is repeated and in operation 3-206, the filter F is flattened into a column vector, in operation 3-208 runs a matrix-multiplication operation through the photonic array, and then in operation 3-210, is reshaped into an output with the appropriate dimensions. In operation 3-212, it is determined whether any further filters F remain. If there are further filters F to pass through the convolutional layer, the process returns to operation 3-206. Otherwise, the process ends. Due to the commutative nature of convolutions, the process 3-200 determines whether filter F is a column vector that is flattened into a column vector in operation 3-206, runs a matrix-multiplication operation through the photonic array in operation 3-208, and then is reshaped into an output with the appropriate dimensions in operation 3-210. mat and the image G is flattened into a column vector and run through the photonic array in operation 3-208.

[0427] The photonic processor may be used to implement any suitable matrix multiplication based algorithm. A matrix multiplication based algorithm reorders and / or expands the input signal so that the calculation can be expressed as a general matrix-matrix multiplication (GEMM) with some pre- and / or post-processing. Some examples of matrix multiplication based algorithms that may be implemented in a photonic processor include im2col, kn2row, and memory-efficient convolution (MEC).

[0428] According to some embodiments, the im2col algorithm may be implemented in a photonic processor. In the im2col algorithm, during pre-processing, an image G is divided into [I h ×I w ] matrix to [(K h ·K w )×(I h I w )] matrix. The filter F can be expanded to [K h ×K w ] matrix to [1×(K h ·K w) row vector. The output can then be generated by a matrix-vector product of the image and the filter, because this preprocessing step ensures that each column can be scaled and all (K h ×K w ) copies of the elements. Therefore, the im2col algorithm takes O(K h ·K w I h I w ) data copy and O(K h ·K w I h I w ) Temporary storage may be required.

[0429] According to some embodiments, the kn2row algorithm may be implemented in a photonic processor. The kn2row algorithm calculates the cross product of the unmodified image and the filtered signal, and returns a cross product of size [(K h ·K w )×(I h I w The kn2row algorithm then adds together certain elements from each row of the cross product to produce a temporary matrix of [1 × (I h I w )] output vector. Therefore, the kn2row algorithm also operates in O(K h ·K w I h I w ) data copy and O(K h ·K w I h I w ) Temporary storage may be required.

[0430] According to some embodiments, the MEC algorithm may be implemented in a photonic processor. The MEC algorithm uses the same algorithm as in the im2col algorithm (K h ·K w ) coefficient, not K h Coefficient or K wThe input image may be expanded by coefficients only. If a smaller filter dimension is chosen for the expansion, the algorithm runs in O(min(K h ,K w )·I h I w ) requires temporary storage and data copies. Unlike im2col or kn2row, which compute one matrix-vector product, the MEC algorithm computes a series of smaller matrix-vector products and concatenates the results.

[0431] In the above-described embodiment, the filter matrix may be expanded during pre-processing rather than the image due to the commutative nature of convolution. The choice of whether it is the image or the filter that is tiled and reshaped into a matrix may depend on which operation is faster and / or requires less computational energy.

[0432] VII. Multidimensional Convolution via 2D Matrix-Matrix Multiplication The inventors have recognized and understood that the matrix multiplication-based algorithms for computing convolutions described above may not be suitable for some computing architectures or applications. The inventors have further recognized and understood that an approach that can combine the computational efficiency of im2col or kn2row with the memory-saving features of the MEC algorithm would be beneficial for computing convolutions and cross-correlation. In particular, the inventors have recognized that these benefits may be achieved by splitting the reordering and reshaping of input and output matrices into pre-processing and post-processing steps, and that such a method may be generalized to N-dimensional convolutions, where N≧2.

[0433] According to some embodiments, multidimensional convolution via a two-dimensional matrix-matrix multiplication algorithm (herein the "cng2" algorithm) involves three steps. At a high level, in a non-limiting example of a two-dimensional circular convolution, the pre-processing step is to calculate [I h ×I w ] by duplicating and rotating the rows of the input matrix [K w ×(I h I w)] matrix, and in some implementations, "rotation" refers to a circular permutation of the elements of a vector, for example rotating ([1,2,3,4],-1) to [2,3,4,1]. In the GEMM step, [K h ×K w ] filter matrix and [K w ×(I h I w )] matrix is ​​calculated. In post-processing, the [K h ×(I h I w )] matrix rows are rotated and added to construct the output.

[0434] According to some embodiments, the cng2 algorithm may be modified to implement other boundary conditions. As an example, during pre- and post-processing, in the case of padding convolutions, vector rows are shifted rather than rotated. That is, during the rotation step, elements that would normally wrap around the row vector are set to zero. Other boundary conditions that may be implemented in the cng2 algorithm include, but are not limited to, symmetric or mirror boundary conditions.

[0435] It may further be noted that the pre-processing step of the cng2 algorithm is not limited to application only to the left-hand side input (here, the image), but rather, according to some embodiments, can be applied to the right-hand side input (here, the filter). In the case of full or valid mode convolution, the operations are commutative, and the pre-processing phase can be applied to either input. In the case of homomodal convolution, I h ≠K h or I w ·K w When , the operation is not commutative and a preprocessing phase can still be applied to the right-hand side, but the filter must first be zero-padded and / or cropped in each dimension to match the output size.

[0436] In some implementations, the cng2 algorithm may include additional steps as described in FIG. 3-3A. Before the pre-processing stage as described above, one or more input filter matrices may need to be reshaped into a filter matrix F having appropriate dimensions, as shown in operation 3-302. This reshaping may be done by concatenating one or more input filter matrices in any suitable manner. In operation 3-304, a pre-processing step is performed to build a circulant matrix H, which will be described in more detail with reference to FIG. 3-3B. In operation 3-306, a GEMM step is performed to create an intermediate matrix X=F×H. Then, a post-processing step may be performed. In operation 3-308, the vector rows of matrix X are rotated and / or shifted to form matrix X′. In operation 3-310, vector row addition is performed on the rows of matrix X′ to form matrix Z. In operation 3-312, depending on the memory layout of the particular processing system, matrix Z may be reshaped into at least one output matrix.

[0437] The manner in which matrix H is constructed may depend on the desired boundary conditions, according to some embodiments, as shown in the extension of act 3-304 in FIG. 3-3B. In act 3-314, it may be determined whether the boundary condition is circular. If it is determined that the boundary condition is circular, the processing system may proceed to act 3-316, where matrix H is created by duplicating and rotating at least one row of the input matrix. Alternatively, if it is determined in act 3-314 that the boundary condition is padding rather than circular, the processing system may proceed to act 3-318, where matrix H is created by duplicating and shifting at least one row of the input matrix as described above. It should be understood that in some embodiments, other boundary conditions than padding boundary conditions may be utilized in act 3-318.

[0438] Alternatively, according to some embodiments, when calculating the cross-correlation, the problem may not need to be explicitly transformed into a convolution as in process 3-300. Instead, the element inversion step 3-302 may be omitted and the pre-processing and post-processing steps of the cng2 algorithm may be modified accordingly. That is, the element inversion step may be combined with the pre-processing and post-processing steps of the cng2 algorithm. How this is done depends on whether the pre-processing extension is applied to the left-hand side input or the right-hand side input. If the left-hand side input is extended, the shifts or rotations in both the pre-processing and post-processing steps may be performed in the opposite direction. If the right-hand side input is extended, each circulant matrix generated during the pre-processing phase may be transposed and concatenated in reverse order, and the i-th row of the GEMM output matrix may be shifted or rotated by (i-n+1)·n elements in the post-processing phase instead of i·n elements. In the case of complex-valued data, the cross-correlation still requires the complex conjugation of one input.

[0439] In some implementations, such as in CNNs, it may be desirable to generalize the above operations so that they can be applied to and / or generated on multi-channel data. For a problem with C input channels and M output channels, the filter matrix has the form [(M K h )×(K w ·C)] and the input matrix has the form [(K w ·C)×(I h I w )] and the output matrix has the form [(M K h )×(I h I w )] is taken.

[0440] With reference to Fig. 3-4A to Fig. 3-4F, an example of a multi-channel input and output process 3-300 according to some embodiments is shown. For a [2x2] filter f with four output channels and a [3x3] image G with three input channels, operation 3-402 is visualized in Fig. 3-4A. However, any size of filter matrix, image matrix and / or any number of input and / or output channels can be implemented. In the example of operation 3-402, a reshaping of filter f into a [6x8] filter matrix F is performed. In this example, the reshaping of filter f is performed by concatenating filter f without changing the order of the matrix elements, although other methods such as rotating, shifting or otherwise modifying the rows of filter f can also be used. The reshaping of filter f ensures that filter matrix F is of the appropriate dimensions for the subsequent GEMM operation. However, in some implementations where memory is appropriately laid out, operation 3-402 may not be necessary before operation 3-404, as described below.

[0441] According to some embodiments, as shown in FIG. 3-4B, after operation 3-402, pre-processing of image G may be performed in operation 3-404. In this example, image G is formed to perform a homomorphic convolution, although any output mode may be used. In this example, a circulant matrix H is formed based on a circulant boundary condition, although any boundary condition may be used, such as a padding boundary condition. After operation 3-404, a GEMM operation F×H=X may be performed in operation 3-406, as shown in FIG. 3-4C. In this example, intermediate matrix X has dimensions [9×8].

[0442] After the GEMM operation, a post-processing step may be performed, as shown in Figures 3-4D to 3-4E. In Figure 3-4D, operation 3-408 is shown, in which the rows of the intermediate matrix X are rotated according to the circular boundary condition in this example to form a matrix X'. Other boundary conditions, such as padding boundary conditions, as a non-limiting example, may be implemented in the pre-processing and post-processing, as long as the boundary conditions in the pre-processing and post-processing steps are identical to each other.

[0443] Referring to FIG. 3-4E where operation 3-410 is shown, the final step of post-processing is to add the rows of matrix X′ to form the output matrix Z. That is, in this example, 00 +x 16 =z 00 , x 01 +x 17 =z 01 etc. In some implementations, depending on how the memory of the processing system is laid out, after operation 3-410, a reshaping of the output matrix Z may need to be performed in operation 3-412. In the example visualization of FIG. 3-4F, the output matrix Z is reshaped into four output matrices A of dimensions [3×3].

[0444] In addition to being generalizable to multiple input channels, according to some embodiments, the cng2 algorithm can also be generalized to higher dimensional signals (i.e., greater than 2). n ×K n-1 ×...×K1] filter tensor with size [I n ×I n-1 For n-dimensional convolution between an image of [K × ... × I1], it is possible to compute the desired output using two-dimensional matrix multiplication using steps similar to those performed for two-dimensional signals. During preprocessing, the input tensor is multiplied by (K a ·K a-1 .......K1) factor, where a can be thought of as the number of dimensions handled during the preprocessing phase, and any value in the range 1≦a≦n-1 can be chosen. In the GEMM step, n ·K n-1 .....K a+1 )×(K a ·K a-1 A multiplication of the partitioned filter tensor as the (K1) matrix and the augmented matrix from the pre-processing step may be performed. During the post-processing step, sub-vectors of the matrix generated during GEMM may be rotated and accumulated.

[0445] The augmented matrix generated by the preprocessing phase is (I i I n-1 .....I a+1) horizontally concatenated sub-matrices, each of which is a nested Toeplitz matrix of order a, where the innermost Toeplitz matrix is ​​defined as in the two-dimensional cng2 implementation. The post-processing phase may perform (na) rounds of rotations and additions, where the i-th round multiplies the matrix produced by the previous round (or by the initial GEMM operation) by a factor of size [K a+i ×(I a+i I a+i-1 .....I1)]. For each submatrix, the following operations are then performed. First, the jth row is (j (I a+i-1 I a+i-2 .....I1)) elements. Then all the rows can be added together.

[0446] Although the above description treats the dimensions in order, i.e. the pre-processing phase expands the data along the first a dimensions and the post-processing phase reduces the data along the last na dimensions, according to some embodiments this need not be the case: the pre-processing phase can expand the data along any a dimensions by permuting the input and output data in the same way as described for the two-dimensional case.

[0447] The cng2 algorithm provides a flexible framework for computing convolutions, and several alternative embodiments are described herein. In some implementations, the overlap region of the input signals at a given point in the output may be shifted by a constant offset. Such an offset may be applied regardless of the output mode, but is most often paired with same-mode outputs. For convolutions (cross-correlation) operating in same mode and with the definitions given above, the boundary conditions are (K h -1)·I w along the left (right) edge of the element and the input image G (K w -1)·I hThis behavior can be modified by redefining the operation with a constant offset between the filter location and the output location. When computing the convolution (cross-correlation), this modification involves subtracting (adding) the offset from (to) the shift or rotation amount in the pre-processing phase, and applying the offset I w can be applied to cng2 by subtracting (adding) from the shift or rotation amount.

[0448] Furthermore, the proposed methods for reducing both the time and storage requirements of the kn2row post-processing step may be applied to the cng2 algorithm as well, according to some embodiments. In the kn2row algorithm, the GEMM operation is performed by dividing a set of K w ·K h The result of the smaller GEMM operations can be accumulated together in succession. This allows the post-processing addition to be optimized with respect to the storage of the final output and performed in-place. In the case of kn2row, this only works if the boundary conditions can be neglected or if an additional (and generally inefficient) so-called hole punching process is introduced. However, in the cng2 algorithm, this process can be applied directly without precision or additional processing, effectively eliminating the computational cost of the post-processing step and reducing the temporary storage required for the cng2 algorithm to O(K w I h I w )

[0449] In some embodiments, the spatial dimensions can be treated in reverse order as described in process 3-300. The cng2 algorithm can be enhanced with a transpose operation applied to both input signals at the start of process 3-300 and a transpose operation on the final output. This still produces the desired result, but it is possible to reduce the size of the filter if the shape is strictly rectangular (i.e., K w ≠K h ), the behavior changes. In this case, the input image is w Instead, the coefficient K hand the post-processing step is O(K h I h I w ) but O(K w I h I w ) additions. An implementation that combines this variant with the low-memory-intensive post-processing variant above reduces the temporary storage required for the cng2 algorithm to O(min(K h ·K w )·I h I w ) can be further reduced.

[0450] As an alternative implementation, the rows and / or columns in the matrices passed to the GEMM operation may be permuted. If the GEMM operation is defined as C=AB, then the rows and / or columns of either the input matrices A or B may be permuted as long as the appropriate permutation is applied to the other input matrix (in the case of permuting the columns of A or the rows of B) or the output matrix (in the case of permuting the rows of A or the columns of B). In particular, permuting the rows of A in the case of multiple output channels may recognize the data-level parallelism available in the post-processing phase, making it well suited for vector processors or single-instruction-multiple-data (SIMD) architectures.

[0451] The convolution calculations may also be calculated using strides according to some embodiments. x and the stride S in the second dimension y For , the convolution operation is defined as follows:

[0452]

number

[0453] This definition is based on the count S x ·S yThis reduces the size of the output signal by , which is equivalent to increasing the step size by which the filter slides across the image at each output point. This can be done by computing the non-strided convolution and then downsampling the result by an appropriate amount in each dimension, but this requires additional O(S x ·S y ) computational steps. At a minimum, this computational penalty is x We modify the preprocessing phase 3-304 to generate only every i·I w Instead, i (I w / S x By modifying the post-processing step to shift or rotate each row by , we can reduce the time to O(S y ) In some implementations, the computational penalty can be eliminated entirely with additional modifications to each phase. First, the pre-processing step 3-304 is performed using S y The core processing phase can then be modified to generate an expanded matrix, where the i-th circulant matrix is ​​assigned to the j-th expanded matrix, for j=i modulo S. y GEMM operations must be performed (one GEMM operation per augmented input matrix). Each operation is a multiplication of K w / S y Post-processing steps 3-308, 3-310 then interleave the rows of the generated matrix, S y Add each group of rows directly (i.e., without shifting or rotating the rows) and subtract the shift or rotation amount i (I w / S x ) to standard post-processing logic w / S y The line can be passed.

[0454] Alternatively, according to some embodiments, the convolution may be stretched. x and the stretch in the second dimension D y In this case, the convolution operation is

[0455]

number

[0456] It is defined as: Stretching expands each field of the filter over a larger patch of the image at each output point and can be thought of as inserting space between the elements of the filter. The cng2 algorithm uses D x and D. y The method may be modified to perform the stretch convolution by increasing the rotation or shift amount by . The stretch convolution may be further restricted to computation using the causal output mode.

[0457] The inventors have further recognized and appreciated that convolution and cross-correlation can be computed using transform-based algorithms. Transform-based algorithms change the nature of the computational problem by first computing an equivalent representation of the input signal in an alternative numerical domain (e.g., the frequency domain), performing an alternative linear operation (e.g., element-wise multiplication), and then computing an inverse transform of the result back to the original numerical domain (e.g., the time domain) of the signal. Examples of such transforms include finite impulse response (FIR) transforms such as the discrete Fourier transform (DFT), discrete sine transform, discrete cosine transform, discrete Hartley transform, non-decimated discrete wavelet transform, Walsh-Hadamard transform, Hankel transform, and Winograd minimum filtering algorithms. An example of a transform-based algorithm based on a DFT is described herein, however, any suitable transform can be implemented with the transform-based algorithm and photonic processor.

[0458] For unitary normalization, the discrete Fourier transform (DFT) of a one-dimensional signal is given by

[0459]

number

[0460] It is calculated as: The inverse T of this transformation 1D -1 can be computed by taking the complex conjugate. Similarly, in two dimensions, the unitary regularized DFT is

[0461]

number

[0462] It can be calculated as: Performing the one-dimensional DFT defined above on a vector of size N yields the matrix-vector product T 1D This can be achieved by calculating (x) = X = Wx. The matrix W is

[0463]

number

[0464] is called the transformation matrix given by

[0465]

number

[0466] The inverse transformation is W -1 can be computed by a similar matrix-vector product where the elements of are complex conjugates. The DFT is a separable transform and can therefore be viewed as computing two one-dimensional transforms along orthogonal axes.

[0467] Therefore, the two-dimensional DFT of an [M × N] (i.e. rectangular) input x is the matrix triple product T 2D (x)=X=WxY T where W is the [M×M] transformation matrix associated with the column, Y is the [N×N] transformation matrix associated with the row, and the superscript T denotes matrix transpose. For a square input of size [N×N], the transpose matrix of column W is the same as the transpose matrix of row Y.

[0468] Equivalently, this first computes a column vector x of size M N for each x row. col and then the following matrix-vector product:

[0469]

number

[0470] where:

[0471]

number

[0472] is the Kronecker product. According to some embodiments, the result vector X col can then be reshaped into an [M×N] two-dimensional array X. 2D (x)=X=reshape(X col ) A similar process is carried out to obtain the forward transformation matrix W and the matrix W associated with the inverse transformation. -1 but may also be implemented with other discrete transformations defined in any suitable way according to the other transformations above.

[0473] In the case of a one-dimensional DFT, the matrix W is a unitary matrix and therefore may be programmed directly into the photonic array according to the embodiments described above. For other discrete transformations, the matrix W may not be unitary and therefore may need to be decomposed before being programmed into the photonic array according to the methods described above. A process 3-500 for performing a one-dimensional transformation on a vector according to some embodiments is shown in FIG. 3-5. In operation 3-502, the matrix W is programmed into the photonic array and in operation 3-504, the vector x is propagated through the array.

[0474] In some implementations, the two-dimensional transform may be calculated as described in process 3-600 of FIG. 3-6. For a two-dimensional input x of size N×N, process 3-500 may be modified to generate the two-dimensional transform defined above. In operation 3-602, a transformation matrix W corresponding to a one-dimensional input of size N is calculated. small Next, in operation 3-604, a pixel of size N is created by tiling W along the diagonals. 2 ×N 2 Then, in operation 3-606, a block diagonal matrix B of size N is created by flattening the input x row by row. 2 Column vector x col In operation 3-608, the multiplication x is created by propagating the vector x through the photonic array. partial In operation 3-610, the matrix X partial may then be reshaped into an N×N matrix, and then in operation 3-612, the N×N matrix is ​​reshaped into a matrix of size N 2 In operation 3-614, X is passed through the photonic array. partial By propagating, multiplication X=BX partial In operation 3-616, the generated vector X may be reshaped into an N×N matrix for output.

[0475] In some embodiments, the two-dimensional transform of an [N×N] input x can then be computed by first programming the matrix W into the photonic array. Secondly, by propagating columns of x through the photonic array, partial =Wx. Third, calculate the partial result X partial Fourth, transpose X through the array. partial T Propagate the sequence of WX partial T Finally, transpose the result to get X=WxW T Generate.

[0476] Some systems, such as one embodiment of the photonic-based system described herein, are limited to the implementation of real unitary matrices (i.e., orthogonal matrices). In such implementations, the transformation can still be calculated, but additional steps are required. The system must keep track of the real and imaginary parts of the transformation matrix and the input vector or image separately. The previously defined embodiment that calculates the product can be adapted to orthogonal matrices, but for every pass through the photonic array described above, the algorithm must perform four passes. If the real part of a variable is denoted as Re(x) and the imaginary part is denoted as Im(x), then the real part of the product is Re(Wx)=Re(W)Re(x)-Im(W)Im(x), and similarly the imaginary part of the product is Im(Wx)=Re(W)Im(x)+Im(W)Re(x). According to some embodiments, the process 3-700 of FIG. 3-7 can be performed in a photocore of a photonic processor that represents only real matrices. Depending on the dimensionality of the input, either process 3-500 or 3-600 may be used in process 3-700. In act 3-702, Re(W) is loaded into the photonic array. In act 3-704, depending on the dimensionality of the input, either process 3-500 or 3-600 may be performed for Re(x) and Im(x). This produces Re(W)Re(x) and Re(W)Im(x). In act 3-706, Im(W) is loaded into the photonic array. In act 3-708, either process 3-500 or 3-600 may be performed for Re(x) and Im(x), again depending on the dimensionality of the input. This produces Im(W)Re(x) and Im(W)Im(x). In operation 3-710, Re(W)Re(x) and Im(W)Im(x) are subtracted to produce Re(Wx), and in operation 3-712, Re(W)Im(x) and Im(W)Re(x) are added to produce Im(Wx).

[0477] Using the above processes 3-500, 3-600 and 3-700, the input matrix can be transformed into its transform. Once the convolution filter F and the image G have been transformed into their transform counterparts, some embodiments can provide the convolution theory. The convolution theory states that the convolution of two signals corresponds to the transformation of the element-wise product of the transforms of the two signals. Mathematically, this is

[0478]

number

[0479] or in other words,

[0480]

number

[0481] wherein:

[0482]

number

[0483] represents element-wise multiplication, and T -1 represents the inverse transform. It should be understood that in some embodiments, the dimensions of the image and the dimensions of the filter may be different, and in such cases, an appropriate dimensional transform may be used to compute each of the forward and inverse transforms. Thus, the matrix multiplication formula representing the one-dimensional convolution of the filter and image with the general transform and general dimensions is:

[0484]

number

[0485] Wherein W B is the matrix associated with the transformation of the filter F, and W D T is the matrix associated with the transformation of image G, and W A Tis the matrix associated with the inverse transform of the combined signal.

[0486] Thus, similarly, the matrix multiplication formula for two-dimensional convolution with a rectangular filter and a general transformation to an image is:

[0487]

number

[0488] Wherein W B and W C T is the matrix associated with the transformation of the filter F, and W D T and W E is the matrix associated with the transformation of image G, and W A T and W F is the matrix associated with the inverse transform of the combined signal.

[0489] 3-8, some embodiments of performing convolution in a photonic processor's photonic core using a transform-based algorithm may include the following operations: In operation 3-802, a transform of image G may be performed using any one of processes 3-500, 3-600, and / or 3-700.

[0490] In some embodiments, the filter F may then be padded with zeros to fit the size of the image G in operation 3-804, after which a transform may be performed on the filter F using any one of processes 3-500, 3-600 and / or 3-700 in operation 3-806. The transformed filter F may then be loaded into an element-wise multiplier of the photonic array in operation 3-808, and the image G may be propagated through the photonic array in operation 3-810. In operation 3-812, an inverse transform may be performed on the result of the previous calculation using any one of processes 3-500, 3-600 and / or 3-700. The result of operation 3-812 may then be reshaped to size G and cropped to generate the final convolved image G*F in operation 3-814.

[0491] In some embodiments, the convolution G*F may be calculated in a divide-and-conquer fashion, where one input is divided into a set of tiles and each tile is separately convolved with the second input. The results of each individual convolution may then be recombined into the desired output, but the algorithm (e.g., overlap-add, overlap-save) implementing this divide-and-conquer approach is not critical. If one input is much smaller than the other and a transform-based algorithm is used for the convolution operation, this approach may be much more efficient than computing the entire convolution in one operation as described above, with the filter padded to fit the size of the image. By performing the transformation of the tiles in a photonic array, it can be seen that such divide-and-conquer algorithms of transform-based convolutions can be implemented in a photonic processor.

[0492] In some embodiments, the filter F and the image G may have multiple channels. As defined above, this means that each channel of the image is convolved with the corresponding channel of the filter tensor and the results are added together element-wise. When the multi-channel convolution is calculated using a transform-based method, the summation across channels can be performed either in the transform domain or in the output domain. In practice, performing the summation in the transform domain is often chosen because this reduces the amount of data to which the output transformation must be applied. In this case, the element-wise multiplication followed by the channel-wise summation can be expressed as a series of matrix-matrix multiplications (GEMMs). Mathematically, this can be expressed as follows:

[0493] Let G be an input signal containing C data channels of an N × N image. Let F be an input signal containing M C data channels of an N × N filter. Let C and M be the number of input and output data channels, respectively. Q m,c Let be the transformed data of the m-th output channel and the c-th input channel of the filter tensor (i.e.,

[0494]

number

[0495] ). Let R be the transformed 3D [C×N×N] input tensor, and let R c Let be the c-th channel of the transformed input tensor (i.e. R C =W D T G C W E ). Thus, the convolution of F and G to produce multiple output channels is

[0496]

number

[0497] It is. S ijIf denotes a column vector consisting of C elements at the (i,j)th position of each channel in a three-dimensional [C×N×N] tensor S, then this is equivalent to

[0498]

number

[0499] It can be expressed as follows. Each Q m ij R ij The matrix-matrix multiplication can be computed in a photonic processor as described above. This can be further combined with the divide and conquer approach described above.

[0500] Aspects of the present application provide methods, procedures and algorithms that may be executed on a processing device, such as a CPU, GPU, ASIC, FPGA, or any other suitable processor. For example, a processing device may execute the procedures described above to generate variable beam splitter and modulator settings for a photocore of a photonic processor described herein. A processing device may also execute the procedures described above to generate input data that is input to a photonic processor described herein.

[0501] An example implementation of a computing device may include at least one processor and a non-transitory computer-readable storage medium. The computing device may be, for example, a desktop or laptop personal computer, a personal digital assistant (PDA), a smart mobile phone, a tablet computer, a server, or any other suitable computing device. The computer-readable medium may be configured to store data to be processed by the processor and / or instructions to be executed by the processor. The processor enables the processing of data and execution of instructions. The data and instructions may be stored in the computer-readable storage medium, for example, to enable communication between components of the computing device. The data and instructions stored in the computer-readable storage medium may include computer-executable instructions implementing techniques that operate according to principles described herein.

[0502] A computing device may further include one or more components and peripherals, including input devices and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output and a speaker or other sound generating device for audible presentation of output. Examples of input devices that may be used for a user interface include a keyboard, a pointing device such as a mouse, a touchpad, and a digitizing tablet. As another example, a computing device may receive input information through speech recognition or in other audible formats. As another example, a computing device may receive input from a camera, a lidar, or other device that generates visual data.

[0503] An embodiment of a computing device may also include a photonic processor, such as described herein. The processor of the computing device may communicate with the photonic processor via one or more interfaces. The information transmitted and received may include settings of variable beam splitters and modulators of the photonic processor and / or measurements from detectors of the photonic processor.

[0504] IV. Photonic Encoder The present inventors have recognized and appreciated that the phase-intensity relationship of an optical signal, and thus the phase and intensity modulation, are interdependent on one another, presenting a problem with respect to precisely encoding vectors into an optical field in optical processing.

[0505] Some optical modulators encode a vector of values ​​into the optical domain, where the values ​​are encoded into the phase or intensity of an optical signal. In an ideal world, an optical intensity modulator can modulate the intensity of an optical signal while keeping the phase of the optical signal unchanged, and an optical phase modulator can modulate the phase of an optical signal while keeping the intensity of the optical signal unchanged. However, in the real world, the phase and intensity of an optical signal are interdependent. Thus, intensity modulation produces a corresponding phase modulation, and phase modulation produces a corresponding intensity modulation.

[0506] For example, consider an integrated photonics platform where the intensity and phase are related to each other by the Kramers-Kronig equation, a bidirectional mathematical relation that connects the real and imaginary parts of a complex analytic function. In these cases, the optical field can be represented by a complex function where the real and imaginary parts of the function (and therefore the intensity and phase) are related to each other. In addition, realistic implementations of intensity and phase modulators generally suffer from dynamic losses, whereby the amount of modulation (phase or intensity) that the modulator imparts depends on the current setting of the modulator. These modulators suffer a certain power loss when no phase modulation is performed and a different power loss when phase modulation is performed. For example, the power loss suffered when there is no phase modulation may be L1, the power loss suffered in the case of π / 2 phase modulation may be L2, and the power loss suffered in the case of π phase modulation may be L3, where L1, L2, and L3 are different from each other. This behavior is undesirable because in addition to the phase modulation, the signal also suffers from intensity modulation.

[0507] An example of the optical modulator (4-10) is shown in FIG. in represents the input light field, and ψ out represents the optical field output by the modulator 4-10, α represents the amplitude modulation coefficient, and θ represents the phase modulation coefficient. Ideally, the modulator 4-10 modulates the optical field ψ in The entry of ψ out = αe iθ where α≦1 and θ∈[0,2π]. Modulators usually have an associated Kramers-Kronig relation and / or suffer from dynamic losses, and simply setting the values ​​of α and θ cannot be easily achieved. Conventionally, two or more modulators with two independent controllable modulation signals are used to encode both the amplitude and phase of the optical field to represent a real signed number. However, having multiple controllable modulation signals requires complex multivariate encoding schemes that often include feedback loops to simultaneously control the phase and intensity of the optical signal.

[0508] Optical modulators capable of modulating intensity without affecting phase exist, including modulators based on electro-optic materials. Unfortunately, electro-optic modulators are difficult to work in large-scale commercial situations due to the use of materials that are impractical to manufacture.

[0509] Conventional computers do not natively perform complex arithmetic. Rather, conventional computers use signed arithmetic, where numbers can be positive or negative, and complex arithmetic can be constructed by performing several calculations and combining the results. The inventors have recognized and understood that this can be used to dynamically simplify the implementation of signed linear conversion of real numbers, even in the presence of dynamic losses and / or non-ideal phase or intensity modulators.

[0510] Some embodiments relate to a technique for encoding a vector of signed real numbers using a non-ideal optical modulator (wherein intensity modulation produces phase modulation and phase modulation produces intensity modulation). In contrast to other implementations, some such embodiments include using one modulation signal to control the phase and intensity of the optical signal. The inventors have realized that when using one modulation signal to encode signed real numbers into an optical field, the exact location of the optical signal in phase space (e.g., real and imaginary parts or amplitude and phase) is not important for decoding. According to some embodiments, what is important for accurate decoding is the projection of the optical signal onto the real axis (or any other axis preselected as the measurement axis). To perform the projection onto the preselected axis, some embodiments use a coherent detection scheme. Optical domain encoding techniques of the type described herein can be used in a variety of contexts, including, but not limited to, high-speed telecommunications for short-, medium- and long-range applications, on-chip phase-sensitive measurements for sensing, communication and computing, and optical machine learning using photonic processors.

[0511] More generally, encoding techniques of the type described herein may be used in any situation in which an optical signal is processed according to a real (as opposed to complex) transformation. Conventional computers do not perform complex arithmetic directly. Rather, conventional computers use signed real arithmetic, where numbers can be positive or negative. In some embodiments, complex arithmetic can be constructed in the optical domain by performing several calculations involving real arithmetic. For example, consider an example optical linear system configured to transform an optical signal according to the following equation: y=Mx=[Re(M)+iIm(M)][Re(x)+iIm(x)]=[Re(M)Re(x)-Im(M)Im(x)]+i[Re(m)Im(x)+Im(M)(Re(x)], where x represents an input vector, M represents a transformation matrix, y represents an output vector, and i represents an input vector. 2 =-1. Considering the real transformation (where Im(M)=0), y is given by:

[0512]

number

[0513] can be rewritten as follows: Based on this equation, note that the real and imaginary parts of the input field x contribute only linearly independent components of the generated field y. The inventors have recognized and appreciated that y can be decoded using a coherent receiver by projecting y onto any axis in the complex plane.

[0514] In some embodiments, encoding techniques of the type described herein may be used as part of a photonic processing system. For example, in some embodiments, optical encoder 1-101 of FIG. 1-1, optical encoder 1-1103 of FIG. 1-11, and / or optical encoder 1-1211 of FIG. 1-12B may implement encoding techniques of the type described herein.

[0515] FIG. 4-2A is a block diagram of a photonic system implementing an optical encoding technique according to some embodiments. The photonic system 4-100 includes a light source 4-102, an encoder 4-104, an optical modulator 4-106 and an optical conversion unit 4-108, a coherent receiver 4-110, a local oscillator 4-112, and a decoder 4-114. In some embodiments, the photonic system 4-100 may include additional or alternative components not shown in FIG. 4-2A. In some embodiments, some or all of the components of FIG. 4-2A may be located on the same semiconductor substrate (e.g., a silicon substrate).

[0516] The light source 4-102 can be implemented in several ways, including, for example, using a coherent light source. In one example, the light source 4-102 includes a laser configured to emit light at a wavelength λ0. The emission wavelength can be in the visible, infrared (near, mid, and far infrared) or ultraviolet portion of the electromagnetic spectrum. In some embodiments, λ0 can be in the O-band, C-band, or L-band. The light emitted by the light source 4-102 is modulated using an optical modulator 4-106. The optical modulator 4-106 is a non-ideal modulator in which phase modulation produces intensity modulation and intensity modulation produces phase modulation. In some embodiments, the phase modulation can be related to the intensity modulation according to the Kramers-Kronig equation. Alternatively or additionally, the modulator 4-106 can be subject to dynamic losses, in which case phase shifts lead to attenuation.

[0517] In some embodiments, the modulator 4-106 is driven by one electrical modulation signal 4-105. Thus, one modulation signal modulates both the phase and the amplitude of the optical field. This is in contrast to modulators traditionally used in optical communications, which encode symbols into the complex amplitude of the optical field, where each symbol represents more than one bit. In such types of modulators, in fact, multiple modulation signals modulate the optical field. Consider, for example, an optical modulator configured to provide a quadrature phase-shift keying (QPSK) modulation scheme. In these types of modulators, one modulation signal modulates the real part of the optical field and one modulation field modulates the imaginary part of the optical field. Thus, in a QPSK modulator, the phase and amplitude of the optical field are jointly modulated using two modulation signals.

[0518] Examples of modulators that may be used for the modulator 4-106 include Mach-Zehnder modulators, electro-optic modulators, ring or disk modulators or other types of resonant modulators, electro-absorption modulators, Frank-Keldysh modulators, acousto-optic modulators, Stark effect modulators, magneto-optic modulators, thermo-optic modulators, liquid crystal modulators, quantum confined optical modulators, and photonic crystal modulators, among other possible types of modulators.

[0519] The encoder 4-104 generates a modulated signal 4-105 based on the real numbers it encodes. For example, in some embodiments, the encoder 4-104 may include a table that maps real numbers to the amplitude of the modulated signal.

[0520] The optical conversion unit 4-108 may be configured to convert the intensity and / or phase of the received optical field. For example, the optical conversion unit 4-108 may include an optical fiber, an optical waveguide, an optical attenuator, an optical amplifier (such as an erbium-doped fiber amplifier), a beam splitter, a beam combiner, a modulator (such as an electro-optic modulator, a Franz-Keldysh modulator, a resonant modulator, or a Mach-Zehnder modulator, among others), a phase shifter (such as a thermal phase shifter or a phase shifter based on the plasma dispersion effect), an optical resonator, a laser, or any suitable combination thereof. In relation to optical communication, the optical conversion unit 4-108 may include an optical fiber communication channel. The optical communication channel may include, for example, an optical fiber and optionally an optical repeater (e.g., an erbium-doped fiber amplifier). In relation to optical processing, the optical conversion unit 4-108 may include a photonic processing unit, an example of which is described in more detail below. In some embodiments, the optical transformation unit 4-108 performs a real transformation such that the imaginary part of the transformation is approximately equal to zero.

[0521] The optical field output by the optical conversion unit 4-108 is provided directly or indirectly (e.g., after passing through one or more other photonic components) to a coherent receiver 4-110. The coherent receiver 4-110 may include a homodyne optical receiver or a heterodyne optical receiver. A reference signal against which the received signal beats may be provided by a local oscillator 4-112, as shown in FIG. 4-2A, or may be provided together with the modulated optical field after passing through the optical conversion unit 4-108. The decoder 4-114 may be configured to extract a real number (or a vector of real numbers) from the signal output by the coherent receiver 4-112.

[0522] In at least some embodiments in which the photonic processing system 1-100 of FIG. 1-1 implements the encoding techniques of the type described herein, the optical modulator 4-106 may be part of the optical encoder 1-101, the optical conversion unit 4-108 may be part of the photonic processor 1-103, and the coherent receiver 4-110 may be part of the optical receiver 1-105.

[0523] FIG. 4-2B is a flow chart illustrating a method for processing real numbers in the optical domain according to some embodiments. Method 4-150 may be performed using the system of FIG. 4-2A or using any other suitable system. Method 4-150 begins at operation 4-152 where a value representing a real number is provided. In some embodiments, the real number may be signed (i.e., positive or negative). The real number may represent a particular environmental variable or parameter, such as a physical condition (e.g., temperature, pressure, etc.), information associated with an object (e.g., position, motion, speed, rotation rate, acceleration, etc.), information associated with a multimedia file (e.g., sound intensity of an audio file, pixel color and / or intensity of an image or video file), information associated with a particular chemical / organic element or compound (e.g., concentration), information associated with a financial asset (e.g., price of a particular security), or any other suitable type of information, including information derived from the examples above. Information represented by signed real numbers may be useful for a variety of reasons, including, for example, training machine learning algorithms, making predictions, performing data analysis, performing troubleshooting, or simply collecting data for future use.

[0524] In operation 4-154, a value representing a real number may be encoded into the optical field. In some embodiments, encoding the value into the optical field includes modulating the phase and intensity of the optical field based on the value. As a result, the phase and amplitude of the optical field reflect the encoded value. Operation 4-154 may be performed in some embodiments using an encoder 4-104 and a modulator 4-106 (see FIG. 4-2A). In some such embodiments, modulating the phase and amplitude based on the value includes driving a modulator with an electrical modulation signal. Thus, the modulation signal modulates both the phase and amplitude of the optical field. Referring again to FIG. 4-2A, by way of example and not limitation, the encoder 4-104 may drive an optical modulator 4-106 using a modulation signal 4-105.

[0525] It should be noted that the use of one modulation signal to drive the modulator does not preclude the use of other control signals to control the environment in which the modulator operates. For example, one or more control signals may be used to control the temperature of the modulator or the temperature of specific parts of the modulator. One or more control signals may be used to power the operation of the modulator. One or more control signals may be used to bias the modulator in a specific regime of operation, such as biasing the modulator in a linear region or setting the wavelength of the modulator to match the wavelength of the light source (e.g., λ0 in FIG. 4-2A).

[0526] A particular type of modulator according to some embodiments is shown in FIG. 4-3A. The modulator 4-206 is a ring resonator modulator. It should be noted that a ring modulator is described here merely by way of example, and any other suitable type of modulation, including those listed above, may alternatively be used. The modulator 4-206 includes a waveguide 4-208, a ring 4-210, and a phase shifter 4-212. The ring 4-210 exhibits a resonant wavelength, the value of which depends, among other parameters, on the length of the ring circumference. The optical field ψ in is launched into the waveguide 4-208, the optical field is in Depending on the wavelength of ψ, it may or may not be coupled to the ring 4-210. For example, in If the wavelength of ψ coincides with the resonant wavelength, in At least a portion of the energy of ψ is transferred to the ring 4-210 via evanescent coupling. The force transferred to the ring oscillates indefinitely inside the ring until it is completely scattered or otherwise dissipated. In contrast, ψ in If the wavelength of does not match the resonant wavelength, ψ in can travel through the waveguide 4-208 without any significant attenuation.

[0527] Therefore, the optical field is ψ for the resonant wavelength in More specifically, the phase and intensity of the optical field can vary with the wavelength of the resonant wavelength ψ inFIG. 4-3B shows the relationship between the intensity (top chart) and phase (bottom chart) of ψ versus the resonant wavelength. in The top chart shows how ψ can vary as a function of wavelength. in The graph shows the intensity spectral response α as a function of all possible wavelengths (λ) that ψ can have. The bottom chart shows the phase spectral response θ as a function of the same wavelength. At the resonant frequency, the intensity response shows a dip and the phase response shows an inflection point. Wavelengths substantially below the resonant frequency experience low intensity attenuation (α to 1) and low phase change (θ to 0). Wavelengths substantially above the resonant frequency experience low intensity attenuation (α to 1) and sign change (θ to -π). At the resonant wavelength, ψ in The intensity of is attenuated by a value corresponding to the dip and undergoes a phase shift equal to the value of the inflection point (i.e., -π / 2). In the example of FIG. 4-3B, ψ in The wavelength (λ) of the in The intensity of α V0 (where 0<α V0 <1) and the phase is V0 (where -π<θ V0 <-π / 2).

[0528] Referring again to FIG. 4-3A, ψ in Modulation of the amplitude and phase of can be performed using a voltage V, which in this case embodies the modulation signal 4-105 of FIG. 4-2A. When a voltage V is applied to the phase shifter 4-212, the refractive index of the phase shifter changes, and therefore the effective length of the perimeter of the ring also changes. This in turn leads to a variation in the resonant frequency of the ring. The extent to which the effective perimeter length (and therefore the resonant frequency) changes depends on the amplitude of V.

[0529] In the example of FIG. 4-3B, the voltage V is set to zero. In the example of FIG. 4-3C, the voltage is set to V1. As shown in FIG. 4-3C, application of V1 to the phase shifter 4-212 shifts the intensity response and phase spectral response of the modulator along the wavelength axis. In this case, the response exhibits a red shift (i.e., a shift toward larger wavelengths). As a result, ψ in The intensity of is given by, where α V1 (α V1 is α V0 (different from ) and the phase is V1 (θ V1 θ V0 Shift by 1 (different from ).

[0530] Therefore, a change in voltage V is in This leads to a change in the intensity and phase of V. In other words, V can be considered a modulation signal. In some embodiments, the voltage V may be the only modulation signal driving the modulator 4-210. In some embodiments, the voltage V may be expressed as follows: V=V DC +V(t), where V DC is a constant, and V(t) varies over time depending on the real number to be encoded. DC is the resonant frequency ψ in The ring can be prebiased so that it is in the vicinity of the wavelength (λ 0 ).

[0531] FIG. 4-3D is an encoding table (e.g., look-up table) that provides an example of how real numbers can be encoded into the phase and intensity of an optical field using the modulator 4-206, according to some embodiments. In this case, by way of example, it is assumed that real numbers from -10 to 10 with increments equal to 1 are encoded into the optical domain (see the column marked "Real Numbers to Encode"). Of course, any suitable set of real numbers can be encoded using the techniques described herein. The column marked "V (Applied Voltage)" indicates the voltage applied to the phase shifter 4-212. The column marked "α (Amplitude Modulation)" indicates the amplitude of ψ in The column labeled "θ (phase modulation)" shows the intensity spectral response at wavelength (λ0) of ψin The graph shows the value of the phase spectral response at a wavelength (λ0) of 100 kHz (see the bottom charts of Figures 4-3B and 4-3C). In this case, the real number "-10" maps to a specific voltage V1, resulting in an amplitude modulation α1 and a phase modulation θ1, the real number "-9" maps to a specific voltage V2, resulting in an amplitude modulation α2 and a phase modulation θ2, and so on. N and the amplitude modulation α N and phase modulation θ N Thus, different real numbers are encoded using different magnitude / phase pairs.

[0532] FIG. 4-3E provides a visual representation of the encoding table of FIG. 4-3D in the complex plane, according to some embodiments. Different points along the line 4-300 represent different intensity / phase relationships according to the spectral response of FIG. 4-3B and FIG. 4-3C as the voltage is changed. For example, the symbol S in represents a particular symbol characterized by an intensity modulation α and a phase modulation θ. in represents a particular row from the table of FIG. 4-3D. Each row in the table maps to a different point on line 4-300.

[0533] Referring again to FIG. 4-2B, in operation 4-156 a transformation is applied to the modulated optical signal output by the modulator. The transformation may include an intensity transformation and / or a phase transformation depending on how one wishes to process the encoded real number. In some embodiments, an optical transformation unit 4-108 may be used to transform the signal output by the modulator 4-106. The optical transformation unit 4-108 may generate a symbol S characterized by an intensity α and a phase θ. in Let S be a symbol S characterized by intensity αβ and phase θ+Δθ. out In other words, the optical transformation unit 4-108 introduces an intensity modulation β and a phase shift Δθ, where the values ​​of β and Δθ depend on the particular optical transformation unit being used.

[0534] For example, the input symbol S in Figure 4-3E inConsidering, Fig. 4-4A to Fig. 4-4C provide an example of how the optical transformation unit can perform the transformation. The transformation in Fig. 4-4A is such that β<1 and Δθ=0. That is, the optical transformation unit introduces attenuation but does not change the phase of the input optical field. This may be the case, for example, for an optical fiber without an optical repeater, where the optical fiber has a length selected to preserve the input phase. As a result, the line 4-300 is transformed into a line 4-401, which is a compressed version of the line 4-300. The output sample S out is S in has the same phase as, but has an intensity equal to αβ (i.e., is attenuated by β).

[0535] The transformation of FIG. 4-4B introduces attenuation and a phase shift. This may be the case, for example, for an optical fiber of any length and without optical repeaters. As a result, the line 4-300 is transformed into a line 4-402, which is a compressed and rotated version of the line 4-300. The output sample S out has phase θ+Δθ and magnitude αβ.

[0536] The transformation of Figure 4-4C includes multi-mode transformation. This can occur when an input light field is combined with one or more other light fields. With multi-mode transformation, the line 4-403 can be reshaped in any suitable manner. Examples of multi-mode transformation include optical processing units, some of which are described in more detail below.

[0537] Referring again to FIG. 4-2B, in operation 4-158, the coherent receiver may mix the transformed modulated optical field with a reference optical signal to obtain an electrical output signal. In some embodiments, the mixing may be performed using the coherent receiver 4-110 of FIG. 4-2A. The reference optical signal may be a signal generated by a local oscillator (e.g., local oscillator 4-112 of FIG. 4-2A) or may be a signal transmitted together with the modulated optical field through the optical transformation unit. The effect of mixing is shown in FIG. 4-5 by way of example and not limitation. Line 4-403 (the same line as shown in FIG. 4-4C) represents all possible intensity / phase pairs resulting from a particular multimode transformation. FIG. 4-5 may, for example, show the effect of propagating an optical signal through the photonic processor 1-103 of FIG. 1-1.

[0538] S out Consider that S is the symbol resulting from the transformation at a particular time. As mentioned above, out is characterized by intensity αβ and phase θ+Δθ. When the transformed field is mixed with a reference signal, points along the line 4-403 are projected onto the reference axis 4-500. Thus, for example, the symbol S out is projected onto point A on axis 4-500. Point A represents the electrical output signal resulting from the mixing, and is between 0 and A.

[0539]

number

[0540] Note that the angle of the axis 4-500 with respect to the real axis (i.e., the angle φ) depends on the phase of the reference signal with respect to the transformed modulated light field. In this example, 0<φ<π / 4.

[0541] Referring again to FIG. 4-2B, in operation 4-160, the decoder may obtain a value representing a decoded real number based on the electrical output signal obtained in operation 4-158. In some embodiments, when decoding symbols obtained through optical transformation, the exact location of the symbols in the complex plane may not be important. What is important for accurate decoding according to some embodiments is the projection of the optical signal onto a known reference axis. In other words, the symbols along line 4-403 are represented by the symbol S out can be mapped to points along the reference axis in the same way that {right arrow over (x)} is mapped to point A. Thus, some embodiments implement an optical decoding scheme in which points along the reference axis correspond to particular symbols in the complex plane.

[0542] The manner in which points along the reference axis are mapped to symbols in the complex plane can be determined using a calibration procedure. During the calibration procedure, a set of input symbols (symbols representing a set of real numbers) of known intensity and phase is passed through a specific optical transformation unit, and the generated symbols are coherently detected using a reference signal with known phase. The amplitudes of the generated electrical output signals (i.e., the amplitudes of the projections along the reference axis) are recorded and stored in a table (e.g., a look-up table). The table can be subsequently used during the operation of decoding the real numbers based on the amplitudes of the projections along the reference axis.

[0543] An example of such a table according to some embodiments is shown in Figures 4-6. The table includes a column for the phase of the reference signal ("reference phase φ"), a column for the projection along the reference axis ("projection

[0544]

number

[0545] , "electrical output signal amplitudes") and a sequence of decoded real numbers ("decoded real numbers"). The table can be filled during a calibration procedure that involves 1) passing symbols of known intensity and phase through a known optical transformation unit, 2) setting the phase of a reference signal to a known value, 3) mixing the transformed symbols with the reference signal thereby projecting the transformed symbols onto a known reference axis, 4) determining the amplitude of the projection along the reference axis, and 5) recording the expected output real numbers. In operation, a user can use the table to infer the decoded real numbers based on the value of the reference phase and the measured projection.

[0546] In the example of Figures 4-6, by way of example and not limitation, only two reference phases are considered: -0 and π / 6. At φ=0, the projections -1, -0.8 and 1 are mapped to the real numbers -9.6, 0.2 and 8.7, respectively. At φ=π / 6, the projections -0.4, 0.6 and 0.9 are mapped to the real numbers 3.1, -5 and 10, respectively. Thus, for example, if a user obtains a projection of -0.4, when using a reference phase of π / 6, the user can infer that the decoded real number is 3.1.

[0547] FIG. 4-7 shows a particular example of a photonic system 4-700 in which an optical conversion unit includes an optical fiber 4-708, according to some embodiments. This example illustrates how encoding techniques of the type described herein can be used in connection with optical communications. In this implementation, the optical fiber 4-708 separates a transmitter 4-701 from a receiver 4-702.

[0548] The transmitter 4-701 includes a light source 4-102, an encoder 4-104, and an optical modulator 4-106 (as described above in connection with FIG. 4-2A), and the receiver 4-702 includes a coherent receiver 4-110 and a decoder 4-114 (also described above in connection with FIG. 4-2A). As mentioned above, in some embodiments, the optical modulator 4-106 may be driven using one modulation signal 4-105. In some such embodiments, the optical modulator 4-106 may implement an on-off keying (OOK) modulation scheme.

[0549] It should be noted that in this implementation, the reference signal used for coherent detection is not generated locally at the receiver 4-702, but is provided directly from the transmitter 4-701 (although other implementations may include a local oscillator at the receiver 4-702). To illustrate this concept, consider for example the plot of FIG. 4-8. FIG. 4-8 shows an example of a power spectral density of the optical field at the output of the transmitter 4-701. As shown, the power spectral density includes a signal 4-800 and a carrier 4-801. The signal 4-800 represents information encoded using the modulator 4-106. The carrier 4-801 represents a tone at the wavelength (λ0) of the light source 4-102. Transmitting this optical field through an optical fiber 4-708, the coherent receiver 4-110 may perform mixing using the carrier 4-801 itself as a reference signal.

[0550] Some embodiments relate to methods of fabricating a photonic system of the type described herein. One such method according to some embodiments is shown in FIG. 4-9. Method 4-900 begins in operation 4-902 where a modulator is fabricated. The modulator may be fabricated to be driven with an electrical modulation signal. An example of a modulator is modulator 4-106 (FIG. 4-2A). In operation 4-904, a coherent receiver is fabricated. In some embodiments, the coherent receiver is fabricated in the same semiconductor substrate as the modulator is fabricated. In operation 4-906, an optical conversion unit is fabricated. The optical conversion unit may be fabricated in the same substrate as the coherent receiver is fabricated and / or in the same semiconductor substrate as the modulator is fabricated. The optical conversion unit may be fabricated coupled between the modulator and the coherent receiver. An example of an optical conversion unit is optical conversion unit 4-108 (FIG. 4-2A).

[0551] V. Differential Receiver The present inventors have recognized and understood that some conventional optical receivers are particularly susceptible to noise generated from voltage sources, noise resulting from photodetectors inevitably generating dark current, and other forms of noise. The presence of noise reduces the signal-to-noise ratio and therefore the ability of these photodetectors to accurately sense incoming optical signals. This can adversely affect the performance of the systems in which these photodetectors are deployed. For example, this can adversely affect the bit error rate and power budget of the system.

[0552] The present inventors have developed an optical receiver with reduced susceptibility to noise. Some embodiments of the present application relate to an optical receiver in which both optical / electrical conversion and amplification are performed differentially. In the optical receiver described herein, subtraction of two separate signals is performed. First, the photocurrents are subtracted from each other to generate a pair of differential currents. The generated differential currents are then further subtracted from each other to generate an amplified differential output. The present inventors have recognized and understood that having an optical receiver with multiple levels of signal subtraction leads to multiple levels of noise cancellation, thus substantially reducing noise from the system. This may have several advantages over conventional optical receivers, including a wider dynamic range, a larger signal-to-noise ratio, a larger output amplitude, and increased power supply noise immunity.

[0553] Optical receivers of the type described herein can be used in a variety of settings, including, for example, telecommunications and data communications (including local area networks, metropolitan area networks, wide area networks, data center networks, satellite networks, etc.), analog applications such as radio over fiber, all-optical switches, lidar, phased arrays, coherent imaging, machine learning, other types of artificial intelligence applications, and other applications. In some embodiments, optical receivers of the type described herein can be used as part of a photonic processing system. For example, in some embodiments, optical receivers of the type described herein can be used to implement optical receiver 1-105 of FIG. 1-1 (e.g., one or more homodyne receivers 1-901 of FIG. 1-9).

[0554] FIG. 5-1 illustrates a non-limiting example of an optical receiver 5-100 according to some non-limiting embodiments of the present application. As illustrated, the optical receiver 5-100 includes photodetectors 5-102, 5-104, 5-106, and 5-108, although other implementations include more than four photodetectors. Photodetector 5-102 may be connected to photodetector 5-104, and photodetector 5-106 may be connected to photodetector 5-108. In some embodiments, the anode of photodetector 5-102 is connected to the cathode of photodetector 5-104 (at node 5-103), and the cathode of photodetector 5-106 is connected to the anode of photodetector 5-108 (at node 5-105). In the example of FIG. 5-1, the cathodes of photodetectors 5-102 and 5-108 are connected to a power supply V DD , and the anodes of the photodetectors 5-104 and 5-106 are connected to a reference potential (e.g., ground). In some embodiments, the reverse configuration is also possible. The reference potential can be a potential equal to zero or -V DD etc. V DD may have any suitable value.

[0555] The photodetectors 5-102 through 5-108 may be implemented in any of a number of ways, including, for example, PN junction photodiodes, PIN junction photodiodes, avalanche photodiodes, phototransistors, photoresistors, etc. The photodetectors may include a material capable of absorbing light at a wavelength of interest. For example, at O-band, C-band, or L-band wavelengths, the photodetectors may have an absorption region made at least partially of germanium, as a non-limiting example. For visible light, the photodetectors may have an absorption region made at least partially of silicon, as another non-limiting example.

[0556] The photodetectors 5-102 to 5-108 may be integrated components monolithically formed as part of the same substrate. In some embodiments, the substrate may be a silicon substrate, such as a bulk silicon substrate or a silicon-on-insulator substrate. Other types of substrates may also be used, including, for example, indium phosphide or any suitable semiconductor material. To reduce variations in the characteristics of the photodetectors due to manufacturing tolerances, in some embodiments, the photodetectors may be positioned near one another. For example, the photodetectors may be positioned within 1 mm of the substrate. 2 Below, 0.1mm 2 Less than or equal to 0.01mm 2 It may be positioned within the following area:

[0557] As further shown in FIG. 5-1, the photodetectors 5-102 through 5-108 are connected to a differential operational amplifier 5-110. For example, the photodetectors 5-102 and 5-104 may be connected to the non-inverting input ("+") of the DOA 5-110, and the photodetectors 5-106 and 5-108 may be connected to the inverting input ("-") of the DOA 5-110. The DOA 5-110 has a pair of outputs. One output is inverting and the other output is non-inverting.

[0558] In some embodiments, as described in more detail in connection with FIG. 5-2, the photodetectors 5-102 and 5-106 may be configured to receive the same optical signal "t" and the photodetectors 5-104 and 5-108 may be configured to receive the same optical signal "b". In some embodiments, the photodetectors 5-102 to 5-108 may be designed to be substantially equal to one another. For example, the photodetectors 5-102 to 5-108 may be formed using the same process steps and the same photomask patterns. In these embodiments, the photodetectors 5-102 to 5-108 may exhibit substantially the same characteristics, such as substantially the same responsivity (ratio of photocurrent to received optical power) and / or substantially the same dark current (current generated when no optical power is received). In these embodiments, the photocurrents generated by the photodetectors 5-102 and 5-106 in response to receiving the signal t may be substantially equal to one another. Such photocurrents are indicated in FIG. 5-1 as "i t". Note that due to the orientation of photodetectors 5-102 and 5-106, the photocurrents generated by photodetectors 5-102 and 5-106 are directed in opposite directions. That is, the photocurrent of photodetector 5-102 is directed toward node 5-103, and the photocurrent of photodetector 5-106 is directed away from node 5-105. Furthermore, the photocurrents generated by photodetectors 5-104 and 5-108 in response to receiving signal b may be substantially equal in value to one another. Such photocurrents may be referred to as "i b ". Due to the orientation of photodetectors 5-104 and 5-108 relative to one another, the photocurrents generated by photodetectors 5-104 and 5-108 are directed in opposite directions. That is, the photocurrent of photodetector 5-108 is directed toward node 5-105 and the photocurrent of photodetector 5-104 is directed away from node 5-103.

[0559] Considering the orientation of the photodetector, the amplitude i t -i b A current with amplitude i emerges from node 5-103. b -i t emerges from node 5-105. The currents therefore have substantially the same amplitude but opposite signs.

[0560] The photodetectors 5-102 to 5-108 may generate dark current. Dark current is usually due to leakage and arises from the photodetectors regardless of whether they are exposed to light or not. Dark current arises even when there is no input optical signal, and therefore contributes substantially to noise in optical receivers. The inventors have realized that the adverse effects of these dark currents can be significantly attenuated by the current subtraction described above. Thus, in the example of FIG. 5-1, the dark current of the photodetector 5-102 and the dark current of the photodetector 5-104 are substantially cancelled out (or at least substantially reduced by each other), as are the dark currents of the photodetectors 5-106 and 5-108. Thus, the noise due to the presence of dark current is significantly attenuated.

[0561] FIG. 5-2 illustrates a photonic circuit 5-200 configured to provide two optical signals to photodetectors 5-102 to 5-108, according to some non-limiting embodiments. The photonic circuit 5-200 may include optical waveguides that route the optical signals to the photodetectors. The optical waveguides may be made of a material that is transparent or at least partially transparent to light of the wavelength of interest. For example, the optical waveguides may be made of silicon, silicon oxide, silicon nitride, indium phosphide, gallium arsenide, or any other suitable material. In the example of FIG. 5-2, the photonic circuit 5-200 includes input optical waveguides 5-202 and 204 and couplers 5-212, 5-214, and 5-216. As further shown, the output optical waveguides of the photonic circuit 5-200 are coupled to the photodetectors 5-102 to 5-108.

[0562] In the example of FIG. 5-2, the couplers 5-212, 5-214, and 5-216 include directional couplers, and evanescent coupling allows for the transfer of optical power between adjacent waveguides. However, other types of couplers, such as Y-junctions, X-junctions, optical crossovers, and reverse-directional couplers, can also be used. In other embodiments, the photonic circuit 5-200 can be implemented using a multimode interferometer (MMI). The couplers 5-212, 5-214, and 5-216 can be 3 dB couplers (having a 50%-50% coupling ratio) in some embodiments, although other ratios, such as 51%-49%, 55%-45%, or 60%-40%, are also possible. It should be understood that due to manufacturing tolerances, the actual coupling ratio may deviate slightly from the intended coupling ratio.

[0563] Signal s1 may be provided to input optical waveguide 5-202, and signal s2 may be provided to input optical waveguide 204. Signals s1 and s2 may be provided to each input optical waveguide using, for example, optical fibers. In some embodiments, s1 represents a reference local oscillator signal, such as a signal generated by a reference laser, and s2 represents the signal to be detected. Thus, the optical receiver may be considered as a homodyne optical receiver. In some such embodiments, s1 may be a continuous wave (CW) optical signal, while s2 may be modulated. In other embodiments, both signals are modulated or both signals are CW optical signals, and the application is not limited to any particular type of signal.

[0564] In the example of Figure 5-2, the signal s1 has an amplitude A LO and phase

[0565]

number

[0566] and the signal s2 has an amplitude A S and phase φ. The combiner 5-212 combines signals s1 and s2 such that signals t and b appear at each output of the combiner 5-212. In an embodiment in which the combiner 5-212 is a 3 dB combiner, t and b are expressed by the following formula:

[0567]

number

[0568] and the powers T and B (for t and b, respectively) can be given by the following formula:

[0569]

number

[0570] It can be given by: Thus, in an embodiment in which the combiners 5-214 and 5-216 are 3 dB combiners, the photodetectors 5-102 and 5-106 may each receive power given by T / 2, and the photodetectors 5-104 and 5-108 may each receive power given by B / 2.

[0571] Referring again to FIG. 5-1, assuming that the responsivities of the photodetectors 5-102 through 5-108 are all equal to one another (although not all embodiments are limited in this respect), the currents emerging from nodes 5-103 and 5-105, respectively, are given by the following equation:

[0572]

number

[0573] It can be given by: The DOA5-110 amplifies the differential signal received at the “+” and “-” inputs and converts it into a voltage V out,n and V out,p In some embodiments, the DOA 5-110 is configured to generate an amplified differential output represented in FIG. 5-1 by: b -i t ,i t -i b ) based on the voltage difference vs. (V out,n ,V out,p ) can be considered a differential transimpedance amplifier. In some embodiments, V out,n , V out,p Each of the currents i t -i b and current i b -i t and thus, the following formula: V out,p =2z(i t -i b ) V out,n =2z(i b -i t ) is produced.

[0574] This differential pair of voltages may be provided as an input to any suitable electronic circuitry, including, but not limited to, an analog-to-digital converter (not shown in FIG. 5-1). Note that the optical receiver 5-100 provides two levels of noise rejection. The first level of noise rejection results from the subtraction of the photocurrents, and the second level of noise rejection results from the subtraction performed in the differential amplification stage. This results in a significant increase in noise rejection.

[0575] In the example of Figure 5-1, the impedances z are shown as being equal to one another, although different impedances may be used in other embodiments. These impedances may include passive electrical components such as resistors, capacitors, and inductors, and / or active electronic components such as diodes and transistors. The components that make up these impedances may be selected to provide desired gain and bandwidth, among other possible characteristics.

[0576] As mentioned above, the optical receiver 5-100 may be monolithically integrated into a substrate. One such substrate according to some non-limiting embodiments is shown in FIG. 5-3A. In this example, the photodetectors 5-102 to 5-108, the photonic circuit 5-200, and the DOA 5-110 are monolithically integrated as part of the substrate 5-301. In other embodiments, the photodetectors 5-102 to 5-108 and the photonic circuit 5-200 may be integrated into the substrate 5-301, and the DOA 5-110 may be integrated into a separate substrate 5-302. In the example of FIG. 5-3B, the substrates 5-301 and 5-302 are flip-chip bonded to each other. In the example of FIG. 5-3C, the substrates 5-301 and 5-302 are wire bonded to each other. In yet another example (not shown), the photodetectors 5-102 through 5-108 and the photonic circuit 5-200 may be fabricated on separate substrates.

[0577] Some embodiments of the present application relate to a method of fabricating an optical receiver, one such method according to some non-limiting examples is shown in Figure 5-4. The method 5-400 begins at operation 5-402 where a plurality of photodetectors are fabricated on a first substrate.

[0578] Once fabricated, the photodetectors can be connected together, for example, in the configuration shown in FIG. 5-1. In some embodiments, the photodetectors are located within 1 mm of the first substrate. 2 Below, 0.1mm 2 Less than or equal to 0.01mm 2 In operation 5-404, a photonic circuit is fabricated on the first substrate. The photonic circuit may be configured to provide pairs of optical signals to the photodetectors, for example, as shown in FIG. 5-2. In operation 5-406, a differential operational amplifier may be fabricated on the second substrate. An example of a differential operational amplifier is the DOA 5-110 of FIG. 5-1. In operation 5-408, the first substrate may be bonded to the second substrate, for example, via flip chip bonding (as shown in FIG. 5-3A), via wire bonding (as shown in FIG. 5-3B), or using any other suitable bonding technique. Once the substrates are bonded, the photodetectors of the first substrate may be electrically connected to the differential operational amplifiers of the second substrate, for example, as shown in FIG. 5-1.

[0579] An example of a fabrication process according to some embodiments is shown generally in Figures 5-4A through 5-4F. Figure 5-4A shows a substrate 5-301 having a lower cladding 5-412 (e.g., an oxide layer such as a buried oxide layer or other type of dielectric material) and a semiconductor layer 5-413 (e.g., a silicon layer, a silicon nitride layer, or other type of material layer). In Figure 5-4B, the semiconductor layer 5-413 is patterned, for example, using photolithographic exposure, to form regions 5-414. The regions 5-414 may be configured to form optical waveguides in some embodiments. In some embodiments, the generated pattern resembles the photonic circuit 5-200 (Figure 5-2), with the waveguides 5-202 and 204 and the couplers 5-212, 5-214, and 5-216 embedded in one or more regions 5-414. In FIG. 5-4C, photodetectors 5-102, 5-104, and 5-108 (and optionally other photodetectors) are formed. In this example, optically absorbing material 5-416 is deposited adjacent region 5-414. The optically absorbing material 5-416 may be patterned to form the photodetectors. The material used for the optically absorbing material may depend on the wavelength to be detected. For example, germanium may be used for L-band, C-band, or O-band wavelengths. Silicon may be used for visible wavelengths. Of course, other materials are possible. The optically absorbing material 5-416 may be positioned optically coupled to region 5-414 in any suitable manner, including, but not limited to, butt coupling, taper coupling, and evanescent coupling.

[0580] In FIG. 5-4D, the DOA 5-110 is formed. In some embodiments, the DOA 5-110 includes several transistors formed via ion implantation. FIG. 5-4D shows implanted regions 5-418, which may form portions of one or more transistors of the DOA 5-110. Although only one ion implant is shown in FIG. 5-4D, in some embodiments, the formation of the DOA 5-110 may include two or more ion implants. Additionally, the DOA 5-110 may be electrically connected to a photodetector, for example, via one or more conductive traces formed in the substrate 5-301.

[0581] The configuration of Figure 5-4D is such that the photonic circuit 5-200, the photodetectors 5-102 to 5-108, and the DOA 5-110 are formed on a common substrate (as shown in Figure 5-3A). Configurations in which the DOA 5-110 is formed on a separate substrate (as shown in Figure 5-3B or Figure 5-3C) are also possible. In one such example, as shown in Figure 5-4E, the DOA 5-110 is formed on a separate substrate 5-302, and the implanted region 5-428 is formed via one or more ion implants.

[0582] Subsequently, the substrate 5-301 is bonded to the substrate 5-302, and the photodetectors 5-102 through 5-108 are connected to the DOA 5-110. In FIG. 5-4F, a conductive pad 5-431 is formed and placed in electrical communication with the optically absorbing material 5-416, and a conductive pad 5-432 is formed and placed in electrical communication with the implanted region 5-428. The conductive pads are electrically connected via wire bonding (as shown in FIG. 5-4F) or flip chip bonding.

[0583] Some embodiments relate to a method of receiving an input optical signal. Some such embodiments may include homodyne detection, although applications are not limited in this respect. Other embodiments may include heterodyne detection. Still other embodiments may include direct detection. In some embodiments, receiving the optical signal may include an optical receiver 5-100 (FIG. 5-1), although other types of receivers may be used.

[0584] An example of a method for receiving an input optical signal according to some embodiments is shown in FIG. 5-5. Method 5-500 begins at operation 5-502 where the input signal is combined with a reference signal to obtain first and second optical signals. The input signal may have data encoded therein, for example in the form of amplitude modulation, pulse width modulation, phase or frequency modulation, among other modulation types. In some embodiments, including homodyne detection, the reference signal may be a signal generated by a local oscillator (e.g., a laser). In other embodiments, data may also be encoded in the reference signal. In some embodiments, the input signal and the reference signal are combined using a photonic circuit 5-200 (FIG. 5-2), although other types of optical couplers may be used, including, but not limited to, MMIs, Y-junctions, X-junctions, optical crossovers, and reverse direction couplers. In embodiments where photonic circuit 5-200 is used, t and b may represent signals obtained from combining the input signal and the reference signal.

[0585] In operation 5-504, the first optical signal is detected using a first photodetector and a second photodetector, and the second optical signal is detected using a third photodetector and a fourth photodetector to generate a differential current pair. In some embodiments, operation 5-504 may be performed using an optical receiver 5-100 (FIG. 5-1). In some such embodiments, the first optical signal is detected using photodetectors 5-102 and 5-106, and the second optical signal is detected using photodetectors 5-104 and 5-108. The generated differential current pair is collectively referred to as current i b -i t and i t -i b Being differential, in some embodiments, the pair of currents may have substantially equal amplitudes but substantially opposite phases (e.g., have a π phase difference).

[0586] In act 5-506, a differential operational amplifier (e.g., DOA5-110 in FIG. 5-1) generates an amplified differential voltage pair using the differential current pair generated in act 5-504. In an embodiment using DOA5-110, the generated differential voltage pair is a voltage Vout,n and V out,p Because they are differential, in some embodiments, the voltage pairs may have substantially equal amplitudes but substantially opposite phases (e.g., have a π phase difference).

[0587] Method 5-500 may have one or more advantages over conventional methods of receiving optical signals, including, for example, wider dynamic range, greater signal-to-noise ratio, greater output amplitude, and increased power supply noise immunity.

[0588] VI. Phase Modulator The present inventors have recognized and understood that certain optical phase modulators suffer from high dynamic loss and low modulation speed, which significantly limit the range of applications in which they can be deployed. More specifically, some phase modulators include a large trade-off between modulation speed and dynamic loss, such that increasing the modulation speed increases the dynamic loss. As used herein, the phrase "dynamic loss" refers to the optical power loss suffered by an optical signal that depends on the degree to which the phase is modulated. An ideal phase modulator is such that the power loss is independent of the phase modulation. However, real-world phase modulators suffer a certain power loss when no modulation is performed and a different power loss when modulation is performed. For example, the power loss suffered when there is no phase modulation may be L1, the power loss suffered when there is π / 2 phase modulation is L2, and the power loss suffered when there is π phase modulation is L3, where L1, L2, and L3 are different from each other. This behavior is undesirable because in addition to phase modulation, the signal also undergoes amplitude modulation.

[0589] Some such phase modulators additionally require a length of several hundred microns in order to provide a sufficiently large phase shift. Unfortunately, such a length makes such phase modulators unsuitable for use in applications where several phase shifters need to be integrated on one chip. The phase modulator alone occupies most of the available space of the chip, thus limiting the number of devices that can be integrated together on the same chip.

[0590] Recognizing the above-mentioned limitations of certain phase modulators, the present inventors have developed a small footprint optical phase modulator capable of providing high modulation speeds (e.g., 6-100 MHz or greater than 1 GHz) while limiting dynamic losses. In some embodiments, the phase modulator occupies an area of ​​only 300 μm 2 Therefore, as an example, 2 A reticle with an area of ​​50 mm 2 It can accommodate as many as 15,000 phase modulators, preserving the memory for other devices.

[0591] Some embodiments relate to a nano-optoelectromechanical system (NOEMS) phase modulator having multiple suspended optical waveguides positioned adjacent to one another and forming multiple slots therebetween. The dimensions of the slots are small enough to form a slot waveguide such that a large portion (e.g., the majority) of the mode energy is confined within the slot itself. These modes are referred to herein as slot modes. Having a large portion of the mode energy within the slot allows for modulation of the effective refractive index of the mode by varying the dimensions of the slot, and consequently, the phase of the optical signal utilizing the mode. In some embodiments, phase modulation can be achieved by applying a mechanical force that varies the dimensions of the slot.

[0592] The inventors have recognized and understood that the modulation speed achievable with the NOEMS phase modulators described herein can be increased by decoupling the mechanical driver from the region where the optical modulation occurs, without significantly increasing the dynamic losses. In phase modulators where the mechanical driver is decoupled from the optical modulation region, the electrical drive signal is applied to the mechanical driver, rather than to the optical modulation region itself. This configuration eliminates the need to make the optical modulation region conductive, thus allowing for reduced doping of this region. Low doping reduces free carriers that would otherwise lead to optical absorption, thus lowering the dynamic losses.

[0593] Furthermore, decoupling the mechanical drive from the optical modulation region allows for greater modulation per unit length, which in turn allows for shorter modulation regions, which in turn allow for higher modulation speeds.

[0594] The inventors have further recognized and understood that the inclusion of multiple slots in the modulation region may allow for further shortening of the length (and thus size) of the phase modulator. Having more than one slot actually allows for a significant shortening of the length of the transition region where light is coupled into the modulation region. This results in a much more compact form factor. Thus, NOEMS phase modulators of the type described herein can have shorter modulation regions and / or shorter transition regions. Phase modulators of the type described herein can have lengths as short as 20 μm or 30 μm in some embodiments.

[0595] As described in more detail below, some embodiments relate to a phase modulator in which a trench is formed in the chip and the modulation waveguide is suspended in air and configured to be freely movable in space.

[0596] The inventors have recognized a potential drawback associated with the use of trenches that arises from the formation of a cladding / air interface. A propagating optical signal encounters a cladding / air interface (or air / cladding interface) as it enters (or exits) the trench. Unfortunately, the presence of the boundary can give rise to optical reflections, which can increase insertion loss. The inventors have realized that the adverse effects of such boundaries can be mitigated by reducing the physical spread of the optical mode in the region through the boundary. This can be accomplished in a variety of ways. For example, in some embodiments, the spread of the optical mode can be reduced by tightly confining the mode within a rib waveguide. The rib waveguide can be sized such that only a small percentage of the mode energy (e.g., less than 20%, less than 10%, or less than 5%) is outside the edges of the waveguide.

[0597] NOEMS phase modulators of the type described herein may be used in a variety of applications including, for example, telecommunications and data communications (including local area networks, metropolitan area networks, wide area networks, data center networks, satellite networks, etc.), analog applications such as radio over fiber, all-optical switches, lidar, phased arrays, coherent imaging, machine learning, and other types of artificial intelligence applications. Additionally, NOEMS modulators may be used as part of an amplitude modulator, for example, when combined with a Mach-Zehnder modulator. For example, a Mach-Zehnder modulator may be provided with a NOEMS phase modulator positioned in one or more of the arms of the Mach-Zehnder modulator. For example, several modulation formats may be possible using NOEMS phase modulators, including amplitude shift keying (ASK), quadrature amplitude modulation (QAM), phase shift keying (BPSK), quadrature phase shift keying (QPSK) and higher order QPSK, offset quadrature phase-shift keying (OQPSK), dual-polarization quadrature phase shift keying (DPQPSK), amplitude phase shift keying (APSK), etc. Additionally, NOEMS phase modulators may be used as phase correctors in applications where the phase of an optical signal tends to drift in an unpredictable manner. In some embodiments, NOEMS phase modulators of the type described herein may be used as part of a photonic processing system.For example, in some embodiments, a NOEMS phase modulator of the type described herein may be used to implement phase modulator 1-207 of FIG. 1-1, to implement a portion of variable beam splitter 1-401 of FIG. 104, to implement phase shifters 1-505, 1-507 and 1-509 of FIG. 1-5, to implement phase modulator 1-601 of FIG. 1-6, to implement a portion of amplitude modulator 1-603 of FIG. 1-6, and / or to implement a portion of amplitude modulator 1-205 of FIG. 1-2.

[0598] FIG. 6-1A is a top view schematic of a nano-opto-electro-mechanical system (NOEMS) phase modulator according to some non-limiting embodiments. The NOEMS phase modulator 6-100 includes an input waveguide 6-102, an output waveguide 6-104, an input transition region 6-140, an output transition region 6-150, a suspended multi-slot optical structure 6-120, mechanical structures 6-130 and 6-132, and mechanical actuators 6-160 and 6-162. The NOEMS phase modulator 6-100 may be fabricated using silicon photonic techniques. For example, the NOEMS phase modulator 6-100 may be fabricated in a silicon substrate, such as a bulk silicon substrate or a silicon-on-insulator (SOI) substrate. In some embodiments, the NOEMS phase modulator 6-100 may further include electronic circuitry configured to control the operation of the mechanical actuators 6-160 and 6-162. The electronic circuitry may be fabricated on the same substrate that hosts the components of Figure 6-1 A or on a separate substrate. If placed on a separate substrate, the substrates may be bonded together in any suitable manner, including 3D bonding, flip chip bonding, wire bonding, etc.

[0599] At least a portion of the NOEMS phase modulator 6-100 is formed in a trench 6-106. As described in more detail below, trenches of the type described herein may be formed by etching a portion of the cladding. In the example of FIG. 6-1A, the trench 6-106 is rectangular, although trenches of any other suitable shape may be used. In this example, the trench 6-106 has four sidewalls. Sidewalls 6-112 and 6-114 are spaced apart from one another along the z-axis (referred to herein as the propagation axis), and the other two sidewalls (not shown in FIG. 6-1A) are spaced apart from one another along the x-axis.

[0600] In some embodiments, the separation along the z-axis between the side walls 6-112 and 6-114 can be 50 μm or less, 30 μm or less, or 20 μm or less. Thus, the modulation region of the NOEMS phase modulator is much shorter than other types of phase modulators that require hundreds of μm to modulate the phase of the optical signal. The relatively short length is made possible by one or more of the following factors: First, having multiple slots improves coupling to the optical modulation region and allows for a reduction in the length of the transition region. The improved coupling can be a result of the enhanced mode symmetry of the multi-slot structure. Second, decoupling the mechanical driver from the optical modulation region allows for a larger modulation per unit length, resulting in a shorter modulation region.

[0601] During operation, an optical signal may be provided to the input waveguide 6-102. In one example, the optical signal may be a continuous wave (CW) signal. Phase modulation may be performed in the suspended multi-slot optical structure 6-120. The phase modulated optical signal may exit the NOEMS phase modulator 6-100 from the output waveguide 6-104. The transition region 6-140 may ensure lossless or near lossless optical coupling between the input waveguide 6-102 and the suspended multi-slot optical structure 6-120. Similarly, the transition region 6-150 may ensure lossless or near lossless optical coupling between the suspended multi-slot optical structure 6-120 and the output waveguide 6-104. The transition regions 6-140 and 6-150 may include tapered waveguides in some embodiments, as described in more detail below. As discussed above, the length of the transition region may be short compared to other implementations.

[0602] The input optical signal may have any suitable wavelength, including, but not limited to, O-band, E-band, S-band, C-band, or L-band wavelengths. Alternatively, the wavelength may be in the 850 nm band or the visible band. It should be understood that the NOEMS phase modulator 6-100 may be made of any suitable material, so long as the material is transparent or at least partially transparent to the wavelengths of interest and the refractive index of the core region is greater than the refractive index of the surrounding cladding. In some embodiments, the NOEMS phase modulator 6-100 may be made of silicon. For example, the input waveguide 6-102, the output waveguide 6-104, the input transition region 6-140, the output transition region 6-150, the suspended multi-slot optical structure 6-120, and the mechanical structures 6-130 and 6-132 may be made of silicon. Given silicon's relatively low optical band gap (approximately 1.12 eV), silicon may be particularly suitable for use with near-infrared wavelengths. In another example, the NOEMS phase modulator 6-100 can be made of silicon nitride or diamond. Given the relatively high optical bandgaps of silicon nitride and diamond (approximately 5 eV and approximately 5.47 eV, respectively), these materials may be particularly suitable for use with visible wavelengths. However, other materials are possible, including indium phosphide, gallium arsenide, and / or any suitable III-V or II-VI alloys.

[0603] In some embodiments, the input waveguide 6-102 and the output waveguide 6-104 may be sized to support one mode at the operating wavelength (although multimode waveguides may also be used). For example, if the NOEMS phase modulator is designed to operate at 1550 nm (although of course not all embodiments are limited in this respect), the input waveguide 6-102 and the output waveguide 6-104 may support one mode at 1550 nm. In this manner, mode confinement within the waveguide may be enhanced, thus reducing optical losses due to scattering and reflection. The waveguides 6-102 and 6-104 may be rib waveguides (e.g., having a rectangular cross section) or may have any other suitable shape.

[0604] As mentioned above, portions of the NOEMS phase modulator 6-100 may be formed in the trench 6-106 such that the waveguide in the modulation region is surrounded by air and free to move through space. A drawback of including a trench is the formation of cladding / air and air / cladding interfaces along the propagation path. Thus, the input optical signal passes through the cladding / air interface (corresponding to the sidewall 6-112) before reaching the region where modulation occurs, and through the air / cladding interface (corresponding to the sidewall 6-114) after the modulation region. These boundaries may introduce reflection losses. In some embodiments, reflection losses may be reduced by positioning the transition region 6-140 inside the trench 6-106, rather than outside (as shown in FIG. 6-1A). In this way, the mode expansion associated with the transition region occurs where the optical signal has already passed the cladding / air interface. In other words, the mode is tightly confined as it passes through the cladding / air interface, but expands in the trench using the transition region to couple to the suspended multi-slot structure 6-120. Similarly, a transition region 6-150 may also be formed within the trench 6-106, thereby spatially reconfining the mode before it reaches the sidewall 6-114.

[0605] FIG. 6-1B shows a suspended multi-slot optical structure 6-120 according to some non-limiting embodiments in more detail. In the example of FIG. 6-1B, the multi-slot optical structure 6-120 includes three waveguides (6-121, 6-122, and 6-123). A slot 6-124 separates the waveguide 6-121 from the waveguide 6-122, and a slot 6-125 separates the waveguide 6-122 from the waveguide 6-123. The width of the slots (d1 and d2) can be less than the critical width (at the operating wavelength) for forming the slot mode, so that the majority of the mode energy (e.g., more than 40%, more than 50%, more than 60%, or more than 75%) is within the slot. For example, each of d1 and d2 can be 200 nm or less, 6-150 nm or less, or 6-100 nm or less. The minimum width can be set by the photolithography resolution.

[0606] FIG. 6-1C is a plot showing an example of optical modes supported by waveguides 6-121, 6-122, and 6-123, according to some non-limiting embodiments. More specifically, the plot shows the amplitude of the mode (e.g., the electric field E x , E y Or E z or magnetic field H x , H y Or H z ) As shown, most of the total energy is confined within the slot, where the mode exhibits an amplitude peak. In some embodiments, there is more optical energy in one of the slots than in any one of the individual waveguides. In some embodiments, there is more optical energy in one of the slots than in all of the waveguides considered together. Outside the outer wall of the external waveguide, the mode energy decays (e.g., exponentially).

[0607] The widths d1 and d2 may be equal to or different from each other. The widths of the slots and the waveguides may be constant (as in FIG. 6-1B) or may vary along the z-axis. In some embodiments, the widths of the waveguides 6-121, 6-122, and 6-123 may be less than the width of the input waveguide 6-102. In some embodiments, when the operating wavelength is within the C-band, the widths of the waveguides 6-121, 6-122, and 6-123 may be within 200 nm to 400 nm, 250 nm to 350 nm, or any other suitable range, whether within or outside such ranges.

[0608] The example of FIG. 6-1B shows a suspended multi-slot optical structure 6-120 with three waveguides and two slots, but any other suitable number of waveguides and slots can be used. In other examples, the suspended multi-slot optical structure 6-120 can include five waveguides and four slots, seven waveguides and six slots, nine waveguides and eight slots, etc. In some embodiments, the structure includes an odd number of waveguides (and therefore an even number of slots) such that only symmetric modes are excited while asymmetric modes remain unexcited. The inventors have realized that enhancing the symmetry of the modes enhances coupling into the slotted structure, thus allowing for a significant reduction in the length of the transition region. However, implementations with an even number of waveguides are also possible.

[0609] As described in more detail below, phase modulation is achieved by moving the external waveguides (6-121 and 6-123 in FIG. 6-1B) along the x-axis relative to the central waveguide (6-122 in FIG. 6-1B). When the waveguide 6-121 moves along the x-axis relative to the waveguide 6-122, the width of the slot 6-124 changes and the shape of the mode supported by the structure changes accordingly. This results in a change in the effective refractive index of the mode supported by the structure and therefore phase modulation. The movement of the external waveguides can be induced using mechanical structures 6-130 and 6-132.

[0610] An example of a mechanical structure 6-130 according to some non-limiting embodiments is shown in FIG. 6-1D. The mechanical structure 6-132 (see FIG. 6-1A) may have a similar configuration. In the example of FIG. 6-1D, the mechanical structure 6-130 includes beams 6-133, 6-134, 6-135, and 6-136. The beam 6-133 connects the mechanical driver 6-160 to the beam 6-134. The beams 6-135 and 6-136 connect the beam 6-134 to an external waveguide. To limit optical losses, the beams 6-135 and 6-136 may be attached to the external waveguide in the transition regions 6-140 and 6-150, respectively, rather than the modulation region (as shown in FIG. 6-1E, described below). However, it is also possible to attach the beams 6-135 and 6-136 to an external waveguide to the modulation region. Alternatively or in addition to those shown in FIG. 6-1D, beams having different shapes, sizes and orientations may be used.

[0611] The mechanical structure 6-130 may transmit the mechanical force generated in the mechanical driver 6-160 to the waveguide 6-121, thereby moving the waveguide 6-121 relative to the waveguide 6-122. The mechanical drivers 6-160 and 6-162 may be implemented in any suitable manner. In one example, the mechanical driver may include a piezoelectric device. In one example, the mechanical driver may include conductive fingers. When a voltage is applied between adjacent fingers, the fingers may undergo acceleration, thus imparting a mechanical force to the mechanical structure. In some embodiments, the mechanical driver may be driven with an electrical signal having an encoded pattern. In this manner, the modulation imparts a pattern to the phase of the input optical signal.

[0612] It should be appreciated that because the waveguides of the suspended multi-slot optical structure 6-120 are driven by an external mechanical driver rather than directly fed by an electrical signal as in certain conventional phase modulators, the waveguides can be made less conductive, thus lowering free carrier absorption losses and, consequently, lowering dynamic losses. This differs from some conventional phase modulators in which the waveguides themselves are doped to act as heaters or carrier accumulation regions. In some embodiments, the waveguides 6-121, 6-122, and 6-123 are made of undoped or lightly doped semiconductor material (e.g., undoped silicon or doped at a concentration of 1000 .mu.m). 14 cm -3 In some embodiments, the resistivity of the material forming the waveguide may be greater than 1300 Ω cm.

[0613] FIG. 6-1E illustrates an example of a transition region 6-140 according to some non-limiting embodiments. In this implementation, the waveguide 6-122 is adjacent to (e.g., continuous with) the input waveguide 6-102. As shown, the waveguide 6-122 tapers in the transition region such that it narrows in width as it approaches the suspended multi-slot optical structure 6-120. In contrast, the waveguides 6-121 and 6-123 taper in the transition region such that it increases in width as it moves away from the suspended multi-slot optical structure 6-120. The tapered waveguides may enable adiabatic coupling between the modes of the input waveguide 6-102 and the modes of the suspended multi-slot optical structure 6-120, thereby limiting coupling losses. A similar configuration may also be used for the transition region 6-150. Due to the enhanced symmetry of the modes supported by the multi-slot structure, the transition regions 6-140 and 6-150 are much shorter than other implementations, in some embodim...

Claims

1. 1. A photonic processing device, comprising: a light source configured to generate a first plurality of optical signals having different wavelengths; a plurality of optical encoders configured to encode a plurality of input vectors into the first plurality of optical signals; an input wavelength division multiplexer configured to couple the first plurality of encoded optical signals to an input waveguide; a photonic processor coupled to the input waveguide, receiving the first plurality of encoded optical signals; and a photonic processor configured to perform a plurality of operations on the encoded first plurality of optical signals to generate a second plurality of optical signals, the plurality of operations performing a matrix multiplication of the plurality of input vectors by a matrix; an output wavelength division multiplexer coupled to the photonic processor and configured to split the second plurality of optical signals onto a plurality of output waveguides; an optical detector configured to detect the second plurality of optical signals received from the plurality of output waveguides.

2. The photonic processing device of claim 1 , wherein the optical detector is configured to detect the second plurality of optical signals using a phase-sensitive detector.

3. The light source is providing a first light to the plurality of optical encoders for use in encoding the first plurality of optical signals; providing a second light to the plurality of optical detectors for use as a local oscillator by the phase sensitive detector; the local oscillator is phase coherent with each of the first plurality of optical signals; The photonic processing device of claim 2 , wherein a first path length of the first plurality of optical signals from the light source to the plurality of optical detectors is equal to a second path length of the local oscillator from the light source to the plurality of optical detectors.

4. One optical encoder of the plurality of optical encoders comprises: encoding an absolute value of a vector component of one of the input vectors into an amplitude of each optical signal of the first plurality of optical signals; and encoding the phases of the vector components of the input vector into the phases of the respective optical signals of the first plurality of optical signals.

5. The photonic processing device of claim 1 , wherein the matrix is ​​an arbitrary unitary matrix.

6. The photonic processor includes: a first array of a first plurality of interconnected variable beam splitters (VBS) including a first plurality of optical inputs and a first plurality of optical outputs; a second array of a second plurality of interconnected VBSs including a second plurality of optical inputs and a second plurality of optical outputs; a plurality of controllable optical elements, each of the plurality of controllable optical elements coupling a single one of the first plurality of optical outputs of the first array to a respective single one of the second plurality of optical outputs of the second array; The photonic processing device of claim 1 comprising:

7. A controller, performing a singular value decomposition (SVD) of the matrix to identify a first SVD matrix, a second SVD matrix, and a third SVD matrix; controlling the first plurality of interconnect VBSs to implement the first SVD matrix; controlling the second plurality of interconnect VBSs to implement the second SVD matrix; controlling the plurality of controllable optical elements to implement the third SVD matrix, the third SVD matrix being a diagonal matrix. The photonic processing device of claim 6 , further comprising a controller configured to:

8. The photonic processing device of claim 7 , wherein the controller further comprises at least one digital-to-analog converter (DAC) for adjusting one or more parameters of the first plurality of interconnect VBSs and the second plurality of interconnect VBSs.

9. Each of the VBSs of the first plurality of interconnected VBSs and each of the VBSs of the second plurality of interconnected VBSs are associated with a respective address; and The photonic processing device of claim 8 , wherein the at least one DAC comprises a single DAC that uses the address to control a plurality of the VBSs of the first plurality of interconnect VBSs and / or the second plurality of interconnect VBSs.

10. The matrix is ​​a first matrix, and the photonic processing device performs the multiplication of a second matrix by the first matrix as (a) identifying the plurality of input vectors from each column of the second matrix; (b) selecting an input vector from the plurality of input vectors; (c) encoding the selected input vector into the first plurality of optical signals using an optical encoder of the plurality of optical encoders; (d) performing the plurality of operations on the first plurality of optical signals associated with the input vector; (e) detecting the second plurality of optical signals associated with the selected input vector; (f) storing a digital detection result based on the detected second plurality of optical signals; (g) repeating operations (b) through (f) for other input vectors of the plurality of input vectors; (h) digitally combining the digital detection results to determine a result matrix resulting from the multiplication of the second matrix by the first matrix. The photonic processing device of claim 1 , further comprising a controller configured to control the photonic processing device to perform by performing:

11. 10. The photonic processing device of claim 1, wherein one optical detector of the plurality of optical detectors includes a low pass filter configured to perform an analog summation of multiple subsequent signals associated with each output waveguide of the plurality of output waveguides.

12. 1. A method for optically performing matrix-vector multiplication, comprising: generating a first plurality of optical signals having different wavelengths; receiving digital representations of a plurality of input vectors; encoding the plurality of input vectors into the first plurality of optical signals using a plurality of optical encoders; coupling the encoded first plurality of optical signals to an input waveguide using an input wavelength division multiplexer; performing, using a photonic processor, a plurality of operations on the encoded first plurality of optical signals to generate a second plurality of optical signals, the plurality of operations comprising: performing matrix multiplication of the plurality of input vectors by a matrix; splitting the second plurality of optical signals onto a plurality of output waveguides using an output wavelength division multiplexer coupled to the photonic processor; detecting the second plurality of optical signals received from the plurality of output waveguides using a plurality of optical detectors; The method includes:

13. The method of claim 12 , wherein the detecting of the second plurality of optical signals is performed using a phase-sensitive detector.

14. Encoding at least one input vector of the plurality of input vectors comprises: encoding absolute values ​​of vector components of the at least one input vector into amplitudes of each optical signal of the first plurality of optical signals; encoding a phase of the vector components of the at least one input vector into a phase of the respective optical signal of the first plurality of optical signals; The method of claim 12, comprising:

15. The method of claim 12 , wherein the matrix is ​​an arbitrary unitary matrix.

16. The matrix-vector multiplication is one of a plurality of matrix-vector multiplications performed to perform a matrix-matrix multiplication, the matrix being a first matrix, and the matrix-matrix multiplication comprising a multiplication of a second matrix by the first matrix, the method comprising: (a) identifying the plurality of input vectors from each column of the second matrix; (b) selecting an input vector from the plurality of input vectors; (c) encoding the selected input vector into the first plurality of optical signals using one of the plurality of optical encoders; (d) performing a plurality of operations on the first plurality of optical signals associated with the input vector; (e) detecting the second plurality of optical signals associated with the selected input vector; (f) storing a digital detection result based on the detected second plurality of optical signals; (g) repeating operations (b) through (f) for other input vectors of the plurality of input vectors; (h) digitally combining the digital detection results to determine a result matrix resulting from the multiplication of the second matrix by the first matrix. The method of claim 12 further comprising:

17. 2. The photonic processing device of claim 1, wherein the first plurality of optical signals includes a first input optical signal having a first wavelength and a second input optical signal having a second wavelength different from the first wavelength, and the light source includes an array of lasers, a first laser of the array configured to generate the first input optical signal and a second laser of the array configured to generate the second input optical signal.

Images