Hybrid analog-digital matrix processor

Abstract

Techniques for computing matrix operations for arbitrarily large matrices on a limited size hybrid analog-digital matrix processor are described. Techniques for gain adjustment in a limited size hybrid analog-digital matrix processor are described that enable the system to achieve higher energy efficiency, greater physical density, and improved numerical accuracy. In some embodiments, these techniques maximize the prediction accuracy of GEMM-based convolutional neural networks using low-precision data representations.

Description

[0001] Cross-references to related applications

[0002] This application claims priority under 35 U.S. SC § 119(e) to U.S. Provisional Patent Application Serial No. 62 / 810851, entitled “GENERAL MATRIX MULTIPLICATION WITH SUB-ARRAY TILING AND MULTIPLE SCALING FOR HYBRID ANALOG-DIGITAL MATRIX PROCESSORS”, filed February 26, 2019, Attorney Dossier No. L0858.70011US00, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to a hybrid analog-to-digital processor configured to perform mathematical matrix operations. Background Technology

[0004] Deep learning, machine learning, latent variable models, neural networks, and other matrix-based differentiable programs are used to solve a wide range of problems, including natural language processing and object recognition in images. Solving these problems using deep neural networks typically requires significant processing time to perform the necessary computations. The most computationally intensive operations in solving these problems are often mathematical matrix operations, such as general matrix multiplication or multichannel convolution. A common approach to accelerating deep learning algorithms is to develop specialized hardware architectures. This is because conventional computer processors, such as central processing units (CPUs), which consist of circuits comprising hundreds of millions of transistors implementing logic gates on bits of information represented by electrical signals, are designed for general-purpose computing and are therefore not optimized for the specific patterns of data movement and computation required by the algorithms used in deep learning and other matrix-based differentiable programs. A common example of specialized hardware for deep learning is the graphics processing unit (GPU), which has a highly parallel architecture that makes it more efficient than CPUs at performing image processing and graphics operations. Following their development for graphics processing, GPUs were found to be more efficient than CPUs at other parallelizable algorithms, such as those used in neural networks and deep learning.

[0005] Deep learning using neural networks typically involves two phases: a training phase and an evaluation phase (sometimes called "inference"). Before a deep learning algorithm can meaningfully execute on a processor (e.g., to classify image or speech samples), it must first be trained during the evaluation phase. The training phase can be very time-consuming and computationally intensive. Summary of the Invention

[0006] Some embodiments relate to a hybrid analog-digital processor, comprising: circuitry including an analog processor, wherein the circuitry is configured to perform mathematical operations using a plurality of paths, wherein for each of the plurality of paths, the circuitry is configured to: determine one or more scaling factors for the path based on a parameter set representing a portion of a matrix; scale at least some parameters of the parameter set based on the one or more scaling factors to produce a scaled parameter set; program the analog processor based on the scaled parameter set; generate a plurality of input analog signals based on an input dataset; generate a plurality of output analog signals based on the plurality of input analog signals and the scaled parameter set; generate a partial output dataset based on the plurality of output analog signals; and scale the partial output dataset based on the one or more scaling factors to produce a scaled partial output dataset, wherein the circuitry is further configured to generate an accumulated output dataset by accumulating the scaled partial output datasets generated by at least two of the plurality of paths, wherein the accumulated output dataset represents the result of the mathematical operation.

[0007] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a scaled parameter set includes performing matrix and matrix multiplication based on multiple input analog signals and a scaled parameter set.

[0008] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a scaled parameter set includes performing convolution based on multiple input analog signals and a scaled parameter set.

[0009] In some embodiments, programming the analog processor based on a scaled parameter set includes setting appropriate gains or attenuations for a plurality of analog amplifiers or attenuators of the analog processor based on the scaled parameter set.

[0010] In some embodiments, the analog processor includes a photonic processor comprising a plurality of programmable photonic devices, and wherein programming the analog processor based on a scaled parameter set includes setting corresponding characteristics for the plurality of programmable photonic devices based on the scaled parameter set.

[0011] In some embodiments, the programmable photonic device includes a Mach-Zehnder interferometer, and wherein setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding optical characteristics for a plurality of Mach-Zehnder interferometers based on a scaled parameter set.

[0012] In some embodiments, the programmable photonic device includes an optical resonator, and wherein setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding optical characteristics for a plurality of optical resonators based on the scaled parameters.

[0013] In some embodiments, the circuit includes multiple digital-to-analog converters (DACs), and the multiple DACs are configured to generate multiple input analog signals based on an input dataset.

[0014] In some embodiments, the circuit includes multiple analog-to-digital converters (ADCs), and the multiple ADCs are configured to generate a partial output dataset based on multiple output analog signals.

[0015] In some embodiments, the plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

[0016] In some embodiments, accumulating scaled partial output datasets generated by at least two of a plurality of paths includes: for at least some paths, adding the scaled partial output datasets generated by the path to the scaled partial output datasets generated by the previous path.

[0017] In some embodiments, programming the simulation processor includes programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing an arbitrary matrix.

[0018] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on the singular value decomposition (SVD) of arbitrary matrices.

[0019] In some embodiments, programming a simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on the dilation of arbitrary matrices.

[0020] In some embodiments, scaling at least some of the parameter set based on one or more scaling factors includes dividing at least some of the parameter set by one or more scaling factors, and scaling a portion of the output dataset based on one or more scaling factors includes multiplying the portion of the output dataset by one or more scaling factors.

[0021] In some embodiments, for each of the plurality of paths, the circuit is also configured to determine one or more scaling factors based on the tiles of the matrix associated with that path.

[0022] In some embodiments, for each of the plurality of paths, the circuit is further configured to determine one or more scaling factors based on one or more rows of a matrix.

[0023] In some embodiments, for each of the plurality of paths, the circuit is also configured to determine one or more scaling factors based on one or more singular values ​​of the matrix.

[0024] In some embodiments, one or more scaling factors include a first scaling factor and a second scaling factor, and scaling at least some parameters of the parameter set based on one or more scaling factors includes: scaling a first subset of the parameter set based on the first scaling factor, and scaling a second subset of the parameter set based on the second scaling factor.

[0025] Some embodiments relate to a method for performing mathematical operations using multiple pathways of an analog processor, the method comprising: for each of the multiple pathways: determining one or more scaling factors for that pathway based on a parameter set representing a portion of a matrix; scaling at least some parameters of the parameter set based on the one or more scaling factors to produce a scaled parameter set; programming the analog processor based on the scaled parameter set; generating multiple input analog signals based on an input dataset; generating multiple output analog signals based on the multiple input analog signals and the scaled parameter set; generating a partial output dataset based on the multiple output analog signals; and scaling the partial output dataset based on the one or more scaling factors to produce a scaled partial output dataset; and generating an accumulated output dataset by summing the scaled partial output datasets generated by at least two of the multiple pathways, wherein the accumulated output dataset represents the result of the mathematical operation.

[0026] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a scaled parameter set includes performing matrix and matrix multiplication based on multiple input analog signals and a scaled parameter set.

[0027] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a scaled parameter set includes performing convolution based on multiple input analog signals and a scaled parameter set.

[0028] In some embodiments, programming the analog processor based on a scaled parameter set includes setting appropriate gains or attenuations for a plurality of analog amplifiers or attenuators of the analog processor based on the scaled parameter set.

[0029] In some embodiments, the analog processor includes a photonic processor comprising a plurality of programmable photonic devices, and wherein programming the analog processor based on a scaled parameter set includes setting corresponding characteristics for the plurality of programmable photonic devices based on the scaled parameter set.

[0030] In some embodiments, the programmable photonic device includes a Mach-Zehnder interferometer, and setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding optical characteristics for a plurality of Mach-Zehnder interferometers based on a scaled parameter set.

[0031] In some embodiments, a programmable photonic device includes an optical resonator, and wherein setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding optical characteristics for a plurality of optical resonators based on a scaled parameter set.

[0032] In some embodiments, generating multiple input analog signals based on an input dataset includes providing the input dataset as input to multiple digital-to-analog converters (DACs).

[0033] In some embodiments, a partial output dataset is generated based on a plurality of output analog signals, the circuit including providing the plurality of output analog signals as inputs to a plurality of analog-to-digital converters (ADCs).

[0034] In some embodiments, the plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

[0035] In some embodiments, accumulating scaled partial output datasets generated by at least two of a plurality of paths includes: for at least some paths, adding the scaled partial output dataset generated by that path to the scaled partial output dataset generated by the previous path.

[0036] In some embodiments, programming the simulation processor includes programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing an arbitrary matrix.

[0037] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on the singular value decomposition (SVD) of arbitrary matrices.

[0038] In some embodiments, programming a simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on an expansion of arbitrary matrices.

[0039] In some embodiments, scaling at least some of the parameters based on one or more scaling factors includes dividing at least some of the parameters by one or more scaling factors, and scaling a portion of the output dataset based on one or more scaling factors includes multiplying the portion of the output dataset by one or more scaling factors.

[0040] In some embodiments, determining one or more scaling factors includes determining one or more scaling factors based on tiles of a matrix associated with the path.

[0041] In some embodiments, determining one or more scaling factors includes determining one or more scaling factors based on one or more rows of a matrix.

[0042] In some embodiments, determining one or more scaling factors includes determining one or more scaling factors based on one or more singular values ​​of a matrix.

[0043] In some embodiments, one or more scaling factors include a first scaling factor and a second scaling factor, and scaling at least some parameters of the parameter set based on one or more scaling factors includes: scaling a first subset of the parameter set based on the first scaling factor, and scaling a second subset of the parameter set based on the second scaling factor.

[0044] Some embodiments relate to a hybrid analog-to-digital processor including: a digital-to-analog converter (DAC) unit including a plurality of DACs; an analog processor arranged to perform matrix and matrix multiplication and coupled to the output of the DAC unit; an analog-to-digital converter (ADC) unit including a plurality of ADCs and coupled to the output of the analog processor; a digital scaling unit coupled to the output of the ADC unit; and a digital accumulator coupled to the output of the digital scaling unit, wherein the digital accumulator includes a memory unit and a digital adder.

[0045] In some embodiments, the hybrid analog-to-digital processor further includes an amplification or attenuation unit coupled between the DAC unit and the ADC unit.

[0046] In some embodiments, the analog processor includes a photonic processor that includes a plurality of programmable photonic devices.

[0047] In some embodiments, the programmable photonic device includes a Mach-Zehnder interferometer.

[0048] In some embodiments, the programmable photonic device includes an optical resonator.

[0049] In some embodiments, the simulation processor is programmed using multiple matrices that collectively represent an arbitrary matrix.

[0050] In some embodiments, multiple matrices jointly represent an arbitrary matrix based on the singular value decomposition (SVD) of the arbitrary matrix.

[0051] In some embodiments, multiple matrices collectively represent an arbitrary matrix based on the expansion of an arbitrary matrix.

[0052] Some embodiments relate to a hybrid analog-to-digital processor configured to perform mathematical operations, including: circuitry comprising an analog processor and an analog scaling unit, wherein the circuitry is configured to: generate a plurality of input analog signals based on an input dataset; set the gain of the analog scaling unit based on a scaling factor; program the analog processor with a parameter set representing a matrix; generate a plurality of output analog signals based on the plurality of input analog signals and the parameter set; generate a plurality of amplified output analog signals by amplifying the plurality of input analog signals and / or the plurality of output analog signals using the analog scaling unit; and generate an output dataset based on the plurality of amplified output analog signals.

[0053] In some embodiments, the analog scaling unit includes an analog amplifier or an attenuator.

[0054] In some embodiments, the hybrid analog-to-digital processor is further configured to perform multipath computation based on mathematical operations, wherein the circuit is further configured to: set the gain of the analog scaling unit to a first value during a first path of the multipath computation; and set the gain of the analog scaling unit to a second value different from the first value during a second path of the multipath computation.

[0055] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing matrix and matrix multiplication based on multiple input analog signals and a parameter set.

[0056] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing convolution based on multiple input analog signals and a parameter set.

[0057] In some embodiments, the circuit includes multiple analog-to-digital converters (ADCs), and the multiple ADCs are configured to generate an output dataset based on multiple output analog signals.

[0058] In some embodiments, the plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

[0059] In some embodiments, programming the analog processor includes programming the analog processor using multiple matrices based on a parameter set, the matrices collectively representing an arbitrary matrix.

[0060] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a parameter set, these matrices collectively representing arbitrary matrices based on the singular value decomposition (SVD) of arbitrary matrices.

[0061] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on an expansion of arbitrary matrices.

[0062] In some embodiments, the circuit is also configured to determine a scaling factor based on a parameter set and an input dataset.

[0063] In some embodiments, determining the scaling factor includes determining the scaling factor based on statistical limits on the parameter set and statistical limits on the input dataset.

[0064] Some embodiments relate to a method for performing mathematical operations, the method comprising: generating a plurality of input analog signals based on an input dataset; setting the gain of an analog scaling unit based on a scaling factor; programming an analog processor using a parameter set representing a matrix; generating a plurality of output analog signals based on the plurality of input analog signals and the parameter set; amplifying the plurality of input analog signals and / or the plurality of output analog signals by using an analog scaling unit to generate a plurality of amplified output analog signals; and generating an output dataset based on the plurality of amplified output analog signals.

[0065] In some embodiments, the analog scaling unit includes an analog amplifier or an attenuator.

[0066] In some embodiments, the hybrid analog-to-digital processor is further configured to perform multipath computation based on mathematical operations, wherein the circuit is further configured to: set the gain of the analog scaling unit to a first value during a first path of the multipath computation; and set the gain of the analog scaling unit to a second value different from the first value during a second path of the multipath computation.

[0067] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing matrix and matrix multiplication based on multiple input analog signals and a parameter set.

[0068] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing convolution based on multiple input analog signals and a parameter set.

[0069] In some embodiments, the circuit includes multiple analog-to-digital converters (ADCs), and the multiple ADCs are configured to generate an output dataset based on multiple output analog signals.

[0070] In some embodiments, the plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

[0071] In some embodiments, programming the analog processor includes programming the analog processor using multiple matrices based on a parameter set, the matrices collectively representing an arbitrary matrix.

[0072] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a parameter set, these matrices collectively representing arbitrary matrices based on the singular value decomposition (SVD) of arbitrary matrices.

[0073] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on an expansion of arbitrary matrices.

[0074] In some embodiments, the method further includes determining a scaling factor based on a parameter set and an input dataset.

[0075] In some embodiments, determining the scaling factor includes determining the scaling factor based on statistical limits on the parameter set and statistical limits on the input dataset.

[0076] Some embodiments relate to a hybrid analog-digital processor, comprising: circuitry including a photonic processor, wherein the circuitry is configured to perform mathematical operations using a plurality of paths, wherein for each of the plurality of paths, the circuitry is configured to: determine one or more scaling factors for the path based on a parameter set representing a portion of a matrix; scale at least some parameters of the parameter set based on the one or more scaling factors to produce a scaled parameter set; program the photonic processor based on the scaled parameter set; generate a plurality of input optical signals based on an input dataset; generate a plurality of output optical signals based on the plurality of input optical signals and the scaled parameter set; generate a partial output dataset based on the plurality of output optical signals; and scale the partial output dataset based on the one or more scaling factors to produce a scaled partial output dataset, wherein the circuitry is further configured to generate an accumulated output dataset by accumulating the scaled partial output datasets generated by at least two of the plurality of paths, wherein the accumulated output dataset represents the result of the mathematical operation.

[0077] In some embodiments, the photonic processor includes a plurality of programmable photonic devices, and programming the analog processor based on a scaled parameter set includes setting corresponding characteristics for the plurality of programmable photonic devices based on the scaled parameter set.

[0078] In some embodiments, a programmable photonic device includes an optical resonator, and wherein setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding optical characteristics for a plurality of optical resonators based on a scaled parameter set.

[0079] In some embodiments, the optical resonator includes an optical ring or disk resonator, and wherein setting the corresponding optical characteristics for a plurality of optical resonators includes setting the corresponding resonant frequencies for the plurality of optical ring or disk resonators based on a scaled parameter set.

[0080] In some embodiments, the programmable photonic device includes a Franz-Kelder modulator, and wherein setting corresponding characteristics for a plurality of programmable photonic devices based on a scaled parameter set includes setting corresponding absorption for a plurality of Franz-Kelder modulators based on a scaled parameter set.

[0081] In some embodiments, the photonic processor includes: a first array of interconnected variable beam splitters (VBSs) including a first plurality of optical inputs and a first plurality of optical outputs; a second array of interconnected VBSs including a second plurality of optical inputs and a second plurality of optical outputs; and a plurality of controllable optical elements, each of which couples a single one of the first plurality of optical outputs of the first array to a corresponding single one of the second plurality of optical inputs of the second array.

[0082] In some embodiments, programming the photonic processor based on a scaled parameter set includes programming a plurality of controllable optical elements based on the scaled parameter set.

[0083] In some embodiments, each VBS of the first and second arrays includes a Mach-Zehnder interferometer, which includes: a first beam splitter;

[0084] A second beam splitter; and at least one phase modulator configured to modulate the phase of light in an optical mode coupled to the first and second beam splitters.

[0085] In some embodiments, each controllable optical element includes a variable attenuator and at least one phase modulator.

[0086] In some embodiments, generating multiple output optical signals based on multiple input optical signals and a scaled parameter set includes performing matrix and matrix multiplication based on multiple input optical signals and a scaled parameter set.

[0087] In some embodiments, programming the photonic processor includes programming the photonic processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing an arbitrary matrix.

[0088] Some embodiments relate to a hybrid analog-to-digital processor including: a digital-to-analog converter (DAC) unit including a plurality of DACs; a photonic processor arranged to perform matrix and matrix multiplication and coupled to the output of the DAC unit; an analog-to-digital converter (ADC) unit including a plurality of ADCs and coupled to the output of the photonic processor; a digital scaling unit coupled to the output of the ADC unit; and a digital accumulator coupled to the output of the digital scaling unit, wherein the digital accumulator includes a memory unit and a digital adder.

[0089] In some embodiments, the hybrid analog-to-digital processor further includes an optical amplification or attenuation unit coupled between the DAC unit and the ADC unit.

[0090] In some embodiments, the photonic processor includes a plurality of optical resonators.

[0091] In some embodiments, the photonic processor includes a first array of interconnected variable beam splitters (VBSs) including a first plurality of optical inputs and a first plurality of optical outputs; a second array of interconnected VBSs including a second plurality of optical inputs and a second plurality of optical outputs; and a plurality of controllable optical elements, each of which couples a single one of the first plurality of optical outputs of the first array to a corresponding single one of the second plurality of optical inputs of the second array.

[0092] In some embodiments, the photonic processor is programmed using multiple matrices that collectively represent an arbitrary matrix.

[0093] Some embodiments relate to a hybrid analog-digital processor configured to perform mathematical operations, including circuitry comprising a photonic processor and at least one amplifier, wherein the circuitry is configured to: generate a plurality of input analog signals based on an input dataset; set the corresponding gain of the at least one amplifier based on a scaling factor; program the photonic processor using a parameter set representing a matrix; generate a plurality of output optical signals based on the plurality of input optical signals and the parameter set; generate a plurality of output analog signals based on the plurality of output optical signals; generate a plurality of amplified output signals by amplifying at least one of the plurality of input optical signals, the plurality of output optical signals, and the plurality of output analog signals using the at least one amplifier; and generate an output dataset based on the plurality of amplified output signals.

[0094] In some embodiments, at least one amplifier includes an optical amplifier and an electronic amplifier, wherein amplifying at least one of a plurality of input optical signals, a plurality of output optical signals, and a plurality of output analog signals using at least one amplifier includes: amplifying the plurality of input optical signals using an optical amplifier; and amplifying the plurality of output analog signals using an electronic amplifier.

[0095] In some embodiments, at least one amplifier includes a laser, and wherein setting the gain of the amplifier includes setting the gain of the laser.

[0096] In some embodiments, the hybrid analog-to-digital processor is further configured to perform multi-path computation based on mathematical operations, wherein the circuit is further configured to: set the gain of at least one amplifier to a first value during a first path of the multi-path computation; and set the gain of at least one amplifier to a second value different from the first value during a second path of the multi-path computation.

[0097] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing matrix and matrix multiplication based on multiple input analog signals and a parameter set.

[0098] In some embodiments, generating multiple output analog signals based on multiple input analog signals and a parameter set includes performing convolution based on multiple input analog signals and a parameter set.

[0099] In some embodiments, the photonic processor includes a plurality of optical resonators.

[0100] In some embodiments, the photonic processor includes: a first array of interconnected variable beam splitters (VBSs) including a first plurality of optical inputs and a first plurality of optical outputs; a second array of interconnected VBSs including a second plurality of optical inputs and a second plurality of optical outputs; and a plurality of controllable optical elements, each of which couples a single one of the first plurality of optical outputs of the first array to a corresponding single one of the second plurality of optical inputs of the second array.

[0101] In some embodiments, programming the analog processor includes programming the analog processor using multiple matrices based on a parameter set, the matrices collectively representing an arbitrary matrix.

[0102] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a parameter set, these matrices collectively representing arbitrary matrices based on the singular value decomposition (SVD) of arbitrary matrices.

[0103] In some embodiments, programming the simulation processor using multiple matrices includes: programming the simulation processor using multiple matrices based on a scaled set of parameters, these matrices collectively representing arbitrary matrices based on an expansion of arbitrary matrices.

[0104] In some embodiments, the circuit is also configured to determine the scaling factor based on the parameter set and the input dataset.

[0105] In some embodiments, determining the scaling factor includes determining the scaling factor based on statistical limits on the parameter set and statistical limits on the input dataset.

[0106] Some embodiments relate to a method of manufacturing a hybrid analog-to-digital processor, comprising: forming a digital-to-analog converter (DAC) unit including a plurality of DACs; forming a photonic processor arranged to perform matrix and matrix multiplication and coupled to the output of the DAC unit; forming an analog-to-digital converter (ADC) unit including a plurality of ADCs and coupled to the output of the photonic processor; forming a digital scaling unit coupled to the output of the ADC unit; and forming a digital accumulator coupled to the output of the digital scaling unit, wherein the digital accumulator includes a memory unit and a digital adder.

[0107] In some embodiments, forming a photonic processor and forming a digital accumulator includes forming the photonic processor and the digital accumulator on a common chip.

[0108] In some embodiments, forming a photonic processor and forming a digital accumulator includes: forming a photonic processor on a first chip; forming a digital accumulator on a second chip; and bonding the first chip to the second chip.

[0109] In some embodiments, forming an ADC unit includes forming a plurality of n-bit ADCs, where n is less than or equal to 12. Attached Figure Description

[0110] Various aspects and embodiments of this application will be described with reference to the following accompanying drawings. It should be understood that these drawings are not necessarily drawn to scale. Items appearing in multiple drawings are indicated by the same reference numerals in all the drawings in which they appear.

[0111] Figure 1-1A This is a representation of matrix multiplication operations based on tile-based computation according to some embodiments.

[0112] Figure 1-1B How to calculate is described according to some embodiments. Figure 1-1A A subpart of the matrix.

[0113] Figure 1-1C This is a heatmap showing the numerical distribution of the input matrix according to some embodiments.

[0114] Figure 1-2A This is a block diagram illustrating a hybrid analog-digital matrix processor according to some embodiments.

[0115] Figure 1-2B This is a diagram illustrating matrix multiplication according to some embodiments.

[0116] Figure 1-2C This is a diagram illustrating matrix multiplication performed using tiling according to some embodiments.

[0117] Figure 1-3This is a flowchart illustrating a method for processing information using a hybrid analog-digital matrix processor according to some embodiments.

[0118] Figure 1-4A This is a diagram illustrating matrix multiplication performed using tiling and scaling based on tiling factors according to some embodiments.

[0119] Figure 1-4B This is a diagram illustrating matrix multiplication performed using tiling and scaling based on tiling row factors according to some embodiments.

[0120] Figure 1-4C This is a diagram illustrating another matrix multiplication performed according to some embodiments using tiling and scaling based on tiling factors.

[0121] Figure 1-4D This is a diagram illustrating another matrix multiplication performed according to some embodiments using tiling and scaling based on tiling row factors.

[0122] Figure 1-5A This is a diagram illustrating a matrix factorized based on singular value decomposition according to some embodiments.

[0123] Figure 1-5B This is a block diagram illustrating an example of a digital accumulator that receives n vector elements according to some embodiments.

[0124] Figure 1-6A This is a plot showing the output signal distribution related to the dynamic range of an analog processor according to some embodiments.

[0125] Figure 1-6B This is a diagram illustrating the effect of varying the gain on the digital representation of data digitized from an analog signal according to some embodiments.

[0126] Figure 1-7A This is a block diagram illustrating another hybrid analog-digital matrix processor according to some embodiments.

[0127] Figure 1-7B This is a block diagram illustrating multiple simulated scaling units according to some embodiments.

[0128] Figure 1-8 This is a flowchart illustrating another method for processing information using a hybrid analog-digital matrix processor, according to some embodiments.

[0129] Figure 1-9A This is a plot showing the functional relationship between the computational accuracy and gain of a neural network according to some embodiments.

[0130] Figure 1-9BThis is a plot showing the functional relationship between mean squared error and gain in the components of a neural network computation according to some embodiments.

[0131] Figure 1-10 This is a block diagram illustrating another hybrid analog-digital matrix processor according to some embodiments.

[0132] Figure 1-11 This is a flowchart illustrating another method for processing information using a hybrid analog-digital matrix processor, according to some embodiments.

[0133] Figure 1-12 This is a block diagram illustrating a photonic hybrid analog-digital matrix processor according to some embodiments.

[0134] Figure 2-1 This is a schematic diagram of a photon processing system according to some embodiments.

[0135] Figure 2-2 This is a schematic diagram of an optical encoder according to some embodiments.

[0136] Figure 2-3 This is a schematic diagram of a photonic processor according to some embodiments.

[0137] Figure 2-4 This is a schematic diagram of an interconnected variable beam splitter array according to some embodiments.

[0138] Figure 2-5 This is a schematic diagram of a variable beam splitter according to some embodiments.

[0139] Figure 2-6 This is a schematic diagram of an embodiment of attenuation and phase shift schemes that can be used to represent a diagonal matrix acting on n optical modes, according to some embodiments.

[0140] Figure 2-7 This is a schematic diagram of an optical attenuator according to some embodiments.

[0141] Figure 2-8 This is a schematic diagram of an optical power tree according to some embodiments.

[0142] Figure 2-9 This is a schematic diagram of an optical receiver according to some embodiments and some non-limiting embodiments.

[0143] Figure 2-10 This is a schematic diagram of a zero-difference detector according to some embodiments and some non-limiting embodiments.

[0144] Figure 2-11 This is a schematic diagram of a folded photon processing system according to some embodiments. Detailed Implementation

[0145] one, Overview

[0146] With the recent surge in deep learning applications, there is a growing demand for digital accelerators specifically designed for high-performance computing in tensor algebra operations, such as convolution or generalized matrix multiplication (GEMM). Typically, digital processor architectures, including graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs), have been used as accelerators for such applications.

[0147] The inventors have recognized that conventional digital accelerators have a major drawback—they are very power-intensive. Consider, for example, a digital accelerator programmed to perform large GEMM operations. Such accelerators rely on a large number of transistors operating at extremely high clock frequencies to compute the required multiplications. For decades, transistor size has shrunk exponentially, resulting in higher performance and lower power consumption—this is what experts commonly refer to as Moore's Law. However, in recent years, this trend has begun to slow. Primarily due to leakage current between extremely small transistors, it has become increasingly difficult to continue scaling performance without sacrificing power consumption.

[0148] Recognizing these limitations, the inventors have developed novel accelerator architectures that improve scalability in terms of performance and energy efficiency. The accelerators developed by the inventors combine the flexibility and programmability of digital controllers with the energy efficiency and inherent speed of analog circuits. These accelerators are referred to herein as hybrid analog-digital processors.

[0149] The hybrid analog-digital processors of the type described herein are particularly well-suited for computational mathematical operations. One such mathematical operation is generalized matrix multiplication (GEMM), and another is convolution, although embodiments of this application are not limited to any particular operation.

[0150] GEMM operations can be expressed as follows:

[0151] D = α(A × B) + βC

[0152] Where α and β are scalars (independent numbers), and A, B, C, and D are matrices. A and B have the same internal dimensions, therefore (A × B) is a matrix multiplication. Some mathematical operations can be decomposed using GEMM operations, thus adapting the accelerator developed by the inventors to various environments.

[0153] Some specific applications, such as deep artificial neural networks, require computations involving massive amounts of data, typically represented as matrices or tensors. One approach to performing operations on large amounts of data is to use a massive hybrid analog-digital processor. For example, to perform linear operations involving matrices or tensors with tens of thousands of rows and columns, a hybrid analog-digital processor with tens of thousands of analog elements can be used.

[0154] However, the inventors have recognized that this approach can be costly because designing a processor with so many analog components may require more chip space or more power than currently available. Furthermore, the inventors have recognized that some applications involve heterogeneous workloads, with some matrix operations having very large dimensions and others having smaller dimensions, resulting in low overall utilization of a processor with sufficient components for maximum computation. Therefore, some embodiments relate to hybrid analog-digital processors capable of processing large amounts of data using a relatively small number of analog components. For example, in some embodiments, a hybrid analog-digital processor with only a few hundred analog components can be used to perform matrix operations involving matrices with thousands of rows and columns (it should be understood that these numbers are provided by way of example only, as different numbers of analog components can be used).

[0155] In some embodiments, matrix operations can be performed by passing multiple passes through a hybrid analog-digital processor. As each pass passes through the processor, the operation is performed on a sub-part of the matrix (referred to herein as a "tile"). Figure 1-1A and 1-1B This demonstrates how to perform matrix multiplication using tiles. First, refer to... Figure 1-1A Matrix C is obtained by multiplying matrix A by matrix B. Matrix A has 1 rows and K columns, and matrix B has K rows and J columns. Therefore, matrix C has 1 rows and J columns. A1 and A2 represent tiles of matrix A. For clarity, in this example, matrix A has been divided into four tiles, although in some embodiments, matrix A may be divided into dozens, hundreds, or thousands of tiles or more. Similarly, in this example, matrix B has been divided into four non-square tiles, although in some embodiments, matrix B may also be divided into dozens, hundreds, or thousands of tiles or more. Each tile A1 and B1 may include N rows and M columns (although tiles of the type described herein need not be square), where N≤1, N≤K, N≤J, M≤1, M≤K, and M≤J. In this example, matrix B is divided into tile rows B1 and B2. Each tile row includes N rows and J columns.

[0156] Each tile row of matrix C is obtained by multiplying the corresponding tile row of matrix A with the corresponding tile row of matrix B and then summing the results. Each multiplication is performed as a path through a hybrid analog-digital processor. Figure 1-1B As shown, for example, tile A1 is multiplied by tile row B1 at the first path through the hybrid analog-to-digital processor, and tile A2 is multiplied by tile row B2 at the second path. The results are then accumulated, such that:

[0157] C1 = A1 × B1 + A2 × B2

[0158] This example involves the summation of two partial results. However, in other embodiments, there may be more partial results, depending on the tile dimension relative to the dimension of the matrix to be multiplied. For GEMM applications, such as those designed for prediction, where the same “weight” matrix (A) is reused with many different batches of data (B), this tile sorting may be beneficial for spatial or temporal localization and reducing the movement of weight data.

[0159] The inventors have also developed a special category of analog processors that utilize optical signals. The inventors have realized that, because optical signals travel at the speed of light in the medium through which light travels, the limitations on delay are far less than those in electrical systems. Furthermore, by increasing the distance the optical signal travels, almost no power consumption is consumed, thus opening up new topologies and processor layouts that are not feasible using electrical signals. Therefore, light-based processors, referred to herein as "photonic processors," can achieve better speed and efficiency performance than conventional electronic processors. It should be understood that the photonic processor is only one possible implementation of the accelerator described herein, and other types of hybrid analog-digital processors can also be used. Examples of such hybrid analog-digital processors will be provided in detail below.

[0160] The inventors also recognized that, in some cases, the matrix representing the input data (B) and / or the "weight" matrix (A) can exhibit large variations in their magnitude values. For example, consider... Figure 1-1C The matrix is ​​represented by darker areas indicating clusters of larger amplitude values ​​and brighter areas indicating clusters of smaller amplitude values. Entry values ​​with large variations in numerical amplitude present significant challenges due to the limited dynamic range of digitizing analog signals, as is done for digital representations in mixed analog-to-digital processors. Smaller amplitude values ​​are rounded to the smallest representable non-zero value, meaning that information in the lower bits can be limited by input quantization noise. Simply amplifying the signal to read the lower bits can produce noisy results.

[0161] The inventors have developed a technique to overcome the aforementioned challenges. The technique developed by the inventors involves normalizing entries in a weight matrix, or entries in an input dataset, to match the dynamic range of the system. Normalization may involve scaling different parts of the matrix by different amounts. Normalization can be tile-based, row-based, tile-based, or in any other suitable manner.

[0162] However, when different parts of a matrix are scaled by different amounts, a problem arises—the values ​​calculated from different matrix parts may no longer be directly summed. To address this issue, inverse scaling is performed in some embodiments. Once scaled back, the results for different matrix parts are restored to the original data range, thus allowing direct summation.

[0163] The inventors also recognized that using a large number of bits to represent digital values ​​while providing high representativeness accuracy can lead to a significant increase in power consumption. Consider, for example, an analog-to-digital converter (ADC). Among other factors, the power consumption of an ADC depends on the number of bits representing the value. The energy required by some ADCs increases exponentially with the number of output bits. Essentially, there is a trade-off between representativeness accuracy and power consumption.

[0164] Conventional digital processors utilize binary number representations such as floating-point, integer, fixed-point, and inflection-point. To achieve higher performance for certain applications, non-standard binary number representations have been developed, including non-linear effective bit mappings (such as logarithmic number representations), binary random numbers, and entropy coding (such as positis, unums, and cone floating-point). Rounding and overflow / underflow using such representations can lead to inconsistencies between the results of digital GEMMs and true arithmetic results, relative to some true matrix multiplications. In floating-point (FP) representations (such as the standard IEEE 754 "binary 64," also known as double precision or "FP64"), this loss of precision is largely negligible. The division of mantissa, exponent, and exponent bias can also be modified relative to the IEEE 754 standard. For example, a 16-bit Brain floating-point, or "bfloat16," representation uses 8 bits for the exponent instead of the 5 bits used in the IEEE standard "half" precision (FP16). To achieve even higher performance and improve energy efficiency, many accelerators perform computations using lower-precision representations (such as 16-bit floating-point (FP16)) and may use mixed-precision representations. For example, NVIDIA's Volta generation tensor kernel computes 4x4 matrices A and B in "half" precision (FP16), and computes GEMM C in half precision or "single" precision (FP32).

[0165] The inventors have recognized that low-precision (e.g., 16 bits or less, 12 bits or less, 10 bits or less) fixed-point binary representations are far less demanding in terms of chip area and power requirements than floating-point binary representations. Furthermore, not all applications require the results of mathematical operations to be represented in full precision. For example, deep learning applications often tolerate significantly lower precision compared to other applications in numerical linear algebra. However, using low-precision representations presents significant challenges because it reduces the precision of the numbers that can be represented. Worse still, reducing the precision of an ADC also reduces its dynamic range. Therefore, there is a trade-off between representing the full dynamic range of a signal and accurately representing smaller output values. When converting a signal from the analog domain to the digital domain, a low-precision ADC may attempt to set the most significant bit of the output. However, this may result in the ADC setting the most significant bit to zero, meaning the ADC is attempting to read a range of signals that contains no meaningful information. The computational characteristics in neural network models further exacerbate this problem—conditions change over time, for example, from one layer of the network to another.

[0166] The inventors have developed accelerator architectures designed to operate using low-precision fixed-point binary representations, thereby limiting chip area and power consumption while providing sufficient accuracy for a variety of practical applications, including deep learning applications. These accelerator architectures can be dynamically reconfigured to account for the dynamic changes in neural network models. The accelerator architectures developed by the inventors are designed to be a range or "window" of signals that a shift ADC can read. The shift readout window allows the ADC to focus the conversion on a portion of the signal that may contain the most meaningful information, or at least some meaningful information.

[0167] In some embodiments, shifting the readout window in this manner may involve using an analog scaling unit (e.g., an amplifier or attenuator). Adjusting the gain of the analog scaling unit causes the readout window to shift, which in turn allows for more accurate readings. For example, considering an 8-bit ADC, setting the gain to 2 causes the readout window to shift 1 bit towards the least significant bit, while setting the gain to 1 / 2 causes the readout window to shift 1 bit towards the most significant bit. In some embodiments, dynamically adjusting the position of the readout window allows the accelerator to reconfigure itself based on the scale of the problem at hand. For example, dynamically changing the position of the readout window between different layers of a neural network model may be desirable to account for changes in the distribution of output data that may occur layer by layer, such as in combination with Figure 1-9B Detailed description.

[0168] two, Matrix tiling

[0169] In some embodiments, matrix operations can be performed by having multiple paths pass through a hybrid analog-digital processor. In each path, the processor performs operations on the tiles of the matrix. Figure 1-2AThis is a block diagram illustrating an example of a hybrid analog-to-digital matrix processor implementing a tiling technique, according to some embodiments. The hybrid analog-to-digital matrix processor 1-100 (also referred to herein as accelerator 1-100) includes a digital-to-analog converter (DAC) unit 1-102, an analog processor 1-104, an analog-to-digital converter (ADC) unit 1-106, a digital scaling unit 1-108, a digital accumulator 1-110, and a controller 1-120. The controller 1-120 controls the operations of the accelerator 1-100. For example, the controller 1-120 may include software instructions that, when executed, cause the accelerator to perform mathematical operations. Figure 1-2A The components and other optional components are collectively referred to as "circuit" in this document. Figure 1-2A The components can be formed on a common chip or on different chips that are bonded together.

[0170] As described in further detail below, accelerator 1-100 is configured to perform mathematical operations by executing several pathways through analog processor 1-104. At each pathway through analog processor 1-104, accelerator 1-100 uses analog processor 1-104 to compute a partial result. The partial results are then summed together using digital accumulator 1-110.

[0171] The analog processor 1-104 can be implemented using analog components in any of a variety of ways. In some embodiments, the analog processor 1-104 is configured to perform linear operations, such as matrix-vector multiplication or matrix-matrix multiplication. Therefore, the processor 1-104 may include multiple analog components designed to perform analog multiplication or other mathematical operations.

[0172] DAC units 1-102 include multiple DACs. Any suitable type of DAC can be used, including, for example, resistive ladder DACs, switched-capacitor DACs, switched-resistor DACs, binary weighted DACs, thermometer-encoded DACs, successive approximation DACs, oversampling DACs, interpolation DACs, hybrid DACs, etc. Similarly, ADC units 1-106 include multiple ADCs. Any suitable type of ADC can be used, including, for example, parallel comparator ADCs, flash ADCs, successive approximation ADCs, Wilkinson ADCs, integrating ADCs, sigma-delta ADCs, pipelined ADCs, integrating ADCs, cyclic ADCs, time-interleaved ADCs, etc.

[0173] Figure 1-2BThis is a representation of matrix-matrix multiplication that can be performed using accelerators 1-100. In some embodiments, matrix A represents the weight matrix, and matrix B represents the input matrix encoded into the analog processor with respect to the input data. In other embodiments, matrix A represents the input matrix, and matrix B represents the weight matrix. In either case, matrix C represents the output data. Entries of matrix A (also referred to herein as “parameters”) can be stored in the accelerator’s memory, and entries of matrix B can be provided to the accelerator as input. C is computed by applying the weight matrix A to the input matrix B. More specifically, matrix C is computed by multiplying matrix A by matrix B. For clarity, matrix A is shown as a 4x4 matrix, matrix B as a 4x3 matrix, and matrix C as a 4x3 matrix. Of course, matrices can have any suitable number of rows and columns. In some embodiments, matrices can have hundreds, thousands, tens of thousands, or more rows and columns. In some embodiments, A and B can be N-dimensional tensors, such that matrix operations will be performed on multiple instances of matrices of the same size. In some embodiments, the number of columns in A is equal to the number of rows in B, thus allowing matrices to be multiplied together. For example, matrix A might have I rows and K columns, and matrix B might have K rows and J columns. Therefore, matrix C has I rows and J columns. This relationship can be expressed as follows:

[0174] dim(A)=(I,K)&dim(B)=(K,J)&dim(C)=(I,J)

[0175] The `dim()` operator, when applied to a matrix, returns the number of rows and columns of the matrix. The number of entries in matrix A is a. ik Where i = 1, 2, ..., I, and k = 1, 2, ..., K. The number of entries in matrix B is b. kj ,

[0176] Where j = 1, 2, ..., J. The entries of matrix C are c ij .

[0177] Multiplying the entirety of matrix A by the entirety of matrix B using a single path requires a large analog processor, especially when I, K, and J are large. Designing a large processor may be impractical because it would likely require more chip space than currently available and could result in significant power consumption. In some embodiments, this is performed using a relatively small analog processor. Figure 1-2B The operations might be more practical. In these embodiments, instead of directly multiplying the matrices as a whole, the multiplication is decomposed into tiles. Figure 1-2CIn the depiction, for example, matrix A is divided into four tiles—A1, A2, A3, and A4. In this particular example, N = 2—each tile of A has two rows and two columns (although other values ​​of N can also be used, as long as N ≤ 1, N ≤ J, and N ≤ K). Matrix B is divided by tile rows B1 and B2, and matrix C is divided by tile rows C1 and C2. The tile rows of matrix C are given by the following expression:

[0178] C1 = A1 × B1 + A2 × B2

[0179] C2 = A3 × B1 + A4 × B2

[0180] Multiplication A1 × B1 can be executed using the first path of the analog processor, multiplication A2 × B2 can be executed using the second path of the analog processor, and the results can then be summed to obtain C1. Similarly, multiplication A3 × B1 can be executed using the third path of the analog processor, multiplication A4 × B2 can be executed using the fourth path of the analog processor, and the results can then be summed to obtain C2.

[0181] Figure 1-3 This is a flowchart illustrating examples of methods for performing mathematical operations according to some embodiments. Methods 1-300 can be executed using accelerators 1-100. Figure 1-2A (Although any other suitable accelerator may be used. It should be understood that the steps of Method 1-300 can be followed...) Figure 1-3 The operations can be performed in the order shown, or in any other suitable order. Specifically, the order of operations can be chosen to utilize spatial and / or temporal positioning. Method 1 consists of over 300 loops, where each loop corresponds to a path through the analog processor. Figure 1-2A The p-th iteration is shown, where p = 1, 2, ..., P.

[0182] In step 1-301, controller 1-120 determines one or more scaling factors for the current path. As described above, in conjunction with... Figure 1-1CIn some embodiments, matrices may exhibit significant magnitude variations across their entries, making it possible to normalize the entries of matrix A or matrix B to match the dynamic range of the analog processor. The determination in steps 1-301 may involve determining the value of a scaling factor that scales the input dataset to match the dynamic range of the ADC unit. This determination can be performed in any of a variety of ways. In some embodiments, this determination may be based on parameters of matrix A. In some embodiments, this determination may be based on parameters of the current tile or other tiles of matrix A. In this case, it may be advantageous to reorder matrix A so that entries with similar dynamic ranges fall into the same tile. The same applies to matrix B. In some embodiments, the dimension of the tiles of matrix A may be chosen depending on the desired granularity of scaling the matrix. For example, if the matrix is ​​expected to exhibit dramatic variations across its values, a smaller tile dimension may be chosen, providing finer resolution. The disadvantage is the need for more paths. In contrast, if matrix B is expected to exhibit relatively slow variations across its values, a larger tile dimension may be chosen, limiting the number of paths.

[0183] The scaling factor can be determined in any suitable manner. In one example, the scaling factor is set to the average value of the entries in the current tile. In another example, the scaling factor is set to the highest value of the entries in the current tile. In yet another example, the scaling factor is set to the lowest value of the entries in the current tile. In yet another example, the scaling factor is set to the maximum absolute value of the rows in the current tile.

[0184] In some embodiments, matrix A can be factored or decomposed in a multiplication matrix. The entries of these matrices can be such that combining (e.g., multiplying) these matrices together yields the original matrix A. Examples of matrix decompositions that can be applied to matrix A include singular value decomposition (SVD), upper-lower decomposition, dilation (e.g., unitary dilation), rank decomposition, Cholesky decomposition, QR decomposition, interpolation decomposition, spectral decomposition, extreme decomposition, etc. In some embodiments, a scaling factor is determined based on parameters of one of the matrices produced by the decomposition of matrix A.

[0185] In one example, matrix A is decomposed based on SVD, and a scaling factor is determined based on one or more singular values ​​of matrix A. For example, the scaling factor could be set to the highest singular value of the current tile, such as the highest absolute value among the singular values ​​of the current tile. In some embodiments:

[0186] t p =max{|σ i |}such that i=1,2,…,N and∑ p =diag[σ i ]

[0187] Where t p ∑ represents the scaling factor of tile p. p Let represent the diagonal matrix associated with tile p, and σ i Represents matrix ∑ p The diagonal value (i.e., the singular value of the p-th tile of A). Quantity | σ i | represents σ i The absolute value of the scaling factor. In another example, the scaling factor is set to the average of the singular values ​​of the current tile. In yet another example, the scaling factor is set to the lowest of the singular values ​​of the current tile.

[0188] For the purpose of determining the scaling factor, matrix A does not need to be divided into tiles. In some embodiments, the scaling factor may be determined based on the rows of tiles in matrix A (tile rows), multiple tiles in matrix A, rows of matrix A, one or more columns of matrix A, or any other suitable portion of matrix A. In some embodiments, the scaling factor may be set to the same value for more than one tile of matrix A.

[0189] In steps 1-302, controller 1-120 scales at least some parameters of matrix A based on the scaling factor determined in steps 1-301. In some embodiments, controller 1-120 scales the parameters (or at least some parameters) of the current tile of matrix A based on the scaling factor. For example, controller 1-120 may scale a first subset of parameters (e.g., one or more rows and / or one or more columns of matrix A) based on a first scaling factor, and may scale a second subset of parameters based on a second scaling factor. Figure 1-4A In the example, the tile parameters are scaled based on a scaling factor determined for that particular tile. Scaling factor t1 indicates the scaling factor determined for tile A1, scaling factor t2 indicates the scaling factor determined for tile A2, scaling factor t3 indicates the scaling factor determined for tile A3, and scaling factor t4 indicates the scaling factor determined for tile A4. In this example, the entries for tile A1 are divided by t1, the entries for tile A2 by t2, the entries for tile A3 by t3, and the entries for tile A4 by t4 (although operations other than division are also possible, including multiplication).

[0190] exist Figure 1-4B In the example, the parameters of the tile rows of matrix A are scaled based on a scaling factor determined for that particular tile row. The scaling factor tr1 indicates the scaling factor determined for tiles A1 and A2, and the scaling factor tr2 indicates the scaling factor determined for tiles A3 and A4. In this example, the entries for the tile rows consisting of tiles A1 and A2 are divided by tr1, and the entries for the tile rows consisting of tiles A3 and A4 are divided by tr2.

[0191] In some embodiments, instead of scaling the parameters of matrix A, controllers 1-120 may scale the entries of matrix B, the input dataset. These entries may be scaled in the digital domain or the analog domain (after passing through DAC units 1-102). When done in the analog domain, scaling may involve attenuating or amplifying the input analog signal based on a scaling factor.

[0192] exist Figure 1-4C In the example, the parameters of the tile rows of matrix B are scaled based on a scaling factor determined for the corresponding tile of matrix A. Here, scaling factor t1 indicates the scaling factor determined for tile A1, and scaling factor t2 indicates the scaling factor determined for tile A2. In this example, the entries of tile row B1 are divided by t1, and the entries of tile row B2 are divided by t2.

[0193] In steps 1-304, controller 1-120 programs analog processor 1-104 based on the parameters of the current tile of matrix A. In at least some embodiments, the parameters of matrix A have been scaled in step 1-302, and controller 1-120 programs analog processor 1-104 based on the scaled parameters of the current tile of matrix A. (Refer to back) Figure 1-2A For example, controller 1-120 programs analog processor 1-104 based on a11 / t1, a12 / t1, a21 / t1, and a22 / t1. Programming the analog processor may involve setting certain characteristics of the analog processor based on parameters. In one example, analog processor 1-104 includes multiple electronic amplifiers (e.g., voltage amplifiers, current amplifiers, power amplifiers, transimpedance amplifiers, transconductance amplifiers, operational amplifiers, or transistor amplifiers, etc.), and programming the analog processor involves setting the gain of the electronic amplifiers based on the parameters of tile A1. In another example, analog processor 1-104 includes multiple electronic attenuators (e.g., voltage attenuators, current attenuators, or power attenuators, etc.), and programming the analog processor involves setting the attenuation of the electronic attenuators based on the parameters of tile A1. In yet another example, analog processor 1-104 includes multiple electronic phase shifters, and programming the analog processor involves setting the phase shift of the electronic phase shifters based on the parameters of tile A1. In another example, the analog processor includes an array of memory devices, such as flash memory or ReRAM, typically laid out in a crossbar configuration. Programming the analog processor involves setting the conductance (or resistance) of each memory cell.

[0194] As described in further detail below, in some embodiments, the analog processor 1-104 is implemented using a photonic processor. In some such embodiments, the analog processor 1-104 includes multiple optical amplifiers (e.g., semiconductor optical amplifiers or other solid-state amplifiers, rare-earth-doped optical amplifiers (such as erbium-doped optical amplifiers), Raman amplifiers, or optical parametric amplifiers, etc.), and programming the analog processor involves setting the gain of the optical amplifiers based on the parameters of tile A1. In another example, the analog processor 1-104 includes multiple optical attenuators (e.g., Mach-Zehnder interferometers, leaky waveguides, loopback attenuators, liquid crystal attenuators, saturable absorbers or Franz-Kedish absorbers, metal-oxide-semiconductor capacitor modulators, carrier depletion-type electro-optic modulators, etc.), and programming the analog processor involves setting the attenuation of the optical attenuators based on the parameters of tile A1. In yet another example, the analog processor 1-104 includes multiple optical phase shifters (e.g., thermo-optical phase shifters, acousto-optic phase shifters, electro-optic phase shifters, plasmon dispersive phase shifters, Pockels-effect phase shifters, nano-optomechanical phase shifters, or nonlinear phase modulators, etc.), and programming the analog processor involves setting the phase shift of the optical phase shifters based on the parameters of tile A1. In yet another example, the analog processor 1-104 includes multiple optical resonators (e.g., ring resonators and disk resonators, photonic crystals, ridge resonators, etc.), and programming the analog processor involves setting the resonant frequencies of the optical resonators based on the parameters of tile A1.

[0195] In step 1-306, accelerator 1-100 receives the input dataset. (Refer to previous section) Figure 1-2C For example, the accelerator may receive entries for tile row B1 in one path and entries for tile row B2 in another path. In some embodiments, controllers 1-120 simultaneously receive and store entries for matrix B, but provide one tile row to DAC units 1-102 at a time (e.g., per path). These entries may represent real numbers. In some embodiments, entries may represent positive real numbers or signed real numbers (both positive and negative).

[0196] In step 1-308, accelerator 1-100 generates multiple input analog signals based on the input dataset received in step 1-302. (Refer to previous step) Figure 1-2A The generation of step 1-304 involves using DAC unit 1-102 to convert the input dataset into an analog signal.

[0197] In steps 1-310, analog processor 1-104 generates multiple output analog signals based on multiple input analog signals and a set of parameters programmed in steps 1-301. In some embodiments, this may involve performing matrix-matrix multiplication. The result is the multiplication of the multiple input analog signals with the respective parameters of the current tile. For the p-th path, the result is expressed as follows:

[0198]

[0199] Where A p B is the p-th tile of matrix A. p It is the p-th tile row of matrix B, t p It is the p-th scaling factor. In other embodiments, generating multiple output analog signals involves convolution based on multiple input analog signals and a parameter set.

[0200] Analog processors 1-104 may include any suitable analog components designed to perform matrix multiplication or convolution. As an example, analog processors 1-104 may include analog mixers. Each mixer mixes an input analog signal with an analog signal encoded (e.g., modulated) based on corresponding parameters of matrix A. As another example, analog processors 1-104 may include amplitude modulators. As another example, analog processors 1-104 may include current control circuitry. Each modulator modulates the input analog signal by an amount obtained based on the corresponding parameters of matrix A. As another example, analog processors 1-104 may include amplifiers. Each amplifier amplifies the input analog signal by a gain obtained based on the corresponding parameters of matrix A. As another example, analog processors 1-104 may include attenuators. Each attenuator attenuates the input analog signal by an attenuation obtained based on the corresponding parameters of matrix A. Analog processors 1-104 may be implemented using any suitable techniques, including, for example, complementary metal-oxide-semiconductor (CMOS) components, radio frequency (RF) components, and microwave components, although other techniques may be used. In some embodiments, the analog processor 1-104 includes a photonic processor, and the multiplication is performed in the optical domain. In some embodiments, the analog processor 1-04 includes a combination of photonic and analog electronic components, and may perform some multiplications in the optical domain while performing other multiplications or additions in the electronic domain. Examples of photonic processors will be described in further detail below.

[0201] In some embodiments, instead of performing operations based on the input analog signal and a scaled parameter set (e.g., matrix multiplication or convolution), steps 1-310 may involve performing operations based on the scaled input analog signal and unscaled parameters (e.g., combining...). Figure 1-4C and 1-4D(As stated above). In any case, the result was... In some embodiments, scaling of matrix B can be performed based on column tiles. That is, different column tiles of matrix B are scaled by different factors.

[0202] In some embodiments, simulated multiplication may involve matrix factorization. The entries of these matrices can be such that combining (e.g., multiplying) the matrices together yields the original matrix A. In some such embodiments, the simulation processor 1-104 can be designed to perform a transformation on the input dataset based on each matrix in which matrix A has been factored. Specific examples of matrix factorization are provided above. SVD is one such example. Figure 1-5A As shown, controller 1-120 can decompose matrix A into a matrix composed of three components: Where U and V are unitary IxI and KxK matrices, respectively. and ), and ∑ is an IxK diagonal matrix with entries of real or complex values. In some embodiments, the values ​​of the diagonal singular values ​​can be further normalized such that the maximum absolute value of the singular values ​​is 1. Values ​​σ1, σ2, ..., σ K This represents singular values. The analog processor 1-104 may include analog hardware designed to transform the input analog signal according to each of the matrices U, Σ, and V.

[0203] Reference Back Figure 1-3 In steps 1-312, accelerator 1-100 generates a partial output dataset based on the multiple output analog signals obtained in steps 1-310. In some embodiments, the generation in step 1-312 involves passing the output analog signals through ADC unit 1-106.

[0204] In steps 1-314, accelerator 1-100 scales a portion of the output dataset based on a scaling factor to produce a scaled partial output dataset. The scaling in steps 1-314 may involve passing the partial output dataset through digital scaling units 1-108. In some embodiments, digital scaling units 1-108 may be configured to apply the inverse of the operation performed in steps 1-308. For example, in at least some embodiments where steps 1-308 involve division, steps 1-314 may involve multiplication. Similarly, in at least some embodiments where steps 1-308 involve multiplication, steps 1-314 may involve division. This restores the entries of the output matrix to their original range.

[0205] In some embodiments, the scaling in steps 1-314 can be based on the same scaling factor used for the scaling in steps 1-302 (although the inverse operation can be applied). Therefore, if the scaling in steps 1-302 involves division by t... pThen the scaling in steps 1-314 involves multiplying by t. p In other embodiments, the scaling in steps 1-314 may involve an additional factor. For example, if the scaling in steps 1-308 involves division by t... p Then the scaling in steps 1-314 may involve multiplying by t. p / r, where factor r can be based on all scaling factors t p Calculation. In one example, the factor r is calculated as follows:

[0206] r = f({t p :p=1,2,…,K / / N)

[0207] The function f(·) can represent any suitable function, such as max(·), min(·), or mean(·). By factor t p Scaling performed by / r has the effect of unifying the range of data by scaling factor.

[0208] In steps 1-316, the scaled partial output dataset generated in steps 1-314 is accumulated. In each pass, accumulation may involve adding the scaled partial output dataset generated by the current pass to the result of the previous accumulation. Therefore, each pass adds the current result to the accumulated result. The accumulation in steps 1-320 can be performed using digital accumulators 1-110. According to some embodiments, examples of accumulators 1-110 are as follows... Figure 1-5B As shown. In this example, accumulator 1-110 includes memory unit 1-520 and digital adder 1-522. Memory unit 1-520 stores the result of previous accumulation. Digital adder 1-522 adds the scaled partial output dataset generated by the current path to the accumulation result.

[0209] In some embodiments, instead of performing an accumulation step in each path, a portion of the result may be temporarily stored in memory, and a single accumulation may be performed after that path has completed. In either case, processing 1-100 may involve at least one accumulation step, in which a scaled partial output dataset generated by at least two of the multiple paths is accumulated.

[0210] In steps 1-318, accelerator 1-100 determines whether additional paths should be executed. For example, this may involve determining whether there are any additional tiles of matrix A that have not yet been processed. That is, in some embodiments, the number of iterations of method 1-300 is equal to the number of tiles of matrix A.

[0211] If accelerator 1-100 determines that additional iterations are needed, method 1-300 executes another path through the accelerator. In the next path, another tile of matrix A is processed. Conversely, if accelerator 1-100 determines that no additional iterations are needed, method 1-300 terminates.

[0212] three, Partial product

[0213] In some embodiments, the simulated processor performs matrix and vector multiplication: in Let A be the input vector, and let A be an n×n matrix. This is the output vector. In index representation, this multiplication is written as... (For all i in [0, N-1]), it is A ij The n elements (iterated j times) and b j The process involves multiplying n elements (iterated j times) and then summing the results. In some embodiments, element A... ij and b j This can be represented using unsigned fixed-point numbers (e.g., INT8). In this representation, if... It is an m1-digit number, and If the number of bits is m2, then a total of m1 + m2 + log2(n) bits are used to fully represent the resulting vector element c. i Generally, without sacrificing precision, the number of bits required to represent the result of a matrix-vector product is greater than the number of bits required to represent the input of that operation. If ADC unit 1-106 cannot read the output vector with full precision, the output vector elements can be rounded to the ADC's precision.

[0214] The inventors have recognized and understood that constructing an ADC with high bit precision may be difficult given the bandwidth corresponding to the rate at which the input vector in analog signal form is transmitted through analog processors 1-104. Therefore, in some embodiments, the bit precision of DACs 1-102 can limit the matrix element A. ij and vector element b j The bit precision of the representation. Therefore, the inventors devised a method to obtain the output vector with full precision by calculating the sum of partial products, the precision of which can be arbitrarily high. For clarity, it is assumed that A represents... ij or b j The required number of bits is the same, i.e., m1 = m2 = m. However, this assumption can be removed and does not limit the scope of embodiments of this disclosure.

[0215] According to some embodiments, as a first action, the method includes combining matrix element Aij and vector element b.j The bit string represents the division into d partitions, where each partition contains k = m / d bits. (If k is not an integer, zeros can be added until m is divisible by d.) Finally, the matrix element A... ij =A ij [0] 2 k(d-1) +A ij [1] 2 k(d-2) +...+A ij [d-1] 2 0 A ij [f] It is A ij The k-th value of the most significant k-bit string of the f-th position. In bit string form, we can write it as A ij =A ij [0] A ij [1] ...A ij [d-1] Similarly, we can also obtain b. j =b j [0] 2 k(d-1) +b j [1] 2 k(d-2) +…+b j [d-1] 2 0 , where vector element b j =b j [0] b j [1] …b j [d-1] Using its bit string format. Multiplication c i =Σ j A ij b j The following division method can be used to decompose the problem: Among them, set S p Let f be the set of all integer values ​​of f and g, where f + g = p.

[0216] As a second action, the method includes controlling simulation processors 1-104 to implement matrix A. ij [f] The input vector b is propagated as an encoded analog signal through the analog processor 1-104. j [g] Each input vector has a precision of only k bits. Using any of the tiling methods described above, the matrix-vector multiplication operation performs c... i [f，g] =Σ jA ij [f] b j [g] The method includes storing an output vector c with an accuracy of 2k+log2(n) bits. i [f，g] .

[0217] The method also includes in set S p Iterate through the different values ​​f and g, repeating the second step for each different value of f and g, and storing the intermediate results c. i [f，g] .

[0218] As a third action, the method includes calculating the final result by summing different iterations of f and g using digital electronic devices (such as processors).

[0219] According to some embodiments of the method, the accuracy of the ADC used to capture fully accurate calculations is only 2k+log2(n) bits, which is less than the originally required 2m+log2(n) bits of accuracy.

[0220] The inventors further recognize and understand that embodiments of the aforementioned method can be extended to operations on tensors. As previously described, hybrid analog-digital processing systems can perform matrix-matrix multiplication using tiles. The above method can be applied to matrix tiles and rows of input dataset tiles to obtain rows of output dataset tiles with full precision.

[0221] As described in further detail below, in some embodiments, the analog processor includes a photonic processor that can represent the matrix in the form of the phase of an optical variable beam splitter (VBS). In some such embodiments, division can be performed on the bit string representing the phase, rather than directly on the matrix elements. In some embodiments, when the mapping from phase to matrix elements is linear, the relationship between the input parameters (in this case, the phase of the VBS and the input vector elements) and the output vector can be linear. When this relationship is linear, the above method still applies. However, in general, according to some embodiments, a nonlinear mapping from the basic representation of the matrix to the photonic representation can be considered. For example, bit string division of Euclidean space matrix elements from their most significant k bit strings to their least significant k bit strings can be used to produce a series of different matrices that are decomposed into phase representations and implemented using a photonic processor.

[0222] It is not necessary to perform division on both the weight matrix entries and the input dataset entries simultaneously. In some embodiments, the photonic processor can propagate multiple input datasets for the same matrix. Performing division only on the input datasets and maintaining VBS control at a set precision (e.g., full precision) may be effective because the digital-to-analog converter (DAC) used for vector preparation can operate at high bandwidth, while the DAC used for VBS can be quasi-static for multiple vectors. Generally, it is more difficult to include a DAC with high bit precision at higher bandwidths than to design a DAC at lower bandwidths. Therefore, in some embodiments, the output vector elements may be more precise than allowed by the ADC, but the ADC will automatically perform some rounding on the output vector values ​​until the bit precision allowed by the ADC is reached.

[0223] Four, Analog gain

[0224] The inventors have recognized that low-precision (16 bits or less, 12 bits or less, 10 bits or less) fixed-point binary representations require significantly less chip area and power than high-precision floating-point binary representations. However, low-precision fixed-point representations result in a significant reduction in representative accuracy compared to high-precision floating-point representations (e.g., FP32) commonly used in modern processors. Furthermore, reduced accuracy can decrease the dynamic range of the ADC. This leads to a trade-off between representing the full dynamic range of a data vector and accurately representing smaller output values. This paper describes techniques for performing matrix operations using a mixed-signal analog-digital processor with a low-precision ADC. More specifically, some accelerator architectures, according to some embodiments, use a readout window technique for shift ADCs. The shift readout window allows the ADC to focus the conversion on a portion of the signal that may have the most meaningful information or at least some meaningful information. Utilizing these techniques, some accelerator architectures are designed to operate using low-precision fixed-point binary representations, thereby significantly reducing power consumption compared to conventional high-precision floating-point representations.

[0225] Figure 1-6AA graph illustrating this technique is shown. This graph represents an example of the signal distribution that can be presented to the ADC as input (see the curve labeled "Input Distribution"). (In this example, the distribution is represented as a function of voltage, but it can also be represented as a function of other quantities.) The input distribution depends on the data to be processed and can vary drastically depending on the data. The ADC's dynamic range (labeled "Dynamic Range") indicates the maximum range of amplitudes the ADC can read. At one end of the dynamic range is the smallest signal (e.g., voltage) the ADC can read. At the other end of the dynamic range is the largest signal the ADC can read. In this particular example, the input distribution extends only over a small portion of the ADC's dynamic range. However, in other cases, the input data can cause the signal to extend primarily outside the dynamic range.

[0226] The techniques described herein allow for adjustment of the input distribution to more closely match the dynamic range of an ADC. In some embodiments, this can be accomplished by scaling (e.g., amplifying or attenuating) the analog signal before it is provided as input to the ADC. In some embodiments, the analog signal can be scaled such that the signal distribution extends from one end of the dynamic range to the other. In some embodiments, such as Figure 1-6A As shown, an analog signal can be scaled such that a small portion of the tail of the distribution is outside the dynamic range. Figure 1-6A In the example, amplification is applied, which stretches the input distribution (the resulting distribution, labeled "amplified distribution") across the entire dynamic range, with a portion of the tail falling outside the dynamic range. Those inputs outside the dynamic range can be clipped to the minimum and maximum values ​​of the ADC range. Figure 1-6A This is illustrated by vertical spikes in the "amplified distribution" at the endpoints of the ADC range. The inventors have recognized that, although... Figure 1-6A The cut-off of a portion of the distribution may result in information loss, but the lost information may not be as meaningful as the information within the dynamic range.

[0227] This concept is as follows Figure 1-6B As shown. For example, consider matrix-vector multiplication with integer (or fixed-point) algorithms. Where A is an n×n matrix. The entries in A are unsigned numbers represented by m1 bits, and the entries in b are unsigned numbers represented by m2 bits. A full-precision representation of the entries in the output vector c requires b... c bit, where b c = log2(n) + m1 + m2 (see bit string 1). However, if using a string with b ADC c ​When a 16-bit ADC reads the output of this calculation, there is a trade-off between representing the full dynamic range of c and representing smaller output values ​​with greater precision. For example, given n=16, m1=2, and m2=4, the full precision would be b. c =10. As further shown in the figure, a 4-bit ADC configured to read out the full dynamic range of c will read the highest 4 bits of the output (see bit string 2). For certain value distributions, this range may contain useful information. However, it is highly likely that the highest bit will not be set in the meantime. Therefore, the conversion will not produce any meaningful information.

[0228] In some embodiments, the ADC readout window can be shifted by amplifying the signal. Bit string 3 is obtained by amplifying a signal with a gain of 4, which shifts the readout window 2 bits to the least significant bit. Therefore, useful information begins to appear within the window. The resulting 4-bit output captures most of the meaningful information of an output with full precision. Bit string 4 is obtained by amplifying a signal with a gain of 64 (which in this case causes ADC saturation), which further shifts the readout window 3 bits to the least significant bit. The saturation result indicates that the true value is greater than the maximum data value that can be represented on the scale of these least significant bits, and is therefore the most accurate representation of the input bit string on that data scale.

[0229] The inventors have recognized that useful information in the lower bits can be limited by input quantization noise. Consider an example where the analog device programs a normalized matrix A (values ​​between 0 and 1) to an accuracy of m1 bits and has 2 -(M1+1) The level of coding noise, and the analog device programs the normalized entries of b (values ​​between 0 and 1) to be accurate to m2 bits, and has 2 -(m2+1) Level coding noise. For n=16, m1=2, and m2=4, the full precision will be b... c =10. However, considering the case where the maximum value of A (maximum value equals 1) is programmed, the (log₂n+m²+1)th bit may be noisy. Similarly, if the maximum value of b (all values ​​equal 1) is programmed, the (log₂n+m¹+1)th bit may be noisy. Amplifying the signal to read this bit and the less significant bits may produce noisy results. However, the values ​​of these noisy bits can be recovered by multiple measurements and averaging. This encoded quantization noise essentially provides a limit to how much amplification of the signal before encountering a noisy bit. In some embodiments, encoded quantization noise is not the only source of noise. The noise floor of the TIA and the optical receiver (in the case of a photonic processor) can limit the amplification before reading out the noisy bit.

[0230] Generally, it can be difficult to predict how much gain is needed to maximize the collection of meaningful information. In some embodiments, the gain can be determined based on statistics of the weight matrix (e.g., matrix A). In some embodiments, this determination can be further based on known statistics of the input vector data (matrix B) or a bound on the statistics of the input vector data. In one example, the gain can be determined as part of the training procedure. That is, the gain is a parameter that can be learned. In other embodiments, the gain parameter can be learned without using machine learning techniques. For example, it can be determined whether a particular gain level is accurate enough for the training data, and if not, it can be tuned until it is accurate enough. Furthermore, it may be desirable to regularize the gain to lower values ​​during the training procedure, for example by adding a regularization loss term to the overall training loss function, to reduce the amount of energy required for evaluation computation in the analog processor, or to run the analog processor at a higher bandwidth while preserving the gain-bandwidth product of the amplifier.

[0231] In some embodiments, the gain setting can be different for each layer of the neural network. In some such embodiments, it may be desirable to set a learning gain for each layer, and the gain can vary as the transition occurs from one layer to the next. In some embodiments, when the accelerator receives new data, the controller can set the gain to a value used for training the new data.

[0232] Figure 1-7A Examples of accelerators based on the above-described technology are shown according to some embodiments. Similar to accelerator 1-100, accelerator 1-700 includes a DAC unit 1-102, an analog processor 1-104, and an ADC unit 1-106. Accelerator 1-700 also includes an analog scaling unit 1-103 located between the DAC unit 1-102 and the analog processor 1-104, and an analog scaling unit 1-105 located between the analog processor 1-104 and the ADC unit 1-106. Figure 1-7A The components, and other optional components, are collectively referred to as "circuits" in this document.

[0233] Unlike accelerators 1-100, accelerators 1-700 are configured to perform mathematical operations in a single path. Therefore, digital accelerators 1-110 are omitted. However, as described further in detail above, techniques for improving the accuracy of low-precision fixed-point representations can be used in accelerators arranged to perform multi-path operations. In particular, multi-path operations can be performed between different levels of precision in the input data and the values ​​encoded in the analog processor.

[0234] Combination Figure 1-6AThe scaling can be performed at any point along the data path from the DAC unit to the ADC unit. Analog scaling units 1-103 can be configured to scale (e.g., amplify or attenuate) the analog signal before it is provided to the analog processor. Analog scaling units 1-105 can be configured to scale the analog signal after the analog processor 1-104. The analog scaling units can include any suitable type of controllable amplifier or attenuator, including voltage amplifiers, current amplifiers, power amplifiers, transimpedance amplifiers, transconductance amplifiers, voltage attenuators, current attenuators, and power attenuators. In a photonic analog processor, as described in further detail below, the analog scaling unit can be one of the aforementioned analog scaling units, optical amplifiers, or variable gain lasers, etc. Figure 1-7B In the example, analog scaling unit 1-103 includes multiple amplifiers or attenuators 1-702. Controller 1-120 sets the gain of these devices. (The gain mentioned herein indicates amplification or attenuation, depending on whether the gain is greater than or less than 1.)

[0235] Figure 1-8 This is a flowchart illustrating examples of methods for performing mathematical operations according to some embodiments. In some embodiments, accelerator 1-700 ( Figure 1-7A Method 1-800 can be executed, but any other suitable accelerator can be used. It should be understood that the steps of Method 1-800 can be followed... Figure 1-8 The order described, or any other suitable order, shall be executed.

[0236] In step 1-801, accelerator 1-700 receives the input dataset. (Refer to previous step) Figure 1-2B For example, an accelerator can receive entries of matrix B.

[0237] In step 1-802, accelerator 1-700 generates multiple input analog signals based on the input dataset received in step 1-801. (Refer to previous step) Figure 1-7A In some embodiments, the generation of steps 1-804 involves using DAC units 1-102 to convert the input dataset into an analog signal.

[0238] In step 1-804, controller 1-120 determines one or more scaling factors. As described above, in some embodiments, the scaling factor can be determined based on a parameter set and an input dataset. For example, the scaling factor can be determined based on statistics of the parameter set and statistics of the input dataset. Alternatively, the scaling factor can be determined based on statistics of the parameter set and assumptions about the statistics of the input dataset. In some embodiments, two scaling factors can be determined, one for simulating scaling unit 1-103 and one for simulating scaling unit 1-105.

[0239] As mentioned above, there can exist an optimal gain value that maximizes the accuracy of computations performed using numerical representations. Although the optimal gain may not be a priori, in some embodiments, the gain can be learned as part of a training process. Figure 1-9A A graph showing the relationship between the accuracy (expressed as a percentage) of a neural network performing an image classification task and its gain is illustrated. As shown, there exists an optimal gain that maximizes accuracy, which can be learned as part of the training process. In some embodiments, this optimal gain can be calculated before performing methods 1-800, and information indicating the optimal gain can be stored in memory or a lookup table for retrieval during methods 1-800.

[0240] In step 1-806, controller 1-120 sets the gain of analog scaling unit 1-103 and / or the gain of analog scaling unit 1-105 based on the scaling factor determined in step 1-806.

[0241] In step 1-808, analog scaling unit 1-103 amplifies or attenuates the input analog signal generated by DAC unit 1-102 based on a scaling factor. As mentioned above, this may result in a closer match between the signal distribution and the dynamic range of the ADC.

[0242] In step 1-810, controller 1-120 programs analog processor 1-104. More specifically, controller 1-120 programs analog processor 1-104 based on a set of parameters (e.g., parameters of matrix A). Programming the analog processor may involve setting certain characteristics of the analog processor based on parameters, as described above. Figure 1-2A and Figure 1-3 As mentioned above.

[0243] In steps 1-812, analog processors 1-104 generate multiple output analog signals based on multiple input analog signals and the parameter set used in steps 1-810 to program the analog processor. In some embodiments, this may involve performing matrix multiplication or convolution. Figure 1-3 As shown, this operation can be performed using matrix decomposition.

[0244] In steps 1-814, accelerator 1-700 amplifies or attenuates the analog signal output by DAC unit 1-102 using the gain or attenuation factor determined in step 1-806. Amplification or attenuation can be performed using analog scaling unit 1-105. In some embodiments, only one step between steps 1-808 and 1-814 is performed. That is, the signal is amplified or attenuated only before or after passing through analog processor 1-104. In other embodiments, both steps are performed.

[0245] In steps 1-816, accelerator 1-700 generates an output dataset based on the multiple output analog signals obtained in steps 1-814. In some embodiments, the generation in steps 1-816 involves passing the output analog signals through ADC unit 1-106.

[0246] The inventors have recognized that, in some embodiments, it may be advantageous to vary the gain or attenuation factor layer by layer in a multi-layer (e.g., multi-path) neural network model. Indeed, the inventors have realized that the error of a low-precision fixed-point representation can be layer-dependent. Figure 1-9B This paper depicts a functional relationship between the mean squared error (MSE) associated with the low-precision fixed-point representation and the gain across three different layers of the neural network. The MSE is calculated relative to the FP32 representation. In this particular example, setting the gain to 2 minimizes the MSE of the second layer, setting the gain to 4 minimizes the MSE of the third layer, and setting the gain to 6 minimizes the MSE of the fourth layer. This is primarily due to the varying data distribution across the different layers of the neural network model. Therefore, in some embodiments, the gain can be varied layer by layer. Additionally, the gain can be varied across any suitable sub-part of the network allocated to the processor, including but not limited to tiles of the matrix within a layer.

[0247] Figure 1-10 The accelerator will combine Figure 1-2A The aspects described by the accelerator and the combination Figure 1-7A The accelerator 1-1000 combines the aspects described herein. Accelerator 1-1000 includes a DAC unit 1-102, an analog scaling unit 1-103, an analog processor 1-104, an analog scaling unit 1-105, an ADC unit 1-106, a digital scaling unit 1-108, a digital accumulation unit 1-110, and a controller 1-120. Accelerator 1-1000 is configured to perform operations as described above. Figure 1-2A The analog processors 1-104 have multiple pathways and are also configured to perform actions such as combining Figure 1-7A The aforementioned amplification or attenuation. Figure 1-10 The components, as well as other optional components, are collectively referred to herein as "circuit".

[0248] Figure 1-11 This is a flowchart illustrating examples of methods for performing mathematical operations according to some embodiments. Accelerator 1-1000 ( Figure 1-10 ) Perform method 1-1100, but any other suitable accelerator may be used. Method 1-1100 is similar to method 1-300, but it also includes some steps of method 1-800.

[0249] five, Optical analog-digital matrix processor example

[0250] The inventors have developed a special class of analog processors that utilize optical signals. They have realized that because optical signals travel at the speed of light in the medium through which light travels, the limitations on delay are far less than those in electrical systems. Furthermore, by increasing the distance traveled by the optical signal, almost no energy is consumed, thus opening up new topologies and processor layouts that are not feasible when using electrical signals. Therefore, photonic processors can be used as part of an accelerator in some embodiments.

[0251] Any of the above techniques can be applied to accelerators that include photonic processors. According to some embodiments, one example of such an accelerator is... Figure 1-12 As shown. Accelerator 1-1200 includes a DAC unit 1-102, one or more lasers 1-1250, an optical encoder 1-1201, an optical scaling unit 1-1203, a photonic processor 1-1204, an optical scaling unit 1-1205, an optical receiver 1-1207, an electronic analog scaling unit 1-105, an ADC unit 1-106, a digital scaling unit 1-108, a digital accumulation unit 1-110, and a controller 1-120. Although using optical signals, accelerator 1-1200 can be configured to operate in a manner similar to accelerator 1-100 (… Figure 1-2A In a manner similar to accelerator 1-700 ( Figure 1-7A In a manner similar to accelerator 1-1000 ( Figure 1-10 The photonic processor 1-1204 can be designed to perform functions similar to the analog processor 1-104 (or any suitable combination of these accelerators). Figure 1-2A and Figure 1-7A The mathematical operations performed are performed in the optical domain. An example of a photonic processor will be described in further detail below. Figure 1-12 The components (whether optical or electronic) and other optional components are collectively referred to herein as “circuit”.

[0252] DAC unit 1-102, electronic analog scaling unit 1-105, ADC unit 1-106, digital scaling unit 1-108, and digital accumulation unit 1-110 have similar combinations Figure 1-2A and Figure 1-7A The characteristics described.

[0253] Laser 1-1250 provides a reference optical signal to optical encoder 1-1201. In some embodiments, controller 1-120 can control the output power of the laser. Optical encoder 1-1201 is configured to convert analog signals received from DAC unit 1-102 into optically coded signals for processing by photonic processor 1-1204. In some embodiments, the optical signal represents the values ​​and signs of associated bit strings as the amplitude and phase of an optical pulse. In some embodiments, the phase can be limited to a binary choice of zero phase shift or π phase shift, representing positive and negative values, respectively. Embodiments are not limited to real-number input vector values. For example, complex vector components can be represented by using more than two phase values ​​when encoding the optical signal.

[0254] Similar to analog scaling unit 1-103 ( Figure 1-7A Optical scaling unit 1-1203 can be configured to scale the optical input signal generated by optical encoder 1-1201 to more closely match the dynamic range of ADC. Furthermore, optical scaling unit 1-1205 can be configured to scale the optical output of processor 1-1204. For example... Figure 1-7A The analog scaling unit 1-105 can be configured to scale the received electrical output to more closely match the dynamic range of the ADC. In some embodiments, the analog scaling unit 1-105 is implemented by a transimpedance amplifier. Additional or alternatively, the signal distribution can be adjusted based on the laser's output power to more closely match the ADC's dynamic range. Effectively, a laser including an optical amplification medium can be utilized as an amplifier. It should be understood that when... Figure 1-12 While examples include optical scaling units 1-1203, 1-1205, and electrical analog scaling units 1-105, other embodiments may include only a subset of these units. In some embodiments, accelerator 1-1200 uses any combination of the following optical amplifiers to generate multiple amplified output optical signals: laser 1-1250, optical scaling units 1-1203, and optical scaling units 1-1205, because amplification or attenuation can be performed at any point along the path to the ADC.

[0255] Optical receiver 1-1207 is configured to convert optical signals into electrical signals. For example, optical receiver 1-1207 may include multiple photodetectors, such as pn junction photodiodes, pin junction photodiodes, avalanche photodiodes, or phototransistors. In some embodiments, the photodetector includes a germanium photodiode.

[0256] like Figure 1-2A The digital scaling units 1-108 can be configured to scale a portion of the output dataset, and the digital accumulation units 1-110 can be configured to accumulate the scaled portion of the output dataset.

[0257] The photonic processor 1-1204 can be implemented in any of a variety of ways. As described in conjunction with the analog processor 1-104, the photonic processor 1-1204 can be designed to perform matrix operations based on matrix decomposition, examples of which are provided below. Different processor architectures can be used to encode the weight matrix. In some embodiments, this can be accomplished by a programmable photonic device such as an optical modulator. Examples of optical modulators are optical resonant modulators, including ring and disk modulators. Another example is a Franz-Kelder modulator. In some embodiments, programming the photonic processor based on the parameters of the matrix involves setting optical characteristics for the programmable photonic device. For example, programming an optical resonator may involve programming the resonant frequency of the resonator, and programming a Franz-Kelder modulator may involve programming the absorption of the modulator.

[0258] We will now discuss other implementation schemes for photonic processors.

[0259] Figure 2-1 A photonics processing system 2-100, which can be used to implement some components of accelerator 1-1200, is shown according to some embodiments. The photonics processing system 2-100 includes an optical encoder 2-101, a photonics processor 2-103, an optical receiver 2-105, and a controller 2-107. The optical encoder 2-100 serves as the optical encoder 1-1201 (…). Figure 1-12 Photon processor 2-103 is used as photon processor 1-1204, optical receiver 2-105 is used as optical receiver 1-1207, and controller 2-107 is used as controller 1-120. Although Figure 2-1 Not shown, but in some embodiments, the optical analog scaling unit may be inserted between the optical encoder 2-101 and the photonic processor 2-103. Alternatively or additionally, the optical analog scaling unit may be inserted between the photonic processor 2-103 and the optical receiver 2-105.

[0260] The photonic processing system 2-100 receives, for example, an input vector represented by a set of input bit strings from an external processor (e.g., a CPU) and produces an output vector represented by a set of output bit strings. For example, if the input vector is an n-dimensional vector, it can be represented by n separate bit strings, each representing a corresponding component of the vector. The input bit strings can be received from the external processor as electrical or optical signals, and the output bit strings can be transmitted to the external processor as electrical or optical signals. In some embodiments, the controller 2-107 does not necessarily output an output bit string after each processing iteration. Instead, the controller 2-107 can use one or more output bit strings to determine a new input bit stream for feedback through components of the photonic processing system 2-100. In some embodiments, the output bit strings themselves can be used as input bit strings for subsequent iterations of the processing performed by the photonic processing system 2-100. In other embodiments, multiple output bit streams are combined in various ways to determine subsequent input bit strings. For example, one or more output bit strings can be added together as part of the determination of subsequent input bit strings.

[0261] Optical encoder 2-101 outputs n individual optical pulses, which are transmitted to photonic processor 2-103. Each output of optical encoder 2-101 is coupled one-to-one with a single input of photonic processor 2-103. In some embodiments, optical encoder 2-101 may be disposed on the same substrate as photonic processor 2-103 (e.g., optical encoder 2-101 and photonic processor 2-103 are located on the same chip). In such embodiments, optical signals can be transmitted from optical encoder 2-101 to photonic processor 2-103 in a waveguide such as a silicon photonic waveguide. In other embodiments, optical encoder 2-101 may be disposed on a separate substrate from photonic processor 2-103. In such embodiments, optical signals can be transmitted from optical encoder 2-101 to photonic processor 2-103 via optical fiber.

[0262] Photonic processor 2-103 performs the multiplication of the input vector with matrix M. As described in detail below, matrix M can be decomposed into three matrices using a combination of singular value decomposition (SVD) and unitary matrix decomposition. In some embodiments, unitary matrix decomposition is performed using an operation similar to Givens rotation in QR decomposition. For example, SVD combined with Haushold decomposition can be used. The decomposition of matrix M into three components can be performed by controller 2-107, and each component can be implemented by a portion of photonic processor 2-103. In some embodiments, photonic processor 2-103 includes three portions: a first array of variable beam splitters (VBS) configured to perform a transformation equivalent to the first matrix multiplication on the input optical pulse array (e.g., see...). Figure 2-3The first matrix module 2-301); a set of controllable optical elements configured to adjust the intensity and / or phase of each optical pulse received from the first array, the adjustment being equivalent to the second matrix multiplied by a diagonal matrix (e.g., see...). Figure 2-3 The second matrix module 2-303); and the second array of the VBS, which is configured to perform a transformation on optical pulses received from a set of controllable electro-optic elements, the transformation being equivalent to a third matrix multiplication (e.g., see the third matrix module 2-305 of Figure 3).

[0263] Photonic processor 2-103 outputs n individual optical pulses, which are transmitted to optical receiver 2-105. Each output of photonic processor 2-103 is coupled one-to-one with a single input of optical receiver 2-105. In some embodiments, photonic processor 2-103 may be disposed on the same substrate as optical receiver 2-105 (e.g., photonic processor 2-103 and optical receiver 2-105 are located on the same chip). In these embodiments, the optical signal can be transmitted from photonic processor 2-103 to optical receiver 2-105 in a silicon photonic waveguide. In other embodiments, photonic processor 2-103 may be disposed on a separate substrate from optical receiver 2-105. In such embodiments, the optical signal can be transmitted from photonic processor 2-103 to optical receiver 2-105 via optical fiber.

[0264] Optical receiver 2-105 receives n light pulses from photonic processor 2-103. Each light pulse is then converted into an electrical signal. In some embodiments, the intensity and phase of each light pulse are measured by a photodetector within the optical receiver. The electrical signal representing these measurements is then output to controller 2-107.

[0265] The controller 2-107 includes a memory 2-109 and a processor 2-111 to control the optical encoder 2-101, the photonic processor 2-103, and the optical receiver 2-105. The memory 2-109 can be used to store input and output bit strings, as well as measurement results from the optical receiver 2-105. The memory 2-109 also stores executable instructions that, when executed by the processor 2-111, control the optical encoder 2-101, perform a matrix factorization algorithm, control the VBS of the photonic processor 2-103, and control the optical receiver 2-105. The memory 2-109 may also include executable instructions that cause the processor 2-111 to determine a new input vector to be sent to the optical encoder based on a set of one or more output vectors determined by measurements performed by the optical receiver 2-105. In this way, controller 2-107 can control an iterative process by multiplying the input vector with multiple matrices by adjusting the settings of photonic processor 2-103 and feeding back the detection information from optical receiver 2-105 to optical encoder 2-101. Therefore, the output vector transmitted from photonic processing system 2-100 to external processor can be the result of multiple matrix multiplications, rather than a simple single matrix multiplication.

[0266] In some embodiments, the matrix may be too large to be encoded using a single path in the photonic processor. In this case, a portion of the larger matrix can be encoded in the photonic processor, and a multiplication operation can be performed on that single portion of the larger matrix. The result of the first operation can be stored in memories 2-109. Subsequently, a second portion of the larger matrix can be encoded in the photonic processor, and a second multiplication operation can be performed. This "blocking" of the larger matrix can continue until multiplication operations have been performed on all portions of the larger matrix. The results of the multiple multiplication operations, which can then be stored in memories 2-109, can be combined to form the final result of multiplying the input vector by the larger matrix.

[0267] In other embodiments, only the collective behavior of the output vectors is used by the external processor. In such embodiments, only collective results, such as the average or maximum / minimum values ​​of the multiple output vectors, are transmitted to the external processor.

[0268] A. Optical encoder

[0269] refer to Figure 2-2 According to some embodiments, the optical encoder includes at least one light source 2-201, a power tree 2-203, an amplitude modulator 2-205, a phase modulator 2-207, a digital-to-analog converter (DAC) 2-209 associated with the amplitude modulator 2-205, and a DAC 2-211 associated with the phase modulator 2-207. DACs 2-209 and 2-211 may be part of DAC unit 1-102. Figure 1-12 Although in Figure 2-2 The amplitude modulator 2-205 and phase modulator 2-207 are shown as a single block with n inputs and n outputs (e.g., each input and output is a waveguide), but in some embodiments, each waveguide may include a corresponding amplitude modulator and a corresponding phase modulator, such that the optical encoder includes n amplitude modulators and n phase modulators. Furthermore, a separate DAC may exist for each amplitude and phase modulator. In some embodiments, instead of associating the amplitude modulator and its respective phase modulator with each waveguide, a single modulator can be used to encode the amplitude and phase information. While using a single modulator to perform such encoding limits the ability to precisely tune both the amplitude and phase of each optical pulse, there are encoding schemes that do not require precise tuning of both the amplitude and phase of the optical pulse. Such schemes will be described later in this document.

[0270] Light source 2-201 can be any suitable coherent light source. In some embodiments, light source 2-201 can be a diode laser or a vertical-cavity surface emitting laser (VCSEL). In some embodiments, light source 2-201 is configured to have an output power greater than 10mW, greater than 25mW, greater than 50mW, or greater than 75mW. In some embodiments, light source 2-201 is configured to have an output power less than 100mW. Light source 2-201 can be configured to emit continuous light waves or light pulses (“optical pulses”) in one or more wavelengths (e.g., C-band or O-band). The duration of the light pulse can be, for example, approximately 100 ps.

[0271] Although Figure 2-2 The light source 2-201 is shown located on the same semiconductor substrate as other components of the optical encoder, but the embodiment is not limited thereto. For example, the light source 2-201 may be a separate laser package edge-bonded or surface-bonded to the optical encoder chip. Alternatively, the light source 2-201 may be completely detached from the chip, and the optical pulses may be coupled to the waveguide 2-202 of the optical encoder 2-101 via an optical fiber and / or a grating coupler.

[0272] Light source 2-201 is shown as two light sources 2-201a and 2-201b, but embodiments are not limited thereto. Some embodiments may include a single light source. Including multiple light sources 201a-b (which may include more than two light sources) can provide redundancy in the event of a failure of one of the light sources. Including multiple light sources can extend the useful lifetime of the photonics processing system 2-100. The multiple light sources 2-201a-b may each be coupled to a waveguide of the optical encoder 2-101 and then combined at a waveguide combiner configured to direct optical pulses from each light source to the power tree 2-203. In these embodiments, only one light source is used at any given time.

[0273] Some embodiments can use two or more phase-locked light sources of the same wavelength simultaneously to increase the optical power entering the optical encoder system. A small fraction of the light from each of the two or more light sources (e.g., obtained via waveguide taps) can be directed to a homodyne detector, where a beat error signal can be measured. The beat error signal can be used to determine the possible phase drift between the two light sources. For example, the beat error signal can be fed into a feedback circuit that controls a phase modulator that locks the output phase of one light source to the phase of another. Phase-locking can be generalized in a master-slave scheme, where N ≥ 1 slave light sources are phase-locked to a single master light source. The result is a total of N+1 phase-locked light sources available for the optical encoder system.

[0274] In other embodiments, each individual light source may be associated with light of a different wavelength. Using light of multiple wavelengths allows some embodiments to be multiplexed, enabling the simultaneous execution of multiple computations using the same optical hardware.

[0275] Power tree 2-203 is configured to split a single optical pulse from light source 2-201 into a spatially separated array of optical pulses. Therefore, power tree 2-203 has one optical input and n optical outputs. In some embodiments, the optical power from light source 2-201 is uniformly divided across n optical modes associated with n waveguides. In some embodiments, power tree 2-203 is an array of 50:50 beamsplitter 2-801, such as... Figure 2-8 As shown. The number of "depths" of power tree 2-203 depends on the number of output waveguides. For a power tree with n output modes, the depth of power tree 2-203 is ceil(log2(n)). Figure 2-8 Power tree 2-203 is shown only when the tree depth is 3 (each level of the tree is marked across the bottom of power tree 2-203). Each level includes 2 m-1There are 3 beam splitters, where m is the number of layers. Therefore, the first layer has a single beam splitter 2-801a, the second layer has two beam splitters 2-801b–2-801c, and the third layer has four beam splitters 2-801d–2-801g.

[0276] Although the power tree 2-203 is shown as an array of cascaded beam splitters, which can be implemented as an evanescent waveguide coupler, the embodiments are not so limited because any optical device that converts a single optical pulse into multiple spatially separated optical pulses can be used. For example, the power tree 2-203 can be implemented using one or more multimode interferometers (MMIs), in which case the equations for the control layer width and depth will be appropriately modified.

[0277] Regardless of the type of power tree 2-203 used, manufacturing the power tree 2-203 to achieve a precisely uniform beam splitting ratio among the n output modes is difficult, if not impossible. Therefore, the amplitude modulator settings can be adjusted to correct for unequal intensities in the n optical pulses output from the power tree. For example, the waveguide with the lowest optical power can be set to the maximum power for any given pulse transmitted to the photonic processor 2-103. Thus, in addition to modulating the amplitude to encode information into the optical pulses, any optical pulse with power higher than the maximum power can be modulated to have a lower power by the amplitude modulator 2-205. A phase modulator can also be placed at each of the n output modes, which can be used to adjust the phase of each output mode of the power tree 2-203 so that all output signals have the same phase.

[0278] Alternatively or additionally, the power tree 2-203 may be implemented using one or more Mach-Zehnder interferometers (MZIs), which may be tuned such that the beam splitting ratio of each beam splitter in the power tree produces substantially equal intensity pulses at the output of the power tree 2-203.

[0279] Amplitude modulator 2-205 is configured to modify the amplitude of each optical pulse received from power tree 2-203 based on a corresponding input bit string. Amplitude modulator 2-205 may be a variable attenuator or any other suitable amplitude modulator controlled by DAC 2-209, which may be further controlled by controller 2-107. Some amplitude modulators are known for use in telecommunications applications and may be used in some embodiments. In some embodiments, a variable beam splitter may be used as amplitude modulator 2-205, wherein only one output of the variable beam splitter is retained, while the other output is discarded or ignored. Other examples of amplitude modulators that may be used in some embodiments include: traveling wave modulators, cavity-based modulators, Franz-Kelder modulators, plasma-based modulators, two-dimensional material-based modulators, and nano-optomechanical switches (NOEMs).

[0280] Phase modulator 2-207 is configured to modify the phase of each optical pulse received from power tree 2-203 based on a corresponding input bit string. The phase modulator may be a thermo-optical phase shifter or any other suitable phase shifter that can be electrically controlled by controller 2-211, which may be further controlled by controller 2-107.

[0281] when Figure 2-2 The amplitude modulator 2-205 and phase modulator 2-207 are shown as two separate components that can be combined into a single element to control both the amplitude and phase of an optical pulse. However, controlling the amplitude and phase of the optical pulse separately has its advantages. That is, since a link is established between amplitude shift and phase shift via the Kramers-Kronig relation, there is a phase shift associated with any amplitude shift. To precisely control the phase of the optical pulse, the phase modulator 2-207 should be used to compensate for the phase shift generated by the amplitude modulator 2-205. For example, the total amplitude of the optical pulse leaving the optical encoder 2-101 is A = a0a1a2, and the total phase of the optical pulse leaving the optical encoder is... Where a0 is the input intensity of the input optical pulse (assuming the phase at the input of the modulator is zero), a1 is the amplitude attenuation of the amplitude modulator 2-205, and Δθ is the phase shift given by the amplitude modulator 2-205 when modulating the amplitude. The phase shift is given by phase modulator 2-207, and a2 is the attenuation associated with the optical pulse passing through phase modulator 2-209. The phase of the optical signal is determined by the propagation of the optical signal. Therefore, setting the amplitude and phase of the optical pulse are not two independent determinations. Instead, in order to accurately encode a specific amplitude and phase in the optical pulse output from the optical encoder 2-101, the settings of both the amplitude modulator 2-205 and the phase modulator 2-207 should be considered for both settings.

[0282] In some embodiments, the amplitude of the optical pulse is directly related to the bit string value. For example, a high-amplitude pulse corresponds to a high bit string value, and a low-amplitude pulse corresponds to a low bit string value. The phase of the optical pulse encodes whether the bit string value is positive or negative. In some embodiments, the phase of the optical pulse output by the optical encoder 2-101 can be selected from two phases separated by 180 degrees (π radians). For example, a positive bit string value can be encoded using a zero-degree phase shift, and a negative bit string value can be encoded using a 180-degree (π radian) phase shift. In some embodiments, the vector is intended to be complex, and therefore the phase of the optical pulse is not only selected from two values ​​between 0 and 2π.

[0283] In some embodiments, controller 2-107 determines the amplitude and phase applied by both amplitude modulator 2-205 and phase modulator 2-207 based on the input bit string and the above equation, wherein the equation relates the output amplitude and output phase to the amplitude and phase applied by amplitude modulator 2-204 and phase modulator 2-207. In some embodiments, controller 2-107 may store a digital value table for driving amplitude modulator 2-205 and phase modulator 2-207 in memory 2-109. In some embodiments, the memory may be placed near the modulators to reduce communication time delay and power consumption.

[0284] DAC 2-209, associated with and communicatively coupled to amplitude modulator 2-205, receives digital drive values ​​from controller 2-107 and converts them into analog voltages to drive amplitude modulator 2-205. Similarly, DAC 2-211, associated with and communicatively coupled to phase modulator 2-207, receives digital drive values ​​from controller 2-107 and converts them into analog voltages to drive phase modulator 2-207. In some embodiments, the DAC may include amplifiers that amplify the analog voltage to a sufficiently high level to achieve the desired extinction ratio (e.g., the highest extinction ratio physically achievable using a particular phase modulator) and the desired phase shift range (e.g., a phase shift range covering the entire range between 0 and 2π) within the amplitude modulator. Figure 2-2 DAC 2-209 and DAC 2-211 are shown as being located within and / or on the chip of optical encoder 2-101, but in some embodiments, DAC 2-209 and 2-211 may be located off-chip while still being communicatively coupled to amplitude modulator 2-205 and phase modulator 2-207, respectively, using conductive traces and / or wires.

[0285] After being modulated by amplitude modulator 2-205 and phase modulator 2-207, n optical pulses are sent from optical encoder 2-101 to photonic processor 2-103.

[0286] B. Photonic Processor

[0287] refer to Figure 2-3 The photonic processor 2-103 performs matrix multiplication on an input vector represented by n input optical pulses and includes three main components: a first matrix module 2-301, a second matrix module 2-303, and a third matrix module 2-305. In some embodiments, as discussed in more detail below, the first matrix module 2-301 and the third matrix module 2-305 include an interconnected array of programmable, reconfigurable variable beam splitters (VBSs) configured to convert the n input optical pulses from an input vector into an output vector, wherein the components of the vector are represented by the amplitude and phase of each optical pulse. In some embodiments, the second matrix module 2-303 includes a set of electro-optic elements.

[0288] The matrix multiplied by the input vector after the input light pulse is transmitted through the photonic processor 2-103 is called A. Matrix A is a general m x n matrix known to the controller 2-107, and is the matrix that should be implemented by the photonic processor 2-103. Therefore, the controller 2-107 decomposes matrix A using singular value decomposition (SVD), such that matrix A is represented by three component matrices: A = V T ∑U, where U and V are real orthogonal nxn and mxm matrices respectively (U T U=UU T =I and V T V = VV T =I), and ∑ is an m x n diagonal matrix with real entries. The superscript "T" in all equations denotes the transpose of the relevant matrix. The SVD of the matrix is ​​known, and the controller 2-107 can use any suitable technique to determine the SVD of matrix A. In some embodiments, matrix A is a complex matrix, in which case matrix A can be decomposed into A = V T ∑U, where V and U are complex unitary n×n and m×m matrices, respectively. and ), and ∑ is an m×n diagonal matrix with real or complex entries. The values ​​of the diagonal singular values ​​can also be further normalized so that the maximum absolute value of the singular values ​​is 1.

[0289] Once the controller 2-107 has determined the matrices U, ∑, and V for matrix A, and if matrices U and V are orthogonal real matrices, the controller can further decompose the two orthogonal matrices U and V into a series of real-valued Givens rotation matrices. The Givens rotation matrix G(i, j, θ) is defined by the following equation per component:

[0290] For k≠i,j,g kk =1

[0291] For k = i, j, g kk =cos(θ)

[0292] g ij =-g ji = -sin(θ),

[0293] Otherwise, g kl =0

[0294] Where g ij Let represent the elements in the i-th row and j-th column of matrix G, and θ be the rotation angle associated with the matrix. Typically, matrix G is an arbitrary 2×2 unitary matrix with determinant 1 (SU(2) group), which is parameterized by two parameters. In some embodiments, these two parameters are the rotation angle θ and another phase value φ. However, matrix G can be parameterized by values ​​other than angle or phase, such as by reflectivity / transmittance or by separation distance (in the case of NOEMS).

[0295] The works of M. Reck et al., “Experimental realization of any discrete unitary operator,” Physical Review Letters 73, 58 (1994) (“Reck”), and WRClements et al., “Optimal design for universal multiport interferometers,” Optica 3, 12 (2016) (“Clements”), are both incorporated herein by reference in their entirety, and at least for the discussion of techniques for decomposing real orthogonal matrices in the manner of Givens rotation. (If any term used herein conflicts with its usage in Reck and / or Clements, the meaning of the term shall be best understood by one of ordinary skill in the art as it is used herein.) The resulting decomposition is given by the following equation:

[0296]

[0297] Where U is an n x n orthogonal matrix, S k It is the set of indices associated with the k-th set of Givens rotations applied (as defined by the decomposition algorithm), θ ij (k) Let S represent the angle of Givens rotation applied between components i and j in the k-th set, and D is a diagonal matrix consisting of +1 or -1 entries, representing the global symbol on each component. Index set S k It depends on whether n is even or odd. For example, when n is even:

[0298] For odd number k, S k ={(1,2),(3,4),...,(n-1,n)}

[0299] For even numbers k, S k ={(2,3),(4,5),...,(n-2,n-1)}

[0300] When n is odd:

[0301] For odd number k, S k ={(1,2),(3,4),...,(n-2,n-1)}

[0302] For even numbers k, S k ={(2,3),(4,5),...,(n-1,n)}

[0303] As an example, and not a limitation, the decomposition of a 4x4 orthogonal matrix can be represented as:

[0304]

[0305] A brief overview of an embodiment of an algorithm implemented using controller 2-107 for decomposing an n x n matrix U in the form of n sets of real-valued Givens rotations is as follows:

[0306] U′←U

[0307] For i from 1 to n-1:

[0308] If i is odd:

[0309] For j = 0 to i-1:

[0310] Looking for G T i-j,i-j+1 (θ) makes element U′ n-j,i-j Invalid, that is, θ = tan -1 (U′ n-j,i-j / U′ n-j,i-j+1 )

[0311] U′←U′G T i-j,i-j+1 (θ)

[0312] Otherwise, if i is even:

[0313] For j = 1 to i:

[0314] Looking for G T n+j-1,n+j-1 (θ) makes element U′ n+j-i,j Invalid, that is, θ = tan -1 (-U′ n+j-i,j /

[0315] U′ n+j-i-1,j )

[0316] U′←G T i-j,i-j+1 (θ)U′

[0317] The resulting matrix U' from the above algorithm is a lower triangular matrix, and its relationship with the original matrix U is expressed by the following equation:

[0318]

[0319] Among them, label S L The two pattern sets connected to the left of U′ by VBS are labeled, and the label S R The two pattern sets connected to the right of U' by VBS are labeled. Since U is an orthogonal matrix, U' is a diagonal matrix with {-1, 1} entries along its diagonal. Therefore, U' = D. U It is known as a "phase screen".

[0320] The next step of the algorithm is to repeatedly search for G. T jk (θ1)D U =D U G jk (θ2), which can be accomplished using the following algorithm, and can be implemented using controller 2-107:

[0321] For S L For each (j,k):

[0322] If U′ j,j and U′ k,k They have different symbols:

[0323] θ2=-θ1

[0324] otherwise

[0325] θ2=θ1

[0326] The above algorithm can also be used to decompose V and / or V T To determine the VBS value of layer m and the associated phase screen.

[0327] The above concept of decomposing an orthogonal matrix into real-valued Givens rotation matrices can be extended to complex matrices, such as unitary matrices instead of orthogonal matrices. In some embodiments, this can be accomplished by including an additional phase in the parameterization of the Givens rotation matrix. Thus, the general form of the Givens matrix with the additional phase term is T(i,j,θ,φ), where

[0328] For k≠i,j,t kk =1

[0329] t ii =e iφ cos(θ),

[0330] t jj =cos(θ),

[0331] t ij = -sin(θ),

[0332] t ji =e iφ sin(θ),

[0333] Otherwise, t kl =0

[0334] Where t ij Let represent the i-th row and j-th column of matrix t, θ be the rotation angle associated with the matrix, and φ be the additional phase. Any unitary matrix can be decomposed into a T-type matrix (i,j,θ,φ). By choosing to set the phase φ = 0, the above-described ordinary real-valued Givens rotation matrix is ​​obtained. Conversely, if the phase φ = π, a set of matrices called Haushold matrices is obtained. The Haushold matrix, H, has the form Where I is an n×n identity matrix, v is a unit vector, and The product is the outer product. The Haushofer matrix represents the reflection of a hyperplane orthogonal to the unit vector v. In this parameterization, the hyperplane is a two-dimensional subspace, rather than an n-1-dimensional subspace as is usually defined for the Haushofer matrix used in QR decomposition. Therefore, decomposing a matrix into Givens rotations is equivalent to decomposing a matrix into Haushofer matrices.

[0335] Based on the aforementioned constraint set of decomposing any unitary matrix into Givens rotations, any unitary matrix can be implemented using a specific sequence of rotations and phase shifts. In photonics, rotations can be represented by variable beam splitters (VBS), and phase shifts are easily implemented using phase modulators. Therefore, for n optical inputs to the photonic processor 2-103, the first matrix module 2-301 and the third matrix module 2-305, representing the unitary matrix of matrix A's SVD, can be implemented using an interconnected array of VBS and phase shifters. Due to the parallel nature of simultaneously transmitting n optical pulses through the VBS array, it is possible to achieve O(IJK / n) time complexity. 2 The second matrix module 2-303 performs matrix multiplication of dimension (I,K)x(K,J) within a time interval. The second matrix module 2-303 is a diagonal matrix of the SVD of matrix A, combined with the diagonal matrix D of each orthogonal matrix associated with the SVD. As mentioned above, each matrix D is called a "phase screen" and can be labeled with a subscript to indicate whether it is a phase screen associated with matrix U or matrix V. Therefore, the second matrix module 303 is matrix ∑′=D V ∑DU By factoring these phase screens out of matrices U and V, we obtain Figure 2-3 The matrices U and V in the matrix.

[0336] In some embodiments, the VBS unit of the photonic processor 2-103 associated with the first matrix module 2-301 and the third matrix module 2-305 can be a Mach-Zehnder interferometer (MZI) with an internal phase shifter. In other embodiments, the VBS unit can be a microelectromechanical system (MEMS) actuator. In some embodiments, an external phase shifter can be used to implement the additional phase required for the Givens rotation.

[0337] Second matrix module 2-303 (representing diagonal matrix D) V ∑D U This can be implemented using amplitude modulators and phase shifters. In some embodiments, a VBS can be used to split a portion of the light, which can be dumped to variably attenuate the optical pulse. Additionally or alternatively, a controllable gain medium can be used to amplify the optical signal. For example, GaAs, InGaAs, GaN, or InP can be used as active gain media for amplifying the optical signal. Other active gain processing can also be used, such as second harmonic generation in materials with crystal inversion symmetry (e.g., KTP and lithium niobate), and four-wave mixing processes in materials lacking inversion symmetry (e.g., silicon). The phase shifter in each optical mode can be used to apply zero-phase shift or π-phase shift, depending on the phase screen implemented. In some embodiments, only a single phase shifter is used per optical mode, rather than one phase shifter per phase screen. This is possible because each matrix D V ,∑, and D U They are all diagonal and therefore can be interchanged. Therefore, the value of each phase shifter in the second matrix module 2-303 of the photonic processor 2-103 is the product of the two phase screens: D V D U .

[0338] refer to Figure 2-4According to some embodiments, the first and third matrix modules 2-301 and 2-305 are implemented as arrays of VBS 2-401. For simplicity, only n = 6 input optical pulses (rows) are shown, resulting in a "circuit depth" (e.g., column number) equal to the number of input optical pulses (e.g., 6). For clarity, only a single VBS 2-401 is labeled with reference numbers. However, VBSs are labeled with subscripts identifying which optical modes are mixed by a particular VBS, and with superscripts indicating the relevant columns. Each VBS 2-401 implements a complex Givens rotation T(i, j, θ, φ), as discussed above, where i and j correspond to the subscript labels of the VBS 2-401, θ is the rotation angle of the Givens rotation, and φ is the additional phase associated with the generalized rotation.

[0339] refer to Figure 2-5 Each VBS 2-401 can be implemented using an MZI 2-510 and at least one external phase shifter 2-507. In some embodiments, a second external phase shifter 2-509 may also be included. The MZI 2-510 includes a first evanescent coupler 2-501 and a second evanescent coupler 2-503 for mixing the two input optical modes of the MZI 2-510. An internal phase shifter 2-505 modulates the phase θ in one arm of the MZI 2-510 to create a phase difference between the two arms. Adjusting the phase θ causes the light intensity output by the VBS 2-401 to change from one output optical mode of the MZI 2-510 to another, thereby producing a controllable and variable beam splitter. In some embodiments, a second internal phase shifter may be applied to the second arm. In this case, the difference between the two internal phase shifters results in a change in the output light intensity. The average value between the two internal phases will impose a global phase on the light entering modes i and j. Therefore, the two parameters θ and φ can be controlled by the phase shifters respectively. In some embodiments, the second external phase shifter 2-509 can be used to correct unwanted differential phase of the output optical mode across VBS due to static phase disorder.

[0340] In some embodiments, phase shifters 2-505, 2-507, and 2-509 may include thermo-optical, electro-optical, or optomechanical phase modulators. In other embodiments, an NOEMS modulator may be used instead of including an internal phase modulator 505 within the MZI510.

[0341] In some embodiments, the number of VBSs increases with the size of the matrix. The inventors have recognized that controlling a large number of VBSs can be challenging, and that sharing a single control circuit across multiple VBSs is advantageous. An example of a parallel control circuit that can be used to control multiple VBSs is a digital-to-analog converter (DAC) that receives a digital string as input, which encodes an analog signal to be applied to a particular VBS. In some embodiments, the circuit also receives a second input, namely the address of the VBS to be controlled. The circuit can then apply an analog signal to the addressed VBS. In other embodiments, the control circuit can automatically scan through multiple VBSs and apply analog signals to multiple VBSs without actively providing addresses. In this case, the addressing sequence is predefined so that it traverses the VBS array in a known order.

[0342] refer to Figure 2-6 The second matrix module 2-303 performs the multiplication of diagonal matrices ∑′=D. V ∑D U This can be accomplished using two phase shifters 2-601 and 2-605 to implement two phase screens and an amplitude modulator 2-603 to adjust the intensity of the associated optical pulse (the adjustment amount is η). As mentioned above, in some embodiments, only the single-phase modulator 2-601 can be used because the two phase screens can be combined together, since the three component matrices forming ∑′ are diagonal and therefore interchangeable.

[0343] In some embodiments, the amplitude modulator 2-603 can be implemented using attenuators and / or amplifiers. If the amplitude modulation value η is greater than 1, the optical pulse is amplified. If the amplitude modulation value η is less than 1, the optical pulse is attenuated. In some embodiments, only attenuation is used. In some embodiments, attenuation can be implemented using a series of integrated attenuators. In other embodiments, such as Figure 2-7 As shown, attenuation 2-603 can be implemented using an MZI comprising two evanescent couplers 2-701 and 2-703 and a controllable internal phase shifter 2-705 to adjust how much input light is transmitted from the input of the MZI to the first output port 2-709 of the MZI. The second output port 2-707 of the MZI may be ignored, blocked, or dumped.

[0344] In some embodiments, controller 2-107 controls the value of each phase shifter in photonic processor 2-103. Each phase shifter discussed above may include a DAC similar to the DAC discussed in conjunction with phase modulator 2-207 of optical encoder 2-101.

[0345] The photonic processor 2-103 can include any number of input optical modes, but the size and complexity of the interconnected VBS arrays 2-301 and 2-305 will increase with the number of input optical modes. For example, if there are n input optical modes, the photonic processor 2-103 will have a circuit depth of 2n+1, where the first matrix module 2-301 and the second matrix module 2-305 each have a circuit depth of n, and the second matrix module 2-303 has a circuit depth of 1. Importantly, the time complexity of performing a single matrix multiplication is linearly related to the number of columns of the input data. If the dimension of the matrix multiplication being processed is (I,K)x(K x J), then the time complexity of performing this multiplication on an optical processor with n optical modes is O(IKJ / n). 2 In some embodiments, this low-order complexity burdened by parallelization results in energy and time efficiencies that are unattainable using conventional electrical processors.

[0346] It should be noted that although the embodiments described herein show the photonic processor 2-103 having n inputs and n outputs, in some embodiments, the matrix A implemented by the photonic processor 2-103 may not be a square matrix. In these embodiments, the photonic processor 2-103 may have different numbers of outputs and inputs.

[0347] It should also be noted that, due to the interconnection topology of the VBS within the first and second matrix modules 2-301 and 2-305, the photonic processor 2-103 can be subdivided into non-interactive row subsets, thereby enabling the simultaneous execution of more than one matrix multiplication. For example, in Figure 2-4 In the VBS array shown, if each VBS 2-401 of coupled optical modes 3 and 4 is set such that optical modes 3 and 4 are completely uncoupled (e.g., as shown in the diagram), Figure 2-4 In VBS 2-401 (without the subscript "34"), the top three optical modes will operate completely independently of the bottom three. This subdivision can be accomplished on a larger scale using photonic processors with a greater number of input optical modes. For example, a photonic processor with n=64 can simultaneously multiply eight eight-component input vectors by their respective 8×8 matrices (each of which is individually programmable and controllable). Furthermore, photonic processor 2-103 does not need to be uniformly subdivided. For example, a photonic processor with n=64 can subdivide into seven different input vectors with 20, 13, 11, 8, 6, 4, and 2 components, each multiplied by its corresponding matrix simultaneously. It should be understood that the numerical examples above are for illustrative purposes only, and any number of subdivisions can be performed.

[0348] Additionally, when the photonic processor 2-103 performs matrix-vector multiplication, where vectors are multiplied by passing optical signals through the VBS array, the photonic processor 2-103 can also be used to perform matrix-matrix multiplication. For example, multiple input vectors can be passed through the photonic processor 2-103 one at a time, where each input vector represents a column of the input matrix. After optically computing each individual matrix-vector multiplication (each multiplication resulting in an output vector corresponding to a column of the output matrix), the results can be digitally combined to form the output matrix produced by the matrix-matrix multiplication.

[0349] C. Optical receiver

[0350] The photonic processor 2-103 outputs n optical pulses that are transmitted to the optical receiver 2-105. The optical receiver 2-105 receives the optical pulses and generates an electrical signal based on the received optical signal. In some embodiments, the amplitude and phase of each optical pulse are determined. In some embodiments, this is achieved using a homodyne or heterodyne detection scheme. In other embodiments, a simple phase-insensitive photodetector can be performed using a conventional photodiode.

[0351] refer to Figure 2-9 According to some embodiments, the optical receiver 2-105 includes a null detector 2-901, a transimpedance amplifier 2-903, and an ADC 2-905. The transimpedance amplifier 2-903 can be used as an electronic analog scaling unit 1-105. Figure 1-12 Furthermore, the ADC2-905 can be part of the ADC unit 1-106. Although the components are... Figure 2-9 The element is shown as a single component for all optical modes, but this is for simplicity. Each optical mode may have a dedicated homodyne detector 2-901, a dedicated transimpedance amplifier 2-903, and a dedicated ADC 2-905. In some embodiments, the transimpedance amplifier 2-903 may be omitted. Instead, any other suitable electronic circuitry that converts current to voltage may be used.

[0352] refer to Figure 2-10 According to some embodiments, the zero-difference detector 2-903 includes a local oscillator (LO) 2-1001, an orthogonal controller 2-1003, a beam splitter 2-1005, and two detectors 2-1007 and 2-1009. The zero-difference detector 2-903 outputs a current based on the difference between the currents output through the first detector 2-1007 and the second detector 2-1009.

[0353] Local oscillator 2-1001 is combined with the input optical pulse at beamsplitter 2-1005. In some embodiments, a portion of light source 2-201 is transmitted to homodyne detector 2-901 via optical waveguide and / or optical fiber. The light from light source 2-201 can itself be used as local oscillator 2-1001, or in other embodiments, local oscillator 2-1001 can be a separate light source that uses light from light source 2-201 to generate phase-matched optical pulses. In some embodiments, MZI can replace beamsplitter 2-1005 so that adjustment can be made between the signal and the local oscillator.

[0354] The quadrature controller 2-1003 controls the cross-sectional angle in the phase space where the measurement takes place. In some embodiments, the quadrature controller 2-1003 may be a phase shifter that controls the relative phase between the input optical pulse and the local oscillator. The quadrature controller 2-1003 is shown as a phase shifter in the input optical mode. However, in some embodiments, the quadrature controller 2-1003 may be in local oscillator mode.

[0355] The first detector 2-1007 detects the light output from the first output of the beam splitter 2-1005, and the second detector 2-1009 detects the light output from the second output of the beam splitter 2-1005. Detectors 2-1007 and 2-1009 can be photodiodes operating with zero bias.

[0356] Subtraction circuit 2-1011 subtracts the current from first detector 2-1007 from the current from second detector 2-1009. The resulting current has amplitude and sign (positive or negative). Transimpedance amplifier 2-903 converts this current difference into a voltage, which can be positive or negative. Finally, ADC 2-905 converts the analog signal into a digital bit string. This output bit string represents the output vector result of the matrix multiplication and is an electronic, digital version of the optical output representation of the output vector output by photonic processor 2-103. In some embodiments, the output bit string may be sent to controller 2-107 for additional processing, as described above, which may include determining the next input bit string based on one or more output bit strings and / or transmitting the output bit string to an external processor.

[0357] The inventors further recognize that the components of the aforementioned photonic processing system 2-100 do not need to be linked back-to-back such that a first matrix module 2-301 is connected to a second matrix module 2-303, which in turn is connected to a third matrix module 2-305. In some embodiments, the photonic processing system 2-103 may include only a single unitary circuit for performing one or more multiplications. The output of the single unitary circuit may be directly connected to the optical receiver 2-105, wherein the result of the multiplication is determined by detecting the output optical signal. In these embodiments, the single unitary circuit may, for example, implement the first matrix module 2-301. The result detected by the optical receiver 2-105 can then be digitally transmitted to a conventional processor (e.g., processor 2-111), whereby the diagonal second matrix module 2-303 is executed in the digital domain using the conventional processor (e.g., 2-111). Then, controller 2-107 can reprogram the individual unitary circuit to execute the third matrix module 2-305, which determines the input bit string based on the result of the digital implementation of the second matrix module, and controls the optical encoder to transmit the optical signal encoded based on the new input bit string through the individual unitary circuit with the reprogrammed settings. The resulting output optical signal detected by optical receiver 105 is then used to determine the result of the matrix multiplication.

[0358] The inventors also recognized the advantages of linking multiple photonic processors 2-103 back-to-back. For example, to implement matrix multiplication M = M1M2, where M1 and M2 are arbitrary matrices, but M2 changes more frequently than M1 due to varying input workloads, a first photonic processor can be controlled to implement M2, while a second photonic processor optically coupled to the first photonic processor can implement the statically held M1. This way, only the first photonic processing system needs frequent updates based on varying input workloads. Such an arrangement not only speeds up computation but also reduces the number of data bits traveling between the controller 2-107 and the photonic processors.

[0359] D. Folded Photon Processing System

[0360] exist Figure 2-1In this arrangement, optical encoder 2-101 and optical receiver 2-105 are positioned on opposite sides of photonics processing system 2-100. In application, feedback from optical receiver 2-105 is used to determine the input to optical encoder 2-101 for future iterations of the process. Data is electronically transmitted from optical receiver 2-105 to controller 2-107 and then to optical encoder 2-101. The inventors have recognized that reducing the distance these electrical signals need to travel (e.g., by reducing the length of electrical traces and / or wires) results in power savings and reduced latency. Furthermore, it is not necessary to place optical encoder 2-101 and optical receiver 2-105 at opposite ends of the photonics processing system.

[0361] Therefore, in some embodiments, the optical encoder 2-101 and the optical receiver 2-105 are positioned close to each other (e.g., on the same side of the photonic processor 2-103) such that the distance the electrical signal must travel between the optical encoder 2-101 and the optical receiver 2-105 is less than the width of the photonic processor 2-103. This can be accomplished by physically interlacing the components of the first matrix module 2-301 and the third matrix module 2-305 so that they are physically located in the same part of the chip. This arrangement is called a “folded” photonic processing system because light first propagates through the first matrix module 2-301 in a first direction until it reaches a physical part of the chip away from the optical encoder 2-101 and the optical receiver 2-105, and then folds so that when the third matrix module 2-305 is implemented, the waveguide causes the light to propagate in the opposite direction to the first direction. In some embodiments, the second matrix module 2-303 is physically located at the fold in the adjacent waveguide. This arrangement reduces the complexity of the electrical traces connecting the optical encoder 2-101, optical receiver 2-105, and controller 2-107, and reduces the total chip area used to implement the photonics processing system 2-100. For example, some embodiments using a folded arrangement only use... Figure 2-1 Back-to-back photonic arrangements require 65% of the total chip area. This can reduce the cost and complexity of photonic processing systems.

[0362] The inventors have recognized that the folded arrangement offers not only electrical advantages but also optical advantages. For example, by reducing the distance the optical signal must travel from the light source used as the local oscillator for null detection, the time-dependent phase fluctuations of the optical signal can be reduced, resulting in higher-quality detection results. In particular, by positioning the light source and null on the same side of the photonic processor, the distance the optical signal used for the local oscillator no longer depends on the size of the matrix. For example, in Figure 2-1In a back-to-back arrangement, the distance traveled by the optical signal targeting the local oscillator is linearly proportional to the size of the matrix, while in a folded arrangement, the travel distance is constant and independent of the matrix size.

[0363] Figure 2-11 This is a schematic diagram of a folded photonics processing system 2-1100 according to some embodiments. The folded photonics processing system 2-1100 includes a power tree 2-1101, multiple optical encoders 2-1103a–2-1103d, multiple homodyne detectors 2-1105a–2-1105d, multiple selector switches 2-1107a–2-1107d, multiple U matrix components 2-1109a–2-1109j, multiple diagonal matrix components 2-1111a–2-1111d, and multiple V matrix components 2-1113a–2-1113j. For clarity, not all components of the folded photonics processing system are shown in the figures. It should be understood that the folded photonics processing system 2-1100 may include components similar to those in the back-to-back photonics processing system 2-100.

[0364] Power tree 2-1101 is similar to power tree 2-203 in Figure 2 and is configured to transmit light from a light source (not shown) to optical encoder 2-1103. However, the difference between power tree 2-1101 and power tree 2-203 is that the power tree transmits the optical signal directly to homodyne detector 2-1105a. In Figure 2, light source 201 transmits the local oscillator signal to the homodyne detector on the other side of the photonic processor by tapping a portion of the optical signal from the light source and guiding the optical signal using a waveguide. Figure 2-11 In the example, power tree 2-1101 includes outputs whose number is twice the number of spatial modes. For example, Figure 2-11 Only four spatial modes of the photonic processor are shown, resulting in eight output modes from the power tree 2-1101—one output directing light to each optical encoder 2-1103, and one output directing light to each homodyne detector 2-1105. The power tree could be implemented, for example, using a cascaded beam splitter or a multimode interferometer (MMI).

[0365] The optical encoder 2-1103 is similar to the power tree optical encoder 2-101 of Figure 1 and is configured to encode information into the amplitude and / or phase of the optical signal received from the power tree 2-1101. This can be implemented, for example, as described in conjunction with the optical encoder 2-101 of Figure 2.

[0366] The zero-difference detector 2-1105 is located between the power tree 2-1101 and the U-matrix component 2-1109. In some embodiments, the zero-difference detector 2-1105 is physically positioned in a column containing the optical encoder 2-1103. In some embodiments, the optical encoder 2-1103 and the zero-difference detector 2-1105 can be interleaved in a single column. In this way, the optical encoder 2-1103 and the zero-difference detector 2-1105 are very close to each other, thereby reducing the distance between the traces (not shown) used to connect the optical encoder 2-1103 and the zero-difference detector 2-1105 and the controller (not shown), which may be physically located in a column adjacent to the optical encoder 2-1103 and the zero-difference detector 2-1105.

[0367] Each of the optical encoders 2-1103 is associated with a corresponding homodyne detector 2-1105. Both the optical encoder 2-1103 and the homodyne detector 2-1105 receive optical signals from the power tree 2-1101. As described above, the optical encoder 2-1103 uses the optical signals to encode the input vector. As described above, the homodyne detector 2-1105 uses the received optical signals from the power tree as a local oscillator.

[0368] Each pair of optical encoders 2-1103 and homodyne detectors 2-1105 is associated with and connected to selector switches 2-1107 via waveguides. Selector switches 2-1107a–2-1107d can be implemented using, for example, conventional 2x2 optical switches. In some embodiments, the 2x2 optical switches are MZIs with internal phase shifters to control the behavior of the MZI from cross to strip. Switches 2-1107 are connected to a controller (not shown) to control whether the optical signal received from the optical encoders 2-1103 is directed toward either the U matrix component 2-1109 or the V matrix component 2-1113. The optical switches are also controlled to direct light received from the U matrix component 2-1109 and / or the V matrix component 2-1113 toward the homodyne detector 2-1105 for detection.

[0369] In the photonic folding photonic processing system 2-1100, the technique used to perform matrix multiplication is similar to the combination described above. Figure 2-3 The back-to-back system described herein. The difference between the two systems lies in the physical layout of the matrix components and the implementation scheme of folding 2-1120, in which the optical signal is in Figure 2-11 The approximate propagation from left to right changes to an approximate propagation from right to left. Figure 2-11 In this context, the connections between components may represent waveguides. In some embodiments, solid lines represent waveguide portions where optical signals propagate from left to right, and in other embodiments, dashed lines represent waveguide portions where optical signals propagate from right to left. Specifically, according to this nomenclature, Figure 2-11The illustrated embodiment is one in which selector switch 2-1107 first directs the optical signal to U matrix component 2-1109. In other embodiments, selector switch 2-1107 may first direct the optical signal to V matrix component 2-1113, in which case the dashed lines would represent the portion of the waveguide from left to right propagating the optical signal, and the solid lines would represent the portion of the waveguide from right to left propagating the optical signal.

[0370] The U matrix of the SVD of matrix A is implemented in the photonic processing system 2-1100 using U matrix component 2-1109, which is interleaved with V matrix component 2-1113. Therefore, with Figure 2-3 Unlike the back-to-back arrangement shown in the embodiment, all U matrix components 2-1109 and V matrix components 2-1113 are not in corresponding independent arrays physically located within a single physical region. Therefore, in some embodiments, the photonic processing system 2-1100 includes multiple columns of matrix components, and at least one column includes both U matrix components 2-1109 and V matrix components 2-1113. In some embodiments, such as Figure 2-11 As shown, the first column may have only the U matrix component 2-1109. The U matrix component 2-1109 is implemented in a manner similar to the first matrix module 2-301 in Figure 3.

[0371] Due to the interleaved structure of the U matrix component 2-1109 and the V matrix component 2-1113, the folded photonic processing system 2-1100 includes waveguide crossovers 2-1110 at different locations between the columns of the matrix elements. In some embodiments, the waveguide crossovers can be constructed using an adiabatic evanescent elevator between two or more layers in an integrated photonic chip. In other embodiments, the U matrix and V matrix can be positioned on different layers of the same chip, and waveguide crossovers are not used.

[0372] After the optical signal propagates through all the U matrix components 2-1109, it propagates to the diagonal matrix component 2-1111, which proceeds in a manner similar to... Figure 2-3 The second matrix module 2-303 is used for implementation.

[0373] After the optical signal propagates through all the diagonal matrix components 2-1111, it propagates to the V matrix component 2-1113, which proceeds in a manner similar to... Figure 2-3 The third matrix module 2-305 is used for implementation. The V matrix of the SVD of matrix A is implemented in the photonic processing system 2-1100 using V matrix component 2-1113 interleaved with U matrix component 2-1109. Therefore, all V matrix components 2-1113 are not physically located in a single independent array.

[0374] After the optical signal propagates through all V matrix components 2-1113, the optical signal returns to the selector switch 2-1107, which directs the optical signal to the zero-difference detector 2-1105 for detection.

[0375] The inventors also recognized that by including a selector switch after the optical encoder and before the matrix components, the folded photonics processing system 2-1100 allows for effective bidirectionality of the circuitry. Therefore, in some embodiments, the controller, such as... Figure 2-1 The controller 2-107 described herein can control whether the optical signal is multiplied by the U matrix first or by the V matrix first. T Matrix. For a VBS array configured to implement a unitary matrix U when propagating optical signals from left to right, the optical signals propagating from right to left implement a matrix U. T Multiplication. Therefore, for the same setup of a VBS array, U can be implemented simultaneously. and U T This depends on how the optical signal propagates through the array, which can be controlled using the selector switch in 2-1107. In some applications, such as backpropagation for training machine learning algorithms, running the optical signal backward through one or more matrices may be desirable. In other applications, bidirectionality can be used to compute the inverse matrix operation on the input vector. For example, for an invertible n×n matrix A, the SVD result is A = V. T ∑U. The inverse of this matrix is ​​A. -1 =U T ∑ -1 V, where ∑ -1 It is the inverse of a diagonal matrix, which can be efficiently computed by inverting each diagonal element. To multiply a vector by matrix A, the switch is configured to guide the optical signal through matrix U in the first direction, then ∑, then V. T To multiply a vector by its inverse A -1 First, set singular values ​​to apply to ∑ -1 The matrix implementation is programmed. This constitutes changing only the settings of one column of the VBS instead of the settings of all 2n+1 columns of the photon processor, which is such as Figure 2-3 The example shown is a unidirectional photon processing system. Then, the optical signal, representing the input vector, propagates through matrix V in a second direction opposite to the first direction. T , then ∑ -1 And then U. Using selector switch 2-1107, the folded photonics processing system 2-1100 can easily switch from first implementing the U matrix (or its transpose) and first implementing the V matrix. T The matrix (or its transpose) is changed.

[0376] six, in conclusion

[0377] Having described several aspects and embodiments of the technology described herein, it should be understood that those skilled in the art will readily make various changes, modifications, and improvements. Such changes, modifications, and improvements are intended to conform to the spirit and scope of the technology described herein. Therefore, it should be understood that the foregoing embodiments are presented by way of example only, and that inventive embodiments may be practiced beyond the specific descriptions within the scope of the appended claims and their equivalents. Furthermore, any combination of two or more features, systems, articles, materials, and / or methods described herein, provided that these features, systems, articles, materials, and / or methods are not inconsistent with each other, is included within the scope of this disclosure.

[0378] Furthermore, as described, some aspects can be embodied as one or more methods. The actions performed as part of a method can be ordered in any suitable manner. Therefore, embodiments can be constructed in which the actions are performed in a different order than shown, which may include performing some actions simultaneously, even though they are shown as sequential actions in the illustrative embodiments.

[0379] All definitions used in this document should be understood as governed by dictionary definitions, definitions in files merged by reference, and / or the general meaning of the definition terms.

[0380] Unless explicitly stated otherwise, the term "a" as used in this specification and claims shall be understood as "at least one".

[0381] The phrase “and / or” as used in this specification and claims should be understood to mean “any one or both” of the elements so combined, that is, elements that exist together in some cases and separately in others.

[0382] The term "circuit" as used in this specification and claims should be understood to mean any type of circuit, whether considered alone or in combination with other circuits, including, for example, analog circuits, digital electronic circuits, and optical circuits. Among other things, optical circuits include solid-state optical elements and combinations thereof, and free-space optical elements and combinations thereof.

[0383] As used herein in the specification and claims, the phrase "at least one" when referring to a list of one or more elements should be understood to mean at least one element selected from any one or more elements in the list, but does not necessarily include at least one of every element specifically listed in the list, nor exclude any combination of elements in the list. This definition also allows for the optional presence of elements other than those specifically identified in the list of elements referred to by the phrase "at least one," whether or not these elements are related to those specifically identified elements.

[0384] The terms "approximately" and "about" can be used in some embodiments to mean within ±20% of the target value, in some embodiments to mean within ±10% of the target value, in some embodiments to mean within ±5% of the target value, and in some embodiments to also mean within ±2% of the target value. The terms "probably" and "approximately" can include the target value.

Claims

1. A hybrid analog-digital processor, comprising: The circuitry includes an analog processor, wherein the circuitry is configured to perform mathematical operations using a plurality of paths, wherein for each of the plurality of paths, the circuitry is configured to: Based on a set of parameters representing a portion of an arbitrary matrix, one or more scaling factors are determined for the path, wherein the one or more scaling factors are configured to scale the data based on the dynamic range of the simulation processor. At least some parameters of the parameter set are scaled based on one or more scaling factors to produce a scaled parameter set; The simulation processor is programmed based on the scaled parameter set; Generate multiple input analog signals based on the input dataset; Multiple output analog signals are generated based on the multiple input analog signals and the scaled parameter set; A partial output dataset is generated based on the multiple output analog signals; as well as The partial output dataset is scaled based on one or more scaling factors to produce a scaled partial output dataset. The circuit is further configured to generate an accumulated output dataset by accumulating the scaled partial output dataset generated by at least two of the plurality of paths, wherein the accumulated output dataset represents the result of the mathematical operation.

2. The hybrid analog-digital processor according to claim 1, wherein, Generating multiple output analog signals based on the multiple input analog signals and the scaled parameter set includes performing matrix and matrix multiplication based on the multiple input analog signals and the scaled parameter set.

3. The hybrid analog-digital processor according to claim 1, wherein, Generating multiple output analog signals based on the multiple input analog signals and the scaled parameter set includes performing convolution based on the multiple input analog signals and the scaled parameter set.

4. The hybrid analog-digital processor according to claim 1, wherein, Programming the analog processor based on the scaled parameter set includes setting corresponding gains or attenuations for multiple analog amplifiers or attenuators of the analog processor based on the scaled parameter set.

5. The hybrid analog-digital processor according to claim 1, wherein, The simulation processor includes a photonic processor, which includes a plurality of programmable photonic devices, and programming the simulation processor based on the scaled parameter set includes setting corresponding characteristics for the plurality of programmable photonic devices based on the scaled parameter set.

6. The hybrid analog-digital processor according to claim 5, wherein, The programmable photonic device includes a Mach-Zehnder interferometer, and wherein setting corresponding characteristics for the plurality of programmable photonic devices based on the scaled parameter set includes: Based on the scaled parameter set, the corresponding optical properties are set for the plurality of Mach-Zehnder interferometers.

7. The hybrid analog-digital processor according to claim 1, wherein, The scaled partial output dataset, generated by summing at least two of the plurality of pathways, includes: For at least some paths, the scaled partial output dataset generated by the path is added to the scaled partial output dataset generated by the previous path.

8. A hybrid analog-to-digital processor configured to perform mathematical operations, comprising: A circuit including an analog processor and an analog scaling unit, wherein the circuit is configured to: Generate multiple input analog signals based on the input dataset; The gain of the analog scaling unit is set based on one or more scaling factors, wherein the one or more scaling factors are configured to scale the data based on the dynamic range of the analog processor; The simulation processor is programmed using a set of parameters representing arbitrary matrices; Multiple output analog signals are generated based on the multiple input analog signals and the parameter set; Multiple amplified or attenuated output analog signals are generated by amplifying or attenuating the multiple input analog signals and / or multiple output analog signals using the analog scaling unit. as well as An output dataset is generated based on the multiple amplified or attenuated output analog signals.

9. The hybrid analog-digital processor according to claim 8, wherein, The analog scaling unit includes an analog amplifier or an attenuator.

10. The hybrid analog-digital processor according to claim 8, wherein, The hybrid analog-to-digital processor is further configured to perform multi-path computation based on the mathematical operation, wherein the hybrid analog-to-digital processor is further configured to: During the first path of the multi-path computation, the gain of the analog scaling unit is set to a first value; and During the second path of the multi-path computation, the gain of the analog scaling unit is set to a second value that is different from the first value.

11. The hybrid analog-digital processor according to claim 8, wherein, Generating multiple output analog signals based on the multiple input analog signals and the parameter set includes: performing matrix and matrix multiplication based on the multiple input analog signals and the parameter set.

12. The hybrid analog-digital processor according to claim 8, wherein, Generating multiple output analog signals based on the multiple input analog signals and the parameter set includes: performing convolution based on the multiple input analog signals and the parameter set.

13. The hybrid analog-digital processor according to claim 8, wherein, The circuit includes multiple analog-to-digital converters (ADCs), and the multiple ADCs are configured to generate the output dataset based on the multiple output analog signals.

14. The hybrid analog-digital processor of claim 13, wherein, The plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

15. The hybrid analog-digital processor according to claim 8, wherein, The circuit is also configured to determine the one or more scaling factors based on the parameter set and the input dataset.

16. A method for performing mathematical operations, the method comprising: Generate multiple input analog signals based on the input dataset; The gain of the analog scaling unit is set based on one or more scaling factors, wherein one or more scaling factors are configured to scale the data based on the dynamic range of the analog processor; The simulation processor is programmed using a set of parameters representing arbitrary matrices; Multiple output analog signals are generated based on the multiple input analog signals and the parameter set; Multiple amplified or attenuated output analog signals are generated by amplifying or attenuating the multiple input analog signals and / or multiple output analog signals using the analog scaling unit. as well as An output dataset is generated based on the multiple amplified or attenuated output analog signals.

17. The method according to claim 16, wherein, The analog scaling unit includes an analog amplifier or an attenuator.

18. The method according to claim 16, wherein, The method further includes performing multi-path computation based on the mathematical operation using a hybrid analog-digital processor, wherein the hybrid analog-digital processor includes circuitry further configured to: During the first path of the multi-path computation, the gain of the analog scaling unit is set to a first value; and During the second path of the multi-path computation, the gain of the analog scaling unit is set to a second value that is different from the first value.

19. The method of claim 16, wherein, Generating multiple output analog signals based on the multiple input analog signals and the parameter set includes: performing matrix and matrix multiplication based on the multiple input analog signals and the parameter set.

20. The method of claim 16, wherein, Generating multiple output analog signals based on the multiple input analog signals and the parameter set includes: performing convolution based on the multiple input analog signals and the parameter set.

21. The method according to claim 18, wherein, The circuit includes multiple analog-to-digital converters (ADCs), and the multiple ADCs are configured to generate the output dataset based on the multiple output analog signals.

22. The method according to claim 21, wherein, The plurality of ADCs includes an n-bit ADC, where n is equal to or less than 12.

23. The method of claim 16, further comprising determining the one or more scaling factors based on the parameter set and the input dataset.

Images