Method and system for performing convolution using an optical network - Patents.com

Optical networks are used to perform convolution operations in the Fourier domain, addressing inefficiencies in CNNs by reducing computational load and memory requirements through discrete Fourier transforms.

JP2024541101AActive Publication Date: 2024-11-06RAYTHEON CO
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024530409
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-11-29
Filing Date
2022-11-21
Publication Date
2024-11-06
Estimated Expiration
2042-11-21

AI Technical Summary

Technical Problem

Existing convolutional neural networks (CNNs) face inefficiencies in terms of data size and computational load due to the use of Toeplitz matrices, leading to increased memory requirements and power consumption.

Method used

Perform convolution operations using optical networks, specifically through discrete Fourier transforms, to reduce computational load and memory requirements by transforming data into the Fourier domain and utilizing optical systems for efficient matrix convolutions.

Benefits of technology

This approach significantly reduces computational complexity and memory usage while maintaining efficient processing of convolution operations, particularly in CNNs, by leveraging optical systems for Fourier domain transformations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541101000001_ABST
    Figure 2024541101000001_ABST
Patent Text Reader

Abstract

The apparatus includes a frequency comb (505a) configured to generate a first carrier signal (505) having at least one first frequency interval. The apparatus also includes a plurality of modulators (345) configured to modulate one of the amplitudes of the first carrier signal to generate a modulated carrier signal. The apparatus further includes a two-dimensional array of optical couplers (330) configured to perform a one-dimensional discrete Fourier transform in a first direction using the modulated carrier signal. The apparatus also includes an array of coherent detectors (625a-625n) and a first demultiplexer (605) optically coupled to an output of the array of optical couplers and to the coherent detector. The apparatus further includes a local oscillator (LO) bank or array (515a) configured to generate a second carrier signal (515) having at least one second frequency interval different from the at least one first frequency interval. In addition, the apparatus includes a second demultiplexer (615) optically coupled to the output of the LO bank or array and to the coherent detector.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure is directed generally to optical systems and, more particularly, to methods and systems for performing convolution using optical networks. [Background technology]

[0002] Convolutional neural networks (CNNs) have applications in a variety of imaging, recommendation, language processing, and other systems. Many convolutional neural networks employ matrix-based convolution to simplify hardware design. In these approaches, the convolution kernels (weight maps) and input feature maps can be converted to Toeplitz matrices and vectors, and multiplication of the Toeplitz matrices and vectors obtains the desired convolution sum. However, these matrix-based convolution techniques are generally insufficient in terms of data size. For example, for the same weight map and input feature map of size N×N, the memory required to store the Toeplitz matrices is 2N 2 From N 4 Reducing the size of the Toeplitz matrix may result in increased power consumption due to duplication and distribution of large input data blocks at high refresh rates. Summary of the Invention

[0003] The present disclosure relates to methods and systems for performing convolution using optical networks.

[0004] In a first embodiment, the apparatus includes a frequency comb configured to generate a plurality of first carrier signals having at least one first frequency interval. The apparatus also includes a plurality of modulators configured to modulate the first carrier signals, each of which is configured to modulate a corresponding one of the first carrier signals based on a time sequence of values ​​from a corresponding portion of the matrix. In addition, the apparatus includes an array of optical couplers configured to perform a one-dimensional (1D) discrete Fourier transform of the portion of the matrix using the modulated first carrier signals, the array of optical couplers configured to output a time sequence of 1D Fourier coefficients for each of the time sequence of values ​​from the corresponding portion of the matrix.

[0005] In a second embodiment, the apparatus includes a frequency comb configured to generate a first carrier signal having at least one first frequency interval. The apparatus also includes a plurality of modulators configured to modulate one of the amplitudes of the first carrier signal to generate a modulated carrier signal. The apparatus further includes a two-dimensional (2D) array of optical couplers configured to perform a one-dimensional discrete Fourier transform in a first direction using the modulated carrier signal. The apparatus also includes an array of coherent detectors and a first demultiplexer optically coupled to an output of the array of optical couplers and to the coherent detector. The apparatus further includes a local oscillator (LO) bank or array configured to generate a second carrier signal having at least one second frequency interval different from the at least one first frequency interval. In addition, the apparatus includes a second demultiplexer optically coupled to an output of the LO bank or array and to the coherent detector.

[0006] In a third embodiment, a method includes obtaining an input feature map and generating a 2D discrete Fourier transform of the input feature map using an optical network to produce a Fourier-space input feature map. The method also includes obtaining a Fourier-space weight map based on the weight map and performing a Hadamard multiplication of the Fourier-space input feature map and the Fourier-space weight map.

[0007] Other technical features may be readily apparent to those skilled in the art from the following drawings, descriptions, and claims.

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: [Brief description of the drawings]

[0009] [Figure 1] 1 illustrates an example process that performs convolution in the Fourier domain. [Diagram 2] 1 illustrates an example process for performing a hybrid two-dimensional discrete Fourier transform according to the present disclosure. [Figure 3A] 1 illustrates an example mathematical structure for computing a one-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 3B] 1 illustrates an example optical system implementing a one-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 4A] 1 illustrates an example two-dimensional discrete Fourier transform calculation using an optical system according to the present disclosure. [Figure 4B] 1 illustrates an example frequency spectrum associated with a two-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 5A] 1 illustrates another example two-dimensional discrete Fourier transform calculation using an optical system according to the present disclosure. [Figure 5B] 1 illustrates an example frequency spectrum associated with a two-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 5C]1 illustrates an example frequency spectrum associated with a two-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 5D] 1 illustrates an example frequency spectrum associated with a two-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 5E] 1 illustrates an example frequency spectrum associated with a two-dimensional discrete Fourier transform in accordance with the present disclosure. [Figure 6] 1 illustrates an example implementation of a column Discrete Fourier Transform block in accordance with the present disclosure. [Figure 7] 1 illustrates an implementation of a column Discrete Fourier Transform block incorporating weighting in accordance with the present disclosure. [Figure 8] 1 illustrates an example method for performing convolution using an optical network according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1A-8 described below and the various embodiments used to illustrate the principles of the present disclosure are merely exemplary and should not be construed as limiting the scope of the present disclosure. Those skilled in the art will understand that the principles of the present disclosure may be implemented in any type of suitably arranged device or system.

[0011] As explained above, convolutional neural networks (CNNs) have applications in a variety of imaging, recommendation, language processing, and other systems. Many convolutional neural networks employ matrix-based convolution to simplify hardware design. In these approaches, the convolution kernels (weight maps) and input feature maps can be converted to Toeplitz matrices and vectors, and multiplication of the Toeplitz matrices and vectors obtains the desired convolution sum. However, these matrix-based convolution techniques are generally insufficient in terms of data size. For example, for the same weight map and input feature map of size N×N, the memory required to store the Toeplitz matrices is 2N 2 From N 4Reducing the size of the Toeplitz matrix may result in increased power consumption due to duplication and distribution of large input data blocks at high refresh rates.

[0012] The present disclosure provides various methods and systems for performing signal processing and other operations involving matrix convolution. As described in more detail below, the methods and systems enable the determination of the convolution of two matrices. As an example, embodiments of the present disclosure provide methods and systems for performing convolution using optical networks. The methods and systems described herein apply to various computing systems, including systems useful in image processing, pattern analysis, signature recognition, recommendations, language processing, and the like. Various benefits or advantages can be achieved using the methods and systems described in the present disclosure compared to previous approaches. For example, embodiments of the present disclosure can calculate matrix convolution with significantly reduced computational load and memory requirements by using an optical system to perform a Fourier transform in the optical domain. Additional features, benefits, and advantages of various embodiments of the present disclosure are described in more detail below.

[0013] It may often be assumed in the following discussion that the optical system is used to perform convolutions on a convolutional neural network, and it should be noted that this is for illustrative and explanatory purposes only. The methods and systems described in this disclosure may be used to perform any desired convolution of a matrix for any suitable purpose. As a result, the methods and systems described in this disclosure may be used to perform convolutions of matrices on a convolutional neural network, and the methods and systems described in this disclosure may be used in any other suitable application.

[0014] FIG. 1 illustrates an example process 100 for performing convolution in the Fourier domain. In other words, the process 100 of FIG. 1 is used to perform convolution in the frequency domain. As shown in FIG. 1, the process 100 for performing convolution generally operates using an input feature map 105 and a weight map 110. The input feature map 105 generally represents a collection of features associated with input data to be processed, such as features of one or more images, audio samples, or other input data. The input feature map 105 is often generated using a feature extractor, which may represent at least a portion of a machine learning model trained to extract features determined to be relevant for a particular task. The weight map 110 generally represents a collection of weights to be applied to the input feature map 105. The weights in the weight map 110 may be determined in any suitable manner, such as when the weights are determined during training of the machine learning model.

[0015] As can be seen in FIG. 1 , the input feature map 105 may be zero padded to form a zero padded input feature map 115, and the weight map 110 may be zero padded to form a zero padded weight map 120. The zero padding operation may involve adding additional entries that encompass "zero" values ​​around the entries of the input feature map 105 and the weight map 110. Notably, zero padding may be used to address aliasing resulting from the Fourier transform process, since the input feature map 105 and the weight map 110 are not assumed to be periodic. Zero padding may increase the data size of the zero padded input feature map 115 and the zero padded weight map 120 relative to the original input feature map 105 and the weight map 110. However, the impact is generally small for the small weight map 110 (compared to conventional convolution methods using Toeplitz matrices and matrix multiplication), and may not be implemented if boundaries are ignored.

[0016] The zero-padded input feature map 115 and the zero-padded weight map 120 may be fed to a first two-dimensional (2D) discrete Fourier transform (DFT) unit 125 and a second 2D discrete Fourier transform unit 130, respectively. The discrete Fourier transform units 125 and 130 operate to transform the zero-padded input feature map 115 and the zero-padded weight map 120 into the Fourier domain (frequency domain), respectively. It will be noted that in some cases, the 2D discrete Fourier transform performed on the weight map 110 or 120 may be performed digitally and stored in memory. This is because for many applications, the weight matrix is ​​identical for different input feature maps 105. In those embodiments, the results of the 2D discrete Fourier transform performed on the weight map 110 or 120 may be retrieved from memory and used whenever additional input feature maps 105 are obtained and processed.

[0017] Once the zero-padded input feature map 115 and the zero-padded weight map 120 are transformed into the Fourier domain, the maps (now in the frequency domain) may be fed into a Hadamard multiplier 135. The Hadamard multiplier 135 generally operates to perform an element-wise multiplication of two frequency domain maps. To transform the results produced by the Hadamard multiplier 135 back into the spatial domain, the output of the Hadamard multiplier 135 may be fed into a 2D inverse discrete Fourier transform (IDFT) unit 140. The inverse discrete Fourier transform unit 140 transforms the results produced by the Hadamard multiplier 135 into the spatial domain. The output of the inverse discrete Fourier transform unit 140 is an output matrix 145 that represents the convolution of the input feature map 105 and the weight map 110.

[0018] In this manner, the process 100 illustrated in FIG. 1 can be used to implement convolution in the Fourier domain. Performing the convolution in the Fourier (frequency) domain can help reduce the complexity of the convolution compared to the traditional matrix multiplication approach. For example, by performing the convolution in the Fourier domain, the circular shift multiply-add operations in the convolution become Hadamard multiplication (element-wise multiplication) operations in the Fourier domain. However, performing a 2D discrete Fourier transform using the discrete Fourier transform units 125 and 130 and performing a 2D inverse discrete Fourier transform using the inverse discrete Fourier transform unit 140 is computationally expensive using conventional techniques. Even approaches using graphic processing units (GPUs), which may typically be implemented using application specific integrated circuits (ASICs), may suffer from high computational loads. To implement the Fourier domain convolution in a more efficient manner, the elements of the process 100 illustrated in FIG. 1 can be performed using one or more optical systems, which results in a reduction in computational expenditures and memory resources.

[0019] FIG. 2 illustrates an example process 200 for performing a hybrid 2D discrete Fourier transform according to the present disclosure. As shown in FIG. 2, the 2D discrete Fourier transform can be implemented using two cascaded one-dimensional (1D) discrete Fourier transforms. In this example, the input feature map 205 (which may represent the input feature map 105 or the zero-padded input feature map 115 described above) is a matrix having a size of M×N, which means that the input feature map 205 has M rows and N columns of matrix elements. The input feature map 205 here can be decomposed into a set of column-wise vectors 210. That is, each column of the input feature map 205 can be used to form a separate vector 210 in the set of column-wise vectors 210.

[0020] A 1D Discrete Fourier Transform (such as an M-point 1D Discrete Fourier Transform) may be performed across one dimension of the input feature map 205, such as when a first 1D Discrete Fourier Transform is performed on each vector 210 in the set of column-wise vectors 210. Thus, the first 1D Discrete Fourier Transform is denoted in FIG. 2 as an “M-point DFT 1D " where the values ​​in a given column are used to perform an M-point discrete Fourier transform, which may also be referred to as a "row DFT" since the values ​​in all rows of a given column are used in the discrete Fourier transform. The output of the first 1D discrete Fourier transform may be arranged as a set of row-oriented vectors 215. For example, the first element in each of the column vectors obtained as the output of the first 1D discrete Fourier transform may be grouped to be the first row-oriented vector 215, and subsequent row-oriented vectors 215 may be obtained by grouping the n-th element in each of the column vectors obtained as the output of the first 1D discrete Fourier transform.

[0021] A second 1D Discrete Fourier Transform (such as an N-point 1D Discrete Fourier Transform) may be performed across two dimensions of the set of row-oriented vectors 215, such as when a second 1D Discrete Fourier Transform is performed for each vector 215 in the set of row-oriented vectors 215. Thus, the second 1D Discrete Fourier Transform is referred to in FIG. 2 as an “N-point DFT 1D " and the values ​​in a given row are used to perform an N-point Discrete Fourier Transform. This completes the generation of the 2D Discrete Fourier Transform of the input feature map 205.

[0022] It is possible to perform a hybrid 2D discrete Fourier transform in the optical domain based on the process 200 illustrated in Figure 2. As discussed more fully below, one or more optical systems can be used to perform a 1D discrete Fourier transform on the M rows of the input feature map 205 as well as a 1D discrete Fourier transform on the N columns of the input feature map 205. This type of approach can help make matrix convolution decisions much more efficient in terms of processing and memory resources.

[0023] 3A and 3B illustrate an example mathematical structure 300 for computing a 1D discrete Fourier transform and an example optical system 320 for implementing the 1D discrete Fourier transform, according to the present disclosure. As shown in FIG. 3A, the mathematical structure 300 represents a 1D discrete Fourier transform performed using an input vector 305. In some cases, the input vector 305 may represent a column-wise vector 210 associated with the input feature map 205 to be processed. A 1D discrete Fourier transform using a total of M points can be expressed as a unitary matrix 310 (which may also be referred to as a unitary transform) of size M×M. Thus, the size of the unitary matrix 310 may be defined by the length M of the input vector 305, as opposed to being defined by the values ​​associated with the elements of the input vector 305. The unitary matrix 310 may be applied to the input vector 305 to generate an output vector 315, which is the output of the 1D discrete Fourier transform applied to the input vector 305. In some cases, the output vector 315 may represent a row-oriented vector 215, where the output vector 315 is a Fourier domain representation of the input vector 305.

[0024] As illustrated in FIG. 3B, the unitary matrix 310 may be mapped to an optical system 320. As shown in FIG. 3B, the optical system 320 is configured to implement the 1D discrete Fourier transform illustrated in FIG. 3A. The optical system 320 may be a passive optical network including a plurality of input ports 325, a plurality of optical couplers 330 coupled to the plurality of input ports 325, and a plurality of output ports 335 coupled to the plurality of optical couplers 330. In some examples, the optical system 320 may be referred to as a "combination network." The optical system 320 includes a predetermined or otherwise defined number of optical couplers 330 consisting of a plurality of optical couplers 330. The optical couplers 330 here are arranged to form a 2D array of optical couplers 330. The size of the unitary matrix 310 is defined by the length M of the input vector 305, and the size of the optical network in the optical system 320 may be the same for different input vectors 305 having elements with different values. In some embodiments, the number of optical couplers 330 in optical system 320 is equal to (M-1) / 2xM. Also, in some embodiments, optical couplers 330 are 50 / 50 optical couplers that utilize evanescent wave coupling or multimode interference, or the like.

[0025] The embodiments of the present disclosure provide several options for changing the input size utilized by a passive optical network, such as by downsizing the size of the DFT matrix. As an example, zero padding as illustrated in FIG. 1 can be utilized to change the input size of the input matrix. As another example, the coupling ratios between the optical couplers 330 in the plurality of optical couplers 330 can be reconfigured to result in an input matrix of smaller size. In some cases, the reconfiguration can include reducing the coupling ratios to zero. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0026] The optical system 320 may receive optical signals from a light source 340, which is optically coupled to the multiple input ports 325 using corresponding waveguides. In some embodiments, the light source 340 represents at least one laser, light emitting diode (LED), pulsed light source, or the like. To load element values ​​of the input vector 305 at the multiple input ports 325, the element values ​​(x[0,0], x[1,0], x[2,0], ..., x[M-1,0]) of the input vector 305 are used to control a corresponding modulator 345. Each modulator 345 controlled by the corresponding element value (x[0,0], x[1,0], x[2,0], ..., x[M-1,0]) modulates the optical intensity (representing a carrier signal) from the corresponding waveguide through the multiple optical couplers 330. For example, each modulator 345 may be used to modulate the amplitude and phase of the optical intensity (carrier signal) provided to the modulator 345. Thus, the electrical signal corresponding to the input vector 305 may be used to modulate the optical signal provided by the light source 340. In some embodiments, the electrical signal corresponding to the input vector 305 may be pulsed to prevent inter-sample interference, where the pulse duration of the electrical signal is shorter than the sample period of the electrical signal. Also, in some embodiments, the light source 340 may be pulsed to prevent inter-sample interference, where the pulse width of the optical intensity is shorter than the sample period of the electrical signal. As described more fully below, the output of each of the plurality of optical couplers 330 is an optical signal provided to a demultiplexer. In some embodiments, the electrical output is provided by an array of coherent receivers, resulting in a system having electrical inputs and electrical outputs.

[0027] In some cases, the input ports 325 may receive fixed and equal optical intensities from the light source 340. As a particular example, a 105 milliwatt laser or LED may be used as the light source 340 to provide 1 milliwatt to each of the multiple input ports 325. In this particular example, modulation of the optical intensity incident on each modulator 345 may result in an output from each modulator 345 ranging from 0 milliwatts to 1 milliwatt depending on the value of the element of the input vector 305 associated with that particular modulator 345. The optical intensity exiting each of the modulators 345 corresponds to the value of the element of the input vector 305 associated with that particular modulator 345. Thus, the values ​​of the elements of the input vector 305 are loaded into the optical network of the optical system 320, which performs a 1D discrete Fourier transform.

[0028] The optical signals exiting the plurality of optical couplers 330 have output values ​​X 0 [0], X 1 [0], X 2 [0], …, X M-1 The input vector 305 is represented by a [0], whose values ​​represent the values ​​of the input vector 305 after transformation into the Fourier domain. As a result, the optical signals present at the multiple output ports 335 can be accessed to retrieve the output vector 315, which is the Fourier transform of the input vector 305.

[0029] 2 can be computed using the optical system 320 by loading the element values ​​of the first column-oriented vector 210 illustrated in FIG. 2 into the modulators 345 associated with the corresponding elements of the column-oriented vector 210. To compute the 1D discrete Fourier transforms of the remaining column-oriented vectors 210, the element values ​​of each remaining column-oriented vector 210 are loaded into the modulators 345 associated with the corresponding elements of each remaining column-oriented vector 210. Thus, after N cycles for the M×N input feature map 205 shown in FIG. 2, the M-point DFT in FIG. 1D2 can be calculated using optical system 320 by loading the values ​​of the elements of the first row-oriented vector 215 illustrated in FIG. 2 into modulators 345 associated with corresponding elements of row-oriented vector 215. To calculate the 1D discrete Fourier transforms of the remaining row-oriented vectors 215, the values ​​of the elements of each remaining row-oriented vector 215 are loaded into modulators 345 associated with corresponding elements of each remaining row-oriented vector 215.

[0030] In this manner, a physical implementation of a 1D discrete Fourier transform is provided in which the optical system 320 computes the discrete Fourier transform of values ​​that are synchronously loaded using a modulator 345 coupled to multiple optical couplers 330. As light propagates through multiple optical couplers 330, complex (real and imaginary) discrete Fourier transform coefficients (X 0 [0], X 1 [0], X 2 An array of weights (e.g., [0]) is computed in the form of optical signals present at multiple output ports 335. As discussed above, embodiments of the present disclosure utilize a physical network (e.g., multiple optical couplers 330) that remains static independent of the values ​​of the elements of the input vector 305. Thus, as the weights change, the strength of the optical signal output of the modulator 345 changes, but the size and layout of the multiple optical couplers 330 remain unchanged. A 1D discrete Fourier transform is computed on the input feature map by loading columns or rows of the input feature map into the optical system 320.

[0031] 4A and 4B illustrate an example 2D discrete Fourier transform computation using optical system 400 and an example frequency spectrum 420 associated with the 2D discrete Fourier transform, according to the present disclosure. As shown in FIG. 4A, values ​​of elements (e.g., column oriented vector 210) in a column of an input feature map (e.g., input feature map 205) can be loaded serially using modulator 345, which may occur in the same or similar manner as discussed above with reference to FIG. 3B. As a result, a time sequence of element values ​​in a time sequence set 405 can be provided to each modulator 345. The values ​​of elements in a column of the input feature map are converted to a time sequence T 0 M values ​​in the first column of the time sequence set 405 at time T 1 M values ​​in the second column of the time sequence set 405 at N-1 This is exemplified by the M value in the Nth column of the time sequence set 405 in

[0032] As can be seen in FIG. 4A , to compute a discrete Fourier transform over the rows of a 2D input feature map (such as input feature map 105 or zero-padded input feature map 115 of FIG. 1 ), samples for rows in each column of the input feature map are calculated over a sampling period T S The values ​​in the time sequence set 405 are fed into multiple input ports of a row DFT block 410, which here represents the optical network formed by the various optical couplers 330 shown in FIG. 3B and described above. Each period T S After that, 1D discrete Fourier transform coefficients along each column of the input feature map are generated. This produces a time sequence of 1D Fourier coefficients for each column in the time sequence set 415. The time sequence of 1D Fourier coefficients in the time sequence set 405 is 0 , T 1 , T 2 , …, T N-1 It is exemplified as being associated with.

[0033] Each time sequence of 1D Fourier coefficients in the time sequence set 415 of 1D Fourier coefficients is expressed as a function y 0 (t), the Fourier transform of the time series gives the frequency spectrum Y 0 (ω) 420. The frequency spectrum 420 is illustrated in FIG. 4B. Thus, each time sequence frequency spectrum 420 in the time sequence set 415 represents a Fourier transform along two dimensions (e.g., a row of the input feature map). Thus, by computing the short-time Fourier transform of the time sequence, the 2D discrete Fourier transform of the input feature map is computed.

[0034] 3B and 4A, a single light source 340 is illustrated, and it will be noted that the single light source 340 may in some cases provide input light at a single wavelength or within a single narrow wavelength range, however, as described below, a multi-wavelength source may also be utilized to perform a short-time Fourier transform of a time sequence.

[0035] 5A-5E illustrate another example 2D discrete Fourier transform computation and associated frequency spectra 520a-520m using optical system 500 according to the present disclosure. As shown in FIG. 5A, rather than digitally computing the short-time Fourier transform (which may increase power consumption), the frequency spectrum Y 0 (ω) to function y 0A second 1D discrete Fourier transform of the 1D Fourier coefficients represented by (t) can be performed by physical Fourier decomposition via coherent detection using a phase-coherent carrier array. In this example, for an M×N input feature map (such as the input feature map 105 or the zero-padded input feature map 115), an N-tone carrier array 505 is used, which in some cases may be provided by a frequency comb 505a with a carrier frequency spacing equal to Δ. That is, the frequency comb 505a may generate N optical signals with frequencies equally spaced by Δ (it is possible for the combs to have different frequency spacings). The carrier array 505 / frequency comb 505a here may be used as an input light source instead of the light source 340 discussed above. Each carrier signal in the N-tone carrier array 505 may be used to modulate the same input value (such as the same element of the input feature map column vector 210), thereby replicating the discrete Fourier transform coefficients calculated in the row DFT block 410 into N spectrally distinct copies. As shown in FIG. 5B, for example, six copies (of the N copies) of the frequency spectrum 420 are illustrated as a frequency domain signature (frequency spectrum 520a). In this example, the spacing between adjacent pairs of each of the six copies is equal to the carrier frequency spacing Δ.

[0036] Because the M values ​​for the input vector are input synchronously, the discrete Fourier transform coefficients calculated to yield each time sequence in the time sequence set 415 are generated synchronously. Thus, as shown in FIGS. 5C to 5E, 1 (nT s ) are included in the frequency domain signature (frequency spectrum 520b), and six copies of the frequency spectrum corresponding to X 2 (nT s ) are included in the frequency domain signature (frequency spectrum 520c), and six copies of the frequency spectrum corresponding to X M-1 (nT s) are included in a frequency-domain signature (frequency spectrum 520m), where N spectrally distinct copies of the frequency spectrum for each time sequence of discrete Fourier transform coefficients are provided in a set of M frequency-domain signatures 520a through 520m.

[0037] Referring again to FIG. 5A, the column DFT block 510 S A coherent detector array (such as a slow coherent detector array) having a bandwidth of δ=N / T from a local oscillator (LO) array or bank 515a can be used to implement a coherent detector array (such as a slow coherent detector array) having a bandwidth of δ=N / T S A column DFT block 510 is used with a phase coherent carrier array 515 having an offset frequency spacing of Δ+δ. Thus, the optical signals in the phase coherent carrier array 515 have frequencies equally spaced by Δ+δ (although it is possible for the optical signals to have different frequency spacings). In this arrangement, the output of each coherent detector set represents the short-time Fourier transform coefficients of the discrete Fourier transform time sequence, thereby resulting in the computation of a second 1D discrete Fourier transform, yielding as output the 2D discrete Fourier transform of the input feature map.

[0038] As an example of this, as shown in FIG. 5B, a first frequency 525 provided in the phase coherent carrier array 515 from the LO array or bank 515a can be used to sample a first frequency spectrum 530 of N spectrally distinct copies of the frequency spectrum (spectrum 520a) present in the first frequency domain signature, a second frequency 535 can be used to sample a second frequency spectrum 540 of N spectrally distinct copies of the frequency spectrum (spectrum 520a) present in the first frequency domain signature, and so on. Thus, since the carrier frequency spacing between the carriers provided by the LO array or bank 515a is equal to Δ+δ, the frequencies provided by the LO array or bank 515a can be used to sample the frequency spectrum across the width of the frequency spectrum. Thus, the output of the column discrete Fourier transform block 510 is a sampled set of Fourier coefficients. Thus, the Fourier coefficients sampled using the LO array or bank 515a are the desired short-time Fourier transform coefficients.

[0039] In the embodiment shown in FIGS. 4A and 4B, the sampled output of the first frequency domain signature is represented by a frequency spectrum Y 0 5A corresponds to (ω). Because all M columns are loaded synchronously, the frequency spectrum associated with each of the M columns is generated synchronously in FIG. 5A, resulting in the 2D discrete Fourier transform of the input feature map as output. It will be noted that the column DFT block 510 follows the row DFT block 410 here, and the positions of the DFT blocks 410 and 510 are reversed, effectively swapping the order of the row DFT and column DFT. As a result, the order of operations can be modified, and the particular implementation illustrated in FIG. 5A is just one possible example implementation.

[0040] FIG. 6 illustrates an implementation of an embodiment of a sequence DFT block 510 according to the present disclosure. As discussed above in connection with FIG. 5A, the sequence DFT block 510 is used to perform a second 1D discrete Fourier transform (or a first 1D discrete Fourier transform if the DFT blocks 410 and 510 are reversed). As shown in FIG. 6, a time sequence 600 (which may represent one of the time sequences in the time sequence set 415) may be fed to a demultiplexer (DMUX) 605 to be split into N spectrally distinct copies (such as the N spectrally distinct copies included in the frequency spectrum 520a shown in FIG. 5B). Thus, each of the spectrally distinct copies illustrated by the first frequency spectrum 530 and the second frequency spectrum 540 in FIG. 5B is demultiplexed to provide a separate copy of the frequency spectrum.

[0041] The output of the demultiplexer 605 (a spectrally distinct copy of the frequency spectrum 420) is provided as an input to a coherent detector array 620. As illustrated in FIG. 6, the coherent detector array 620 includes a plurality of coherent detectors 625a-625n, which may also be referred to as coherent receivers. The signal 610 from the LO array or bank 515a (such as the phase coherent carrier array 515) may be fed to a demultiplexer 615, which may feed a first frequency provided by the LO array or bank 515a to a first coherent detector 625a, a second frequency provided by the LO array or bank 515a to a second coherent detector 625b, an Nth frequency provided by the LO array or bank 515a to an Nth coherent detector 625n, and so on. 5B by first frequency 525 and second frequency 535 can be demultiplexed to provide frequencies present in phase coherent carrier array 515. In some cases, the number of carrier signals in N-tone carrier array 505, the number of carrier signals in phase coherent carrier array 515, and the number of coherent detectors 625a-625n in coherent detector array 620 are equal.

[0042] Each of the coherent detectors 625a-625n in the coherent detector array 620 can multiply the frequency spectrum output by the demultiplexer 605 with the frequencies output by the demultiplexer 615. Because the coherent detectors 625a-625n are band-limited receivers, this multiplication is effectively an integration of the frequency spectrum over a narrow frequency range associated with the particular frequency output by the demultiplexer 615. Thus, as illustrated by frequency spectrum 630a and frequency 635a, frequency spectrum 630b and frequency 635b, and frequency spectrum 630n and frequency 635n, the frequencies provided by the LO array or bank 515a can be used to sample the frequency spectrum 420 across the width of the frequency spectrum 420 and generate short-time Fourier transform coefficients. As shown in Figure 5A, the process in Figure 6 can be repeated M times to complete a 2D discrete Fourier transform of the M x N input feature maps, such that there are a total of M time sequences from row DFT block 410 to column DFT block 510. In some embodiments, the structure in Figure 6 can be replicated M times, so that the M time sequences in the time sequence set 415 can be processed synchronously. In those embodiments, there can be an array of demultiplexers 605, an array of demultiplexers 615, and multiple coherent detector arrays 620.

[0043] 7 illustrates an example implementation of a column DFT block 510 incorporating weighting according to the present disclosure. As an extension of the column DFT block 510 discussed in connection with FIG. 6, a 2D convolution can be further computed within the coherent detector array 620 of the column DFT block 510 by modulating each frequency provided by the LO array or bank 515a with a pre-computed or otherwise defined Fourier equivalent of a weighting matrix (such as the weight map 110 or the zero-padded weight map 120 illustrated in FIG. 1).

[0044] As shown in FIG. 7, the frequencies output by the demultiplexer 615 are multiplied with corresponding weighting factors, thereby multiplying each 2D discrete Fourier transform coefficient with a corresponding weighting factor. In some embodiments, the weighting factors are Fourier space weights. In the example shown in FIG. 7, the first frequency provided by the LO array or bank 515a is multiplied with a first weight W using a first weighting element 710a before delivery to the first coherent detector 625a. 0,0 and the second frequency provided by LO array or bank 515a is weighted by a second weight W using second weighting element 710b prior to delivery to second coherent detector 625b. 0,1 and the Nth frequency provided by LO array or bank 515a is weighted by the Nth weight W using Nth weighting element 710n before delivery to Nth coherent detector 625n. 0,N-1 In some embodiments, weighting elements 710a-710n can be implemented using tunable optical attenuators or phase shifters or the like, or as elements that modify the phase and / or amplitude of the frequencies output by demultiplexer 615. In this manner, the frequencies provided by LO array or bank 515a can be used to sample frequency spectrum 420 across the width of frequency spectrum 420 and to weight the sampling using corresponding weights, thereby reducing the frequency of W 0,0 X 0,0 , W 0,1 X 0,1 , ..., W 0,N-1 X 0,N-1 7 may be replicated M times, so that the M time sequences in the time sequence set 415 may be processed synchronously. In those embodiments, there may be an array of demultiplexers 605, an array of demultiplexers 615, multiple coherent detector arrays 620, and multiple sets of weighting elements 710a-710n. Thus, for example, weights W 1,0 -W1,N-1 is the time series X 1,0 -X 1,N-1 is used by the weight W N-1,0 -W N-1,N-1 is the time series X N-1,0 -X N-1,N-1 Used by.

[0045] Since the weights may be relatively invariant, in some embodiments, a 2D discrete Fourier transform of the weights may be performed once and the results may be stored for later use in various embodiments of the systems described herein. Thus, row DFT block 410 and column DFT block 510 may be utilized to generate a Fourier space weight map, and the coefficients of the Fourier space weight map may be stored for later use. Alternatively, since the weights are relatively invariant, a conventional system may be utilized to calculate the Fourier space weight map, and the Fourier space weight map may be stored in memory. In FIG. 7, the weights W exemplified are 0,0 , W 0,1 , ..., W 0,N-1 corresponds to the Fourier space weight map present at the output of the second 2D discrete Fourier transform unit 130 illustrated in Figure 1. Thus, the output provided by the coherent detector array 620 corresponds to the output of the Hadamard multiplier 135 illustrated in Figure 1.

[0046] The resulting convolution product provided by the coherent detector array 620 may be transformed from Fourier space to real space, such as by passing the output of the coherent detector array 620 through a second optical system 640. The second optical system 640 may be similar to the one described in Figures 5A and 6, such as the compound optical system of Figure 5A without a weighting matrix. The second optical system 640 may perform an inverse two-dimensional discrete Fourier transform, such as by rotating its input accordingly. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0047] As one particular example use case of the described approach, a video stream can be received and processed using embodiments of the present disclosure. A video stream containing rapidly changing data can be received as one or more input feature maps 105. A weight map suitable for image processing, which may contain data that is not rapidly changing, can be received as weight map 110. Using embodiments of the present disclosure, a video stream can be processed at high speed and with low computational complexity using the optical system described above. It will be noted that in addition to image processing applications, other applications utilizing neural networks, convolutional networks, or other convolutions (such as systems supporting voice analysis and signature analysis, etc.) can use various implementations of the embodiments described above.

[0048] The embodiments of the present disclosure can provide significant benefits and advantages over conventional optical matrix multipliers, such as with respect to system size and complexity. For example, for a 64×64 input feature map and a 3×3 weight map, the optical matrix multiplier may utilize a 4,356×4,356 U matrix (corresponding to 9,485,190 tunable optical couplers), a 4,096×4,096 V matrix (corresponding to 8,386,560 tunable optical couplers), and a 64×64 D matrix (corresponding to 64 variable optical attenuators). Assuming an optical coupler size of 15 μm×15 μm limited by bump pitch and a detector size of 50 μm×105 μm, the total size of the optical matrix multiplier is approximately 4,122 mm. 2 The order of magnitude of the

[0049] In some implementations, embodiments of the present disclosure providing the same functionality may implement two 66×66 discrete Fourier transform matrices to perform the discrete Fourier transform and the inverse discrete Fourier transform, respectively. This corresponds to 4,290 fixed optical couplers, four 66×66 channel wavelength division multiplexing (WDM) arrays (two for the discrete Fourier transform and two for the inverse discrete Fourier transform) corresponding to 17,424 ring filters, and a 66×66 weight matrix corresponding to 4,356 complex (e.g., I / Q) modulators. Assuming an optical coupler size of 15 μm×15 μm limited by bump pitch, a ring filter size of 30 μm×10 μm, and a coherent detector size of 105 μm×105 μm, the total size of this implementation is approximately 93.3 mm. 2 In addition to the significant size savings, the power consumption of the embodiments of the present disclosure is much lower than that associated with optical matrix multipliers, because many of the components in the embodiments of the present disclosure are passive.

[0050] Although Figures 1-7 illustrate examples of discrete Fourier transforms, optical systems implementing the discrete Fourier transforms, signal spectra, and related details, various modifications may be made to Figures 1A-7. For example, each of the input feature maps 105, 115, 205 and weight maps 110, 120 may have any suitable dimensions. Also, the optical systems may include any suitable number of each of the components shown in the figures. In addition, in each of the illustrated optical systems, various components may be combined, further subdivided, duplicated, omitted, or rearranged, and additional components may be added, depending on particular needs.

[0051] 8 illustrates an example method 800 of performing convolution using an optical network in accordance with the present disclosure. For ease of explanation, the method 800 may be performed using any of the optical systems described above. However, the method 800 may be performed using any other suitable optical system designed in accordance with the present disclosure.

[0052] As shown in FIG. 8, in step 810, an input feature map is obtained, and in step 815, a weight map is obtained. This may include, for example, receiving, generating, or obtaining an input feature map 105 associated with the input data to be processed, and receiving, generating, or obtaining a weight map 110 to be applied to the input data to be processed. In step 820, a two-dimensional discrete Fourier transform of the input feature map is generated to produce a Fourier-space input feature map. This may include, for example, using any of the optical networks described above to convert the input feature map 105, 205 (or the zero-padded input feature map 115) to a Fourier-space input feature map. In step 825, a two-dimensional Fourier transform of the weight map is generated to produce a Fourier-space weight map. This may include, for example, using any of the optical networks described above to convert the weight map 110 (or the zero-padded weight map 120) to a Fourier-space weight map. In step 830, a Hadamard multiplication of the Fourier-space input feature map and the Fourier-space weight map is performed, which may include, for example, an optical network outputting an optical signal representing a 2D discrete Fourier transform of the input feature map.

[0053] In some embodiments, a second optical network may be used to perform a two-dimensional inverse discrete Fourier transform of the output produced by the Hadamard multiplication in step 835. For example, the optical system 640 may be used to perform a 2D IDFT process. The output of the 2D inverse discrete Fourier transform may result in a convolution of the input feature map 105 and the weight map 110. Also, in some embodiments, zero padding of the input map may be performed to generate the input feature map, and zero padding of the weight map may be performed to generate the weight map (which may be performed before or during steps 810 and 815). Additionally, in some embodiments, in step 825, a Fourier-space weight map may be generated and stored in memory, and the Fourier-space weight map may be retrieved from memory and utilized to weight the local oscillator carrier.

[0054] Although FIG. 8 illustrates one embodiment of a method 800 for performing convolution using an optical network, various modifications may be made to FIG. 8. For example, while shown as a series of steps, various steps in FIG. 8 may overlap, be performed in parallel, be performed in a different order, or be performed any number of times. Moreover, individual steps in FIG. 8 may include multiple sub-steps that may be performed in various sequences as appropriate for the individual step. Additionally, various steps may be added to or removed from method 800 depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0055] The following describes example embodiments of the present disclosure that implement or relate to methods and systems for performing convolution using an optical network, although other embodiments may be used in accordance with the teachings of the present disclosure.

[0056] In a first embodiment, the apparatus includes a frequency comb configured to generate a plurality of first carrier signals having at least one first frequency interval. The apparatus also includes a plurality of modulators configured to modulate the first carrier signals, each of which is configured to modulate a corresponding one of the first carrier signals based on a time sequence of values ​​from a corresponding portion of the matrix. In addition, the apparatus includes an array of optical couplers configured to perform a one-dimensional (1D) discrete Fourier transform of the portion of the matrix using the modulated first carrier signals, the array of optical couplers configured to output a time sequence of 1D Fourier coefficients for each of the time sequence of values ​​from the corresponding portion of the matrix.

[0057] Any single one or any suitable combination of the following features may be used with the first embodiment. The array of optical couplers may be configured to replicate each time sequence of the 1D Fourier coefficients into multiple spectrally distinguishable copies based on different frequencies of a first carrier signal. The apparatus may also include a coherent detector configured to sample the time sequence of the 1D Fourier coefficients based on a second carrier signal, the second carrier signal may have at least one second frequency interval greater than the at least one first frequency interval. The coherent detector may be configured to output short-time Fourier transform coefficients of the time sequence of the 1D Fourier coefficients, the short-time Fourier transform coefficients may represent a 2D discrete Fourier transform of the matrix. The apparatus may also include a first demultiplexer configured to separate the multiple spectrally distinguishable copies of the time sequence of the 1D Fourier coefficients from the array of optical couplers for each time sequence of the 1D Fourier coefficients and provide different ones of the spectrally distinguishable copies to different ones of the coherent detectors. The apparatus may also include a local oscillator (LO) bank or array configured to generate a second carrier signal, and a second demultiplexer configured to separate the second carrier signal for each time sequence of 1D Fourier coefficients from the array of optical couplers and provide a different one of the second carrier signals to a different one of the coherent detectors. The apparatus may also include a weighting element configured to adjust the second carrier signal based on different weights and provide the adjusted second carrier signal to the coherent detector. The matrix may include an input feature map, and the weights may be from a weight map. The time sequence of values ​​from a corresponding portion of the matrix may correspond to a column-wise vector from the matrix, and the time sequence of 1D Fourier coefficients from the array of optical couplers sampled by the coherent detector may include a row-wise vector.

[0058] In a second embodiment, the apparatus includes a frequency comb configured to generate a first carrier signal having at least one first frequency interval. The apparatus also includes a plurality of modulators configured to modulate one of the amplitudes of the first carrier signal to generate a modulated carrier signal. The apparatus further includes a two-dimensional (2D) array of optical couplers configured to perform a one-dimensional discrete Fourier transform in a first direction using the modulated carrier signal. The apparatus also includes an array of coherent detectors and a first demultiplexer optically coupled to an output of the array of optical couplers and to the coherent detector. The apparatus further includes a local oscillator (LO) bank or array configured to generate a second carrier signal having at least one second frequency interval different from the at least one first frequency interval. In addition, the apparatus includes a second demultiplexer optically coupled to an output of the LO bank or array and to the coherent detector.

[0059] Any single one or any suitable combination of the following features may be used with the second embodiment. Each coherent detector may be configured to receive an output of the first demultiplexer and an output of one of the second demultiplexers and generate Fourier coefficients. The array of coherent detectors may be configured to output a second 1D discrete Fourier transform in a second direction to complete a 2D discrete Fourier transform on the input of the device. The optical coupler may include a 50 / 50 optical coupler. The number of first carrier signals, the number of second carrier signals, and the number of coherent detectors in the array of coherent detectors may be equal. The device may also include a weighting element configured to modulate the second carrier signal according to a weighting matrix. The Fourier equivalent weight matrix of the weighting matrix may be pre-calculated and stored in the device.

[0060] In a third embodiment, a method includes obtaining an input feature map and generating a 2D discrete Fourier transform of the input feature map using an optical network to produce a Fourier-space input feature map. The method also includes obtaining a Fourier-space weight map based on the weight map and performing a Hadamard multiplication of the Fourier-space input feature map and the Fourier-space weight map.

[0061] Any single one or any suitable combination of the following features may be used with the third embodiment. The method may also include generating a 2D inverse discrete Fourier transform of an output of the Hadamard multiplication using a second optical network. The output of the 2D inverse discrete Fourier transform may include a convolution of the input feature map and the weight map. The method may also include generating a 2D discrete Fourier transform of the weight map using an optical network to produce a Fourier-spatial weight map, and storing coefficients of the Fourier-spatial weight map. The method may also include zero padding the input map to produce the input feature map and the weight map to provide the weight map. The method may also include receiving a Fourier-spatial weight map, storing the Fourier-spatial weight map, retrieving the Fourier-spatial weight map from the storage device, and utilizing the Fourier-spatial weight map to weight the LO carrier signal.

[0062] In some embodiments, various functions described in this patent document are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium that can be accessed by a computer, such as read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable media excludes wired, wireless, optical, or other communication links that transmit transient electrical or other signals. Non-transitory computer readable media include media on which data can be stored permanently, and media on which data can be stored and later overwritten, such as rewritable optical disks, or erasable storage devices.

[0063] It may be advantageous to provide definitions of certain words and phrases used throughout this patent document. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, procedures, functions, objects, classes, instances, associated data, or portions thereof, adapted for implementation in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "including" and "comprising" and their derivatives are meant to include without limitation. The term "or" is inclusive and / or. The term "associated with" and its derivatives may mean including, contained within, interconnected with, containing, housed within, connected to or with, coupled to or with, communicable with, associated with, interleaved with, juxtaposed to, proximate to, bound to or with, having, having properties of, relating to or with, and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the listed items may be used, and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A and B and C. No description in this disclosure should be read as implying that any particular element, step, or function is an essential or critical element that must be included within the scope of the claim. The scope of patented subject matter is defined solely by the claims as allowed. Moreover, no claim invokes 35 U.S.C. 112(f) with respect to any of the appended claims or claim elements, unless the precise phrase "means for" or "step for" is expressly used in a particular claim, followed by a participial phrase identifying the function.The use of terms such as "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller" in the claims is understood and intended to refer to structures known to those skilled in the art, as further modified or enhanced by the features of the claims themselves (without being limited thereto), and is not intended to invoke 35 U.S.C. § 112(f).

[0064] While this disclosure has described certain embodiments and generally associated methods, alterations and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the disclosure. Other changes, substitutions, and alterations are possible without departing from the spirit and scope of the disclosure, as defined by the following claims.

Claims

1. 1. An apparatus comprising: a frequency comb configured to generate a plurality of first carrier signals having at least one first frequency interval; a plurality of modulators configured to modulate the first carrier signals, each modulator configured to modulate a corresponding one of the first carrier signals based on a time sequence of values ​​from a corresponding portion of a matrix; an array of optical couplers configured to perform a one-dimensional (1D) discrete Fourier transform of a portion of the matrix using the modulated first carrier signal, the array of optical couplers configured to output a time sequence of 1D Fourier coefficients for each time sequence of values ​​from a corresponding portion of the matrix; An apparatus comprising:

2. 2. The apparatus of claim 1, wherein the array of optical couplers is configured to replicate each time sequence of 1D Fourier coefficients into multiple spectrally distinct copies based on different frequencies of the first carrier signal.

3. a coherent detector configured to sample the time sequence of 1D Fourier coefficients based on a second carrier signal, the second carrier signal having at least one second frequency interval greater than the at least one first frequency interval; the coherent detector is configured to output short-time Fourier transform coefficients of the time sequence of the 1D Fourier coefficients, the short-time Fourier transform coefficients representing a two-dimensional (2D) discrete Fourier transform of the matrix; 2. The apparatus of claim 1.

4. 4. The apparatus of claim 3, further comprising: a first demultiplexer configured to separate, for each time sequence of 1D Fourier coefficients from the array of optical couplers, a plurality of spectrally distinct copies of the time sequence of 1D Fourier coefficients and provide different ones of the spectrally distinct copies to different ones of the coherent detectors.

5. a local oscillator (LO) bank or array configured to generate the second carrier signal; a second demultiplexer configured to separate the second carrier signals for each time sequence of 1D Fourier coefficients from the array of optical couplers and provide different ones of the second carrier signals to different ones of the coherent detectors; The apparatus of claim 4 further comprising:

6. 6. The apparatus of claim 5, further comprising a weighting element configured to adjust the second carrier signal based on a different weight and provide an adjusted second carrier signal to the coherent detector.

7. The matrix includes an input feature map, The weights are from a weight map.

7. The apparatus of claim 6.

8. a time sequence of values ​​from a corresponding portion of the matrix corresponds to a column-wise vector from the matrix; the time sequence of 1D Fourier coefficients from the array of optical couplers sampled by a coherent detector comprises a row directional vector; 2. The apparatus of claim 1.

9. 1. An apparatus comprising: a frequency comb configured to generate a first carrier signal having at least one first frequency interval; a plurality of modulators configured to modulate the amplitude of one of the first carrier signals to generate a modulated carrier signal; a two-dimensional (2D) array of optical couplers configured to perform a one-dimensional (1D) discrete Fourier transform in a first direction using the modulated carrier signal; an array of coherent detectors; a first demultiplexer optically coupled to an output of the array of optical couplers and to the coherent detector; a local oscillator (LO) bank or array configured to generate a second carrier signal having at least one second frequency interval different from the at least one first frequency interval; a second demultiplexer optically coupled to the output of the LO bank or array and to the coherent detector; The device comprising:

10. Each coherent detector is configured to receive an output of the first demultiplexer and an output of one of the second demultiplexers and generate Fourier coefficients; the array of coherent detectors is configured to output a second one-dimensional (1D) discrete Fourier transform in a second direction to complete a 2D discrete Fourier transform on the input of the device.

10. The apparatus of claim 9.

11. The apparatus of claim 9 , wherein the optical coupler comprises a 50 / 50 optical coupler.

12. The apparatus of claim 9 , wherein a number of the first carrier signals, a number of the second carrier signals, and a number of coherent detectors in the array of coherent detectors are equal.

13. 10. The apparatus of claim 9, further comprising a weighting element configured to modulate the second carrier signal according to a weighting matrix.

14. The apparatus of claim 13 , wherein a Fourier equivalent of the weighting matrix is ​​pre-computed and stored in the apparatus.

15. Obtaining an input feature map; generating a two-dimensional (2D) discrete Fourier transform of the input feature map using an optical network to generate a Fourier-space input feature map; obtaining a Fourier-space weight map based on the weight map; performing a Hadamard multiplication of the Fourier-space input feature map and the Fourier-space weight map; A method comprising:

16. The method of claim 15 , further comprising generating a 2D inverse discrete Fourier transform of an output of the Hadamard multiplication using a second optical network.

17. The method of claim 16 , wherein the output of the 2D inverse discrete Fourier transform comprises a convolution of the input feature map and the weight map.

18. generating a 2D discrete Fourier transform of the weight map using the optical network to produce the Fourier-space weight map; storing the coefficients of the Fourier-space weighting map; 20. The method of claim 17, further comprising:

19. The method of claim 15 , further comprising: zero padding an input map to generate the input feature map and a weighting map to provide the weight map.

20. receiving the Fourier-space weights map; storing said Fourier-space weight map; Retrieving the Fourier-space weights map from storage; and weighting a local oscillator (LO) carrier signal utilizing the Fourier-space weighting map; The method of claim 15 further comprising:

Citation Information

Patent Citations

  • High-speed and low-jitter communication system

    JP2015515159A

  • Photonic processing systems and methods

    JP2021523508A

  • Large-Scale Artificial Neural-Network Accelerators Based on Coherent Detection and Optical Data Fan-Out

    US20210357737A1

  • High-speed low-jitter communication system

    WO2013116644A1

  • Photonic processing systems and methods

    WO2019222185A1