Photonic Neural Network System

Abstract

To provide a system and method, which greatly increase speed, resolution, and power efficiency as compared to digital spatial convolutions for optically processing images.SOLUTION: A photonic neural network system 10 for convolving and adding frames of data includes a first sensor display device 14 and a second sensor display device 26. Each sensor display device comprises an array of transmit-receive modules. Each transmit-receive module comprises a light sensor element, a light transmitter element, and a memory bank. A radial modulator device 20 is positioned where transmission of light fields including frames of data are Fourier transformed. Filters implemented by modulator elements of the radial modulator device 20 convolve fields of light including the frames of data, which are then sensed on a pixel-by-pixel basis by the light sensor elements, which accumulate charges, thus sum pixel values of sequential convolved frames of data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to neural networks, and more particularly to convolutional neural networks with optical processing. State of the prior art

[0002] Neural networks are well known as computing systems with many simple, highly interconnected processing elements that process information through dynamic state responses to external inputs. Neural networks are useful for recognizing patterns and clustering and classifying data. Computers can utilize neural networks to perform machine learning, in which the computer learns to perform a task by analyzing training examples. These examples are typically pre-labeled by a user. For example, a neural network configured as an object or image recognition system could be fed thousands of example images labeled "cat" or "no cat," and then use its results to identify cats in other images or, in some cases, to indicate the absence of cats in other images. Alternatively, such a neural network configured as an object recognition system could be fed thousands of examples of images with various objects, such as cats, cows, horses, pigs, sheep, cars, trucks, boats, and airplanes, labeled as such, and then use its results to identify whether other images contain cats, cows, horses, pigs, sheep, cars, trucks, boats, or airplanes.

[0003] A convolutional neural network (CNN) is a type of neural network that uses many identical copies of the same neuron, allowing the network to have many neurons and represent computationally large models while requiring only a small number of real values ​​that describe how the neurons behave. Convolution is a method of combining two signals to form a third signal. CNNs are typically implemented in software or programmable digital hardware.

[0004] Deep learning is the term used for stacked neural networks, i.e., networks containing several layers. Layers are composed of nodes. Nodes (or nodes) are where computations occur; they are loosely patterned on neurons in the human brain and fire when they encounter sufficient stimuli. Nodes combine inputs from data with a set of coefficients or weights that amplify or attenuate the input, thereby assigning significance to the task the algorithm is trying to learn—for example, which inputs are most useful for classifying data without error. These input-weight products are summed, and the sum passes through the node's activation function to determine whether and to what extent the signal will proceed further through the network to affect the final outcome, such as the act of classification. Node layers are rows of neuron-like switches that turn on or off as inputs are fed through the network. The output of each layer simultaneously serves as the input for subsequent layers, starting with the initial input layer, which receives data. Three or more node layers are considered "deep" learning. In a deep learning network, each layer of nodes trains a distinct set of features based on the output of the previous layer; therefore, the more layers data (e.g., pictures, images, audio, etc.) passes through, the more complex the features the node can recognize. During training, a process called backpropagation is used to adjust the network to improve its chances of predicting the same type of image the next time. This data processing and backpropagation is repeated multiple times until predictions become reasonably accurate and no further improvement can be seen. The neural network can then be utilized in inference mode to classify new input data and predict outcomes inferred from its training.

[0005] In addition to input and output layers, a typical convolutional neural network has four essential neuron layers: convolution, activation, pooling, and fully connected. In the first one or more convolutional layers, thousands of neurons act as the first set of filters, scouring every part and pixel in an image for patterns. As more and more images are processed, each neuron gradually learns to filter specific features, which improves accuracy. In effect, one or more convolutional layers decompose the image into distinct features. The activation layer then emphasizes salient features, such as those likely to have value or importance in the final classification result. For example, eyes are more likely to indicate a face rather than a frying pan.

[0006] All of the convolutions and activations over the entire image generate a large amount of data that can overwhelm the computational capacity of a computer, so pooling is used to compress the data into a more manageable form. Pooling is the process of selecting the best data and discarding the rest, resulting in a lower resolution dataset. Several types of pooling can be used, some of the more common types are "max pooling" and "average pooling."

[0007] Finally, in a fully connected layer, each reduced or "pooled" feature map or data is connected to an output node (neuron) that represents the items the neural network is learning about or being enabled to identify, for example, cats, cows, horses, pigs, sheep, cars, trucks, boats, and airplanes. the above When the feature map or data is run through these output nodes, each node generates a list of any features that the node identifies as a particular item. featureVote on the map or data. The final output of the network of image data passed through the network is based on the votes of the individual nodes. Early in the training of the network, voting may produce more erroneous outputs, but as the number of images and backpropagation increases to adjust the weights and refine the training, accuracy improves, and therefore, ultimately, the prediction or inference of outcomes from the input data can be very accurate.

[0008] The foregoing examples of the related art and limitations associated therewith are illustrative of the present subject matter, but are not intended to be exclusive or exhaustive. Other aspects and limitations of the related art will become apparent to those skilled in the art upon reading this specification and studying the drawings.

[0009] The accompanying drawings, which are incorporated in and form a part of this specification, depict some exemplary embodiments and / or features, but not the sole or exclusive embodiments and / or features. It is intended that the embodiments and drawings disclosed herein be considered illustrative and not limiting. [Brief explanation of the drawings]

[0010] FIG. 1 is a functional block diagram of an exemplary photonic neural network system.

[0011] FIG. 2 is an isometric view of an exemplary photonic convolution assembly for optically processing and convolving images for the photonic neural network system of FIG. 1, with a portion of the second Fourier transform lens shown broken away to reveal an array of optical sensor-display components of the second sensor-display device.

[0012] FIG. 3 is a front view of an example radial modulator in the example photonic neural network of FIG.

[0013] FIG. 4 is an isometric view of a central portion of the example radial modulator of FIG. 3, along with an expanded view of an optical modulator segment of the example radial modulator.

[0014] FIG. 5 is an isometric view illustrating the convolution function of an example photonic convolution assembly of an example photonic neural network system.

[0015] FIG. 6 is a schematic top view of the example photonic convolutional assembly of FIG. 2 showing a first sensor-display device displaying a frame of data (image) and a second sensor-display device sensing the convoluted frame of data.

[0016] FIG. 7 is a schematic top view of the example photonic convolutional assembly of FIG. 2 showing a second sensor-display device displaying a frame of data (image) and a first sensor-display device sensing a convoluted frame of data.

[0017] FIG. 8 is a functional block diagram of an array of transmitting and receiving modules (or transmitting and receiving modules) in the first sensor display device. External Interface, Column Controls, and Row Controls are external interfaces, column controls, and row controls, respectively.

[0018] FIG. 9 is an enlarged isometric view of a portion of an array of transmit and receive modules.

[0019] FIG. 10 is an enlarged isometric view of an exemplary transceiver module.

[0020] FIG. 11 is a perspective view of an example optical transmitter element of the example transceiver module of FIGS.

[0021] Figure 12 is a functional block diagram of an exemplary system interface to the external interface of the sensor display device. To All RedFives is to all RedFives. Output is the output, and Analog in is the analog input.

[0022] 13 is a functional block diagram of an exemplary external interface of the sensor display device, where Digital Interface is a digital interface, Analog Interface is an analog interface, Row & Column Controls are row and column controls, Global Controls are global controls, Analog Bus Buffers are analog bus buffers, and Trixel Array is a Trixel array.

[0023] FIG. 14 is a schematic diagram of a row and column control line register (or row and column control line register) for an array of transmit and receive modules.

[0024] FIG. 15 is a schematic diagram of the analog data lines to the transceiver module.

[0025] Figure 16 is a schematic diagram of some of the transmit and receive modules (trixels) in the array interconnected in a Pooling Chain.

[0026] FIG. 17 is an enlarged schematic diagram of the pooling boundaries (or pooling boundary lines) of the transmitting and receiving modules (pixels) and the interconnections between adjacent transmitting and receiving modules (trixels).

[0027] 18 is an example memory shift driver for a memory bank in an example transmit / receive module of the example photonic neural network system 10. MEM SHIFT DRIVER is a memory shift driver.

[0028] Figure 19 is a schematic diagram of an exemplary analog memory read interface for a memory bank. MEM READ PAD is the memory read pad, APPLY ReLU is the apply ReLU, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0029] Figure 20 is a schematic diagram showing the analog memory read average of the transmit / receive module (trixel) to the pooling chain. MEM READ PAD is the memory read pad, APPLY ReLU is the apply ReLU, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0030] Figure 21 is a schematic diagram showing the analog memory read maximum of the transmit / receive module (trixel) to the pooling chain. MEM READ PAD is the memory read pad, APPLY ReLU is the apply ReLU, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0031] Figure 22 is a schematic diagram showing the analog memory of the transceiver module (trixel) read out to the external data line. MEM READ PAD is the memory read pad, APPLY ReLU is the ReLU application, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0032] Figure 23 is a schematic diagram showing the analog memory peak value storage of the transmission / reception module (trixel). MEM READ PAD is the memory read pad, APPLY ReLU is the ReLU application, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0033] Figure 24 shows the analog memory peak value reset of the transmission / reception module (trixel). MEM READ PAD is the memory read pad, APPLY ReLU is the ReLU application, READ AMP is the read amplifier, and Pooling Chain is the pooling chain.

[0034] Figure 25 shows a graphical representation of an example of a ReLU (rectified linear unit) response, where Max Voltage is the maximum voltage, Min Voltage is the minimum voltage, Linear (Non-ReLU) Response is the linear (non-ReLU) response, Classical ReLU Response is the classical ReLU response, and Soft ReLU Response is the soft ReLU response.

[0035] Figure 26 is a schematic diagram showing writing to the analog memory of the transceiver module (trixel). MEM DRIVER is the MEM (memory) driver, MEM WRITE PAD is the MEM write pad, and Pooling Chain is the pooling chain.

[0036] Figure 27 is a schematic diagram showing loading analog memory from external data lines. MEM DRIVER is the MEM (memory) driver, MEM WRITE PAD is the MEM write pad, and Pooling Chain is the pooling chain.

[0037] Figure 28 is a schematic diagram showing the flag memory write circuit. MEM DRIVER is a MEM (memory) driver, MEM WRITE PAD is a MEM write pad, and Pooling Chain is a pooling chain.

[0038] 29 is a schematic diagram showing the flag memory read circuit. MEM DRIVER is a MEM (memory) driver, MEM WRITE PAD is a MEM write pad, and Pooling Chain is a pooling chain.

[0039] Figure 30 is a schematic diagram showing the optical control line setup for reading the transmitter / receiver module (trixel) sensor to the pooling chain. SNSR READ PAD is the SNSR (sensor) read pad, SNSR AMP is the SNSR (sensor) read pad, and Pooling Chain is the pooling chain.

[0040] Figure 31 is a schematic diagram showing the optical control lines for resetting the transmitter / receiver module (trixel) sensor. SNSR READ PAD is the SNSR (sensor) read pad, SNSR AMP is the SNSR (sensor) read pad, and Pooling Chain is the pooling chain.

[0041] Figure 32 is a schematic diagram showing the optical control line settings for writing optical transmitter elements (modulators) from the pooling chain. MOD READ PAD is the MOD read pad, MOD DRIVER is the MOD driver, and Pooling Chain is the pooling chain.

[0042] 33A-B show schematic diagrams of the transmission / reception module (trixel) circuit. MEM READ PAD is a memory read pad, READ AMP is a read amplifier, APPLY ReLU is ReLU application, and MEM SHIFT DRIVER is a MEM shift driver.

[0043] FIG. 34 illustrates an exemplary photonic convolution assembly having a Fourier optical sensor device for Fourier transforming frames of correction data in training mode.

[0044] [Figure 35] FIG. 35 is a diagrammatic isometric view of an example Fourier optical sensor device.

[0045] FIG. 36 shows an example of a photonic convolution assembly with a camera lens embodiment for introducing a real-world frame of data (image) into the photonic convolution assembly. DETAILED DESCRIPTION OF THE INVENTION

[0046] A functional block diagram of an exemplary photonic neural network system (or photonic neural network system) 10 is shown in FIG. 1 , and an isometric view of an exemplary photonic convolution assembly 12 for optically processing and convolving images for the photonic neural network system 10 is shown in FIG. 2 . Convolution by this photonic neural network system 10 is performed using an optical Fourier transform, which significantly increases speed, resolution, and power efficiency compared to digital spatial convolution. Therefore, creating and using a neural network can be done much faster and with much less power consumption than typical computer algorithmic processing and computational convolution. Because all of the convolution and summation is entirely analog, full-frame photonic computation, power consumption is very low. As described below, summing is achieved by building charge within capacitive optical sensors, which is an analog process. Because the sensors have very low noise and are free of clocking and other transient noise sources, summing is a very quiet process. Photonic neural network 10 can accept and process any data, such as images, video, audio, speech patterns, or anything typically processed by convolutional neural networks, and supports all existing convolutional neural net architectures and training methods. Photonic neural network 10 also provides full-frame image parallelism with an architecture in which every data element is in its ideal position for the next stage, with full resolution processed at the speed of light, and is therefore nearly 100 percent efficient. Other advantages can be seen from this description.

[0047] Referring to both FIGS. 1 and 2, the optical processing of images, for example for a photonic neural network system 10, is performed in a photonic convolution assembly 12. Essentially, the first sensor display device 14 projects frames of data (e.g., images or optical representations of sound, speech patterns, video, or other data) as a modulated light field 15 via a first Fourier transform lens 16 and a polarizer 18 onto a radial modulator 20 positioned at the focal plane of the lens 16, as described in more detail below. The frames of data projected by the first sensor display device 14 are formed by the first sensor display device 14 based on values ​​or signals provided to the first sensor display device 14 by support electronics (described in more detail below) via an electronic data interface 22. The Fourier transform lens 16 can be a diffractive lens, a solid convex lens, or any other form of Fourier transform lens. Additionally, a fiber faceplate (not shown) can be placed in front of the lens 16 to collimate the light before it enters the lens 16.

[0048] Lens 16 transforms a frame of data (eg, an image) into its Fourier equivalent at the focal plane (also called the Fourier transform plane), and therefore at the surface of radial modulator 20 . The radial modulator 20 generates an optical field 15 containing the Fourier equivalent of a frame of data based on a pattern (also called a "filter") loaded into the radial modulator 20 by support electronics (described in more detail below) via an electronic data interface 24. In the Fourier transform plane The reflected light field, including the modulated frame of data, is inverse Fourier transformed back into the spatial domain at a distance from the radial modulator 20 to the second sensor and display device 26, and the modulated frame of data incident on the second sensor and display device 26 is thus reflected back into the spatial domain at a distance from the radial modulator 20 to the second sensor and display device 26, and thus the modulated frame of data incident on the second sensor and display device 26 is ... In the Fourier transform plane The spatial domain features of the unfiltered data frame are displayed on a second sensor per pixel. Device26, where light incident on each pixel generates a charge proportional to the intensity of the light and the time the light is incident on the pixel. 14 Each frame of data coming out of the second sensor display device can be modulated with one or more filters (patterns) by the radial modulation device 20. 26 may receive from the first sensor display device 14 one or more frames of data modulated by one or more filters applied to the radial modulation device 20. Thus, the charge accumulation of each pixel in the second sensor display device 26 may be the sum of one or more modulated (i.e., filtered) patterns of one or more frames of data, as described in more detail below, thereby constituting a convolution of the one or more frames of data projected by the first sensor display device 14.

[0049] For example, one frame of data is projected sequentially by the first sensor display device 14, first in red, then in green, then in blue, and then by the radial modulator device (or radial modulator) 20 can apply the same or different filters (pattern modulation) to each of the red, green, and blue projections. All of these modulated frames of data are modulated by the light from each of the sequentially modulated frames of data. Second sensor display device 26 The charges for each pixel are added together and can be detected sequentially by the second sensor display device 26. Second sensor display device 26 The charges for each pixel are of each pixel of the second sensor display device 26 are transferred to each memory cell, and those memory cells Second sensor display device 26 These summation results for each pixel are stored, thereby comprising stored pixel values ​​of the convolution of the frame of data in the spatial domain projected into the spatial domain by the first sensor display device 14 and convolved into the Fourier transform domain by the filter in the radial modulator device 20.

[0050] The process can be repeated for red, green, and blue projections of the same frame of data from the first sensor display device 14, but with different filters in the radial modulator 20, and therefore with different modulation patterns from the Fourier transform domain reflected by the radial modulator 20 to the second sensor display device 26, resulting in another set of accumulated pixel values ​​and another summation result for another convolved frame of data in the memory bank of the second sensor display device 26. The accumulation of the convolved frame of data in the second sensor display device 26 forms a 3D convolution block from the frames of data projected by the first sensor display device 14 for all of these different filter applications by the radial modulator 20. In summary, the frame of data from the first sensor display device 14 is multiplied by the radial modulator 20 by a series of filters in the Fourier plane and summed by the second sensor display device 26 in a sequence that builds a 3D convolution block in the memory of the second sensor display device 26. Assuming sufficient memory capacity to store all of the pixel values ​​for all of the convolutional frames of data in the 3D convolutional block, any number of such convolutional frames of data can be stored in the 3D convolutional block, which can be thought of as the first level in a neural network.

[0051] For the next convolution block or level, the first sensor display device 14 and the second sensor display device 26 swap functions. The 3D convolution block in the memory of the second sensor display device 26 becomes a frame of data for the next convolution sequence. For example, each convolution frame of data stored in the 3D convolution block in the memory of the second sensor display device 26 is projected by the second sensor display device 26 through the second Fourier transform lens 28 to the radial modulation device 20, where it is multiplied by a filter and reflected back to the first sensor display device 14. The first sensor display device 14 detects and sums the series of such convolved and summed frames of data to construct the next 3D convolution block in the memory of the first sensor display device 14.

[0052] This process cycle is shown diagrammatically in FIG.

[0053] These convolution processes can be repeated as many times as necessary for any convolutional neural network architecture, with frames of data projected back and forth (or to each other) between the first sensor and display device 14 and the second sensor and display device 26. As more filters are applied in subsequent cycles, the convolutions can be pooled by feeding the accumulated charge from multiple pixel detections into a single memory cell, instead of feeding the accumulated charge from each pixel detection into an individual memory cell, as described in more detail below. Thus, convolutional neural networks with many levels of abstraction can be developed using the exemplary photonic neural network 10.

[0054] A front view of an exemplary radial modulator device 20 is shown in FIG. 3, and an exemplary segmented optical modulator of the exemplary radial modulator device 20 is shown in FIG. Element 40A perspective view of a central portion of an exemplary radial modulator device 20 with an expanded view of the radial modulator device 20 is shown in FIG. 4. The radial modulator device 20 has an optically active region 30 comprising a plurality of light-modulating wedge segments 32 (wedge-shaped segments), each of which is independently operable to modulate light incident on the respective wedge segment 32. In the example radial modulator device 20 shown in FIGS. 2, 3, and 4, the wedge segments 32 are grouped into a plurality of wedge sectors 34, each extending radially outward from a central component 36, which together form the optically active region 30 of the radial modulator device 20. In FIGS. 3 and 4, the wedge segments 32 and sectors 34 are not shown in order to avoid cluttering the drawings. 34 Although only some of the wedge segments 32 and wedge sectors 34 are labeled with these numbers in this illustration, one skilled in the art would understand where all of the wedge segments 32 and wedge sectors 34 are located within the exemplary radial modulator device 20. In the example radial modulator device 20 shown in Figures 3 and 4, the wedge segments 32 are arranged to form a circular optically active area 30, although other shapes can be used.

[0055] As mentioned above, each of the wedge segments 32 is optically active, in the sense that it can be activated to transmit light, block light, or modulate the transmission of light between complete transmission and blockage. Thus, a beam or field of light incident on the optically active region 30 can be modulated by any combination of one or more wedge segments 32. Spatial light modulators can be designed and manufactured to modulate light in many ways. For example, U.S. Patent No. 7,103,223, issued September 5, 2006, to Rikk Crill, illustrates the use of birefringent liquid crystal materials to modulate the wedge segments in a radial spatial light modulator similar to the radial modulator 20 of FIGS. 2 and 3. Zhang et al., "Active metasurface modulator with electro-optic polymer using bimodal plasmonic resonance," Optics Express, Vol. 25, No. 24, November 17, 2017, describes an electrically tunable metal grating with an electro-optic polymer that modulates light and is suitable for ultrathin surface-normal applications. The optically active wedge segment 32 in the exemplary radial modulator device 20 Such a metasurface light modulator element configured for use as an exemplary segmented light modulator 40 having a metal grating structure 42 for is shown in FIG. The grating structure 42 is sandwiched between a bottom metal (e.g., Au) layer 46 and an interdigitated top thin film metal (e.g., Au) grating layer 48, all built on a substrate 50. Interdigitated Electro-optic polymer 44. Diffraction grating 42 The period of is such that diffraction is prohibited incident light LThe thickness of the top metal layer 48 is greater than the skin depth to eliminate direct coupling of the incident light L into the electro-optic polymer 44. The bottom metal layer 46 is also of the same thickness so that it acts as a nearly perfect reflective mirror. Essentially, light L enters the top of the metasurface light modulator element 40, is phase shifted within the electro-optic polymer 44, which is periodically poled by application of a poling voltage 45, reflects off the bottom metal layer 46, and exits the top surface with the polarization of the light further phase shifted and rotated 90 degrees during its second (i.e., reflected) pass. Other wedge segments 32 in the example radial modulation device 20 have the same type of light modulation element 40. can is sized and shaped to fit and substantially fill each particular wedge segment 32. The central component 36 may also include a light-modulating element 40.

[0056] 2-7 and described above, for example, radial modulator 20 is a reflective device, where incident light is modulated and reflected by wedge segments 32. However, radial modulators may alternatively be transmissive devices, where incident light is modulated and transmitted through the radial modulator. Of course, the positions of optical components, such as sensor-display devices, lenses, and polarizers, must be rearranged to route the light field in the proper sequence to the respective optical components, but one skilled in the art, once familiar with the exemplary photonic neural network 10 described above, will understand how to perform such rearrangements.

[0057] As shown in FIGS. 3 and 4 and briefly described above, the optically active wedge segments 32 are grouped into multiple wedge sectors 34 that extend radially from a round central component 36 around the optically active region 30. The wedge segments 32 are also arranged in concentric rings around the central component 36. Each concentric ring of wedge segments 32, except for the innermost concentric ring, has an outer radius twice that of the immediately adjacent inner ring, which corresponds to the scale distribution in the Fourier transform. Thus, each subsequent wedge segment 32 radially outward within a wedge sector 34 is twice as long as the previous wedge segment 32. A detailed description of a radial modulator, which functions as a filter in the Fourier transform plane of an image, can be found, for example, in U.S. Patent No. 7,103,223, issued September 5, 2006, to Rikk Crill. Here, the optical energy from higher spatial frequency shape content in the spatial domain is filtered by the lower spatial frequency content. Light energy from than In the Fourier transform plane Suffice it to say that the angular orientation and intensity of the light energy from the higher spatial frequency feature content and from the lower spatial frequency content is preserved in the Fourier transform of the image. Thus, light energy transmitted (or transmitted) by a particular wedge segment 32 positioned at a particular angular orientation and a particular radial distance from the center (optical axis) of the Fourier transform image in the Fourier transform plane is inverse Fourier transformed in the projection to display only feature content (features) from the original image that have the same angular orientation as the particular wedge segment 32, and that have spatial frequencies within the range corresponding to the radial extent to which such light energy was distributed in the Fourier transform plane. Returning to the spatial domainThe light intensities (brightness) of these inverse Fourier transformed features (shape content) correspond to the light intensities (brightness) that these features (shape content) had in the original image, and they are in the same locations as in the original image. Of course, any shape content (features) contained in the light energy of the original image that was blocked and not transmitted by a particular wedge segment 32 in the Fourier transform plane will be missing back into the spatial domain in the inverse Fourier transform image. Also, any shape content (features) comprised of light energy that was only partially blocked, and therefore partially transmitted, by a particular wedge segment 32 in the Fourier transform plane will be inverse Fourier transformed into the spatial domain with the same angular orientation and particular spatial frequency as described above, but with reduced intensity (brightness). (Alternatively, shape content (features) consisting of light energy that is only partially blocked, and therefore partially transmitted, by a particular wedge segment 32 in the Fourier transform plane may be inverse Fourier transformed into the spatial domain at the same angular orientation and particular spatial frequency as described above, but at a reduced intensity (brightness). Thus, as explained above and described in more detail below, some of the shape content (features) of the original image may be preserved in the inverse Fourier transform image at full or partial intensity (brightness), and some of the shape content (features) may be preserved in the inverse Fourier transform image. Partially or The completely removed inverse Fourier transform image back to the spatial domain is the convolved image that is detected and used in the construction of the 3D convolution block of the neural network, as shown in Figure 5.

[0058] 5, a first filter 54 is loaded into the radial modulator device 20 via the data interface 24, such that the wedge segments 32 are configured to block or fully or partially transmit light in the pattern set by the first filter 54. For example, a first frame of data including an image of a pile of LEGO® toy building blocks 52 is loaded into the first sensor and display device 14 via the data interface 22, such that a display component (not shown in FIG. 5) within the first sensor and display device 14 is configured to display the frame of data including the image of the LEGO® toy building blocks 52, as seen in FIG. 5. Laser illumination 13 is directed onto the first sensor and display device 14 essentially through a first Fourier transform lens 16 and through a polarizer 18 which illuminates the frame of data 50 onto the radial modulator device 20, which is positioned a focal distance Fl from the first Fourier transform lens 16, i.e., in the focal plane of the first Fourier transform lens 16, as also shown in FIG. 6. The Fourier transform lens 16 focuses the light field 15 containing the image 50 onto the surface of the radial modulator 20. The frame of data 50 containing the image of the LEGO® toy building block 52 is expressed in the Fourier transform domain as Radial modulation device 20The light is convolved with a filter by the wedge segment 32, which, as explained above, either fully or partially reflects some light or blocks some light that constitutes the image. The wedge segment 32 phase-shifts the reflected light, thus rotating its polarization, so that the light reflected by the radial modulator device 20 is reflected by the polarizer 18 to the second sensor and display device 26, as shown by the reflected light field 56. Thus, as explained above, some shape content (features) of the original frame of data 50 of the image of the LEGO® toy building block 52 are absent or less intense, i.e., filtered out, in the convolved image incident on the second sensor and display device 26, as shown in FIG. 5 . That convolved frame of data (image) is detected by the second sensor and display device 26 and summed with several subsequent convolved images at the second sensor and display device 26 to form a first convolved sum data frame (image) 58. The first convolved data frame (image) 58 is fed to the 3D convolution block 6 as shown schematically in FIG. 5 can be transferred to a memory bank for accumulation with subsequent convolved data frames (images) to construct

[0059] The second sensor display device 26 and the first sensor display device 14 then swap roles as described above, with the second sensor display device 26 in display mode and the first sensor display device 14 in sensor mode. Placing the second sensor display device 26 in display mode then causes the first convolved and summed data frame (image) 58 to be projected by the second sensor display device 26 onto the radial modulator 20, where it is convolved with an additional filter, and then reflected by the radial modulator 20 back to the first sensor display device 14, as shown schematically in FIG. 7 . This role swap is shown diagrammatically in FIG. 7 , where the second sensor display device 26 switches to display mode and the first sensor display device 14 switches to sensor mode. In display mode, the display component of the second sensor display device 26 is programmed to display its first convolved and summed data frame (image) 58. Thus, laser illumination 60 on second sensor and display device 26 illuminates first convolved and summed data frame (image) 58 along second optical axis 62 through second Fourier transform lens 28 onto polarizer 18, which reflects light field 64 along first optical axis 61 to radial modulator 20. The optical distance between second Fourier transform lens 28 and radial modulator 20 along second optical axis 62 and first optical axis 61 is equal to the focal length of second Fourier transform lens 28. Thus, light field 64 at radial modulator 20 in the Fourier transform plane is the Fourier transform of first convolved and summed data frame (image) 58. The radial modulator 20 applies a filter to the Fourier transform of the first convolved and summed data frame to provide a second convolution to the data frame, which it reflects with a phase shift as described above and propagates along the first optical axis 61 to the first sensor and display device 14. The first sensor and display device 14, in turn, in the exchanged role of detector as described above, detects the frame of data (image) convolved with the filter applied by the radial modulator 20.The convolved frame of data (image) detected by the first sensor display device 14 is then summed by the first sensor display device 14 with several other convolved frames of data (images) subsequently detected by the first sensor detection device 14, and these convolved summed frames of data (images) are transferred to a memory bank and used to construct a second 3D convolution block 66, which is shown in FIG. 5 is shown diagrammatically in

[0060] The roles of the first and second sensor and display devices 14, 26 are then swapped again, and the convolved and summed frame of data sensed and summed by the first sensor and display device 14 is backprojected through the system in the same manner as described above, convolved by the radial modulator device 20, and then sensed and summed by the second sensor and display device 26 to form the first 3D convolution block. 65 , which continues to be built and sent back through the system to be convolved with additional filters and sums to build the second 3D convolution bank 66. This process is repeated as many times as desired to build deeper and deeper convolutions, or until the inference neural network is complete.

[0061] The first sensor and display device 14 and the second sensor and display device 26 each include a 3D convolution block, as described in more detail below. 65, 66 and subsequent convolution blocks may have memory banks for storing convolved frames of data (images) that receive and sum successively received convolution frames of data (images). Thus, except for the first frame of data (image) loaded into the system, an input frame of data may always reside in one memory bank of the sensor display devices 14, 26 from the previous convolution cycle. A set of filters 68 are loaded into the radial modulator 20 in synchronization with the display of the frame of data (image) by the respective first and second sensor display devices 14, 26 for convolution of the frame of data (image) with the filters.

[0062] Except for being computed optically in the Fourier transform domain, the convolutions in this photonic neural network system 10 are identical to convolutions computed by traditional digital methods. However, as explained in more detail below, whole-frame parallelism at any resolution, processed at the speed of light in an architecture where every data element is ideally positioned for the next convolution stage in the cycle, is nearly 100 percent efficient; therefore, as mentioned above and explained in more detail below, building convolution blocks in the example photonic neural network system 10 offers far more power and speed than convolutions computed by traditional digital methods.

[0063] As described above, each of the first and second sensor-display devices 14, 26 has both light detection and image display capabilities on a pixel-by-pixel basis. Because the first sensor-display device 14 and the second sensor-display device 26 have essentially the same components and structure in this example photonic neural network system 10, details of these devices will be described below primarily with reference to the first sensor-display device 14, with the understanding that such details also represent the second sensor-display device 26. Accordingly, in the following description, the first sensor-display device 14 may be referred to simply as the sensor-display device 14. A functional block diagram of an example sensor-display device 14 is shown in FIG. 8 and includes an array 80 of transceiver modules (transmit-receive modules) 82, each of which has light-transmitting and light-detecting elements and a memory bank, as described in more detail below. Row and column control for the transceiver modules 82 in the array 80, as well as a mixed analog and digital interface 24 to an external control circuit (not shown in FIG. 8), are provided for inputting and outputting data, as described in more detail below. An enlarged portion of the array 80 shown diagrammatically in Figure 9 illustrates example transmit / receive modules 82 within the array 80, and a further enlarged diagrammatic representation of the example transmit / receive modules 82 is shown in Figure 10. Each of the example transmit / receive modules 82 includes both micro-optical transmitter elements 84 and micro-optical detector (sensor) elements 86, which together form the transmit / receive module 82are sufficiently small and close enough together to effectively function as light transmitters and light receivers at substantially the same pixel locations of an image or frame of data, with sufficient resolution to produce neural network results at least as useful as neural network results from typical computational convolutions and processing by computer algorithms. For example, for a neural network using exemplary photonic neural network system 10 to produce results as useful as neural network results from typical computational convolutions and processing by computer algorithms, micro-optical transmitting elements 84 and micro-optical detecting elements 86 may be offset from each other by 40 micrometers or less, and both may fit within a transceiver module 82 having an area of ​​160 square micrometers or less.

[0064] 10, in addition to the optical transmitter elements 84 and optical sensor or optical receiver detector elements 86, an exemplary transmit / receive module 82 includes a modulator driver 88, a memory bank 90, a memory interface 92, analog and digital control elements 94, a pooling connection 96 for making pooling connections with adjacent transmit / receive modules 82 in the array 80, a pooling control element 98, and a sense amplifier 100. In a display mode, for example, a first sensor display device 14 (Figures 2, 5, and 6) As mentioned above, when a frame of data (image) is projected onto the radial modulator 20, laser illumination is directed toward the back of the transceiver module 82, as shown in Figures 2, 5, and 6. The first frame of data (image) consists of pixel values ​​for each pixel in the frame of data (image). These pixel values ​​are calculated based on the laser illumination. 13 (See Figures 2, 5, and 6) Light field of 15 Array 80 with a pattern to generate a first frame of data (image) in (Figures 8-10)10, for each of the transmit / receive modules 82, the pixel value is provided to an analog and digital control element 94, which shifts the pixel value to a modulator driver 88. The modulator driver 88 shifts the pixel value to the optical transmitter element 84 according to the pixel value. (Figure 10) By modulating the voltage above, Other transceiver modules 82 in the array 80 (FIGS. 8-10) The other phototransmitter elements 84 transmit light for each pixel while the laser is irradiated. 13 (Figures 2, 5, and 6) modulates the laser illumination incident on the optical transmitter elements 84 in a manner that transmits the pixels of the array 80, which together form a laser light field transmitted through the array 80. 15 (Figures 2, 5, and 6) After the first frame of data (image) is transmitted by the first sensor and display device 14 and the convolved frame of data (image) is returned to the first sensor and display device 14, the light field containing that frame of convolved data (image) is incident on the sensors 86 of all the transceiver modules 82 in the array 80 of the first sensor and display device 14. Thus, the array 80 The sensors 86 on each transceiver module 82 in the system detect pixels of the incident light field, and thus pixels of a frame (image) of data (or date) constituted by (or contained in) the incident light field. Those skilled in the art understand how light sensors, such as charge-coupled devices (CCDs), are constructed and function, and such or similar light sensors can be used for the sensors 86. Essentially, each light sensor has a light-sensitive photodiode or capacitive component that responds to incident photons by absorbing a majority of the energy in the photons, generating a charge proportional to the incident light intensity, and storing that charge in the capacitive component. The longer light is incident on the sensor, the more charge accumulates in the capacitive component. Thus, light energy incident on each pixel of the sensor 86 causes a charge to build up in that sensor 86, the magnitude of the charge being proportional to the intensity of the incident light at that pixel and the time that light at that pixel is incident on the sensor 86.

[0065] As described above, when a series of convolved frames of data (images) are transmitted to and received by the sensor display device 14, the light energy (photons) of the successive light fields comprising the successive (sequence) frames of data (images) induces electrical charges in the sensors 86, such that successive pixels of the light field energy from the successive light fields sensed by each sensor 86 can be accumulated, i.e., added to the capacitive component of that individual sensor 86, resulting in an accumulated charge in the sensor 86 that is the sum of the light energies from the sequence of light fields at that particular pixel location. Thus, the sequence of convolved frames of data (images) received by the sensor display device 14 are sensed and summed pixel by pixel by the array 80 of transceiver modules 82 of the sensor display device 14. Then, as described above, once a predetermined number of individual convolved frames of data (images) have been received and summed, the accumulated (summed) charges in the sensors 86 of each individual transceiver module 82 are shifted to the memory bank 90 of that individual transceiver module 82. The same action of shifting the accumulated (summed) charge in the sensor 86 to the memory bank 90 also 80 Therefore, when the shift operation is performed, the 82 Array of 80 is a module that transmits and receives a complete convolved and summed frame of data (image) resulting from that series or successive convolutions and summations of input frames of data (images). 82 10 , the pixel values ​​for that particular transceiver module 82's pixel location for that initial (or first) convolved and summed frame of data (image) are shifted from the sensor 86 to the first memory cells 102. Thus, the combination of all the first memory cells 102 in the memory banks 90 of all transceiver modules 82 in the array 80 comprises a pixel-by-pixel convolved and summed frame of data (image).

[0066] Then, as described above, when a subsequent second series or sequence of frames of data (image) is convolved, the charge stored in the sensor 86 for that pixel of the resulting second convolved frame of data (image) is shifted into the first memory cell 102 as the charge from that pixel of the first convolved frame of data (image) is simultaneously shifted into the second memory cell 104 of the memory 90. Those skilled in the art will understand how such a shift register memory is made and operated, so a detailed explanation is not necessary for purposes of this description. This same process occurs simultaneously in the other transmit / receive modules 82 in the array 80. Thus, the combination of all first and second memory cells 102, 104 in the memory banks 90 of all transmit / receive modules 82 in the array 80 comprises the first and second convolved frames of data (image) for each pixel.

[0067] As more subsequent series or sequences of frames of data (images) are convolved and summed as described above, the summed pixel values ​​of successive (or sequential) convolved and summed frames of such data (images) are sequentially shifted into the first memory cells 102, while each preceding pixel value is further shifted along the memory cells of the memory bank 90, the process occurring simultaneously in all of the transmit / receive modules 82 of the array 80 as described above. Thus, all of such convolved and summed frames of data (images) from all of the series or sequences of convolutions and sums are stored, pixel by pixel, in the memory cells of the memory banks 90 in the array 80 of the transmit / receive modules 82. Each such convolved and summed frame of data (images) may be referred to as a convolution, and thus the array 80 of the transmit / receive modules 82 can hold as many convolutions, per pixel, as there are individual memory cells in the individual memory banks 90 of the transmit / receive modules 82. For example, the exemplary transceiver modules 82 shown schematically in FIGS. 9 and 10 each have a memory bank 90 comprised of 64 individual memory cells 102, 104, ..., n. Thus, the exemplary array 80 of transceiver modules 82 can hold as many as 64 frames of data (images) at full resolution, pixel by pixel. When the optical transmitting elements 84 and optical sensor elements 86 within a transceiver module 82 (see FIG. 1 ) are pooled with the optical transmitting elements 84 and optical sensor elements 86 of adjacent transceiver modules 82, as described in more detail below, all of the optical transmitting elements 84 and optical sensor elements 86 within the pooled group display the same brightness for a coarser representation of the frame of data (image). Under such pooling conditions, the memory banks 90 of the transceiver modules 82 within the pooled group can be used sequentially to store the sensed and summed results for the transceiver modules 82 across the pooled group, thereby increasing the effective memory capacity and depth.For example, if transmit / receive modules 82 each having a memory bank 90 containing 64 memory cells are pooled into 5 x 5 groups, i.e., 25 transmit / receive modules 82 per group, the effective memory capacity or depth of each group is 1,600 memory cells (64 x 25 = 1,600). Thus, sequential convolution-added frames of data (images) are first fed to one of the transmit / receive modules 82 in the group until the memory bank 90 of that transmit / receive module 82 is filled, then a further sequential convolution-added frame of data (image) is fed to a second transmit / receive module 82 in the group until the memory bank 90 of that second transmit / receive module 82 is also filled, and then the memory bank 90 of each of the remaining transmit / receive modules 82 in the group can continue to be sequentially filled. If the memory banks 90 of all transmit / receive modules 82 in a group are filled, then that block of convolutions in memory will be 1,600 deep. The collection of convolutions in memory 90 of transceiver module 82 in array 80 is, for example, 3D convolution block 65, shown diagrammatically in Figure 5. Once the desired number of such convolutions have been accumulated in the array for the final 3D convolution block, the neural network results can be read out pixel by pixel from memory bank 90 for transmission by sensor and display device 14 and, at the end of the process, back through electronic data interface 22 to output the neural network results.

[0068] However, it is important to reiterate that during the deep learning process of iteratively convolving and summing frames of data using the example photonic neural network system 10, the pixel values ​​of the most recently formed convolutional blocks still reside within the memory cells of the individual memory banks 90 within the individual transceiver modules 82. Thus, when the sensor and display device 14 switches from a sensor mode, in which convolutional blocks are stored within the memory 90 of the transceiver module 82, to a display mode, in which the convolutional blocks are transmitted back through the optical components of the system 10 for deeper convolution processing, the pixel values ​​for each of the convolved and summed frames of data (images) comprising (or comprising) the convolutional blocks can be read out (shifted) directly from the memory cells 102, 104,...,n of the memory 90 to the modulator driver 88 without further processing and transfer of data to or from external computer processing, memory, and other components or functions. Instead, when switching from sensor mode to display mode, the pixel values ​​of each frame of data (image) comprising (or making up) the convolution block are sequentially read (shifted) out of memory 90 directly into modulator driver 88, which drives optical transmitter elements 84 to modulate the laser light incident on transceiver modules 82 in a manner that writes (imposes) onto the light field the pixel values ​​of the frame of data (image) to be further convolved in that convolution cycle. Thus, as all of the transceiver modules 82 in the array 80 simultaneously switch to display mode and the pixel values ​​in each of them are written (imposed) onto the laser light field, the combination of those pixel values ​​in the light field transmitted by sensor display device 14 replicates the previously convolved frame of data (image) summed and stored in memory banks 90 of the transceiver modules 82 in the array 80. The light field with the previously convolved and summed frame (image) of data is projected through a Fourier transform lens 16 onto a radial modulator 20 for further convolution with a filter in the Fourier transform plane, and then detected by another (e.g., second) sensor and display device 26 as previously described.

[0069] As explained above, these convolution and summation processes are repeated many times over many cycles with many filters. The first sensor-display device 14 and the second sensor-display device 26 are aligned on their respective optical axes 61, 62 (see Figures 2, 6, and 7) so that the transmit / receive module 82 of the first sensor-display device 14 is optically aligned with the corresponding transmit / receive module 82 of the second sensor-display device 26. As a result, there is perfect optical alignment between the arrays 80 of the first and second sensor-display devices 14, 26, including between the corresponding transmit / receive modules. Thus, the exemplary photonic neural network 10 performs full-frame, full-resolution, fully parallel convolution at the speed of light. Other effects, such as gain, thresholding (ReLU), max- or average-pooling, and other functions, are performed simultaneously in dedicated circuitry, as described in more detail below, and these effects do not introduce additional time delays. Virtually any convolutional neural network architecture can be accommodated, including, for example, VGG16 or Inception-Resnet-v2. All processing occurs entirely on the sensor and display devices 14, 26, without repositioning frames of data (images) to or from these devices. For inference operations, the user application only needs to load the images and accept the results after a few microseconds.

[0070] The micro optical transmitter element 84 in the transceiver module 82 can be any optical modulator device that emits or modulates light. The description of the exemplary photonic neural net system 10 above includes an optical transmitter element 84 that modulates laser light incident on the back surface of the optical transmitter element by allowing or inhibiting the light from passing through the optical transmitter element. However, the optical transmitter element 84 can be replaced with a reflective optical modulator device that modulates the incident light and reflects it, which would require the laser illumination to be incident on the same surface of the optical transmitter element from which the light is reflected and would require a rearrangement of the optical elements, as would be understood by one skilled in the art after becoming familiar with the photonic neural network example described above. As another alternative, the optical transmitter element 84 can be replaced with a light emitter, thereby eliminating the need for the laser light field to be incident on the back surface and pass through a modulator.

[0071] An example optical transmitter element 84 is shown in FIG. 11 , which performs phase modulation of incident light similar to the metasurface optical modulator element 40 shown in FIG. 4 and described above. However, the optical transmitter element 84 in this example of FIG. 11 is an optically transmissive element instead of an optically reflective element of FIG. 4 . Therefore, the bottom metal layer 46 crosses the top electrode 48, creating gaps between the grating structures. As a result, incident light L is phase-modulated by the electro-optic polymer 44 and can either be blocked by the electro-optic polymer 44 or pass through the grating structure 42. The substrate 50 is transparent to the light L. The poling voltage 45 is driven by the modulator driver 88 of the transceiver module 82 according to the pixel value imposed by the optical field L, as described above. Further details of such a transmissive optical modulator can be found in Kosugi et al., “Surface-normal electro-optic polymer modulator with silicon subwavelength grating,” IEICE Electronics Express, Vol. 13, No. 17, pp. 1–9, September 10, 2016.

[0072] The back of the transceiver module 82 has an opaque cover or mask (not shown) covering the back to prevent laser illumination on the back of the transceiver module 82 from transmitting through the transceiver module 82, except for an opening that allows light to reach and pass through the optical transmitting element 84. Optical components including a Fourier transform lens 16 and a fiber faceplate that collimates the light in front of the Fourier transform lens 16 can be bonded to the front of the first sensor and display device 14. Similarly, a Fourier transform lens 28 and a fiber faceplate can be bonded to the front of the second sensor and display device 26.

[0073] Referring now to FIG. 1 , in addition to the first and second sensor-display devices 14, 26 and radial modulator device 20 of the photonic convolution assembly 12, the exemplary photonic neural network system 10 also includes external control components, including, for example, (i) a circuit block 110 that implements pulse outputs for driving the radial modulator device 20, (ii) a high-speed analog-to-digital circuit block 112 through which digital data is loaded to and received from the first and second sensor-display devices 14, 26, a high-bandwidth memory (HBM2) 114, and a field-programmable gate array (FPGA) 116, which is the basic control and interface device for other system components. The HBM2 114 provides storage for filters, state machine steps, and image data. The circuit block 110, HBM2 114, and FPGA 116 reside on a multi-chip module (MCM) 118, and the user interface to the system 10 is nominally via a PCI-Express bus 120.

[0074] A functional block diagram of an exemplary system interface 122 between a field programmable gate array (FPGA) 116 and the first sensor and display device 14 is shown in FIG. 12, which is also representative of the system interface between the FPGA 116 and the second sensor and display device 26. For convenience and brevity in the drawings, the diagram and circuit block 110 Output circuit device 111 (Figure 1) In the relevant description for the sensor display device 14, 26 (Figures 1, 2, 5-7, and 10) For convenience and brevity, the term "Sensay" (an abbreviation for Sensor and Display) or "RedFive" is sometimes used. (Figures 8-10) are sometimes called "Trixels." ("Trixels" is an abbreviation for "transmitting and receiving pixels.")

[0075] RedFives 111 Some of the sensor memory bank It is responsible for generating analog data to load the 90. s 111 The 14, 26 I / O are interfaced through the memory module (HBM2) 114 because the HBM2 114 is the state machine source managed by the FPGA 116. Analog and digital input / output (I / O) are interfaced through the FPGA 116 because they are used to control feedback loops. Some unused bits are wrapped back to the FPGA 116 as status flags for synchronization. Sensay Digital 14, 26 I / O are RedFives 111 Although it uses the same memory lines as some of the ADCs, they are not accessed simultaneously, so this dual use of memory lines is not a conflict. Also, some of the output analog lines from the RedFives111 are accessed by the ADC. 112 The ADC is used to read the data and pass it to the FPGA 116. 112 The number is implementation dependent.

[0076] Sensor display device ( Sensay) 14, 26 External Interface (See Figure 8) A functional block diagram of the system is shown in FIG. 13. In FIG. 13, "Sx" is prefixed to "Sensay A" 14 or "Sensay B" 26 (to distinguish signals associated with either Sensay 14, 26). The digital input lines in FIG. 8 can be grouped into three general categories: row and column control load sets of latches within a Sensay (see FIG. 14). Global control lines (or rows) have various functions, each of which will be explained in the context of their use. Global rows can be routed along rows or columns. Global control lines route to all transmit / receive modules (trixels) 82 and are not specific to a particular column or row.

[0077] SxPeakreset resets the analog peak-hold circuit used for external gain control. This signal is asynchronous, but should only be asserted when SxFreeze is asserted to avoid data contention (1).

[0078] Figure 13 SxSnsreset is the sensor 86 (Figure 10) Resets the sensor to the level of the analog SxLevel line. 86 is designed to store charge and therefore requires this mechanism to drop to a preset level. This reset can be used as a global bias to preset the charge level of the sensor 86 (and therefore the modulator level for the next pass).

[0079] SxPoolmode determines average (1) or max (0) behavior in pooling.

[0080] Figure 13 SxFreeze is a global memory Bank 90 (Figure 10) Enables or disables access. When asserted (1), all Trixels memory drives 92 are set to a safe state, Bank 90 No accesses or shifts are allowed. SxFreeze is used when configuring other control lines to prevent data corruption before the lines have settled. In the following description, the function of SxFreeze is not always mentioned, its behavior is always the rule.

[0081] Figure 13 SxRWDir is valid memory Bank 90 (Figure 10) determines whether to read or write. When set to "1", the memory Bank 90 When set to "0", data is written to the memory. Bank 90 Data is read from the FET 84. This also gates the operation of the sensor (light detecting element) 86 and the modulator (light transmitting element) 84. This represents modulator mode (0) or sensor mode (1).

[0082] SxFlagRD, SxFlagWR, and SxFlagRST control the digital flag memory used for semantic labeling. SxFlagRST is the global address reset for all flag memories. SxFlagRD and SxFlagWR control memory access.

[0083] SxShift0,1,2 are externally driven in a three-phase sequence to move charge in either a clockwise or counterclockwise direction in the shift register memory 90 only in the addressed trixels (transmit-receive modules) 82. If a trixel is not addressed, its memory drive lines are forced into a safe state and do not affect the charging of the memory.

[0084] SxExtemal determines whether the SxAnalog and SxData lines are active (1) or whether data movement and access is simply internal (0).

[0085] Consider four combinations of these signals:

[0086] Image Load: SxFreeze=0, SxRWDir=1, SxExternal=1. This means that the addressed trixels82 memory cells will take data from the external SxAnalog lines and energize the trixels memory bank90 via the internal SxData lines. Because there are 120 SxAnalog lines, this action can be up to 120 wide. For implementations where a 120-wide DAC set is not appropriate, the lines can be externally connected in groups, and narrower accesses can be accommodated by simply enabling the MEMR lines in sequence. Regardless of the implemented external row width, typically only one MEMR line is enabled at a time to avoid contention (although the same DAC value can be sent to an entire row at once, if necessary).

[0087] ResultSave: SxFreeze=0, SxRWDir=0, SxExternal=l. This means that the addressed trixels82 memory cell will send data to the external SxAnalog line for conversion with the external ADC. Again, this can be up to 128 wide, but narrower implementations can be accommodated without design changes to the Sensay. Regardless of the implemented external row width, only one MEMC will be active at a time to avoid contentions (this is not optional for reads to avoid data contentions).

[0088] Sensor Mode: SxFreeze=0, SxRWDir=1, SxExternal=0. This means that any memory cell in the addressed trixels 82 will acquire data from the sensor 86 (via the pooling chain described below and in conjunction with SxShiftO,1,2) while shifting the existing voltage as a shift register set of memory values, saving the voltage as a new memory charge.

[0089] Modulator mode: SxFreeze=0, SxRWDir=0, SxExternal=0. This means that any memory cell in the addressed trixels 82 will send data (via the pooling chain) to the modulator (light transmitting element) 84, which will shift the voltage present as a shift register set of memory charges in relation to SxShiftO,1,2. Memory reads are non-destructive.

[0090] Exemplary row and column control line registers for trixels (transmit / receive modules) 82 are shown schematically in FIG. 14. In this example, the row and column control line registers comprise 235 individually addressed static 64-bit latches arranged as five row and five column lines per trixel. Their outputs are always active and are zeroed at power-up. These row and column control lines are used by each trixel 82 to configure its function relative to its neighbors. Each latch is individually addressed by asserting data on SxControl, setting an 8-bit address on SxAddr, and pulsing SxLatch.

[0091] A trixel's memory 90 is said to be "addressed" when its MEMR and MEMC are both asserted. Similarly, its light sensor is said to be "addressed" when its OPTC and OPTR are both asserted. Other trixels 82 functions are disabled when its ENBR and ENBC are deasserted. To completely disable a trixel 82, its MEMR, MEMC, OPTR, OPTC, FLAGR, and FLAGC are also deasserted.

[0092] The pool boundary lines 86 (POOLC and POOLR) affect entire columns and rows of trixels 82 and define the boundaries of the super trixels, as explained in more detail below. The rightmost and bottommost lines are always enabled, so only the 1079 POOLR and 1919 POOLC lines are present. Unused lines of the 64-bit latch are unconnected.

[0093] The *_SL and *_SR lines shift the respective register left or right on a rising edge.

[0094] SxReLUl and SxReLU2 (Figure 13) are driven by an external DAC. They are global to all trixels 82 and are applied to the sensor 86 readings to filter out weak information. SxLevel (Figure 13) is also driven by an external DAC. It is used by all trixels 86 as a preset level and is also added to the modulator driver 88 level, where it is used as a phase offset. Because the sensor display unit 14 or 26 is in either sensor or modulator (transmit) mode at any given time, as described above, there is no contention. The SxPeak (Figure 13) analog output signal is the signal from all trixels 82 (transmit / receive modules). As explained in more detail below, each trixels memory cell passes its highest value to a common trace. The value of this trace represents the highest global value seen by the entire trixels array since SxPeakreset was last asserted. This is used by external circuitry for system gain and normalization.

[0095] An example of an analog interface is shown schematically in Figure 15. The SxAnalog lines are 120 traces that connect each of the nine adjacent SxData rows. In other words, internally, the SxData0000 through SxData0008 row traces all connect to output pin SxAnalog00. The SxData0009 through SxData0017 row traces all connect to output pin SxAnalog001, etc. All SxAnalog pins are hardwired to the nine internal SxData traces. Only one trixels memory bank 90 at a time can drive or sense its local trace (forced by the external controller). When TMS is asserted, all SxAnalog and SxData lines are connected together.

[0096] Note that, as mentioned above, because the control lines are individually controllable, input or output schemes of any size, from 1 to 120 wide, can be implemented by simply connecting these lines together external to the Sensay (sensor-display device) and allowing only the appropriate trixels82 to match the architecture. The wider the interface, the faster the load and unload operations will be, but more external circuitry will be required. This allows for a high degree of customization without modifying the design.

[0097] The architecture of the sensor-display devices 14, 26 in the illustrative photonic neural network system 10 is built around pooling chains. As shown in Figures 9, 10, and 16, each of the transmit / receive modules (trixels) 82 in the array 80 has pooling boundary lines 96 along two of its edges, e.g., along the right and bottom edges, when Figures 9, 10, and 16 are oriented on paper. All sensor, modulator, memory read, and memory write accesses use the pooling chain to pass analog data within and between the trixels (sensor-display devices) 82. The function of the pooling boundary lines 96 is to connect or disconnect adjacent transmit / receive modules (trixels) 82 from the pooling chain, creating super-trixels or "islands." The pooling chain circuit connection of each adjacent trixel 82 to the boundary line 96 is shown in the expanded schematic diagram of the connection in Figure 17, at virtual locations nnnn, mmmm within the array 80 of trixels 82. When POOLC=0, all east-west trixels pooling chain connections for the entire column are open. When POOLR=0, all north-south trixels pooling chain connections for the entire row are open. All other trixels pooling chain connections remain closed. The effect of this pooling structure is to create an island of connected pooling chain lines. All trixels on the super trixels island share this chain, which is essentially a single low-impedance "trace." When POOLR is asserted, the transistor connecting the pooling chain for this trixels conducts, connecting the pooling chain to the trixels 82 south of it in the next row. When POOLC is asserted, POOLC connects to the pooling chain for trixels 82 east.

[0098] As mentioned above, the memory banks 90 in each of the transmit / receive modules (trixels) 82 are essentially shift registers, and shift register design and technology are well understood and readily available to those skilled in the art. Figure 18 shows the analog memory shift driver scheme. When addressed (MEMC and MEMR are both asserted) and unfrozen (SxFreeze is deasserted), any combination of SxShiftO,1,2 simply propagates to the output (MemShiftO,1,2) that actually drives the analog memory cell shift plates. If either MEMC or MEMR is deasserted for a Trixel, or if SxFreeze is asserted, the analog memory driver is automatically placed in a safe state (MemShiftO,1,2 = 0).

[0099] Figure 19 is a schematic diagram of an exemplary analog memory read interface for memory bank 90 (Figure 10). The memory can be read, and the analog data can be sent to the external SxAnalog interface via the internal SxData lines, or sent to the pooling chain 126 (if greater than SxReLU, otherwise zeroed) via either a maximum (diode) or average (resistor) circuit path. The unmodified value read from the analog memory is also used to charge the diode isolation capacitor (sample and hold circuit) and ultimately drive the SxPeak value across the sensor display device (sensay) 14, 26 (used externally for system gain control). Examples of these modes are shown schematically in Figures 20-24. Figure 20 shows the trixel analog memory read average to the pooling chain. Figure 21 shows the trixel analog memory read maximum to the pooling chain. Figure 22 shows the analog memory being read to the external data lines. Figure 23 shows the trixel analog memory peak value storage. Figure 24 shows the analog memory peak value reset.

[0100] A rectified linear unit (ReLU) is often applied to the data to suppress weak responses. The first sensor display device 14 and the second sensor display device 26 (sensay, 14, 26) are shown in Figure 1. 25 The sensor 14, 26 has a flexible dual-slope ReLU implementation that can produce a variety of responses, ranging from no effect (Example A) to conventional cutoff (Example B) to variable slope cutoff (Example C). Two external analog voltages driven by DACs control the transfer function. Because the sensors 14, 26 are unipolar designs, the "zero" position is nominally in the middle of the voltage range of the memory bank 90.

[0101] Writing to the analog memory 90 is simpler than reading. When the analog memory 90 of a transmit / receive module (trixels) 82 is addressed (both MEMC and MEMR are asserted, and SxRWDir=1), whatever the value on the local pooling chain is placed on the write pad as shown in Figure 26. To actually store the value in the analog memory cell, the shift line is cycled. Loading the analog memory 90 from the external data lines is shown in Figure 27.

[0102] The flag memory is a 640-bit LIFO (Last-In-First-Out) device (i.e., "stack") in each trixel 82 used to implement semantic labeling. If SxFlagRST=1, the internal address pointer is unconditionally set to zero for all trixels 82. The value does not need to be zeroed. Except for reset, the memory is only active when FLAGRmmmm=1 and FLAGCNNnn=1 for virtual trixel position nnnnnn,mmmm. If either FLAGRmmmm=0 or FLAGCNNnn=0, the signal has no effect on the memory. See Figure 14 for FLAGR and FLAGC.

[0103] Schematics of the Flag Memory Write and Flag Memory Read are shown in Figure 28 and Figure 29, respectively. If SxFlagWR=1, the comparator output is valid at the "D" memory input. On the falling edge of SxFlagWR going from "1" to "0", if SxFlagRD=0, the current flag bit, determined by the state of the trixel's current read value compared to the value on the pooling chain, is pushed onto the stack. That is, if the analog memory read voltage matches the pooling chain voltage, this trixel 82 is the "master" and a "1" is stored, otherwise a "0" is stored. See Figure 19 for FlagVAL.

[0104] Because the hysteresis is very small, it is possible for multiple trixels82 to see themselves as the "master" if they have very similar voltage levels. In such a case, what is ultimately read is the average voltage of the enabled trixels82 in this pooling group during the expansion pass. Since the "competing" voltages were nearly identical, this would have little practical effect.

[0105] On the rising edge of SxFlagRD=1, while SxFlagWR=0, the last bit written (i.e., the top of the stack) is read and applied as enabled FlagEN=1 to the Trixel memory read circuit (see Figure 19). The output is enabled as long as SxFlagRD=1.

[0106] If SxFlagWR=0 and SxFlagRD=0, then FlagEN=1. This is applied. SxFlagWR=1 and SxFlagRD=1 is illegal and should not be applied by the external controller. To avoid contention between the memory output and the comparator output, Flag EN is tri-stated in such cases.

[0107] Examples of optical control line configurations for reading the sensor 86 of a transceiver module (trixel) 82 into the pooling chain, reconfiguring the sensor 86, and writing the modulator (optical transmitter element) 84 from the pooling chain are shown in Figures 30, 31, and 32, respectively. The function of the optical control lines is to connect the trixels 82 at the intersection of the enabled OPTR and OPTC lines to their optical elements (modulators 84 or sensors 86) in the pooling chain. If SxRWDir=0 and SxExternal=0, data is read from the pooling chain to drive this trixel's modulator 84. If SxRWDir=1 and SxExternal=0, data is buffered from this trixel's sensor 86 and placed on the pooling chain. If SxExternal=1, both the modulator 84 and the sensor 86 are disconnected. Multiple sensors 86 can be enabled simultaneously, and an average of their values ​​will appear on the pooling chain for lower noise. Note also that when the sensor 86 is summing optical signals (frames of data) as described above, there is no other activity on the detectors 14, 26 (no clocks, etc.), resulting in a very low-noise measurement.

[0108] In modulator mode (SxRWDir=0) and internally driven (SxExternal=0), the outputs of all addressed trixel memory banks 90 are automatically pooled, and all light transmitting elements (modulators) 84 in the same super trixel (connected to the same pooling chain) "shine" with the same brightness. This constitutes resampling by replication.

[0109] Local ENB (ENBRmmmm=1 and ENBCNNn=1) allows individual optical transmitting elements (modulators) 84 to be disabled.

[0110] The drive level DL of the optical transmitter element (modulator) 84 is the sum of the pooling chain PC multiplied by the calibrated sensor value CS+1 plus SxLevel, given by the formula DL=(PC*(CS+1))+SxLevel. If Sxlnvert=1, the drive is inverted, i.e., 100% level is 0% modulation, 90% is 10%, etc.

[0111] The schematic diagrams of Figures 33A-B show an overview of the transceiver module (trixel) circuitry.

[0112] The above description is based on photonic neural network processing in inference mode, e.g., when the trained neural network is used to recognize images, speech, sounds, etc. Training a neural network using a photonic neural network, e.g., the photonic neural network system 10 described above, has several differences compared to a digital convolutional network system. As described above, during training of a typical digital convolutional neural network system, adjustments are made using a process called backpropagation to increase the network's likelihood of predicting the same type of image the next time. In a typical digital convolutional neural network, this data processing and backpropagation is performed multiple times until the predictions become reasonably accurate and no longer improve. The neural network can then be utilized in inference mode to classify new input data and predict results that will be inferred from its training. In digital convolutional neural networks, training is relatively simple because the backpropagation terms (or terms) and filters are all in the spatial domain. Taking the "right answer" and going back through the structure to calculate correction terms is slow, but still does not require a change of domain. Training in a photonic neural network is not as straightforward because the terms that need to be trained are in the frequency domain and the convolution results are in the spatial domain. While it is possible to use spatial domain data and calculate correction terms using a Fast Fourier Transform (FFT) algorithm and apply them to the Fourier filters used in radial modulator device 20, such calculations are very computationally intensive.

[0113] Alternatively, the example photonic neural network system 10 described above may be adapted to convert the training correction terms into Fourier transform terms that can then be added by the radial modulator 20 to a filter applied to the convolution in an iterative training process. An example adaptation for performing such a conversion optically instead of digitally includes adding a specialized Fourier optical sensor device 130 to the photonic convolution assembly 12, as shown in FIG. 34. The Fourier optical sensor device 130 is positioned on the opposite side of the polarizer 18 from the second sensor and display device 26 on the optical axis 62, and in axial alignment with the second sensor and display device 26. The Fourier optical sensor device 130 may also be Second Fourier transform lens 28 The focal length F2 is equal to Second Fourier transform lens 28 is located in the Fourier transform plane at a distance from . Second The Fourier optical sensor device 130 is Second Fourier transform lens 28 In that Fourier transform plane, Second The Fourier optical sensor device 130 can detect the Fourier transform of the frames of light data or images emanating from the second sensor and display device 26. Thus, the photonic neural network system 10 (Figure 1) The correction terms needed to train the second sensor-display device are calculated in spatial domain frames of correction data. 26 , which then displays (projects) the frames of correction data in the light field 132 onto the Fourier light sensor device 130. Thus, the frames of correction data in the light field 132 are fed to the Fourier transform lens 28 , i.e., the frame of correction data in the spatial domain is Fourier transformed into the Fourier domain at the speed of light in the Fourier light sensor device 130. The frame of correction data in the Fourier transform domain is then detected by the Fourier light sensor device 130 and used to adjust the filters of the radial modulator 20.

[0114] Typically, in inference mode, 3D convolution blocks are shifted out of memory and sent back through the photonic convolution assembly 12 for further levels of convolution and summation cycles in the memory banks 90 of the transceiver module (trixel) 82. (Figure 10) In a particular iterative convolution cycle, data frames present in the memory bank 90 are lost and refilled with subsequent 3D convolution blocks, all of which occurs very quickly as described above. However, in training mode, these intermediate frames of data are stored in the first and second sensor and display devices 14, 26. Transmitting and receiving module 82 The correction terms are extracted from memory bank 90 and transferred to external memory for use in performing the backpropagation digital calculations and writing correction terms in the spatial domain. These correction terms are then provided to second sensor and display device 26 as frames of correction data in the spatial domain for projection onto Fourier optical sensor device 130 and Fourier transformation, as shown in FIG. 34 and described above, so that the Fourier transformed frames of correction data can be detected by Fourier optical sensor device 130 in the Fourier domain for use as filters in radial modulator device 20 for further convolution cycles. This training mode extraction of intermediate correlation data, backpropagation digital calculations, and writing of correction terms takes some time, thus slowing the iterative convolution addition cycles compared to the inference mode of operation, but is still much faster than digital convolutional neural network training.

[0115] To incorporate the Fourier light sensor device 130 into the photonic convolution assembly 12, for example, as shown in FIG. 34, a half-wave variable polarizer 134 is positioned between the second sensor display device 26 and the polarizer 18 to rotate the plane of polarization by 90 degrees when frames of correction data are being projected by the second sensor display device 26 onto the Fourier light sensor 130. For example, in a normal inference mode of operation, the second sensor display device 26 displays P-polarized light reflected from the polarizer 18 onto the radial modulator device 20, and then, to display or project frames of correction data onto the Fourier light sensor 130 for training, the half-wave variable polarizer 134 is activated to rotate the plane of polarization of the projected light field by 90 degrees to become S-polarized, with the resulting light field 132 passing through the polarizer 18 to the Fourier light sensor 130.

[0116] The frames of corrected data are then filtered to train the neural network. Radial modulation device 20 (see Figure 3) It has a value that needs to be provided to a particular wedge segment 32. Thus, Figure 34 The frames of correction data provided to the second sensor and display unit 26 for projection onto the Fourier light sensor unit 130 are provided in a format that corresponds to the wedge segments 32 of the radial modulator device 20 (see FIG. 3) that need to be modulated in a corrected manner to train the neural network, so that these correction data are ultimately placed in a filter that drives the appropriate wedge segments 32 in a corrected manner. Thus, the Fourier light sensor unit 130 detects light 132 from the second sensor and display unit 26 in the same pattern as the wedge segments 32 of the radial modulator unit 20, so that the correction data for light 132 is detected, processed, and provided to the appropriate wedge segments 32 of the radial modulator unit 20.

[0117] The light projected from the second sensor display device 26 The radial modulation device 20 (FIG. 3)To facilitate detection according to the same pattern as the wedge segments 32, as described above, the exemplary Fourier optical sensor device 130 may be configured with a radial modulation device 20, as shown, for example, in FIG. (Figure 3) The optical sensor board 135 has a plurality of optical sensor elements 136 arranged in an optical sensor array 138 corresponding to the pattern of the wedge segments 32 and wedge sectors 34. As shown in Figure 35, A radial array lens plate 140 is placed in front of the photosensor array 138 and is connected to the radial modulator 20. (Figure 3) The radial modulator device 20 has a plurality of individual lens elements 142 arranged in a radial pattern of wedges and sectors that correspond to the wedge segments 32 and sectors 34 of the second sensor and display device 26. These lens elements 142 capture incident light 132 from the second sensor and display device 26 in a radial pattern that corresponds to the radial pattern of the wedge segments 32 and wedge sectors 34 of the radial modulator device 20, thereby capturing frames of correction data for the incident light 132 when formulated and programmed into the second sensor and display device 26. The segments of light captured by each lens element 142 are focused by the lens elements 142 as individual sub-beams 138 onto respective photosensor elements 136 and converted into electrical signals corresponding to the intensity of the light incident on the sensor elements 136, thus converting the frames of correction data in the incident light 132 into electrical data signals corresponding to the frames of correction data. As shown in Figure 34, These analog electrical data signals are then fed to the FPGA 116 The frames of data in the incident light 132 can be converted to a digital signal for processing into a correction filter by a Fourier transform lens, which can then be provided by the circuit block 110 for connection to the radial modulator device 20 via the interface 24. Again, the frames of data in the incident light 132 are passed through a Fourier transform lens 28and detected by the sensor elements 136 in the Fourier transform domain; therefore, the correction data in the signal sent from the Fourier light sensor unit 130 to the FPGA 116 or other electronic processing component is in the Fourier domain as needed to drive the wedge segments 32 of the radial modulator device 20. Due to the placement of optical components, the frames of correction data supplied to the second sensor and display unit 26 may need to be inverted so that the segments of light captured by the sensor elements 136 and the corresponding signals generated correspond to the appropriate wedge segments 32 of the radial modulator device 20. However, as noted above, because the correction terms are calculated in the spatial domain, no algorithmic constraints are imposed on the training. Once the normal training backpropagation calculations are performed, the optical system described above and shown in FIGS. 34 and 35 converts the spatial domain correction terms to their radial Fourier domain equivalents.

[0118] In another embodiment, illustrated in FIG. 36, a camera lens 150 is used to capture, as described above, for example, the photonic neural network 10 (Figure 1) A camera lens 150 is mounted on the photonic convolution assembly 12 in a manner that illuminates a real-world scene 152 onto the photonic convolution assembly 12 in the spatial domain as frames of data (images) for processing. For example, as shown in FIG. 35 , the camera lens 150 is mounted on the optical axis 62 so as to be axially aligned with the second sensor and display device 26 on the opposite side of the polarizer 18 from the second sensor and display device 26. A polarizer 154 is disposed between the camera lens 150 and the polarizer 18 to polarize the light field 156 transmitted by the camera lens 150 into a polarization plane that reflects from the polarizer 18. Thus, the light field 156 is reflected by the polarizer 18 to the first sensor and display device 14, as shown in FIG. 36 . The light sensor elements 86 in the transceiver module 82 (see FIGS. 9 and 10) of the first sensor and display device 14 detect and capture frames of data (images) in the light field 156 and store them in the memory bank 90 in the first sensor and display device 14, as described above. (Figure 10) The shutter device 158 on the camera lens 150 then closes on the camera lens 150 to terminate light transmission through the camera lens 150, and the first sensor and display device 14 can then begin processing the frame of data (image) through the photonic convolution assembly 12 in either the inference or training operations described above.

[0119] A bandpass filter 160 can also be provided with the camera lens 150 to allow only specific spectral frequencies of light to be transmitted to the photonic convolution assembly 12 as desired. The bandpass filter 160 can be a variable bandpass filter, if desired, such that various spectral frequency bands of light from the real-world scene 152 can be sequentially transmitted from the camera lens 150 to the photonic convolution assembly, while frames of data (images) within each frequency band are sequentially captured, thereby providing a set of hyperspectral images for convolution sequentially through the photonic convolution assembly. Such variable bandpass filters are well known. For example, a variable half-wave retarder can be used in combination with a fixed polarizer as a variable bandpass filter. Such a variable half-wave retarder in combination with a fixed polarizer can also be used as a shutter.

[0120] The foregoing description is believed to illustrate the principles of the present invention. Moreover, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the invention to the exact structures and processes shown and described above. Recourse may therefore be had to all suitable modifications and equivalents that are within the scope of the present invention. When used herein, the terms "comprise," "include," "including," and "having" are intended to specify the presence of stated features, integers, components, or steps, but they do not exclude the presence or addition of one or more other features, integers, components, steps, or groups thereof. The following is the invention as originally described in the application. <Claim 1> a first sensor display device comprising an array of transceiver modules, each transceiver module comprising a light sensor element, a light transmitter element, and a memory bank having a plurality of memory cells; A second sensor display device comprising an array of transceiver modules, each transceiver module comprising a light sensor element, a light transmitting element, and a memory bank having a plurality of memory cells; a radial modulator device having a plurality of modulating elements arranged at a plurality of radial distances and angular directions relative to an optical axis; a first Fourier transform lens disposed between the optical transmitting element of the first sensor-display device and the optical transmitting element of the radial modulator device; a second Fourier transform lens disposed between the optical transmitting element of the first sensor-display device and the radial modulator device; and 1. A system for convolving and summing data frames, wherein the radial modulator device is positioned at a focal length from a first Fourier transform lens and a focal length from a second Fourier transform lens such that the radial modulator device is positioned at a Fourier transform plane of both the first Fourier transform lens and the second Fourier transform lens. <Claim 2> 10. The system of claim 1, further comprising a system control component for forming and providing a filter to the radial modulator device, controlling the sequence of transmission of light fields comprising frames of data from the first and second sensor and display devices, convolving the frames of data with the filter of the radial modulator device, and detecting a light field comprising the convolved frames of data from the radial modulator device. <Claim 3> 10. The system of claim 1, wherein the light sensor element is a capacitive light sensor that accumulates charge from sensed light. <Claim 4> 1. A method of convolving and adding frames of data for a convolutional neural network, comprising: sequentially projecting the frames of data as a light field in a spatial domain along a first optical axis; Fourier transforming the light field in a sequential Fourier transform plane; sequentially convolving the light field at the Fourier transform plane with a light modulator having light-modulating segments spaced at various radial distances and angular orientations relative to an optical axis; Inverse Fourier transforming the sequence of convolved light fields into the spatial domain at the first sensor display location; sensing each of the convolved light fields pixel by pixel at the first sensor display location with a capacitive light sensor at a pixel location capable of storing charge in the spatial domain; causing a capacitive light sensor to accumulate charge resulting from sequentially sensing the convolved light field at the first sensor display location. A method having the following. <Claim 5> 5. The method of claim 4, further comprising shifting the charge accumulated in each sensor to a memory cell in the memory bank after sensing multiple convolved light fields such that the memory cell has accumulated charge resulting from light detected at a particular pixel location for sequentially sensed light fields. <Claim 6> convolving additional sequences of light fields with frames of data at said light modulator applying different filters; sensing additional sequences of the convolved light field pixel by pixel with the capacitive sensor and accumulating charge resulting from the sensing at each pixel location; shifting the stored charge in each sensor to a memory cell having pre-stored charge after sensing the plurality of convolved light fields, while shifting the pre-stored charge to another memory cell in the memory bank; repeating the above process to build a 3D convolution block of frames of convolved and summed data in a memory bank at each pixel location at the first sensor display location. 6. The method of claim 5, comprising: <Claim 7> Fourier transforming the convolved and summed frames of data forming a 3D convolution block in the sequential light field from the pixel locations at the first sensor display location and transmitting the frames back to the modulator in the Fourier transform plane; sequentially convolving the light field at the Fourier transform plane with an optical modulator having optical modulation segments spaced at different radial distances and angular orientations relative to an optical axis; Inverse Fourier transforming the sequence of convolved light fields into the spatial domain at a second sensor-display location; sensing each of the convolved light fields in the spatial domain pixel by pixel with a capacitive light sensor at the pixel location capable of storing charge at a second sensor display location; causing a capacitive light sensor to accumulate charge resulting from sequentially sensing the convolved light field at a second sensor display location; convolving additional sequences of light fields with frames of data at the light modulator applying different filters; sensing an additional sequence of the convolved light fields for each pixel at a second sensor display location with a capacitive sensor and accumulating charge resulting from the sensing at each pixel location; shifting the charge stored in each sensor at a second sensor-receiver location after sensing the plurality of convolved light fields to a memory cell having a previously stored charge while shifting the previously stored charge to another memory cell in the memory bank; repeating the above process to build a 3D convolution block of frames of convolved and summed data in a memory bank at each pixel location at a second sensor display location. 7. The method of claim 6, comprising: <Claim 8> The method of claim 7 , further comprising repeating the process for additional cycles. <Claim 9> The method of claim 8 , further comprising pooling a plurality of the sensors and memory banks together in repeated cycles of the process. <Claim 10> The method of claim 8 , comprising max pooling of the plurality of sensors and memory banks. <Claim 11> 8. The method of claim 7, further comprising transmitting a frame of convolved and summed data for each pixel at the first sensor display location with a phototransmitter element at the pixel location.

Claims

1. 1. A sensor and display device comprising an array of transceiver modules, each of the transceiver modules in the array comprising: an optical transmitter; a modulation driver connected to the optical transmitter for driving the optical transmitter; an optical receiver that generates charge from light received by the optical receiver; a memory bank; and a memory interface that connects the optical receiver to the memory bank, shifts charge from the optical receiver to the memory bank as data, and shifts the data as charge from the memory bank to the modulation driver, and drives the optical transmitter to transmit light based on the charge.

2. 2. The sensor display device of claim 1, wherein said memory bank comprises a shift register.

3. 2. The sensor display device of claim 1, wherein each of the transceiver modules in the array is configured to (i) transmit light as pixels of a transmitted image, whereby a combination of the pixels of light transmitted by the array of transceiver modules comprises the transmitted image, and (ii) receive light as pixels of an incident image, whereby a combination of the pixels of light received by the array of transceiver modules comprises the incident image.

4. 2. The sensor display device of claim 1, wherein the array has the transceiver modules arranged in rows and columns.

5. 5. The sensor display device of claim 4, including a pooling chain connectable to the memory banks of each of the transceiver modules in the array, the pooling chain allowing data from multiple transceiver modules in the array to be read together and data to be written to multiple transceiver modules in the array together.

6. 4. The sensor display device of claim 3, wherein the optical transmitter and optical receiver within the transceiver module are offset from each other by no more than 40 micrometers and both are fitted within the transceiver module having an area of ​​no more than 160 square micrometers.

Images