data processing array

Abstract

This disclosure relates to data processing arrays. A system (10) for convolving and summing data frames includes a first sensor-display device (14) and a second sensor-display device (26). Each sensor-display device (14, 26) includes an array (80) of transmit-receive modules (82). Each transmit-receive module (82) includes a light sensor element (86), a light emitter element (84), and a storage unit (90). A radial modulator device (20) is positioned to perform a Fourier transform on the transmission of the light field including the data frames. A filter implemented by the modulator element of the radial modulator device (20) convolves the light field including the data frames, which is then sensed pixel-by-pixel by the charge-accumulating light sensor element (86), thus summing the pixel values ​​of the sequentially convolved data frames.

Description

[0001] This application is a divisional application of patent application 201880075224.5, filed on September 20, 2018, entitled "Photonic Neural Network System". Technical Field

[0002] This invention relates to neural networks, and more particularly to convolutional neural networks with optical processing. Background Technology

[0003] Neural networks are well-known as computational systems, comprising many simple, highly interconnected processing elements that process information through their dynamic state responses to external inputs. Neural networks recognize patterns and are useful for clustering and classifying data. Computers can utilize neural networks to perform machine learning, where they learn to perform tasks by analyzing training examples. Typically, the examples have been pre-labeled by the user. For example, thousands of example images labeled "cat" or "no cat" can be fed into a neural network configured as an object recognition system or an image recognition system, and the results can then be used to identify cats in other images or indicate the absence of cats in other images, as appropriate. Alternatively, thousands of example images with various objects (e.g., cats, cows, horses, pigs, sheep, cars, trucks, boats, and airplanes) can be fed into such a neural network configured as an object recognition system and labeled accordingly, and the results can then be used to identify whether any cat, cow, horse, pig, sheep, car, truck, boat, or airplane is present in other images.

[0004] A Convolutional Neural Network (CNN) is a type of neural network that uses many identical copies of the same neurons. This allows the network to have a large number of neurons and represent computationally large models with a small number of actual values ​​describing how a neuron behaves (which needs to be learned). Convolution is a way of combining two signals to form a third signal. CNNs are typically implemented in software or programmable digital hardware.

[0005] Deep learning is the term used for stacked neural networks (i.e., networks consisting of multiple layers). A layer consists of nodes. Nodes are where computation takes place; they are loosely arranged neurons in the human brain that are triggered when they receive sufficient stimulation. A node combines inputs from data with a set of coefficients (or weights that amplify or suppress that input), thereby assigning importance to the inputs for the task the algorithm is trying to learn—for example, which input is most helpful in classifying data without error. These inputs are multiplied by their weights and summed, then the sum is passed through the node's activation function to determine whether and to what extent the signal proceeds further through the network to influence the final result, such as the classification action. A layer of nodes is a row of these neuron-like switches that turn on or off as inputs are fed through the network. Starting with the initial input layer that receives data, the output of each layer is simultaneously the input to the next layer. More than three layers of nodes are considered "deep" learning. In deep learning networks, each layer of a node is trained on a different set of features based on the output of the previous layer. Therefore, the more layers data (e.g., images, pictures, speech, etc.) passes through, the more complex the features a node can recognize. During training, a process called backpropagation is used to provide adjustments, increasing the likelihood that the network will predict the same type of image next time. This data processing and backpropagation are repeated until the predictions are fairly accurate and there is no further improvement. The neural network can then be used in inference mode to classify new input data and predict the results inferred from its training.

[0006] A typical convolutional neural network (CNN) has four basic layers of neurons in addition to the input and output layers: convolutional layers, activation layers, pooling layers, and fully connected layers. In the initial one or more convolutional layers, thousands of neurons act as the first set of filters, sifting through every part and pixel in the image to find patterns. As more and more images are processed, each neuron gradually learns to filter for specific features, which improves accuracy. Efficiently, one or more convolutional layers decompose an image into distinct features. Activation layers then highlight salient features—features that may be valuable or important in the final identification result. For example, eyes might indicate a face rather than a frying pan.

[0007] All convolutions and activations across the entire image generate massive amounts of data and can overwhelm a computer's computational power. Therefore, pooling is used to compress the data into a more manageable form. Pooling is the process of selecting the best data and discarding the rest, resulting in a lower-resolution dataset. Several types of pooling can be used, some of the more common types being "max pooling" and "average pooling."

[0008] Finally, in the fully connected layers, each reduced or "pooled" feature map or data is connected to an output node (neuron) representing an item the neural network is learning or has been enabled to identify, for example, cats, cows, horses, pigs, sheep, cars, trucks, boats, and airplanes. As the feature map or data passes through these output nodes, each node votes for any feature map or data that identifies it as its specific item. The final output of the network for the image data that has passed through the network is based on the votes of individual nodes. In the early stages of network training, voting may produce more erroneous outputs; however, as the number of images and backpropagation data increases to adjust the weights and optimize training, accuracy improves, and eventually, predictions or inferences about results based on the input data can be quite accurate.

[0009] The foregoing examples of related technologies and the limitations thereof are intended to be illustrative rather than exclusive or exhaustive of the subject matter. Other aspects and limitations of the related art will become apparent to those skilled in the art upon reading the specification and studying the accompanying drawings. Attached Figure Description

[0010] The accompanying drawings, which are incorporated herein by reference and form a part of this specification, illustrate some, but not unique or exclusive, example embodiments and / or features. The embodiments and drawings disclosed herein are intended to be illustrative rather than restrictive. In the drawings:

[0011] Figure 1 This is a functional block diagram for an example photonic neural network system;

[0012] Figure 2 It is used for Figure 1 An isometric view of an example photonic convolution assembly of a photonic neural network system performing optical processing and convolution on an image, with a portion of a second Fourier transform lens shown detached to display an array of light sensor-display components of a second sensor-display device;

[0013] Figure 3 yes Figure 1 Front view of an example radial modulator in an example photonic neural network;

[0014] Figure 4 yes Figure 3 An isometric view of the central portion of an example radial modulator and an enlarged view of an example optical modulator segment of a radial modulator.

[0015] Figure 5 It is an isometric plot showing the convolution function of an example photonic convolution assembly of an example photonic neural network system;

[0016] Figure 6 yes Figure 2The example photonic convolution assembly is shown in a schematic top plan view, which illustrates a first sensor-display device that displays data frames (images) and a second sensor-display device that senses the convolved data frames;

[0017] Figure 7 yes Figure 2 The schematic top plan view of the example photonic convolution assembly shows a second sensor-display device that displays data frames (images) and a first sensor-display device that senses the convolved data frames;

[0018] Figure 8 This is a functional block diagram of the array of transmit-receive modules in the first sensor-display device;

[0019] Figure 9 This is an enlarged isometric view of a portion of the transmitter-receiver module array;

[0020] Figure 10 This is an enlarged isometric view of an example transmit-receive module;

[0021] Figure 11 yes Figure 9 and Figure 10 A perspective view of an example optical transmitter element in an example transmit-receive module;

[0022] Figure 12 This is a functional block diagram of an example system interface for an external interface to a sensor-display device;

[0023] Figure 13 This is a functional block diagram of an example external interface for a sensor-display device;

[0024] Figure 14 This is a schematic diagram of the row and column control line registers for the array used in the transmit-receive module;

[0025] Figure 15 This is a schematic diagram of the analog data lines to the transmit-receive module;

[0026] Figure 16 This is a schematic representation of some of the transmit-receive modules (trixels) in an array interconnected in a pooling chain;

[0027] Figure 17 This is an enlarged schematic diagram of the pooling boundary line of the transmit-receive module (pixel) and the interconnection of the adjacent transmit-receive module (trixel);

[0028] Figure 18 This is an example memory shift driver used in the example transmit / receive module of the example photonic neural network system 10;

[0029] Figure 19 This is a schematic diagram of an example simulated memory read interface for a storage device;

[0030] Figure 20 This is a schematic diagram illustrating simulated memory reads from the transmit-receive module (trixel) for pooling chain averaging;

[0031] Figure 21 This is a schematic diagram illustrating simulated memory reads from the largest transmit-receive module (trixel) in the pooling chain;

[0032] Figure 22 This is a schematic diagram illustrating the analog memory read from the transmit-receive module (trixel) to the external data line;

[0033] Figure 23 This is a schematic diagram illustrating the peak storage of the analog memory in the trixel transmitter-receiver module;

[0034] Figure 24 The peak reset of the analog memory in the transmit-receive module (trixel) is shown;

[0035] Figure 25 A graphical representation of the response of an example rectified linear unit (ReLU) is shown;

[0036] Figure 26 This is a schematic diagram showing the writing to the analog memory of the transmit-receive module (trixel);

[0037] Figure 27 This is a schematic diagram showing the loading of analog memory from an external data line;

[0038] Figure 28 This is a schematic diagram showing the flag memory write circuit;

[0039] Figure 29 This is a schematic diagram showing the flag memory read circuit;

[0040] Figure 30 This is a schematic diagram showing the optical control line setup for reading the transmitter-receiver module (trixel) sensor into the pooling chain;

[0041] Figure 31 This is a schematic diagram showing the optical control lines used to reset the trixel sensor;

[0042] Figure 32 This is a schematic diagram showing the optical control line setup for writing from the pooling chain to the optical emitter element (modulator);

[0043] Figures 33A-33BA schematic diagram showing an overview of the trixel circuit is provided.

[0044] Figure 34 An example photonic convolution assembly of a Fourier optical sensor device is shown, which has a Fourier transform for corrected data frames in training mode.

[0045] Figure 35 This is a schematic isometric view of an example Fourier optical sensor device; and

[0046] Figure 36 An example photonic convolution assembly is shown, featuring a camera lens embodiment for incorporating real-world data frames (images) into the photonic convolution assembly. Detailed Implementation

[0047] Figure 1 A functional block diagram of an example photonic neural network system 10 is shown, and Figure 2 An isometric view of an example photonic convolution assembly 12 used for optical processing and convolution of images from a photonic neural network system 10 is shown. Convolution using this photonic neural network system 10 is performed via optical Fourier transform, which significantly improves speed, resolution, and power efficiency compared to digital spatial convolution. Therefore, generating and using a neural network can be done several orders of magnitude faster and with much lower power consumption compared to typical computational convolution and processing using computer algorithms. The power consumption is very low because all convolutions and sums are fully analog, full-frame photonic computations. Summations are accomplished by building charges in a capacitive optical sensor, as will be explained below; this is an analog process. The sensor is very low-noise and has no timing or other transient noise sources, making the summation a very low-noise process. The photonic neural network 10 can accept and process any data (e.g., images, videos, sounds, speech patterns, or anything typically processed by convolutional neural networks), and it supports all existing convolutional neural network architectures and training methods. The photonic neural network 10 also provides full-frame image parallelism at full resolution, processed at the speed of light, and has an architecture in which each data element is in its ideal position for the next stage, thus achieving near 100% efficiency. Other advantages can be understood from this description.

[0048] refer to Figure 1 and Figure 2Both utilize the photonic convolution assembly 12 to perform optical processing on the image of the example photonic neural network system 10. Essentially, the first sensor-display device 14 projects a data frame (e.g., an optical representation of an image or other data such as sound, voice patterns, video, etc.) as a modulated light field 15, which passes through the first Fourier transform lens 16 and through a polarizer 18 to a radial modulator device 20 placed in the focal plane of the lens 16, as will be explained in more detail below. The first sensor-display device 14 forms the data frame projected by itself based on values ​​or signals provided to it via electronic data interface 22 by supporting electronics (described in more detail below). The Fourier transform lens 16 can be a diffractive lens, a solid convex lens, or any other form of Fourier transform lens that may be used. Furthermore, a fiber panel (not shown) can be placed in front of the lens 16 to collimate the light before it enters the lens 16.

[0049] Lens 16, located at the focal plane (also known as the Fourier transform plane), transforms the data frame (e.g., an image) at the surface of the radial modulator device 20 into its Fourier equivalent. The radial modulator device 20 modulates the light field 15, comprising the Fourier equivalent of the data frame, based on a pattern (also known as a “filter”) loaded into the radial modulator device 20 via electronic data interface 24 by supporting electronics (described in more detail below), and reflects the modulated data frame to the second sensor-display device 26 of the detection result. The reflected light field, comprising the modulated data frame, undergoes an inverse Fourier transform back into the spatial domain over the distance from the radial modulator device 20 to the second sensor-display device 26, thus the modulated data frame incident on the second sensor-display device 26 is a spatial domain characteristic of the data frame that has already passed through the radial modulator device 20 (i.e., not filtered out by the radial modulator device 20). The result is detected by the second sensor-display module 26 on a pixel-by-pixel basis, where the light incident on each pixel generates a charge proportional to the intensity of the light and the time the light is incident on the pixel. Each data frame transmitted from the first sensor-display device can be modulated by the radial modulator device 20 using one or more filters (modes). Furthermore, the second sensor-display device can receive one or more data frames modulated by one or more filters applied in the radial modulator device 20 from the first sensor-display device 14. Therefore, the charge constructed for each pixel in the second sensor-display device 26 can be the sum of one or more modulated (i.e., filtered) modes of one or more data frames, as will be described in more detail below, thereby constituting a convolution of one or more data frames projected by the first sensor-display device 14.

[0050] For example, a data frame can be sequentially projected by the first sensor-display device 14, first in red, then in green, and then in blue, and the radial modulator device 20 can apply the same or different filters (mode modulation) to each of the red, green, and blue projections. All these modulated data frames can be sequentially detected by the second sensor-display device 26, where light from each of those sequentially modulated data frames is added to a charge for each corresponding pixel. These charges for each corresponding pixel are then transferred to corresponding memory units that store the summed results for each corresponding pixel, thereby including the stored pixel values ​​of the convolution of the data frames projected in the spatial domain by the first sensor-display device 14 and convolved in the Fourier transform domain by the filters in the radial modulator device 20.

[0051] This process can be repeated for the red, green, and blue projections of the same data frame from the first sensor-display device 14, but using different filters in the radial modulator device 20. Therefore, different modulation patterns from the Fourier transform domain are reflected from the radial modulator device 20 to the second sensor-display device 26, thereby generating another summed result and another set of stored pixel values ​​for another convolved data frame in the memory of the second sensor-display device 26. Based on the data frames projected by the first sensor-display device 14 to all those different filters of the radial modulator device 20, the accumulated convolved data frames in the second sensor-display device 26 form 3D convolutional blocks. In summary, the data frames from the first sensor-display device 14 are multiplied by the radial modulator device 20 with a series of filters in the Fourier plane, and summed by the second sensor-display device in a sequence that constructs 3D convolutional blocks in the memory of the second sensor-display device 26. Assuming sufficient memory capacity to store all pixel values ​​for all convolved data frames in a 3D convolutional block, any number of such convolved data frames can be accumulated into a 3D convolutional block. This 3D convolutional block can be considered the first stage in a neural network.

[0052] For the next convolutional block or the next stage, the first sensor-display device 14 and the second sensor-display device 26 exchange functions. The 3D convolutional block in the memory of the second sensor-display device 26 is transformed into data frames for the next convolutional sequence. For example, each accumulated convolutional data frame in the 3D convolutional block in the memory of the second sensor-display device 26 is projected by the second sensor-display device 26 onto the radial modulator device 20 through the second Fourier transform lens 28, where the data frame is multiplied by a filter and reflected back to the first sensor-display device 14. The first sensor-display device 14 detects and sums a series of such convolutional and summed data frames to construct the next 3D convolutional block in the memory of the first sensor-display device 14.

[0053] The process cycle is in Figure 5 It is shown schematically in the diagram.

[0054] For any convolutional neural network architecture, those convolution process cycles that repeatedly project data frames between the first sensor-display device 14 and the second sensor-display device 26 can be repeated as needed. As more filters are applied in subsequent cycles, pooling can be performed by feeding the accumulated charge from multiple pixel detections into a single memory cell instead of feeding the accumulated charge from each pixel detection into individual memory cells, as will be described in more detail below. Therefore, the example photonic neural network 10 can be used to develop convolutional neural networks with many levels of abstraction.

[0055] Figure 3 The image shows a front view of an example radial modulator device 20, and... Figure 4 The image shows a perspective view of the central portion of an example radial modulator device 20, and an enlarged view of an example segment optical modulator within the example radial modulator device 20. The radial modulator device 20 has an optically active region 30 comprising a plurality of optically modulated wedge segments 32 (wedge segments), each of which is independently actuated to modulate light incident on a corresponding wedge segment 32. Figure 2 , Figure 3 and Figure 4 In the example radial modulator device 20 shown, wedge segments 32 are grouped into a plurality of wedge sectors 34, each of which extends radially outward from the central component 36 and together forms the optical active region 30 of the radial modulator device 20. Figure 3 and Figure 4In this diagram, only some of the markings in wedge segments 32 and sectors 43 are marked with those reference numerals to avoid confusion in the drawings. However, using this illustration, those skilled in the art will see and understand the location of all wedge segments 32 and wedge sectors 34 in the example radial modulator device 20. Figure 3 and Figure 4 In the example radial modulator device 20 shown, the wedge segment 32 is arranged to form a circular optical active region 30, but other shapes may also be used.

[0056] As explained above, each of the wedge segments 32 is optically active in the sense that each wedge segment 32 can be activated to transmit light, block light, or modulate the transmission of light between complete transmission and blocking. Therefore, any combination of one or more wedge segments 32 can be used to modulate a light beam or light field incident on the optically active region 30. Spatial light modulators can be designed and constructed to modulate light in a variety of ways. For example, U.S. Patent No. 7,103,223, granted to Rikk Crill on September 5, 2006, describes a method for modulating light in conjunction with… Figure 2 and Figure 3 The radial modulator device 20, similar to radial spatial light modulators, uses birefringent liquid crystal materials to modulate wedge-shaped segments. The article "Active metasurface modulator with electro-optic polymer using bimodal plasmonic resonance" (Optics Express, Vol. 25, No. 24, November 17, 2017) by Zhang et al. describes an electrically tunable metallic grating with an electro-optic polymer, which modulates light and is suitable for ultrathin surface normal applications. Figure 4The diagram illustrates a metasurface optical modulator element 40 shaped to serve as an optical active wedge segment 32 in an example radial modulator device 20. The metasurface optical modulator element 40 includes a metallic grating structure 42. The grating structure 42 includes an electro-optic polymer 44 sandwiched between a bottom metal (e.g., Au) layer 46 and interleaved top thin metal (e.g., Au) grating layers 48, all constructed on a substrate 50. The period of the grating 44 is shorter than the wavelength of light, thus preventing diffraction. The thickness of the top metal layer 48 is greater than the skin depth to eliminate direct coupling from the incident light L to the electro-optic polymer 44. The bottom metal layer 46 also has the same thickness, making it function as a near-perfect reflector. Essentially, light L enters the top of the metasurface light modulator element 40, undergoes a phase shift in the electro-optic polymer 44 (which is periodically polarized by the application of a polarization voltage 45), is reflected away from the bottom metal layer 46, undergoes a further phase shift during its second (i.e., reflection) passage, and exits the top surface with a 90-degree polarization rotation. Other wedge segments 32 in the example radial modulator device 20 have the same type of light modulator element 40, but the size and shape of the light modulator element 40 are configured to fit and substantially fill each particular wedge segment 32. The central assembly 36 may also have a light modulator element 40.

[0057] Figure 2-7 The example radial modulator device 20 shown and described above is a reflecting device, in which incident light is modulated by the wedge segment 32 and reflected. However, the radial modulator can alternatively be a transmitting device, in which incident light is modulated and transmitted through the radial modulator. Of course, the positions of the optical components (e.g., sensor-display devices, lenses, and polarizers) must be rearranged to route the light field to the corresponding optical components in an appropriate sequence, but those skilled in the art will know how to perform such a rearrangement once they become familiar with the example photonic neural network 10 described above.

[0058] like Figure 3 and Figure 4As shown and briefly explained above, the optically active wedge segments 32 are grouped into multiple wedge sectors 34 that extend radially from the circular central component 36 to the periphery of the optically active region 30. The wedge segments 32 are also arranged in concentric rings around the central component 36. Except for the innermost concentric ring, the outer radius of each concentric ring of the wedge segments 32 is twice the outer radius of the immediately adjacent inner ring, matching the proportional distribution in the Fourier transform. Therefore, the length of each radially outward-facing subsequent wedge segment 32 in the wedge sector 34 is twice the length of the immediately preceding wedge segment 32. A detailed explanation of how the radial modulator functions as a filter on the Fourier transform plane of the image can be found, for example, in U.S. Patent No. 7,103,223 to Rikk Crill, issued September 5, 2006. It is sufficient to say here that light energy from higher spatial frequency shape contents in the spatial domain diffuses radially further than light energy from lower spatial frequency contents, while the angular orientation and intensity of light energy from lower and higher spatial frequency contents are preserved in the Fourier transform of the image. Therefore, light energy transmitted by a specific wedge segment 32 placed at a specific angular orientation and a specific radial distance from the center (optical axis) of the Fourier-transformed image in the Fourier transform plane will be inversely Fourier-transformed back into the spatial domain in the projection, displaying only the shape contents (features) in the original image that have the same angular orientation as the specific wedge segment 32, and only the shape contents (features) in that angular orientation that have spatial frequencies corresponding to the radial extent to which such light energy diffuses in the Fourier transform plane. The light intensity (brightness) of these inversely Fourier-transformed features (shape contents) will correspond to the light intensity (brightness) of those features (shape contents) in the original image, and these inversely Fourier-transformed features will be located in the same positions they are in the original image. Of course, the shape content (features) included in the light energy of the original image that are blocked by specific wedge segments 32 and not transmitted in the Fourier transform plane will be lost in the inverse Fourier transform image returned to the spatial domain. Moreover, the shape content (features) included in the light energy that are only partially blocked by specific wedge segments 32 and therefore partially transmitted in the Fourier transform plane will be inversely transformed back to the spatial domain with the same angular orientation and specific spatial frequency as explained above, but with reduced intensity (brightness). Therefore, as explained above and as will be explained in more detail below, the inverse Fourier transform image returned to the spatial domain is constructed for... Figure 5 The 3D convolutional blocks of the neural network shown are detected and used in convolutional images, where some of the shape content (features) of the original image are preserved in the inverse Fourier transform image with full or partial intensity (brightness), and some of the shape content (features) are completely removed in the inverse Fourier transform image.

[0059] Therefore, refer to Figure 5 A first filter 54 is loaded into the radial modulator device 20 via data interface 24. This first filter 54 configures the wedge segment 32 to either completely or partially transmit or block light in a mode set by the first filter 54. This includes, for example, a set of LEGO bricks. TM The first data frame 50 of the image of the toy building blocks 52 is loaded into the first sensor-display device 14 via the data interface 22, thus displaying the display components in the first sensor-display device 14. Figure 5 (Not shown in the text) Set to show including LEGO TM Data frames of images of toy building blocks 52, such as Figure 5 As can be seen from this, laser illumination 13 is directed at the first sensor-display device 14, and this laser illumination 13 essentially illuminates the LEGO display. TM The light from the data frame 50 of the toy building block image passes through the first Fourier transform lens 16 and is projected onto the radial modulator device 20 via the polarizer 18. The radial modulator device 20 is positioned in the Fourier transform plane at a distance F1 from the focal length F1 of the first Fourier transform lens 16, that is, in the focal plane of the first Fourier transform lens 16, as... Figure 6 As schematically shown, the Fourier transform lens 16 focuses the light field 15, including the image 50, onto a focal point on the surface of the radial modulator device 20. (Including LEGO) TM The data frame 50 of the image of the toy building blocks 52 is convolved in the Fourier transform domain by a filter through a wedge segment 32 of the radial modulator device 20. This filter either completely or partially reflects some light or blocks some light included in the image, as explained above. The wedge segment 32 is phase-shifted, thus rotating the polarization of the reflected light, such that the light reflected by the radial modulator device 20 is reflected by polarizer 18 to the second sensor-display device 26, as indicated by the reflected light field 56. Therefore, as explained above, in the convolved image incident on the second sensor-display device 26, LEGO TM Some shape content (features) of the original data frame 50 of the image of toy building blocks 52 is lost or weakened, i.e., filtered out, such as Figure 5 As shown in the diagram, a convolved data frame (image) is detected at the second sensor-display device 26 and summed with some subsequent convolved images in the second sensor-display device 26 to form a first convolved and summed data frame (image) 58. This first convolved and summed data frame (image) 58 can be transferred to storage for accumulation with subsequent convolved and summed data frames (images) to construct a 3D convolutional block 64, as shown in the diagram. Figure 5 It is shown schematically in the middle.

[0060] Then, the second sensor-display device 26 and the first sensor-display device 14 switch roles as explained above, such that the second sensor-display device 26 enters display mode and the first sensor-display device 14 enters sensor mode. With the second sensor-display device 26 in display mode, the first convolved and summed data frame (image) 58 is then projected back from the second sensor-display device 26 to the radial modulator device 20, as... Figure 7 As schematically shown, the first convolved and summed data frame (image) 58 is convolved in the radial modulator device 20 using an additional filter, and then reflected back to the first sensor-display device 14 by the radial modulator device 20. This role reversal occurs in... Figure 7 The diagram schematically illustrates that the second sensor-display device 26 is switched to display mode, and the first sensor-display device 14 is switched to sensor mode. In display mode, the display components of the second sensor-display device 26 are programmed to display a first convolved and summed data frame (image) 58. Therefore, the laser illumination 60 on the second sensor-display device 26 projects light from the first convolved and summed data frame (image) 58 through a second Fourier transform lens 28 along the second optical axis 62 onto a polarizer 18, which reflects the light field 64 along the first optical axis 61 to the radial modulator device 20. The optical distance between the second Fourier transform lens 28 and the radial modulator device 20 along the second optical axis 62 and the first optical axis 61 is equal to the focal length of the second Fourier transform lens 28. Therefore, the light field 64 at the radial modulator device 20 in the Fourier transform plane is the Fourier transform of the first convolved and summed data frame (image) 58. Radial modulator device 20 applies a filter to the Fourier transform of the first convolved and summed data frame to provide a second convolution to the data frame, and reflects the data frame in the case of phase shift, as explained above, so that it can then be propagated along the first optical axis 61 to the first sensor-display device 14. The first sensor-display device 14, now in the role of detector as explained above, detects the data frame (image) convolved by the filter applied by radial modulator device 20. The convolved data frame (image) detected by the first sensor-display device 14 is then summed by the first sensor-display device 14 with some other convolved data frames (images) subsequently detected by the first sensor-detector device 14, and this convolved and summed data frame (image) is transferred to storage and used to construct the second 3D convolutional block 66, which... Figure 7 It is shown schematically in the diagram.

[0061] Then, the roles of the first sensor-display device 14 and the second sensor-display device 26 are switched again, and the convolved and summed data frames sensed and summed by the first sensor-display device 14 are projected back by the system in the same manner as described above to perform the following operations: convolved by the radial modulator device 20 and then detected and summed by the second sensor-display device 26 to continue building the first 3D convolutional block 64; and sent back by the system to convolve and sum using additional filters to continue building the second 3D convolutional block 66. This process is repeated multiple times as needed to build increasingly deeper convolutions, or until the inference neural network is complete.

[0062] The first sensor-display device 14 and the second sensor-display device 26 may each have a storage (as explained in more detail below) for storing the received data frames (images) that are summed with subsequently received convolved data frames (images) to construct 3D convolutional blocks 64, 66 and subsequent convolutional blocks. Therefore, except for the first data frame (image) loaded into the system, the input data frame may always reside in the storage of one of the sensor-display devices 14, 26 from a previous convolution cycle. A series of filters 68 are loaded into the radial modulator device 20 synchronously with the data frames (images) displayed by the respective first sensor-display device 14 and second sensor-display device 26 for convolving the data frames (images) using the filters.

[0063] Aside from being optically computed in the Fourier transform domain, the convolutions from this photonic neural network system 10 are identical to those computed by conventional digital methods. However, the efficiency of full-frame parallelism at any resolution (as explained in more detail below) at light speed, utilizing an architecture where each data element is ideally positioned for its next convolutional stage for a period, approaches 100%, thus, constructing convolutional blocks using the example photonic neural network system 10, as described above and in more detail below, offers higher power and speed than convolutions computed by conventional digital methods.

[0064] As explained above, each of the first sensor-display device 14 and the second sensor-display device 26 has both light detection capability and image display capability on a pixel-by-pixel basis. In this example photonic neural network system 10, the first sensor-display device 14 and the second sensor-display device 26 have essentially the same components and structures; therefore, the details of these devices are described below primarily with reference to the first sensor-display device 14, but it should be understood that these details also refer to the second sensor-display device 26. Therefore, in the following description, the first sensor-display device 14 is sometimes simply referred to as sensor-display device 14. A functional block diagram of the example sensor-display device 14 is shown in... Figure 8 The diagram shows an array 80 including transmit-receive modules 82, each of which has light transmission and light detection elements and a storage unit, as will be described in more detail below. Row and column controls for the transmit-receive modules 82 in the array 80, as well as external control circuitry, are provided. Figure 8 A hybrid analog and digital interface 24 (not shown in the image) is used for inputting and outputting data, which will be described in more detail below. Figure 9 The enlarged portion of array 80, schematically shown in the diagram, illustrates an example transmit-receive module 82 within array 80, and Figure 10 A further enlarged schematic representation of the example transmit-receive module 82 is shown. Each of the example transmit-receive modules 82 includes both a miniature light emitter element 84 and a miniature light detector (sensor) element 86, which are small enough for the transmit-receive module and close enough to be efficiently used as a light emitter and a light receiver at substantially the same pixel location in an image or data frame of sufficient resolution for operating a useful neural network, for example, whose results are at least as useful as those from typical computational convolutions and processed using computer algorithms. For example, for a neural network system 10 in which the example photonic neural network system 10 is as useful as those from typical computational convolutions and processed using computer algorithms, the miniature light emitter element 84 and the miniature light detector element 86 may be offset from each other by no more than 40 micrometers, and both may be housed within a transmit-receive module 82 having an area of ​​no more than 160 square micrometers.

[0065] like Figure 10As best seen in the example, besides the light emitter element 84 and the light sensor or detector element 86, the example transmit-receive module 82 also includes a modulator driver 88, a memory bank 90, a memory interface 92, analog and digital control elements 94, pooling connections 96 for pooling connections with adjacent transmit-receive modules 82 in the array 80, a pooling control element 98, and a sensing amplifier 100. In display mode, for example, when the first sensor-display device projects a data frame (image) onto the radial modulator 20 as explained above, the laser illumination is directed towards the back of the transmit-receive module 82, as... Figure 2 , Figure 5 and Figure 6 As shown in the diagram, the first data frame (image) consists of pixel values ​​for each pixel of the data frame (image). These pixel values ​​are fed into the corresponding transmit-receive modules 82 of the array 80 in a pattern that will create the first data frame (image) in the light field of laser illumination. For example... Figure 10 In the transmit-receive module 82 shown, a single transmit-receive module has its pixel values ​​fed to an analog-digital control element 94, which shifts the pixel values ​​to a modulation driver 88. The modulation driver 88 modulates the voltage on the light emitting element 84 according to the pixel values, thereby modulating the laser illumination incident on the light emitting element 84 in such a way that the pixels emitting laser illumination simultaneously with the corresponding pixels of other light emitting elements 84 are emitting light, and these pixels together apply a data frame (image) to the laser field emitted through the array 80. After a first data frame (image) is emitted by the first sensor-display device 14 and a convolved data frame (image) is guided back to the first sensor-display device 14, the light field including the convolved data frame (image) is incident on the sensors 86 of all the transmit-receive modules 82 in the array 80 of the first sensor-display device 14. Therefore, the sensor 86 on each transmit-receive module 82 in the array detects the pixels of the incident light field, and thus detects the pixels of the data frame (image) included in the incident light field. Those skilled in the art will understand how optical sensors (e.g., charge-coupled devices (CCDs)) are constructed and operate, and that such optical sensors, or similar optical sensors, can be used in sensor 86. Essentially, each optical sensor has a photosensitive photodiode or capacitive component that responds to incident photons by absorbing most of the energy in a photon and creating a charge proportional to the intensity of the incident light, and accumulating the charge in the capacitive component. The longer the light is incident on the sensor, the more charge accumulates in the capacitive component. Therefore, the amount of light energy incident on each pixel of sensor 86 causes charge to build up in that sensor 86, and the magnitude of the charge is proportional to the intensity of the incident light in that pixel and the time the light in that pixel is incident on sensor 86.

[0066] As explained above, if a series of convolved data frames (images) are transmitted to and received by the sensor display device 14, the light energy (photons) of the sequential light field of the sequential data frames (images) will bring charge into the sensor 86. Therefore, the sequential pixel of the light field energy from the sequential light field sensed by the individual sensor 86 can be accumulated (i.e., added) to the capacitive component of that individual sensor 86, resulting in the accumulation of charge in the sensor 86, which is the sum of the light energy of the sequence of light fields at that particular pixel location. Therefore, the array 80 of the transmitter-receiver module 82 of the sensor-display device 14 senses and sums the sequence of convolved data frames (images) received by the sensor-display device 14 on a pixel-by-pixel basis. Then, as explained above, when a predetermined number of individual convolved data frames (images) have been received and summed, the accumulated (summed) charge in the sensor 86 of each individual transmitter-receiver module 82 is shifted to the storage 90 of that individual transmitter-receiver module 82. The same operation of shifting the accumulated (summed) charge in sensor 86 to storage 90 occurs simultaneously in all transmit-receive modules 82 of array 90. Therefore, during this shift operation, the array 90 of transmit-receive modules 92 has already stored in the individual storage 90 of each transmit-receive module 92 a complete convolved and summed data frame (image) resulting from the series or sequence of convolutions and summations of the input data frame (image). Reference Figure 10 The pixel value of the first convolved and summed data frame (image) at the pixel location of the specific transmit-receive module 82 is shifted from the sensor 86 to the first memory cell 102. Therefore, the composite of all the first memory cells 102 in the storage bank 90 of all transmit-receive modules 82 of the array 80 includes convolved and summed data frames (images) on a pixel-by-pixel basis.

[0067] Then, when the subsequent second series or sequence of data frames (images) is convolved and summed as described above, the accumulated charge in the sensor 86 for the pixels of the resulting second convolved and summed data frame (image) is shifted to the second memory cell 104 of memory 90, as the charge of the pixels from the first convolved and summed data frame (image) is simultaneously shifted to the second memory cell 104 of memory 90. Those skilled in the art understand how such shift register memories are manufactured and operated, therefore a detailed description is not necessary for the purposes of this explanation. This same process occurs simultaneously in the other transmit-receive modules 82 in array 80. Therefore, the collection of all first memory cells 102 and second memory cells 104 in the storage bank 90 of all transmit-receive modules 82 of array 80 includes, on a pixel-by-pixel basis, the first convolved and summed data frame (image) and the second convolved and summed data frame (image).

[0068] As described above, as an increasing series or sequence of data frames (images) is convolved and summed, the summed pixel values ​​of these sequentially convolved and summed data frames (images) are shifted sequentially into the first memory cell 102, while each preceding pixel value is shifted further along the memory cells of the storage bank 90, and as explained above, this process occurs simultaneously in all transmit-receive modules 82 of the array 80. Therefore, all such convolved and summed data frames (images) from all series or sequences of convolution and summation are stored on a pixel-by-pixel basis in the memory cells of the storage bank 90 of the array 80 of the transmit-receive modules 82. Each of these convolved and summed data frames (images) is sometimes referred to as a convolution, so the array 80 of the transmit-receive modules 82 can maintain as many convolutions as the individual memory cells present in the individual storage banks 90 of the transmit-receive modules 82. For example, Figure 9 and Figure 10 The example transmit-receive modules 82, schematically shown, each have a storage bank 90 consisting of 64 individual memory units 102, 104, ..., n. Therefore, on a full-resolution pixel-by-pixel basis, the example array 80 of the transmit-receive modules 82 can maintain as many convolutions as a data frame (image) of 64 convolutions. When the transmit-receive modules 82 (see...) Figure 1When the light emitter element 84 and light sensor element 86 in the transmit-receive module 82 are pooled together with the light emitter element 84 and light sensor element 86 of the adjacent transmit-receive module 82, as described in more detail below, all the light emitter elements 84 and light sensor elements 86 in the pooled group show the same brightness for a coarse representation of the data frame (image). Under such pooling conditions, the memory bank 90 of the transmit-receive module 82 in the pooled group can be used sequentially to store the sensed and summed results for the entire pooled group of transmit-receive modules 82, thereby increasing the effective memory capacity and depth. For example, if transmit-receive modules 82, each having a memory bank 90 comprising 64 memory cells, are pooled into 5×5 groups, i.e., 25 transmit-receive modules 82 per group, then the effective memory capacity or depth of each group is 1600 memory cells (64×25=1600). Therefore, sequentially convolved and summed data frames (images) can first be fed into one of the transmit-receive modules 82 in the group until its memory 90 is filled. Then, more sequentially convolved and summed data frames (images) can be fed into a second transmit-receive module 82 in the group until its memory 90 is also filled, and then the corresponding memory 90s of the remaining transmit-receive modules 82 in the group are sequentially filled. If the memory 90s of all transmit-receive modules 82 in the group are filled, the depth of the block of convolutions in memory will be 1600. The convolutions in the memory 90s of the transmit-receive modules 82 in the array 80 are aggregated together to form a convolution block, for example, Figure 5 The 3D convolution block 65 is schematically shown in the diagram. When the desired number of such convolutions have been accumulated in the array for the last 3D convolution block, these convolutions can be read from the storage 90 on a pixel-by-pixel basis for the sensor-display device 14 to send back via the electronic data interface 22 at the end of processing to output the neural network results.

[0069] However, it is important to reiterate that during the deep learning process of repeatedly convolving and summing data frames using the example photonic neural network system 10, the pixel values ​​for the newly formed convolutional blocks remain in the memory cells of the individual storage bank 90 within the individual transmit-receive module 82. Therefore, when the sensor-display device 14 switches from sensor mode (where convolutional blocks accumulate in the memory 90 of the transmit-receive module 82) to display mode (where convolutional blocks are sent back via the optical components of system 10 for deeper convolutional processing), the pixel values ​​for each of the convolved and summed data frames (images) including the convolutional blocks can be directly read (shifted) from the memory cells 102, 104, ..., n of the memory 90 to the modulator driver 88 without any further processing or transfer of the data to or from external computer processing, memory, or other components or functions. Conversely, when switching from sensor mode to display mode, the pixel values ​​of individual data frames (images) including convolution blocks are sequentially read directly (shifted) from memory 90 to modulation driver 88, which drives light emitting element 84 to modulate the laser incident on the transmit-receive module 82 in such a way that the pixel values ​​of the data frames (images) to be further convolved in that convolution cycle are written (applied) into the light field. Therefore, since all transmit-receive modules 82 in array 80 are simultaneously switched to display mode, and the pixel values ​​of each of these modules are written (applied) into the laser field, the collection of these pixel values ​​in the light field transmitted by sensor-display device 14 replicates the previously convolved data frames (images) summed and stored in the memory 90 of the transmit-receive modules 82 in array 80. The light field, which includes the previously convolved and summed data frame (image), is then projected through the Fourier transform lens 16 onto the radial modulator 20 for further convolution in the Fourier transform plane using a filter, and the light field is then detected at another (e.g., a second) sensor-display device 26, as explained above.

[0070] As explained above, these convolution and summation processes are repeated repeatedly through numerous filters over numerous cycles. Furthermore, the first sensor-display device 14 and the second sensor-display device are aligned along their respective optical axes 61, 62 (see...). Figure 2 , Figure 6 and Figure 7The transmit-receive module 82 in the first sensor-display device 14 is optically aligned with its corresponding counterpart in the second sensor-display device 26, such that there is complete optical alignment between the respective arrays 80 of the first sensor-display device 14 and the second sensor-display device 26 (including between the corresponding transmit-receive modules). Therefore, the example photonic neural network 10 performs full-frame, full-resolution, fully parallel convolutions at the speed of light. Other effects, such as gain, thresholding (ReLU), max or average pooling, and other functions, as will be described in more detail below, are performed simultaneously with dedicated circuitry, and these effects represent no additional time delay. In fact, any convolutional neural network architecture can be accommodated, including, for example, VGG16 or Inception-Resnet-v2. All processing is performed entirely on the sensor-display devices 14, 26, without repositioning the data frames (images) into or out of these devices. During inference operations, the user application only needs to load the image and receive the result a few microseconds later.

[0071] The miniature optical emitter element 84 in the transmit-receive module 82 can be any optical modulator device that emits or modulates light. The description of the example photonic neural network system 10 described above includes an optical emitter element 84 that modulates laser light incident on the back side of the optical emitter element by allowing or preventing light transmission through it. However, as those skilled in the art will understand once they grasp the example photonic neural network described above, the optical emitter element 84 can be replaced by a reflective optical modulator device that modulates and reflects the incident light. This would require laser illumination incident on the same surface of the optical emitter element from which the light is reflected and would require rearrangement of the optics. As another alternative, the optical emitter element 84 can be replaced by an optical emitting tube, which would eliminate the need for a laser field contrasted on the back side of the modulator and passing through it.

[0072] Figure 11 An example light emitter element 84 is shown, which is similar to Figure 4 The super-surface light modulator element 40 shown and described above modulates the phase of the incident light. However, due to Figure 11 The example light emitter element 84 in the example is a light transmission element rather than a light transmission element. Figure 4The light-reflecting element in the grating structure 42 has gaps between the bottom metal layers 46, which are interleaved like the top electrodes 48, so that the incident light L is phase-modulated by the electro-optic polymer 44 to either be blocked by the electro-optic polymer 44 or allowed to pass through the grating structure 42. The substrate 50 is transparent to the light L. The polarization voltage 45 is driven by the modulator driver 88 of the transmit-receive module 82 according to the pixel value to be applied to the light field L, as explained above. More details of this transmissive light modulator can be found in the article "Surface-normal electro-optic-polymer modulator with silicon subwavelength grating" by Kosugi et al. (IEICE ElectronicsExpress, Vol.13, No.17, pp.1-9, September 10, 2016).

[0073] The back side of the transmitter-receiver module 82 has an opaque covering or mask (not shown) covering the back surface to prevent laser illumination on the back side of the transmitter-receiver module 82 from transmitting through the transmitter-receiver module 82 except through the aperture that allows light to reach and pass through the light emitter element 84. Optical components, including a Fourier transform lens 16 and a fiber optic panel for collimating the light in front of the Fourier transform lens 16, can be integrated into the front of the first sensor-display device 14. Similarly, a Fourier transform lens 28 and a fiber optic panel can be integrated into the front of the second sensor-display device 26.

[0074] Now for reference Figure 1 In addition to the first sensor-display device 14 and the second sensor-display device 26 of the photonic convolution assembly 12 and the radial modulator device 20, the example photonic neural network system 10 also includes external control components, including, for example: (i) a circuit block 110 that implements pulse output to drive the radial modulator device 20; (ii) a high-speed analog-to-digital circuit block 112 through which digital data is loaded into and received from the first sensor-display device 14 and the second sensor-display device 26; a high-bandwidth memory (HBM2) 114; and a field-programmable gate array (FPGA) 116, which serves as a basic control and interface device for other system components. The HBM2 114 provides storage for filters, state machine steps, and image data. The circuit block 110, HBM2 114, and FPGA 116 are on a multi-chip module (MCM) 118 and nominally connected to the user interface of the system 10 via a PCI-Fast bus 120.

[0075] exist Figure 12The diagram shows a functional block diagram of an example system interface 122 between a field-programmable gate array (FPGA) 116 and a first sensor-display device 14. Figure 12 The functional block diagram also illustrates the system interface between FPGA 116 and the second sensor-display device 26. For convenience and brevity in the figures, the arbitrary term "RedFive" is sometimes used in the figures and associated descriptions for circuit block 110, and the arbitrary term "Sensay" (an abbreviation for sensor and display) is sometimes used for sensor-display devices 14, 26. Also for convenience and brevity, the transmit-receive module 82 is sometimes referred to as a "trixel" ("trixel" is an abbreviation for "transmit-receive pixel").

[0076] Some of the RedFives are used to generate analog data to load Sensay memory 90. Since the memory module (HBM2) 114 is a state machine source managed by FPGA 116, these RedFives are coupled through HBM2 114. Since FPGA 116 is used to control the feedback loop, analog and digital input / output (I / O) are coupled through FPGA 116. Some unused bits are packaged back to FPGA 116 as status flags for synchronization. Sensay digital I / O uses some of the same memory lines as in the RedFives, but they cannot be accessed simultaneously, so this dual use of memory lines is not conflicting. Furthermore, some of the outgoing analog lines from the RedFives are shared as incoming analog lines to the ADCs. The number of ADCs used to read data and pass it to FPGA 116 depends on the implementation.

[0077] Figure 13 The diagram shows the functional block diagram of Sensay's external interface. Figure 13 In this context, “Sx” is used for the preceding “SensayA”14 or “SensayB”26 (when discussing the system, it is used to distinguish the signal associated with either Sensay 14 or Sensay 26). Figure 8 The numeric input lines in Sensay can be grouped into three general categories. Row and column controls load a set of latches in Sensay (see...). Figure 14 The global control line has various functions, each of which will be described in the context of its use. Global lines can be routed along rows or columns. Global control lines are routed to all transmit-receive modules (trixels) 82 and are not unique for any particular column or row.

[0078] SxPeakreset resets the analog peak hold circuit used for external gain control. This signal is asynchronous, but to avoid data contention, it should only be active when SxFreeze is active (1).

[0079] SxSnsreset resets the sensor to the level of the analog SxLevel line. Sensors are designed to accumulate charge, so this mechanism is necessary to reduce the charge level to a preset level. Note that this reset can also be used as a global bias to preset the sensor charge level (and thus preset the modulator level on the next pass).

[0080] SxPoolmode determines the average (1) operation or the maximum (0) operation in pooling.

[0081] SxFreeze enables and disables global memory access. If SxFreeze is enabled (1), all trixel memory drives are set to a safe state, and no memory access or shifting is permitted. SxFreeze is used when configuring other control lines to prevent data contamination before the lines are scheduled. The function of SxFreeze is not always mentioned in the explanation below, but its actions are always overridden.

[0082] SxRWDir determines whether to read or write to the enabled memory. If set to "1", data is written to memory, and if set to "0", data is read from memory. SxRWDir also gates the operation of the sensor (photodetector element) 86 and the modulator (light emitter element) 84. SxRWDir indicates modulator mode (0) or sensor mode (1).

[0083] SxFlagRD, SxFlagWR, and SxFlagRST control the digital flag memory used for semantic tagging. SxFlagRST is a global address reset for all flag memories. SxFlagRD and SxFlagWR control memory access.

[0084] SxShift0, SxShift1, and SxShift2 are driven externally in a three-phase sequence to move the charge of the shift register memory 90 clockwise or counterclockwise only at the addressed trixel (transmit-receive module) 82. If the trixel is not addressed, its memory drive line is forced into a safe state, and the memory charge is unaffected.

[0085] SxExtemal determines whether the SxAnalog and SxData lines are active (1) or whether data movement and access are internal (0).

[0086] Consider four combinations of these signals:

[0087] Image loading: SxFreeze = 0, SxRWDir = 1, SxExternal = 1. This means that any memory cell at the addressed trixel 82 will fetch data from the external SxAnalog line and place voltage in the trixel bank (multiple) via the internal SxData line. Since there are 120 SxAnalog lines, this operation can be up to 120 wide. For implementations where a 120-wide setting is unsuitable for the DAC, these lines can be externally grouped, and narrower accesses can be accommodated simply by enabling MEMC lines sequentially. Regardless of the external line width of the implementation, typically only one MEMC line is enabled at a time to avoid contention (but the same DAC value can be sent to the entire line at once if needed).

[0088] Results are saved as follows: SxFreeze = 0, SxRWDir = 0, SxExternal = 1. This means that any memory cell at the addressed trixel 82 will send data to the external SxAnalog line for conversion with an external ADC. Again, this can be up to 128 wide, but a narrower implementation can be accommodated without requiring design changes to Sensay. Regardless of the implemented external line width, only one MEMC is enabled at a time to avoid contention (this is not optional for reads to avoid data contention).

[0089] Sensor mode: SxFreeze = 0, SxRWDir = 1, SxExternal = 0. This means that any memory cell at the addressed trixel 82 will (via the pooling chain, which will be described below) acquire data from sensor 86 and, in conjunction with SxShift0, SxShift1, and SxShift2, save the voltage as a new memory charge while the existing voltage shift is used as a set of shift registers for the memory value.

[0090] Modulator mode: SxFreeze = 0, SxRWDir = 0, SxExternal = 0. This means that any memory cell at the addressed trixel 82 will send data (via pooling chain) to the modulator (optical transmitter element) 84, and in conjunction with SxShift0, SxShift1, and SxShift2, will use the existing voltage shift as a set of shift registers for memory charge. Memory reads are non-destructive.

[0091] exist Figure 14The diagram schematically illustrates example row and column control line registers for a trixel (transmit-receive module) 82. In this example, the row and column control line registers comprise 235 individually addressable static 64-bit latches, arranged with five row lines and five column lines for each trixel. These outputs are always active and are set to zero upon power-up. Each trixel 82 uses these row and column control lines to configure itself relative to its neighbors for its functionality. Each latch is individually addressed by validating data at SxControl, setting an 8-bit address at SxAddr, and applying a pulse to SxLatch.

[0092] When both MEMR and MEMC of the Trixel are enabled, the Trixel's memory 90 is considered "addressable". Similarly, when both OPTC and OPTR of the Trixel are enabled, its optical sensor is "addressable". When both ENBR and ENBC of the Trixel are disabled, the miscellaneous Trixel 82 functions are disabled. To completely disable the Trixel 82, similarly disable its MEMR, MEMC, OPTR, OPTC, FLAGR, and FLAGC.

[0093] Pooling boundary lines 86 (POOLC and POOLR) affect the entire column and row of trixel 82 and define the boundaries of the supertrix, as will be described in more detail below. The rightmost and bottommost lines are always enabled, resulting in only 1079 POOLR lines and 1919 POOLR lines. Lines not used in the 64-bit latch are not connected.

[0094] The *_SL and *_SR lines shift their respective registers to the left or right on the rising edge.

[0095] SxReLU1 and SxReLU2 ( Figure 13 Driven by an external DAC. They are global for all Trixel 82s and apply sensor 86 read values ​​to eliminate weak information. SxLevel( Figure 13 It is also driven by an external DAC. The SxLevel is used as a preset level by all trixel sensors 86, and is also summed to the modulator drive level 88, where it is used as a phase offset. As explained above, since the sensor (sensor-display device) 14 or 26 is in either sensor mode or modulator (transmit) mode at any given time, there is no contention. SxPeak ( Figure 13The analog output signal comes from all 82 trixel (transmit-receive modules). As described in more detail below, each trixel memory cell passes its highest value to a common trace. The value of this trace represents the highest global value seen by the entire trixel array since the last time SxPeakreset was enabled. This is used by external circuitry for system gain and normalization.

[0096] Figure 15 The example analog interface is schematically illustrated. The SxAnalog line consists of 120 traces, each connecting nine adjacent SxData rows. In other words, internal SxData0000 to SxData0008 traces are all connected to the output pin SxAnalog000. SxData0009 to SxData0017 traces are all connected to the output pin SxAnalog001, and so on. Each SxAnalog pin is hardwired to one of the nine internal SxData traces. Only one trixel memory bank 90 is allowed to drive or sense its local trace at a time (forced by an external controller). If TMS is enabled, all SxAnalog and SxData lines are connected together.

[0097] It is worth noting that, as discussed above, because the control lines are individually controllable, input or output schemes of any size from 1 to 120 can be implemented simply by connecting these lines together externally to the Sensay (sensor-display device) and enabling only the appropriate Trixel 82 to match the architecture. Wider interfaces allow for faster loading and unloading operations, but require more external circuitry. This allows for a high degree of customization without altering the design.

[0098] In the example photonic neural network system 10, the Sensay (sensor-display device) 14, 26 architecture is built around a pooling chain. For example... Figure 9 , Figure 10 and Figure 16 As shown, each of the transmit-receive modules (trixels) 82 in array 80 has two along its edge (e.g., in those...). Figure 9 , Figure 10 and Figure 16 Pooling boundary line 96 (along the right and bottom edges when oriented on paper). All sensor, modulator, memory read, or memory write accesses use pooling chains to transfer analog data within and between trixels (sensor-display devices) 82. The function of pooling boundary line 96 is to connect or disconnect nearby transmit-receive modules (trixels) 82 from the pooling chains, thereby creating supertrixels or "islands". Figure 17In the enlarged schematic diagram of the connections, at assumed positions nnnn, mmmm in the array 80 of trixel 82, the pooling chain circuit connections to the boundary lines 96 of the corresponding adjacent trixel 82 are shown. If POOLC = 0, all east-west trixel pooling chain connections for the entire column are open. If POOLR = 0, all north-south trixel pooling chain connections for the entire row are open. All other trixel pooling chain connections remain closed. The effect of this pooling structure is to create islands of connected pooling chain lines. All trixels on a supertrixel island share this chain, which is essentially a single low-impedance "trace". If POOLR is enabled, the transistor connecting the pooling chain of that trixel will conduct, thus connecting the pooling chain to the trixel 82 in the next row to its south. If POOLC is enabled, the pooling chain is connected to the trixel 82 to its east.

[0099] As mentioned above, the memory bank 90 in each of the transmit-receive modules (trixel) 82 is essentially a shift register, and shift register design and technology are well understood and readily available to those skilled in the art. Figure 18 The analog memory shift driver scheme is illustrated. In the addressable (MEMC and MEMR enabled) and unfrozen (SxFreeze disabled) state, any combination of SxShift0, SxShift1, and SxShift2 will only propagate to the outputs (MemShift0, MemShift1, MemShift2) that actually drive the analog memory cell shift board. If MEMC or MEMR is disabled for trixel, or if SxFreeze is enabled, the analog memory driver will automatically be placed in a safe state (MemShift0, MemShift01, MemShift02 = 010).

[0100] Figure 19 It is used for storage 90 ( Figure 10 A schematic diagram of an example analog memory read interface is provided. The memory can be read and analog data routed to the external SxAnalog interface via the internal SxData line, or sent to pooling chain 126 via either the maximum (diode) circuit path or the average (resistor) circuit path (if the analog data is greater than SxReLU, otherwise it is set to zero). The unmodified value read from the analog memory is also used to charge the diode-isolated capacitor (sample-and-hold circuit), ultimately driving the SxPeak value for the entire sensor-display device (sensay) 14, 26 (for system gain control used externally). Examples of these modes are shown in... Figure 20-24 It is shown schematically in the diagram. Figure 20 The Trixel analog memory reads averaged for the pooling chain are shown. Figure 21 The maximum Trixel simulated memory read for the pooling chain is shown. Figure 22 This illustrates an analog memory read from an external data line. Figure 23 The peak storage of the Trixel analog memory is shown. Figure 24 The simulation shows a peak memory reset.

[0101] Rectified linear units (ReLUs) are typically used in data processing to suppress weak responses. The first sensor-display device 14 and the second sensor-display device 26 (sensay 14, sensay 26) each have flexible dual-slope ReLU implementations, which can lead to... Figure 24 The various responses shown range from no effect (Example A) to conventional cutoff (Example B) to variable slope cutoff (Example C). The transfer functions are controlled by two external analog voltages driven by the DAC. Since Sensay 14 and Sensay 26 are unipolar designs, the "zero" position is nominally located at the center of the 90-volt range of the memory bank.

[0102] Writing to analog memory 90 is simpler than reading. When addressing analog memory 90 of the transmit-receive module (trixel) 82 (with both MEMC and MEMR enabled and SxRWDir = 1), any value on the local pooling chain will be placed on the write pad, such as... Figure 26 As shown in the diagram. In order to actually store the value in the analog memory cell, the shift line is looped. Figure 27 The diagram shows the loading of analog memory 90 from an external data line.

[0103] The flag memory is a 640-bit Last-In-First-Out (LIFO) device (i.e., a "stack") at each transmit-receive module (trixel) 82 used for semantic tagging implementation. If SxFlagRST = 1, the internal address pointers are unconditionally set to zero for all transmit-receive modules (trixels) 82. Setting the values ​​to zero is not necessary. Aside from resetting, the memory is only active for the assumed trixel positions nnnn, mmmm, FLAGRmmmm = 1 and FLAGCnnnn = 1. If FLAGRmmmm = 0 or FLAGCnnnn = 0, no signal affects the memory. For FLAGR and FLAGC, see [link to relevant documentation]. Figure 14 .

[0104] exist Figure 28 and Figure 29The diagrams show the write and read operations for the flag memory. When SxFlagWR = 1, the comparator output is valid at the "D" memory input. Simultaneously with SxFlagRD = 0, on the falling edge of SxFlagWR transitioning from "1" to "0", the current flag bit, determined by comparing the current read value of the trixel with the value on the pooling chain, is pushed onto the stack. In other words, if the analog memory read voltage matches the pooling chain voltage, the trixel 82 is "master" and stores "1"; otherwise, it stores "0". For FlagVAL, see [link to relevant documentation]. Figure 19 .

[0105] Because of the very small hysteresis, if more than one trixel 82 has very similar voltage levels, they may treat themselves as the "master". In this case, the final read will be the average voltage of the trixels 82 enabled in that pooling group during the expansion pass. Since the "competition" voltages are almost identical, this will have little practical effect.

[0106] On the rising edge of SxFlagRD = 1, when SxFlagWR = 0, the last bit written (i.e., at the top of the stack) is read and applied as enable FlagEN = 1 to the trixel memory read circuit (see [link]). Figure 19 Output is enabled as long as SxFlagRD = 1.

[0107] If SxFlagWR = 0 and SxFlagRD = 0, then FlagEN = 1. This is acceptable. SxFlagWR = 1 and SxFlagRD = 1 is illegal, and the external controller should not apply it. To avoid contention between memory output and comparator output, FlagEN is tri-state in this case.

[0108] exist Figure 30 , Figure 31 and Figure 32Examples of optical control line settings are shown below, used to read the sensor 86 of the transmit-receive module (trixel) 82 into the pooling chain, reset the sensor 86, and write data from the pooling chain to the modulator (optical emitter element) 84. The optical control lines function to connect the optical element (modulator 84 or sensor 86) of the trixel 82 at the intersection of the enabled OPTR and OPTC lines to the pooling chain. If SxRWDir = 0 and SxExternal = 0, data is read from the pooling chain to drive the trixel modulator 84. If SxRWDir = 1 and SxExternal = 0, data is buffered from the sensor 86 of the trixel and placed on the pooling chain. If SxExternal = 1, both the modulator 84 and the sensor 86 are disconnected. Multiple sensors 86 can be enabled simultaneously; the average value of the sensor values ​​will appear on the pooling chain to reduce noise. It should also be noted that, as explained above, when sensor 86 is summing the optical signal (data frame), there is no other activity on sensor 14 and sensor 26 (no clock, etc.), which will result in very low noise measurements.

[0109] If in modulator mode (SxRWDir=0) and internally driven (SxExternal=0), all addressed trixel memory bank 90 outputs will be automatically pooled, and all light emitter elements (modulators) 84 in the same supertrixel (connected to the same pooling chain) will "emit" at the same brightness. This constitutes resampling through duplication.

[0110] The local ENB (ENBRmmmm=1 and ENBCnnnn=1) allows the individual optical transmitter element (modulator) to be disabled 84.

[0111] The drive level DL for the optical transmitter element (modulator) 84 is the sum of the pooling chains PC multiplied by the calibration sensor value CS+1 plus SxLevel; as the formula DL = (PC * (CS+1)) + SxLevel. If SxInvert = 1, the driver is inverted; that is, 100% of the level will be 0% modulation, 90% of the level will be 10% modulation, and so on.

[0112] Figures 33A-33B The diagram shows an overview of the trixel circuit.

[0113] The above explanation is based on photonic neural network processing in inference mode, for example, where a trained neural network is used to recognize images, speech, sounds, etc. Training a neural network using a photonic neural network (e.g., the photonic neural network system 10 described above) differs from training a digital convolutional neural network system in several ways. As mentioned above, during the training of a typical digital convolutional neural network system, a process called backpropagation is used to provide adjustments to increase the likelihood that the network will predict the same type of image next time. In a typical digital convolutional neural network, this data processing and backpropagation are performed repeatedly until the prediction is fairly accurate and there is no further improvement. The neural network can then be used in inference mode to classify new input data and predict the results inferred from its training. In digital convolutional neural networks, training is relatively simple because the backpropagation terms and filters are in the spatial domain. Taking the "correct answer" and returning it through the structure to compute the correction terms is slow, but still does not require changing the domain. Training in a photonic neural network is not so straightforward because the terms to be trained are in the frequency domain, while the convolution result is in the spatial domain. One can use spatial domain data to compute correction terms using the Fast Fourier Transform (FFT) algorithm and apply them to the Fourier filter used in the radial modulator device 20, but this computation is very computationally intensive.

[0114] Conversely, the example photonic neural network system 10 described above is adapted to convert the correction terms used for training into Fourier transform terms, which can then be added to the filters of the convolutions applied by the radial modulator device 20 during the iterative training process. An example adaptation that optically implements this conversion rather than digitally computes includes adding a dedicated Fourier light sensor device 130 to the photonic convolution assembly 12, such as... Figure 34As shown in the diagram, the Fourier transform optical sensor device 130 is placed on the optical axis 62, aligned with the axis of the second sensor-display device 26, on the opposite side of the second sensor-display device 26 from the polarizer 18. The Fourier transform optical sensor device 130 is also placed in the Fourier transform plane at a distance from the Fourier transform lens 26 equal to the focal length F2 of the Fourier transform lens 26. Therefore, the Fourier transform optical sensor device 130 is positioned at the Fourier transform plane of the Fourier transform lens 26. In this Fourier transform plane, the Fourier transform optical sensor device 130 can detect the Fourier transform of a data frame or image in the light emitted from the second sensor-display device 26. Therefore, the correction terms required for training the photonic neural network system 10 can be fed into the second sensor-display device 10 in the spatial domain of the correction data frame, which the second sensor-display device 10 then displays (projects) the correction data frame onto the Fourier transform optical sensor device 130 in the light field 132. Therefore, when the correction data frame in the light field 132 arrives at the Fourier optical sensor device 130, it is subjected to a Fourier transform by the Fourier transform lens 26. That is, the correction data frame in the spatial domain is Fourier transformed at the speed of light to the Fourier domain at the Fourier optical sensor device 130. Then, those correction data frames in the Fourier transform domain are detected by the Fourier optical sensor device 130 and used to adjust the filter for the radial modulator device 20.

[0115] Typically, in inference mode, when 3D convolutional blocks are shifted out of memory and sent back via photonic convolution assembly 12 for further levels of convolution and summation cycles, and when memory 90 is refilled with subsequent 3D convolutional blocks, data frames present at specific iterations of convolution in memory 90 of the transmit-receive module 82 are lost; all of this occurs very rapidly, as explained above. However, for training mode, those intermediate data frames are extracted from memory 90 of the first sensor-display device 14 and the second sensor-display device 26 and transferred to external memory for backpropagation digital computation and for writing correction terms in the spatial domain. These correction terms are then fed as correction data frames in the spatial domain to the second sensor-display device 26 for projection and Fourier transform to Fourier optical sensor device 130, as... Figure 34 As shown and explained above, the Fourier-transformed corrected data frames can therefore be detected in the Fourier domain by the Fourier optical sensor device 130 and used as filters in the radial modulator device 20 for further convolution cycles. This extraction of training patterns from intermediate correlated data, backpropagation digital computation, and writing of the corrected terms takes time, thus slowing down the iterative convolution-summing cycle compared to the inference pattern of the operation, but is still much faster than training a digital convolutional neural network.

[0116] In order to accommodate the Fourier light sensor device 130 (e.g., as shown in the photonic convolution assembly 12) Figure 34 As shown in the diagram, a half-wave variable polarizer 134 is placed between the second sensor-display device 26 and the polarizer 18 to rotate the polarization plane by 90 degrees when a correction data frame is projected from the second sensor-display device 26 onto the Fourier optical sensor 130. For example, if in normal inference mode of operation, the second sensor-display device 26 displays P-polarized light reflected from the polarizer 18 to the radial modulator device 20, and is then used to display or project a correction data frame onto the Fourier optical sensor 130 for training, the half-wave variable polarizer 134 is activated to rotate the polarization plane of the projected light field by 90 degrees to S-polarization, such that the resulting light field 132 passes through the polarizer 18 and reaches the Fourier optical sensor 130.

[0117] The correction data frames have values ​​that need to be provided by the filter to a specific wedge segment 32 for training the neural network. Therefore, those correction data frames provided to the second sensor-display device 26 for projection onto the Fourier light sensor device 130 are used in conjunction with the wedge segment 32 of the radial modulator device 20 (see...). Figure 3 A corresponding format is provided, which needs to be modulated in a corrected manner for training the neural network, such that the corrected data will eventually be fed into a filter that drives the appropriate wedge segment 32 in a corrected manner. Therefore, the Fourier optical sensor device 130 detects light 132 from the second sensor-detector device 26 in the same mode as the wedge segment 32 in the radial modulator device 20, such that the corrected data in the light 132 is detected, processed, and fed to the appropriate wedge segment 32 of the radial modulator device 20.

[0118] To facilitate the detection of light projected from the second sensor-display device 26 according to the same pattern as the wedge segment 32, as explained above, the example Fourier light sensor device 130 has a light sensor plate 135 comprising a plurality of light sensor elements 136 arranged in a light sensor array 138 corresponding to the patterns of the wedge segment 32 and the wedge sector 34 of the radial modulator device 20, such as, for example... Figure 35 As shown in the diagram, the radial array lens plate 140 is positioned in front of the light sensor array 138, as... Figure 35As shown, the device has multiple individual lens elements 142 arranged in a radial pattern that matches the wedge segments 32 and sectors 34 of the radial modulator device 20. These lens elements 142 capture incident light 132 from the second sensor-display device 26 in a radial pattern corresponding to the radial pattern of the wedge segments 32 and wedge sectors 34 of the radial modulator device 20, thus capturing correction data frames in the incident light 132 when they are formulated and programmed into the second sensor-display device 26. Segments of light captured by the respective lens elements 142 are focused by the lens elements 142 as individual sub-beams 138 onto the respective light sensor elements 136 and converted into electrical signals with intensities corresponding to the intensity of the light incident on the sensor elements 136, thus converting the correction data frames in the incident light 132 into electrical data signals corresponding to the correction data frames. The analog electrical data signals can be converted into digital signals for processing into a correction filter by FPGA 110, which can then be fed by circuit block 110 for connection to radial modulator device 20 via interface 24. Again, the data frames in the incident light 132 are Fourier transformed in the Fourier transform domain by Fourier transform lens 26 and sensed by sensor element 136. Therefore, the correction data in the signal sent from Fourier light sensor device 130 to FPGA 116 or other electrical processing components is in the Fourier domain as needed to drive the wedge segment 32 of radial modulator device 20. Due to the arrangement of the optical components, the correction data frames fed to the second sensor-display device 26 may need to be inverted so that the segment of light captured by sensor element 136 and the corresponding signal generated match the appropriate wedge segment 32 of radial modulator device 20. However, as explained above, the correction terms are calculated in the spatial domain and therefore do not impose any algorithmic constraints on training. Once normal training backpropagation calculations are performed, the above-described and... Figure 34 and Figure 35 The optical component shown converts the spatial domain correction term into a radial Fourier domain equivalent.

[0119] exist Figure 36 In another embodiment shown, a camera lens 150 is mounted on the photonic convolution assembly 12 in such a manner that the camera lens 150 projects light from the real-world scene 152 as a data frame (image) in the spatial domain into the photonic convolution assembly 12 for processing as described above in the example photonic neural network 10. For example, as Figure 35As shown, camera lens 150 is mounted on optical axis 62 to be axially aligned with the second sensor-display device 26 and is on the opposite side of polarizer 18. Polarizer 154 is placed between camera lens 150 and polarizer 18 to polarize the light field 156 transmitted by camera lens 150 to a polarization plane reflected from polarizer 18, so that light field 156 is reflected by polarizer 18 to the first sensor-display device 14, as shown. Figure 36 As shown in the diagram. First sensor-display device 14 (see...) Figure 9 and Figure 10 The light sensor element 86 in the transmit-receive module 82 detects and captures data frames (images) in the light field 156, and processes the data frames (images) into the storage 90 in the first sensor-display device 14, as explained above. Then, the shutter device 158 on the camera lens 150 closes the camera lens 150 to terminate the transmission of light through the camera lens 150, and then the first sensor-display device 14 can begin processing the data frames (images) through the photonic convolution assembly 12 in inference or training operations, as explained above.

[0120] The bandpass filter 160 may also be equipped with a camera lens 150 to allow only certain spectral frequencies of light to pass through the photonic convolution assembly 12 when needed. When needed, the bandpass filter 160 can be a variable bandpass filter, allowing various spectral bands of light from the real-world scene 152 to be transmitted sequentially from the camera lens 150 into the photonic convolution assembly, while simultaneously capturing data frames (images) in each band sequentially, thereby providing a hyperspectral image set for sequential convolution through the photonic convolution assembly. Such variable bandpass filters are known. For example, a variable half-wave delay can be combined with a fixed polarizer as a variable bandpass filter. Such a combination of a variable half-wave delay and a fixed polarizer can also be used as a shutter.

[0121] The foregoing description is considered to be an illustration of the principles of the invention. Furthermore, since many modifications and alterations will readily occur to those skilled in the art, it is not intended to limit the invention to the exact constructions and processes shown and described above. Therefore, all suitable modifications and equivalents falling within the scope of the invention can be employed. When used in this specification, the terms “comprise,” “comprises,” “comprising,” “include,” “including,” and “includes” are intended to specify the presence of the stated features, integrals, components, or steps, but they do not exclude the presence or inclusion of one or more other features, integrals, components, steps, or groups thereof.

Claims

1. A data processing array, comprising: An array of modules arranged in multiple rows and columns, each module in the array including a read / write memory; A pooling chain includes a first pooling boundary line extending along a first side of each module in each column of the array and a second pooling boundary line extending along a second side of each module in the array, wherein the first boundary line extending along the first side of each module intersects and connects with the second boundary line extending along the second side of the module to form a boundary pooling circuit for that module; and wherein the boundary pooling circuit of each module is connected to an adjacent pooling circuit of an adjacent module in the same column via a first openable and closeable switch, and is connected to an adjacent pooling circuit of an adjacent module in the same row via a second openable and closeable switch; The memory pooling interface in each module is used to connect the memory in the module to the boundary pooling circuit of the module, so as to realize the reading of data from the memory of the module into the pooling circuit and the writing of data from the boundary pooling circuit into the memory.

2. The data processing array according to claim 1, comprising: A pooled row switch control line connected to each first switch in the column of the module is used to simultaneously turn on and off all the first switches in the column; as well as A pooled column switch control line connected to each of the second switches in the row of the module is used to simultaneously turn on and off all the second switches in the row.

3. The data processing array according to claim 2, wherein, Each module includes an optical modulator, a modulation driver connected to the optical modulator, and a modulation pooling interface that connects the modulation driver to the boundary pooling circuit of the module to write data from the boundary pooling circuit, thereby driving the optical modulator.

4. The data processing array according to claim 3, wherein, Each module includes a light sensor and a sensor interface, the sensor interface being used to feed data from the light sensor onto the boundary pooling circuit.

5. The data processing array according to claim 1, wherein, Each module includes a data interface for connecting analog data lines to the module's memory, for reading data from the memory into the analog-to-digital converter, and for writing data from the digital-to-analog converter into the memory.

Images