Optical image processing

Abstract

An apparatus (2) for use in optical image processing is disclosed. The apparatus (2) comprises a medium (7) capable of supporting overlapping optical modes (10) such that in response to a spatially structured illumination (4), a set of overlapping optical modes (10) in the medium (7) are non-uniformly excited and output a spatially structured response (6).

Description

Technical Field

[0001] This invention relates to optical image processing. Background Technology

[0002] One promising approach to machine learning is the use of physical neuromorphic computing schemes. In physical neuromorphic computing schemes, hardware architectures are used to simulate the function of neurons and / or neural networks.

[0003] Various approaches have been explored to implement physical neuromorphic computing schemes; however, these schemes may have one or more drawbacks. Memristor arrays may suffer from poor dimensionality, lack of image input capability, and rapid physical degradation. Optical delay line systems may require kilometers of fiber optic cable reels, thus limiting miniaturization and applicability to remote, non-data center use cases such as autonomous vehicles. Vertical cavity surface-emitting lasers (“VCSELs”) may lack large input dimensionality capability and do not allow for mode separation.

[0004] Furthermore, the data inputs in many existing neuromorphic computing schemes are typically one-dimensional or low-dimensional. This can be a drawback because many modern machine learning problems (such as those related to computer vision) involve high-dimensional data.

[0005] Physical neuromorphic computing schemes have been implemented in a range of all-optoelectronic and optoelectronic systems.

[0006] T. Zhou et al., “Large-scale neuromorphic optoelectronic computing with areconfigurable diffractive processing unit”, Nature Photonics, Vol. 15, p. 367 (2021), describe an optoelectronic fusion computing architecture based on optical diffraction.

[0007] D. Pierangeli, G. Marcucci and C. Conti, “Photonic extreme learning machine by free-space optical propagation”, Photonics Research, Vol. 9, p. 1446 (2021), describes a neuromorphic photonic scheme that can be realized by using an optical encoder and coherent wave propagation in free space.

[0008] US 2021 / 0285819 A1 describes a system for machine learning using optical data. The system includes a diffuse physical medium for scattering light from a light source coupled to an adjustable spatial light modulator.

[0009] Also referenced is M. Matuszewski et al., “Energy-Efficient Neural Network Inference with Microcavity Exciton Polaritons”, Physical Review Applied, Vol. 16, p. 024045 (2021), which proposed an all-optical neural network with an energy efficiency of 4 × 10⁻⁶ watts per second. 16 Subsynaptic operations, and a performance density of 10 per second per square millimeter. 16 Secondary synaptic operation.

[0010] Software-based methods may also have one or more drawbacks. Part of software machine learning involves a so-called "non-linear activation" step, where data passed through a software neural network undergoes some non-linear transformation to increase processing power; typical transformations are modified linear activation functions (hereinafter referred to as "ReLU") or sigmoid transformations. Manually designing software networks to enable these transformations to be performed in parallel can be quite costly. Summary of the Invention

[0011] According to a first aspect of the invention, an apparatus is provided for use in optical image processing. The apparatus includes a medium capable of supporting overlapping optical modes, such that, in response to spatially structured illumination, a set of overlapping optical modes in the medium is non-uniformly excited and outputs a spatially structured response.

[0012] This arrangement allows the device to process data, including that contained in spatially structured lighting, in a manner that preserves the data's characteristics. This is because no preprocessing is required before the data is provided to the device for processing, such as by pixel rasterization (in other words, by taking pixels as a one-dimensional (1D) input as a time series) or by image transformation.

[0013] Different optical modes in a set of overlapping optical modes excited in response to spatially structured illumination may have one or more of the following: different frequencies; different spatial mode shapes; and different spatial positions within the medium.

[0014] The medium can be disposed on the surface. The surface can be the surface of a substrate. The surface can be flat. The surface can be non-flat, for example, it can be patterned, curved, or have a non-uniform surface.

[0015] The device may include a substrate. The medium may be disposed in a matrix of another material.

[0016] The substrate can be rigid. The substrate can be flexible. The substrate can be made of plastic material or primarily made of plastic material. The substrate can be planar. The substrate can be non-planar, for example, curved.

[0017] The substrate may comprise, consist of, or primarily consist of a single layer, or may comprise a single layer (e.g., a surface layer) comprising, consisting of, or primarily consist of a dielectric or semiconductor, such as indium tin oxide (ITO), silicon dioxide (SiO2), hafnium dioxide (HfO2), titanium dioxide (TiO2), zinc oxide (ZnO), magnesium fluoride (MgF2), or diamond. The substrate may be an oxide material. The substrate may comprise, consist of, or primarily consist of a semiconductor material, such as silicon (Si). The substrate may comprise transparent materials and / or transparent material layers (e.g., surface layers), such as dielectric or semiconductor layers, such as ITO, SiO2, HfO2, TiO2, ZnO, MgF2, or diamond.

[0018] The medium may include an optical gain medium. The optical gain medium may be embedded in the medium. Alternatively, the medium may be an optical gain medium.

[0019] The optical gain medium can be a solid-state gain medium. A solid-state gain medium may comprise a semiconductor material embedded in a polymer material or dye molecules embedded in a polymer material. Alternatively, the optical gain medium can be a liquid-phase dye medium.

[0020] The medium may have an optical gain of less than 1, equal to 1, or greater than 1. In other words, the device can be used to amplify incident light and / or to attenuate incident light.

[0021] The medium can be spatially non-uniform. The medium can be spatially non-uniform along at least one in-plane direction. The medium can be patterned in-plane. The medium can be spatially non-uniform along out-of-plane directions. For example, the medium can be a multilayer heterogeneous structure. The medium can include voids. The medium can include a suspended structure. The medium can be 3D patterned.

[0022] The medium can be spatially homogeneous. The medium can be spatially homogeneous along at least one in-plane direction. The medium can have a complex refractive index, which is spatially structured in response to a mask in the form of a spatially structured beam being projected onto the medium. The mask can be provided by a second light source, different from the light source configured to provide spatially structured illumination. The mask can be superimposed on the spatially structured illumination. In other words, a single light source can provide both spatially structured illumination and a mask.

[0023] The medium can be configured as a panel. The medium across the panel can be spatially non-uniform. The medium on the panel can be spatially uniform. The medium can be configured as a screen. The medium on the screen can be spatially non-uniform. The medium across the screen can be spatially uniform.

[0024] The device can be configured to operate at a designed wavelength. The designed wavelength can be between 10 nm and 400 nm (or "ultraviolet"). The designed wavelength can be between 400 nm and 780 nm (or "visible"). The designed wavelength can be between 780 nm and 1.4 μm (or "IR-A"). The designed wavelength can be between 1.4 μm and 3 μm (or "IR-B"). The designed wavelength can be between 3 μm and 1 mm (or "IR-C").

[0025] The design wavelength refers to the wavelength of spatially structured lighting. The design wavelength can correspond to an energy that is less than or equal to the bandgap energy of the material providing the medium.

[0026] The medium may include a first photonic network, which includes radiatively coupled network links at nodes. In other words, the medium may contain a first photonic material and have a first topology in which network links are radiatively coupled at nodes.

[0027] Photonic networks may include or take the form of photonic waveguide networks.

[0028] At least a first subset of the network links of the first photonic network can be interconnected at the nodes of the first photonic network. At least a second subset of the network links of the first photonic network can be separated at the nodes of the first photonic network by a distance less than the design wavelength.

[0029] The first photonic network may include at least two closed-loop paths. These at least two closed-loop paths can be used to transmit light with a designed wavelength.

[0030] The medium may include a second photonic network comprising radiatively coupled network links at nodes, the second photonic network being spatially separated from the first photonic network. The second photonic network may contain a second photonic material and have a second topology in which the network links are radiatively coupled at nodes.

[0031] At least a first subset of the network links of the second photonic network can be interconnected at the nodes of the second photonic network. At least a second subset of the network links of the second photonic network can be separated at the nodes of the second photonic network by a distance less than the design wavelength.

[0032] The second photonic network may include at least two closed-loop paths. These at least two closed-loop paths can be used to transmit light with a designed wavelength.

[0033] The first photonic material may be different from the second photonic material. The first photonic material may be the same as the second photonic material. The first topological structure may be different from the second topological structure. The first topological structure may be the same as the second topological structure.

[0034] The medium may include three or more spatially separated photonic networks.

[0035] The medium may include a photonic waveguide. The network link may include a photonic waveguide. The photonic waveguide may be embedded in the network link. The network link may be a photonic waveguide. The photonic waveguide may be an active photonic waveguide.

[0036] The term "active" in waveguides can refer to the waveguide's ability to provide gain, thereby amplifying light.

[0037] The network link and / or photonic waveguide may have at least one dimension less than 1000 nm. At least one dimension may include width or diameter. At least one dimension may include thickness.

[0038] Network links and / or photonic waveguides may have at least one dimension greater than or equal to one-tenth of the design wavelength. Network links and / or photonic waveguides may have at least one dimension less than or equal to 10 times the design wavelength. Network links and / or photonic waveguides may have at least one dimension less than or equal to 100 times the design wavelength.

[0039] Network links and / or photonic waveguides may have a length distribution. The length distribution may be centered on an average length value. In the case of design wavelengths between 400 nm and 780 nm (or "visible light"), and / or in the case of design wavelengths between 780 nm and 1.4 μm (or "IR-A"), the average length value may be between 3 μm and 20 μm. The average length value may be less than 100 μm. The average length value may be less than 1 mm.

[0040] In a photonic network, the degree of a node can be between 3 and 6.

[0041] Photonic waveguides can have a diameter distribution. The diameter distribution can be centered on an average diameter value. Photonic waveguides can have an average diameter value between 1 μm and 10 μm, and an average length value between 100 μm and 500 μm.

[0042] The medium may include low-dimensional structures. Low-dimensional structures may include quantum wells. Quantum wells may be encapsulated quantum wells.

[0043] The medium may contain semiconductor materials.

[0044] Semiconductor materials can be inorganic semiconductor materials. Semiconductor materials can be III-V group semiconductor materials. III-V group semiconductor materials can be direct bandgap III-V group semiconductor materials. III-V group semiconductor materials can be indium phosphide (InP), gallium arsenide (GaAs), or indium gallium arsenide (InGaAs).

[0045] Semiconductor materials can be organic semiconductor materials. Organic semiconductor materials can be Rhodamine 6G dye. Organic semiconductor materials can be Rhodamine B dye.

[0046] A set of overlapping optical modes may include strongly coupled optical modes. A set of overlapping optical modes may be strongly coupled.

[0047] A set of overlapping optical modes may include at least 10 optical modes. A set of optical modes may include up to 100 or up to 1000 optical modes.

[0048] A set of overlapping optical modes may include two or more optical modes that are lasing modes. A set of overlapping optical modes may include only optical modes that are not lasing modes. A set of overlapping optical modes may include only one optical mode or consist of only one optical mode, which varies non-linearly with the intensity of spatially structured illumination.

[0049] Spatial structured lighting may include input data, such as images for classification. The input data may be encoded. The image may be a 2D image. The image may be a pixelated image. The image may have a size of at least 20 pixels × 20 pixels, such as 28 pixels × 28 pixels. The image may have a size of at least 1000 pixels × 1000 pixels. The image may be optically projected onto the medium in a native pixel-mapped state. In the native pixel-mapped state, local information about neighboring pixels and image features is fully preserved.

[0050] An n×m pixel image can be projected onto a medium having a length L and a width W, such that each pixel of the projected image spans an area of ​​L / n×W / m on the physical surface of the medium.

[0051] The device may be neuromorphic computing hardware. The device can be used in neuromorphic optical computing. The device may be able to provide a test accuracy score equal to or greater than 95% as measured by a modified National Institute of Standards and Technology (“MNIST”) handwritten digit classification task (e.g., a version available at least as of November 27, 2023). The device may be able to provide a test accuracy score equal to or greater than 97.7% as measured by the MNIST handwritten digit classification task. The device may be able to provide a test accuracy score equal to or greater than 98.15% as measured by the MNIST handwritten digit classification task.

[0052] The device can be configured to modulate the optical response of the medium. Modulating the optical response can mean modifying the interaction between spatially structured lighting and the medium, as well as within the medium, thereby resulting in a change in the spatially structured response provided for a given spatially structured lighting input.

[0053] Spatial structured lighting can be modulated by applying a field to the medium through stimulated emission processes and through the interaction of photoluminescence with the medium.

[0054] The device can be configured to modulate the optical response of the device based on adjusting a field applied to the medium. The field can be a local field or a global field. The field can be an electric field, a magnetic field, illumination different from spatially structured illumination, or a temperature field. The field can include at least two selected from the group consisting of an electric field, a magnetic field, illumination different from spatially structured illumination, and a temperature field. The device can be configured to modulate the optical response of the medium by electrically gating the entire medium. The device can also be configured to modulate the optical response of the medium by locally electrically gating at least one defined spatial sub-region of the medium.

[0055] The device may include a set of first electrodes configured to apply a locally reconfigurable electric field to a medium. The device may be configured to use the first electrodes to apply local electrical gating. The reconfigurable electric field may be used to modulate an optical response. The reconfigurable electric field may be used to modulate the lasing threshold of the medium.

[0056] The first electrode can be patterned below the dielectric. For example, the first electrode can be interposed between the dielectric and the substrate. The first electrode can also be patterned above the dielectric. For example, the dielectric can be interposed between the electrode and the substrate. In other words, the first electrode can be disposed on the upper surface of the dielectric opposite to the substrate. The first electrode can be patterned at a laterally adjacent location on the dielectric. For example, the first electrode can be offset from the dielectric in a plane in which the dielectric at least primarily resides. In other words, the first electrode can be coplanar with the dielectric.

[0057] The first electrode may have a two-dimensional (2D) arrangement, and the first electrode is configured to apply a locally reconfigurable electric field across the medium. The 2D arrangement may define a 2D mesh.

[0058] The device may be configured to use a first electrode for training a medium using the method described in accordance with the sixth aspect below. The device may be configured to store a configuration of a locally reconfigurable electric field corresponding to a stable (at least local) minimum of the loss function. The device may be configured to retrieve and apply the stored configuration of the locally reconfigurable electric field.

[0059] When the operation of the device includes a lasing mode, the device can be configured to apply a reconfigurable electric field using a first electrode, the reconfigurable electric field modulating the lasing threshold of the medium. When the operation of the device is based on photoluminescence, the device can be configured to apply a reconfigurable electric field using a first electrode, the reconfigurable electric field modulating the way spatially structured illumination interacts with different regions of the medium.

[0060] Alternatively, the device may include a second set of electrodes configured to inject charge carriers into the dielectric. In other words, the second electrodes are not separated from the dielectric by an electrically insulating layer. Therefore, rectifier contacts, tunneling barrier contacts, Schottky contacts, etc., may not be considered electrically insulating layers. Electrical insulation can mean that current (above leakage current) may not flow in response to an applied field below the irreversible electrical breakdown field. The device may be configured to use the second electrodes to apply a localized current through the dielectric. The injection of charge carriers can be used to modulate the optical response. The injection of charge carriers can be used to modulate the lasing threshold of the dielectric.

[0061] The second electrode can be patterned below the dielectric. For example, the second electrode can be interposed between the dielectric and the substrate. The second electrode can also be patterned above the dielectric. For example, the dielectric can be interposed between the second electrode and the substrate. In other words, the second electrode can be disposed on the upper surface of the dielectric opposite to the substrate. The second electrode can be patterned at a laterally adjacent location on the dielectric. For example, the second electrode can be offset from the dielectric in a plane in which the dielectric at least primarily resides. In other words, the second electrode can be coplanar with the dielectric.

[0062] The second electrode may have a 2D arrangement, and the second electrode is configured to apply a locally reconfigurable current across the dielectric. The 2D arrangement may define a 2D mesh.

[0063] The device may be configured to use a second electrode for training the medium using the method described in the sixth aspect below. The device may be configured to store a locally reconfigurable current corresponding to a stable (at least local) minimum of the loss function. The device may be configured to retrieve and apply the stored locally reconfigurable current.

[0064] In other examples, the device may be configured to include a set of electrodes comprising: a subset of first electrodes configured to apply a locally reconfigurable electric field to a dielectric, and a subset of second electrodes configured to inject charge carriers into the dielectric. The set of electrodes may consist of a subset of the first electrodes and a subset of the second electrodes.

[0065] The device can be configured to modulate the optical response based on providing an interference effect in the medium. The optical response can be a lasing response.

[0066] The device can be configured to modulate the optical response based on projecting a mask, which takes the form of a spatially structured beam from a second light source, onto at least a portion of the medium. The device can be configured to provide spatial structuring to the spatially structured beam using a digital micromirror device (DMD) or a spatial light modulator (SLM), through optical interference or holography, or through spatial focusing.

[0067] The apparatus can be configured to project a trained pixel grid from a mask onto a medium, superimposed on spatially structured illumination, to modulate the complex refractive index of the medium. In other words, the medium is trained using a pixel grid from a mask that takes the form of a spatially structured beam from a second light source.

[0068] Alternatively, the device can be configured to modulate the optical response based on superimposing a mask with spatially structured illumination. The mask can be superimposed with spatially structured illumination before the output of the illumination. For example, an image corresponding to the mask can be combined with an image corresponding to the spatially structured illumination before being output to a DMD, SLM, etc. The mask can be superimposed with spatially structured illumination via addition. The mask can be superimposed with spatially structured illumination via convolution.

[0069] Alternatively, the mask can be superimposed on spatially structured illumination at a point along the optical path leading to the medium. For example, a beam combiner can combine spatially structured illumination with a spatially structured beam from a second source.

[0070] A mask may comprise a set of mask pixels corresponding to the maximum illumination intensity, or take the form of a set of mask pixels corresponding to the maximum illumination intensity. A mask may comprise a set of mask pixels corresponding to the minimum illumination intensity, or take the form of a set of mask pixels corresponding to the minimum illumination intensity. A mask may comprise a set of mask pixels, or take the form of a set of mask pixels, each mask pixel corresponding to an illumination intensity less than the maximum illumination intensity and greater than the minimum illumination intensity. A mask may take the form of an image formed by mask pixels corresponding to varying illumination intensities.

[0071] The apparatus may be configured to use a spatially structured beam from a second light source for training a medium using the method described in accordance with the sixth aspect below. The apparatus may be configured to use superimposed masks for training the medium using the method described in the sixth aspect below. The apparatus may be configured to store a configuration of masks corresponding to stable (at least local) minima of the loss function. The apparatus may be configured to use a spatially structured beam from the second light source to retrieve and apply the stored configuration of the masks.

[0072] The device can be configured to modulate the optical response based on local magnetic control. The device can be configured to modulate the optical response based on local magnetic plasmon control. The device can be configured to modulate the optical response based on global magnetic control. The device can be configured to modulate the optical response based on global magnetic plasmon control. 。

[0073] The device may include a magnetic element arranged to provide a locally reconfigurable magnetic field across a medium. The locally reconfigurable magnetic field can be used for localized magnetic control or localized magnetic plasma control. The magnetic element may include nanomagnets. The device may be configured to use the magnetic element to apply a locally reconfigurable magnetic field across a medium. The magnetic state of the magnetic element can locally control how much spatially structured illumination is absorbed or enhanced, and can be used to modulate the optical response. The magnetic element may be configured such that its magnetic state can be reconfigured by at least one of an applied magnetic field, current / voltage control, or an all-optical magnetic switch.

[0074] The magnetic element can be a permanent magnet. In this way, the training state of the medium can be directly stored in the device itself.

[0075] Magnetic elements can be disposed on the upper surface of the medium. The medium can be inserted between the magnetic elements and the substrate. The magnetic elements can be disposed on the upper surface of the medium opposite to the substrate. The magnetic elements can be arranged adjacent to the medium, i.e., offset from the medium in a plane in which the medium at least predominates. In other words, the magnetic elements can be coplanar with the medium. The magnetic elements can be arranged to define a 2D grid.

[0076] The apparatus may be configured to use magnetic elements for training the medium using the method described in the sixth aspect below. The apparatus may be configured to store a configuration of locally reconfigurable magnetic fields corresponding to stable (at least local) minima of the loss function. The apparatus may be configured to use magnetic elements to retrieve and apply the stored configuration of the locally reconfigurable magnetic fields.

[0077] The device can be configured to modulate an optical response based on changing the temperature of a medium. The device may include one or more heaters configured to change the temperature of the medium. Each heater may be an on-chip heater. The device can be configured to control the heaters to change the temperature of the medium locally or globally.

[0078] The apparatus may be configured to use a heater for training the medium using the method described in the sixth aspect below. The apparatus may be configured to store locally varying temperatures corresponding to stable (at least local) minimum values ​​of the loss function. The apparatus may be configured to use a heater to retrieve and apply the stored locally varying temperatures.

[0079] The device can be configured to determine ambient temperature based on modulation of the optical response of the sensing medium. In other words, the ambient temperature can be inferred based on changes in the spatially structured response of the sensing medium to one or more reference modes of spatially structured lighting as the ambient temperature changes.

[0080] The methods described for modulating optical responses can be combined with each other.

[0081] According to a second aspect of the present invention, a method for manufacturing an apparatus as described in the first aspect is provided. The method includes: providing a first substrate having a main surface; providing a heteroepitaxial wafer having an intermediate sacrificial layer interposed between a second substrate and an upper layer; bonding an upper surface of the heteroepitaxial wafer to the main surface of the first substrate; forming an adhesive medium comprising the first substrate and the upper layer, the adhesive medium comprising etching the sacrificial layer to remove the second substrate and the sacrificial layer; and photolithographically patterning a structure on the surface of the adhesive medium from which the sacrificial layer has been etched.

[0082] The method described in the second aspect may include features corresponding to any feature of the apparatus as described in the first aspect.

[0083] According to a third aspect of the present invention, an optical image processing system is provided. The optical image processing system includes the apparatus as described in the first aspect. The optical image processing system further includes an imaging spectrometer configured to detect a spatially structured response.

[0084] Readouts from an optical image processing system can be performed spatially uniformly (i.e., using a single spatial integral spectrum for the entire medium for each input image) or spatially with resolution (i.e., using a spatial integral spectrum for each of multiple defined spatial sub-regions of the medium for each input image).

[0085] An imaging spectrometer can be configured to perform readout over a defined spatial sub-region of the medium. Readout can be performed over this defined spatial sub-region using a lens arrangement in the detection path between the medium and the spectrometer. Alternatively, readout can be performed over the entire medium.

[0086] The optical image processing system may also include a neutral density filter configured to maintain the pump power of spatially structured illumination.

[0087] The optical image processing system may also include a digital micromirror device configured to provide spatially structured illumination. The digital micromirror device can reflect a geometrically arranged configuration.

[0088] Digital micromirror devices can provide structured spatial illumination by selectively reflecting unstructured light provided by a first light source. The first light source may include or be in the form of a laser source. The first light source may also include or be in the form of a light-emitting diode (LED).

[0089] The system may be configured such that most functional elements (e.g., all functional elements, including the dielectric) are arranged in an on-chip package, in which functional optical elements are assembled in a multilayer vertical stack on a flat substrate (such as a silicon (Si) chip). The on-chip package (hereinafter referred to as a "photonic chip") may be a flat on-chip package. In the on-chip package, the spatially structured illumination source may include a spatially uniform flat shape factor light source, such as a microstrip laser.

[0090] The optical image processing system may also include a spatial light modulator configured to provide spatially structured illumination.

[0091] Spatial light modulators can be used to impart spatial structure to light output from an illumination source. Alternatively, spatially structured illumination can be provided by optical pumps capable of outputting spatially structured illumination, such as nanolaser arrays, vertical cavity surface-emitting laser arrays, or miniature light-emitting diode arrays. Readout hardware may include an on-chip spectrometer on a photonic chip. The photonic chip may also include a dielectric and an illumination source. The illumination source may include a photodiode.

[0092] Optical image processing systems can be configured as non-trainable repositories for data processing.

[0093] Optical image processing systems can be configured as fully trainable deep neural networks for inference.

[0094] The system as described in the third aspect may include features corresponding to any features of the apparatus as described in the first aspect and / or the method as described in the second aspect.

[0095] According to a fourth aspect of the present invention, a computer vision product is provided. The computer vision product includes an optical image processing system as described in the third aspect. In other words, the computer vision product includes the apparatus as described in the first aspect and an imaging spectrometer configured to detect a spatially structured response.

[0096] Although the response of a medium is spatially structured, the detection of the spatially structured response using an imaging spectrometer does not require spatial discretization. For example, an imaging spectrometer can be configured to detect the spectrum of the spatially structured response corresponding to the entire medium or a single region thereof. Alternatively, an imaging spectrometer can be configured to detect the spectrum of the spatially structured response corresponding to each of two or more distinct regions of the medium.

[0097] The computer vision product also includes a processor configured to determine changes in the spectrum provided by a spatially resolved spectrometer. The spectrum can be a spatially resolved spectrum.

[0098] Computer vision products can be selected from the following groups: autonomous vehicles; autonomous drones, autonomous aircraft, autonomous ships, low-latency image processing systems, closed-circuit television (CCTV) camera processing systems, industrial process control systems, medical imaging systems, and surgical control systems. CCTV camera processing systems can be used for crowd recognition. Industrial process control systems can be used to monitor production lines, such as food production lines or component production lines. Medical imaging systems can be used to process data including MRI images or tumor images. Surgical control systems can be used to process endoscopic camera feeds.

[0099] Computer vision products can be autonomous vehicles or systems used for object detection and classification.

[0100] The vehicle can be an autonomous car, an autonomous drone, an autonomous aircraft, or an autonomous vessel. The object can be a human face. The system can be a low-latency image processing system, an industrial process control system, a medical imaging system, or a surgical control system.

[0101] The computer vision product as described in the fourth aspect may include features corresponding to any feature of the apparatus as described in the first aspect, the method as described in the second aspect, and / or the system as described in the third aspect.

[0102] According to a fifth aspect of the present invention, a method is provided for operating an apparatus as described in the first aspect, an optical image processing system as described in the third aspect, or a computer vision product as described in the fourth aspect. The method includes providing spatially structured illumination to the apparatus. The method further includes detecting a spatially structured response.

[0103] A spatially structured response can correspond to a computational task. This spatially structured response can be reconfigured to correspond to another computational task using training media.

[0104] The method of operating the device can be an optical image processing method.

[0105] The method may include modulating the optical response of the modulating medium. Modulation of the optical response may be performed in any manner described with respect to the first and sixth aspects.

[0106] The method described in the fifth aspect may include features corresponding to any feature of the apparatus described in the first aspect, the method described in the second aspect, the system described in the third aspect, and / or the computer vision product described in the fourth aspect.

[0107] According to a sixth aspect of the present invention, a method is provided for training an apparatus as described in the first aspect, an optical image processing system as described in the third aspect, or a computer vision product as described in the fourth aspect. The method includes: determining a loss function of the apparatus; determining whether the loss function has reached a stable minimum; and modulating the optical response in the apparatus.

[0108] The method iteratively includes: determining the loss function of the device; determining whether the loss function has reached a stable minimum; and the optical response in the modulation device.

[0109] The optical response can be iteratively modulated using either gradient descent or backpropagation algorithms.

[0110] The method can be continued until it is determined that the loss function has reached a stable minimum value.

[0111] The loss function for determining the device may include: evaluating the performance of a given task; and generating the loss function based on the task performance.

[0112] Modulation of the optical response can include providing an interference effect in the medium. The optical response can be a lasing response. Spatially structured illumination can be modulated by applying a field to the medium through stimulated emission processes and through the interaction of photoluminescence with the medium.

[0113] The optical response in a modulation device may include modulating the field applied to the medium.

[0114] A field can be a local field or a global field. A field can be an electric field, a magnetic field, lighting that is different from spatially structured lighting, or a temperature field. A field can include at least two of the following groups: electric field, magnetic field, lighting that is different from spatially structured lighting, and temperature field.

[0115] Reconfigurable training can be provided by modulating the interaction between spatially structured lighting and the medium by applying a field to the medium.

[0116] The modulated optical response may include projecting a mask, which takes the form of a spatially structured beam from a second light source, onto at least a portion of the medium.

[0117] Modulating the optical response may include electrically gating the entire medium or locally electrically gating at least one defined spatial sub-region of the medium. A set of electrodes may be used to apply the local electrical gating. The set of electrodes may be patterned at laterally adjacent locations, above, or below the medium. The set of electrodes may have a 2D arrangement and be configured to apply a locally reconfigurable electric field across the medium. The 2D arrangement may define a 2D grid. The reconfigurable electric field may be used to modulate the optical response. The reconfigurable electric field may be used to modulate the lasing threshold of the medium.

[0118] The modulated optical response may include at least one of local magnetic control, local magnetoplasmic control, global magnetic control, and global magnetoplasmic control. Magnetic elements, such as nanomagnets, may be used to provide local magnetic control or local magnetoplasmic control. The magnetic elements may be configured to apply a locally reconfigurable magnetic field across the medium. The magnetic state of the magnetic elements can locally control how much spatially structured illumination is absorbed or enhanced and can be used to modulate the optical response. The magnetic state may be reconfigurably programmed by at least one of the applied magnetic field, current / voltage control, or all-optical magnetic switching.

[0119] Modulating the optical response can include changing the temperature of the medium. Changing the temperature of the medium can be done using a heater, such as an on-chip heater. Changing the temperature of the medium can also be done using ambient temperature. The temperature can be changed locally or globally.

[0120] The method described in the sixth aspect may include features corresponding to any feature of the apparatus described in the first aspect, the method described in the second aspect, the system described in the third aspect, the computer vision product described in the fourth aspect, and / or the method described in the fifth aspect. Attached Figure Description

[0121] Some embodiments of the invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 This is a schematic perspective view of a first optical image processing system, which includes a device containing a medium. Figure 2A yes Figure 1 A schematic plan view of a portion of the medium; Figure 2B It is along Figure 2A A schematic cross-sectional view of the line A-A' in the diagram.

[0122] Figure 3 It is a schematic plan view of a medium including a spatially separated photonic network; Figure 4 It is a schematic side view of a medium placed on an uneven surface; Figure 5 This is a schematic side view of the medium as a multilayered heterogeneous structure; Figure 6 It is a schematic side view of the medium including the voids; Figure 7 It is a schematic side view of a medium including a suspended structure; Figure 8 It is a schematic perspective view of a 3D patterned medium; Figure 9 It is a process flow diagram of the method for manufacturing the device; Figure 10 This is a schematic block diagram of the second optical image processing system; Figure 11 This is a schematic block diagram of the third optical image processing system; Figure 12 This is a schematic block diagram of the fourth optical image processing system; Figure 13 It is a schematic side view of a device including electrodes for training; Figure 14 This is a schematic side view of a device including nanomagnets for training; Figure 15 This is a process flow diagram of the training device method; Figure 16 Examples of correct (top) and incorrect (bottom) classifications are shown; Figure 17 The test accuracy score of 97.7% is shown, determined using the device and through the MNIST handwritten digit classification task. Figure 18A Exemplary images are shown for comparison of optical physical calculations performed using a device provided with raw image data with logistic regression performed on the same raw image data. Figure 18A ); Figure 18B Is to provide Figure 18A The photonic network of the image; Figure 18C It comes from Figure 18B Spatial-resolved spectra of photonic networks; Figure 18D The use of the device is shown. Figure 18C The spatially resolved spectrum of the spectrum determined a test accuracy score of 98.15%. Figure 18E The test accuracy score of 90.5% is shown, determined by logistic regression on the original image data. Figure 18F This is a perspective view example of optical physics calculations; Figure 19 It is a schematic perspective view of a device that receives spatially structured lighting as a spatially structured response from another device; Figure 20 This is a schematic block diagram of the fifth optical image processing system; Figure 21 This is a schematic block diagram of the sixth optical image processing system; Figure 22 This is a schematic block diagram of the seventh optical image processing system; Figure 23A This shows the experimental data for the input image; Figure 23B and Figure 23C This shows experimental data illustrating the extracted feature / edge maps provided by two different optical modes; Figure 23D This is experimental data showing feature / edge maps extracted from multiple optical modes, where logistic regression has been used to train which combination of modes to use for image edge extraction; Figures 24A to 24D This is an example of simulated data using four different optical modes to detect different edges (top / right / left / bottom). The color bars correspond to the presence of the desired edge, where 1 (white) indicates the presence of the desired edge and 0 (black) indicates the absence of the desired edge. Figure 25A It is a labeled planar SEM micrograph of a medium in the form of a photonic network patterned from III-V semiconductor materials; Figure 25B This is a schematic projection of the first configuration of electrodes for electro-optic modulation of the dielectric. Figures 26A to 26G A method for manufacturing a first configuration of electrodes for electro-optic modulation of a dielectric is schematically illustrated. Figures 27A to 27E A method for manufacturing a second configuration of electrodes for electro-optic modulation of a dielectric is schematically illustrated; Figures 28A to 28E A method for manufacturing a third configuration of electrodes for electro-optic modulation of a dielectric is schematically illustrated. Figure 29 shows the corresponding figures. Figure 28A , Figure 28C and Figure 28E A schematic floor plan; Figure 30A A low-density photonic network with network links of average length of 7 μm is schematically illustrated. Figure 30B For a pair of different conditions Figure 30A The simulated lasing spectrum of the photonic network shown; Figure 31A A high-density photonic network with network links of average length of 2 μm is schematically illustrated. Figure 31B For a pair of different conditions Figure 31A The simulated lasing spectrum of the photonic network shown; Figures 32A to 32G Experimental data for an image classification task performed without masks are presented; and Figures 33A to 33G This demonstrates the execution of the code in the presence of a mask. Figures 32A to 32G Experimental data for the same image classification task. Detailed Implementation

[0123] In the following text, the same parts are indicated by the same reference numerals.

[0124] introduction This document describes an apparatus for use in optical image processing. The apparatus is driven by projecting spatially structured illumination, including input image data, onto a medium of the apparatus. As a result of the interaction between the spatially structured illumination and the medium, and within the medium, a spatially structured response is output (which is typically also structured in frequency). By modulating the interaction between the spatially structured illumination and the apparatus, and within the apparatus, or alternatively by processing the input image data, the apparatus can be used to perform neuromorphic optical computations.

[0125] First optical image processing system 1 refer to Figure 1 The first optical image processing system 1 (hereinafter referred to as the "first system") is shown.

[0126] The first system 1 includes a device 2 for optical image processing, a first light source 3 for providing spatially structured illumination 4, including input image data, to the device 2, and an imaging spectrometer 5 for detecting a spatially structured response 6 from the device 2. The device 2 includes a medium 7 and a substrate 8. The medium 7 is disposed on a surface 9 of the substrate 8 and is capable of supporting overlapping optical modes 10. In response to receiving the spatially structured illumination 4, a set of overlapping optical modes 10 is excited in the medium 7 and outputs a spatially structured response 6.

[0127] Although described as a spatially structured response 6, it should be understood that the response 6 of medium 7 will typically be structured both spatially and frequency-wise. While the response of medium 7 is spatially structured, the detection of the spatially structured response 6 using imaging spectrometer 5 does not require spatial discretization. For example, imaging spectrometer 5 may be configured to detect the spectrum of the spatially structured response 6 corresponding to the entire medium 7 (i.e., the global region) or a single region thereof. Alternatively, imaging spectrometer 5 may be configured to detect the spectrum of the spatially structured response 6 corresponding to each of two or more different regions of medium 7.

[0128] Medium 7 includes a first photonic network 11, in which input image data is processed and spatially structured response 6 is output from the first photonic network.

[0129] Also refer to Figure 2A and Figure 2B This shows a portion of the first photonic network 11.

[0130] The first photonic network 11 includes nodes 131, 132, ..., 133. NMultiple network links 121, 122, ..., 12 are radiatingly coupled. N And it is set on a flat surface 9.

[0131] The spatially structured response 6 may include contributions arising from the interaction between the spatially structured illumination 4 and the medium 7 in various ways. First, the spatially structured response 6 may include contributions arising from scattering of the spatially structured illumination 4 from the medium 7. Second, the spatially structured response 6 may include contributions from photoluminescence, that is, absorption of the spatially structured illumination 4 by the medium 7 resulting in the emission of photons via fluorescence or phosphorescence. Third, the spatially structured response 6 may include contributions from stimulated emission processes, such as lasing or amplified spontaneous emission (“ASE”) driven by the interaction between the spatially structured illumination 4 and the medium 7.

[0132] Different optical modes 10 in a set of overlapping optical modes 10 excited in response to spatially structured illumination 4 may differ in one or more of the following aspects: frequency; spatial mode shape; and spatial position within medium 7.

[0133] A set of overlapping optical modes 10 includes two or more optical modes that are lasing modes. Spatial structuring of the spatially structured illumination 4 can cause an imbalance in mode competition between lasing modes. However, this is not mandatory, and a set of overlapping optical modes 10 may alternatively include only optical modes that are not lasing modes.

[0134] The resonant mode capable of inducing lasing can be provided by using a first light source 3. The first light source 3 may include a suitable pulsed pump laser coupled to an optical microscope. The suitability of the pulsed pump laser is determined at least in part by its ability to provide a sufficiently large pulse flux to induce lasing within the medium 7. If the medium 7 comprises a semiconductor material, a suitable pulsed pump laser may be a femtosecond pulsed laser. If the medium 7 comprises a polymer material, a suitable pulsed pump laser may be a nanosecond pulsed laser.

[0135] By performing physical transformations in parallel and inherently utilizing its internal (optical and optoelectronic) physics, device 2 can achieve higher processing speeds than software networks. Specifically, the speed of the stimulated emission process means that device 2 can process input images in a short time of approximately 10 ps. Furthermore, since device 2 processes data in the form of light, it can achieve higher energy efficiency than conventional CMOS / GPU hardware.

[0136] The spectrum provided by the imaging spectrometer 5 based on the spatially structured response 6 is sensitive to minute changes in the spatially structured illumination 4. Shifts in these spectra can form the basis for performing image processing AI tasks using the device 2. Minor changes in the spatially structured illumination 4 may cause changes in the relative amplitude or even the identity of a set of excited overlapping optical modes 10, resulting in changes in the spectrum obtained by the imaging spectrometer 5.

[0137] The same input data (in the form of spatially structured lighting 8) can be provided to device 2 multiple times in the form of transformations such as translation, rotation, reflection, spatial magnification, or reduction. Providing the same input data to device 2 multiple times in such a transformation and combining the spatially structured response 6 for each time the input data is provided can help enhance the computational performance of device 2 (that is, the test accuracy score determined by the MNIST handwritten digit classification task).

[0138] In lasing-based operating modes, the power range used for computational tasks is defined by the lasing light input-output curve of measuring device 2. The minimum power is defined by the lasing threshold (at which at least one lasing mode exists in the spectrum of the spatially structured response 6), and the maximum power is defined by either the power at which the number of lasing modes in the spectrum of the spatially structured response 6 is maximized, or the power at which the light intensity output decreases due to heating / material degradation, whichever is lower. In these modes, the power range is typically much higher than the minimum power and much lower than the maximum power.

[0139] Network links 121, 122, ..., 12 N Through nodes 131, 132, ..., 13 N They are physically in contact with each other and radiatively coupled. However, this is not necessary, and network links 121, 122, ..., 12... N At least one subset of the data can be found in nodes 131, 132, ..., 13 N The spatial separation between the device and other network links is less than the design wavelength at which device 2 is configured to operate. In this way, network links 121, 122, ..., 12... N Direct physical contact between them is not necessary.

[0140] Network links 121, 122, ..., 12 N It is an active photonic waveguide. However, this is not mandatory, and network links 121, 122, ..., 12 N It may include components other than active photonic waveguides, or may include low-dimensional features such as quantum wells instead of active photonic waveguides. Network links may include or take the form of particle chains. Network links may include or take the form of reflective elements. Network links may include or take the form of passive photonic waveguides.

[0141] refer to Figure 3 This shows a second example of device 2.

[0142] Medium 7 may include a second photonic network 112 spatially separated from the first photonic network 11 by a distance greater than the designed wavelength. In this way, device 2 can provide a set of parallel outputs. Combining parallel outputs can help improve the computational performance of device 2.

[0143] The second photonic network 112 may have the same materials and topology as the first photonic network 11 (optionally rotated or otherwise transformed). Alternatively, the second photonic network 112 may differ from the first photonic network 11 in one or both of its materials and topology.

[0144] refer to Figure 4 The third example of device 2 is shown.

[0145] The dielectric 7 can be disposed on the substrate 8 and on an uneven surface 14. The surface 14 can have a wide variety of uneven profiles. In particular, the surface 14 may include flat portions connected by inclined portions, or it may be undulating. Furthermore, the surface 14 may not be the surface of the substrate 8, but may instead be defined by a combination of the substrate 8 and a patterned layer (not shown), such as a set of gate electrodes of a set of magnetic elements (in...) interposed between the substrate 8 and the dielectric 7. Figure 13 (Best example in China).

[0146] Spatial structuring of medium 7 refer to Figure 5 The fourth example of device 2 is shown.

[0147] Medium 7 may include dielectric layers 151, 152, ..., 15 N A multilayer heterogeneous structure. Dielectric layers 151, 152, ..., 15 N It can have different thicknesses and / or compositions, and does not have to be planar.

[0148] Each dielectric layer 151, 152, ..., 15 N Medium 7 is provided, thereby allowing a single input to spatially structured lighting to be provided by each medium layer 151, 152, ..., 15 N Parallel processing. If the dielectric layers are 151, 152, ..., 15... N The gaps between them are very small, so the adjacent dielectric layers 151, 152, ..., 15 N The interactions between them may also become significant. Preferably, the dielectric layers 151, 152, ..., 15 N They differ from each other in at least one (and possibly more) of the materials, topology, and orientation.

[0149] refer to Figure 6 The fifth example of device 2 is shown.

[0150] The pores 16 may be arranged non-uniformly within the medium 7. The pores 16 may have a regular shape (such as an ellipsoid) or an irregular shape. The pores 16 do not have to have the same shape and may have any shape.

[0151] The gap 16 can add nonlinear optical loss to the medium 7. This can be used to introduce additional computing power to the device 2.

[0152] refer to Figure 7 The sixth example of device 2 is shown.

[0153] The voids 16 can be arranged non-uniformly within the medium in a manner that defines the suspension structure 17. The suspension structure 17 does not need to be of a regular shape.

[0154] The levitation structure 17 can be configured to achieve 3D coupling between regions of the medium 7 that are not close to each other (e.g., regions of the photonic network 11). The levitation structure 17 can also have lower optical losses compared to waveguides formed on the substrate. This is because there is typically a larger refractive index difference between the medium 7 and the air filling the voids 16 compared to the refractive index difference between the medium 7 and the substrate 8.

[0155] refer to Figure 8 The seventh example of device 2 is shown.

[0156] The medium 7 does not need to be planar. In other words, the medium 7 can be 3D patterned because it has both in-plane and out-of-plane textures. In particular, expanded polystyrene foam beads can be used to provide so-called "inverse opal" style 3D patterning. Alternatively, 3D patterning can be provided via optical interference writing, or via nanoscale 3D printing (such as so-called "two-photon lithography" or focused ion beam deposition), or via 3D nanoscale self-assembly.

[0157] As described above, medium 7 has a spatially non-uniform physical structure. However, this is not mandatory, and medium 7 can have a spatially uniform physical structure along three orthogonal directions, achieved by having a spatially non-uniform complex refractive index. In particular, the first photonic network 11 can be replaced by a uniform thin film with a complex refractive index, and a second light source (not shown) can be used to provide the spatially non-uniform complex refractive index by projecting a spatially structured beam onto the uniform thin film. In other words, the first light source 3 is used to provide the spatially structured illumination 4 as a probe, and the second light source is used to provide the spatial structuring of the complex refractive index of medium 7.

[0158] For example, a uniform thin film with complex refractive index can be formed from a uniform film of indium tin oxide (ITO), indium phosphide (InP), or another active material illuminated / pumped with a non-uniform spatial pattern.

[0159] Equipment manufacturing The dielectric 7 can contain a wide variety of semiconductor materials. In particular, the dielectric 7 can contain III-V group semiconductor materials (such as indium phosphide (InP)) or organic semiconductor materials (such as rhodamine 6G dye). Therefore, the dielectric can be a solid-state gain dielectric or a liquid-phase dye dielectric.

[0160] Device 2 can be manufactured in a wide variety of ways, such as: photolithography on a thin III-V semiconductor layer bonded to a SiO2-coated Si substrate; electrospinning dye-mixed polymer nanofibers onto a TEM grid and then post-annealing to create network links 121, 122, ..., 12 N They are fused together to form at least one photonic network 11, 112; or by nanoimprint lithography of a soft polymer doped with laser dye.

[0161] Media such as Medium 7 can be produced cheaply, are virtually infinitely configurable, and are easily compatible with widely used semiconductor and polymer manufacturing technologies.

[0162] refer to Figure 9 The method for manufacturing apparatus 2 will now be described.

[0163] The method includes: providing a first substrate having a main surface (step S1.1); providing a heteroepitaxial wafer having an intermediate sacrificial layer interposed between a second substrate and an upper layer (step S1.2); bonding the upper layer to the main surface of the first substrate (step S1.3); and etching the sacrificial layer to remove the second substrate and the sacrificial layer, thereby forming an adhesive medium comprising the first substrate and the upper layer (step S1.4). The method may then include: patterning a structured photolithography pattern onto the surface of the adhesive medium from which the sacrificial layer has been etched (step S1.5). The adhesive medium is an example of device 2. Specifically, the first substrate may be substrate 8, and the main surface may be surface 9 or surface 14.

[0164] The first substrate may include or may be a silicon (Si) wafer. The main surface may be the surface of the silicon wafer or the surface of a silicon dioxide (SiO2) layer deposited on silicon. The second substrate may include an indium phosphide (InP) wafer. The intermediate sacrificial layer may include indium gallium arsenide (InGaAs). The top layer may include indium phosphide (InP).

[0165] Second optical image processing system 18 refer to Figure 10The second optical image processing system 18 (hereinafter referred to as the "second system") is shown.

[0166] The second system 18 differs from the first system 1 in that it uses an external spatially structured illumination source 19 instead of the first light source 3, and includes a spectrometer 20 that does not necessarily have to be an imaging spectrometer (such as an imaging spectrometer 5). In other words, the spectrometer 20 does not have to be spatially resolved.

[0167] In the second system 18, device 2 receives spatially structured illumination 4 from spatially structured illumination source 19 and outputs its spatially structured response 6 to spectrometer 20. The readouts from spectrometer 20 are provided to computer system 1 for processing.

[0168] Since spectrometer 20 does not necessarily have to be spatially resolved, it can provide readouts corresponding to the spatially structured response 6 from the entire medium 7. However, if spectrometer 20 is an imaging spectrometer (such as imaging spectrometer 5), spectrometer 20 can provide readouts corresponding to the spatially structured response 6 from at least one defined spatial sub-region of the medium 7.

[0169] In spectrometer 20, the spatially structured response 6 can be collected by a microscope objective (not shown), filtered to remove contributions corresponding to the spatially structured illumination 4, dispersed using a grating (not shown), and focused into a CCD camera (not shown). Spatial-resolved readout can be achieved by including a lens arrangement (not shown) between the medium 7 and the spectrometer 20.

[0170] The second system 18 can use a neutral density filter (not shown) to maintain the pump power of the spatially structured illumination 4 by monitoring the pump power and correcting it relative to a reference power value. Achieving this real-time correction can help improve computational performance.

[0171] Third optical image processing system 22—Free space implementation method refer to Figure 11 The third optical image processing system 22 (hereinafter referred to as the "third system") is shown.

[0172] The third system 22 has a free space implementation, in which light is projected through the free space between its constituent parts.

[0173] The third system 22 differs from the second system 18 in that it includes a digital micromirror device (“DMD”) 23 arranged in a reflective geometry. Specifically, the DMD 23 is arranged to receive input light 24 from an illumination source 25 external to the third system 22 and to provide spatially structured illumination 4 to the device 2 via a beam splitter 26 and an objective lens 27. The spatially structured response 6 of the device 2 is detected by a spectrometer 20, and its readout is provided to a computer system 21 for processing.

[0174] In other examples, DMD 23 may alternatively be arranged to illuminate medium 7 through the transmission geometry.

[0175] Using a DMD to impart spatial structure (such as spatial structure corresponding to an image) to the input light (such as input light 24) helps make the third system 22 cheaper and easier to manufacture. This is because DMD components are widely available and industrially produced. Using a DMD 23 also helps make the third system 22 suitable for operation at frequencies from GHz to THz.

[0176] Fourth optical image processing system 28—On-chip implementation refer to Figure 12 The fourth optical image processing system 28 (hereinafter referred to as the "fourth system") is shown.

[0177] The fourth system 28 differs from the third system 22 in that it has an on-chip implementation rather than a free-space implementation. The fourth system 28 also differs from the third system 22 in that it uses a spatial light modulator (“SLM”) 29 in the transmission geometry instead of a DMD (such as DMD 23) to provide spatially structured illumination 4.

[0178] The components of the fourth system 28 (i.e., SLM 29, device 2, and spectrometer 20) are arranged in an on-chip package (also referred to herein as a "photonic chip"). In the on-chip package, the components of the fourth system 28 are arranged in a vertical stack of multiple layers on a flat substrate (such as a silicon (Si) chip). The flatness of the on-chip package helps to make it easier to integrate the system 28 into larger components.

[0179] SLM 29 is used to impart spatial structure to the input light 24 provided by illumination source 25. Illumination source 25 can be a spatially uniform, flat shape factor light source, such as a microstrip laser. However, it is not necessary to use an SLM to impart spatial structure to the spatially structured illumination 4, and the spatially structured illumination 4 can instead be provided directly from illumination source 25, such as a nanolaser array, a vertical cavity surface-emitting laser (“VCSEL”) array, or a micro light-emitting diode (“LED”) array, that is, illumination source 25 capable of directly outputting the spatially structured response 4.

[0180] Neuromorphic computing scheme The device 2 described in this paper can form the basis for implementing one or more neuromorphic computing schemes.

[0181] Storage calculation Device 2 can be used as a reservoir for reservoir computation. In this scheme, device 2 does not necessarily need to be trainable, and the input data in the spatially structured illumination 4 is transformed into a spatiotemporal gain distribution in device 2 due to the absorption of the spatially structured illumination 4 by the medium 7. Light emitted from the medium 7 is guided within the medium 7 and propagates across multiple paths within the medium 7. Constructive interference of light across at least two closed paths within the medium can generate optical modes 10, which can be amplified by optical gain. When the optical gain of the modes is balanced with the mode loss, lasing from these modes 10 occurs at a threshold. The optical modes 10 have non-zero spatial overlap and are therefore competitive for gain. When two or more optical modes 10 are lased, mode competition due to spatial burning and nonlinear coupling can cause the mode intensity and wavelength to vary nonlinearly with the pump power of the spatially structured illumination 4 across the medium 7.

[0182] In this way, the system including device 2 can perform multiple linear and nonlinear operations on the input data and output a unique spectrum, such as a lasing spectrum, formed by these operations. Storage computations can be performed by processing these spectra (i.e., the storage output). Specifically, the spectral output can be regressed to directly produce computational outputs (see, for example, [link to relevant documentation]). Figure 17 The results shown are as described below. A reservoir computation scheme can be trained for a given computational task by optimizing the processing of these spectra for one or more given inputs.

[0183] The input data can be preprocessed to take the form of spatially structured lighting 4. Specifically, analog time-series data can be converted into grayscale pixel lighting values ​​and illuminated onto medium 7. One way to preprocess the input data is to preprocess the input data into a positive numerical matrix, binarize the preprocessed input data into a pattern projection on a DMD (such as DMD 23) or SLM (such as SLM 29), and project the spatially structured lighting 6 from the DMD or SLM onto device 2.

[0184] Input data preprocessing may include pixel dithering and / or noise representation to achieve grayscale and / or analog value input. These types of input data preprocessing can help improve computational performance.

[0185] Deep Neural Networks Device 2 can be used in a fully trainable deep neural network (“DNN”) for inference. Specifically, device 2 can be configured as one or more layers in a neural network, such as a convolutional neural network (“CNN”). In this scenario, device 2 must be trainable.

[0186] The device 2 can be used as a convolution kernel by projecting image data (such as image fragments) onto the medium 7. In this scheme, the spectral output from the spectrometer 20 is passed to another network layer for further processing, rather than being processed by regression to directly produce a computational output.

[0187] The spatially resolved response from the first network layer, including device 2, can be projected as an image onto another network layer (in... Figure 19 (Best example shown). The other network layer may be a different physical network having different properties from the first network layer. The other network layer may include hardware or hardware such as device 2, memristor network or nanomagnetic network.

[0188] If direct physical information transfer between the first network layer and another network layer is desired, the other network layer will need to be designed (e.g., by selecting appropriate materials) to accept the spatially structured response output of the first network layer as input. Alternatively, an intermediate measurement step can be used, for example, to measure the spatially structured response of the first network layer using an imaging spectrometer 5, and based on this, to prepare spatially structured illumination for projection onto the other network layer. The latter option may be preferred to maintain the signal-to-noise ratio across a sequence of two or more network layers.

[0189] There may be software and / or hardware layers before the first network layer, and the input data may be the output of the preceding or following network layers, rather than image data (in...). Figure 21 and Figure 22 (Best example shown). In this way, device 2 can be used as a pre- / post-processor as part of a larger computing system. This increases memory and thus allows the device to be used for video processing.

[0190] The output data can be post-processed in external circuitry or an external CPU (such as computer system 21). The output data can be extracted directly from the spatially structured response 6, for example, using spatially resolved spectra (such as optical lasing spectra) read from spectrometer 20.

[0191] train Device 2 can be trained in a wide variety of ways.

[0192] The first mode for training device 2 uses a second light source (not shown) to provide spatial structuring of the complex refractive index of medium 7.

[0193] refer to Figure 13 The device 2 is shown, which includes electrodes 30 for training the medium 7 inserted between the medium 7 and the substrate 8.

[0194] However, device 2 does not require this specific arrangement of electrodes 30, and the medium 7 can instead be inserted between electrodes 30 and substrate 8. In other words, electrodes 30 can be disposed on the upper surface of the medium 7 opposite to substrate 8. Alternatively, electrodes 30 can be arranged adjacent to the medium 7, that is, offset from the medium 7 in the plane in which the medium 7 is at least primarily located.

[0195] Electrode 30 can be arranged to define a 2D grid (or “pixel grid”). This arrangement increases the ease with which electrode 30 can be addressed.

[0196] In the second mode for training device 2, electrode 30 can be used to apply a locally reconfigurable electric field to medium 7, thereby allowing local electrical control of medium 7.

[0197] refer to Figure 14 The device 2 is shown, which includes a magnetic element 31 disposed on the upper surface of the medium 7.

[0198] However, device 2 does not require this specific arrangement of magnetic element 31, and medium 7 can instead be inserted between magnetic element 31 and substrate 8. In other words, magnetic element 31 can be disposed on the upper surface of medium 7 opposite to substrate 8. Alternatively, magnetic element 31 can be arranged adjacent to medium 7, that is, offset from medium 7 in the plane in which medium 7 is at least primarily located.

[0199] The magnetic element 31 can be arranged to define a 2D grid. This arrangement can improve the ease with which the magnetic element 31 can be addressed.

[0200] In the third mode for use in the training device, the magnetic element 31 can be used to apply a locally reconfigurable magnetic field across the medium 7, thereby allowing local magnetic control and / or local magnetic plasma control of the medium 7.

[0201] As will be discussed below, the fourth mode for training device 2 includes applying heat to medium 7.

[0202] refer to Figure 15 The method for training device 2 will now be described.

[0203] The method includes: initiating training (step S2.1); determining the loss function of device 2 (step S2.2); determining whether the loss function has reached a stable minimum value (step S2.3); if the determination is no, modulating the optical response in device 2 (step S2.4) and repeating steps S2.2 and S2.3 in the order stated; if the determination is yes, ending training (step S2.5).

[0204] Determining the loss function of device 2 may include: evaluating the device's performance for a given task; and generating the loss function based on the task performance.

[0205] The precise choice of the loss function will depend on the task for which Device 2 is trained. An example of a loss function suitable for image classification is cross-entropy loss. Another example would be maximizing the specific pattern corresponding to the input class while minimizing the pattern strength of other patterns.

[0206] The optical response in modulation device 2 implies a modification of the interaction between spatially structured illumination 4 and medium 7, as well as within the medium, resulting in a change in the spatially structured response 6 provided for a given spatially structured illumination 4 input.

[0207] The loss function is high when accuracy is low. The loss function is low when accuracy is high. The higher the loss, the greater the change in physical response between training iterations. After multiple iterations of the method, the loss reaches a stable minimum, and the computational performance of device 2 can be described as optimized for a given computational task. In other words, device 2 can be described as being trained for a given computational task when the loss is at (at least locally) a stable minimum.

[0208] The optical response in modulation device 2 includes modulating the field applied to medium 7. The field can be applied locally or globally to medium 7. In other words, the field can be applied to only a part of medium 7 or to the entire medium 7.

[0209] In a first mode for training device 2, the field takes the form of a mask, which is a spatially structured beam from a second light source projected onto at least a portion of medium 7. The spatially structured beam from the second light source can be spatially structured by DMD or by SLM, by optical interference or holography, or by spatial focusing. In the first mode for training device 2, the trained pixel grid in the mask is superimposed on the spatially structured illumination 4 on medium 7 to modulate the complex refractive index of medium 7. In other words, in the first mode for training device 2, the medium 7 is trained using a pixel grid in the mask taking the form of a spatially structured beam from the second light source. Alternatively, instead of using a spatially structured beam from the second light source, the trained pixel grid in the mask can be superimposed on the spatially structured illumination (see also...). Figures 32A to 33G (and the corresponding description).

[0210] In the second mode of the training device 2, electrode 30 is used to apply an electric field to the medium 7. When the operating mode of device 2 includes a lasing mode, electrode 30 can be used to apply a reconfigurable electric field that modulates the lasing threshold of the medium 7. When the operating mode of device 2 is based on photoluminescence, electrode 30 can be used to apply a reconfigurable electric field that modulates the interaction between the spatially structured illumination 4 and different regions of the medium 7. In this way, different regions of the medium 7 can respond to the spatially structured illumination 4 with higher or lower and / or offset sensitivity.

[0211] In the third mode used for training device 2, the field is a magnetic field, and magnetic elements 31 are used to apply magnetic control and / or magnetic plasma control 7 to the medium 7. Each magnetic element 31 has a locally modified magnetic state in which the spatially structured illumination 4 is absorbed and / or enhanced. The magnetic state can be reconfigurably programmed in a wide variety of ways, such as by the applied magnetic field, by current / voltage control, and / or by all-optical magnetic switching.

[0212] In the fourth mode for training device 2, the field takes the form of heat applied to medium 7. Heat can be applied to medium 7 using a heater (such as an on-chip heater) or by localized heating (such as by laser heating). Alternatively, heating can be provided by ambient temperature in such a way that device 2 is a sensor that changes its physical dynamics in response to changes in its environment. Such a sensor can be applied to atmospheric or weather sensing and / or vehicle or process control sensing.

[0213] Computing performance of device 2 The computational performance of Device 2 has been evaluated using the MNIST handwritten digit classification task.

[0214] refer to Figure 16 This shows examples of correct (top) and incorrect (bottom) classifications of handwritten digits.

[0215] The errors are primarily caused by obscure numbers that may be difficult for human readers to discern.

[0216] Also refer to Figure 17 The test accuracy score of 97.7% was determined using device 2 and through the MNIST handwritten digit classification task.

[0217] The 97.7% test accuracy score is based on Figure 17 The confusion matrix shown is calculated based on the classification of 10,000 images, of which 233 images were misclassified and 9,767 images were correctly classified.

[0218] Due to floating-point precision, the sum of the values ​​in each row and column shown is not always exactly equal to 1.000. For example, the sum of the values ​​in the row with actual label 0 is equal to 1.000, the sum of the values ​​in the row with actual label 1 is equal to 1.001, and the sum of the values ​​in the row with actual label 8 is equal to 0.999.

[0219] Also refer to Figures 18A to 18F This illustrates a comparison between optical physics calculations performed using a device provided with raw image data and logistic regression performed on the same raw image data.

[0220] exist Figure 18A The image shown is an exemplary one used in the MNIST handwritten digit classification task. Figure 18B The image shown in the panel is a planar view of an example of photonic network 11. The image shown in the panel is one of 10,000 images projected onto device 2 during the MNIST handwritten digit classification task. Figure 18C The image shows the spatially resolved spectrum read from the imaging spectrometer 5. The upper spectral profile indicates data from a defined spatial sub-region of medium 7, and corresponds to... Figure 18C The region within the bright rectangular frame. The lower spectral line profile indicates data from the entire medium 7, and corresponds to... Figure 18C The region within the dark square box. Logistic regression is performed by computer system 21 on the readouts from imaging spectrometer 5, and... Figure 18D In the calculation, an accuracy of 98.15% was obtained based on the confusion matrix shown. Figure 18E In this study, logistic regression was directly performed on the same raw image data used in the MNIST handwritten digit classification task, and a lower accuracy of 90.5% was calculated based on the confusion matrix shown. Figure 18D Compared to the values ​​in the confusion matrix, Figure 18E The values ​​in the confusion matrix shown are not normalized.

[0221] Therefore, device 2 can provide a test accuracy score of 98.15%, as measured by the MNIST handwritten digit classification task. This test accuracy score is an improvement over the test accuracy score achievable by performing logistic regression directly on the same original image data.

[0222] Specific examples of neuromorphic computing schemes including device 2 DNN As described above, device 2 can be used in fully trainable DNNs for inference. In particular, device 2 can be configured as one or more layers in a neural network, such as a CNN.

[0223] refer to Figure 19The diagram shows the first network layer 11 and the second network layer 12.

[0224] The first network layer 11 and the second network layer 12 are each modified versions of the first system 1, and each includes devices 21 and 22.

[0225] The first network layer 11 differs from the first system 1 only in that it does not include an imaging spectrometer 11. The second network layer 12 differs from the first system 1 in that the spatially structured illumination provided to the device 22 is the spatially structured response 61 of the first network layer 11. The spatially structured response 61 of the first network layer 11 and the spatially structured response 62 of the second network layer 12 are both detected by the imaging spectrometer (not shown) of the second network layer 12.

[0226] The first network layer 11 and the second network layer 12 are connected in series. However, this is not mandatory, and more network layers (not shown) may be included, and the network layers may be connected in feedback loops, wherein the spatially structured response 62 of the second network layer serves as the spatially structured illumination 41 of the first network layer 11. The network layers may be connected to form a feedforward neural network (such as a CNN), or they may be connected to form a recurrent neural network.

[0227] The first network layer 11 and the second network layer 12 do not necessarily include separate devices 21 and 22. Instead, the first network layer 11 may include a first photonic network 11, and the second network layer 12 may include a second photonic network 112. The first photonic network 11 and the second photonic network 112 are included in the same device 2 (in Figure 3 (Best example in China).

[0228] Combining parallel outputs (e.g., parallel outputs from the first network layer 11 and the second network layer 12) can help improve computational performance.

[0229] Fifth Optical Image Processing System 32—Independent Processor refer to Figure 20 The fifth optical image processing system 32 (hereinafter referred to as the "fifth system") is shown.

[0230] The fifth system 32 includes only one device 2. Therefore, the fifth system 32 is implemented as a standalone processor. The fifth system 32 can be an example of the first system 1, which is implemented as a standalone processor and includes only one device 2.

[0231] The device 2 of the fifth system 32 can be used as a reservoir or as a network layer in a deep neural network (DNN).

[0232] Sixth Optical Image Processing System 33—Preprocessor refer to Figure 21 The sixth optical image processing system 33 (hereinafter referred to as the "sixth system") is shown.

[0233] The sixth system 33 is a generalized version of the example DNN described above, which includes a first network layer 11 and a second network layer 12. In particular, the sixth system 33 includes two network layers connected in series, wherein only the first network layer includes the device 2.

[0234] In the sixth system 33, device 2 receives spatially structured lighting 4 as input and provides spatially structured response 6 as output. The second network layer 34 receives the spatially structured response as input and provides output 35.

[0235] In this way, device 2 is used as a preprocessor within system 33.

[0236] Seventh Optical Image Processing System 36—Post-processor refer to Figure 22 The seventh optical image processing system 36 (hereinafter referred to as the "seventh system") is shown.

[0237] The seventh system 36 is a modified version of the sixth system 33, in which device 2 is used as a post-processor instead of a pre-processor.

[0238] In the seventh system 36, the first network layer 37 receives input 38 and provides output 39. Device 2 receives spatially structured lighting 4 (which is provided based on output 39) as input and provides spatially structured response 6 as output.

[0239] In these ways, device 2 can be used as a pre- / post-processor as part of a larger computing system. As described above, combining device 2 as a network layer with another network layer (including memristor networks or nanomagnetic networks) can increase memory and thus allow device 2 to be used for video processing.

[0240] In System 6 33 and System 7 36, network layers do not need to be connected in series, and two or more network layers can be connected in parallel.

[0241] Edge / Feature Detection Device 2 can directly perform convolutional edge detection using its internal physical properties without regression or post-processing. Specifically, the first system 1 can be used simultaneously as a convolutional kernel layer and a classifier layer in a convolutional neural network. The kernel layer can serve as a standalone edge detection / feature extraction system, treating the optical power of each lasing mode as a separate feature extraction kernel. A first optical mode of one wavelength may respond more strongly in the presence of vertical edges, while a second optical mode of another wavelength may respond more strongly in the presence of horizontal edges. In this way, the mode power can be used to construct a feature map, regardless of whether the kernel is pre-trained / regressed. The edge / feature detection functionality can serve as a layer within a larger image processing and / or computational system, such as the initial kernel layer of a convolutional neural network. Alternatively, the edge / feature detection functionality can act as a standalone processor that extracts edges / features from an image and returns an edge / feature map as its output.

[0242] refer to Figures 23A to 23C The input image (letters ICL) is shown. Figure 23A ) and the extracted feature / edge maps provided by two different optical modes 10 ( Figure 23B and Figure 23C Examples of ).

[0243] Figure 23B and Figure 23C Each shows the corresponding Figure 23A The contrast of the edges or features of the input image.

[0244] Feature maps are generated using the following process: • Transfer the image ( Figure 23A It is divided into multiple different windows (e.g., 5×5 pixels).

[0245] • Each of these windows is illuminated onto the medium, and the response is recorded.

[0246] • Output the amplitude of a specific pattern of the window.

[0247] • Repeat the process for all possible 5×5 windows in the input image, thereby allowing the generation of a 'feature map', i.e., the amplitude of a specific pattern across each 5×5 input.

[0248] This is similar to how convolutional neural networks work, where the kernel is scanned across the entire image by a raster. Of course, the nature of the processing in this instance, which involves the interaction between spatially structured lighting 4 and medium 7, is significantly different from processing images using kernels executed by a conventional floating-point processor or graphics processing unit (GPU).

[0249] Figure 23D Additional experimental data for other examples of edge detection are shown, where regression was used to train which optical modes were used to train the edge map.

[0250] A single device 2 can provide many nuclei in parallel by multiplexing nuclei in the spectral frequency space via mode / nucleus correspondence.

[0251] Also refer to Figures 24A to 24D This demonstrates the use of four different optical modes 10 ( Figures 24A to 24B An example of detecting all four edges (one for each of the four edges).

[0252] Figures 24A to 24D Each shows the contrast of at least one edge of the simulated square.

[0253] Therefore, device 2 can be used for edge detection and feature detection.

[0254] application Device 2 can be used in a wide variety of computer vision applications, from autonomous vehicle driving to object detection.

[0255] In the first example, device 2 is used in an autonomous vehicle, where image data from an onboard camera is optically projected onto at least one device 2 pre-configured and trained for hazard detection. In this application, response time is critical, and the fast processing speed of device 2 relative to conventional software neural network processing can be particularly beneficial. An autonomous vehicle including device 2 can respond to an observed oncoming hazard by providing spectral readouts (which include one or more characteristic features indicating a warning), which act as triggers in the autonomous driving circuitry to activate evasive maneuvers, thereby protecting the safety of passengers and pedestrians. In a related example, device 2 can be used in another autonomous vehicle, such as an autonomous drone, autonomous aircraft, or autonomous vessel. By providing a processor, such as computer system 21, to determine changes in the spectral readouts of spectrometer 20, device 2 can be particularly well-suited for use in such remote, non-data center use cases.

[0256] In a second example, device 2 is included in a CCTV camera processing system that responds to the observation of certain objects (such as crowds or faces) by providing spectral readouts (which include one or more characteristic features indicating the observed object), similar to how a warning spectrum is provided in an autonomous vehicle. In related examples, device 2 may be included in another processing system, such as an industrial manufacturing control system, a medical imaging system, or a surgical control system, for example, for processing endoscopic camera images and / or video data.

[0257] Device 2 can also be used in a wide variety of other machine learning applications, where data used for non-linear processing can be mapped onto images.

[0258] Optical response modulation using electric field Figure 13 A device including an electrode 30 for applying a locally reconfigurable electric field to a medium 7 is schematically shown and described above.

[0259] The example of medium 7 described herein can be excited to support hyperspectral modes that are spatially distributed across multiple network waveguides and interfere with each other in a complex manner.

[0260] As described herein, even minute variations induced in the material properties (e.g., refractive index) of medium 7 or in parts / regions of medium 7 can lead to significant changes in the output signal. This opens up the prospect of dynamically reconfigurable medium 7 (e.g., photonic network 11).

[0261] For example, a small change in the refractive index of a segment of photonic network 11 corresponds to an increase or decrease in the optical path length seen by the optical modes intersecting the segment. This will impart an optical phase difference within this segment, thereby affecting interference with overlapping and / or closely adjacent optical modes.

[0262] Unlike conventional modulators or linear networks based on Mach-Zehnder interferometers, modulation of the optical response of medium 7 does not require a full π (180°) phase shift. Destructive interference is not required; only the complex interference pattern between overlapping optical modes excited by a given spatially structured illumination needs to be affected. Therefore, a small change in the complex refractive index should suffice. Various methods can be applied to medium 7 as described herein to generate such a local refractive index change (some of which have already been described above), including but not limited to: thermal, electric field (Poukes and Kerr effects), plasma dispersion via free carrier injection, or magnetic field modulation.

[0263] For example, also refer to Figure 25A and Figure 25B This illustrates a modulator with a first configuration 101 having electrodes 30 for electro-optic modulation of the medium 7. Figure 25A It is a labeled planar SEM micrograph of a medium 7 in the form of a photonic network 11 patterned from III-V semiconductor materials. Figure 25B This is a schematic projection of the first configuration 101.

[0264] A dielectric layer 102 is supported on a substrate 8. For example, the substrate 8 may be a Si carrier wafer, and the dielectric layer 102 may be a buried oxide layer. A photonic network 11 formed by network links 12 bonded (or closely spaced) at nodes 13 is supported on the dielectric layer 102. For example, the photonic network 11 may be patterned from InP.

[0265] Electrodes 30 are patterned in pairs, each pair surrounding at least a portion of the length of a network link 12. The number of network links 12 with corresponding electrode pairs can range from a single network link 12 to each network link. The inventors have observed that even changing a single network waveguide can have a considerable impact on the entire network 11. In practice, for a given task and photonic network 11, a balance needs to be struck between the complexity of fabricating and addressing the device 2 and the degree of modulation of the spatially structured response 6 achievable with the same spatially structured illumination 4 by changing the applied electric field.

[0266] Optionally, the gap 16 may be formed in the network link 12 surrounded by the electrode pair 30. The gap 16 may take the form of a physical gap or may be an inclusion of other materials / regions of different refractive indices. The gap 16 may introduce optical loss, whereby light is lost from the waveguide (such as the network link 12) connecting the gap 16. This loss provides additional nonlinearity, which can be used for processing / computation. Additionally or alternatively, the gap 16 may be arranged to embody a photonic crystal mirror. A combination of photonic crystal mirrors formed by appropriately arranged gaps 16 can be used to form a resonant structure that enhances the modulation efficiency of a device including the first configuration 101. The gap 16 is not necessary for the operation of the first configuration 101.

[0267] Potential difference applied between a pair of electrodes 30 V Local electric field generated across corresponding network link 12 E Potential difference V This is achieved via conductive traces (not shown) connected to each electrode 30 and allowing for individual addressing of each pair. For example, one electrode 30 in each pair could be connected to system ground, while the other electrode could be connected to a signal output. This is merely an example, and alternative addressing schemes may be used.

[0268] An electric field E applied across a network link can cause local Pockels effect and / or local Kerr effect.

[0269] The Pockels effect (or linear electro-optic effect) is related to the magnitude of the applied electric field. E Proportional to: (1) in Δn pockels It is the refractive index n Induced changes, n 0 is the unperturbed (zero electric field) refractive index, and r ij It is the third-order electro-optic tensor of the Pockels coefficient.

[0271] LiNbO3 is commonly used to obtain the Pockels effect in commercial high-speed electro-optic modulators. It is typically smaller in group III-V semiconductors (such as InP). r ij The magnitude of the component can be partially influenced by a higher refractive index. n Zero-value compensation. Therefore, equivalent modulation effective values ​​can be achieved in InP as in LiNbO3.

[0272] InP also offers the additional advantage of being easier to pattern, dope, and contact. Therefore, it is possible to utilize photonic design techniques to optimize the overlap between the optical mode 10 and the modulation region (between electrodes 30). The InP network link 12 can utilize the Kerr effect, which is typically small (although relatively large in InP) and depends on the secondary field | E | 2 Therefore, if the field is also concentrated (e.g., by using a photonic crystal (e.g., patterned using a gap 16)), the Kerr effect can also be used to modulate the refractive index.

[0273] like Figure 25A and Figure 25B As illustrated in the first configuration 101, the Pockels modulator is based on an electric field. Gating is sufficient and no contact with network link 12 is required, which significantly simplifies manufacturing. The modulator with the first configuration 101 can be embedded in an individual network link 12 to provide local phase modulation of optical mode 10 using said network link. The modulator with the first configuration 101 can be combined with photonic crystal designs, for example, using gap 16 (or other inclusions) to pattern mirrors or resonators.

[0274] By applying a bias voltage across the selected network link 12, the refractive index n within the network link changes (at least in the electric field). E (In the affected region). Therefore, the interference pattern excited by the given spatially structured illumination 4 is modified, and thus the corresponding spatially structured response 6 (e.g., hyperspectral lasing mode) is also modified. Thus, a modulator arranged, for example, according to the first configuration 101, can act as a reconfigurable physical weight within the medium 7 used for physically implementing neuromorphic computation. For example, in a random lasing network.

[0275] The field effect is inherently fast, with operable frequencies up to GHz or higher. Even without direct contact with InP (which would promote carrier recombination), speeds in the MHz range should be possible. This refers to the speed at which modulation applied to medium 7 changes from one state to another—the speed at which medium 7 processes spatially structured illumination 4 into spatially structured response 6 does not deviate from the inherently high speed described above.

[0276] Also refer to Figures 26A to 26GAn exemplary method for manufacturing a modulator having a first configuration 101 is schematically illustrated.

[0277] Special Reference Figure 26A A uniform layer 103 is deposited over a dielectric layer 102 supported by a substrate 8. The substrate 8 can be any material discussed herein, such as, for example, silicon (Si). The dielectric layer 102 can be any insulating material, such as, for example, silicon dioxide (SiO2), with a thickness between 0.5 μm and 2 μm. The uniform layer 103 can be formed from any material suitable for patterning to provide the dielectric 7 (such as, for example, a III-V semiconductor). Examples include indium phosphide (InP), gallium arsenide (GaAs), etc. Generally, the uniform layer 103 can be formed from any material described herein for providing the dielectric 7 and patternable by photoresist etching. Preferably, the material forming the uniform layer 103 should simultaneously satisfy: • It can serve as an active gain material, i.e., it has a direct bandgap; and • It has a Pockels coefficient or a Kerr coefficient. (For example, silicon would be unsuitable).

[0278] The uniform layer 103 may be pre-formed on top of the dielectric layer 102 before the start of the manufacturing process. In other words, the starting point may be a laminate formed by the substrate 8, the dielectric layer 102, and the uniform layer 103. One or more additional layers (not shown) may be disposed between any pair of illustrated layers for a variety of purposes, including but not limited to: diffusion blocking, improving lattice matching, creating quantum wells, improving adhesion, etc.

[0279] A first photoresist layer 104 is deposited over a uniform layer 103. Commonly used photoresist materials include polymers or oxides (such as hydrogen silsesquioxane (HSQ)). The first photoresist layer 104 is patterned into a desired shape having a photonic network 11.

[0280] Special Reference Figure 26B The uniform layer 103 is etched to form a photonic network 11 (formed by network links 12 and nodes 13) with the same contour as the first photoresist layer 104. Although in Figure 26B The diagram shows a complete etching through the uniform layer 103, but in some examples, the etching does not need to be deep enough to expose the dielectric layer 102: the network can be patterned on top of the remaining uniform layer (not shown).

[0281] Special Reference Figure 26C The first photoresist layer 104 is removed, and a second dielectric layer 105 is deposited over the network 11 and in the gaps between the networks 11. In other words, the second dielectric layer 105 can be described as a “blanket” layer. The second dielectric layer 105 can be an oxide layer, such as, for example, silicon dioxide.

[0282] Special Reference Figure 26D A second photoresist layer 106 is deposited over the second dielectric layer 105. The second photoresist layer 106 is patterned as the inverse (or negative) phase of the desired pattern of the electrodes 30 and any connecting traces used (not shown). Portions of network 11, such as some network links 12 and / or nodes 13 that are not intended to have adjacent electrodes 30 for modulation, may be completely covered by the second photoresist layer 106 (see, for example, [link to relevant documentation]). Figure 26D (Part of network 11 shown on the right).

[0283] Special Reference Figure 26E Electrodes 30 are deposited within the gaps of the second photoresist layer 106 using a stripping process. The electrode material deposited above the second photoresist layer 106 is not shown because it will be removed along with the second photoresist layer 106. Electrodes 30 can be formed from any suitable conductive material (such as, for example, gold (Au), aluminum (Al), indium tin oxide (ITO), etc.).

[0284] Special Reference Figure 26F The second photoresist layer 106 is then removed to complete the stripping process, leaving the electrode 30 for modulation according to the first configuration 101.

[0285] Special Reference Figure 26G The network 11 and the electrode 30 may be covered by a third dielectric layer 107, which provides electrical insulation and protects the network (and optionally also serves for device planarization).

[0286] The top metallization layer 108 is connected to the electrode 30 via a via 109 formed through the third dielectric layer 107, thereby allowing addressing of the electrode 30. The third dielectric layer 107 may be an oxide layer, such as, for example, silicon dioxide. Figure 26G The diagram only shows contact with one of the electrodes 30. Contact with the other electrode 30 is located outside the illustrated cross-sectional plane.

[0287] In other examples, some or all of the connections to electrode 30 may be made using conductive traces (not shown) supported on the second dielectric layer 105. In yet another example, some or all of the connections to the electrode may be made using vias (not shown) extending from the underside of substrate 8 (in other words, from the surface of substrate 8 opposite to dielectric layer 102) to network 11. This helps reduce reflections from the structure of contact electrode 30 to spatially structured lighting.

[0288] The materials used for the second dielectric layer 105 and the third dielectric layer 107 should be transparent to the wavelength ranges corresponding to the spatially structured illumination 4 and the spatially structured response 6 (these two wavelength ranges may be different and do not need to overlap).

[0289] In the first configuration 101, electrode 30 is isolated from the network 11 material by a second dielectric layer 105. Electrode 30 surrounding a given network link 12 forms a capacitor, thereby applying a substantially uniform electric field across the network link 12. In other examples, the second dielectric layer 105 may be omitted, and an electrical contact may be formed between electrode 30 and the network 11 material. In this way, electrode 30 surrounding a given network link 12 can be injected with charge carriers, thus opening up alternative mechanisms for modulation of the optical response. In such examples, the electrode 30 material should be selected to provide an appropriate type of contact, such as an ohmic contact, a rectifier contact, etc., and the electrodes 30 forming a single pair may be formed from different materials. Different materials for the electrodes 30 are also possible in the first configuration 101, but this would only complicate manufacturing.

[0290] Modulator with second configuration Also refer to Figures 27A to 27E The fabrication and structure of a modulator having a second configuration 110 of electrodes 30 for electro-optic modulation of the medium 7 are shown. Figures 27A to 27D This is a schematic cross-sectional view illustrating the manufacturing stage of the second configuration 110, in which Figure 27D A second configuration 110 of the manufacture of the electrode 30 for electro-optic modulation is shown. Figure 27E This is a schematic plan view of the second configuration 110 of the electrode 30 used for electro-optic modulation. Figure 27D Corresponding to along Figure 27E The cross section shown is taken by the line marked B-B'.

[0291] The materials used for the substrate 8, dielectric layer 102, network 11, and electrode 30 are the same as those described with respect to the first configuration 101.

[0292] Special Reference Figure 27A In the same manner as described for the first configuration 101 (see also) Figure 26A (and corresponding description), a uniform layer 103 is deposited on top of a dielectric layer 102 supported by a substrate 8.

[0293] Special Reference Figure 27B Electrode 30 is deposited above uniform layer 103. Electrode 30 can be patterned using any suitable technique (including, but not limited to, stripping and selective etching methods).

[0294] Special Reference Figure 26C A first photoresist layer 104 is deposited over the uniform layer 103 and the electrode 30. In the same manner as the first configuration 101, the first photoresist layer 104 is patterned into a desired shape having a photonic network 11 (but the specific topology of the photonic network 11 may be different).

[0295] Special Reference Figure 26D and Figure 27EThe uniform layer 103 is etched to form the photonic network 11. After etching, the first photoresist layer 104 is removed, leaving a second configuration 110 of the fabrication of the electrode 30. Similar to the first configuration 101, although the etching is shown to extend completely through the uniform layer 103, this is not always necessary, and in other examples a residual uniform layer (not shown) thinner than the network 11 may be left.

[0296] Network 11 is etched such that the modulation region 111 of the support electrode 30 is left wider than the rest of the corresponding network link 12. For example, as for... Figure 27E The first network link 121 is shown. Modulation region 111 is a part of the first network link 121.

[0297] As described above, it is not necessary to form a modulation electrode 30 for each network link 12. For example, as also shown in Figure 27E The second network links to the fourth network links 122, 123, 124 (and coupled at node 13) may not include the modulation electrode 30. Similarly, some or all of the second network links to the fourth network links 122, 123, 124 may include the modulation electrode, as long as it is outside the illustrated portion of network 11.

[0298] In the same manner as in the first configuration 101, in the second configuration 110, the electrode 30 may alternatively be configured for charge injection.

[0299] Modulator with a third configuration The first configuration 101 and the second configuration 110 described above require prior knowledge of the location of the network links 12 and which network links 12 will be surrounded by the electrodes 30.

[0300] However, the configuration of the electrodes 30 used for electro-optic modulation is not limited to electrode pairs located on both sides of the network link 12.

[0301] Also refer to Figures 28A to 28E The fabrication and structure of a modulator having a third configuration 112 of electrodes 30 for electro-optic modulation of the medium 7 are shown. Also referenced is... Figures 29A to 29C The corresponding figures are shown respectively. Figure 28A , Figure 28C and Figure 28E A schematic floor plan.

[0302] The materials used for the substrate 8, dielectric layer 102, network 11, and electrodes 30, etc., may be the same as those described with respect to the first configuration 101. However, since at least one set of electrodes 30 is preferably transparent, a wider range of material choices may be allowed.

[0303] Special Reference Figure 28A and Figure 29AThe first electrode 301 is deposited on top of the dielectric layer 102 supported by the substrate 8. For visual clarity, Figure 29A The shape of the first electrode 301 is only shown in the example.

[0304] The first electrode 301 is in the first direction ( Figure 29A shown x Extending parallel to each other on the axis, and in the second perpendicular direction ( Figure 29A shown y The first electrode 301 can be patterned using any suitable method, including but not limited to stripping, etching, mask sputtering, transfer, etc.

[0305] Special Reference Figure 28B Additional dielectric material is deposited to increase the thickness of the dielectric layer 102, covering the first electrode 301 and planarizing the top surface. This allows the first electrode 301 to be embedded within the dielectric layer 102.

[0306] To minimize manufacturing complexity, the added dielectric material is preferably the same as the original dielectric layer 102, such as... Figure 28B As illustrated. However, since its primary function is for planarization of the top surface, it can also be used as an electrical insulator, although it is important that dielectric 7 is conductive (e.g., an indium-based III-V material) and charge carrier injection is not desired. Different dielectric materials can be used equally to form a multilayer dielectric (not shown) with electrodes 30 embedded at the interface (not shown).

[0307] Special Reference Figure 28C and Figure 29B The dielectric 7 is deposited above the dielectric layer 102 in which the first electrode 301 is embedded.

[0308] Any method described herein for producing the medium can be used. For example, the medium 7 can take the form of a photonic network 11 illustrated in Figures 28 and 29. The photonic network 11 can be produced as described herein, for example, by photolithography, by electrospinning and post-annealing dye mixing of polymer nanofibers or semiconductor nanowires, or by any other means capable of producing the medium 7 described herein.

[0309] Special Reference Figure 28D A second dielectric layer 105 is deposited over the photonic network 11. Unlike the thin second dielectric layer 105 of the first configuration 101, in the third configuration, the second dielectric layer 105 may be thicker and is preferably at least partially planarized.

[0310] Special Reference Figure 28E and Figure 29C The second electrode 301 is deposited above the second dielectric layer 105. Figure 29CIn order to clearly show the relative positions of electrodes 301, 302 and network 11, the substrate 8, dielectric layer 102 and second dielectric layer 105 are omitted.

[0311] The second electrode 302 is in the second direction ( Figure 29C shown y Extending parallel to each other on the axis, and in the first direction ( Figure 29C shown y The second electrode 302 can be patterned using any suitable method, including but not limited to stripping, etching, mask sputtering, etc.

[0312] At least one of the multiple sets of electrodes 301, 302 should be transparent. For example, the second electrode 302 should be transparent when illuminated from above the substrate 8. The transparent electrodes 30, 301, 302 can be manufactured using, for example, transparent conductors such as ITO, thin conductive materials similar to graphene, etc.

[0313] In this way, the first electrode 301 and the vertical second electrode 302 form a grid. Local modulation of the optical response of the medium 7 can be achieved by connecting one or more first electrodes 301 and one or more second electrodes 302 to different potentials.

[0314] Additionally, in the third configuration 112, the medium 7 is not limited to the photonic network 11 illustrated in Figures 28 and 29. The medium 7 may take the form of a uniform film of a material having a complex refractive index, which can be modulated using an applied electric field generated in a patterned manner by electrodes 301 and 302.

[0315] A more complex grid arrangement of electrodes 30 can be considered a variation of the third configuration 112. For example, the upper electrode layer can be a uniformly transparent anti-electrode connected to system ground or any other fixed potential. The lower electrode layer can be controlled by a transistor layer (e.g., a thin-film transistor TFT layer) such that each lower electrode forms a storage capacitor with the upper electrode as a common terminal. Similar circuitry and addressing schemes known from TFT liquid crystal displays can be used to set each lower electrode to a corresponding potential, thereby effectively allowing electro-optic modulation as an "image" to be applied immediately before input spatially structured illumination.

[0316] In the same manner as in the first configuration 101, in the third configuration 112, the electrode 30 may alternatively be configured for charge injection.

[0317] Simulation of refractive index modulation Also refer to Figure 30A A schematic diagram of a low-density photonic network is shown, featuring network links 12 with an average length of 7 μm. Also referenced... Figure 30BThe diagram illustrates the variation in simulated lasing spectra for simulated refractive index shifts, with different network links 12 exhibiting refractive index shifts of varying magnitudes. The maximum refractive index shift is used. n = 0.05, where the relative refractive index offset applied to each network link 12 n via the network link 12 relative to the y-axis (e.g.) Figure 30A Angles in the vertical direction (as shown in the example) θ To determine, and use for this angle θ Calculated cos( θ) To obtain a scaling factor between 0 and 1. Multiply this scaling factor by the maximum refractive index offset of 0.05 to obtain the refractive index offset for each network link 12. n These refractive index shifts can be achieved via any of the modulation methods described herein (including electric and magnetic fields), where n Modulation of 0.05 is experimentally feasible.

[0318] Simulations were performed by solving Maxwell's equations for light passing through network 11 under uniformly pumped illumination (i.e., the entire network is uniformly illuminated). Solving the equations identified the lasing mode frequencies and amplitudes, thus generating the lasing spectrum of the network. InP was simulated as the network material.

[0319] Also refer to Figure 30B For the refractive index shift to its maximum n The first case 113, with a refractive index of 0.05, and the second case 114, using a uniform (unmodified) refractive index for each network link 12, present the results according to... Figure 30A The simulated lasing spectrum of network 11 is shown. A refractive index shift can be observed in the simulation. n This is sufficient to generate a clear and distinguishable shift between lasing spectra 113 and 114.

[0320] Also refer to Figure 31A A schematic diagram of a high-density photonic network is shown, featuring network links 12 with an average length of 2 μm. Also referenced... Figure 31B The diagram illustrates the variation in the simulated lasing spectrum for the simulated refractive index shift, with different network links 12 exhibiting refractive index shifts of varying magnitudes. The maximum refractive index shift is used again. n = 0.05, where the relative refractive index offset applied to each network link 12 n Such as about Figure 30A and Figure 30B The description determines this.

[0321] Also refer to Figure 31B For the refractive index shift to its maximum n The first case 113, with a refractive index of 0.05, and the second case 114, using a uniform (unmodified) refractive index for each network link 12, present the results according to... Figure 31A The simulated lasing spectrum of network 11 is shown.

[0322] By comparison Figure 30A and Figure 30B and Figure 31A and Figure 31B It can be observed that the two networks 11 with different link densities demonstrate the effectiveness of using network topology (such as...) Figure 30A The lower density (with an average waveguide length of 7 μm and the higher density (with an average waveguide length of 2 μm) shown can exert control over network modulation. For the lower density network ( Figure 30A ), refractive index n modulation Δn This results in a shift in mode frequency. However, for higher density networks ( Figure 31A ), refractive index n modulation Δn This generates more complex spectral variations, including the emergence of new lasing modes. This demonstrates that the response of network 12 to modulation can be controlled and influenced through network topology / design.

[0323] Modulation simulation using structured photomasks Simulations were performed to demonstrate modulation using structured photomasks. These simulations were conducted using a 'digital twin' of experimental network 11, thereby training the neural network on experimental data to model the network 11 response under different spatially structured illumination 4.

[0324] To improve classification accuracy, a mask illumination pattern is generated, where a genetic algorithm training process is used to determine the mask design. Mask illumination ( Figure 33B (As shown) has multiple mask pixels 120 within the illumination area 115. Figure 33B The plurality of mask pixels are set to the maximum pump illumination intensity. In the following example, instead of using a second light source to supply the structured beam, mask pixels 120 are superimposed on the image / data input of the spatial structured illumination 4 input to the network.

[0325] Also refer to Figures 32A to 32G This illustrates the baseline case without a structured photomask. It also references... Figures 33A to 33G An example is shown using a structured photomask.

[0326] Special Reference Figure 32A An example of an input image used to pattern spatial structured lighting 4 is shown. The image shown is inverted for application to network 12. In other words, when projected onto the network, white areas of the image correspond to low pumped illumination intensity, and black areas of the image correspond to high pumped illumination intensity.

[0327] Special Reference Figure 32B The illuminated area 115 is shown as a white dashed outline superimposed on the experimental SEM image of the network 11 topology used to obtain experimental data for training digital twins. Figure 32B The coloring of network link 12 shown corresponds to a single mode of | E | 2 The square of the electric field strength, calculated by solving Maxwell's equations, is superimposed on the experimental scanning electron microscope image of network 11, showing an example of the shape / size of one optical mode in optical mode 10. The optical mode 10 shown includes a relatively low-intensity region 116 and a relatively high-intensity region 117.

[0328] Special Reference Figure 32C This shows that for Figure 32B Network 11 shown responds to the corresponding Figure 32A The image shown is spatially structured illumination 4, a simulated lasing spectrum in the maskless case. The "spatial" axis corresponds to... Figure 32B The vertical direction. Figure 32C The coloring corresponds to the intensity (AU) of light detected at the corresponding wavelength and spatial (vertical) position. The response includes a relatively low-intensity region "L" and a relatively high-intensity region "H".

[0329] Special Reference Figures 32D to 32G It shows how to... Figure 32C The simulated lasing spectrum was calculated in the same manner as the spectrum shown, to compare the responses represented by different figures. In this case, Figure 32D and Figure 32E The lasing spectra generated in response to different representations of the number "2" were compared. Figure 32D yes Figure 33C (The repetition is used for side-by-side comparisons), and Figure 32F and Figure 32G The lasing spectra generated in response to different representations of the digit "6" were compared. Figures 32D to 32G In all cases shown, no mask was used, and the image providing spatially structured illumination 4 was reproduced above the corresponding spectrum.

[0330] For comparison, Figures 33A to 33GSimulated results are presented for the same network 11 and input image, with spatially structured illumination 4 superimposed on a pattern of mask pixels 120 trained to improve classification accuracy. Each mask pixel 120 corresponds to the maximum pump illumination intensity. (Special Reference) Figure 33B The illustrated mask pixel pattern is used to generate Figures 33C to 33G The pattern of the simulated lasing spectrum is presented in the image.

[0331] Compare Figure 32C (without mask) and Figure 33C (With a mask), a more focused spatial-spectral response can be observed. Furthermore, comparisons were made separately. Figures 32D to 32G (without mask) and Figures 33D to 33G (With masking), it can be observed that the response using the trained mask is not only more focused than the response without masking, but also shows a higher degree of similarity in responses to different representations of the same number.

[0332] There is a mask ( Figures 33C to 33G ) and maskless ( Figures 32C to 32G The simulated spectra of ) indicate that modulation of the optical response of medium 7 can be used to reduce or even completely eliminate the need for a software regression step to explain the spatially structured response.

[0333] Unbound by theory, it is believed that the pattern of mask pixels 120 guides the lasing mode to a specific spectral region, thus allowing the region with the highest amplitude to be used as the computational output. It can be observed that the effect of the mask on the response of network 11 is similar to previous findings. Figures 30A to 31B The described effect of electrical gating is somewhat similar. Similar effects are expected for other modulation techniques described above (including magnetic modulation and thermal modulation).

[0334] In this way, by using various methods to modulate the optical response, it is possible to reconfigure / reprogram the intrinsic nonlinear physical properties of medium 7 (such as network 11), thereby changing the refractive index. n Changes in the competitive landscape, such as altering the competitive dynamics.

[0335] Despite about Figures 32A to 33G The simulation results shown depict mask pixel 120 set to maximum intensity, but this is not required, and patterns with mask pixel 120 at zero intensity can be trained in other examples. Similarly, the illumination intensity of mask pixel 120 is not limited to a binary pattern, and the mask can utilize grayscale intensity levels (between zero and the maximum value). In some examples, the mask can be trained in the form of an image.

[0336] Revise It will be understood that various modifications can be made to the embodiments described above. Such modifications may involve equivalents and other features already known in the fields of optical image processing and neuromorphic computing hardware, and may replace or supplement the features already described herein. A feature of one embodiment may be replaced or supplemented by a feature of another embodiment.

[0337] Although imaging spectrometer 5 has been described, in some examples, a spectrometer that does not require imaging / spatial selectivity can be used. For example, a spectrometer with only frequency resolution can be used to detect spatially structured responses from the entire medium 7 or a single region thereof (predefined by the selection of the input coupling optics).

[0338] Although an imaging spectrometer 5 has been described, in some examples, a spectrometer is not required and a camera can be used instead. For example, when the spatial distribution of the spatially structured response 6 is sufficient to encode the output of the computational task. When using a red-green-blue (RGB) camera, the intensity differences between color channels combined with the spatial distribution of the spatially structured response 6 can provide the output of the computational task.

[0339] Although claims have been drafted in this application for specific combinations of features, it should be understood that the scope of disclosure of this invention also includes any novel feature or combination of novel features or any generalization thereof explicitly or implicitly disclosed herein, whether or not it is identical to the invention currently claimed in any claim, and whether or not it alleviates any or all of the same technical problems as this invention. The applicant hereby notifies that during the filing of this application or any other application derived therefrom, new claims may be formulated as these features and / or combinations of these features.

Claims

1. An apparatus for use in optical image processing, the apparatus comprising: • A medium capable of supporting overlapping optical modes, such that in response to spatially structured illumination, a set of overlapping optical modes in the medium is non-uniformly excited and outputs a spatially structured response.

2. The apparatus of claim 1, wherein the medium comprises: • Optical gain medium.

3. The apparatus of claim 1 or 2, wherein the medium is spatially non-uniform.

4. The apparatus of any one of claims 1 to 3, wherein the medium is configured as a panel.

5. The apparatus of any one of claims 1 to 4, wherein the medium comprises a first photonic network.

6. The apparatus of claim 5, wherein the medium comprises a second photonic network spatially separated from the first photonic network.

7. The apparatus of any one of claims 1 to 6, wherein the medium comprises a photonic waveguide.

8. The apparatus of any one of claims 1 to 7, wherein the medium comprises a low-dimensional structure.

9. The apparatus of claim 8, wherein the low-dimensional structure comprises a quantum well.

10. The apparatus of any one of claims 1 to 9, wherein the medium comprises a semiconductor material.

11. The apparatus of claim 10, wherein the semiconductor material is an inorganic semiconductor material.

12. The apparatus of claim 10 or 11, wherein the semiconductor material is a III-V group semiconductor material.

13. The apparatus of claim 10, wherein the semiconductor material is an organic semiconductor material.

14. The apparatus of any one of claims 1 to 13, wherein it is configured to modulate the optical response of the medium.

15. The apparatus of any one of claims 1 to 14, comprising a first set of electrodes configured to apply a locally reconfigurable electric field to the medium.

16. The apparatus of any one of claims 1 to 15, further comprising a second set of electrodes configured to inject charge carriers into the medium.

17. The apparatus of any one of claims 1 to 16, comprising a second light source configured to project a mask in the form of a spatially structured beam onto the medium.

18. The apparatus of any one of claims 1 to 16, wherein the apparatus is configured to overlay the mask with the spatially structured lighting.

19. The apparatus of any one of claims 1 to 18, comprising a magnetic element arranged to provide a locally reconfigurable magnetic field across the medium.

20. The apparatus of any one of claims 1 to 19, comprising one or more heaters configured to change the temperature of the medium.

21. An optical image processing system, comprising: • The apparatus as described in any one of claims 1 to 20; and • Imaging spectrometer, which is configured to detect spatially structured responses.

22. The optical image processing system of claim 21, wherein the imaging spectrometer is configured to perform readout over a defined spatial sub-region of the medium.

23. The optical image processing system of claim 21 or 22, further comprising: • Neutral density filter, which is configured to maintain the pump power of spatially structured illumination.

24. The optical image processing system of any one of claims 21 to 23, further comprising: • A digital micromirror device configured to provide the space-structured illumination.

25. The optical image processing system of any one of claims 21 to 23, further comprising: • A spatial light modulator, which is configured to provide the spatial structured illumination.

26. The optical image processing system of any one of claims 21 to 25, wherein the optical image processing system is configured as a non-trainable store for data processing.

27. The optical image processing system of any one of claims 21 to 26, wherein the optical image processing system is configured as a fully trainable deep neural network for inference.

28. A computer vision product, comprising: • An optical image processing system as described in any one of claims 21 to 27; and • Processor, configured to determine variations in the spatially resolved spectrum provided by the spatially resolved spectrometer.

29. The computer vision product of claim 28, wherein the computer vision product is an autonomous vehicle or a system for object detection and classification.

30. A method of operating the apparatus as claimed in any one of claims 1 to 20 or the optical image processing system as claimed in any one of claims 21 to 27, the method comprising: Provide the spatial structured lighting to the device; as well as Detect the spatial structured response.

31. The method of claim 30, further comprising modulating the optical response of the medium.

32. A method for training an apparatus as claimed in any one of claims 1 to 20 or an optical image processing system as claimed in any one of claims 21 to 27, the method comprising: Determine the loss function of the device. Determine whether the loss function has reached a stable minimum, and Modulate the optical response in the device.

33. The method of claim 32, wherein modulating the optical response in the device comprises adjusting the field applied to the medium.

Images