Optical image processing

Abstract

A device (2) for use in optical image processing is disclosed. The device (2) comprises a medium (7) capable of supporting overlapping optical modes (10) such that, in response to spatially-structured illumination (4), a set of overlapping optical modes (10) in the medium (7) are non-uniformly excited and output a spatially-structured response (6).

Description

[0001] Optical image processing

[0002] Field of the Invention

[0003] The present invention relates to optical image processing.

[0004] Background

[0005] A promising approach to machine learning is to use physical neuromorphic computing schemes. In physical neuromorphic computing schemes, hardware architectures are used to emulate the function of neurons and / or neural networks.

[0006] Various approaches have been made to implement physical neuromorphic computing schemes; however, these can have one or more drawbacks. Memristor arrays can suffer from poor dimensionality, lack a capability for image input, and rapidly degrade physically. Optical delay-line systems can involve reels of kilometres of optical cable and thereby be incompatible with miniaturisation and being employed in remote non- data-centre use cases such as in self-driving cars. Vertical emitting cavity lasers ("VCSELs") can lack large input dimensionality capability and not allow for mode separation.

[0007] Moreover, the data input in many existing neuromorphic computing schemes is typically one-dimensional or few-dimensional. This can be a drawback because many modern machine learning problems, such as those relating to computer vision, concern high-dimensional data.

[0008] Physical neuromorphic computing schemes have been implemented in a range of all- optical and optoelectronic systems.

[0009] T. Zhou et al. '. "Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit", Nature Photonics, volume 15, page 367 (2021) describes an optoelectronic fused computing architecture based on the diffraction of light.

[0010] D. Pierangeli, G. Marcucci, and C. Conti: "Photonic extreme learning machine by free- space optical propagation", Photonics Research, volume 9, page 1446 (2021) describes a neuromorphic photonic scheme which can be implemented by using an optical encoder and coherent wave propagation in free space. US 2021 / 0285819 Al describes a system for machine learning using optical data. The system includes a diffusive physical medium for scattering light from an optical source coupled to an adjustable spatial light modulator. Reference is also made to M. Matuszewski et al. '. "Energy-Efficient Neural Network Inference with Microcavity Exciton Polaritons", Physical Review Applied, volume 16, page 024045 (2021), which proposes all-optical neural networks having an energy efficiency of energy efficiency of 4x io16synaptic operations per second per watt and a performance density 1016synaptic operations per second per millimetre squared.

[0011] Software- based approaches can also have one or more drawbacks. A part of software machine learning is the so-called "non-linear activation" step, where data passed through the software neural network undergoes some non-linear transform to add processing power, typically a rectified linear activation function (hereinafter referred to as "ReLU") or a sigmoid transform. Manually engineering software networks so that these transforms can occur in a parallel fashion can have a substantial cost.

[0012] Summary

[0013] According to a first aspect of the present invention there is provided a device for use in optical image processing. The device comprises a medium capable of supporting overlapping optical modes such that, in response to spatially-structured illumination, a set of overlapping optical modes in the medium are non-uniformly excited and output a spatially-structured response.

[0014] This arrangement can allow the device to process data included in the spatially- structured illumination in such a way that the features of the data are conserved. This is because there is no need pre-process the data, for example by pixel rastering (in other words, ID input of pixels as a time series) or by image transforms, before providing it to the device for processing.

[0015] Different optical modes of the set of overlapping optical modes excited in response to the spatially-structured illumination may have one or more of different frequencies; different spatial mode shapes; and different spatial locations within the medium.

[0016] The medium may be disposed on a surface. The surface may be a surface of the substrate. The surface may be flat. The surface may be non-flat, for example, it may be patterned, curved, or have a non-uniform surface.

[0017] The device may include the substrate. The medium may be disposed in a matrix of another material.

[0018] The substrate may be rigid. The substrate may be flexible. The substrate may consist of or consist predominantly of a plastic material. The substrate may be planar. The substrate may be non-planar, for example curved.

[0019] The substrate may include, consist of, or consist predominantly of, or may include a layer (for example, a surface layer) which includes, consists of, or consists predominantly of a dielectric or a semiconductor, such as indium tin oxide (ITO), silicon dioxide (SiOz), hafnium dioxide (HfOz), titanium dioxide (TiOz), zinc oxide (ZnO), magnesium fluoride (MgFz), or diamond. The substrate may be an oxide material. The substrate may include, consist of, or consist predominantly of a semiconductor material such as silicon (Si). The substrate may include a transparent material and / or a layer (for example, a surface layer) of transparent material, such as a layer of a dielectric or a semiconductor, such as ITO, SiOz, HfOz, TiOz, ZnO, MgFz, or diamond. The medium may comprise an optical gain medium. The optical gain medium may be embedded in the medium. Alternatively, the medium may be an optical gain medium.

[0020] The optical gain medium may be a solid-state gain medium. The solid state-gain medium may comprise semiconductor material embedded in a polymer material or dye-molecules embedded in a polymer material. Alternatively, the optical gain medium may be a liquid-phase dye medium.

[0021] The medium may have an optical gain which is less than 1, equal to 1, or greater than 1. In other words, the device may be for amplification of incident light and / or for attenuation of incident light.

[0022] The medium may be spatially non-uniform. The medium may be spatially non-uniform along at least one in-plane direction. The medium may have in-plane patterning. The medium may be spatially non-uniform along an out-of-plane direction. For example, the medium may be a multi-layered heterostructure. The medium may include voids. The medium may include suspended structures. The medium may be 3D patterned.

[0023] The medium may be spatially uniform. The medium may be spatially uniform along at least one in-plane direction. The medium may have a complex refractive index that, in response to a mask taking the form of a spatially-structured light beam being projected onto the medium, is spatially structured. The mask may be provided by a second light source that is different to a light source configured to provide the spatially-structured illumination. The mask may be superposed with the spatially- structured illumination. In other words, a single light source may provide both the spatially-structured illumination and the mask.

[0024] The medium may be configured as a panel. The medium may be spatially non-uniform across the panel. The medium may be spatially uniform across the panel. The medium may be configured as a screen. The medium may be spatially non-uniform across the screen. The medium may be spatially uniform across the screen.

[0025] The device may be configured for operation at a design wavelength. The design wavelength may be between 10 nm and 400 nm (or "ultraviolet light"). The design wavelength may be between 400 nm and 780 nm (or "visible light"). The design wavelength may be between 780 nm and 1.4 pm (or 'TR-A"). The design wavelength may be between 1.4 pm and 3 pm (or"IR-B"). The design wavelength may be between 3 pm and 1 mm (or 'TR-C").

[0026] The design wavelength refers to the wavelength of the spatially-structured illumination. The design wavelength may correspond to an energy which is less than or equal to a bandgap energy of a material providing the medium.

[0027] The medium may comprise a first photonic network, the first photonic network comprising network links that are radiatively coupled at nodes. Expressed differently, the medium may comprise a first photonic material and have a first topology in which network links are radiatively coupled at nodes.

[0028] A photonic network may include, or take the form of a network of optical waveguides.

[0029] At least a first sub-set of the network links of the first photonic network may be interconnected at the nodes of the first photonic network. At least a second sub-set of the network links of the first photonic network may be separated by a distance less than the design wavelength at the nodes of the first photonic network.

[0030] The first photonic network may comprise at least two closed loop paths. The at least two closed loop paths may be for transmitting light having the design wavelength.

[0031] The medium may comprise a second photonic network, the second photonic network comprising network links that are radiatively coupled at nodes, the second photonic network being spatially separated from the first photonic network. The second photonic network may comprise a second photonic material and have a second topology in which network links are radiatively coupled at nodes.

[0032] At least a first sub-set of the network links of the second photonic network may be interconnected at the nodes of the second photonic network. At least a second sub-set of the network links of the second photonic network may be separated by a distance less than the design wavelength at the nodes of the second photonic network.

[0033] The second photonic network may comprise at least two closed loop paths. The at least two closed loop paths may be for transmitting light having the design wavelength. The first photonic material may be different to the second photonic material. The first photonic material may be the same as the second photonic material. The first topology may be different to the second topology. The first topology may be the same as to the second topology.

[0034] The medium may comprise three or more spatially separated photonic networks.

[0035] The medium may comprise photonic waveguides. The network links may comprise the photonic waveguides. The photonic waveguides may be embedded in the network links. The network links may be photonic waveguides. The photonic waveguides may be active photonic waveguides.

[0036] The term "active" in relation to a waveguide may refer to that waveguide being capable to provide gain, amplifying the light.

[0037] The network links and / or photonic waveguides may have at least one dimension less than 1000 nm. The at least one dimension may include or be a width or a diameter. The at least one dimension may include or be a thickness.

[0038] The network links and / or photonic waveguides may have at least one dimension greater than or equal to a tenth of the design wavelength. The network links and / or photonic waveguides may have at least one dimension less than or equal to 10 times the design wavelength. The network links and / or photonic waveguides may have at least one dimension less than or equal to 100 times the design wavelength.

[0039] The network links and / or photonic waveguides may have a distribution of lengths. The distribution of lengths may be centred around an average length value. In the case that the design wavelength is between 400 nm and 780 nm (or "visible light"), and / or in the case that the design wavelength is between 780 nm and 1.4 pm (or "IR-A"), the average length value may be between 3 pm and 20 pm. The average length value may be less than 100 pm. The average length value may be less than 1 mm.

[0040] The degree of nodes in the photonic network may be between 3 and 6.

[0041] The photonic waveguides may have a distribution of diameters. The distribution of diameters may be centred around an average diameter value. The photonic waveguides may have an average diameter value between 1 pm and 10 pm and the average length value may be between 100 pm and 500 pm. The medium may comprise low-dimensional structures. The low-dimensional structures may comprise quantum wells. The quantum wells may be encapsulated quantum wells.

[0042] The medium may comprise a semiconductor material.

[0043] The semiconductor material may be an inorganic semiconductor material. The semiconductor material may be a III-V semiconductor material. The III-V semiconductor material may be a direct bandgap III-V semiconductor material. The III-V semiconductor material may be indium phosphide (InP), gallium arsenide (GaAs), or indium gallium arsenide (InGaAs).

[0044] The semiconductor material may be an organic semiconductor material. The organic semiconductor material may be Rhodamine 6G dye. The organic semiconductor material may be Rhodamine B dye.

[0045] The set of overlapping optical modes may include optical modes which are strongly coupled. The set of overlapping optical modes may be strongly coupled.

[0046] The set of overlapping optical modes may include at least 10 optical modes. The set of optical modes may include up to 100 or up to 1000 optical modes.

[0047] The set of overlapping optical modes may include two or more optical modes that are lasing modes. The set of overlapping optical modes may comprise only optical modes which are not lasing modes. The set of overlapping optical modes may comprise or consist of only one optical mode that varies non-linearly with the intensity of the spatially-structured illumination.

[0048] The spatially-structured illumination may include input data such as an image for classification. The input data may be encoded. The image may be a 2D image. The image may a pixellated image. The image may have a size of at least 20 pixels by 20 pixels, such as 28 pixels by 28 pixels. The image may have a size of at least 1000 pixels by 1000 pixels. The image may be projected optically onto the medium in a native pixel-map state. In the native pixel-map state, local information on neighbouring pixels and image features may be fully preserved. An n by m pixel image may be projected onto a medium having length L and width W such that each pixel of the projected image spans an area of L / n by W / m on the physical surface of the medium.

[0049] The device may be neuromorphic computing hardware. The device may be for use in neuromorphic optical computing. The device may be capable of providing a test accuracy score, measured by the Modified National Institute of Standards and Technology ("MNIST") handwritten digit classification task (for example at least the version available on 27 November 2023), equal to or greater than 95%. The device may be capable of providing a test accuracy score, measured by the MNIST handwritten digit classification task, equal to or greater than 97.7%. The device may be capable of providing a test accuracy score, measured by the MNIST handwritten digit classification task, equal to or greater than 98.15%.

[0050] The device may be configured for modulating of an optical response of the medium. Modulating an optical response may mean that the interaction(s) of the spatially- structured illumination with and within the medium are modified, resulting in a change in spatially-structured response provided for a given spatially-structured illumination input.

[0051] The interaction of the spatially-structured illumination with the medium through stimulated emission processes and through photoluminescence may be modulated via application of a field to the medium.

[0052] The device may be configured to modulate an optical response in the device based on adjusting a field applied to the medium. The field may be a local field or a global field. The field be an electric field, a magnetic field, illumination that is different to the spatially-structured illumination, or a temperature field. The field may include at least two of the group consisting of an electric field, a magnetic field, illumination that is different to the spatially-structured illumination, and a temperature field. The device may be configured to modulate the optical response of the medium by electrically gating the entire medium. The device may be configured to modulate the optical response of the medium by locally electrically gating at least one defined spatial subregion of the medium.

[0053] The device may include a set of first electrodes configured for applying a locally reconfigurable electric field to the medium. The device may be configured to use the first electrodes to apply local electrical gating. The reconfigurable electric field may be used to modulate the optical response. The reconfigurable electric field may be used to modulate a lasing threshold of the medium.

[0054] The first electrodes may be patterned below the medium. For example, the first electrodes may be interposed between the medium and the substrate. The first electrodes may be patterned above the medium. For example, the medium may be interposed between the electrodes and the substrate. Expressed differently, the first electrodes may be disposed on an upper surface of the medium opposite to the substrate. The first electrodes may be patterned laterally adjacent to the medium. For example, the first electrodes may be offset from the medium in a plane in which the medium at least predominantly lies. In other words, the first electrodes may be co-planar with the medium.

[0055] The first electrodes may have a 2D arrangement configured to apply the locally reconfigurable electric field across the medium. The 2D arrangement may define a 2D grid.

[0056] The device may be configured to use the first electrodes for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to store a configuration of the locally reconfigurable electric field which corresponds to a stable (at least locally) minima of a loss function. The device may be configured to retrieve and apply a stored configuration of the locally reconfigurable electric field.

[0057] When the operation of the device includes lasing modes, the device may be configured to use the first electrodes to apply a reconfigurable electric field that modulates a lasing threshold of the medium. When the operation of the device is based on photoluminescence, the device may be configured to use the first electrodes to apply a reconfigurable electric field that modulates how the spatially-structured illumination interacts with different regions of the medium.

[0058] Alternatively, the device may include a second set of electrodes configured for injecting charge carriers to the medium. In other words, the second electrodes are not separated from the medium by electrically insulating layers. For this purpose, rectifying contacts, tunnel barrier contacts, Schottky contacts and so forth may not be regarded as electrically insulating layers. Electrically insulating may mean that currents (beyond leakage currents) may not flow in response to applied fields below an irreversible electrical breakdown field. The device may be configured to use the second electrodes to apply local electrical currents through the medium. The injection of charge carriers may be used to modulate the optical response. The injection of charge carriers may be used to modulate a lasing threshold of the medium.

[0059] The second electrodes may be patterned below the medium. For example, the second electrodes may be interposed between the medium and the substrate. The second electrodes may be patterned above the medium. For example, the medium may be interposed between the second electrodes and the substrate. Expressed differently, the second electrodes may be disposed on an upper surface of the medium opposite to the substrate. The second electrodes may be patterned laterally adjacent to the medium. For example, the second electrodes may be offset from the medium in a plane in which the medium at least predominantly lies. In other words, the second electrodes may be co-planar with the medium.

[0060] The second electrodes may have a 2D arrangement configured to apply locally reconfigurable electrical currents across the medium. The 2D arrangement may define a 2D grid.

[0061] The device may be configured to use the second electrodes for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to store a configuration of the locally reconfigurable electrical currents which corresponds to a stable (at least locally) minima of a loss function. The device may be configured to retrieve and apply a stored configuration of the locally reconfigurable electrical currents.

[0062] In other examples, the device may be configured to include a set of electrodes which includes: a subset of first electrodes configured for applying locally reconfigurable electric fields to the medium, and a subset of second electrodes configured for injecting charge carriers to the medium. The set of electrodes may consist of the subset of first electrodes and the subset of second electrodes.

[0063] The device may be configured to modulate the optical response based on providing interference effects in the medium. The optical response may be a lasing response.

[0064] The device may be configured to modulate the optical response based on projecting a mask taking the form of the spatially-structured light beam from the second light source onto at least a part of the medium. The device may be configured to provide spatial structuring to the spatially structured light beam using a digital micromirror device (DMD) or using a spatial light modulator (SLM), by optical interference or holography, or by spatial focussing.

[0065] The device may be configured to project a trained grid of pixels in the mask on the medium, superimposed over the spatially-structured illumination, to modulate the complex refractive index of the medium. Expressed differently, a grid of pixels in a mask taking the form of the spatially-structured light beam from the second light source is used to train the medium.

[0066] Alternatively, the device may be configured to modulate the optical response based on superposing the mask with the spatially structured illumination. The mask may be superposed with the spatially structured illumination prior to output of the spatially structured illumination. For example, an image corresponding to the mask may be combined with an image corresponding to the spatially structured illumination, before being output to a DMD, SLM, and so forth. The mask may be superposed with the spatially structured illumination by addition. The mask may be superposed with the spatially structured illumination by convolution.

[0067] Alternatively, the mask may be superposed with the spatially structured illumination at a point along an optical path to the medium. For example, a combiner may merge the spatially structured illumination with a spatially-structured light beam from a second light source.

[0068] The mask may include, or take the form of, a set of mask pixels corresponding to a maximum illumination intensity. The mask may include, or take the form of, a set of mask pixels corresponding to a minimum illumination intensity. The mask may include, or take the form of, a set of mask pixels, each corresponding to an illumination intensity less than the maximum illumination intensity and greater than the minimum illumination intensity. The mask may take the form of an image formed by mask pixels corresponding to varying illumination intensities.

[0069] The device may be configured to use the spatially-structured light beam from the second light source for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to use the superposed mask for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to store a configuration of the mask which corresponds to a stable (at least locally) minima of a loss function. The device may be configured to retrieve and apply a stored configuration of the mask using the spatially-structured light beam from the second light source.

[0070] The device may be configured to modulate the optical response based on local magnetic control. The device may be configured to modulate the optical response based on local magneto-plasmonic control. The device may be configured to modulate the optical response based on global magnetic control. The device may be configured to modulate the optical response based on global magneto-plasmonic control.

[0071] The device may comprise magnetic elements arranged to provide a locally reconfigurable magnetic field across the medium. The locally reconfigurable magnetic field may be used for local magnetic control or local magneto-plasmonic control. Magnetic elements may include nanomagnets. The device may be configured to use the magnetic elements to apply a locally reconfigurable magnetic field across the medium. The magnetic state of the magnetic elements may locally control how much of the spatially-structured illumination is absorbed or enhanced, and may be used to modulate the optical response. The magnetic elements may be configured such that the magnetic state thereof can be reconfigured by at least one of an applied magnetic field, current / voltage control, or all-optical magnetic switching.

[0072] The magnetic elements may be permanent magnets. In this way, a trained state of the medium may be stored directly in the device itself.

[0073] Magnetic elements may be disposed on an upper surface of the medium. The medium may be interposed between the magnetic elements and the substrate. The magnetic elements may be disposed on an upper surface of the medium opposite to the substrate. The magnetic elements may be arranged to be adjacent to the medium, that is, offset from the medium in a plane in which the medium at least predominantly lies. In other words, the magnetic elements may be co-planar with the medium. The magnetic elements may be arranged to define a 2D grid.

[0074] The device may be configured to use the magnetic elements for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to store a configuration of the locally reconfigurable magnetic field which corresponds to a stable (at least locally) minima of a loss function. The device may be configured to retrieve and apply a stored configuration of the locally reconfigurable magnetic field using the magnetic elements. The device may be configured to modulate the optical response based on varying the temperature of the medium. The device may include one or more heaters configured to vary the temperature of the medium. The (or each) heater may be an on-chip heater. The device may be configured to control the heater(s) to vary the temperature of the medium locally or globally.

[0075] The device may be configured to use the heater(s) for training the medium using the method according to the sixth aspect described hereinafter. The device may be configured to store a configuration of a locally varied temperature which corresponds to a stable (at least locally) minima of a loss function. The device may be configured to retrieve and apply a stored configuration of the locally varied temperature using the heater(s).

[0076] The device may be configured to determine an environmental temperature based on sensing modulation of the optical response of the medium. In other words, based on detecting changes in the spatially-structured response of the medium to one or more reference patterns of spatially-structured illumination as the environmental temperature changes, the environmental temperature may be inferred.

[0077] The described approaches to modulating the optical response may be combined with one another.

[0078] According to a second aspect of the present invention there is provided a method of fabricating the device of the first aspect. The method includes: providing a first substrate having a principal surface; providing a hetero-epitaxial wafer having an intermediate sacrificial layer interposed between a second substrate and an upper layer; bonding the upper surface of the hetero-epitaxial wafer to the principal surface of the first substrate; forming a bonded medium, the bonded medium comprising the first substrate and the upper layer, forming the bonded medium comprising etching the sacrificial layer to remove the second substrate and the sacrificial layer; and lithographically patterning structures on the surface of the bonded medium from which the sacrificial layer was etched.

[0079] The method of the second aspect may include features corresponding to any features of the device of the first aspect.

[0080] According to a third aspect of the present invention there is provided an optical image processing system. The optical image processing system comprises the device of the first aspect. The optical image processing system also comprises an imaging spectrometer configured to detect the spatially-structured response.

[0081] Readout from the optical image processing system may be performed spatially uniformly (that is, with one single spatially integrated spectra for each input image for the entirety of the medium) or with spatial resolution (that is, with a spatially integrated spectra for each input image for each of a plurality of defined spatial-sub- regions of the medium).

[0082] The imaging spectrometer may be configured to perform readout on defined spatial sub-regions of the medium. Readout may be performed on defined spatial sub-regions of the medium using an arrangement of lenses in a detection path between the medium and the spectrometer. Alternatively, readout may be performed on the whole medium.

[0083] The optical image processing system may further comprise a neutral density filter configured to maintain a pump power of the spatially-structured illumination.

[0084] The optical image processing system may further comprise a digital micromirror device configured to provide the spatially-structured illumination. The digital micromirror device may be arranged in a reflection geometry.

[0085] The digital micromirror device may provide the spatially-structured illumination by selective reflection of unstructured light provided by a first light source. The first light source may include, or take the form of, a laser source. The first light source may include, or take the form of, a light emitting diode.

[0086] The system may have a configuration in which a majority of functional elements, for example all functional elements, including the medium are arranged in an on-chip package in which functional optical elements are assembled in a multi-layer vertical stack on a flat substrate such as a silicon (Si) chip or the like. The on-chip package (hereinafter referred to as the "photonics chip") may be a flat on-chip package. In the on-chip package, the source of spatially structured illumination may include a spatially uniform flat form factor light source such as a microstrip laser.

[0087] The optical image processing system may further comprise a spatial light modulator configured to provide the spatially-structured illumination. The spatial light modulator may be used to impart a spatial structure on the light output by a source of illumination. Alternatively, the spatially structured illumination may be provided by an optical pump capable of outputting spatially structured illumination, such as a nanolaser array, a vertical-cavity surface-emitting laser array, or a miniature light-emitting diode array. Readout hardware may include an on-chip spectrometer of the photonics chip. The photonics chip may further include the medium and the source of illumination. The source of illumination may include photodiodes.

[0088] The optical image processing system may be configured for use as a non-trainable reservoir for data processing.

[0089] The optical image processing system may be configured for use as a fully trainable deep-neural network for inference.

[0090] The system of the third aspect may include features corresponding to any features of the device of the first aspect and / or the method of the second aspect.

[0091] According to a fourth aspect of the present invention there is provided a computer vision product. The computer vision product comprises the optical image processing system of the third aspect. Expressed differently, the computer vision product comprises the device of the first aspect and an imaging spectrometer configured to detect the spatially-structured response.

[0092] Although the response of the medium is spatially structured, the detection of the spatially-structured response using the imaging spectrometer need not be spatially discretised. For example, the imaging spectrometer may be configured to detect a frequency spectrum of the spatially-structured response corresponding to the medium overall, or a single region thereof. Alternatively, the imaging spectrometer may be configured to detect a frequency spectrum of the spatially-structured response corresponding to each of two or more different regions of the medium.

[0093] The computer vision product also comprises a processor configured to determine variations in spectra provided by the spatially-resolved spectrometer. The spectra may be spatially-resolved spectra.

[0094] The computer vision product may be selected from the group consisting of a selfdriving car; an autonomous drone, an autonomous aircraft, an autonomous sea vessel, a low-latency image processing system, a CCTV camera processing system, an industrial process control system, a medical imaging system, and a surgical control system. The CCTV camera processing system may be for crowd recognition. The industrial process control system may be for observing a production line, such as a production line for food or a production line for components. The medical imaging system may be for processing data comprising MRI images or tumour images. The surgical control system may be for processing endoscope camera feeds.

[0095] The computer vision product may be a vehicle capable of being autonomously piloted or a system for object detection and classification.

[0096] The vehicle may be the self-driving car, the autonomous drone, the autonomous aircraft, or the autonomous sea vessel. The object may be a human face. The system may be the low-latency image-processing system, the industrial process control system, the medical imaging system, or the surgical control system.

[0097] The computer vision product of the fourth aspect may include features corresponding to any features of the device of the first aspect, the method of the second aspect, and / or the system of the third aspect.

[0098] According to a fifth aspect of the present invention there is provided a method of operating the device of the first aspect, the optical image processing system of the third aspect, or the computer vision product of the fourth aspect. The method comprises providing the spatially-structured illumination to the device. The method also comprises detecting the spatially-structured response.

[0099] The spatially-structured response may correspond to a computational task. The spatially-structured response may be reconfigured to correspond to another computational task by training the medium.

[0100] The method of operating the device may be a method of optical image processing.

[0101] The method may comprise modulating an optical response of the medium. The modulation of the optical response may be carried out in any manner described in relation to the first and sixth aspects. The method of the fifth aspect may include features corresponding to any features of the device of the first aspect, the method of the second aspect, the system of the third aspect, and / or the computer vision product of the fourth aspect.

[0102] According to a sixth aspect of the present invention there is provided a method of training the device of the first aspect, the optical image processing system of the third aspect, or the computer vision product of the fourth aspect. The method comprises: determining a loss function of the device, determining whether the loss function has reached a stable minima, and modulating an optical response in the device.

[0103] The method may comprise iteratively: determining a loss function of the device, determining whether the loss function has reached a stable minima, and modulating an optical response in the device.

[0104] The optical response may be iteratively modulated using a gradient-descent algorithm or a backpropagation algorithm.

[0105] The method may be carried out until a determination that the loss function has reached a stable minimum is made.

[0106] Determining a loss function of the device may comprise assessing task performance for a given task and generating the loss function based on the task performance.

[0107] Modulating the optical response may include providing interference effects in the medium. The optical response may be a lasing response. The interaction of the spatially-structured illumination with the medium through stimulated emission processes and through photoluminescence may be modulated via application of a field to the medium.

[0108] Modulating an optical response in the device may comprise adjusting a field applied to the medium.

[0109] The field may be a local field or a global field. The field be an electric field, a magnetic field, illumination that is different to the spatially-structured illumination, or a temperature field. The field may include at least two of the group consisting of an electric field, a magnetic field, illumination that is different to the spatially-structured illumination, and a temperature field. Reconfigurable training may be provided by modulating the interaction of the spatially-structured illumination with the medium via application of the field to the medium.

[0110] Modulating the optical response may include projecting the mask taking the form of the spatially-structured light beam from the second light source onto at least a part of the medium.

[0111] Modulating the optical response may include electrically gating the entire medium or locally electrically gating at least one defined spatial sub-region of the medium. A set of electrodes may be used to apply the local electrical gating. The set of electrodes may be patterned laterally adjacent to the medium, above the medium, or below the medium. The set of electrodes may have a 2D arrangement and be configured to apply a locally reconfigurable electric field across the medium. The 2D arrangement may define a 2D grid. The reconfigurable electric field may be used to modulate the optical response. The reconfigurable electric field may be used to modulate a lasing threshold of the medium.

[0112] Modulating the optical response may include at least one of local magnetic control, local magneto-plasmonic control, global magnetic control, and global magneto- plasmonic control. Magnetic elements such as nanomagnets may be used to provide the local magnetic control or local magneto-plasmonic control. The magnetic elements may be configured to apply a locally reconfigurable magnetic field across the medium. The magnetic state of the magnetic elements may locally control how much of the spatially-structured illumination is absorbed or enhanced and can be used to modulate the optical response. The magnetic state may be reconfigurably programmed by at least one of an applied magnetic field, current / voltage control, or all-optical magnetic switching.

[0113] Modulating the optical response may include varying the temperature of the medium. Varying the temperature of the medium may be carried out using a heater such as an on-chip heater. Varying the temperature of the medium may be carried out by environmental temperature. The temperature may be varied locally or globally.

[0114] The method of the sixth aspect may include features corresponding to any features of the device of the first aspect, the method of the second aspect, the system of the third aspect, the computer vision product of the fourth aspect, and / or the method of the fifth aspect. Brief Description of the Drawings

[0115] Certain embodiments of the present invention will now be described, by way of example, with reference to the accompanying drawings in which:

[0116] Figure 1 is a schematic perspective view of a first optical image processing system comprising a device that comprises a medium;

[0117] Figure 2A is a schematic plan view of a portion of the medium of Figure 1;

[0118] Figure 2B is a schematic cross-section view along the line A-A' in Figure 2A.

[0119] Figure 3 is a schematic plan view of a medium comprising spatially separated photonic networks;

[0120] Figure 4 is a schematic side view of a medium disposed on a surface that is not flat;

[0121] Figure 5 is a schematic side view of a medium that is a multi-layered heterostructure;

[0122] Figure 6 is a schematic side view of a medium comprising voids;

[0123] Figure 7 is a schematic side view of a medium comprising suspended structures;

[0124] Figure 8 is a schematic perspective view of a medium that is 3D patterned;

[0125] Figure 9 is a process flow diagram of a method of fabricating a device;

[0126] Figure 10 is a schematic block diagram of a second optical image processing system;

[0127] Figure 11 is a schematic block diagram of a third optical image processing system;

[0128] Figure 12 is a schematic block diagram of a fourth optical image processing system;

[0129] Figure 13 is a schematic side view of a device comprising electrodes for training;

[0130] Figure 14 is a schematic side view of a device comprising nanomagnets for training;

[0131] Figure 15 is a process flow diagram of a method of training a device;

[0132] Figure 16 shows examples of correct (top) and incorrect (bottom) classifications;

[0133] Figure 17 shows a test accuracy score of 97.7% determined using a device and by the MNIST handwritten digit classification task;

[0134] Figure 18A shows an exemplary image (Figure 18A) to be used in a comparison of optical physical computing using a device provided with raw image data with logistic regression performed on the same raw image data;

[0135] Figure 18B is a photonic network to which the image of Figure 18A is provided;

[0136] Figure 18C is a spatially-resolved spectrum from the photonic network of Figure 18B;

[0137] Figure 18D shows a test accuracy score of 98.15% determined using spatially-resolved spectra including the spectrum of Figure 18C from a device;

[0138] Figure 18E shows a test accuracy score of 90.5% determined by logistic regression on raw image data;

[0139] Figure 18F is a perspective view illustration of optical physical computing;

[0140] Figure 19 is a schematic perspective view of a device receiving spatially-structured illumination that is a spatially-structured response from another device;

[0141] Figure 20 is a schematic block diagram of a fifth optical image processing system;

[0142] Figure 21 is a schematic block diagram of a sixth optical image processing system; Figure 22 is a schematic block diagram of a seventh optical image processing system; Figure 23A is experimental data showing an input image;

[0143] Figures 23B and 23C are experimental data showing extracted feature / edge maps provided by two different optical modes;

[0144] Figure 23D is experimental data showing feature / edge maps being extracted by multiple optical modes, where logistic regression has been used to train which combination of modes are used to extract image edges;

[0145] Figures 24A to 24D are simulated data illustrating detecting different edges (top / right / left / bottom) using four different optical modes, colour bars correspond to presence of the desired edge with 1 (white) indicating presence of the desired edge and 0 (black) indicating absence of the desired edge;

[0146] Figure 25A is an annotated plan-view SEM micrograph of a medium in the form of a photonic network patterned from III-V semiconductor material;

[0147] Figure 25B is a schematic projection view of a first configuration of electrodes for electro-optic modulation of a medium;

[0148] Figures 26A to 26G schematically illustrate a method of fabricating the first configuration of electrodes for electro-optic modulation of a medium;

[0149] Figures 27A to 27E schematically illustrate a method of fabricating a second configuration of electrodes for electro-optic modulation of a medium;

[0150] Figures 28A to 28E schematically illustrate a method of fabricating a third configuration of electrodes for electro-optic modulation of a medium;

[0151] Figures 29 show schematic plan views corresponding to Figures 28A, 28C and 28E respectively;

[0152] Figure 30A schematically illustrates a low-density photonic network having network links 7 pm long on average;

[0153] Figure 30B compares simulated lasing spectra for the photonic network shown in Figure 30A for a pair of different conditions;

[0154] Figure 31A schematically illustrates a high-density photonic network having network links 2 pm long on average;

[0155] Figure 31B compares simulated lasing spectra for the photonic network shown in Figure 31A for a pair of different conditions;

[0156] Figures 32A to 32G present experimental data for an image classification task performed without masking; and

[0157] Figures 33A to 33G present experimental data for the same image classification task as Figures 32A to 32G, performed with masking.

[0158] Detailed Description of Certain Embodiments

[0159] In the following, like parts are denoted by like reference numerals. Introduction

[0160] Herein, devices for use in optical image processing are described. The devices can be driven by projecting spatially-structured illumination including input image data onto a medium of the device. As a result of the interactions of the spatially-structured illumination with and within the medium, a spatially-structured response is output (the response is also typically structured in frequency). By modulating the interactions of the spatially-structured illumination with and within the device, or alternatively by processing the input image data, the devices can be used to perform neuromorphic optical computing.

[0161] First optical image processing system 1

[0162] Referring to Figure 1, a first optical image processing system 1 (hereinafter referred to as the "first system") is shown.

[0163] The first system 1 includes a device 2 for optical image processing, a first light source 3 for providing spatially-structured illumination 4 including input image data to the device 2, and an imaging spectrometer 5 for detecting a spatially-structured response 6 from the device 2. The device 2 includes a medium 7 and a substrate 8. The medium 7 is disposed on a surface 9 of the substrate 8 and can support overlapping optical modes 10. In response to receiving the spatially-structured illumination 4, a set of overlapping optical modes 10 are excited in the medium 7 and output the spatially-structured response 6.

[0164] Although described as a spatially-structured response 6, it should be appreciated that the response 6 of the medium 7 will typically be structured both spatially and in frequency. Although the response of the medium 7 is spatially structured, the detection of the spatially-structured response 6 using the imaging spectrometer 5 need not be spatially discretised. For example, the imaging spectrometer 5 may be configured to detect a frequency spectrum of the spatially-structured response 6 corresponding to the medium 7 overall (i.e. globally), or a single region thereof. Alternatively, the imaging spectrometer 5 may be configured to detect a frequency spectrum of the spatially-structured response 6 corresponding to each of two or more different regions of the medium 7.

[0165] The medium 7 includes a first photonic network 11 in which the input image data is processed and from which the spatially-structured response 6 is output. Referring also to Figures 2A and 2B, a portion of the first photonic network 11 is shown.

[0166] The first photonic network 11 comprises a plurality of network links 12i, 12z, 12N that are radiatively coupled at nodes 13i, 132, 13N and is disposed on the flat surface 9.

[0167] The spatially-structured response 6 can include contributions arising from the spatially-structured illumination 4 interacting with the medium 7 in a variety of ways. First, the spatially-structured response 6 can include a contribution arising from scattering of the spatially-structured illumination 4 from the medium 7. Secondly, the spatially-structured response 6 can include a contribution from photoluminescence, that is, absorption of the spatially-structured illumination 4 by the medium 7 leading to emission of photons by florescence or phosphorescence. Thirdly, the spatially- structured response 6 can include a contribution from stimulated emission processes such as lasing or amplified spontaneous emission ("ASE") that is driven by the interaction of the spatially-structured illumination 4 with the medium 7.

[0168] Different optical modes 10 of the set of overlapping optical modes 10 excited in response to the spatially-structured illumination 4 may differ in one or more of frequency; spatial mode shape; and spatial location within the medium 7.

[0169] The set of overlapping optical modes 10 includes two or more optical modes that are lasing modes. The spatial-structuring of the spatially structured illumination 4 can unbalance mode completion between the lasing modes. This need not be the case, however, and the set of optical overlapping modes 10 may instead comprise only optical modes which are not lasing modes.

[0170] Resonant modes capable of leading to lasing can be provided by using the first light source 3. The first light source 3 can include a suitable pulsed pump laser coupled into an optical microscope. The suitability of a pulsed pump laser is determined, at least in part, by whether it can provide a pulse fluence that is sufficiently large to induce lasing within the medium 7. In the case that the medium 7 includes a semiconductor material, the suitable pulsed pump laser can be a femtosecond pulsed laser. In the case that the medium 7 includes a polymer material, the suitable pulsed pump laser can be a nanosecond pulsed laser. By performing physical transforms in a parallel fashion and intrinsically using its internal (optical and optoelectronic) physics, the device 2 can have a higher processing speed than software networks. In particular, the speed of stimulated emission processes can mean that the device 2 can process an input image in as little as around 10 ps. Moreover, since the device 2 processes data taking the form of light, it can have a higher energy efficiency than conventional CMOS / GPU hardware.

[0171] Spectra provided by the imaging spectrometer 5 based on the spatially-structured response 6 are sensitive to small variations in the spatially-structured illumination 4. Shifts in these spectra can form the basis on which image processing Al tasks can be performed using the device 2. Small variations in the spatially-structured illumination 4 may cause changes in the relative amplitude, or even the identity, of the excited set of overlapping optical modes 10, producing the changes in the spectra obtained by the imaging spectrometer 5.

[0172] The same input data, in the form of spatially-structured illumination 8, may be provided to the device 2 multiple times, with transformations such as translations, rotations, reflections, spatial enlargement or shrinking. Providing the same input data to the device 2 multiple times with such transformations and combining the spatially- structured responses 6 for each time the input data is provided can help to enhance the computational performance (that is, the test accuracy score determined by the MNIST handwritten digit classification task) of the device 2.

[0173] In modes of operation based on lasing, the power range for computing tasks is defined by measuring lasing light-in light-out curves for the device 2. The minimum power is defined by the lasing threshold (at which there is at least one lasing mode in the spectrum of the spatially-structured response 6) and the maximum power is defined by the power at which there is a maximum number of lasing modes in the spectrum of the spatially-structured response 6 or by the power at which there is a drop in light intensity output due to heating / material degradation, whichever of the two is lower. In these modes, the power range is typically well above the minimum power and well below the maximum power.

[0174] The network links 12i, 122, ..., 12N are radiatively coupled by way of being in physical contact with each other at the nodes 13i, 132, ..., 13N. This need not be the case, however, and at least a sub-set of the network links 12i, 122, ..., 12N can be spatially separated from other network links at the nodes 13i, 132, ..., 13N by a distance less than a design wavelength at which the device 2 is configured to operate. In this way, direct physical contact between the network links 12i, 122, 12N is not required.

[0175] The network links 12i, 122, ..., 12N are active photonic waveguides. This need not be the case, however, and the network links 12i, 122, ..., 12N may include components other than the active photonic waveguides or may include low-dimensional features such as quantum wells instead of active photonic waveguides. Network links may include, or take the form of, particle chains. Network links may include, or take the form of, reflective elements. Network links may include, or take the form of, passive photonic waveguides.

[0176] Referring to Figure 3, a second example of the device 2 is shown.

[0177] The medium 7 can include a second photonic network 112 that is spatially separated from the first photonic network 11 by a distance greater than the design wavelength. In this way, the device 2 can provide a set of parallel outputs. Combining parallel outputs can help to improve the computational performance of the device 2.

[0178] The second photonic network II2 may have the same materials and topology as the first photonic network 11 (optionally rotated or otherwise transformed). Alternatively, the second photonic network II2 may differ from the first photonic network 11 in one or both of materials and topology.

[0179] Referring to Figure 4, a third example of the device 2 is shown.

[0180] The medium 7 can be disposed on a substrate 8 and on a surface 14 that is not flat. The surface 14 can have a wide variety of profiles that are not flat. In particular, the surface 14 may include flat portions connected by sloped portions or may be undulant. Furthermore, the surface 14 may not be a surface of the substrate 8 and may instead be defined by the combination of the substrate 8 and a patterned layer (not shown) such as a set of gate electrodes of a set of magnetic elements interposed between the substrate 8 and the medium 7 (best illustrated in Figure 13).

[0181] Spatial structuring of the medium 7

[0182] Referring to Figure 5, a fourth example of the device 2 is shown. The medium 7 can be a multi-layered heterostructure including medium layers 15i, 152, 15N. The medium layers 15i, 152, ..., 15N can have different thickness and / or composition and are not required to be planar.

[0183] Each medium layer 15i, 152, ..., 15N provides a medium 7, allowing a single input of spatially-structured illumination to be processed in parallel by each medium layer 15i, 152, ..., 15N. If the gaps between medium layers 15i, 152, ..., 15N are small, then interactions between adjacent medium layers 15i, 152, ..., 15N may also become significant. Preferably, the medium layers 15i, 152, ..., 15N differ from one another by at least one (and possibly more) of material, topology and orientation.

[0184] Referring to Figure 6, a fifth example of the device 2 is shown.

[0185] Voids 16 may be arranged non-uniformly within the medium 7. The voids 16 may have a regular shape, such as an ellipsoidal shape, or an irregular shape. The voids 16 are not required to have the same shape and can have any shape.

[0186] Voids 16 can add nonlinear optical losses to the medium 7. This can be applied to introduce additional computational power to the device 2.

[0187] Referring to Figure 7, a sixth example of the device 2 is shown.

[0188] Voids 16 can be arranged non-uniformly within the medium in such a way that suspended structures 17 are defined. The suspended structures 17 need not be regularly shaped.

[0189] Suspended structures 17 can be configured to enable 3D coupling between regions of the medium 7 (for example regions of a photonic network 11) which are not close to each other. Suspended structures 17 may also have lower optical losses in comparison to a waveguide formed on a substrate. This arises because there is typically a larger difference in refractive index between the medium 7 and air filling the void 16, compared to a difference in refractive index between the medium 7 and the substrate 8.

[0190] Referring to Figure 8, a seventh example of the device 2 is shown.

[0191] The medium 7 need not be planar. Expressed differently, the medium 7 may have a 3D patterning in that it has both an in-plane texture and an out-of-plane texture. In particular, expanded polystyrene foam beads can be used to provide so-called "inverse opal" style 3D patterning. Alternatively, the 3D patterning can be provided via optical interference writing, or via nanoscale 3D printing such as so-called "two-photon lithography" or focused ion-beam deposition, or via 3D nanoscale self-assembly.

[0192] Hereinbefore, it has been described that the medium 7 has a spatially non-uniform physical structure. This need not be the case, however, and the medium 7 can have a physical structure which is spatially-uniform along three orthogonal directions by way of it having a complex refractive index which is spatially-non-uniform. In particular, the first photonic network 11 can replaced with a uniform thin film having a complex refractive index and a second light source (not shown) can be used to provide the spatially non-uniform complex refractive index by projecting a spatially-structured light beam onto the uniform thin film. Expressed differently, the first light source 3 is used to provide the spatially-structured illumination 4 as a probe and the second light source can be is used to provide the spatial structuring of the complex refractive index of the medium 7.

[0193] For example, a uniform thin film having a complex refractive index may be formed from Indium Tin Oxide (ITO), a uniform film of Indium Phosphide (In P), or another active material which is being illuminated / pumped with a nonuniform spatial pattern.

[0194] Device fabrication

[0195] The medium 7 can include a wide variety of semiconductor materials. In particular, the medium 7 can be include a III-V semiconductor material such as indium phosphide (InP) or an organic semiconductor material such as Rhodamine 6G dye. Thus, the medium can be a solid-state gain medium or a liquid-phase dye medium.

[0196] The device 2 can be fabricated in a wide variety of ways, such as by lithography on a thin III-V semiconductor layer that is bonded on a SiCh-coated Si substrate, by electrospinning dye-mixed polymer nanofibers onto a TEM grid and post-annealing to fuse the network links 12i, 122, ..., 12N together to form at least one photonic network 11, II2, or by nanoimprint lithography of soft polymers doped with laser dye.

[0197] Media such as the medium 7 can be cheap to produce, near-infinitely configurable, and readily compatible with widespread semiconductor and polymer fabrication techniques.

[0198] Referring to Figure 9, a method of fabricating the device 2 will now be described. The method includes providing a first substrate having a principal surface (step Sl. l), providing a hetero-epitaxial wafer having an intermediate sacrificial layer interposed between a second substrate and an upper layer (step SI.2), bonding the upper layer to the principal surface of the first substrate (step SI.3), and etching the sacrificial layer to remove the second substrate and the sacrificial layer to form a bonded medium that includes the first substrate and the upper layer by (step SI.4). The method may then include lithographically patterning structures onto the surface of the bonded medium from which the sacrificial layer was etched (step SI.5). The bonded medium is an example of the device 2. In particular, the first substrate can be the substrate 8 and the principal surface can be the surface 9 or the surface 14.

[0199] The first substrate may include or be a silicon (Si) wafer. The principal surface may be a surface of the silicon wafer or a surface of a silicon dioxide (SiOz) layer deposited on the silicon. The second substrate may include an indium phosphide (InP) wafer. The intermediate sacrificial layer may include indium gallium arsenide (InGaAs). The upper layer may include indium phosphide (InP).

[0200] Second optical image processing system 18

[0201] Referring to Figure 10, a second optical image processing system 18 (hereinafter referred to as the "second system") is shown.

[0202] The second system 18 is different to the first system 1 in that it uses an external source of spatially structured illumination 19 instead of in place of the first light source 3 and includes a spectrometer 20 that is not required to be an imaging spectrometer such as the imaging spectrometer 5. Expressed differently, the spectrometer 20 is not required to be spatially-resolved.

[0203] In the second system 18, the device 2 receives spatially-structured illumination 4 from the source of spatially structured illumination 19 and outputs its spatially-structured response 6 to the spectrometer 20. The readout of the spectrometer 20 is provided to a computer system 1 for processing.

[0204] Because the spectrometer 20 is not required to be spatially-resolved, it can provide a readout which corresponds to the spatially-structured response 6 from the whole of the medium 7. In the case that the spectrometer 20 is an imaging spectrometer such as the imaging spectrometer 5, however, the spectrometer 20 can provide a readout which corresponds to the spatially-structured response 6 from at least one defined spatial sub-region of the medium 7.

[0205] In the spectrometer 20, the spatially-structured response 6 can be collected by a microscope objective (not shown), filtered to remove a contribution corresponding to the spatially-structured illumination 4, spectrally dispersed using a grating (not shown), and focussed into a CCD camera (not shown). Spatially-resolved readout can be enabled by including an arrangement of lenses (not shown) between the medium 7 and the spectrometer 20.

[0206] The second system 18 can use a neutral density filter (not shown) to maintain a pump power of the spatially-structured illumination 4 by monitoring and correcting the pump power with respect to reference power values. Implementing such real-time correction can help to improve computational performance.

[0207] Third optical image processing system 22 - free-space implementation Referring to Figure 11, a third optical image processing system 22 (hereinafter referred to as the "third system") is shown.

[0208] The third system 22 has a free-space implementation in which light is projected through free space between its component parts.

[0209] The third system 22 is different to the second system 18 in that it includes a digital micromirror device ("DMD") 23 arranged in a reflection geometry. In particular, the DMD 23 is arranged to receive input light 24 from a source of illumination 25 external to the third system 22, and to provide the spatially-structured illumination 4 to the device 2 via a beam splitter 26 and an objective 27. The spatially-structured response

[0210] 6 of the device 2 is detected by the spectrometer 20, the readout of which is provided to the computer system 21 for processing.

[0211] In other examples, the DMD 23 may alternatively be arranged illuminate the medium

[0212] 7 in a transmission geometry.

[0213] Using a DMD to impart a spatial structure such as a spatial structure corresponding to an image onto input light such as the input light 24 can help to make the third system 22 cheaper and easier to manufacture. This is because DMD components are widely available and industrially produced. Using a DMD 23 can also help to make the third system 22 suitable for operating at GHz and even THz frequencies. Fourth optical image processing system 28 - on-chip implementation

[0214] Referring to Figure 12, a fourth optical image processing system 28 (hereinafter referred to as the "fourth system") is shown.

[0215] The fourth system 28 is different from the third system 22 in that it has an on-chip implementation rather than a free-space implementation. The fourth system 28 is also different from the third system 22 in that it uses a spatial light modulator ("SLM") 29 in a transmission geometry instead of a DMD such as the DMD 23 to provide the spatially-structured illumination 4.

[0216] The component parts of the fourth system 28, namely the SLM 29, the device 2, and the spectrometer 20, are arranged in an on-chip package (herein also referred to as a "photonics chip"). In the on-chip package, the component parts of the fourth system 28 are arranged in a multi-layer vertical stack on a flat substrate such as a silicon (Si) chip. The on-chip package being flat can help to make the system 28 easier to integrate integration into larger assemblies.

[0217] The SLM 29 is used to impart a spatial structure on the input light 24 provided by the source of illumination 25. The source of illumination 25 can be a spatially-uniform flat form factor light source such as a microstrip laser. The SLM need not be used to impart the spatial structure to the spatially-structured illumination 4, however, and the spatially-structured illumination 4 can instead be provided directly from a source of illumination 25 such as a nanolaser array, a vertical-cavity surface-emitting laser ("VCSEL") array, or a miniature light-emitting diode ("LED") array, that is, a source of illumination 25 capable of directly outputting the spatially structured response 4.

[0218] Neuromorphic computing schemes

[0219] The device 2 described herein can form the basis for implementing one or more neuromorphic computing schemes.

[0220] Reservoir computing

[0221] The device 2 can be used as a reservoir for reservoir computing. In this scheme, the device 2 is not required to be trainable and the input data in the spatially-structured illumination 4 translates into a spatiotemporal gain profile in the device 2 due to absorption of the spatially-structured illumination 4 by the medium 7. Light emitted from the medium 7 is guided in the medium 7 and propagates across multiple paths within the medium 7. Constructive interference of light across the at least two closed paths within the medium can result in the optical modes 10, which can be amplified with optical gain. Lasing from these modes 10 occurs at the threshold when optical gain for a mode balances the mode losses. The optical modes 10 have a non-zero spatial overlap and so can compete for gain. When two or more optical modes 10 lase, mode competition due to spatial hole burning and non-linear coupling can result in non-linear variation of the mode intensity and the wavelength with the pump-power of the spatially-structured illumination 4 across the medium 7.

[0222] In this way, a system including the device 2 may perform multiple linear non-linear operations on the input data and output unique spectra such as lasing spectra formed from these operations. Reservoir computing can be performed by processing these spectra, that is, the output of the reservoir. In particular, the spectral output can be processed by regression to directly produce a computational output (see for example the results shown in Figure 17 and described hereinafter). The reservoir computing scheme can be trained for a given computational task by optimising the processing of these spectra for one or more given inputs.

[0223] Input data can pre-processed to take the form of the spatially-structured illumination 4. In particular, analogue time-series data can be converted to grayscale pixel illumination values and illuminated onto the medium 7. One way to pre-process the input data is to pre-process the input data as a matrix of positive values, binarize the pre-processed input data into pattern projections on a DMD such as the DMD 23 or a SLM such as the SLM 29, and to project the spatially-structured illumination 6 from the DMD or SLM onto the device 2.

[0224] Input data pre-processing can include pixel dithering and / or noise representation to enable grayscale and / or analogue value input. These types of input data preprocessing can help to improve computational performance.

[0225] Deep neural networks

[0226] The device 2 can be used in a fully trainable deep neural network ("DNN") for inference. In particular, the device 2 can be configured as a layer or layers in a neural network such as a convolutional neural network ("CNN"). In this scheme, the device 2 is required to be trainable.

[0227] By projecting image data such as image fragments onto the medium 7, the device 2 can be used as a convolutional kernel. In this scheme, the spectral output from the spectrometer 20 is passed to another network layer for further processing instead of being processed by regression to directly produce a computational output.

[0228] The spatially-resolved response from a first network layer including the device 2 can be projected as an image onto a further network layer (best illustrated in Figure 19). The further network layer can be a distinct physical network with different properties to the first network layer. The further network layer can include or be hardware such as the device 2, a memristive network, or a nanomagnetic network.

[0229] If a direct, physical transfer of information between the first network layer and a further network layer is desired, then the further network layer would need to be designed (for example by choice of appropriate materials) to accept the spatially- structured response output of the first network layer as input. Alternatively, an intermediate measurement step may be used, for example using an imaging spectrometer 5 to measure the spatially-structured response of the first network layer, and using this as the basis to prepare a spatially-structured illumination for projection onto the further network layer. The latter option may be preferable to maintain signal-to-noise ratio through a sequence of two or more network layers.

[0230] The first network layer may have software and / or hardware layers before it, and instead of image data, the input data may be the output of preceding or subsequent network layers (best illustrated in Figures 21 and 22). In this way, the device 2 can be used as a pre / post-processor as part of a larger computing system. This can add memory and thereby can allow the device to be used for video processing.

[0231] Output data can be post- processed in external circuits or an external CPU such as the computer system 21. Output data can be extracted directly from the spatially- structured response 6, for example using spatially-resolved spectra such as optical lasing spectra read out from the spectrometer 20.

[0232] Training

[0233] The device 2 can be trained in wide a variety of ways.

[0234] A first mode for training the device 2 uses the second light source (not shown) to provide spatial structuring of the complex refractive index of the medium 7.

[0235] Referring to Figure 13, a device 2 including electrodes 30 for training the medium 7 interposed between the medium 7 and the substrate 8 is shown. The device 2 need not include this specific arrangement of electrodes 30, however, and the medium 7 may instead be interposed between the electrodes 30 and the substrate 8. Expressed differently, the electrodes 30 may be disposed on an upper surface of the medium 7 opposite to the substrate 8. Alternatively, the electrodes 30 may be arranged to be adjacent to the medium 7, that is, offset from the medium 7 in a plane in which the medium 7 at least predominantly lies.

[0236] The electrodes 30 can be arranged to define a 2D grid (or "pixel grid"). This arrangement can improve the ease with which the electrodes 30 can be addressed.

[0237] In a second mode for training the device 2, the electrodes 30 can be used to apply a locally reconfigurable electric field to the medium 7 and thereby can allow for local electrical control of the medium 7.

[0238] Referring to Figure 14, a device 2 including magnetic elements 31 disposed on an upper surface of the medium 7 is shown.

[0239] The device 2 need not include this specific arrangement of magnetic elements 31, however, and the medium 7 may instead be interposed between the magnetic elements 31 and the substrate 8. Expressed differently, the magnetic elements 31 may be disposed on an upper surface of the medium 7 opposite to the substrate 8. Alternatively, the magnetic elements 31 may be arranged to be adjacent to the medium 7, that is, offset from the medium 7 in a plane in which the medium 7 at least predominantly lies.

[0240] The magnetic elements 31 can be arranged to define a 2D grid. This arrangement can improve the ease with which the magnetic elements 31 can be addressed.

[0241] In a third mode for training the device, the magnetic elements 31 can be used to apply a locally reconfigurable magnetic field across the medium 7 and thereby can allow for local magnetic and / or magneto-plasmonic control of the medium 7.

[0242] As will be discussed hereinafter, a fourth mode for training the device 2 includes applying heat to the medium 7.

[0243] Referring to Figure 15, a method of training the device 2 will now be described. The method includes: initiating training (step S2.1); determining a loss function of the device 2 (step S2.2); making a determination of whether the loss function has reached a stable minimum (step S2.3); in the case of a negative determination, modulating an optical response in the device 2 (step S2.4) and repeating steps S2.2 and S2.3 in that order; in the case of a positive determination, ending training (step S2.5).

[0244] Determining a loss function of the device 2 can include assessing task performance of the device for a given task and generating the loss function based on the task performance.

[0245] The precise choice of loss function will depend on the task for which the device 2 is being trained. One example of a loss function suitable for image classification is cross entropy loss. Another example would be to maximise specific modes which correspond to an input class, whilst minimising mode intensity of other modes.

[0246] Modulating an optical response in the device 2 means that the interaction(s) of the spatially-structured illumination 4 with and within the medium 7 are modified, resulting in a change in spatially-structured response 6 provided for a given spatially- structured illumination 4 input.

[0247] In the case that accuracy is low, the loss function is high. In the case that accuracy is high, the loss function is low. The higher the loss, the greater the change in the physical response between iterations of the training. After a number of iterations of the method have been carried out, the loss reaches a stable minimum and the computational performance of the device 2 can be described as optimised for the given computational task. In other words, in the case that the loss is at a stable minimum (at least locally), the device 2 can be described as being trained for the given computational task.

[0248] Modulating the optical response in the device 2 includes adjusting a field applied to the medium 7. The field can be applied to the medium 7 locally or globally. Expressed differently, the field can be applied to only a portion of the medium 7 or to the entirety of the medium 7.

[0249] In the first mode for training the device 2, the field takes the form of a mask taking the form of the spatially-structured light beam from the second light source that is projected onto at least a part of the medium 7. The spatially structured light beam from the second light source can be spatially-structured by a DMD or by a SLM, by optical interference or holography, or by spatial focussing. In the first mode for training the device, a trained grid of pixels in the mask are superimposed over the spatially-structured illumination 4 on the medium 7 to modulate the complex refractive index of the medium 7. Expressed differently, in the first mode of training the device 2, a grid of pixels in a mask taking the form of the spatially-structured light beam from the second light source is used to train the medium 7. Alternatively, instead of using a spatially-structured light beam from the second light source, the trained grid of pixels in the mask may instead be superposed with the spatially- structured illumination (see also Figures 32A to 33G and the corresponding description).

[0250] In the second mode for training the device 2, the electrodes 30 are used to apply a field that is an electrical field to the medium 7. In modes of operation of the device 2 which include lasing modes, the electrodes 30 can be used to apply a reconfigurable electric field that modulates a lasing threshold of the medium 7. In modes of operation of the device 2 based on photoluminescence, the electrodes 30 can be used to apply a reconfigurable electric field that modulates how the spatially-structured illumination 4 interacts with different regions of the medium 7. In this way, different regions of the medium 7 can respond with higher or lower, and / or shifted, sensitivity to the spatially- structured illumination 4.

[0251] In the third mode for training the device 2, the field is a magnetic field and the magnetic elements 31 are used to exert magnetic and / or magnetoplasmonic control 7 over the medium 7. Each magnetic element 31 has a magnetic state that locally modifies how much of the spatially-structured illumination 4 is absorbed and / or enhanced. The magnetic states can be reconfigurably programmed in a wide variety of ways such as by an applied magnetic field, by current / voltage control, and / or by all- optical magnetic switching.

[0252] In the fourth mode for training the device 2, the field takes the form of heat applied to the medium 7. The heat can be applied to the medium 7 using a heater such as an on- chip heater or by local heating such as by laser heating. Alternatively, the heating may be provided by environmental temperature in such a way that the device 2 is a sensor that varies its physical dynamics in response to changes in its environment. Such a sensor can be applied in atmospheric or weather sensing and / or in vehicular or process control sensing. Computational performance of the device 2

[0253] The computational performance of the device 2 has been assessed using the MNIST handwritten digit classification task.

[0254] Referring to Figure 16, examples of correct (top) and incorrect (bottom) classifications of handwritten digits are shown.

[0255] The errors are dominated by ambiguous digits that human readers may struggle with.

[0256] Referring also to Figure 17, a test accuracy score of 97.7% determined using the device 2 and by the MNIST handwritten digit classification task is shown.

[0257] The test accuracy score of 97.7% was calculated from the confusion matrix shown in Figure 17 and based on classifying 10,000 images, 233 of which were classified incorrectly and 9767 of which were classified correctly.

[0258] Due to floating point precision, the sum of the values in each row and column shown is not always exactly 1.000. For example, the sum of the values in the row having an actual label of 0 is equal to 1.000, the sum of the values in the row having an actual label of 1 is 1.001, and the sum of the values in the row having an actual label of 8 is 0.999.

[0259] Referring also to Figures 18A to 18F, a comparison of optical physical computing using a device provided with raw image data with logistic regression performed on the same raw image data is shown.

[0260] In Figure 18A, an exemplary image used in the MNIST handwritten digit classification task is shown. In Figure 18B, a plan view of an example of the photonic network 11 is shown. The image shown in panel a was one of the 10,000 images projected onto the device 2 whilst the MNIST handwritten digit classification task was being performed. In Figure 18C, a spatially-resolved spectrum read out by the imaging spectrometer 5 is shown. The upper line profile indicates data from a defined spatial sub-region of the medium 7 and corresponds to the region of Figure 18C within the light rectangular box The lower line profile indicates data from the entirety of the medium 7 and corresponds to the region of Figure 18C within the dark square box. Logistic regression is performed on the readout from the imaging spectrometer 5 by the computer system 21 and, in Figure 18D, an accuracy of 98.15% is calculated from the confusion matrix shown. In Figure 18E, logistic regression on the same raw image data as was used in the MNIST handwritten digit classification task was performed directly and a lower accuracy of 90.5% is calculated from the confusion matrix shown. In contrast to the values in the confusion matrix of Figure 18D, the values in the confusion matrix shown of Figure 18E are not normalised.

[0261] Thus, the device 2 can provide a test accuracy score, measured by the MNIST handwritten digit classification task, equal to 98.15%. This test accuracy score is an improvement on the test accuracy score that can be achieved by directly performing logistic regression on the same raw image data.

[0262] Specific examples of neuromorphic computing schemes including the device 2 DNN

[0263] Hereinbefore it has been described that device 2 can be used in a fully trainable DNN for inference. In particular, the device 2 can be configured as a layer or layers in a neural network such as a CNN.

[0264] Referring to Figure 19, a first network layer li and a second network layer I2 are shown.

[0265] The first network layer li and the second network layer I2 are each modified versions of the first system 1 and each include a device 2i, 22.

[0266] The first network layer li is different to the first system 1 only in that it does not include an imaging spectrometer 11. The second network layer I2 is different to the first system 1 in that the spatially structured illumination provided to the device 22 is the spatially-structured response 61 of the first network layer li. The spatially structured response 61 of the first network layer li and the spatially structured response 62 of the second network layer I2 are both detected by the imaging spectrometer (not shown) of the second network layer I2.

[0267] The first network layer li and the second network layer I2 are connected in series. This need not be the case, however, and further network layers (not shown) can be included, and network layers can be connected in a feedback loop with the spatially- structured response 62 of the second network layer used as the spatially-structured illumination 4i of the first network layer li. Network layers can be connected to form feed-forward neural networks such as CNNs or can be connected to form recurrent neural networks. It need not be the case that the first network layer li and the second network layer I2 include separate devices 2i, 22. Instead, the network layer li can include the first photonic network 11 and the second network layer I2 can include the second photonic network II2. The first photonic network 11 and the second photonic network II2 are included in the same device 2 (best illustrated in Figure 3).

[0268] Combining parallel outputs, for example from the first network layer li and the second network layer I2, can help to improve computational performance.

[0269] Fifth optical image processing system 32 - stand-alone processor

[0270] Referring to Figure 20, a fifth optical image processing system 32 (hereinafter referred to as the "fifth system") is shown.

[0271] The fifth system 32 comprises only one device 2. Thus, the fifth system 32 is implemented as a stand-alone processor. The fifth system 32 can be used is an example of the first system 1 which is implemented as a stand-alone processor and comprises only one device 2.

[0272] The device 2 of the fifth system 32 can be used either as a reservoir or as a network layer in a deep neural network (DNN).

[0273] Sixth optical image processing system 33 - pre-processor

[0274] Referring to Figure 21, a sixth optical image processing system 33 (hereinafter referred to as the "sixth system") is shown.

[0275] The sixth system 33 is a generalised version of the example DNN example described hereinbefore as including the first network layer li and the second network layer I2. In particular, the sixth system 33 includes two network layers connected in series, only the first of which includes a device 2.

[0276] In the sixth system 33, the device 2 receives the spatially-structured illumination 4 as an input and provides the spatially-structured response 6 as an output. The second network layer 34 receives the spatially-structured response as an input and provides an output 35.

[0277] In this way, the device 2 is used within the system 33 as a pre-processor. Seventh optical image processing system 36 - post-processor

[0278] Referring to Figure 22, a seventh optical image processing system 36 (hereinafter referred to as the "seventh system") is shown.

[0279] The seventh system 36 is a modified version of the sixth system 33 in which the device 2 is used as a post-processor instead of a pre-processor.

[0280] In the seventh system 36, a first network layer 37 receives an input 38 and provides an output 39. The device 2 receives the spatially-structured illumination 4 as an input, the spatially-structured illumination provided based on the output 39 and provides the spatially-structured response 6 as an output.

[0281] In these ways, the device 2 can be used as a pre / post-processor as part of a larger computing system. As described hereinbefore, using the device 2 as a network layer in combination with a further network layer including for example a memristive network or a nanomagnetic network can add memory and thereby can allow the device 2 to be used for video processing.

[0282] In the sixth system 33 and the seventh system 36, network layers need not be connected in series, and two or more network layers may be connected in parallel.

[0283] Edqe / feature detection

[0284] The device 2 can be used to perform convolutional edge detection directly using its internal physics, with no requirement for regression or post-processing. In particular, the first system 1 can serve as both the convolutional kernel layer and classifier layers of a convolutional neural network. The kernel layer can be used as a standalone edge- detection / feature-extraction system, which treats the optical power of each lasing mode as a separate feature extraction kernel. A first optical mode at one wavelength may respond more strongly when vertical edges are present, a second optical mode at another wavelength may respond more strongly when horizontal edges are present. In this way, mode powers can be used to construct feature maps, either with or without prior training / regression of the kernels. The edge / feature detection functionality can act as a layer within a larger image processing and / or computing system, such as an initial kernel layer of a convolutional neural network. Alternatively, the edge / feature detection functionality can act as a standalone processor which extracts edges / features from images and returns maps of the edges / features as its output. Referring to Figures 23A to 23C, examples of an input image (the letters ICL) (Figure 23A) and extracted feature / edge maps provided by two different optical modes 10 (Figures 23B and 23C) are shown.

[0285] Each of Figures 23B and 23C show contrast corresponding to edges or features of the input image of Figure 23A.

[0286] Feature maps were generated using the following procedure:

[0287] • The image (Figure 23A) was split into multiple different windows (for example 5 x 5 pixels).

[0288] • Each such window was illuminated onto the medium and the response recorded.

[0289] • The amplitude of a particular mode for that window was taken as the output.

[0290] • The process was repeated over all possible 5 x 5 windows in the input image, allowing generation of the 'feature-map' i.e. the amplitude of a particular mode across each 5 x 5 input.

[0291] This is similar to how a convolution neural network works in which a kernel is raster scanned across an entire image. Of course, the nature of the processing in this instance by the interaction of spatially-structured illumination 4 with the medium 7, differs significantly compared to execution of a kernel to process an image using a conventional floating point processor or graphics processing unit (GPU).

[0292] Further experimental data of other examples of edge-detection in which regression has been used to train which of the optical modes are used to train edge maps is shown in Figure 23D.

[0293] A single device 2 can provide many kernels in parallel by multiplexing kernels in the spectral frequency space via the mode / kernel correspondence.

[0294] Referring also to Figures 24A to 24D, examples of detecting all four edges using four different optical modes 10 (one in each of Figures 24A to 24B) is shown.

[0295] Each of Figures 24A to 24D show contrast indicating at least one of the edges of a simulated square.

[0296] Thus, the device 2 can be for edge detection and feature detection. Applications

[0297] The device 2 can be used in a wide variety of computer vision applications ranging from autonomous piloting of vehicles to object detection.

[0298] In a first example, the device 2 is used in a self-driving car in which image data from car mounted cameras is projected optically onto at least one device 2 that is preconfigured and trained for hazard detection. In this application, where response time is critical, the rapid processing speed of the device 2, relative to conventional software neural network processing, can be particularly beneficial. A self-driving car including the device 2 can respond to observed oncoming hazards by providing a spectral readout which includes one or more characteristic features indicative of a warning, that spectral readout acting as a trigger for activating evasive manoeuvres in the selfdriving circuits, keeping passengers and pedestrians safe. In related examples, the device 2 can be used in another other autonomous vehicle, such as an autonomous drone, an autonomous aircraft, or an autonomous sea vessel. By providing a processor such as the computing system 21 for determining variations in the spectral readout of the spectrometer 20, the device 2 can be particularly suitable for use in such remote non-data-centre use cases.

[0299] In a second example, the device 2 is included in a CCTV camera processing system which can respond to observing certain objects, such as crowds or faces, by providing a spectral readout which includes one or more characteristic features indicative of the object(s) observed, similarly to how the warning spectra is provided in the self-driving car. In related examples, the device 2 can be included in another processing system, such as an industrial manufacture control system, a medical imaging system, or a surgical control system for example for processing endoscope camera image, and / or video data.

[0300] The device 2 can also be used in a wide variety of other machine learning applications in which data for non-linear processing can be mapped onto an image.

[0301] Optical response modulation using electric fields

[0302] A device including electrodes 30 for applying a locally reconfigurable electric field to the medium 7 is shown schematically in Figure 13 and described hereinbefore.

[0303] Examples of mediums 7 described herein may be excited to support hyperspectral modes which are spatially distributed across multiple network waveguides and interfere with each other in a complex manner. As described herein, even inducing small changes in the properties (for example refractive index) of the material(s) of the medium 7, or portions / regions of the medium 7, can result in significant changes in the output signals. This opens up the prospect of the medium 7, for example a photonic network 11, being dynamically reconfigurable.

[0304] Small variations in the refractive index of, for example a segment of a photonic network 11 corresponds to an extension or reduction of that optical path length seen by an optical mode intersecting that segment. This will impart a difference in the optical phase within this segment, influencing the interference with overlapping and / or closely adjacent optical modes.

[0305] Unlike conventional modulators or linear networks based on Mach-Zehnder interferometers, modulating of the optical response of the medium 7 does not require a full n (180°) phase-shift. It is not necessary to create destructive interference, but simply to impact the complex interference patterns between the overlapping optical modes excited by a given spatially-structured illumination. A small variation of the complex refractive index should therefore be sufficient. A number of methods can be applied to mediums 7 described herein to create such local variations of the refractive index (some already described hereinbefore), including but not limited to: thermal, electric field (Pockels & Kerr effect), plasma dispersion by free carrier injection or magnetic field modulation.

[0306] For example, referring also to Figures 25A and 25B a modulator having a first configuration 101 of electrodes 30 for electro-optic modulation of the medium 7 is shown. Figure 25A is an annotated plan-view SEM micrograph of a medium 7 in the form of a photonic network 11 patterned from III-V semiconductor material. Figure 25B is a schematic projection view of the first configuration 101.

[0307] A dielectric layer 102 is supported on the substrate 8. For example, the substrate 8 may be a Si carrier wafer and the dielectric layer 102 may be a buried oxide layer. A photonic network 11 formed of network links 12 joined (or closely space) at nodes 13 is supported on the dielectric layer 102. For example, the photonic network 11 may be patterned from InP.

[0308] Electrodes 30 are patterned in pairs, each pair bracketing at least a portion of the length of a network link 12. The number of network links 12 which have a corresponding electrode pair may vary from a single network link 12 up to every network link. The inventors have observed that changing even a single network waveguide can produce a considerable effect on the whole network 11. In practice, for a given task and photonic network 11, a balance will need to be found between complexity of fabricating and addressing the device 2 and the degree of modulation of the spatially-structured response 6 obtainable for the same spatially-structured illumination 4 by varying applied electric fields.

[0309] Optionally, voids 16 may be formed in network links 12 which are bracketed by an electrode pair 30. Voids 16 may take the form of physical voids, or may be inclusions of other materials / regions of differing refractive index. Voids 16 may introduce optical losses, whereby light is lost from waveguides such as network links 12 which connect the voids 16. This loss provides additional nonlinearity which can be useful for processing / computation. Additionally or alternatively, voids 16 may be arranged to embody a photonic crystal mirror. A combination of photonic crystal mirrors formed by appropriately arranged voids 16 could be used to form a resonant structure enhancing the modulation efficiency of a device including the first configuration 101. The voids 16 are not necessary for operation of the first configuration 101.

[0310] A potential difference 1 / applied between a pair of electrodes 30 generates a localised electric field E across the corresponding network link 12. The potential difference 1 / is applied via conductive traces (not shown) connecting to each electrode 30 and allowing individual addressing of each pair. For example, one electrode 30 of each pair may be connected to system ground whilst the other electrode is connected to a signal output. This is only one example and alternative addressing schemes may be used.

[0311] The electric field E applied across the network link may give rise to a localised Pockels effect and / or a localised Kerr effect.

[0312] The Pockels effect (or linear electro-optic effect), is proportional to the magnitude of the applied electric field, |E | : In which Aripockeis is the induced change in the refractive index n, no is the unperturbed (zero electric field) refractive index and nj is the third-rank electro-optical tensor of the Pockels coefficient.

[0313] LiNbCh is commonly used for obtaining the Pockels effect in commercial high-speed electro-optic modulators. The typically lower magnitudes of nj components in III-V semiconductors such as InP may be partially compensated by a higher value of the refractive index, no. Hence comparable effective values of modulation may be achieved in InP as in LiNbCh.

[0314] InP has the further advantage of being much easier to pattern, dope and contact. Hence, it is possible to exploit photonic design techniques to optimize optical mode 10 overlap with the modulated region (between the electrodes 30). InP network links 12 may exploit the Kerr effect which is often smaller (though relatively large in InP), and depends on the quadratic field | E |2. Therefore if the field is also concentrated, for example by the use of a photonic crystal (e.g. patterned using voids 16), then the Kerr effect may also be used to contribute to modulation of the refractive index.

[0315] Pockels modulators, as illustrated in the first configuration 101 of Figures 25A and 25B, are electric-field based. Gating is sufficient and no contacts are needed to the network link 12, which can significantly simplify fabrication. Modulators having the first configuration 101 can be embedded with individual network links 12, to provide local phase modulation of optical modes 10 using that network link. Modulators having the first configuration 101 may be combined with photonic crystal designs, for example using voids 16 (or other inclusions) to pattern a mirror or resonator.

[0316] By applying a bias across selected network links 12, the refractive index n within the network link is varied (at least in the region affected by the electric field E). Consequently, the interference pattern excited by a given spatially-structured illumination 4 is modified, and hence the corresponding spatially-structured response 6 (for example, a hyperspectral lasing mode). Modulators, for example arranged according to the first configuration 101, may therefore act as reconfigurable physical weights within a medium 7 used to physically implement a neuromorphic computation. For example in a random lasing network.

[0317] The field effect is inherently fast, operable up to frequencies of GHz or more. Even direct without contacts to the InP which will facilitate carrier recombination, speeds in the MHz regime should be possible. This refers to the speed of changing the modulation applied to the medium 7 from one state to a different state - the speed of the medium 7 processing the spatially-structured illumination 4 to the spatially- structured response 6 does not deviate from the inherently high speed described hereinbefore.

[0318] Referring also to Figures 26A to 26G, an exemplary method of fabricating a modulator having the first configuration 101 is schematically illustrated.

[0319] Referring in particular to Figure 26A, a uniform layer 103 is deposited over a dielectric layer 102 supported by a substrate 8. The substrate 8 may be of any material discussed herein, such as, for example silicon, Si. The dielectric layer 102 may be any insulating material such as, for example, silicon dioxide, SiCh, having a thickness between 0.5 and 2 pm. The uniform layer 103 may be formed from any material suitable for pattering to provide the medium 7, such as, for example, a III-V semiconductor. Examples include indium phosphide, InP, gallium arsenide, GaAs, and so forth. In general the uniform layer 103 may be formed from any material described herein for providing the medium 7 and which can be patterned by photoresist etching. Preferably, the material forming the uniform layer 103 should both:

[0320] • be able to act as active gain material, i.e have a direct bandgap; and

[0321] • possess a Pockels or Kerr coefficient, (silicon, for example, would be unsuitable).

[0322] The uniform layer 103 may have been pre-formed over the dielectric layer 102 in advance of starting the fabrication process. In other words, the starting point may be a laminate formed from the substrate 8, the dielectric layer 102 and the uniform layer 103. One or more additional layers (not shown) may be provided between any pair of the illustrated layers for a variety of purposes including, but not limited to, diffusion barriers, improving lattice matching, the creation of quantum wells, improving adhesion, and so forth.

[0323] A first photoresist layer 104 is deposited over the uniform layer 103. Commonly used photoresist materials include polymers, or an oxide such as hydrogen silsesquioxane (HSQ). The first photoresist layer 104 is patterned to have the desired shape of a photonic network 11.

[0324] Referring in particular to Figure 26B, the uniform layer 103 is etched to form the photonic network 11 (formed of network links 12 and nodes 13) having the same outline as the first photoresist layer 104. Although shown as being etched completely through the uniform layer 103 in Figure 26B, in some examples the etching does not need to be deep enough to expose the dielectric layer 102: the network may be patterned on top of a residual uniform layer (not shown).

[0325] Referring in particular to Figure 26C, the first photoresist layer 104 is removed, and a second dielectric layer 105 is deposited over the network 11 and in the gaps between the network 11. In other words, the second dielectric layer 105 may be described as a "blanket" layer. The second dielectric layer 105 may be an oxide layer such as, for example, silicon dioxide.

[0326] Referring in particular to Figure 26D, a second photoresist layer 106 is deposited over the second dielectric layer 105. The second photoresist layer 106 is patterned as the inverse (or negative) of the desired pattern of electrodes 30 and any connecting traces (not shown) where used. Parts of the network 11, for example some network links 12 and / or nodes 13 which are not desired to have proximate electrodes 30 for modulation, may be entirely covered by the second photoresist layer 106 (see for example the portion of the network 11 shown in the right hand side of Figure 26D).

[0327] Referring in particular to Figure 26E, electrodes 30 are deposited within the gaps of the second photoresist layer 106 using a lift-off process. Electrode material deposited over the second photoresist layer 106 is not shown, since this will be removed along with the second photoresist layer 106. The electrodes 30 may be formed from any suitable conductive material such as, for example, gold, Au, aluminium, Al, indium tin oxide, ITO, and so forth.

[0328] Referring in particular to Figure 26F, the second photoresist layer 106 is then removed to complete the lift-off process, leaving electrodes 30 for modulation in accordance with the first configuration 101.

[0329] Referring in particular to Figure 26G, the network 11 and electrodes 30 may be covered over by a third dielectric layer 107 providing electrical insulation and protecting the network (and optionally also serving for planarization of the device).

[0330] A top metallization layer 108 is connected to electrodes 30 using vias 109 formed through the third dielectric layer 107, allowing addressing of the electrodes 30. The third dielectric layer 107 may be an oxide layer such as, for example, silicon dioxide. Only a contact to one of the electrodes 30 is shown in Figure 26G. The contact to the other electrode 30 lies outside of the illustrated cross-section plane.

[0331] In other examples, some or all connections to the electrodes 30 may be made using conductive traces (not shown) supported on the second dielectric layer 105. In still further examples, some of all of the connections to the electrodes may be made using via (not shown) extending from the underside of the substrate 8 to the network 11 (in other words, from the opposite face of the substrate 8 to the dielectric layer 102. This may help to reduce reflection of the spatially-structured illumination from structures contacting the electrodes 30.

[0332] The materials used for the second dielectric layer 105 and the third dielectric layer 107 should be transparent to wavelength ranges corresponding the spatially-structured illumination 4 and the spatially-structured response 6 (these two wavelength ranges may differ and need no overlap).

[0333] In the first configuration 101, electrodes 30 are isolated from the network 11 material by second dielectric layer 105. The electrodes 30 bracketing a given network link 12 form a capacitor, applying a broadly uniform electric field across that network link 12. In other examples, the second dielectric layer 105 may be omitted, and electrical contact formed between the electrodes 30 and the network 11 material. In this way, the electrodes 30 bracketing a given network link 12 may inject charge carriers, opening up alternative mechanisms for modulation of the optical response. In such examples, the electrode 30 materials should be chosen to provide an appropriate type of contact, for example ohmic, rectifying and so forth, and the electrodes 30 forming a single pair may be formed from different materials. Different materials for electrodes 30 are also possible in the first configuration 101, but would serve only to complicate fabrication.

[0334] Modulator having a second configuration

[0335] Referring also to Figures 27A to 27E, fabrication and structure of a modulator having a second configuration 110 of electrodes 30 for electro-optic modulation of the medium 7 are shown. Figures 27A to 27D are schematic cross-sections illustrating stages of the fabrication of the second configuration 110, with Figure 27D showing the fabricated second configuration 110 of electrodes 30 for electro-optic modulation. Figure 27E is a schematic plan view of the second configuration 110 of electrodes 30 for electro-optic modulation. Figure 27D corresponds to a cross-section along the line labelled B-B' shown in Figure 27E. Materials for substrate 8, dielectric layer 102, network 11 and electrodes 30 and so forth are the same as described in relation to the first configuration 101.

[0336] Referring in particular to Figure 27A, a uniform layer 103 is deposited over a dielectric layer 102 supported by a substrate 8, in the same way described for the first configuration 101 (see also Figure 26A and corresponding description).

[0337] Referring in particular to Figure 27B, electrodes 30 are deposited over the uniform layer 103. The electrodes 30 may be patterned using any suitable technique, including but not limited to, lift-off and selective etching methods.

[0338] Referring in particular to Figure 26C, a first photoresist layer 104 is deposited over the uniform layer 103 and electrodes 30. In the same way as for the first configuration 101, the first photoresist layer 104 is patterned to have the desired shape of a photonic network 11 (though the precise topology of the photonic network 11 may differ).

[0339] Referring in particular to Figures 26D and 27E, the uniform layer 103 is etched to form the photonic network 11. After the etch, the first photoresist layer 104 is removed, leaving the fabricated second configuration 110 of electrodes 30. Similarly to the first configuration 101, although the etch is shown is extending entirely through the uniform layer 103, this is not always essential and in other examples a residual uniform layer (not shown) thinner than the network 11 may be left.

[0340] The network 11 is etched so that modulation regions 111 supporting the electrodes 30 are left wider than the rest of the corresponding network link 12. For example, as shown for a first network link 12i shown in Figure 27E. The modulation region 111 is a part of the first network link 12i.

[0341] As described hereinbefore, modulation electrodes 30 need not be formed for every network link 12. For example, second to fourth network links 12z, 123, 124 also shown in Figure 27E (and coupled at node 13) may include no modulation electrodes 30. Equally, some or all of the second to fourth network links 12z, 123, 124 could include modulation electrodes outside of the portion of the network 11 which is illustrated.

[0342] In the same way as the first configuration 101, electrodes 30 may alternatively be configured for charge injection in the second configuration 110. Modulator having a third configuration

[0343] The first configuration 101 and second configuration 110 described hereinbefore require prior knowledge of both the locations of network links 12 and also which network links 12 are to be bracketed by electrodes 30.

[0344] However, configurations of electrodes 30 for electro-optic modulation are not limited to electrode pairs positioned either side of a network link 12.

[0345] Referring also to Figures 28A to 28E, fabrication and structure of a modulator having a third configuration 112 of electrodes 30 for electro-optic modulation of the medium 7 are shown. Referring also to Figures 29A to 29C, schematic plan views are shown corresponding to Figures 28A, 28C and 28E respectively.

[0346] Materials for substrate 8, dielectric layer 102, network 11 and electrodes 30 and so forth may be the same as described in relation to the first configuration 101.

[0347] However, as at least one set of electrodes 30 are preferable transparent, potentially enabling a wider choice of materials.

[0348] Referring in particular to Figures 28A and 29A, first electrodes 30i are deposited over the dielectric layer 102 supported by substrate 8. Only the shapes of the first electrodes 30i are illustrated in Figure 29A for visual clarity.

[0349] The first electrodes 30i extend in a first direction (x-axis shown in Figure 29A) parallel to one another, and are spaced apart (for example regularly spaced) in a second, perpendicular direction (y-axis shown in Figure 29A). The first electrodes 30i may be pattered using any suitable method, including, but not limited to, lift-off, etching, sputtering through a mask, transfer printing and so forth.

[0350] Referring in particular to Figure 28B, additional dielectric material is deposited to extend the thickness of the dielectric layer 102, cover the first electrodes 30i and planarize the top surface. This leaves the first electrodes 30i embedded within the dielectric layer 102.

[0351] In order to minimise fabrication complexity the additional dielectric material is preferably the same as the original dielectric layer 102, as illustrated in Figure 28B. However, since the function is primarily for planarization of the top surface, though it may also serve as electrical insulation when the medium 7 is conductive (for example Indium based III-V materials) and charge carrier injection is not desired. It is equally to use a different dielectric material, resulting in a multi-layer dielectric (not shown) with the electrodes 30 embedded at the interface (not shown).

[0352] Referring in particular to Figures 28C and 29B, the medium 7 is deposited over the dielectric layer 102 having the first electrodes 30i embedded therein.

[0353] Any method for producing the medium described herein may be used. For example, the medium 7 may take the form of a photonic network 11 as illustrated in Figures 28 and 29. The photonic network 11 may be produced as described herein, for example by lithographic methods, by electrospinning and post-annealing of dye-mixed polymer nanofibers or semiconductor nanowires, or in any other way able to produce a medium 7 as described herein.

[0354] Referring in particular to Figure 28D, a second dielectric layer 105 is deposited over the photonic network 11. Unlike the thin second dielectric layer 105 of the first configuration 101, in the third configuration the second dielectric layer 105 may be thicker, and is preferably at least partly planarizing.

[0355] Referring in particular to Figures 28E and 29C, second electrodes 30i are deposited over the second dielectric layer 105. The substrate 8, dielectric layer 102 and second dielectric layer 105 are omitted in Figure 29C for visual clarity of the relative positions of the electrodes 30i, 302 and network 11.

[0356] The second electrodes 302 extend in second direction (y-axis shown in Figure 29C) parallel to one another, and are spaced apart (for example regularly spaced) in the first direction (y-axis shown in Figure 29C). The second electrodes 302 may be pattered using any suitable method, including, but not limited to, lift-off, etching, sputtering through a mask, and so forth.

[0357] At least one of the sets of electrodes 30i, 302 should be transparent. For example, when illuminated from above the substrate 8, the second electrodes 302 should be transparent. Transparent electrodes 30, 30i, 302 may be produced using, for example, transparent conductors such as ITO, thin conductor-like graphene, and so forth.

[0358] In this way, the first electrodes 30i and the perpendicular second electrodes 302 form a grid. Local modulation of the optical response of the medium 7 may be implemented by connecting one or more first electrodes 30i and one or more second electrodes 302 to different potentials.

[0359] Additionally, in the third configuration 112, the medium 7 is not limited to a photonic network 11 as illustrated in Figures 28 and 29. The medium 7 could take the form of a uniform film of a material having a complex refractive index which can be modulated using applied electrical fields generated in a pattern using the electrodes 30i, 302.

[0360] More complex grid arrangements of electrodes 30 may be considered as variations of the third configuration 112. For example, an upper electrode layer could be a uniform, transparent counter electrode connected to system ground or any other fixed potential. A lower electrode layer could be controlled by a transistor layer (for example a thin-film transistor, TFT, layer), so that each lower electrode forms a storage capacitor with the upper electrode as a common. Analogous circuits and addressing schemes known from TFT liquid crystal displays could be used to set each lower electrode to a corresponding potential, effectively allowing the electro-optic modulation to be applied as an "image", immediately prior to input of the spatially- structured illumination.

[0361] In the same way as the first configuration 101, electrodes 30 may alternatively be configured for charge injection in the third configuration 112.

[0362] Simulations of refractive index modulation

[0363] Referring also to Figure 30A, a schematic of low-density photonic network is shown, having network links 12 with an average length of 7 pm. Referring also to Figure 30B, changes in the simulated lasing spectrum are presented for simulated refractive index shifts, having different magnitudes for different network links 12. A maximum refractive index shift of An = 0.05 was used, with the relative refractive index shift An applied to each network link 12 determined by the angle 0 of that network link 12 relative to the y-axis (vertical as illustrated in Figure 30A), and using cos(0) calculated for this angle 9 to obtain a scaling factor between 0 and 1. This scaling factor was multiplied by the maximum index shift of 0.05 to obtain the refractive index shift An for each network link 12. These refractive index shifts are achievable via any of the modulation methods described herein, including electrical field and magnetic field, with a modulation of An = 0.05 being experimentally achievable.

[0364] The simulations were performed by solving Maxwell's equations for light through the network 11 under uniform pump illumination (e.g. all of the network is uniformly illuminated). The equations were solved to identify the lasing mode frequencies and amplitudes, producing lasing spectra for the network. InP was simulated as the network material.

[0365] Referring also to Figure 30B, the lasing spectra calculated from the simulations of the network 11 shown in Figure 30A are presented for a first case 113 of the refractive index shift varying up to a maximum of An = 0.05, and for a second case 114 in which a uniform (unmodified) refractive index was used for every network link 12. It may be observed that the simulated refractive index shifts An is sufficient to generate a clearly resolvable shift between the lasing spectra 113, 114.

[0366] Referring also to Figure 31A, a schematic of high-density photonic network is shown, having network links 12 of length 2 pm on average. Referring also to Figure 31B, changes in the simulated lasing spectrum are presented for simulated refractive index shifts, having different magnitudes for different network links 12. A maximum refractive index shift of An = 0.05 was again used, with the relative refractive index shift An applied to each network link 12 determined as described in relation to Figures 30A and 30B.

[0367] Referring also to Figure 31B, the lasing spectra calculated from the simulations of the network 11 shown in Figure 31A are presented for a first case 113 of the refractive index shift varying up to a maximum of An = 0.05, and for a second case 114 in which a uniform (unmodified) refractive index was used for every network link 12.

[0368] From a comparison of Figures 30A and 30B to Figures 31A and 31B, it may be observed that the two networks 11 of differing link 12 densities demonstrate the control which can be exerted over network modulation by using the network topology (lower density with 7 pm average waveguide length as shown in Figure 30A, and higher density with 2 pm average waveguide length). For the lower density network (Figure 30A), the refractive index n modulation An produces a shift of mode frequency. However, for the higher density network (Figure 31A), the refractive index n modulation An generates more complex spectral changes, including creating new lasing modes. This demonstrates that via network topology / design, it is possible to control and influence the response of a network 12 to modulation.

[0369] Simulations of modulation using a structured light mask

[0370] Simulations were performed to demonstrate modulation using a structured light mask. These simulations were performed using a 'digital twin' of an experimental network 11, whereby a neural network was trained on experimental data to model the network 11 response under different spatially-structured illuminations 4.

[0371] Mask illumination patterns were generated to improve classification accuracy, with the mask design determined using a genetic algorithm training process. The mask illumination (shown in Figure 33B) has a number of mask pixels 120 (Figure 33B) within an illumination area 115 set to a maximum pump illumination intensity.

[0372] Instead of using a second light source to supply a structured light beam, in the following examples the mask pixels 120 were superimposed on image / data inputs in the spatially-structured illumination 4 input to the network.

[0373] Referring also to Figures 32A to 32G a baseline case without a structured light mask is illustrated. The case using the structured light mask is then illustrated referring also to Figures 33A to 33G.

[0374] Referring in particular to Figure 32A, an example of an input image used to pattern the spatially-structured illumination 4 is shown. The image is shown was inverted for application to the network 12. In other words, white areas of the image corresponded to low pump illumination intensity when projected onto the network, and black areas of the image corresponded to high pump illumination intensity when projected onto the network.

[0375] Referring in particular to Figure 32B, the illumination area 115 is shown as a dashed white outline superimposed on an experimental SEM image of the network 11 topology which was used to obtain the experimental data for training the digital twin. The shading of network links 12 shown in Figure 32B corresponds to the |E |2(electric field intensity squared) of a single mode, calculated by solving Maxwell's equations, superimposed on the experimental scanning electron microscope image of the network 11, showing an example of the shape / size of one of the optical modes 10. The optical mode 10 shown includes regions 116 of relatively low intensity and regions 117 of relatively high intensity.

[0376] Referring in particular to Figure 32C, simulated lasing spectra are shown for the network 11 shown in Figure 32B, in response to spatially-structured illumination 4 corresponding to the image shown in Figure 32A, without masking. The "Spatial" axis corresponds to the vertical direction of Figure 32B. The shading in Figure 32C corresponds to the intensity (A.U.) of light detected for the corresponding wavelength and spatial (vertical) position. The response includes regions "L" of relatively low intensity and regions "H" of relatively high intensity.

[0377] Referring in particular to Figures 32D to 32G, simulated lasing spectra calculated in the same was as spectra shown in Figure 32C are shown, to compare responses different representations of the same digits. In this case Figures 32D and 32E compare lasing spectra generated in response to different representations of the digit "2"(Figure 32D is a repeat of Figure 33C for side-by-side comparison), and Figure 32F and 32G compare lasing spectra generated in response to different representations of the digit "6". In all cases shown in Figure 32D to 32G, no masking was used, and the image providing the spatially-structured illumination 4 is reproduced above the corresponding spectra.

[0378] For comparison, Figures 33A to 33G present simulation results for the same network 11 and input images, for the case that the spatially-structured illumination 4 was superimposed with a pattern of mask pixels 120 trained to improve classification accuracy. Each mask pixel 120 corresponded to maximum pump-illumination intensity. Referring in particular to Figure 33B, the pattern of mask pixels illustrated was the pattern used for generating the simulated lasing spectra presented in Figures 33C to 33G.

[0379] Comparing Figures 32C (unmasked) and 33C (masked), it may be observed that the spatial-spectral response is more focused. Moreover, comparing Figures 32D to 32G (unmasked) with Figures 33D to 33G (masked) respectively, it may be observed that the responses in the case of using the trained mask were not only more focused compared to the unmasked cases, but also exhibit a higher degree of similarity in the responses to different representations of the same digit.

[0380] The simulated spectra with (Figures 33C to 33G) and without (Figures 32C to 32G) masking demonstrate that modulation of the optical response of a medium 7 can be used to reduce, or even remove entirely, the need for a software regression step to interpret the spatially-structured response.

[0381] Without wishing to be bound by theory, it is believed that the pattern of mask pixels 120 directs lasing modes to particular spectral regions, allowing the region with highest amplitude to be taken as the computational output. It may be observed that the impact of masking on the network 11 response is somewhat similar to the effect of electric gating previously described in relation to Figures 30A to 31B. Similar effectiveness is expected for other modulation techniques described hereinbefore, including magnetic and thermal modulation.

[0382] In this way, using various approaches to modulate the optical response, it is possible to reconfigure / reprogram the intrinsic nonlinear physics of a medium 7 (such as a network 11), changing refractive index n, changing the mode-coupling competition landscape, and so forth.

[0383] Although the mask pixels 120 described in relation to the simulation results shown in Figures 32A to 33G were set to maximum intensity, this is not essential, and in other example a pattern of mask pixels 120 having zero intensity could be trained.

[0384] Similarly, mask pixel 120 illumination intensities are not limited to binary patterns, and a mask could utilise greyscale intensity levels (between zero and maximum). In some examples, masks is the form of images could be trained.

[0385] Modifications

[0386] It will be appreciated that various modifications may be made to the embodiments hereinbefore described. Such modifications may involve equivalent and other features which are already known in the fields of optical image processing and neuromorphic computing hardware, and which may be used instead of or in addition to features already described herein. Features of one embodiment may be replaced or supplemented by features of another embodiment.

[0387] Although an imaging spectrometer 5 has been described, in some examples a spectrometer may be used which need not be capable of imaging / spatial selectivity. For example, a spectrometer having only frequency resolution could be used to detect the spatially-structured response from all, or a single region of the medium 7 (predefined by the selection of in-coupling optics).

[0388] Although an imaging spectrometer 5 has been described, in some examples a spectrometer is not necessary, and a camera may be used instead. For example, when spatial distribution of the spatially-structured response 6 is sufficient to encode the output of a computational task. When a red-green-blue (RGB) camera is used, the differences in intensity between colour channels may, in combination with the spatial distribution of the spatially-structured response 6, provide the output of a computational task Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure of the present invention also includes any novel features or any novel combination of features disclosed herein either explicitly or implicitly or any generalization thereof, whether or not it relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as does the present invention. The applicants hereby give notice that new claims may be formulated to such features and / or combinations of such features during the prosecution of the present application or of any further application derived therefrom.

Claims

Claims1. A device for use in optical image processing, the device comprising:■ a medium capable of supporting overlapping optical modes such that, in response to spatially-structured illumination, a set of overlapping optical modes in the medium are non-uniformly excited and output a spatially-structured response.

2. The device of claim 1, wherein the medium comprises:■ an optical gain medium.

3. The device of claim 1 or 2, wherein the medium is spatially non-uniform.

4. The device of any one of claims 1 to 3, wherein the medium is configured as a panel.

5. The device of any one of claims 1 to 4, wherein the medium comprises a first photonic network.

6. The device of claim 5, wherein the medium comprises a second photonic network that is spatially separated from the first photonic network.

7. The device of any one of claims 1 to 6, wherein the medium comprises photonic waveguides.

8. The device of any one of claims 1 to 7, wherein the medium comprises lowdimensional structures.

9. The device of claim 8, wherein the low-dimensional structures comprise quantum wells.

10. The device of any one of claims 1 to 9, wherein the medium comprises a semiconductor material.

11. The device of claim 10, wherein the semiconductor material is an inorganic semiconductor material.

12. The device of claim 10 or 11, wherein the semiconductor material is a III-V semiconductor material.

13. The device of claim 10, wherein the semiconductor material is an organic semiconductor material.

14. The device of any one of claims 1 to 13, configured for modulating of an optical response of the medium.

15. The device of any one of claims 1 to 14, comprising a first set of electrodes configured for applying a locally reconfigurable electric field to the medium.

16. The device of any one of claims 1 to 15, comprising a second set of electrodes configured for injecting charge carriers to the medium.

17. The device of any one of claims 1 to 16, comprising a second light source configured to project a mask taking the form of a spatially-structured light beam onto the medium.

18. The device of any one of claims 1 to 16, wherein the device is configured to superpose a mask with the spatially structured illumination.

19. The device of any one of claims 1 to 18, comprising magnetic elements arranged to provide a locally reconfigurable magnetic field across the medium.

20. The device of any one of claims 1 to 19, comprising one or more heaters configured to vary the temperature of the medium.

21. An optical image processing system, comprising:■ the device of any one of claims 1 to 20; and■ an imaging spectrometer configured to detect the spatially-structured response.

22. The optical image processing system of claim 21, wherein the imaging spectrometer is configured to perform readout on defined spatial sub-regions of the medium.

23. The optical image processing system of claims 21 or 22, further comprising :■ a neutral density filter configured to maintain a pump power of the spatially- structured illumination.

24. The optical image processing system of any one of claims 21 to 23, further comprising:■ a digital micromirror device configured to provide the spatially-structured illumination.

25. The optical image processing system of any one of claims 21 to 23, further comprising:■ a spatial light modulator configured to provide the spatially-structured illumination.

26. The optical image processing system of any one of claims 21 to 25, wherein the optical image processing system is configured for use as a non-trainable reservoir for data processing.

27. The optical image processing system of any one of claims 21 to 26, wherein the optical image processing system is configured for use as a fully trainable deep-neural network for inference.

28. A computer vision product, comprising :■ the optical image processing system of any one of claims 21 to 27; and■ a processor configured to determine variations in spatially-resolved spectra provided by the spatially-resolved spectrometer.

29. The computer vision product of claim 28, wherein the computer vision product is a vehicle capable of being autonomously piloted or a system for object detection and classification.

30. A method of operating the device of any one of claims 1 to 20 or the optical image processing system of any one of claims 21 to 27, the method comprising: providing the spatially-structured illumination to the device; and detecting the spatially-structured response.

31. The method of claim 30, comprising modulating of an optical response of the medium.

32. A method of training the device of any one of claims 1 to 20 or the optical image processing system of any one of claims 21 to 27, the method comprising:determining a loss function of the device, determining whether the loss function has reached a stable minima, and modulating an optical response in the device.

33. The method of claim 32, wherein modulating an optical response in the device comprises adjusting a field applied to the medium.

Images