Photonic digital signal processor

Abstract

Described herein is a data processing device (100) including an input (102) for receiving an input signal including a data stream to be processed. A reconfigurable transversal filter (106) is adapted to perform one or more temporal signal processing functions on the input signal to produce one or more processed output signals (110-114). Filter (106) includes an optical frequency comb generator (108) to generate a plurality of discrete, equally spaced frequency channels (116). Filter (106) also includes a modulator (132) configured to modulate the input signal with each frequency channel (116) to generate modulated frequency channels having the input data stream encoded therein. An optical delay device (136) selectively applies a frequency dependent temporal delay to each of the modulated frequency channels such that each of the modulated frequency channels are temporally shifted by a predefined temporal period to produce delayed modulated frequency channels. Filter (106) also includes an optical spectral shaper (138) is configured to selectively apply respective amplitude weightings to the frequency channels (116) before or after modulation and delay. Filter (106) further includes one or more photodetectors (140-144) configured to sum one or more of the delayed modulated frequency channels to generate the one or more processed output signals. Device (100) also includes a controller (130) configured to dynamically control the amplitude weightings to the frequency channels (116) to define the signal processing functions.

Description

PHOTONIC DIGITAL SIGNAL PROCESSORFIELD OF THE INVENTION

[0001] The present application relates to data processing devices and in particular to microwave photonics-based data processing.

[0002] Embodiments of the present invention are particularly adapted for providing an optical microcomb-based data processing system for processing video images. However, it will be appreciated that the invention is applicable in broader contexts and other applications.BACKGROUND

[0003] Signal processing has become central to many fields, from coherent optical telecommunications, where it is used to compensate signal impairments, to video image processing. Image processing is particularly important for applications such as observational astronomy, medical imaging and diagnosis, robotic vision, remote drones, autonomous driving, ultra high speed imaging, holographic three-dimensional displays, big data and artificial intelligence. Many of these require real-time processing of massive real-world information, placing extremely high demands on the processing speed (bandwidth) and throughput of image processing systems.

[0004] For these applications, signal processing traditionally has mainly been performed electronically. However these, as well as other new applications, particularly those involving real time video image processing, are creating unprecedented demand for ultrahigh performance, including high bandwidth and reduced energy consumption.

[0005] While electrical digital signal processing (DSP) technologies are well established, they face intrinsic limitations in energy consumption and processing speed such as the well- known von Neumann bottleneck.

[0006] To overcome these limitations, optical signal processing offers the potential for much higher speeds, and this has been achieved using a variety of techniques including silicon photonic crystal metasurfaces

[0001] , surface plasmonic structures [2], and topological interfaces [3]. These free-space, spatial-light devices offer many attractions such as compact footprint, low power consumption, and compatibility with commercial cameras and optical microscopes. However, they tend to be non-reconfigurable fixed systems designed to perform a single fixed function. On a more advanced level, human action recognition through processing of video image data has been achieved using photonic computers [4, 5]. However,these were achieved either in comparatively low speed systems [4] or in high bandwidth (multi-TeraOP regime) systems based on bulk-optics that is incompatible with integration [5].

[0007] To date, optical systems, especially those compatible with integration, still have not demonstrated capability of processing of large data sets of high-definition video images and at ultra-high speeds - enough for real-time video image processing.

[0008] Any discussion of the background art throughout the specification should in no way be considered as an admission that such art is widely known or forms part of common general knowledge in the field.SUMMARY OF THE INVENTION

[0009] In accordance with a first aspect of the present invention, there is provided a data processing device, including: an input for receiving an input signal including a data stream to be processed; a reconfigurable transversal filter adapted to perform one or more temporal signal processing functions on the input signal to produce one or more processed output signals, wherein the reconfigurable transversal filter includes: an optical frequency comb generator to generate a plurality of discrete, equally spaced frequency channels; a modulator configured to modulate the input signal with each frequency channel to generate modulated frequency channels having the input data stream encoded therein; an optical delay device to selectively apply a frequency dependent temporal delay to each of the modulated frequency channels such that each of the modulated frequency channels are temporally shifted by a predefined temporal period to produce delayed modulated frequency channels; an optical spectral shaper configured to selectively apply respective amplitude weightings to the frequency channels before or after modulation and delay; and one or more photodetectors configured to sum one or more of the delayed modulated frequency channels to generate the one or more processed output signals; anda controller configured to dynamically control the amplitude weightings to the frequency channels to define the signal processing functions.

[0010] In some embodiments, the data streams include image data. In some embodiments, the data streams include video image data. In some embodiments, input image frames are flattened into a one dimensional vector to form the input signal.

[0011] In some embodiments, the device includes a feedback loop to feed back measured peak powers of frequency channels to the controller, wherein the controller compares the measured peak powers of the frequency channels to reference filter tap weights and updates the amplitude weightings of the frequency channels based on this comparison.

[0012] In some embodiments, the signal processing functions include performing differentiation. In some embodiments, the signal processing functions include performing differentiation edge detection. In some embodiments, the signal processing functions include performing a Hilbert transform. In some embodiments, the signal processing functions include performing edge enhancement. In some embodiments, the signal processing functions include performing integration.

[0013] In some embodiments, the device includes a spectral shaping device configured to substantially equalize the amplitudes of the frequency channels. In some embodiments, the spectral shaping device and the optical spectral shaper are the same device.

[0014] In some embodiments, the device is configured to perform a plurality of simultaneous image processing functions and wherein a subset of the frequency channels are allocated for each image processing function. In some embodiments, 5 to 20 frequency channels are allocated for each image processing function. In some embodiments, the device is configured to perform 5 to 40 functions simultaneously.

[0015] In some embodiments, the optical frequency comb generator is configured to generate 50 to 150 frequency channels. In some embodiments, the frequency channels are spaced apart by 50 GHz.

[0016] In some embodiments, the device is integrated onto a single chip.

[0017] In some embodiments, the device includes a signal pre-processing module configured to process the input signal to generate a one-dimensional vector for input to the reconfigurable transversal filter.

[0018] In some embodiments, the optical spectral shaper is configured to demultiplex the delayed modulated frequency channels such that subsets of the delayed frequency channels are directed along different optical paths for separate processing by respective photodetectors. In other embodiments, the device includes a demultiplexer configured to split subsets of the delayed modulated frequency channels along different optical paths for separate processing by respective photodetectors.

[0019] In some embodiments, the device includes one or more optical power splitters configured to split the optical frequency channels into two or more groups of optical frequency channels. In some embodiments, the device includes a plurality of optical spectral shapers, each configured to selectively apply respective amplitude weightings to subsets of the frequency channels before or after modulation and delay.

[0020] In accordance with a second aspect of the present invention, there is provided an image processing device utilizing a microcomb-based transversal filter.BRIEF DESCRIPTION OF THE FIGURES

[0021] Example embodiments of the disclosure will now be described, by way of example only, with reference to the accompanying drawings in which:Figure 1 is a schematic illustration of a data processing device according to a first embodiment;Figure 2 is a schematic diagram showing the device of Figure 1 processing a sequence of video images to perform edge detection, motion blur (integration) and edge enhancement;Figure 3 is a schematic diagram of a data processing device according to a second embodiment performing 34 simultaneous image processing functions on an input video sequence;Figure 4 is a schematic diagram of a data processing device according to a third embodiment performing 3,078 simultaneous image processing functions on an input video sequence;Figure 5 is a table of detailed parameters for the device of Figure 3;Figures 6a-6c illustrate experimental results of image processing using different image processing functions in the device of Figure 3;Figure 7 illustrates a comparison of Berkeley Segmentation Database (BSD) images processed using the Sobel’s algorithm with video image processor after edge detection in the device of Figure 3;Figures 8a and 8b illustrate measured video processing of real-time video images in the device of Figure 3;Figure 9 compares simulated and measured higher order derivatives on a frame of video image data in the device of Figure 3; andFigure 10 illustrates schematically a sequence of input data reorganization required for performing 2-D derivatives.DESCRIPTION OF THE INVENTION

[0022] Embodiments of the present invention described herein relate to the processing of streams of video images to perform image processing functions. However, it will be appreciated that the invention is applicable more broadly to data processing in general and not specifically to image processing.

[0023] The inventors have previously developed an optical microcomb-based system for the separate application of neuromorphic processing and convolution of data. This is described in Australian patent application 2022200006, filed on 4 January 2022 and entitled “Data processing device incorporating an optical frequency comb”. The contents of AU 2022200006 are incorporated herein by way of cross reference.System overview

[0024] Referring initially to Figures 1 and 2, there is illustrated a data processing device 100. Figure 1 illustrates the primary hardware components of device 100 while Figure 2 illustrates the device in operation processing video image inputs. As best shown in Figure 1 , device 100 includes an input 102 for receiving an input signal including a data stream to be processed. In some embodiments, the input signal is an electrical signal having the data stream encoded into an electrical carrier signal via conventional techniques such as amplitude, frequency or phase modulation. Where the input signal is an electrical signal, input 102 may take the form of an electrical input port such as an Ethernet port, coaxial cable port or wireless interface. In other embodiments, the input signal may comprise an optical signal having the data stream modulated into an optical carrier signal by conventional techniques such as amplitude, frequency, phase or polarization modulation. Where the inputsignal is an optical signal, input 102 may take the form of an optical input port such as an optical fiber coupler. For the purpose of simplicity, in the present specification, the input signal will be described as an electrical signal.

[0025] The input data stream is preferably in the form of a vector. Where the data stream is not in vector form (such as 2D images), a pre-processing module 104 is included to vectorize the input data. By way of example, for applications to image processing, the input data is in the form of two or three dimensional matrices and so it needs to be flattened into vectors. Here, the pre-processing module 104 performs matrix flattening to form vectors for processing by device 100. In some embodiments, the raw input matrix is first sliced horizontally and vertically into multiple rows and columns respectively, which are flattened into vectors and connected head to tail. This image pre-processing is illustrated schematically in Figure 2. Preprocessing module 104 may comprise any data processing modules such as one or more general purpose computers, microprocessors, video processing hardware, cloud-based systems embedded systems or machine learning systems.

[0026] Where multiple input signals are provided through separate inputs (such as different video signals or different communications sources), pre-processing module 104 is configured to combine the input signals into a single data stream for subsequent processing. This may be achieved through signal combining modules such as multiplexers and couplers.

[0027] Once vectorized, input signal 104 is in the form of an input data vector X having a length L and encoded as the intensities of temporally spaced symbols in a serial electrical waveform by a high speed analog to digital converter. In some embodiments, the analog to digital converter has a resolution of 8 bits per symbol at a sampling rate of 64 gigabaud. In principle, for analog input signals this A / D and D / A step can be avoided as device 100 is capable of performing analog signal processing. However, for video processing, A / D and D / A conversion allows for dramatically increasing the speed of the video signal over standard video rates in order to fully exploit the ultrahigh speed of device 100.

[0028] Device 100 also includes a reconfigurable RF photonic transversal filter 106 adapted to perform one or more temporal signal processing functions on the input signal to produce a plurality of processed or filtered output signals 1 10-1 14. As described below, filter 106 includes a number of processing elements. The transfer function of filter 106 is given by:H(c^ = ^ h (n)e-jwnT(1 )

[0029] Where to is the RF angular frequency, T is the time delay between adjacent filter taps (frequency channels in the present invention), and h(n) is the tap coefficient of the nthtap.

[0030] To perform this function, filter 106 includes an optical frequency comb generator 108 configured to generate a plurality of discrete, equally spaced frequency channels 1 16. Optical frequency comb generator 108 includes a pump laser 1 18 to generate an input pump signal 120. The pump signal 120 is input to a microresonator in the form of micro-ring resonator 122. Resonator 122 is configured to produce the frequency channels 1 16. Although not illustrated, an Erbium Doped Fibre Amplifier (EDFA) may be included in generator 108 to boost the pump signal. In other embodiments, the micro-ring resonator 122 may be replaced with other microresonator devices to generate other types of frequency combs.

[0031] The pump laser 1 18 may be a Tunics 100S-HP external cavity laser manufactured by Yenista Optics. However, it will be appreciated that various other laser sources will be applicable. The EDFA may be a PMFA-37 polarization maintaining optical amplifier manufactured by PriTel, Inc. However, other types of optical amplifier may be used in place of the EDFA to initiate the parametric oscillation within micro-ring resonator 122.

[0032] Optical frequency combs generate an output spectrum that is composed of discrete and equally spaced frequency lines. In the present application, these frequency lines, referred to as channels, are used as discrete taps for a RF photonic transversal filter. Microcombs offer the full power of conventional optical frequency combs, but in an integrated form with much smaller footprint.

[0033] In some embodiments, micro-ring resonator 122 is adapted to generate soliton crystal microcombs with a channel spacing of about 48.9 GHz, with 90 frequency channels occupying 36 nm across the telecommunications C-band (1 ,540-1 ,570 nm), offering adiabatically generated low-noise frequency comb lines with a small footprint of <1 mm2and potentially low power consumption (<100 mW). In one embodiment, the micro-ring resonator 122 used to generate soliton crystal microcombs was fabricated based on a complementary metal-oxide-semiconductor (CMOS) compatible doped silica glass platform. It has a radius of ~592 pm, a high quality factor of ~1.5 million, and a free spectral range (FSR) of ~0.393 nm (i.e., ~48.9 GHz). The low FSR results in a large number of wavelengthchannels, which are used as discrete taps in the RF photonic transversal filter. The crosssection of the waveguide was 3 pm x 2 pm, resulting in anomalous dispersion in the C-band. The input and output ports of the MRR were coupled to a fibre array via specially designed mode converters, yielding a low fibre-chip coupling loss of 0.5 dB / facet. However, these values are exemplary only and, in other embodiments, micro-ring resonator 122 is capable of generating soliton crystal microcombs with different channel spacings.

[0034] Soliton crystal microcomb devices are formed by generating self-localized light pulses within micro-ring resonators. They are naturally formed in micro-cavities with appropriate mode crossings, without the need for complex dynamic pumping and stabilization schemes. Soliton crystal microcombs are characterized by distinctive ‘fingerprint’ optical spectra which arise from spectral interference between the tightly packaged solitons circulating along the ring cavity.

[0035] This category of soliton microcomb features deterministic soliton formation originating from the mode-crossing-induced background wave and the high intracavity power. This, in turn, enables simple and reliable initiation via adiabatic pump wavelength sweeping that can be achieved with manual detuning. The ability to adiabatically sweep the pump laser lies in the fact that the intra-cavity power is over thirty times higher than for single-soliton states (dissipative Kerr solitons), and very close to that of spatiotemporal chaotic states. Thus, the soliton crystal displays much less thermal detuning or instability resulting from the ‘soliton step’ that makes resonant pumping of dissipative Kerr soliton states more challenging.

[0036] In other embodiments, optical frequency combs may be used that generate greater or fewer numbers of frequency channels. By way of example, frequency comb generator 108 may produce between 25-1000, 20-500, 100-500 or 50-100 frequency channels.

[0037] Filter 106 includes an optical spectral shaper device 126 configured to substantially equalize the amplitudes of the frequency channels 1 16. In particular, the amplitudes of frequency channels 116 are preferably flattened or normalized prior to subsequent amplitude shaping of the channels to produce filter tap weights. The flattened frequency channels 128 are illustrated in the inset of Figure 1 . By way of example, this flattening may be performed by a Waveshaper device such as a Finisar 16000S WaveShaper device developed by ll-VI Incorporated. However, other devices may be used to perform gain flattening across the frequency channels 128. In some embodiments, this channel flattening process is performed later in the optical train.

[0038] Filter 106 further includes a modulator 132 configured to modulate the input signal with each frequency channel to generate modulated frequency channels. In particular, the input data stream contained in the input signal is multicast onto each frequency channel such that each optical sub-signal has the input data stream encoded therein. The modulated frequency channels represent replicas of the input data stream but weighted by the corresponding weight of weight vector W of each frequency channel. Thus the weighted optical sub-signal at the ith frequency channel is given as W[ / ? - i + 1] • X[n],

[0039] By way of example, suitable electro-optic modulators include an iXblue 40 GHz Lithium Niobate Mach-Zehnder intensity electro-optic modulator. However, it will be appreciated that other types of electro-optic modulators may be used in the present invention.

[0040] The output from modulator 132 is a single multiplexed optical signal 134 having the modulated frequency channels contained therein as different spectral components.

[0041] The tap delays required by Equation 1 are achieved by passing the multiplexed optical signal 134 through an optical delay device in the form of a length of dispersive standard single mode fibre (SMF) 136. SMF 136 selectively applies a frequency dependent temporal delay (through chromatic dispersion) to each of the modulated frequency channels such that each of the modulated frequency channels are temporally shifted by a predefined temporal period to produce delayed modulated frequency channels. Preferably, the predefined temporal period matches the data symbol duration T of the stream of input data (inverse of the Baud rate of X[n], However, longer or shorter delays are possible.

[0042] For microcombs with multiple equally spaced comb lines transmitted over the dispersive SMF, in Eq. (1 ) T is given by T=DxLxAA, where D is the dispersion coefficient of the SMF, L is the length of the SMF, AA is the spectral separation between adjacent comb lines (in our case 48.9 GHz) and the RF bandwidth of the system is given by f = 1 / T.

[0043] The optical delay provided by SMF 136 plays a crucial role in achieving simultaneous processing by introducing frequency dependent controlled time delays to the different channels, enabling the functions to be processed independently and in parallel. These time delays coincide with the requirements of the transversal filter function (Equation 1 ) and ensure that the input signals for each function are properly aligned and synchronized.

[0044] In other embodiments, SMF136 may be replaced with other optical delay devices such as Bragg gratings, dispersive waveguides and / or tuneable dispersion compensators. Insome embodiments, the modulated frequency channels may be spatially separated along different optical paths (e.g. through a multipath waveguide) that have different optical path lengths and / or different dispersive properties.

[0045] Filter 106 also includes an optical spectral shaper 138 to selectively apply respective amplitude weightings 128 to the frequency channels 116 (either raw frequency channels, modulated frequency channels or modulated and delayed frequency channels). As described below, spectral shaper 138 may also operate as an optical demultiplexer. The amplitude weightings of the frequency channels provide tap coefficients for transversal filter 106. The spectral transfer function of a transversal filter can be engineered via the design of the tap coefficients for different channels, thus allowing for highly reconfigurable filter shapes. The weights are input to spectral shaper 138 by way of a control signal from a control module 130. Control module 130 is configured to dynamically control the amplitude weightings to the frequency channels 1 16 to define the signal processing functions. Control module 130 may comprise an internal controller within spectral shaper 138 or may comprise external control from a computer, microcontroller or other external or networked device. Under control by module 130, spectral shaper 138 selectively and dynamically controls the optical power of each frequency channel produced by comb generator 1 16 such that each channel has a particular amplitude or peak optical power.

[0046] The control signal from control module 130 may be defined, in part, from user input at a user interface to select specific signal processing functions. In this manner, a user may be able to vary the signal processing that is performed by device 100 through a user interface.

[0047] By way of example, spectral shaper 138 may include a WaveShaper 4000S programmable optical filter device manufactured by ll-VI Incorporated. However, it will be appreciated that spectral shaper 136 may include various optical filter devices capable of applying selective filtering to different frequencies such as fibre Bragg gratings.

[0048] Once weighting has been applied by spectral shaper 120, the weighted frequency channels define one or more weight vectors W or filter kernels that represent the filter tap coefficients (h(n) in Equation 1 ) of device 100. The coefficients are defined by the number and configuration of frequency channels. Depending on the particular application, one or more weight vectors may be generated. The number and length of the weight vectors are arbitraryand can be scaled depending on the application. By way of example, using 90 frequency channels, 10x9 weight vectors can be produced.

[0049] As will be described below, the allocation of frequency channels and dynamic control of the channel weightings allows multiple signal processing functions to be implemented simultaneously on the input signal.

[0050] Here, the term "functions" refers to signal processing operations comprised of fundamental mathematical (e.g. vector) operations that are performed by the system. In the case of image processing, these functions relate to image processing operations such as object edge detection, edge enhancement and motion blur. These functions operate on the input signal to extract or enhance these key characteristics and include both integral and fractional order differentiation, fractional order Hilbert transforms, and integration. For differentiation and Hilbert transforms, both integral order and a continuous range of fractional order transforms can be performed. Other possible signal processing functions in the context of image processing include low-pass filtering, high-pass filtering, band-pass filtering, Fourier and other transforms, convolution, auto-correlation, cross-correlation, modulation, sampling or resampling, interpolation or decimation, windowing, normalization, noise reduction, peak detection, spectral analysis and dynamic range compression as examples. The range of possible functions that can be performed by device 100 is unlimited given that the system can process a continuous range of arbitrary fractional and high-order differentiation and fractional Hilbert transforms. Signal processing functions and signal processing operations may be used interchangeably.

[0051] Although the amplitude weighting of the frequency channels is described as occurring at this stage, in other embodiments, the amplitude weighting may occur earlier in the process. In these latter embodiments, spectral shaper 138 is located further upstream in the optical train such as before SMF 136 and modulator 132.

[0052] Given that transversal filters are linear in nature, the different operations of comb line weighting, signal modulation and channel delay can be performed in different order. As such, in some embodiments, SMF 136 may be positioned after spectral shaper 138 and / or modulator 132. However, where spectral shaper 138 is located prior to SME 136 and modulator 132, a separate demultiplexing module is required to separate different outputs corresponding to signal processing functions.

[0053] Spectral shaper 138 may be configured to operate as a demultiplexer and split subsets of the delayed modulated frequency channels along different optical paths for separate processing. This demultiplexing functionality is only required where multiple simultaneous signal processing functions are to be applied to the input signal. Spectral shaper 138 thus performs the function of splitting out the signals that are subject to different signal processing functions.

[0054] In other embodiments, a separate demultiplexer device may be incorporated downstream of spectral shaper 138 to separately perform demultiplexing. In these embodiments, the demultiplexer may be in the form of a wavelength selective switch or other optical switch.

[0055] Filter 106 includes one or more photodetectors 140-144 configured to sum one or more of the delayed modulated frequency channels to generate the one or more processed output signals. The number of photodetectors is equal to the number of optical paths split from demultiplexer 138. Although 5 photodetectors 140-144 are illustrated in Figure 1 , it will be appreciated that any number of photodetectors may be implemented. By way of example, photodetectors 132a-j may include XPDV2020 high speed photodetectors manufactured by II- VI, Incorporated.

[0056] Photodetectors detect and sum the delayed modulated frequency channels to produce processed electrical outputs. This summing process completes the filter function summarised in Equation 1 above. The detection process effectively sums the aligned symbols of the different sub-band signals, which are delayed with respect to each other. As such, the photodetectors 140-144 act to collectively convolve each of the delayed modulated frequency channels of a given subset into a convolved dataset. A similar convolution process is performed for each signal processing operation (subset of delated modulated frequency channels).

[0057] The output signals 1 10-1 14 represent processed versions of the input signal, each having a respective signal processing operation applied thereto. The output signals 110-114 are one dimensional electronic signals. Where the input data includes image or video data, an image reconstruction module 150 is capable of reconstructing the images from the output one dimensional data to produce processed images.

[0058] In device 100, the amplitude weightings can be dynamically varied to change the number and type of signal processing functions being performed. This dynamic control is performed by control module 130 providing different control signals to spectral shaper 138. The number of signal processing functions that can be performed is limited by the number of available frequency comb channels and the number of output paths and photodiodes implemented in the system. The number of frequency comb channels can be increased by using an optical power splitter such as a coupler.

[0059] Referring now to Figure 3, there is illustrated an alternate signal processing device 300 configured to simultaneously perform 34 different signal processing functions on input data. Device 300 operates in a similar manner to that of device 100 but includes an optical coupler 302 for splitting the delayed modulated frequency channels into two outputs for doubling the number of frequency channels for use in the signal processing (at the cost of a reduction in signal power). In device 300, two spectral shaping devices 304 and 306 act as both spectral shapers for applying the amplitude weightings to the frequency channels and as demultiplexers for sending the processed signals to separate photodetectors for generating processed output signals.

[0060] The outputs of device 300 are 34 processed video signals which can be reconstructed to provide real-time simultaneous processed video image streams. Detailed parameters of device 300 to provide the 34 different signal processing functions are summarised in Figure 5. In Figure 5, DIFF represents differentiation), HT represents Hilbert transformation and BHT represents Bandpass Hilbert transformation.

[0061] Referring to Figure 4, the device can be scaled up arbitrarily to much larger data rates. In Figure 4, a large scale data processing device 400 includes a single microcomb generator 402 which generates a frequency comb of 405 frequency channels. 5 channels were used for each signal processing function. These are passed through an optical power splitter 404, which splits the signals 19 times along 19 independent optical paths. Each path includes a respective modulator 410a-c, single mode fibre 412a-c to provide an optical delay, spectral shaper / demultiplexer 414a-c and photodiodes 416 to sum the processed signal.

[0062] Each demultiplexer / spectral shaper 412a-c has 81 outputs and performs similar functions to that of spectral shaper 138 of Figure 1. Furthermore, the optical components are polarization sensitive such that information can be encoded onto two orthogonal polarizations of the wavelength channels. This dual-polarization effectively doubles the processingcapability of device 400. This provides for 19x81 x2=3,078 simultaneous outputs. For a modulation rate of 64 GigaBaud and a symbol rate of 8 bits per symbol, device 400 is capable of providing a processing speed of 1 .575 Petabits / s.

[0063] As shown in Figure 1 , in order to improve the comb shaping accuracy, a feedback control loop 180 may be employed for spectral shaper 138. RF Gaussian pulses may be used as the system input and replicas of the input pulses in different frequency channels may be measured. Next, peak intensities of the system impulse response may be extracted to obtain accurate RF-to-RF tap coefficients. Finally, the extracted tap coefficients can be subtracted from the ideal tap coefficients to obtain an error signal, which can be used to calibrate the loss of spectral shaper 138.

[0064] After several iterations of the comb shaping loop, an accurate impulse response that compensates for the non-ideal impulse response of the system can be obtained. This can significantly improve the accuracy of the data processing in device 100.

[0065] Preferably at least 5 frequency channels (filter taps) are used for each signal processing function. However, the greater the number of frequency channels, the greater the accuracy in reconstructing a filter function that performs the desired signal processing function. The number of available frequency channels (filter taps) can be increased by using MRRs with smaller FSRs or optical amplifiers with broader operation bandwidths.

[0066] The use of wavelength division multiplexing and WaveShaper-type spectral shapers allows for very flexible allocation of frequencies and highly reconfigurable tuning. By employing these components in a carefully designed configuration where each function only requires a limited number of frequency channels, multiple functions can be simultaneously processed. By configuring the WaveShapers appropriately according to the transfer function, different functions can be applied to different groups of frequency channels, facilitating simultaneous processing of multiple functions across the entire microcomb spectrum.

[0067] Although the devices described above use off-the-shelf commercial components, in practice these can be realized using various techniques compatible with integration, such as cascaded Mach-Zehnder interferometers or programmable phase modulators. These components enable precise control over the spectral phase and amplitude profiles of the optical signals, allowing the realization of fractional order operations. As such, in someembodiments, devices 100, 300 or 400 may be entirely integrated onto a single chip such as a silicon wafer, integrated circuit or other type of integrated system-on-chip device.

[0068] The devices described above are also capable of performing convolution operations for deep learning neural networks. This opens up new opportunities for image or video processing applications in robotic vision and machine learning. In particular, each parallel function can be trained and performed with as many as 34 kernels with a size of 5 by 1 for the convolutional neural network, therefore a neural network could be achieved, avoiding the bandwidth limitation given by the analogue-to-digital converters.

[0069] As compared with spatial-light devices used for image processing, the present described systems are not only ultra-high speed but highly reconfigurable and programable, able to perform many different functions without any change to the physical hardware. The devices can be used as an optical real-time signal processor for video images that is reconfigurable and compatible with integration. It is based on components that are either already integrated or have been demonstrated in integrated form, and operates at an ultra- high bandwidth of 17 Terabits / s. This is sufficient to process approximately 400,000 (399,061 ) video signals both concurrently and in real-time, performing up to 34 functions on each signal simultaneously.

[0070] The present disclosed system is comparable to electrical digital signal processing (DSP) systems but with the important advantages that it operates at multi-terabit / s speeds, enabled by massively parallel processing. It is also very general, flexible, and highly reconfigurable - able to perform a wide range of functions without requiring any change in hardware.Example results

[0071] The above described devices 100, 300 and 400 were tested and compared to theoretical results. Experimental results agree well with theory, demonstrating that the processor is a powerful approach for ultrahigh-speed video image processing for robotic vision, machine learning, and many other emerging applications.

[0072] Figures 6a-6c illustrate experimental results of image processing. Panels (a) - (c) illustrate results for edge detection based on differentiation with order of 0.5, 0.75, and 1 , respectively. Panels (d) - (f) show results for motion blur based on integration with tap number of 15, 45, and 75, respectively. Panels (g) - (I) show results for edge enhancement based onHilbert transformation with operation bandwidth of 18 GHz, 12 GHz, and 38 GHz, respectively. In (a) - (I), (i) shows the designed and measured optical spectra of the shaped microcomb, (ii) shows the measured and simulated spectral response of the video image processing system, and (iii) shows the measured and simulated high definition (HD) video images after processing.

[0073] Figure 7 illustrates a comparison of Berkeley Segmentation Database (BSD) images processed using the Sobel’s algorithm with video image processor after edge detection. Differentiation with different orders of 0.2, 0.4, 0.6, 0.8, and 1 .0 are used for the edge detection with our video image processor. The Sobel results were performed electronically.

[0074] Figures 8a and 8b illustrate measured video processing of real-time video images. The first original video had a resolution of 568 x 320 pixels and a frame rate of 30 frames per second. Panel (a) shows original video, panel (b) shows processed video after 0.5 order differentiation, panel (c) shows processed video after integration and panel (d) shows processed video after Hilbert transformation with 90-degree phase shift.

[0075] Figure 9 compares simulated and measured higher order derivatives on a frame of video image data.

[0076] Figure 10 illustrates schematically a sequence of input data reorganization required for performing 2-D derivatives. Normally, processing functions such as differentiation, operating on video signals, only result in a one dimensional process - since it acts on individual lines of the video raster image. However, by appropriately pre-processing the video signal it is possible to obtain a fully two-dimensional derivative.

[0077] The raw input matrices were first sliced horizontally and vertically into multiple rows and columns, respectively, which were flattened into vectors and connected head-to-tail. After that, the data was passed through the above described data processing device. Finally, the processed video is restored into the original size of the matrix and the average of horizontally and vertically processed video is taken and formed into a two-dimensional processed video sequence.References[1] Y. Zhou, H. Zheng, I. I. Kravchenko, and J. Valentine, “Flat optics for image differentiation,” Nature Photonics 14, 316-323 (2020).[2] T. Zhu, Y. Zhou, Y. Lou, H. Ye, M. Qiu, Z. Ruan, and S. Fan, “Plasmonic computing of spatial differentiation,” Nature Communications 8, 15391 (2017).[3] T. Zhu, C. Guo, J. Huang, H. Wang, M. Orenstien, Z. Ruan, and S. Fan, “Topological optical differentiator,” Nature Communication 12, 680 (2021 ).[4] P. Antonik, N. Marsal, D. Brunner, and D. Rontani, “Human action recognition with a large-scale brain-inspired photonic computer”, Nature Machine Intelligence 1 , 530- 537 (2019).[5] T. Zhou, X. Lin, J. Wu, Y. Chen, H. Xie, Y. Li, J. Fan, H. Wu, Lu. Fang, and Q. Dai, “Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit”, Nature Photonics 15, 367-373 (2021 ).Interpretation

[0078] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as "processing," "computing," "calculating," “determining”, analyzing” or the like, refer to the action and / or processes of a computer or computing system, or similar electronic computing device, that manipulate and / or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

[0079] In a similar manner, the term “controller” or "processor" may refer to any device or portion of a device that processes electronic data, e.g., from registers and / or memory to transform that electronic data into other electronic data that, e.g., may be stored in registers and / or memory. A “computer” or a “computing machine” or a "computing platform" may include one or more processors.

[0080] Reference throughout this specification to “one embodiment”, “some embodiments” or “an embodiment” means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, appearances of the phrases “in one embodiment”, “in some embodiments” or “in an embodiment” in various places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more embodiments.

[0081] As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0082] In the claims below and the description herein, any one of the terms comprising, comprised of or which comprises is an open term that means including at least the elements / features that follow, but not excluding others. Thus, the term comprising, when used in the claims, should not be interpreted as being limitative to the means or elements or steps listed thereafter. For example, the scope of the expression a device comprising A and B should not be limited to devices consisting only of elements A and B. Any one of the terms including or which includes or that includes as used herein is also an open term that also means including at least the elements / features that follow the term, but not excluding others. Thus, including is synonymous with and means comprising.

[0083] It should be appreciated that in the above description of exemplary embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single embodiment, Fig., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this disclosure.

[0084] Furthermore, while some embodiments described herein include some but not other features included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the disclosure, and form different embodiments, as would be understood by those skilled in the art. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0085] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

[0086] Embodiments described herein are intended to cover any adaptations or variations of the present invention. Although the present invention has been described and explained in terms of particular exemplary embodiments, one skilled in the art will realize that additional embodiments can be readily envisioned that are within the scope of the present invention.

Claims

What is claimed is:1 . A data processing device, including: an input for receiving an input signal including a data stream to be processed; a reconfigurable transversal filter adapted to perform one or more temporal signal processing functions on the input signal to produce one or more processed output signals, wherein the reconfigurable transversal filter includes: an optical frequency comb generator to generate a plurality of discrete, equally spaced frequency channels; a modulator configured to modulate the input signal with each frequency channel to generate modulated frequency channels having the input data stream encoded therein; an optical delay device to selectively apply a frequency dependent temporal delay to each of the modulated frequency channels such that each of the modulated frequency channels are temporally shifted by a predefined temporal period to produce delayed modulated frequency channels; an optical spectral shaper configured to selectively apply respective amplitude weightings to the frequency channels before or after modulation and delay; and one or more photodetectors configured to sum one or more of the delayed modulated frequency channels to generate the one or more processed output signals; and a controller configured to dynamically control the amplitude weightings to the frequency channels to define the signal processing functions.

2. The device according to claim 1 wherein the data streams include image data.

3. The device according to claim 2 wherein the data streams include video image data.

4. The device according to claim 1 or claim 2 wherein input image frames are flattened into a one dimensional vector to form the input signal.

5. The device according to any one of the preceding claims including a feedback loop to feed back measured peak powers of frequency channels to the controller, wherein the controller compares the measured peak powers of the frequency channels to referencefilter tap weights and updates the amplitude weightings of the frequency channels based on this comparison.

6. The device according to any one of the preceding claims wherein the signal processing functions include performing differentiation.

7. The device according to any one of the preceding claims wherein the signal processing functions include performing differentiation edge detection.

8. The device according to any one of the preceding claims wherein the signal processing functions include performing a Hilbert transform.

9. The device according to any one of the preceding claims wherein the signal processing functions include performing edge enhancement.

10. The device according to any one of the preceding claims wherein the signal processing functions include performing integration.1 1 . The device according to any one of the preceding claims including a spectral shaping device configured to substantially equalize the amplitudes of the frequency channels.

12. The device according to claim 1 1 wherein the spectral shaping device and the optical spectral shaper are the same device.

13. The device according to any one of the preceding claims configured to perform a plurality of simultaneous image processing functions and wherein a subset of the frequency channels are allocated for each image processing function.

14. The device according to claim 13 wherein 5 to 20 frequency channels are allocated for each image processing function.

15. The device according to claim 13 or claim 14 configured to perform 5 to 40 functions simultaneously.

16. The device according to any one of the preceding claims wherein the optical frequency comb generator is configured to generate 50 to 150 frequency channels.

17. The device according to any one of the preceding claims wherein the frequency channels are spaced apart by 50 GHz.

18. The device according to any one of the preceding claims integrated onto a single chip.

19. The device according to any one of the preceding claims including a signal pre-processing module configured to process the input signal to generate a one-dimensional vector for input to the reconfigurable transversal filter.

20. The device according to any one of the preceding claims wherein the optical spectral shaper is configured to demultiplex the delayed modulated frequency channels such that subsets of the delayed frequency channels are directed along different optical paths for separate processing by respective photodetectors.21 . The device according to any one of the preceding claims including a demultiplexer configured to split subsets of the delayed modulated frequency channels along different optical paths for separate processing by respective photodetectors.

22. The device according to any one of the preceding claims including one or more optical power splitters configured to split the optical frequency channels into two or more groups of optical frequency channels.

23. The device according to claim 22 including a plurality of optical spectral shapers, each configured to selectively apply respective amplitude weightings to subsets of the frequency channels before or after modulation and delay.

24. An image processing device utilizing a microcomb-based transversal filter.

Images