An end-to-end optical computing chip based on multi-modal analog signal fusion processing

Abstract

The application discloses an end-to-end optical computing chip based on multi-modal analog signal fusion processing, and belongs to the technical field of optical computing. The chip comprises a multi-modal input fusion front-end module, a fiber input interface and an end-to-end inference module. The multi-modal input fusion front-end module can convert different types of original analog signals such as images, spectra and radio frequencies into unified broadband spectral input signals. The end-to-end inference module builds a deep optical neural network, which comprises a "sensing-convolution integrated" unit realized by an arrayed waveguide grating (AWG). The end-to-end inference module also comprises an optoelectronic nonlinear-pooling integrated unit, which simultaneously realizes the functions of average pooling and nonlinear activation, and effectively compensates for optical path loss through injection of a light source. Finally, the signals processed by multiple layers are integrated by a full connection layer, and a classification result is output by an output layer. The chip architecture realizes direct and efficient processing of multi-modal analog signals.

Description

Technical Field

[0001] This invention belongs to the field of optical computing technology, and more specifically, relates to an end-to-end optical computing chip based on multimodal analog signal fusion processing. Background Technology

[0002] Edge computing, as a crucial carrier for technologies such as artificial intelligence and the Internet of Things, presents severe challenges in processing multimodal analog signals originating from the physical world, including images, spectra, and radio frequencies, requiring real-time processing, high efficiency, and low power consumption. Traditional digital electronic computing architectures, due to their inherent analog-to-digital conversion overhead, processing latency, high energy consumption, and the von Neumann bottleneck, are increasingly showing their limitations in meeting the extreme requirements of edge devices in terms of power consumption, size, and response speed.

[0003] Integrated photonics, with its high bandwidth, low latency, and parallel processing potential, offers a new path to overcome the bottlenecks of electronic computing, giving rise to Optical Neural Networks (ONNs). However, current ONN technology still faces several obstacles in practical applications. First, existing solutions are not yet mature in terms of direct and efficient input and on-chip fusion of multimodal raw analog signals. Most integrated photonic systems still rely on preprocessed or single-mode optical signals, lacking a unified, low-overhead interface to directly and efficiently load diverse raw analog signals from the physical world into optical chips for processing. This limits the full potential of optical computing in analog domain information processing.

[0004] Secondly, in the design of core computing units, a "sensing-computing separation" model is currently prevalent. For example, when processing spectral information, a separate spectral sensing module typically acquires data first, and the extracted features (often digitized) are then fed into subsequent computing units. This separation model not only increases the redundancy of on-chip resources and the overhead of signal conversion, but also fails to fully exploit the potential of certain optical structures to simultaneously perform sensing and computing tasks, thus limiting the chip's integration and end-to-end processing efficiency. In particular, when feature extraction operations such as convolution need to be performed directly on the raw spectral information, there is a lack of efficient integrated hardware implementation solutions.

[0005] Furthermore, the core bottleneck in constructing practically applicable deep optical neural networks lies in the power attenuation of optical signals during multi-layer cascaded transmission and computation. Optical signals incur losses in on-chip waveguides, couplers, and each computational layer. These losses accumulate progressively in deep networks, resulting in excessively low optical signal intensity at the network's depths. This makes it impossible to effectively drive nonlinear activation units or be accurately read by detectors, severely limiting the effective depth and achievable model complexity of ONNs. Existing on-chip optoelectronic nonlinear activation schemes, without relying on external electrical amplifiers, often fail to provide sufficient net signal gain to compensate for the aforementioned losses. This hinders the construction of truly deep-cascaded, fully on-chip integrated optical intelligent systems.

[0006] Therefore, to meet the urgent needs of future edge intelligence applications, it is imperative to develop a novel photonic edge intelligence chip architecture. This architecture should be able to directly and efficiently fuse and process multi-source heterogeneous raw analog signals, achieving a high degree of integration between sensing and computing functions at both the physical and functional levels. Furthermore, it should be able to construct optical neural networks that can effectively overcome optical power attenuation and support deep network cascading, thereby providing a powerful hardware foundation with low latency, high throughput, and high energy efficiency. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the present invention aims to provide an end-to-end optical computing chip based on multimodal analog signal fusion processing. This chip addresses the problems of existing technologies in efficiently fusing and processing various types of raw analog signals, achieving integrated sensing and computing functions, and constructing deep cascadeable optical neural networks that can overcome optical power attenuation bottlenecks. It also improves the real-time processing capability, integration, and energy efficiency of multimodal information in edge computing scenarios.

[0008] To achieve the above objectives, the present invention provides an end-to-end optical computing chip based on multimodal analog signal fusion processing, comprising:

[0009] A multimodal input fusion front-end module, the module being used to convert at least two different types of raw analog input signals (including but not limited to image signals, spectral signals, and radio frequency signals) into a unified broadband spectral input signal suitable for on-chip optical path processing;

[0010] An end-to-end inference module has its input end optically coupled to the output end of the multimodal input fusion front-end module via an optical fiber input interface. The end-to-end inference module includes a sensing layer, multiple alternating cascaded convolutional layers and nonlinear activation layers, a fully connected layer, and an output layer. The sensing layer is used for spectral sensing of broadband spectral input signals. The multiple alternating cascaded convolutional layers and nonlinear activation layers are used to perform optical convolution operations and nonlinear activation operations on the spectrum sensed by the sensing layer, respectively. The fully connected layer is used to perform feature integration and classification tasks on the high-level abstract feature vectors extracted by the multiple alternating cascaded convolutional layers and nonlinear activation layers, and finally outputs the classification result through the output layer.

[0011] Furthermore, the multimodal input fusion front-end module includes a random scattering device for scattering the spatial light field carrying image information and mapping it to the spectral domain, converting it into a part of the broadband spectral input signal. The random scattering device includes one or more waveguide arrays, each waveguide having a grating structure with random or pseudo-random parameters.

[0012] Furthermore, the multimodal input fusion front-end module includes an electro-optic phase modulator for phase modulation of the optical carrier of the radio frequency signal, and the resulting spectral sidebands are converted into a part of the broadband spectral input signal.

[0013] Furthermore, the sensing layer includes an arrayed waveguide grating, which utilizes its inherent wavelength dispersion characteristics and multi-port spatial separation capability to achieve spectral sensing of the input spectral signal.

[0014] Furthermore, the convolutional layer includes a tunable weighting unit, which is a Mach-Zehnder interferometer (MZI) array. The MZI array is used to load convolution kernel weights. The convolutional layer, through its multiple-input multiple-output optical path routing characteristics, converges and superimposes optical signals from different input paths that have passed through the MZI array and have been loaded with convolution kernel weights at its output port to complete the optical convolution operation.

[0015] Furthermore, the convolutional layer is an arrayed waveguide grating.

[0016] Furthermore, the nonlinear activation layer unit includes a photodetector, an optical modulator, and a supply light source. The photodetector is used to convert the optical signal output by the optical convolution operation into an electrical signal. The optical modulator is driven by the electrical signal. The supply light source provides light input to the optical modulator, enabling the nonlinear activation layer to achieve nonlinear activation function. The injection of the supply light source effectively compensates for optical path loss or achieves net signal gain.

[0017] Furthermore, the photodetector in the nonlinear activation layer is a multi-port photodetector, or the output currents of multiple photodetectors are combined in the electrical domain to converge the optical signals from multiple output ports of the convolutional layer, thereby achieving the average pooling function.

[0018] Furthermore, the optical modulator in the nonlinear activation layer is a microring resonator or a nanobeam optical cavity modulator.

[0019] Furthermore, the fully connected layer employs a Mach-Zehnder interferometer (MZI) array to perform matrix multiplication operations.

[0020] Compared with the prior art, the present invention provides a multimodal photonic edge intelligence architecture that integrates spectral, imaging, and radio frequency signals within a monolithic silicon photonics platform. This architecture focuses on solving the integration method and implementation problem of end-to-end optical convolutional neural network inference, encompassing key technologies such as spectral-visual-RF front-end sensing, on-chip sensing-convolutional integrated units, and cascadeable all-optical nonlinear operators. It can be widely applied to low-latency, high-throughput edge visual recognition, real-time spectral analysis, radar sensing, and other multimodal intelligent terminal scenarios. Specifically, it can achieve the following beneficial effects:

[0021] 1. Achieves direct fusion and efficient on-chip processing of multimodal raw analog signals: This invention, through a unified spectral domain conversion front end, can directly process various raw analog signals such as images, spectra, and radio frequencies, avoiding complex off-chip preprocessing and multiple signal conversions, and improving the system's integration and response speed.

[0022] 2. Improved integration and efficiency of sensing and computing: The AWG-based sensing-convolution unit design enables the spectral sensing and convolution operation functions to be tightly integrated in a single core device, reducing the number of devices and on-chip optical path complexity, reducing signal loss, and improving processing efficiency.

[0023] 3. Overcomes the power attenuation bottleneck of deep optical network cascading: The innovative cascadeable optoelectronic nonlinear-average pooling integrated unit, especially through the "supply light" injection mechanism, can effectively compensate for or even increase the power of optical signals, making it possible to build deeper and more complex optical neural networks, thereby improving the expressive power and computational performance of the model.

[0024] 4. Possesses high performance and low power consumption potential: By fully utilizing the high bandwidth and low latency characteristics of photonics and the parallel processing capabilities of the architecture of this invention, it is expected to achieve high computing throughput and low unit computing power consumption, meeting the stringent performance and power consumption requirements of edge intelligent applications.

[0025] 5. Enhanced system scalability and process compatibility: The modular design and compatibility with standard silicon photonics processes provide a foundation for large-scale chip scaling and low-cost manufacturing. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the overall architecture of an end-to-end optical computing chip based on multimodal analog signal fusion processing according to the present invention, showing the multi-layer cascaded processing unit inside the modal fusion front-end module and the monolithic integrated optical chip;

[0027] Figure 2 This is a schematic diagram of the structure of the cascaded multi-layer convolutional network and the cascaded opto-nonlinear-average pooling integrated unit in the optical computing chip of the present invention. (a) is a schematic diagram of the logical structure of a general deep convolutional neural network. (b) is a schematic diagram of the implementation method of cascading multi-layer optical convolutional layers and nonlinear activation layers in the present invention, highlighting the introduction of supply light at each level. (c) is a detailed structural schematic diagram of the cascaded opto-nonlinear-average pooling integrated unit, showing a multi-port photodetector for pooling and an optical modulator driven by the supply light that can realize nonlinear activation (as shown in the figure, a nanobeam electro-optic modulator without FSR constraints).

[0028] Figure label:

[0029] 1-Modal fusion front-end module, 2-Fiber optic input interface, 3-End-to-end inference module, 4-Sensing layer, 5-Convolutional layer, 6-Nonlinear activation layer, 7-Fully connected layer, 8-Output layer. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0031] like Figure 1 As shown, the end-to-end optical computing chip based on multimodal analog signal fusion processing proposed in this invention mainly includes a mode fusion front-end module 1, an end-to-end inference module 3, and an optical fiber input interface 2 connecting the two. The mode fusion front-end module 1 is responsible for converting analog signals from different sources, such as image signals (converted through scattering), spectral signals (directly input or processed), and radio frequency signals (converted through phase modulation), into broadband spectral signals. These fused spectral signals are fed into the monolithic integrated optical chip (i.e., the end-to-end inference module 3) through the optical fiber input interface 2 (or other optical coupling methods).

[0032] The monolithic integrated optical chip internally constructs a multi-layered cascaded optical neural network processing core. This core typically includes a sensing layer 4, multiple alternating cascaded convolutional layers 5 (i.e., AWG sensing-convolutional integrated units), and a nonlinear activation layer 6 (i.e., optoelectronic nonlinear-average pooling integrated units). After multi-layer feature extraction and nonlinear processing, the signal can be sent to the final fully connected layer 8 for final classification or regression decision, and the calculation result is output through the output layer 9. The entire chip constitutes an end-to-end inference module 3. The signal mainly flows in the form of optical signals inside the chip, undergoes photoelectric-optical conversion within the nonlinear activation unit, and the effective transmission of the signal in the deep network is ensured by the optical supply mechanism.

[0033] To convert two-dimensional or three-dimensional image signals into one-dimensional spectral signals, this invention employs space-spectral mapping techniques based on random scattering media or specially designed diffractive optical elements. In a preferred embodiment, an integrated random scattering chip can be used. This scattering chip typically includes one or more waveguide arrays, with grating structures etched on the waveguides having random or pseudo-random parameters, and may include coupling regions between the waveguides.

[0034] Its working principle is as follows: First, a broadband light source illuminates the original image to be identified or processed. The spatial light field modulated by the image information is precisely projected onto the surface of the random scattering chip through an optical relay system. When the light wave carrying the spatial information of the image interacts with the complex micro-nano structure inside the scattering chip, multiple scattering, diffraction, and interference occur. Because these optical processes are highly sensitive to the wavelength and spatial position of the incident light, the scattering chip uniquely encodes (maps) the light information from different spatial positions of the input image onto the intensity or phase of different wavelengths (or narrow bands) of the output light, forming specific spectral features related to the input image. Finally, the output light collected from a specific port of the scattering chip is a broadband spectral signal carrying the original image information. This spectral signal is then coupled to the AWG sensing-convolution unit of the monolithic integrated optical chip for further processing.

[0035] For time-domain radio frequency (RF) signals, this invention employs a technique based on a high-speed electro-optic phase modulator to map their information content onto the optical spectrum. In a preferred embodiment, a wide-bandgap, low-drive-voltage electro-optic phase modulator is used.

[0036] The workflow is as follows: First, the RF analog electrical signal to be processed is loaded onto the RF drive port of the phase modulator. Simultaneously, one or more continuous beams of light are injected into the phase modulator as optical carriers from the optical input port. Voltage variations in the RF signal cause corresponding changes in the refractive index of the optical waveguide material inside the modulator, thereby modulating the phase of the passing optical carrier in real time. This time-domain phase modulation manifests in the frequency domain (spectral domain) as a series of symmetrical modulation sidebands on both sides of the original optical carrier frequency. The frequency spacing of these sidebands is related to the main frequency components in the RF signal, while the amplitude and phase distribution of the sidebands are determined by the specific waveform and modulation depth of the RF signal. In this way, the time-domain information of the original RF signal is effectively converted (encoded) into the optical spectrum composed of the optical carrier and its modulation sidebands. Subsequently, this composite spectral signal containing the characteristics of the RF signal is fed into the AWG sensing-convolution unit for analysis and processing.

[0037] In some applications, the input signal itself is spectral information. In this case, these spectral signals can be directly input to the AWG sensing-convolution unit of a monolithic integrated optical chip through a standard fiber optic interface or an on-chip integrated optical coupling structure. If there are spectral signals from multiple sources in the system, appropriate spectral management may be required at the input end. For example, they can be combined into the same input fiber / waveguide using a passive optical multiplexer, or they can be input separately into different parallel processing channels of the AWG sensing-convolution unit.

[0038] The core of convolutional layer 5 in this invention is the AWG (Optical Array Girder). AWGs are typically made of silicon or other materials suitable for optical waveguides, and their typical structure includes: a set of input waveguides, an input star coupler, a set of array waveguides with precisely equal optical path lengths, an output star coupler, and a set of output waveguides. The optical path difference between adjacent waveguides in the array waveguide is crucial for achieving its wavelength selection and routing capabilities. The design of the AWG considers the center operating wavelength, required channel spacing, free spectral range (FSR), and the number of input / output ports to match specific application requirements. To reduce device size, compact designs, such as overlapping input and output star couplers, can be employed.

[0039] The spectral sensing capability of the AWG stems from its inherent wavelength dispersion and spatial separation characteristics. When an input optical signal containing multiple wavelengths (or a continuous broadband spectrum) enters the input star coupler (FPR1) through an input waveguide, the light diffracts in FPR1 and is uniformly distributed to the various entrances of the array waveguide. Due to the precise optical path difference between the array waveguides, light of different wavelengths accumulates different phases when it reaches a specific convergence point of the output star coupler (FPR2) through different array waveguides. Only wavelengths that satisfy a specific phase-matching condition (i.e., light from all array waveguides interferes in phase at that point) can form a focused spot at that convergence point (corresponding to the entrance of a certain output waveguide) and be efficiently coupled out. Each output port of the AWG corresponds to a specific center transmission wavelength. Therefore, when a broadband spectral signal is input into the AWG, its different spectral components will be automatically and passively routed by the AWG to different spatially separated output ports. By monitoring the optical power of these output ports, the intensity distribution of the input spectral signal in each wavelength channel can be directly obtained, thereby achieving parallel sensing and sampling of the input spectrum.

[0040] This invention utilizes the aforementioned sensing characteristics of AWG and its parallel routing capability to perform optical convolution operations.

[0041] Kernel Loading and Weighting: The weights (kernel elements) in the convolution operation are loaded using an on-chip integrated array of tunable optical weighting units (e.g., a set of Mach-Zehnder interferometers, MZIs). The optical signal input to the AWG (or its separated wavelength components) is first weighted in parallel by these MZIs. Each MZI controls the transmittance of the input optical signal by adjusting the state of its internal phase shifter, thereby weighting the optical signal.

[0042] Parallel computation of convolution: Multiple optical signals, weighted by an MZI array, are fed in parallel into different input ports of the AWG. Leveraging its specific optical path routing characteristics from multiple input ports to multiple output ports, the AWG converges optical signals from different input paths at each (or part of) the output port. The superposition of these optical signals at the output port (typically intensity superposition) corresponds to the dot product operation (multiplication and addition) of the input spectral vector and the convolution kernel at specific relative positions. Since the AWG has multiple output ports, these output ports can output different elements of the series of results generated by the entire convolution operation in parallel, thus achieving a high degree of parallelization of the convolution operation.

[0043] Two-dimensional convolution mapping: For convolutions of two-dimensional data such as images, a method of mapping them to one-dimensional convolutions can be used to adapt to the AWG architecture. An N×N two-dimensional input feature map can be flattened into a one-dimensional vector. Similarly, an M×M two-dimensional convolution kernel can also be flattened and a specific number of zeros inserted between its elements to form a one-dimensional equivalent convolution kernel with a specific structure. The length of this equivalent one-dimensional convolution kernel and the position of the zero padding need to be determined according to the stride and sliding method of the original two-dimensional convolution to ensure that the shift operation of the one-dimensional convolution can accurately simulate the sliding coverage of the two-dimensional convolution kernel on the input feature map. Through this mapping, the two-dimensional convolution problem is transformed into a one-dimensional convolution problem that AWG can efficiently handle.

[0044] The core of "integrated sensing-convolution" lies in the fact that while the AWG performs wavelength separation (i.e., spectral sensing) on ​​the input broadband spectral signal, these separated (sensed) wavelength components carrying the original spectral information can be directly used as input features for convolution operations, participating in subsequent weighting and superposition processes. The output of the sensing process is directly fed into or constitutes the input data for convolution calculation, avoiding separate sensing and computation modules. This design simplifies the structure, reduces on-chip signal transmission, and improves overall processing efficiency and integration.

[0045] Nonlinear activation layer 6 follows each convolutional layer 5 and is responsible for implementing nonlinear activation, optional average pooling, and lossless or gain cascading of signals. Its basic workflow is as follows: it receives multiple parallel optical output signals from the preceding AWG sensing-convolutional unit; these optical signals are converted into corresponding electrical signals (photocurrents) by a photodetector (PD) array; these electrical signals are then used to drive an optical modulator array; each optical modulator is provided with optical input by an independent "supply light" and modulates the supply light according to the driving current; the modulated supply light is then fed as the output of that unit into the next stage processing unit.

[0046] PD type and characteristics: High-speed, high-response photodetectors compatible with silicon photonics technology can be used, such as PIN photodiodes based on germanium-silicon (Ge-on-Si), whose response spectrum range needs to cover the operating wavelength of the chip.

[0047] Multi-port PD for average pooling: To achieve average pooling, a specially designed PD structure can be used, such as... Figure 2 (c) on the left illustrates "Multi-port photodetectors used for pooling". A PD unit can be designed with a large photosensitive area and configured to simultaneously receive optical signals from multiple adjacent output ports of the preceding AWG (logically corresponding to a pooling window). The total photocurrent generated by the PD approximately represents the average or sum of the light intensities at these ports. Alternatively, the currents of multiple independent single-port PDs can be combined in the electrical domain to achieve pooling.

[0048] Modulator Types and Principles: Various types of on-chip integrated optical modulators can be used, such as microring resonators (MRMs), Mach-Zehnder modulators (MZMs), or more advanced ones. Figure 2 The example on the right of (c) is a "FSR-free nanobeam electro-optic modulator". These modulators use electrical signals to change optical properties such as the refractive index of an optical waveguide, thereby modulating the transmitted optical signal.

[0049] Advantages of devices such as nanobeam modulators: Using modulators such as nanobeam modulators, which have no obvious free spectral range (FSR) limitation or have a very large FSR, is beneficial for supporting multi-wavelength parallel operation over a wider spectral range, or for integrating more parallel modulation channels at a higher density, making them suitable for large-scale parallel processing.

[0050] Supplying light is the core mechanism for achieving cascadability.

[0051] Characteristics of the supply light: The supply light is typically a continuous beam of relatively stable and high power. Its wavelength needs to be precisely matched with the operating wavelength of the selected optical modulator.

[0052] Injection method: The supply light can be distributed and injected into each modulation unit in the optical modulator array from an external light source or an on-chip integrated light source through an on-chip integrated waveguide network.

[0053] Function (achieving net signal gain / loss compensation): The photocurrent generated by the PD drives the optical modulator, thereby modulating the high-power "supply light". The power of the optical signal output from the nonlinear activation unit depends primarily on the original power of the supply light and the modulation depth of the optical modulator under this driving current. By setting the power of the supply light to a suitable high level, the power of the output optical signal can be made greater than the original signal power input to the PD, achieving a "net gain" in signal power, or at least compensating for the losses in the preceding stage.

[0054] Nonlinear activation is achieved by driving an optical modulator with the photocurrent generated by the photodiode (PD), utilizing the modulator's inherent nonlinear electro-optic response characteristics. For example, when the driving current changes, the modulator's transmission or reflection power to the supplied light will exhibit a nonlinear relationship with the input photocurrent, thus realizing nonlinear activation. By precisely designing the modulator's operating point and the characteristics of the supplied light, the desired shape of the nonlinear activation function can be customized to a certain extent.

[0055] This invention utilizes the aforementioned "AWG sensing-convolution integrated unit" (such as...) Figure 2 (b) “Optical convolutional layer 1”, “Optical convolutional layer 2”, up to “Optical convolutional layer N”) and “Cascadeable optoelectronic nonlinear-average pooling integrated unit” (e.g. Figure 2 (b) The “non-linear activation layer” after each convolutional layer is alternately connected to construct a deep convolutional neural network architecture.

[0056] The specific cascading method is as follows: The first-layer AWG sensing-convolutional unit processes the raw spectral signal from the multimodal input fusion front-end. Its output parallel optical signal is fed into the first-layer photoelectric nonlinear-average pooling unit. The output optical signal of this nonlinear-pooling unit is then fed as input into the second-layer AWG sensing-convolutional unit. This process is repeated layer by layer, with the signal undergoing convolutional feature extraction and nonlinear / pooling processing. The independent injection of "supply light" at each stage is crucial to ensuring that the signal can be effectively transmitted throughout the entire deep network.

[0057] After multi-layer convolution and nonlinear pooling operations, the extracted high-level abstract feature vectors can be fed into one or more optical fully connected layers for final feature integration and classification or regression tasks. The optical fully connected layers can be implemented using optical matrix multipliers based on MZI arrays. The matrix weights can be set by precisely controlling the state of each phase shifter in the MZI array.

[0058] At the final output of the optical neural network, a high-speed, high-sensitivity photodetector array is deployed to read the final optical calculation results. These detectors convert the final optical calculation results into corresponding electrical signals. These electrical signals may then require further electrical signal processing, such as transimpedance amplification, analog-to-digital conversion, and digital logic decision-making.

[0059] To enable the optical computing chip of this invention to function properly and achieve reconfigurability, a stable, precise, and programmable control and bias unit is required. This unit is typically composed of off-chip components (such as FPGAs or microcontrollers) or some on-chip integrated auxiliary circuitry. Its main functions include driving and biasing tunable optical elements (such as MZI phase shifters, modulator biasing, and supplying optical power), data loading and reading, and, in the case of supporting on-chip training, algorithm execution and parameter updates.

[0060] The optical computing chip described in this invention is designed to be highly compatible with current mainstream and mature CMOS microelectronics manufacturing processes, enabling large-scale, high-yield, and low-cost production using the advanced infrastructure of the existing semiconductor industry. A typical preferred implementation platform is a silicon-on-insulator (SOI) wafer.

[0061] The main process steps in chip manufacturing may include: wafer preparation; precisely defining the patterns of passive optical components such as waveguides, AWGs, and MZIs through photolithography and etching processes and transferring them to the top silicon layer; forming P-type and N-type doping distributions in specific areas of the top silicon layer through multiple ion implantation steps; for photodetectors, selective epitaxial growth of germanium thin films on silicon waveguides may be required; forming metal microheaters (for thermo-optic modulation) and metal electrodes for all active devices through thin film deposition and patterning processes; constructing multilayer metal interconnect wiring; and finally, surface passivation and packaging preparation. The entire manufacturing process is carried out in an ultra-clean environment with strict process control.

[0062] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An end-to-end optical computing chip based on multimodal analog signal fusion processing, characterized in that, include: The system comprises a multimodal input fusion front-end module (1), an end-to-end inference module (3), and an optical fiber input interface (2) connecting the two. The multimodal input fusion front-end module (1) is used to convert at least two different types of original analog input signals into a unified broadband spectral input signal. The end-to-end inference module (3) includes a sensing layer (4), multiple alternating cascaded convolutional layers (5) and nonlinear activation layers (6), a fully connected layer (7), and an output layer (8). The sensing layer (4) is used to perform spectral sensing on the broadband spectral input signal. The multiple alternating cascaded convolutional layers (5) and nonlinear activation layers (6) are respectively used to process the spectrum sensed by the sensing layer (4). Optical convolution operation and nonlinear activation; the convolutional layer (5) includes a tunable weighting unit, which is a Mach-Zehnder interferometer (MZI) array. The MZI array is used to load convolution kernel weights. The convolutional layer (5) uses its multi-input multi-output optical path routing characteristics to converge and superimpose optical signals from different input paths that have passed through the MZI array and loaded with convolution kernel weights at its output port to complete the optical convolution operation; the fully connected layer (7) is used to perform feature integration and classification tasks on the high-level abstract feature vectors extracted by the multi-layer alternating convolutional layer (5) and nonlinear activation layer (6), and finally outputs the classification result through the output layer (8).

2. The chip according to claim 1, characterized in that, The multimodal input fusion front-end module (1) includes a random scattering device for scattering the spatial light field carrying image information and mapping it to the spectral domain, converting it into a part of the broadband spectral input signal. The random scattering device includes one or more waveguide arrays, each waveguide having a grating structure with random or pseudo-random parameters.

3. The chip according to claim 1, characterized in that, The multimodal input fusion front-end module (1) includes an electro-optic phase modulator for phase modulation of the optical carrier of the radio frequency signal, and the resulting spectral sideband is converted into a part of the broadband spectral input signal.

4. The chip according to claim 1, characterized in that, The sensing layer (4) includes an arrayed waveguide grating, which is used to realize spectral sensing of the input spectral signal by utilizing its inherent wavelength dispersion characteristics and multi-port spatial separation capability.

5. The chip according to claim 1, characterized in that, The convolutional layer (5) is an arrayed waveguide grating.

6. The chip according to claim 5, characterized in that, The nonlinear activation layer (6) unit includes a photodetector, an optical modulator, and a supply light source. The photodetector is used to convert the optical signal output by the optical convolution operation into an electrical signal. The optical modulator is driven by the electrical signal. The supply light source provides light input to the optical modulator, so that the nonlinear activation layer (6) realizes the nonlinear activation function and effectively compensates for optical path loss or realizes signal net gain through the injection of the supply light source.

7. The chip according to claim 6, characterized in that, The photodetector in the nonlinear activation layer (6) is a multi-port photodetector, or the output currents of multiple photodetectors are combined in the electric domain to converge the optical signals from multiple output ports of the convolutional layer (5), thereby realizing the average pooling function.

8. The chip according to claim 6 or 7, characterized in that, The optical modulator in the nonlinear activation layer (6) is a micro-ring resonator or a nanobeam optical cavity modulator.

9. The chip according to claim 1, characterized in that, The fully connected layer (7) uses a Mach-Zehnder interferometer (MZI) array to perform matrix multiplication.

Images