Ultra-high-speed end-to-end all-optical intelligent computing chip architecture and system

Through all-optical parallel computing array chips, image processing, transmission and reconstruction are solved in the optical domain, bandwidth limitation and energy efficiency bottlenecks in the prior art are achieved, high-energy-efficient and high-speed image processing is achieved, and edge computing and intelligent robot applications are supported.

CN118537204BActive Publication Date: 2024-12-03TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410755805.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-12
Publication Date
2024-12-03
Estimated Expiration
2044-06-12

AI Technical Summary

Technical Problem

The prior art has bandwidth limitations and energy efficiency bottlenecks in image processing, transmission and reconstruction, especially the frame rate and energy efficiency limits caused by frequent photoelectric conversion of electronic circuits, and lacks a unified optical computing model and parallel processing capabilities.

Method used

Using all-optical parallel computing array chip (OPCA chip) to realize end-to-end processing, transmission and reconstruction of images in the optical domain, focus the input spatial light through the microlens array, and use all-optical parallel computing array to perform multi-spectral detection and calculation, and transmit The module transmits the results to the all-optical reconstruction module for light field reconstruction, realizing the processing of nanosecond response time and terahertz bandwidth.

Benefits of technology

It realizes high-efficiency and high-speed image processing and transmission, supports edge computing and intelligent robot applications, reduces the energy consumption and delay caused by photoelectric conversion, and improves the frame rate and energy efficiency of image processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118537204B_ABST
    Figure CN118537204B_ABST
Patent Text Reader

Abstract

The present invention discloses an ultra-high-speed end-to-end all-optical intelligent computing chip architecture and system, which includes a microlens array for focusing input spatial light onto a sensing unit; an all-optical parallel computing array including a plurality of sensing units, the sensing unit for responding to the input spatial light and outputting all-optical computing results through multi-spectral all-optical computing based on the input multi-spectral detection light; a transmission module for transmitting the all-optical computing results to an all-optical reconstruction module; and an all-optical reconstruction module for performing light field reconstruction based on the all-optical computing results to obtain light field reconstruction results. The computing process of the present invention is conducive to achieving high energy efficiency and high-speed computing of the light field.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, transmission and reconstruction, and in particular to an ultra-high-speed end-to-end all-optical intelligent computing chip architecture and system. Background Art

[0002] Image processing, transmission, and reconstruction constitute the main part of information technology. The rapid development of edge devices and data centers has put forward higher requirements on the bandwidth and efficiency of image processing, transmission, and reconstruction. The frequent conversion of serial signals between the optical and electronic fields and the gradual saturation of electronic processor performance have become the bottleneck of end-to-end machine vision.

[0003] With the widespread deployment of sensing and computing modules at the edge of the Internet, image processing, transmission, and reconstruction are becoming increasingly important. Optics plays a key role in imaging, communication, and display, while digital CMOS electronics dominates the field of image sensing and processing. The development of optics and electronics has successfully improved their respective transmission bandwidths and processing speeds. However, the frame rate and energy efficiency of operations such as spatial image processing, transmission, and reconstruction are approaching their limits. These challenges mainly come from two aspects: (1) The performance of electronic circuits is gradually saturated, limiting the potential for significant improvements in signal processing speed and energy efficiency. (2) Image operations require frequent serial conversions between the electronic and optical domains during image sensing and transmission. These conversions result in a lot of energy consumption, which limits the frame rate of these operations.

[0004] In order to alleviate the bottleneck problem of integrated electronic circuits, a potential solution is to perform image processing in parallel directly in the optical domain. At present, some researchers use spatial optical modulation elements to pre-process images and then input them into the electronic system for post-processing. Among them, the more representative works are optoelectronic neural networks and all-optical neural networks, both of which have been developed for spatial image classification. However, none of the existing photonic computing methods can directly process spatial intensity images. Most optical image processors use electronic cameras to either record spatial images and input them into optical computing modules, or to collect pre-processing outputs and input them into electronic post-processing systems.

[0005] Another line of research focuses on developing smart sensors that transfer some digital operations to or near the sensor. This approach does not extract raw image pixels, but only reads valid data in them, including spatial image features and temporal dynamics, alleviating bandwidth pressure between sensing and computing. Although these sensors do not need to read out raw pixels, the output still needs to be read out sequentially because current cannot be multiplexed. This limits the increase in sensing and processing frame rates. More importantly, the transmission and reconstruction operations must undergo EO (electrical / optical) / OE (optical / electrical) and DA (digital / analog) / AD (analog / analog) conversions, which are limited by the current saturation of electronic circuit development.

[0006] All-optical computing paradigms have the potential to address the bandwidth limitations inherent in hybrid optoelectronic processing, transmission, and reconstruction. Nevertheless, several technical challenges remain. (1) Intensity image recording and image processing are separate. While the effectiveness of optical computing has been demonstrated, direct processing of optical intensity inputs remains challenging because only fully positive convolutions or dot products are currently achievable. (2) Reconfigurable and parallel optical image processing is difficult to achieve. Currently, optical images are read out serially via electronic cameras, limiting the frame rate of imaging and image processing. (3) The lack of a unified optical computing model that integrates image processing, transmission, and reconstruction results in separate models and frequent optoelectronic conversions. Summary of the invention

[0007] The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

[0008] To this end, the present invention proposes an all-optical parallel computational array chip (OPCA chip) in an embodiment of the present invention for end-to-end processing, transmission and reconstruction of light field images. The response time of the OPCA chip is 6 nanoseconds, and the optical bandwidth is at least 160 nanometers. All computing operations in the OPCA chip are implemented in the optical domain, so there is no energy consumption or delay caused by photoelectric conversion. The all-optical computing sensor proposed in the present invention opens the door to extremely high-speed processing, transmission and reconstruction of visible content with nanosecond response time and terahertz bandwidth. Based on the chip, an ultra-high-speed end-to-end all-optical intelligent computing chip system is proposed.

[0009] Another object of the present invention is to provide an ultra-high-speed end-to-end all-optical intelligent computing method.

[0010] a microlens array for focusing input spatial light onto the sensing unit;

[0011] An all-optical parallel computing array comprises a plurality of the sensing units, wherein the sensing units are used to respond to the input spatial light and output all-optical computing results through multi-spectral all-optical computing based on the input multi-spectral detection light;

[0012] A transmission module, used for transmitting the plenoptic calculation result to a plenoptic reconstruction module;

[0013] The plenoptic reconstruction module is used to perform light field reconstruction based on the plenoptic calculation result to obtain a light field reconstruction result.

[0014] The ultra-high-speed end-to-end all-optical intelligent computing chip system of the embodiment of the present invention may also have the following additional technical features:

[0015] In one embodiment of the present invention, one of the sensing units is a resonant ring, and a plurality of the sensing units are physically connected via a bus waveguide.

[0016] In one embodiment of the present invention, the microlens array comprises a microlens array with matching period and numerical aperture, and the microlens array is also used to confine the input spatial light to the sensing area of ​​the resonant ring.

[0017] In one embodiment of the present invention, the resonator of the resonant ring is used to perform phase and / or amplitude modulation on the light propagating in the bus waveguide.

[0018] In one embodiment of the present invention, the sensing unit is further configured to respond to the same wavelength when the multi-spectral detection light is input into the bus waveguide, so as to output the coherently combined multi-spectral detection light through the output end of the sensing unit.

[0019] In one embodiment of the present invention, the plenoptical reconstruction module comprises a plenoptical renderer, wherein the plenoptical renderer is used to:

[0020] Constructing the all-optical parallel computing array and the all-optical renderer based on the blazed grating and the diffraction network;

[0021] Reflecting multi-spectral detection light of different wavelengths to a preset spatial position through the blazed grating;

[0022] The diffraction network is used to diffract the reflected multi-spectral detection light of different wavelengths through different phase masks at corresponding spatial positions through an all-optical renderer to reconstruct the light field.

[0023] In one embodiment of the present invention, the all-optical parallel computing array includes a plurality of all-optical parallel computing units connected via a bus.

[0024] In one embodiment of the present invention, the all-optical parallel computing unit is used to obtain a gradient operator based on calculation results of addition and subtraction on multiple all-optical parallel computing units; wherein the gradient operator is used for image feature extraction of the input spatial light.

[0025] In one embodiment of the present invention, a pump-probe experimental device is also included, and the experimental device is used to measure the response time of the all-optical parallel computing unit.

[0026] To achieve the above object, the present invention further provides an ultra-high-speed end-to-end all-optical intelligent computing method, comprising:

[0027] The input spatial light is focused onto the sensing unit through a microlens array;

[0028] Using the sensing unit of the all-optical parallel computing array to respond to the input spatial light, and outputting the all-optical computing result through multi-spectral all-optical computing based on the input multi-spectral detection light;

[0029] Light field reconstruction is performed based on the plenoptic calculation result to obtain a light field reconstruction result.

[0030] The ultra-high-speed end-to-end all-optical intelligent computing chip system and method of the embodiments of the present invention are based on an all-optical parallel computing array chip (OPCA chip), which performs calculations completely in the optical domain, is conducive to high-efficiency and high-speed computing, and supports edge computing, intelligent robots and other applications.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0033] Figure 1 is a structural diagram of an ultra-high-speed end-to-end all-optical intelligent computing chip system according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a network for end-to-end image processing, transmission and reconstruction using an all-optical computing sensor according to an embodiment of the present invention;

[0035] Figure 3 is a schematic diagram of the working principle of an OPCA unit according to an embodiment of the present invention;

[0036] Figure 4 is a schematic diagram of response time measurement of an OPCA unit according to an embodiment of the present invention;

[0037] Figure 5 is a network diagram for image processing using OPCA according to an embodiment of the present invention;

[0038] Figure 6 is a schematic diagram of a network for end-to-end image sensing, processing, transmission and reconstruction using an OPCA chip according to an embodiment of the present invention;

[0039] Figure 7 It is a flow chart of an ultra-high-speed end-to-end all-optical intelligent computing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0041] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] The following describes an ultra-high-speed end-to-end all-optical intelligent computing chip system and method according to an embodiment of the present invention with reference to the accompanying drawings.

[0043] Figure 1 It is a schematic diagram of the structure of an ultra-high-speed end-to-end all-optical intelligent computing chip system according to an embodiment of the present invention.

[0044] like Figure 1 As shown, the system 10 includes:

[0045] A microlens array 100 for focusing input spatial light onto a sensing unit;

[0046] The all-optical parallel computing array 200 includes a plurality of sensing units, the sensing units being used to respond to input spatial light and output all-optical computing results through multi-spectral all-optical computing based on the input multi-spectral detection light;

[0047] The transmission module 300 is used to transmit the plenoptic calculation result to the plenoptic reconstruction module 400;

[0048] The plenoptic reconstruction module 400 is used to perform light field reconstruction based on the plenoptic calculation result to obtain a light field reconstruction result.

[0049] In an embodiment of the present invention, the incident light can be directly irradiated onto a high-Q value (high quality factor) resonant ring on the OPCA chip based on the OPCA chip. The intensity information carried by the visible light modulates the detection light signal and is processed on the chip; by moving the resonant wavelength of the high-Q value resonant ring, the encoded detection light signal can perform constructive or destructive interference, which is equivalent to addition and subtraction operations; due to the large optical bandwidth, different light spots in the sensor unit can be calculated in parallel through wavelength multiplexing; the calculation results can be transmitted to the far field in real time and reconstructed by a jointly designed multi-wavelength photonic processor. The response time of the sensor is 6 nanoseconds and the optical bandwidth is 160 nanometers (1480 nanometers to 1640 nanometers). Since all operations are implemented in the optical domain, the calculation process is conducive to high-energy efficiency and high-speed computing, supporting applications such as edge computing and intelligent robots.

[0050] Figure 2 The present invention utilizes an all-optical computing sensor for end-to-end image processing, transmission and reconstruction. Figure 2 (a) Schematic diagram of an end-to-end multi-wavelength all-optical neural network composed of an all-optical parallel computing array chip (OPCA chip). It can be understood that the image incident on the all-optical sensor and each unit is focused onto the resonant ring, which modulates the multi-spectral detection light (neuron activation). The spectral responsivity of each ring is adjusted, and the external spatial input is spectrally selectively weighted and fully connected (all-optical computing). The multi-spectral output containing the calculation results can be demultiplexed for classification decisions, or transmitted to the far field for further reconstruction (all-optical reconstruction). Figure 2 (b) is an optical microscope image of an 8×8 OPCA chip. Scale bar: 0.3 mm.

[0051] In one embodiment of the present invention, spatial light is focused by a lens array and input into an OPCA chip; the OPCA chip responds to the input information and outputs the calculation results through multi-spectral all-optical calculation. The calculation results of OPCA are transmitted over long distances through optical fibers. The calculation results of OPCA are input into a jointly designed all-optical reconstruction module, and the original scene is finally reconstructed.

[0052] Specifically, the schematic diagram of the end-to-end multi-wavelength neural network based on the all-optical parallel computing array chip (OPCA chip) is as follows: Figure 2 As shown in (a).

[0053] Exemplarily, each sensing unit in the computational array is a high-Q all-pass resonant ring. The units are physically connected via bus waveguides. Above the unit array, a microlens array with matching period and numerical aperture (NA) confines the input light field to the sensing area of ​​the resonant ring. During inference, the object is imaged onto the sensor array, represented as x i,j(i=1,2,..,M;j=1,2,...,N), where M and N represent the number of rows and columns of the sensor array respectively. The light from each unit area is focused onto the resonant ring through the corresponding microlens. The focused light is absorbed by the silicon waveguide to generate photogenerated carriers. Due to the high Q value of the resonator, the refractive index change caused by the carriers is amplified, thereby phase and / or amplitude modulating the light propagating in the bus waveguide. Figure 2 In (a), x i,j The phase and / or amplitude change caused is denoted as h i,j Different resonators are biased to different resonant wavelengths. When multispectral probe light is input into the bus waveguide, it can be understood that (the multispectral probe light is the carrier of the system calculation; the original input is the external ambient light.) A group of units will respond to the same wavelength, and the output light through the output port of the bus waveguide is coherently combined. The operating wavelength in the multispectral probe light is denoted as λ k (k=1,2,...,K), where K represents the number of working wavelengths. This process establishes a linear connection between these units. At the output end of the chip array, the spectral intensity information represents the decision output and can be demultiplexed and read out. It can be known that after the multi-spectral detection light passes through the system, the light components of different wavelengths will be different. Through the filter and optical power meter (or optical detector) at the output end of the system, the intensity of light of different wavelengths can be measured separately, and these intensity changes represent the system output.

[0054] For example, the output of the chip array, i.e., the multi-spectral detection light containing spectral intensity information, will be transmitted to a remote end for reconstruction. The all-optical renderer uses a blazed grating and a diffraction network designed together with the sensor array to achieve reconstruction: the blazed grating reflects light of different wavelengths to a specific spatial position, represented as h′ k (k=1,2,..,K). The reflected light will then be diffracted by different phase masks at the corresponding positions, and finally the light field will be reconstructed. The reconstruction result is Figure 2 In (a), it is represented as x′ ij .

[0055] It can be understood that the computing sensor of the OPCA of the embodiment of the present invention basically establishes a spectral-spatial optical neural network. The spatial input activates the spectral neurons, and through the weighted connection layer of the spectrum, parallel processing of the entire optical domain is achieved. The optical microscope image of the OPCA chip proposed by the present invention is shown in FIG. Figure 2 In (b), light of different wavelengths is input into the device through the detection port, and external light of different intensities is input into the device through the excitation port. The output light intensity of the device under different conditions is recorded in turn to obtain the transmission efficiency change curve.

[0056] Further, Figure 3The working principle of the OPCA unit is shown below. Figure 3 (a) Schematic diagram of a single OPCA unit. The light and dark curves represent the responses of the OPCA unit with and without an external focused beam, respectively. Figure 3 (b) The transmission response of the OPCA unit to incident light of different light intensities. Left: Coherent light input. Right: Incoherent light input. (cd) Coherent constructive responses of the same incident light Figure 3 (c) and coherence cancellation Figure 3 (d) bias in the figure. The right sub-figures are, from left to right, the responses of the first unit with and without an external beam, the responses of the second unit with and without an external beam, and the cumulative responses of the two units connected in series with and without an external beam. Magenta curves of different colors represent the responses when external beams of different intensities are input, and the lighter the magenta, the greater the light intensity. Figure 3 (e) Experimental results of image convolution using OPCA chip. (From left to right: original image, horizontal gradient, vertical gradient and gradient magnitude).

[0057] In one embodiment of the present invention, Figure 3 The working principle of a single OPCA unit and the gradient operator obtained by combining two OPCA units are demonstrated, wherein the OPCA units are arranged in an array and connected through a bus to obtain the above-mentioned OPCA chip. The gradient operator is a basic operation of image processing, which can obtain the edge of the image and help extract image features. Figure 3 (a) in the figure describes the working mechanism of the OPCA unit: the core component consists of a racetrack ring resonator. The incident light beam is focused on the waveguide core. Due to the carrier dispersion effect, the refractive index of the waveguide changes. This change changes the spectral transmission characteristics of the resonant ring waveguide, thereby changing the transmission efficiency. When the external light beam is focused on the OPCA unit, the transmission efficiency of different wavelengths is different. Figure 3 Taking the four wavelengths λ1 to λ4 in (a) as an example, the transmission efficiency of λ1 and λ4 remains unchanged, while the transmission efficiency of λ2 increases, and the transmission efficiency of λ3 decreases. Figure 3 (b) shows the responses of two OPCA units under coherent and incoherent input light, respectively. The coherent input is a 532nm continuous wave (CW) laser, and the incoherent input is a light-emitting diode (LED). It can be seen that the transmission efficiency of the OPCA unit changes with the change of the incident light intensity.

[0058] Specifically, according to the response of the transmission efficiency to the incident light intensity, the embodiment of the present invention develops two basic calculation modes of OPCA: addition and subtraction. When the bias wavelength of the two units is exactly the resonant wavelength, both elements are in the minimum transmission state. Since the index change caused by the carriers will cause the resonant wavelength to redshift, the incident light on the two resonant sensing units will coherently increase the transmission efficiency at this wavelength, which is equivalent to an addition operation, such as Figure 3 Conversely, when one sensing unit is in a resonant bias state and the resonant wavelength of the other unit is blue-shifted, the external incident light will enhance the transmission efficiency of the first unit and reduce the transmission efficiency of the second unit, resulting in a subtraction operation, as shown in (c) in Fig. Figure 3 As shown in (d) in the figure. The embodiment of the present invention performs basic image processing operations on a 48×64 pixel image by constructing addition and subtraction operations. The image is divided into several sub-blocks, and each sub-block is subjected to addition and subtraction operations by two OPCA units. The experimental results are shown in Figure 3 As shown in (e) in the figure, an edge detection filter is applied to obtain the intensity gradient of the image in the horizontal and vertical directions. The two edge filters together form a "Sobel-type" image filter to obtain the gradient of the image.

[0059] Further, Figure 4 The response time of the OPCA unit is measured. The measured full width at half maximum (FWHM) τ(pump) = 12.4ns, τ(detection) = 18.4ns. The probe measurement results show that the response time is about 6ns.

[0060] Specifically, in order to measure the response time of the OPCA unit, a pump-probe experimental device was constructed in the embodiment of the present invention. The output of the mode-locked laser was split by a 50:50 splitter, one part of which was monitored by a 5GHz bandwidth photodiode, and the other half of the beam was focused and incident on the OPCA unit. The laser wavelength was set near the resonance peak of the ring. In the experiment, the signal collected by the oscilloscope was as follows: Figure 4 As shown. From the signal collected by the oscilloscope, it can be obtained that the full width at half maximum (FWHM) of the pump light and the detection light are τ(pump) = 12.4ns and τ(detection) = 18.4ns respectively, so the response time is about 6.0ns. In addition, the embodiment of the present invention also measures the working wavelength range of the OPCA unit, and the unit performs well within the output wavelength range of the tunable laser (1480nm to 1640nm).

[0061] Further, Figure 5 Schematic diagram of image processing using OPCA. Figure 5 (a) is a photograph of the packaging results of the manufactured chip array. Figure 5 (b) in Figure 5 Optical microscope image of the chip array in (a). Figure 5 (c) Spectral responses of 16 sensing units and four operating wavelengths. Figure 5 (d) Variation of working wavelength light intensity of two samples in the test set. Figure 5 (e) Confusion matrix of experimental classification results. Scale bar: 0.3 mm.

[0062] Specifically, in Figure 5 In (a)-(b), an embodiment of the present invention demonstrates a fully packaged OPCA integrated circuit, which consists of a 4×4 array of sensing and computing units connected by a waveguide sequence. Each unit is controlled by a metal heater to bias the spectral response. During the entire experiment, the input light field is imaged onto the OPCA chip through a microscope. The optical intensity image is then focused onto the sensing area through a microlens array, generating a response in each OPCA unit. The on-chip computational results are accumulated at the output end of the waveguide and multi-wavelength output is achieved through time division multiplexing in the experiment. These multi-wavelength outputs can also be separated simultaneously through a DWDM (dense wavelength division multiplexing) demultiplexer.

[0063] The present invention evaluates the classification performance on four handwritten digits "0", "1", "2" and "3" in the MNIST dataset. In the experiment, the present invention constructs a neural network based on the spectral response of the OPCA and the measured pump light modulation characteristics. 22731 samples are used for training and 200 samples are used for testing. Specifically, the present invention configures the thermal bias of the light ring and the input working wavelength as parameters for image classification training. Figure 5 (c) in FIG. 5 shows the spectral response of 16 OPCA units and four working wavelengths. The embodiment of the present invention randomly selected 200 images from the test set to evaluate the accuracy. The overall experimental accuracy of the 200 samples was 73.50%, which is comparable to the simulation test accuracy of 75.50%. Figure 5 (d) in shows two examples from the first row to the last row. The confusion matrix is ​​shown in Figure 5 (e) in .

[0064] further, Figure 6 Schematic diagram of end-to-end image sensing, processing, transmission, and reconstruction using OPCA chips. Figure 6 (a) Experimental setting for end-to-end reconstruction. Figure 6 (b) Wavelength settings for the two reconstruction tasks. Figure 6 (c) The trained spatial light modulator (SLM) phase mask. Figure 6 (d) Input image in . Figure 6 (e) Experimental results. Scale bar: 0.5 mm.

[0065] It is understandable that since the sensing and calculation results are both in the waveguide optical domain, they can be transmitted to the far field through optical fiber without the need for photoelectric conversion. Figure 2 As shown in (a) of FIG. 1 , an end-to-end multi-wavelength all-optical neural network is developed in the embodiment of the present invention, which integrates processing, transmission and reconstruction. The experimental setup of the network is as follows: Figure 6 As shown in (a), it consists of a processing module and a rendering module. The experimental setup of the processing module is similar to the setup of all-optical image recognition, except that a 3×3 OPCA sub-chip is used. The processing module will output modulated multi-wavelength light, which is transmitted to the rendering module through optical fiber. After optical fiber collimation, the light is irradiated onto the blazed grating and thereby reflected to different areas on the phase modulator. By training the phase mask together with the OPCA chip, light of different wavelengths is diffracted on the camera to form different patterns. Figure 6 In the embodiment of the present invention, two all-optical reconstruction schemes are experimentally tested on the network: "T" and "H" letter compression reconstruction and 3×3 pixel-by-pixel reconstruction. Figure 6 (b) shows the wavelength settings for these tasks. Figure 6 (c) in Figure 1 shows the trained phase mask, clearly showing the corresponding regions at different wavelengths. Figure 6 (d) in the experiment, the experimental results are shown in Figure 6 (e) in the figure. The average signal-to-noise ratio (SNR) of the two tasks is 11.69dB and 11.32dB respectively.

[0066] In summary, the present invention provides an all-optical parallel computing array chip (OPCA chip) for image sensing, processing, transmission and reconstruction. The OPCA unit utilizes the carrier dispersion effect caused by the incident light beam to perform intelligent image processing operations on daily scene images. The measured response time is 6.0ns, the bandwidth is 160nm, and it supports hundreds of optical frequency combs with on-chip Kerr frequency comb teeth

[21] . The high response speed and large optical bandwidth help to achieve sub-THz spatial optical processing bandwidth. The classification performance of the 4×4 OPCA chip was evaluated on the MNIST dataset in the embodiment of the present invention. The overall experimental accuracy of 200 samples was 73.5%, which shows that the OPCA chip can reason about natural scenes without photoelectric conversion. In addition, the OPCA chip enables the embodiment of the present invention to build an end-to-end multi-wavelength all-optical neural network that integrates perception, computing and reconstruction. The research results of the embodiment of the present invention show that the network performs well in compressed reconstruction and pixel-by-pixel reconstruction, with signal-to-noise ratios of 11.69dB and 11.32dB, respectively.

[0067] In the existing photoelectric sensor-based processing flow, the imaging speed is limited by the serial readout and digital transmission bandwidth, while the image processing speed is limited by the clock frequency of the digital electronic device. Frequent conversions between the analog and digital domains and between the electronic and optical domains will bring huge energy overhead. The all-optical natural image processing sensor proposed in the present invention has many advantages: first, the image can be processed in parallel directly in the sensor without being limited by serial readout. Second, end-to-end operations, including image sensing, processing, transmission and reconstruction, all occur in the optical domain, thus reducing the burden of O / E and E / O conversions and A / D and D / A conversions, thereby supporting high-energy efficiency and high-speed processing.

[0068] Given the exponential growth of images stored on edge devices, image processing has increasingly higher computational requirements for power consumption and processing time. Therefore, it is crucial to find an efficient image processing method that is both energy-efficient and fast. The OPCA proposed in the present invention has the unique characteristics of all-optical computing and has significant advantages in processing speed and energy efficiency. The embodiments of the present invention predict that the all-optical neural network driven by the OPCA chip will be widely used in edge computing, intelligent robots and other fields.

[0069] In the ultra-high-speed end-to-end all-optical intelligent computing chip system of the embodiment of the present invention, the incident light directly irradiates the high-Q value resonant ring on the OPCA chip. The intensity information carried by the visible light modulates the detection light signal and is processed on the chip; by moving the resonant wavelength of the high-Q value resonant ring, the encoded detection light signal can perform constructive or destructive interference, which is equivalent to addition and subtraction operations; due to the large optical bandwidth, different light spots in the sensor unit can be calculated in parallel through wavelength multiplexing; the calculation results can be transmitted to the far field in real time, and high-quality calculation reconstruction can be performed through a jointly designed multi-wavelength photon processor.

[0070] In order to implement the above embodiment, Figure 7 As shown, this embodiment also provides an ultra-high-speed end-to-end all-optical intelligent computing method, including:

[0071] S1, focusing the input spatial light onto the sensing unit through a microlens array;

[0072] S2, using the sensing units of the all-optical parallel computing array to respond to the input spatial light, and outputting all-optical computing results through multi-spectral all-optical computing based on the input multi-spectral detection light;

[0073] S3, performing light field reconstruction based on the plenoptic calculation result to obtain a light field reconstruction result.

[0074] In the ultra-high-speed end-to-end all-optical intelligent computing method of the embodiment of the present invention, the incident light is directly irradiated onto the high-Q resonant ring on the OPCA chip. The intensity information carried by the visible light modulates the detection light signal and is processed on the chip; by moving the resonant wavelength of the high-Q resonant ring, the encoded detection light signal can perform constructive or destructive interference, which is equivalent to addition and subtraction operations; due to the large optical bandwidth, different light spots in the sensor unit can be calculated in parallel through wavelength multiplexing; the calculation results can be transmitted to the far field in real time, and high-quality calculation reconstruction can be performed through a jointly designed multi-wavelength photonic processor.

[0075] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0076] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. An ultra-high-speed end-to-end all-optical intelligent computing chip system, characterized in that: include: a microlens array for focusing input spatial light onto the sensing unit; An all-optical parallel computing array comprises a plurality of the sensing units, wherein the sensing units are used to respond to the input spatial light and output all-optical computing results through multi-spectral all-optical computing based on the input multi-spectral detection light; A transmission module, used for transmitting the plenoptic calculation result to a plenoptic reconstruction module; The plenoptic reconstruction module is used to perform light field reconstruction based on the plenoptic calculation result to obtain a light field reconstruction result; Each sensing unit in the computing array is a high-Q all-pass resonant ring, the sensing units are physically connected through a bus waveguide, and above the unit array, a microlens array with matching period and numerical aperture is used to confine the input light field to the sensing area of ​​the resonant ring; During inference, the object is imaged onto the sensor array, denoted as x i,j (i=1, 2, ..., M; j=1, 2, ..., N), where M and N represent the number of rows and columns of the sensor array respectively; the light of each unit area is focused onto the resonant ring through the corresponding microlens, and the focused light is absorbed by the silicon waveguide to generate photogenerated carriers; based on the high Q value of the resonator, the refractive index change caused by the carriers is amplified, and the phase and / or amplitude modulation of the light propagating in the bus waveguide is performed, which is determined by x i,j The phase and / or amplitude change caused is denoted as h i,j , different resonators are biased to different resonant wavelengths; when multi-spectral detection light is input into the bus waveguide, a group of units respond to the same wavelength, and the output light through the output port of the bus waveguide is coherently combined. The operating wavelength in the multi-spectral detection light is represented by λ k (k=1, 2, ..., K), where K represents the number of operating wavelengths.

2. The system according to claim 1, characterized in that One of the sensing units is a resonant ring, and a plurality of the sensing units are physically connected via a bus waveguide.

3. The system according to claim 1, characterized in that The microlens array includes a microlens array with matching period and numerical aperture, and the microlens array is also used to limit the input spatial light to the sensing area of ​​the resonant ring.

4. The system according to claim 1, characterized in that The resonator of the resonant ring is used to perform phase and / or amplitude modulation on the light propagating in the bus waveguide.

5. The system according to claim 1, characterized in that The sensing unit is also used to respond to the same wavelength when the multi-spectral detection light is input into the bus waveguide, so as to output the coherently combined multi-spectral detection light through the output end of the sensing unit.

6. The system according to claim 1, characterized in that The full optical reconstruction module includes a full optical renderer, and the full optical renderer is used to: Constructing the all-optical parallel computing array and the all-optical renderer based on the blazed grating and the diffraction network; Reflecting multi-spectral detection light of different wavelengths to a preset spatial position through the blazed grating; The diffraction network is used to diffract the reflected multi-spectral detection light of different wavelengths through different phase masks at corresponding spatial positions through an all-optical renderer to reconstruct the light field.

7. The system according to claim 1, characterized in that The all-optical parallel computing array includes a plurality of all-optical parallel computing units connected via a bus.

8. The system according to claim 7, characterized in that The all-optical parallel computing unit is used to obtain a gradient operator based on calculation results of addition and subtraction on a plurality of the all-optical parallel computing units; wherein the gradient operator is used for image feature extraction of the input spatial light.

9. The system according to claim 8, characterized in that Also included is a pump-probe experimental setup for measuring the response time of an all-optical parallel computing unit.

10. A method for the ultra-high-speed end-to-end all-optical intelligent computing chip system according to any one of claims 1 to 9, characterized in that: include: The input spatial light is focused onto the sensing unit through a microlens array; Using the sensing unit of the all-optical parallel computing array to respond to the input spatial light, and outputting the all-optical computing result through multi-spectral all-optical computing based on the input multi-spectral detection light; Light field reconstruction is performed based on the plenoptic calculation result to obtain a light field reconstruction result.