Spatially diffractively reduced dimensionality assisted optical neuromorphic computing architecture

Abstract

The application discloses a kind of space diffraction dimensionality reduction assisted optical neurostat computing architecture, belong to optical computing field.Space diffraction dimensionality reduction assisted optical neurostat computing architecture includes input module, image replication device, linear multiplication device, linear cumulative device, nonlinear activation device, on-chip coupling device, on-chip neural network chip, wherein on-chip neural network chip includes on-chip linear multiplication device and on-chip nonlinear activation device.The specific implementation method is to encode the image information to be identified as a multi-wavelength optical matrix through the input module, the encoded matrix is subjected to dimensionality reduction operation and feature extraction through a plurality of image replication devices, linear multiplication devices, linear cumulative devices and nonlinear activation devices, and the dimensionality reduced matrix light information is coupled into the on-chip neural network chip through the on-chip coupling device to realize the related neural network function.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of optical computing, and more particularly, to a spatial diffraction dimension-reduction assisted optical neuromorphic computing architecture. BACKGROUND

[0002] With the rapid development of the Internet of Things and big data, how to efficiently store, transmit and process massive data has become a major challenge in the current information field. In the past decade, thanks to the significant improvement of GPU chip computing power and the development of deep learning algorithms, artificial intelligence technology (AI) based on artificial neural networks has flourished and made great achievements in image recognition, speech recognition and autonomous driving. However, it is worth noting that as the data dimension rises, high-performance deep learning algorithms require a large amount of storage and computing resources. Although in the past 50 years, semiconductor technology has gradually matured, from the initial 180nm process technology to the current 5nm advanced technology, and the 3nm technology being developed, the integrated circuit technology has developed rapidly, and the performance of the electrical computing system has improved rapidly, but due to the strict requirements of processing precision, this state cannot continue forever. As the size of the transistor gradually decreases, the power consumption and heat dissipation of the chip gradually become a problem, so the traditional electrical computing system cannot continue to meet the demand for high speed, low energy consumption and large-scale computing. In order to solve this problem, researchers have proposed high-performance optical computing systems, such as optical processing chips based on neuromorphic computing network architecture. Typically, a neuromorphic computing network is composed of about 10 11 Personal artificial neurons, each based on an adjustable optical switch to adjust the weight. For input samples with large matrix dimension, the neural network needs to perform matrix dimension reduction before training to reduce the training cost. The device that implements this operation in the traditional neural network is called an autoencoder.

[0003] In recent years, photonic AI acceleration chips have become a breakthrough point in the field of high-performance intelligent computing, thanks to the advantages of high speed, parallelism, low crosstalk, low energy consumption and high interconnection bandwidth of photons, as well as the rapid development of integrated optoelectronics. In AI acceleration chips, artificial neural networks composed of synapses and neurons are the hardware foundation for data computation and storage. Among them, the synapse array is the core hardware that enables large-scale parallel operation and storage-computing integration in AI acceleration chips. However, the number of optical neural synapses that can be inherited in current optical AI acceleration chips is far less than that of electrical neural networks, making it impossible to implement neural network training on high-dimensional data matrices. Therefore, some scholars have proposed an all-optical neural network encoder based on a spatial light modulator (SLM).

[0004] The input image of the system is first copied on N' input image spaces by a microlens array to prepare data for subsequent image processing and complete data dimensionality lifting; the microlens array is an array composed of lenses with micrometer-level light apertures and relief depths. Like traditional lenses, the minimum functional unit can also be a spherical mirror, an aspherical mirror, a cylindrical lens, a prism, etc., and can also realize focusing, imaging, beam transformation and other functions in the field of micro-optics. Because of the small size and high integration of the unit, it can form many new optical systems and complete functions that traditional optical elements cannot complete. The structure of the microlens array can be divided into single-row type, M*N arrangement type and full type according to the arrangement method of the minimum functional unit, and can also be divided into single-surface array and double-surface array. The microlens array can be divided into two types: refractive microlens array and diffractive microlens array. Taking the diffractive microlens array as an example, based on the diffraction principle of physical optics, the wavefront phase of light is modulated and changed by the surface relief structure of the lens array, thereby realizing the modulation and transformation of light waves. After the laser passes through each diffractive unit, diffraction occurs, and interference occurs at a certain distance (usually infinite or the focal plane of the lens), forming a specific light intensity distribution and completing the replication of the image. The SLM forms a weight matrix, and the N' copy images are projected to different regions of the SLM, covering all the pixels on the SLM, and the transmittance of each pixel of the SLM is modulated according to the element size of the weight matrix, and the matrix point multiplication algorithm is realized by proportionally attenuating the input image. Finally, the light components after operation are focused by using a lens to realize the summation of each output vector element, the image output by the SLM is reduced by using the focusing effect of the lens, and the first data dimensionality reduction process is completed. Finally, the light-light nonlinear activation layer (OONA) operation of each element of the output vector is realized by using an image intensifier.

[0005] However, only relying on this spatial light dimensionality reduction data mode cannot realize the function of a complex neural network. For a high-dimensional data matrix, the feature matrix after the dimensionality reduction process also needs to be subjected to convolution, full connection and other operations to realize corresponding neural network recognition, detection and other functions. SUMMARY

[0006] In view of the defects of the prior art, the purpose of the present application is to provide a spatial diffraction dimensionality reduction assisted optical neural pseudo-state computing architecture, which aims to provide an effective solution for high-dimensional data all-optical computing acceleration.

[0007] To achieve the above object, the application provides a spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, which comprises an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens and an on-chip neural network chip. The on-chip neural network chip comprises an on-chip linear multiplication device for linear weight configuration and an on-chip nonlinear activation device with a nonlinear function function. The specific implementation method is that the input image information output by the input module encodes the image information to be recognized into a multi-wavelength optical matrix, and the encoded multi-wavelength optical matrix is copied into MxN matrixes with smaller size by the image copying device, wherein M and N are integers. The copied image copies are subjected to weight adjustment by the linear multiplication device, and then the linear mapping results of different copied image copies are subjected to full connection addition operation in the linear accumulation device; the results of linear operation are subjected to nonlinear mapping of data by the nonlinear activation device, so as to realize dimension reduction processing of input information once, and finally realize dimension reduction operation and feature extraction of high-dimensional data matrix through several same operation steps. The dimension reduction matrix is coupled into the on-chip neural network chip by the focusing lens, and the convolution matrix and the full connection matrix on the on-chip neural network chip further process the dimension reduction matrix to realize the function of the related neural network. For the traditional ridge waveguide, the change of mode energy distribution caused by the phase change of the phase change film is not obvious, and the non-volatile multi-level adjustable photonic neural synapse device based on the phase change material proposed in the application can realize significant change of mode energy distribution.

[0008] Further, the input image information requires that the initial high-dimensional data matrix can be wave division multiplexing encoded, thereby improving the parallel processing capability of the neural network. The specific method is that a broadband light source of a specific working waveband is selected, a spatial light path is adjusted to irradiate on a related optical intensity\phase encoder (SLM, DMD, etc.), thereby obtaining a multi-wavelength matrix carrying the same intensity information; a spatial filter array is used to perform specific filtering processing on each matrix element, and only the single wavelength information required for calculation is retained. Adjusting the output of the optical intensity\phase encoder can realize wave division multiplexing encoding. Compared with the traditional continuous waveguide structure, the non-volatile multi-level adjustable photonic neural synapse device based on the phase change material has a greater output intensity difference when the phase change material is in different phase states.

[0009] Further, the image copying device requires that the multi-wavelength high-dimensional matrix information can be compressed and copied, and the compression ratio and the number of copies are the results optimized by offline training of the neural network. By optimizing the imaging quality of the image copying device in the Fourier plane, the spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture can retain the output features to the greatest extent after image compression.

[0010] Further, the linear multiplication device needs to meet the intensity modulation law under the spatial diffraction theory. According to the Huygens principle, the light field response of each pixel is defined as a secondary wave source to emit approximate spherical light waves, which are radiated to each unit of the linear multiplication device in the next layer and form neural connections. By adjusting the fixed amplitude and phase response of the previous diffraction unit, the response is superimposed on the neural connection, thereby realizing the adjustment of the intensity of the neural connection.

[0011] Further, the on-chip neural network chip adopts a parallel network architecture based on cross waveguides. The architecture requires multiple channel inputs, which can fully utilize the wavelength dimension of light, use different wavelength inputs, ensure incoherence between different channels, and have great expansibility to design larger network architectures. The optical encoding information in different channels is coupled with a specific optical neural synapse after passing through the coupling region, and the output intensity is related to the weight coefficient of the optical neural synapse.

[0012] The spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, the image duplicating device can duplicate the multi-wavelength optical matrix into MxN (M can be equal to N) low-dimensional matrices.

[0013] The spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, the linear multiplication device can be a spatial light modulator (SLM) with dynamically configured weights, a digital micromirror device (DMD), or a fixed weight optical plane such as a multi-phase plane and a metasurface.

[0014] The spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, the linear accumulation device can accumulate the linear operation results of different duplicated images, and as a preferred embodiment, the application adopts a focusing lens.

[0015] The spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, the nonlinear activation device is an all-optical nonlinear activation device, which further ensures the processing speed of the data dimension reduction process, and as a preferred embodiment, the application adopts an image intensifier.

[0016] The spatial diffraction dimension reduction assisted optical neural quasi-state computing architecture, the input module, image duplicating device, linear multiplication device, linear accumulation device, and nonlinear activation device can extract feature information of the input signal, and the feature information dimension is the same as the feature signal of the input dimension of the on-chip neural network chip. The feature information is coupled into the on-chip neural network chip through the input coupling device, which can be end face coupling or vertical coupling, or special arranged array optical fiber can be used to couple the optical signal into the on-chip neural network chip through end face coupling or vertical coupling.

[0017] The on-chip linear multiplication device and the on-chip nonlinear activation device included in the on-chip neural network chip can be reconfigurable coding, which can be a thermal optical device based on a traditional heating method, or an optical control device based on a phase change material, a ferroelectric material, a magneto-optical material, and the like, and as a preferred embodiment, the present application adopts a non-volatile device based on a phase change material.

[0018] The on-chip linear multiplication device has an insertion loss of less than 3 dB, and an intensity modulation range of greater than 3 dB in a wavelength range corresponding to multi-wavelength matrix information.

[0019] The on-chip nonlinear activation device can be an optical nonlinear activation device or an electrical nonlinear activation device, but at least three nonlinear activation function functions need to be realized.

[0020] The input channel of the cross-mth row of the on-chip neural network chip, when the input signal X n After passing through a coupler with a coupling intensity coefficient k1, the signal intensity entering the first cross waveguide is X n ×(1-k1).

[0021] The on-chip neural network chip 7 is processed on a silicon platform, a silicon nitride platform, an indium phosphide platform, or a lithium niobate platform.

[0022] Compared with the prior art, the spatial diffraction dimensionality reduction assisted optical neural quasi-state computing architecture can complete all optical domain processing from image acquisition to processing result output, so that high-resolution optical image processing tasks can be completed in a very short time delay. According to the measurement, the time delay of the architecture for a >=1K*1K image is expected to be within 1ms. In comparison, the time delay of a traditional electrical image processing system for completing the same scale image processing task is tens of ms. In addition, the present application can fully exploit the wideband and low delay advantages of the all-optical architecture to realize ultra-low delay real-time intelligent computing of image data, greatly improving the operation speed, and the reconfigurable parallel processing capability is suitable for various complex application requirements. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a schematic diagram of an optical signal computing architecture of the spatial diffraction dimensionality reduction assisted optical neural quasi-state computing architecture.

[0024] Figure 2is the input image information encoding schematic diagram of the spatial diffraction dimension reduction assisted optical neuromorphic computing architecture.

[0025] Figure 3 is the on-chip neural network 4-input-4-output architecture schematic diagram (a) and data multi-channel convolution process inference diagram (b) of the spatial diffraction dimension reduction assisted optical neuromorphic computing architecture.

[0026] Figure 4 Reconfigurable neural synapse scheme schematic diagram in the on-chip neural network of the spatial diffraction dimension reduction assisted optical neuromorphic computing architecture. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0028] The present application provides a spatial diffraction dimension reduction assisted optical neuromorphic computing architecture, comprising an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens, and an on-chip neural network chip. The on-chip neural network chip comprises an on-chip linear multiplication device for linear weight configuration and an on-chip nonlinear activation device with nonlinear function function. The specific implementation method is to encode the image information to be recognized into a multi-wavelength optical matrix by the input image information output by the input module, and the encoded multi-wavelength optical matrix is copied into MxN matrix size smaller optical matrix after the image copying device, wherein M and N are integers, the linear multiplication device is used for linear modulation of the low-dimensional matrix to obtain linear operation result, the linear operation result is accumulated by the linear accumulation device, the nonlinear activation device realizes the nonlinear mapping result of the matrix, and the dimension reduction matrix light information is obtained, the dimension reduction matrix light information is coupled into the on-chip neural network chip through the on-chip coupling device to realize the neural network recognition and classification function.

[0029] EMBODIMENT

[0030] As Figure 1As shown, the high-dimensional data matrix full-optical computing system of the embodiment includes an input module, an image duplicating device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens, and an on-chip neural network chip. The on-chip neural network chip includes an on-chip linear multiplication device and an on-chip nonlinear activation device. The specific implementation method is to encode the image information to be recognized into a multi-wavelength optical matrix through input image information. The encoded matrix creates N copies on the input image space through the image duplicating device. The copied image copies are subjected to weight adjustment through the linear multiplication device, and then the linear mapping results of different copied image copies are subjected to full-connection addition operation in the linear accumulation device; the results of linear operation are subjected to nonlinear mapping of data through the nonlinear activation device. After several identical operation steps, the dimension reduction operation and feature extraction of the high-dimensional data matrix are realized. The reduced dimension matrix is coupled into the on-chip neural network chip through the focusing lens, and the convolution matrix and the full-connection matrix on the on-chip neural network chip further process the reduced dimension matrix to realize the related neural network function.

[0031] For the high-dimensional data matrix full-optical computing system in the embodiment, the optical wavelength parallel dimension is used to improve the computing efficiency. Therefore, the input matrix needs to be encoded in multiple wavelengths. The full-optical input encoder to be used in the embodiment is shown in Figure 2 The specific method is to select a broadband light source of a specific working wavelength, adjust the spatial light path to irradiate on a spatial light modulator, thereby obtaining a multi-wavelength matrix carrying the same intensity information; use a spatial filter array to perform specific filtering processing on each matrix element, and only retain the single wavelength information required for calculation. The output of the optical intensity\phase encoder can realize wavelength division multiplexing encoding.

[0032] Further, the encoded multi-wavelength input matrix is still a high-dimensional matrix information, which cannot be directly used as the input matrix of the neural network and subjected to linear and nonlinear processing. In the training of the traditional neural network, the data needs to be encoded. The purpose of pre-encoding is to compress the input data, extract the feature information in the data, and reduce the cost of subsequent neural network training. For the high-dimensional data matrix full-optical computing system in the embodiment, the data dimension reduction operation is performed in an optical manner, specifically including an image duplicating device, a linear multiplication device, a linear accumulation device, and a nonlinear activation device. A full-optical encoder is designed. The full-optical encoder to be used in the embodiment is a multi-phase surface architecture (or a spatial light modulator architecture). The input high-dimensional data matrix is mapped into a target vector after passing through multiple encoders, and the target vector carries the feature information of the input data.

[0033] Further, the compressed target vector of the high-dimensional data matrix needs to be input to the on-chip neural network for related convolution / fully connected operation, so as to realize the neural network functions such as decision, recognition or detection. In the embodiment, the target vector dimension is the same as the number of input ports of the on-chip neural network, which is coupled into the on-chip neural network through the focusing lens for convolution / fully connected operation. In the embodiment, the on-chip neural network convolution principle is introduced by taking a 4-input-4-output architecture as an example, as shown in Figure 3 . The input vector dimension is 1x4, wherein the intensity information of each element X m is represented by the light intensity amplitude of different wavelengths l m , so as to realize parallel data processing of four wavelength information. The input light of channel n enters the reconfigurable neural synapse through the coupler with a coupling intensity coefficient k1, and since the reconfigurable neural synapse has a specific loss coefficient a nm , the output intensity of the signal with wavelength l n after passing through the reconfigurable neural synapse is X n x k1 x a nm . When the output signal of the reconfigurable neural synapse again passes through the coupling region with a coupling intensity coefficient k2, it will be coupled into the cross waveguide array again, and the intensity is X n x k1 x k2 x a nm . The input signals of different channels pass through the corresponding reconfigurable neural synapses and are output at the output ports of the cross waveguide array, wherein the intensity information of the mthcolumn output port represents the operation result between the input vector (X1, X2, X3, X4) T and the convolution kernel n. The specific data operation process is shown in (b) of Figure 3 .

[0034] Further, the loss coefficient a nm of the reconfigurable neural synapse is related to the material loss coefficient, and the embodiment realizes the all-optical reconfigurable neural synapse by heating the phase change material through external pulse light signals, as shown in Figure 4 . The pulse light signal is input into the reconfigurable neural synapse waveguide channel in the form of grating coupling. It should be noted that the wavelength of the pulse light signal and the encoding wavelength of the input vector should be different, so as to ensure the incoherence between the pulse light signal and the calculation information in the calculation process. The light signal X n in the input channel n is partially coupled into the reconfigurable neural synapse when passing through the coupling region L DC1 , and the multiplication operation of different weight coefficients is completed by changing the crystallization state of the phase change material. After the multiplication operation, the light signal passes through the coupling region L DC2After coupling into the waveguide array again, the signal light is finally accumulated and output in other channels. Since the convolution operation needs to be completed in the waveguides of m rows and different columns, we need to realize uniform splitting in the m-column waveguide array, the core of which is to control the different coupling lengths L DC1 The control of the splitting ratio is realized, and the splitting ratio of the ith column is 1 / (m+1-i). The light is split from the main waveguide to the reconfigurable neural synapses to perform the multiplication operation of the weight coefficient, and each reconfigurable neural synapse weight can also be reset to switch different state numbers.

[0035] After the data on-chip convolution operation, the high-dimensional input matrix is finally compressed into a 1xM vector (the size of M is equal to the number of output ports of the neural network chip). This feature vector (spot array intensity) can realize data judgment, classification, identification and other operations.

[0036] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present application, and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A spatially diffractively reduced dimensionality assisted optical neuromorphic computing architecture, characterized in that, The input module (1), the image copying device (2), the linear multiplication device (3), the linear accumulation device (4), the nonlinear activation device (5), the focusing lens (6), and the on-chip neural network chip (7) are included, wherein the on-chip neural network chip (7) includes an on-chip linear multiplication device (7-1) for linear weight configuration and an on-chip nonlinear activation device (7-2) with a nonlinear function function; the input module (1) is used to encode image information to be identified into a multi-wavelength optical matrix, and the encoded multi-wavelength optical matrix is copied into M×N matrix size smaller optical matrix after the image copying device (2), wherein M and N are integers, the linear multiplication device (3) is used to linearly modulate the low-dimensional matrix to obtain a linear operation result, the linear operation result is accumulated through the linear accumulation device (4), the nonlinear activation device (5) realizes the nonlinear mapping result of the matrix, and a reduced dimension matrix light information is obtained, and the reduced dimension matrix light information is coupled into the on-chip neural network chip (7) through the on-chip coupling device (6) to realize the neural network recognition and classification function.

2. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The optical information carried by the input module (1) is optical information carrying special intensity, phase, polarization distribution or optical information encoded by an electro-optical encoder.

3. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The image copying device (2) is a metasurface or a microlens array.

4. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The linear multiplication device (3) is a spatial light modulator SLM, a digital micromirror device DMD, a multi-phase surface or a metasurface.

5. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The linear accumulation device (4) is a focusing lens.

6. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The coupling mode is end face coupling or vertical coupling.

7. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 1, wherein, The on-chip linear multiplication device (7-1) and the on-chip nonlinear activation device (7-2) are used for reconfigurable coding, and the on-chip linear multiplication device (7-1) and the on-chip nonlinear activation device (7-2) are thermal optical devices or optical control devices based on phase change materials, ferroelectric materials and magneto-optical materials.

8. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 7, wherein, The on-chip linear multiplication device (7-1) has multi-bit modulation capability in the wavelength range corresponding to the multi-wavelength matrix information.

9. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 7, wherein, The on-chip nonlinear activation device (7-2) is an optical nonlinear activation device or an electrical nonlinear activation device, which has the pulse response characteristics of a bionic neural synapse.

10. The spatial-diffraction dimensionality-reduced auxiliary-optical neuromorphic computing architecture of claim 7, wherein, The on-chip neural network chip (7) is processed on a silicon platform, a silicon nitride platform, an indium phosphide platform or a lithium niobate thin film platform.

Images