Optical neural mimicry computing architecture assisted by space diffraction dimension reduction

Abstract

The invention discloses an optical neural mimicry computing architecture assisted by space diffraction dimension reduction, and belongs to the field of optical computing. The spatial diffraction dimensionality reduction assisted optical neural mimicry computing architecture comprises an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, an on-chip coupling device and an on-chip neural network chip, wherein the on-chip neural network chip comprises an on-chip linear multiplication device and an on-chip nonlinear activation device. According to the specific implementation method, image information needing to be recognized is coded into a multi-wavelength optical matrix through an input module, and dimension reduction operation and feature extraction of the coded matrix are achieved through a plurality of image copying devices, a linear multiplication device, a linear accumulation device and a nonlinear activation device. And the dimension reduction matrix optical information is coupled into the on-chip neural network chip through the on-chip coupling device to realize related neural network functions.

Description

Technical Field

[0001] The present invention belongs to the field of optical computing, and more specifically, to an optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction. Background Art

[0002] With the rapid development of the Internet of Things and big data, how to efficiently store, transmit and process massive amounts of data has become a major challenge in the current information field. In the past decade, thanks to the substantial increase in GPU chip computing power and the development of deep learning algorithms, artificial intelligence technology (AI) based on artificial neural networks has flourished and has shined in areas such as image recognition, speech recognition and autonomous driving. However, it is worth noting that with the increase in data dimensions, high-performance deep learning algorithms require a large amount of storage and computing resources. Although semiconductor technology has gradually matured in the past 50 years, from the initial 180nm process technology to the current 5nm advanced process, and the 3nm process currently under development, the integrated circuit process has developed rapidly and the performance of electrical computing systems has increased rapidly, but this state cannot continue to develop due to the strict requirements of processing accuracy. As the size of transistors gradually decreases, problems such as chip power consumption and heat dissipation gradually emerge. Therefore, traditional electrical computing systems cannot continue to meet the needs of high-speed, low-energy, and large-scale computing. To solve this problem, researchers have proposed high-performance optical computing systems, such as optical processing chips based on neuromorphic computing network architectures. Typically, a neuromorphic computing network consists of approximately 10 11 The neural network consists of artificial neurons, each of which adjusts its weight based on a dimmable light switch. For input samples with large matrix dimensions, the neural network needs to perform matrix dimensionality reduction before training to reduce the training cost. The device that implements this operation is called an autoencoder in traditional neural networks.

[0003] In recent years, photonic AI acceleration chips have become a breakthrough in the field of high-performance intelligent computing due to the advantages of high speed, parallelism, low crosstalk, low energy consumption and high interconnection bandwidth of photons, as well as the rapid development of integrated optoelectronics. In AI acceleration chips, artificial neural networks composed of synapses and neurons are the hardware basis for data calculation and storage. Among them, the synaptic array is the core hardware that enables AI acceleration chips to achieve large-scale parallel computing and storage. However, the number of optical neural synapses that can be inherited in optical AI acceleration chips is far less than that of electrical neural networks, and neural network training for high-dimensional data matrices cannot be achieved. Therefore, some scholars have proposed an all-optical neural network encoder based on a spatial light modulator (SLM).

[0004] The input image of the system first creates N' copies of the input image space through a microlens array to prepare data for subsequent image processing and complete data dimensionality upgrade; the microlens array is an array composed of lenses with a clear aperture and relief depth of micrometers. Like traditional lenses, its smallest functional unit can also be a spherical mirror, an aspherical mirror, a cylindrical mirror, a prism, etc. It can also achieve focusing, imaging, beam transformation and other functions at a micro-optical angle. Moreover, due to its small unit size and high integration, it can form many new optical systems and complete functions that traditional optical elements cannot complete. The structure of the microlens array can be divided into single-row, M*N arrangement, full-coverage, etc. according to the arrangement method of the smallest functional unit, and can also be divided into single-sided array and double-sided array. The microlens array can be divided into two categories: refractive microlens array and diffractive microlens array. Taking the diffractive microlens array as an example, based on the diffraction principle of physical optics, the light is modulated by the surface relief structure of the lens array to change the wavefront phase, thereby realizing the modulation and transformation of the light wave. The laser diffracts after passing through each diffraction unit and generates interference at a certain distance (usually infinity or the focal plane of the lens), forming a specific light intensity distribution and completing the image replication. The SLM forms a weight matrix, and N' copies of the image are projected to different areas of the SLM, covering all pixels on the SLM. The transmittance of each pixel of the SLM is modulated according to the size of the elements of the weight matrix, and the matrix dot multiplication algorithm is implemented by proportionally attenuating the input image. Finally, the lens is used to focus the light components that have been calculated to realize the summation of each output vector element, and the focusing effect of the lens is used to reduce the image output by the SLM, and the first data dimensionality reduction processing is completed by summing. Finally, the image intensifier is used to realize the optical-optical nonlinear activation layer (OONA) operation of each element of the output vector.

[0005] However, relying solely on this spatial optical dimensionality reduction method cannot achieve complex neural network functions. For high-dimensional data matrices, the feature matrix after the data dimensionality reduction process still needs to undergo convolution, full connection and other operations to achieve corresponding neural network recognition, detection and other functions. Summary of the invention

[0006] In view of the defects of the prior art, the purpose of the present invention is to provide an optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, aiming to provide an effective solution for all-optical computing acceleration of high-dimensional data.

[0007] To achieve the above-mentioned purpose, the present invention provides an optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, including an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens, and an on-chip neural network chip. The on-chip neural network chip includes an on-chip linear multiplication device for linear weight configuration and an on-chip nonlinear activation device with a nonlinear function function. The specific implementation method is to encode the image information to be identified into a multi-wavelength optical matrix through the input image information output by the input module, and the encoded multi-wavelength optical matrix is ​​copied into M×N optical matrices with smaller matrix sizes after passing through the image copying device, where M and N are integers. The copied image copies are weighted by a linear multiplication device, and then the linear mapping results of different copied image copies are fully connected in the linear accumulation device. The result of the linear operation is passed through a nonlinear activation device to realize nonlinear mapping of the data, so as to realize a dimensionality reduction process of the input information, and after several identical operation steps, the dimensionality reduction operation and feature extraction of the high-dimensional data matrix are finally realized. The reduced-dimensional matrix is ​​coupled into the on-chip neural network chip through a focusing lens, and the convolution matrix and the fully connected matrix on the on-chip neural network chip further process the reduced-dimensional matrix to realize the relevant neural network functions. For traditional ridge waveguides, the change in mode energy distribution caused by the phase change of the phase change film is not obvious. The non-volatile multi-level adjustable photonic neural synapse device based on phase change material proposed in the present invention can achieve significant changes in mode energy distribution.

[0008] Furthermore, the input image information requires that the initial high-dimensional data matrix can be encoded by wavelength division multiplexing, thereby improving the parallel processing capability of the neural network. The specific method is to select a broadband light source of a specific working band, adjust the spatial light path to illuminate the relevant optical intensity\phase encoding device (SLM, DMD, etc.), thereby obtaining a multi-wavelength matrix carrying the same intensity information; use the spatial filter array to perform specific filtering on each matrix element, and only retain the single wavelength information required for the calculation. Adjusting the output of the optical intensity\phase encoding device can realize wavelength division multiplexing encoding. Compared with the traditional continuous waveguide structure, the non-volatile multi-level adjustable photonic neural synapse device based on phase change material proposed in the present invention has a larger output intensity difference when the phase change material is in different phases.

[0009] Furthermore, the image replicator is required to be able to compress and replicate multi-wavelength high-dimensional matrix information. The compression ratio and the number of copies are the results of offline neural network training optimization. By optimizing the imaging quality of the image replicator in the Fourier plane, the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction can retain the output features to the greatest extent after image compression.

[0010] Furthermore, the linear multiplication device needs to comply with the intensity modulation law under the spatial diffraction theory. Specifically, according to the Huygens principle, the light field response of each pixel is defined as a secondary wave source emitting an approximate spherical light wave, which radiates to each unit in the next layer of the linear multiplication device and forms a neural connection. By adjusting the fixed amplitude and phase response of the previous diffraction unit, the response is superimposed on the neural connection, thereby adjusting the strength of the neural connection.

[0011] Furthermore, the on-chip neural network chip adopts a parallel network architecture based on cross-waveguides. The architecture requires multi-channel input, which can make full use of the wavelength dimension of light and use different wavelengths of input, which not only ensures the incoherence between different channels, but also has great scalability to design a larger network architecture. The optically encoded information in different channels is coupled with specific optical synapses after passing through the coupling area, and the output intensity is related to the weight coefficient of the optical synapse.

[0012] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the image replication device can replicate the multi-wavelength optical matrix into M×N (M can be equal to N) low-dimensional matrices.

[0013] In the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the linear multiplication device can be a spatial light modulator (SLM) or a digital micromirror device (DMD) with dynamically configured weights, or an optical plane with fixed weights, such as a multi-phase surface and a metasurface.

[0014] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the linear accumulation device can perform accumulation operations on the linear calculation results of different replicated images. Preferably, the present invention adopts a focusing lens.

[0015] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the nonlinear activation device is an all-optical nonlinear activation device, thereby ensuring the processing speed of the data dimensionality reduction process. Preferably, the present invention adopts an image enhancer.

[0016] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the input module, image copying device, linear multiplication device, linear accumulation device, and nonlinear activation device can extract characteristic information of the input signal, the characteristic information dimension is a characteristic signal with the same input dimension as the on-chip neural network chip, and the characteristic information is coupled into the on-chip neural network chip through an input coupling device, specifically, it can be directly coupled into the on-chip neural network chip by end face coupling or vertical coupling, and the optical signal can also be coupled into the on-chip neural network chip by end face coupling or vertical coupling using a specially arranged array optical fiber.

[0017] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the on-chip linear multiplication device and the on-chip nonlinear activation device included in the on-chip neural network chip can both be reconfigured for encoding, specifically, they can be thermally adjustable optical devices based on traditional heating methods, or they can be optical control devices based on phase change materials, ferroelectric materials, magneto-optical materials, etc. Preferably, the present invention adopts non-volatile devices based on phase change materials.

[0018] The spatial diffraction dimensionality reduction-assisted optical neuromorphic computing architecture has an on-chip linear multiplication device with an insertion loss of less than 3 dB and an intensity modulation range of greater than 3 dB within the wavelength range corresponding to the multi-wavelength matrix information.

[0019] In the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the on-chip nonlinear activation device can be an optical nonlinear activation device or an electrical nonlinear activation device, but it is necessary to implement at least three nonlinear activation function functions.

[0020] The spatial diffraction dimension reduction-assisted optical neuromorphic computing architecture, the cross-mth row input channel of the on-chip neural network chip, when the input signal X n After passing through the coupler with a coupling strength coefficient of k1, the signal strength entering the first cross waveguide is X n ×(1-k1).

[0021] The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, the on-chip neural network chip 7 is processed on a silicon platform, a silicon nitride platform, an indium phosphide platform or a lithium niobate platform.

[0022] Through the above technical solutions conceived by the present invention, compared with the prior art, the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction proposed by the present invention can achieve the completion of everything from image acquisition to processing result output in the optical domain. Therefore, high-resolution optical image processing tasks can be completed within a very short delay. According to calculations, the latency of this architecture ≥1K×1K images is expected to be less than 1ms. In comparison, the latency of traditional electrical image processing systems to complete image processing tasks of the same scale is tens of ms. In addition, the present invention can fully explore the broadband and low-latency advantages of the all-optical architecture, realize ultra-low latency real-time intelligent computing of image data, greatly improve the computing speed, and its reconfigurable parallel processing capabilities are suitable for a variety of complex application requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the optical signal computing architecture of the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction.

[0024] Figure 2It is a schematic diagram of the input image information encoding of the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction.

[0025] Figure 3 This is a schematic diagram of the on-chip neural network 4-input-4-output architecture of the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction (a) and the data multi-channel convolution process inference diagram (b).

[0026] Figure 4 Schematic diagram of the reconfigurable synapse scheme in the on-chip neural network of the optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0028] The present invention provides an optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, including an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens, and an on-chip neural network chip. The on-chip neural network chip includes an on-chip linear multiplication device for linear weight configuration and an on-chip nonlinear activation device with a nonlinear function function. The specific implementation method is to encode the image information to be identified into a multi-wavelength optical matrix through the input image information output by the input module, and the encoded multi-wavelength optical matrix is ​​copied into M×N optical matrices with a smaller matrix size after passing through the image copying device, wherein M and N are integers, and the linear multiplication device is used to linearly modulate the low-dimensional matrix to obtain a linear operation result, and the linear operation result is accumulated through the linear accumulation device, and the nonlinear activation device realizes the nonlinear mapping result of the matrix to obtain the reduced-dimensional matrix light information, and the reduced-dimensional matrix light information is coupled into the on-chip neural network chip through the on-chip coupling device to realize the neural network recognition and classification function.

[0029] Example

[0030] like Figure 1As shown, the high-dimensional data matrix all-optical computing system of this embodiment includes an input module, an image copying device, a linear multiplication device, a linear accumulation device, a nonlinear activation device, a focusing lens, and an on-chip neural network chip. The on-chip neural network chip includes an on-chip linear multiplication device and an on-chip nonlinear activation device. The specific implementation method is to encode the image information to be identified into a multi-wavelength optical matrix by inputting image information, and the encoded matrix is ​​passed through the image copying device to create N copies in the input image space. The copied image copies are weighted by the linear multiplication device, and then the linear mapping results of different copied image copies are fully connected in the linear accumulation device. The results of the linear operation are passed through the nonlinear activation device to realize the nonlinear mapping of the data. After several identical operation steps, the dimensionality reduction operation and feature extraction of the high-dimensional data matrix are realized. The dimensionality reduction matrix is ​​coupled into the on-chip neural network chip through the focusing lens, and the convolution matrix and the fully connected matrix on the on-chip neural network chip further process the dimensionality reduction matrix to realize the relevant neural network function.

[0031] For the high-dimensional data matrix all-optical computing system in this embodiment, it is intended to use the parallel dimension of optical wavelengths to improve computing efficiency. Therefore, the input matrix needs to be encoded with multiple wavelengths. The all-optical input encoder to be used in this embodiment is as follows: Figure 2 The specific method is to select a broadband light source with a specific working band, adjust the spatial light path to illuminate the spatial light modulator, and thus obtain a multi-wavelength matrix carrying the same intensity information; use the spatial filter array to perform specific filtering on each matrix element, and only retain the single wavelength information required for calculation, and adjust the output of the optical intensity\phase encoding device to achieve wavelength division multiplexing encoding.

[0032] Furthermore, the encoded multi-wavelength input matrix is ​​still high-dimensional matrix information, and it is impossible to directly use it as the input matrix of the neural network and perform linear and nonlinear processing. In traditional neural network training, this kind of data needs to be encoded. The purpose of precoding is to compress the input data, extract the characteristic information in the data, and reduce the cost of subsequent neural network training. For the high-dimensional data matrix all-optical computing system in this embodiment, it is planned to use optical methods to perform data dimensionality reduction operations, specifically including image copying devices, linear multiplication devices, linear accumulation devices, and nonlinear activation devices. Design an all-optical encoder. The all-optical encoder to be used in this embodiment is a multi-phase plane architecture (or spatial light modulator architecture). The input high-dimensional data matrix is ​​mapped to a target vector after passing through multiple encoders, and the target vector carries the characteristic information of the input data.

[0033] Furthermore, the target vector after compression of the high-dimensional data matrix needs to be input into the on-chip neural network for related convolution / full connection operations, thereby realizing neural network functions such as judgment, recognition or detection. In this embodiment, the dimension of the target vector is the same as the number of input ports of the on-chip neural network, and is coupled into the on-chip neural network through the focusing lens for convolution / full connection operations. In this embodiment, Figure 3 As shown in the figure, the 4-input-4-output architecture is used as an example to introduce the principle of on-chip neural network convolution. The input vector dimension is 1×4, where each element X m The intensity information is obtained by different wavelengths l m The input light of channel n enters the reconfigurable neural synapse through a coupler with a coupling strength coefficient of k1. Since the reconfigurable neural synapse has a specific loss coefficient a nm , so the wavelength is l n The output strength of the signal after passing through the reconfigurable neural synapse is X n ×k1×a nm When the output signal of the reconfigurable neural synapse passes through the coupling region with a coupling strength coefficient of k2 again, it will be coupled into the cross waveguide array again with an intensity of X n ×k1×k2×a nm The input signals of different channels are output at the output ports of the cross-waveguide array after passing through the corresponding reconfigurable neural synapses, where the intensity information of the output port in the mth column represents the input vector (X1, X2, X3, X4) T The operation result between and convolution kernel n. The specific data operation process is as follows Figure 3 As shown in (b) in .

[0034] Further, the loss coefficient a of the reconstructable synapse nm Related to the material loss coefficient, this embodiment realizes all-optical reconfigurable neural synapses by heating the phase change material through an external pulse light signal. Figure 4 As shown in Figure 2, the pulsed light signal is input into the reconfigurable synaptic waveguide channel by grating coupling. It should be noted that the wavelength of the pulsed light signal must be different from the encoding wavelength used by the input vector, thereby ensuring the incoherence between the pulsed light signal and the calculation information during the calculation process. The optical signal X in the input channel n n After the coupling region L DC1 When part of the light intensity is coupled into the reconfigurable neural synapse, the multiplication operation of different weight coefficients is completed by changing the crystal state of the phase change material. After the multiplication operation, the optical signal passes through the coupling area L DC2After that, it is coupled into the waveguide array again, and finally outputted by accumulating the signal light of other channels. Since the waveguides in m rows and different columns need to complete the convolution operation, we need to achieve uniform light splitting in the total m-column waveguide array. The core is to control the different coupling lengths L. DC1 The splitting ratio is controlled, and the splitting ratio of the i-th column is 1 / (m+1-i). The light is split from the main waveguide and coupled to the reconfigurable synapse for multiplication of the weight coefficient. The weight of each reconfigurable synapse can also be reset to switch between different states.

[0035] After the on-chip convolution operation, the high-dimensional input matrix is ​​finally compressed into a 1×M vector (the size of M is equal to the number of output ports of the neural network chip). This feature vector (spot array intensity) can realize the judgment, classification, and recognition of the data to be tested.

[0036] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. Optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction, characterized in that: The invention comprises an input module (1), an image copying device (2), a linear multiplication device (3), a linear accumulation device (4), a nonlinear activation device (5), a focusing lens (6), and an on-chip neural network chip (7), wherein the on-chip neural network chip (7) comprises an on-chip linear multiplication device (7-1) for linear weight configuration and an on-chip nonlinear activation device (7-2) having a nonlinear function function; the input module (1) is used to encode image information to be identified into a multi-wavelength optical matrix, and the encoded multi-wavelength optical matrix is ​​copied into M×N optical matrices with a smaller matrix size after passing through the image copying device (2), wherein M and N are integers; the linear multiplication device (3) is used to linearly modulate the low-dimensional matrix to obtain a linear operation result, and the linear operation result is accumulated through the linear accumulation device (4); the nonlinear activation device (5) realizes a nonlinear mapping result of the matrix to obtain reduced-dimensional matrix optical information, and the reduced-dimensional matrix optical information is coupled into the on-chip neural network chip (7) through the on-chip coupling device (6) to realize the neural network identification and classification function.

2. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The optical information carried by the input module (1) is optical information carrying special intensity, phase, and polarization distribution, or optical information encoded by an electro-optical encoding device.

3. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The image replication device (2) is a metasurface or a microlens array.

4. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The linear multiplication device (3) is a spatial light modulator SLM, a digital micromirror device DMD, a multi-phase surface or a metasurface.

5. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The linear accumulation device (4) is a focusing lens.

6. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The coupling mode is end face coupling or vertical coupling.

7. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 1, characterized in that: The on-chip linear multiplication device (7-1) and the on-chip nonlinear activation device (7-2) are both used for reconfigurable coding. The on-chip linear multiplication device (7-1) and the on-chip nonlinear activation device (7-2) are thermally adjustable optical devices, or optical control devices based on phase change materials, ferroelectric materials, or magneto-optical materials.

8. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 7, characterized in that: The on-chip linear multiplication device (7-1) has a multi-bit modulation capability within the wavelength range corresponding to the multi-wavelength matrix information.

9. The spatial diffraction dimension reduction-assisted optical neuromorphic computing architecture according to claim 7, characterized in that: The on-chip nonlinear activation device (7-2) is an optical nonlinear activation device or an electrical nonlinear activation device, and has the impulse response characteristics of a bionic neural synapse.

10. The optical neuromorphic computing architecture assisted by spatial diffraction dimensionality reduction according to claim 7, characterized in that: The on-chip neural network chip (7) is processed on a silicon platform, a silicon nitride platform, an indium phosphide platform or a lithium niobate thin film platform.

Images