Neuromorphic intelligent optical computing architecture system and apparatus

Abstract

The application discloses a neuromorphic intelligent light computing architecture system and device, and relates to the field of machine learning, and specifically discloses the following technical scheme: a multi-channel representation module, which encodes original input target light field signals into coherent light of different wavelengths through a multi-spectrum laser; an attention-aware light neural network module, which comprises a BU and a TD light attention module, wherein the coherent light of different wavelengths is input into the BU light attention module, and the attention-aware light neural network is trained, so that the TD light attention module modulates the multi-dimensional sparse features extracted by the BU light attention module in terms of spectral and spatial transmittance to obtain final spatial light output based on the trained attention-aware light neural network; and an output module, which detects and identifies the final spatial light output on an output plane to obtain the positioning and recognition result of the target in the light field. The application has unprecedented capability and scalability by adaptively allocating computing resources, and for the first time utilizes a light neural network to solve high-complexity machine learning problems.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neuromorphic computing, in particular to a neuromorphic intelligent light computing architecture system and device. BACKGROUND

[0002] Optical neuromorphic computing has demonstrated its potential in high energy efficiency and parallel computing. However, existing optical architectures apply dense optical neuron connections and pursue higher capacity by simply expanding or deepening the network, resulting in a highly complex and redundant network, and thus can only be used to solve simple tasks. Instead, the human brain allows highly efficient analysis of various complex tasks using an event-driven attention mechanism and sparse neuron connections.

[0003] Artificial intelligence has made great progress and has had a wide impact on machine vision, autonomous driving, intelligent robots, etc. Modern machine intelligence tasks require complex algorithms and large-scale computing, resulting in a growing demand for computing resources. As Moore's law stagnates, energy efficiency has become a major obstacle to electronic-based neural networks, hindering the wider application of today's artificial intelligence technology. Recently, due to the inherent high speed and high energy efficiency of light propagation, optical neural networks (ONN) that use light instead of electricity for computing have proven their potential as the next generation of computing models, and small-scale all-optical systems have successfully verified basic visual processing tasks such as handwritten digit recognition and saliency detection. Deep optical, Fourier neural networks and hybrid optical-electronic CNNs integrate electronic components into optical architectures to enhance ordinary ONNs. Other works multiplex optical computing units, trying to process larger inputs and achieve better performance. In essence, these methods maintain the original dense optical neuron connections and pursue higher capacity by simply expanding or deepening the network, unfortunately, this results in severe computational redundancy, making optical networks unable to complete advanced real-world tasks. Instead, the human brain uses an event-driven attention mechanism and applies spectral and spatial sparse neuron connections for extremely efficient parallel computing of general complex tasks. In fact, optical computing has inherent sparsity and parallelism due to its massive optical connections, which can naturally generalize the features of biological neurons to optical neurons. SUMMARY

[0004] The present application aims to at least partially solve one of the technical problems in the related art.

[0005] To this end, the present application proposes a neuromorphic intelligent optical computing architecture system, designs an attention-aware optical neural network (AttnONN), a hierarchical optical network architecture that applies spectral and spatial sparse optical convolution simultaneously, in which optical neurons are only activated when there is a signal to be processed. AttnONN can adaptively allocate computing resources, has unprecedented ability and scalability, and the present application first uses optical neural networks to solve high-complexity machine learning problems. Experimental results confirm that AttnONN has high performance and high energy efficiency on various challenging tasks, and compared with the original optical network, the learning ability is improved by 8 times, and the efficiency is more than 2 orders of magnitude higher than that of a representative electrical neural network (ResNet-18, etc.).

[0006] Another object of the present application is to propose a neuromorphic intelligent optical computing architecture device.

[0007] To achieve the above object, in one aspect, the present application proposes a neuromorphic intelligent optical computing architecture system, which comprises a multi-channel representation module, an attention-aware optical neural network module and an output module, wherein,

[0008] The multi-channel representation module is used to encode the original input target light field signal into coherent light of different wavelengths by a multi-spectral laser;

[0009] The attention-aware optical neural network module comprises a BU and a TD optical attention module, the different wavelengths of coherent light are input into the BU optical attention module, and the attention-aware optical neural network is trained to modulate the multi-dimensional sparse features extracted by the BU optical attention module by the TD optical attention module based on the trained attention-aware optical neural network to obtain the final spatial light output;

[0010] The output module is used to detect and identify the final spatial light output on the output plane to obtain the positioning and identification results of the target in the light field.

[0011] In addition, the neuromorphic intelligent optical computing architecture system according to the above embodiments of the present application can also have the following additional technical features:

[0012] Further, in one embodiment of the present application, the BU optical attention module comprises a first BU optical attention module and a second BU optical attention module, and the TD optical attention module takes the output features of the first BU optical attention module as input and processes feedback to adjust the second BU optical attention module.

[0013] Further, in an embodiment of the present application, the output of the TD light attention module is obtained by controlling the connection of the light neurons in the second BU light attention module based on the super-surface based light filter, while the light neurons are spectrally and spatially modulated to obtain the light neuron connection result and the modulation result.

[0014] Based on the light neuron connection result and the modulation result and the light attention factor, the intensity sensor detects and identifies the final spatial light output in the output plane to obtain the positioning and identification result of the target in the light field.

[0015] Further, in an embodiment of the present application, each unit of the super-surface based light filter is composed of two layers of films, the first layer is a GST unit, and the second layer is an intensity mask unit, the GST unit includes amorphous and crystalline states, corresponding to different transmission spectra, and the two states are switched instantaneously by conversion light.

[0016] Further, in an embodiment of the present application, the BU and TD light attention modules are constructed by inserting a multi-layer sparse light convolution unit into the Fourier plane of the 4f optical system under the coherent light of different wavelengths; the preset is the input light field of the first BU light attention module at the i-th wavelength, and the first 2f optical system under the coherent light is used to perform Fourier transform on the input to obtain the first feature:

[0017]

[0018] wherein, represents the optical feature in the Fourier domain, F represents the Fourier transform matrix; and the second feature is converted again:

[0019]

[0020] wherein, represents the converted attention feature, T represents the complex conversion matrix performed;

[0021] The first BU light attention module is propagated based on diffraction, and the second feature is transmitted to the next layer as input to obtain output data and transmitted to the TD light attention module and the second BU light attention module.

[0022] Further, in an embodiment of the present application, the input of the TD light attention module and the second BU light attention module is transformed as:

[0023]

[0024]

[0025] use represents the feature of the kth layer, which is based on the propagation of the TD light attention module modulating each second BU light attention module in the Fourier space:

[0026]

[0027] wherein, represents the attention feature of the modulated second BU light attention module, and M k respectively represent the spectral and spatial modulation functions determined by the TD light attention module;

[0028] The preset TD light attention module and the second BU light attention module are m layers, and the spectrum is set to n wavelengths, so as to calculate the final spatial light output through an activation function and perform Fourier transform to the real space by using a second 2f optical system:

[0029]

[0030] wherein, represents the corresponding nonlinear function of the photorefractive crystal, and P represents the output of the entire framework.

[0031] Further, in an embodiment of the present application, the loss function during network training of the attention-aware light neural network is defined as:

[0032] L(T) = ||P - Γ(G) ||

[0033] wherein, G is the true value, Γ represents a spatial inversion operation due to the use of two optical Fourier transforms, and the finally obtained loss will be back propagated to optimize the spectral and spatial coefficients of the BU and TD branches.

[0034] Further, in an embodiment of the present application, each 3 layers of the first light attention module, the TD light attention module and the second BU light attention module are defined as an attention unit, and the size of each attention unit is 2*2 μm, and in each layer, each spectral channel contains 800*800 size diffraction neurons for training.

[0035] Further, in an embodiment of the present application, the gap between the layers of the first BU light attention module, the TD light attention module and the second BU light attention module is set to 100 μm, and each channel of the network channel is allocated a wavelength between 500-1500 nm; the intensity threshold of all intensity mask units is set to 0.3, and the light neurons lower than the intensity threshold are set to be not activated.

[0036] To achieve the above object, another aspect of the present application provides a neuromorphic intelligent light computing architecture device, comprising a multi-spectral laser, a beam splitter, a mirror, a lens, a first BU light attention module, a second BU light attention module, a light filter, a TD light attention module and an intensity sensor.

[0037] A target light field signal is input to the multi-spectral laser to output coherent light of different wavelengths, which is guided by the beam splitter, the mirror and the lens to propagate based on diffraction, and after propagation, the TD light attention module takes the multi-dimensional sparse features output by the first BU light attention module as input and processes feedback to adjust the second BU light attention module to control the connection of light neurons in the second BU light attention module through the light filter and perform spectral and spatial transmittance modulation on the light neurons, and based on a light attention factor, the intensity sensor is used to detect the positioning and recognition result of the target in the light field.

[0038] The neuromorphic intelligent light computing architecture system and device of the embodiment of the present application complete attention-aware sparse learning to adaptively allocate computing resources, and can run large-scale complex machine vision applications at the speed of light.

[0039] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter in the description of the application. BRIEF DESCRIPTION OF DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description of embodiments, taken in conjunction with the accompanying drawings, in which:

[0041] Figure 1 is a structural schematic diagram of a neuromorphic intelligent light computing architecture system according to an embodiment of the present application;

[0042] Figure 2 is a principle schematic diagram of a neuromorphic intelligent light computing architecture AttnONN according to an embodiment of the present application;

[0043] Figure 3 is a structural schematic diagram of a light filter based on a metasurface according to an embodiment of the present application;

[0044] Figure 4 is a performance evaluation schematic diagram of an AttnONN on a target detection task according to an embodiment of the present application;

[0045] Figure 5 is a working schematic diagram of an AttnONN on a 3D target classification task according to an embodiment of the present application;

[0046] Figure 6is a structural schematic diagram of a neuromorphic intelligent light computing architecture device according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0048] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0049] The neuromorphic intelligent light computing architecture system and device according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0050] Figure 1 is a structural schematic diagram of a neuromorphic intelligent light computing architecture system according to an embodiment of the present application.

[0051] As shown in Figure 1 , the system 10 includes a multi-channel representation module 100, an attention-aware light neural network module 200, and an output module 300, wherein,

[0052] The multi-channel representation module 100 is configured to encode the original input target light field signal into coherent light of different wavelengths by a multi-spectral laser.

[0053] The attention-aware light neural network module 200 includes a BU and a TD light attention module. The coherent light of different wavelengths is input to the BU light attention module, and the attention-aware light neural network is trained to modulate the multi-dimensional sparse features extracted by the BU light attention module by the TD light attention module based on the trained attention-aware light neural network to obtain the final spatial light output.

[0054] The output module 300 is configured to detect and identify the final spatial light output on the output plane to obtain the positioning and identification result of the target in the light field.

[0055] Figure 2 The principle of the neuromorphic intelligent light computing architecture of the present application. Figure 2(a) in the diagram shows the bottom-up (BU) and top-down (TD) visual attention flow in the human brain. The BU attention flow transfers raw sensory input to salient features with potential importance, such as paying attention to salient areas in the background; the TD attention flow shifts BU attention to prior knowledge based on long-term cognition, such as finding a car. Together, they locate the target location and identify its category. Figure 2 (b) in the figure shows the hierarchical optical neuromorphic structure of AttnONN. The BU optical attention module extracts multidimensional sparse features of different wavelengths; the TD optical attention module modulates the spectral and spatial transmittance of the BU attention, thereby obtaining the target's localization and recognition results in the light field.

[0056] Understandably, the human visual system relies on two distinct attentional processes. For example... Figure 2 As shown in (a), the internal brain processes depict the cortical pathways involved in visual attention, while the external processes illustrate the integration of corresponding computational stages in the bottom-up (BU) and top-down (TD) approaches. Specifically, the visual scene captured by the eye, along with its multidimensional information (e.g., color, view, or intensity), is fed to the sparsely connected prefrontal cortex (PFC), posterior parietal cortex (PPC), and visual cortex (VC) for processing. The BU attention stream can be modulated by the TD attention stream to the current behavioral goal and prior knowledge. Ultimately, attention is focused on the location of the most salient activity / object in the visual scene, which can be used for reasoning in higher semantic tasks such as visual detection. During attention stream processing, neuronal connections are sparse and parallel, and synapses only activate when there are relevant signals that need to be processed.

[0057] In one embodiment of the present invention, the architecture of AttnONN is designed as follows: Figure 2 As described in (b) above. The multi-channel characterization of the input signal is encoded onto different wavelengths of the optical field and divided into TD and BU branches. A multispectral laser, beam splitter (BS), mirror (M), and lens (L) are used to generate and guide diffraction-based light propagation. The BU and TD optical attention modules are constructed by inserting sparse optical convolutional units into the Fourier plane of a 4f optical system under coherent light. The sparse optical convolutional units are formed by stacking multiple layers, where each layer transmits sparse features as input to the next layer. The TD optical attention module takes the output features of BU optical attention module 1 (BU1) as input and processes the feedback to further modulate BU optical attention module 2 (BU2).

[0058] Furthermore, the TD output controls the optical neuron connections in BU2 via a metasurface-based optical filter, simultaneously performing spectral and spatial modulation on them. This is combined with the optical attention factor U... bu1 U td and U bu2The final result can be detected using an intensity sensor on the output plane. Preferably, the network architecture proposed in this invention is configured to use wavelengths of 500-1500 nm, which provides a wide range for spectral selection.

[0059] Figure 3 The structure of the proposed metasurface-based optical filter is shown, such as Figure 3 As shown, each unit consists of two thin films: the first layer is a 2×2 μm GeSbTe (GST) film grown on a transparent silicon substrate, and the second layer is an intensity mask. GST has two states (amorphous and crystalline), corresponding to different transmission spectra, and the two states can be instantaneously switched using conversion light. The intensity mask unit is based on a digital micromirror device (DMD) for spatial modulation. Preferably, for nonlinear activation functions in network propagation, this invention uses a photorefractive crystal (SBN:60) as the optical nonlinear layer. SBN:60 can adaptively change its refractive index according to changes in light intensity distribution, thereby providing the computation of all-optical activation functions in optical neuron connections. It is understood that... Figure 3 A metasurface optical filter based on phase change material GST and an intensity mask is proposed. Adaptive spectral modulation is achieved by switching the GST unit between different states during phase transition, while spatial modulation is achieved by changing the on / off state of the intensity mask.

[0060] In one embodiment of the present invention, it is assumed that The input light field of the BU1 optical attention module at the i-th wavelength is used in a 2f system under coherent light. The input is Fourier transformed as follows: in Let F represent the optical features in the Fourier domain, and let F represent the Fourier transform matrix. Subsequently, the features are transformed into: in Let T represent the transformed attention features, and T represent the complex transformation matrix performed. Each attention layer performs diffraction-based propagation and passes the features as input to the next layer, finally obtaining the output. This is then passed to the subsequent TD and BU2 modules. Similarly, the input transformation of the first layer of TD and BU2 is as follows: and use Representing the features of the k-th layer, as the TD propagates, each BU2 layer is simultaneously modulated by the TD in Fourier space:

[0061]

[0062] in This represents the modulated BU2 attention features. and M kTD attention-decided spectral and spatial modulation functions, which can adaptively prune and activate the connections of optical neurons to achieve sparse optical convolution. Given the TD and BU2 modules for m layers, and the spectral setting for n wavelengths, the final output is computed by a complex activation function and applied another 2f system Fourier transform back to real space:

[0063]

[0064] wherein represents the application of the corresponding nonlinear function of the photorefractive crystal, P represents the output of the entire framework.

[0065] During the network training of the attention-aware optical neural network, the input data is encoded in real time into the information of the complex light field, and the output is measured by the intensity sensor. The loss function of the network training can be defined as L(T) = ||P - Γ(G)||, wherein G is the true value, and Γ represents the spatial inversion operation due to the use of two optical Fourier transforms. The final obtained loss is back-propagated to optimize the spectral and spatial coefficients of the BU and TD branches.

[0066] Further, in the experiment, the present application applies 3 attention units with 9 layers of 800x800 optical neurons for evaluation (every 3 layers from the BU1, TD and BU2 modules are defined as an attention unit), and the size of each attention unit is 2x2μm. In addition to the attention neurons, in each layer, there are 800x800 trainable diffractive neurons in each spectral channel.

[0067] Further, the aperture of the double 2f system is set to match the layer size so that the intensity sensor can better capture the network output. The present application determines the gap between the layers to be 100μm, which provides higher spatial usage efficiency for network calculation. The number of network channels depends on the input data structure, and each channel is assigned a specific wavelength ranging from 500-1500nm. The present application sets the intensity threshold of all intensity mask units to 0.3, and the optical neurons below the threshold will be set as inactive on the mask.

[0068] Figure 4 Schematic diagram for performance evaluation of AttnONN on target detection task. Figure 4 (a) in the AttnONN inference process for target detection and its projection-interference-prediction workflow. Figure 4 (b) in the representative result comparison between different benchmarks of the original ONN, the AttnONN applying BU attention, and the AttnONN applying BU and TD attention. The present application observes that the proposed architecture can quickly and accurately detect single-target and multi-target scenes.

[0069] In one embodiment of the present application, the present application first evaluates the performance of AttnONN on the challenging object detection task based on the complex KITTI dataset. This experiment applies a 2D object subset, of which 7,500 images are used for training and 2,500 images are used for validation. The present application adjusts the image resolution from the original 1225x375 to 800x800 as the network input, and the 4-channel RGBD (red, green, blue, and depth) representation is separated and encoded into wavelengths of 500nm, 600nm, 700nm, and 800nm. As shown in (a) of Figure 4 In the inference process of object detection by AttnONN, the present application converts and records the attention features, as shown in (a) of

[0070] For comparison, the present application constructs a 9-layer 800x800 original ONN and a high-performance ResNet-18 network based on electronics to learn using the same configuration. The representative detection results under different settings are shown in (b) of Figure 4 The detection true value is designed as a bright square on a blank background, which matches the target position on the original image. Compared with the original ONN, the BU light attention module prunes the most redundant light connections and retains significant features, while the TD light attention module further modulates attention to the region of interest and locates the target.

[0071] For quantitative evaluation of accuracy, the present application calculates the precision-recall (PR) curve between the detection results and the true value. The results show that the accuracy of AttnONN reaches 64.8%, 71.9%, and 79.0% without applying BU and TD attention, applying BU attention, and applying BU and TD attention, respectively. The peak performance of AttnONN is 39.8% higher than that of the original ONN.

[0072] In addition, the proposed AttnONN only activates 12.1% of the light neurons for light propagation, which is more than 8 times the learning ability compared with the original ONN using traditional dense connections, and the energy use efficiency is 2 orders of magnitude higher than that of the electrical network. The present application concludes that the proposed architecture obtains the unique advantages brought by sparse light convolution, which is the first time to detect objects on complex real-world data through ONN.

[0073] Figure 5The workflow of AttnONN on 3D object classification task. The multi-channel slices of 3D data are projected and encoded into light field, processed by BU and TD light attention modules, and finally generate classification patterns on the output plane.

[0074] In one embodiment of the present application, the performance of the proposed architecture on 3D object classification task is further evaluated. Figure 5 The inference workflow of AttnONN on ShapeNet dataset is illustrated, which contains 55 common object categories and over 50,000 3D models. The present application selects 5 categories as the subset for AttnONN functional verification. Each input 3D model is cropped into l slices and all are resized to 800x800 resolution, where l is set to 9 in the experiment. The multi-channel input is encoded using 9 different wavelengths in the range of 600-1400 nm, with a separation of every 100 nm. After propagation using BU and TD light attention modules, the classification output is measured by a sensor with a fixed pattern set.

[0075] It can be known that the present application applies the quantitative method of classification accuracy, and the present application measures that when the number of channels gradually increases, AttnONN can obtain higher accuracy, however when l>5, the accuracy of the original ONN sharply decreases. The highest accuracy achieved by AttnONN and ResNet-18 based on electronics is 93.8% and 94.3% respectively, which proves that the proposed architecture has competitive performance on complex tasks. In all experiments, AttnONN successfully utilizes the inherent sparsity and parallelism of light, providing significant optimization for optical computing.

[0076] In summary, the proposed neuromorphic intelligent optical computing architecture system completes attention-aware sparse learning, and can run large-scale complex machine vision applications at the speed of light. The present application verifies the high accuracy and high energy efficiency of AttnONN on challenging object detection and 3D object classification tasks through various experimental evaluations. As an embedded system, the proposed architecture can be manufactured and deployed to edge / terminal imaging systems including microscopes, cameras and smartphones, thereby building a more powerful optical computing system for modern advanced machine intelligence.

[0077] The neuromorphic intelligent optical computing architecture system according to the embodiment of the present application has high accuracy and high energy efficiency on challenging object detection and 3D object classification tasks, adaptively allocates computing resources, has unprecedented ability and scalability, and for the first time uses optical neural networks to solve high-complexity machine learning problems.

[0078] In order to realize the above-mentioned embodiments, as Figure 6As shown, the embodiment also provides a neuromorphic intelligent light computing architecture device 1, the device 1, including a multi-spectral laser 2, a beam splitter 3, a mirror 4, a lens 5, a first BU light attention module 6, a second BU light attention module 7, a light filter 8, a TD light attention module 9 and an intensity sensor 10;

[0079] The target light field signal is input to the multi-spectral laser 2 to output coherent light of different wavelengths, the coherent light of different wavelengths is guided based on diffraction light propagation through the beam splitter 3, the mirror 4 and the lens 5, after propagation, the TD light attention module 9 takes the multi-dimensional sparse features output by the first BU light attention module 6 as input and processes feedback to adjust the second BU light attention module 7, to control the connection of the light neurons in the second BU light attention module 7 through the light filter 8 and perform spectral and spatial transmittance modulation on the light neurons, and based on the light attention factor, the target positioning and recognition result in the light field are detected by using the intensity sensor 10.

[0080] The neuromorphic intelligent light computing architecture device according to the embodiment of the present application has high precision and high energy efficiency on challenging target detection and 3D target classification tasks, adaptively allocates computing resources, has unprecedented ability and scalability, and for the first time uses a light neural network to solve high-complexity machine learning problems.

[0081] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, different embodiments or examples described in the present specification and the features of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0082] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A neuromorphic intelligent photonic computing architecture system, characterized in that, The system comprises a multi-channel representation module, an attention-aware optical neural network module and an output module, wherein The multi-channel representation module is configured to encode an original input target light field signal into coherent light of different wavelengths by a multi-spectral laser. The attention-aware optical neural network module comprises a BU and a TD optical attention module, the coherent light of different wavelengths is input into the BU optical attention module, and the TD optical attention module is trained based on the trained attention-aware optical neural network to modulate the multi-dimensional sparse features extracted by the BU optical attention module in terms of spectral and spatial transmittance to obtain a final spatial light output. The output module is configured to detect and identify the final spatial light output on an output plane to obtain a positioning and identification result of a target in a light field. The BU optical attention module comprises a first BU optical attention module and a second BU optical attention module, the TD optical attention module takes the output features of the first BU optical attention module as input and performs processing feedback to adjust the second BU optical attention module. The output of the TD optical attention module is obtained by controlling the connection of optical neurons in the second BU optical attention module based on a super-surface-based optical filter, and the optical neurons are modulated in terms of spectrum and space to obtain the connection result and the modulation result of the optical neurons. Based on the connection result and the modulation result of the optical neurons and the optical attention factor, an intensity sensor detects and identifies the final spatial light output on an output plane to obtain a positioning and identification result of a target in a light field.

2. The neuromorphic intelligent photonic computing architecture system of claim 1, wherein, Each unit of the super-surface-based optical filter is composed of two layers of films, the first layer is a GST unit, and the second layer is an intensity mask unit, the GST unit comprises amorphous and crystalline states corresponding to different transmission spectra, and the two states are switched instantaneously by conversion light.

3. The neuromorphic intelligent photonic computing architecture system of claim 1, wherein, The BU and TD light attention modules are constructed by inserting a multi-layer sparse light convolution unit into optical system under the coherent light of different wavelengths; preset is the input light field of the first BU light attention module at the first wavelength, and the first feature is obtained by performing Fourier transform on the input by the first optical system under the coherent light. wherein represents the optical feature in the Fourier domain, represents the Fourier transform matrix; converting again to the second feature: wherein, denotes the converted attention feature, denotes the executed complex conversion matrix; propagating the first BU optical attention module based on diffraction and transmitting the second feature as input to a next layer to obtain output data and to the TD optical attention module and the second BU optical attention module.

4. The neuromorphic intelligent photonic computing architecture system of claim 3, wherein, The input of the TD optical attention module and the second BU optical attention module is transformed as follows: Using denotes the characteristic of the layer, based on the propagation of the TD light attention module, which modulates each second BU light attention module in Fourier space: wherein, denote the attention features of the modulated second BU light attention module, and denote the spectral and spatial modulation functions decided by the TD light attention module, respectively. The preset TD light attention module and the second BU light attention module are layer, and the spectral setting is wavelength, to calculate the final spatial light output by an activation function, and utilize the second optical system to perform Fourier transform to real space: wherein, () represents the corresponding nonlinear function of the photorefractive crystal, represents the output of the entire framework.

5. The neuromorphic intelligent photonic computing architecture system of claim 1, wherein, The loss function during the network training of the attention-aware optical neural network is defined as follows: wherein is a true value, denotes a spatial inversion operation due to the use of two optical Fourier transforms, the resulting loss will be back-propagated to optimize the spectral and spatial coefficients of the BU and TD branches.

6. The neuromorphic intelligent photonic computing architecture system of claim 1, wherein, Each 3 of the first optical attention module, the TD optical attention module and the second BU optical attention module are defined as an attention unit, and the size of each attention unit is In each layer, each spectral channel contains 800*800 size of diffractive neurons for training.

7. The neuromorphic intelligent photonic architecture system of claim 6, wherein, The gap between layers of the first BU light attention module, the TD light attention module and the second BU light attention module is set as , each channel of the network channel allocates a wavelength between 500-1500nm; the intensity threshold of all intensity mask units is set to 0.3, and the light neuron lower than the intensity threshold is set to be inactivated.

Images