Analog photonic convolutional neural network image recognition system with arrayed input

Abstract

This invention discloses an array-based analog photonic convolutional neural network image recognition system. The system comprises an optical input module for acquiring the light field of a single-wavelength spectral image; an optical processing module for processing the acquired light field; an optical computation module for performing full neural network computation on the light field; and an optical detection module for classifying the computation results to output the image recognition result. This solution utilizes a spatially multiplexed array channel to replicate the input for the single-wavelength spectral image light field, significantly reducing the diffraction distance of the all-optical computation. Even without diffraction distance, a certain level of accuracy is maintained, achieving high accuracy at a diffraction distance of 2mm. Furthermore, a microlens array is used to fan out the input image to create identical copies. Regional weighting of the spatial light modulator elements enables spatial multiplexing of the spatial light modulator, further improving its efficiency and network accuracy.

Description

Technical Field

[0001] This invention belongs to the field of image recognition, specifically relating to an image recognition system with arrayed input analog photonic convolutional neural network. Background Technology

[0002] Image recognition technology is an important field of artificial intelligence. It refers to the technology of object recognition in images to identify targets and objects of various patterns, and has significant applications in many fields such as navigation, map and terrain registration, natural resource analysis, weather forecasting, environmental monitoring, and physiological disease research. However, current image recognition methods are computationally complex and computationally intensive, requiring enormous computing resources and incurring considerable latency. Optical neural networks can effectively alleviate some of the computational resource pressure on both software and electronic hardware, providing a promising alternative to artificial neural networks. The most energy-intensive and time-consuming part of artificial neural networks is dense matrix multiplication. However, in optical neural networks, matrix multiplication can be performed at the speed of light. The nonlinearity in artificial neural networks can also be achieved in optical neural networks using nonlinear optical elements. Furthermore, once the optical neural network is trained, this structure can perform optical signal calculations without additional energy input.

[0003] All-optical neural networks refer to networks that, after undergoing certain modulation techniques, load information onto the intensity, phase, or polarization characteristics of light, and directly perform neural network calculations through the interference, diffraction, polarization, or scattering properties of light. Diffraction neural networks based on space optics primarily utilize devices such as static memory masks, metasurface arrays, or spatial light modulators (SLMs) to modulate the light field in space through diffraction and other methods to achieve neural network calculations. They offer advantages such as high speed, low energy consumption, and strong parallel computing capabilities. However, existing photonic neural networks still have some drawbacks.

[0004] The challenges include: 1) Previous methods used multiple or multiplexed reflective SLMs, resulting in long optical paths and large space requirements. Furthermore, the long diffraction distance of current optical neural networks, due to accuracy requirements, hinders the miniaturization and integration of all-optical neural networks. 2) Currently, there is no good method to achieve all-optical multi-dimensional convolution calculations. In summary, optical diffraction neural networks remain a hot topic in space optics, with broad application prospects, but also facing many challenges that urgently need to be addressed. Summary of the Invention

[0005] To address the above-mentioned problems, the present invention aims to provide a solution.

[0006] The specific technical solution for achieving the objective of this invention is as follows:

[0007] An arrayed input analog photonic convolutional neural network image recognition system includes an optical input module, an optical processing module, an optical computing module, and an optical detection module;

[0008] The optical input module is used to acquire the light field of a single-wavelength spectral image;

[0009] The light processing module is used to process the light field of the acquired single-wavelength spectral image;

[0010] The optical computing module is used to perform full neural network calculations on the optical field;

[0011] The optical detection module is used to classify the calculation results, thereby outputting image recognition results;

[0012] The optical input module is connected to the optical processing module and the optical computing module, and the optical computing module is connected to the optical processing module and the optical detection module.

[0013] Furthermore, the optical input module utilizes optical elements to obtain a single-wavelength spectral image light field with the same incident phase and a certain light intensity distribution.

[0014] Furthermore, the light processing module is used to process the acquired single-wavelength spectral image light field, including:

[0015] Filter out stray ambient light;

[0016] The light field is polarized to obtain linearly polarized light.

[0017] Furthermore, the light processing module includes a first processing unit and a second processing unit;

[0018] The first processing unit is set as a polarizer between the optical input module and the optical processing module, and the second processing unit is set as an analyzer between the optical computing module and the optical detection module.

[0019] Furthermore, the optical computing module includes a spatial multiplexing array and an optical modulation array;

[0020] The spatial multiplexing array receives the light field processed by the optical processing module and uniformly splits the incident light field to generate spatial beam arrays at different positions, forming multiple array-type image channels, which are then output to the optical modulation array.

[0021] The optical modulation array receives the array-type image channel output by the spatial multiplexing array, performs optical fully connected calculations, modulates the amplitude and phase information of the incident light field, and performs multiply-accumulate calculations on the incident light field, simulating the role of neurons in a traditional neural network, realizing convolutional neural network calculations, and outputting the calculated light field.

[0022] Furthermore, the spatial multiplexing array includes a plurality of uniformly arranged microlenses;

[0023] The microlenses are arranged in M ​​rows and N columns to form MN channels. The light field after passing through the light processing module is divided into MN channel images after being spatially multiplexed by an array and then input into an optical modulation array for all-optical neural network calculation.

[0024] Furthermore, the optical modulation array is used to perform fully connected optical computation on the input multi-channel image;

[0025] The optical modulation array adopts any one of static memory mask, metasurface array, and liquid crystal cell array. The modulation information is one or both of phase and amplitude. Multiple optical modulation arrays are vertically integrated to form a series structure, thereby realizing the multiplication of two or more neural networks.

[0026] Furthermore, multiple different filters or polarizers are placed between the spatial multiplexing array and the optical modulation array to perform wavelength-selective transmission or polarization selection on each channel image, thereby realizing frequency division multiplexing or polarization multiplexing.

[0027] Furthermore, the optical processing module employs one of a polarizer, a polarizing prism, or a waveplate.

[0028] Furthermore, the optical detection module is either a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor.

[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0030] (1) Compared with the general optical neural network, which uses multiple reflective SLMs, has a long optical path and occupies a large space volume, the optical modulation array of the present invention adopts a transmissive spatial light modulator, with the optical path on the same horizontal axis. The image is transmitted in a straight line from the input end, passes through a series of optical elements and reaches the output end. It does not need to pass through the reflective device, which greatly reduces the space size of the system.

[0031] (2) Compared with the general optical neural network that directly performs neural network calculations on the image, the present invention uses a microlens array to fan out the input image to create the same copy, and loads weights on the array elements of the spatial light modulator in different regions to realize the spatial multiplexing of the spatial light modulator, which further improves the utilization efficiency of the spatial light modulator and improves the accuracy of the network.

[0032] (3) Compared with general photoelectric neural networks, this scheme uses spatial multiplexing array channels to copy the input for single-wavelength spectral image light field. Essentially, it expands the information of the input in the vertical plane of space. Therefore, the calculation required in the propagation direction is reduced, and the distance requirement is reduced. This greatly reduces the diffraction distance of all-optical calculation. It still has a certain accuracy under the condition of no diffraction distance. High accuracy can be achieved under the condition of 2mm diffraction distance, which further improves the integration of the network.

[0033] The present invention will be further described below with reference to specific embodiments. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the image recognition system architecture of the analog photonic convolutional neural network with arrayed input according to the present invention.

[0035] Figure 2 This is a schematic diagram of an arrayed input analog photonic convolutional neural network image recognition system according to an embodiment of the present invention.

[0036] Figure 3 This is a schematic diagram of the arrangement of detection units in the optical detection module in an embodiment of the present invention.

[0037] Figure 4 This is a schematic diagram of the network recognition accuracy when the array quantity is 2*2-7*7 and the diffraction distance is 0mm in Embodiment 2 of the present invention.

[0038] Figure 5 This is a schematic diagram of the network recognition accuracy when the array number is 5*5-7*7 and the diffraction distance is 2mm in Embodiment 3 of the present invention. Detailed Implementation

[0039] Combination Figure 1 and Figure 2 An arrayed input analog photonic convolutional neural network image recognition system includes an optical input module 1, an optical processing module 2, an optical computing module 3, and an optical detection module 4.

[0040] The light input module 1 is used to acquire the light field of a single-wavelength spectral image;

[0041] The light processing module 2 is used to process the light field of the acquired single-wavelength spectral image;

[0042] The optical computing module 3 is used to perform full neural network calculations on the optical field;

[0043] The optical detection module 4 is used to classify the calculation results, thereby outputting image recognition results;

[0044] The optical input module 1 is connected to the optical processing module 2 and the optical computing module 3, and the optical computing module 3 is connected to the optical detection module 4 through the optical processing module 2.

[0045] The optical input module 1 uses optical elements to obtain a single-wavelength spectral image light field with the same incident phase and a certain light intensity distribution. It can be a combination of a laser and a digital baffle or a screen.

[0046] Generally speaking, a combination of laser, lens group and digital baffle can be used. The laser beam is converted into parallel light by the lens group. The parallel light is incident on the digital baffle and a light field with uniform intensity distribution and equal incident phase is obtained at the digital baffle. A4 paper is placed on the back of the digital baffle so that the incident light undergoes diffuse reflection and finally obtains the input image.

[0047] The emitted light from the light input module 1 is directly input into the light processing module 2 in free space. The light processing module 2 is used to process the acquired single-wavelength spectral image light field, including:

[0048] By filtering out stray light from the environment and polarizing the light field, linearly polarized light is obtained.

[0049] The light processing module 2 includes a first processing unit and a second processing unit;

[0050] The first processing unit is set as a polarizer between the optical input module 1 and the optical processing module 2, that is, after the digital baffle. The second processing unit is set as an analyzer between the optical computing module 3 and the optical detection module 4.

[0051] The optical processing module 2 filters out stray ambient light, ensuring that the light in the optical path is linearly polarized, thus guaranteeing the working effect of the spatial light modulator.

[0052] Typically, the light processing module 2 uses one of a polarizer, a polarizing prism, or a waveplate.

[0053] The optical computing module 3 includes a spatial multiplexing array 5 and an optical modulation array 6;

[0054] The spatial multiplexing array 5 receives the light field processed by the light processing module 2 and uniformly splits the incident light field to generate spatial beam arrays at different positions, forming multiple array-type image channels, and then outputs them to the optical modulation array 6.

[0055] The optical modulation array 6 receives the array-type image channel output by the spatial multiplexing array 5, performs optical fully connected calculations, modulates the amplitude and phase information of the incident light field, and performs multiplication and addition calculations on the incident light field to simulate the function of neurons in a traditional neural network, realize convolutional neural network calculations, and output the calculated light field.

[0056] The optical modulation array 6 is placed on the focal plane of the photodetector module 4. The computational light field output by the optical computing module 3 is directly acquired by the detection surface of the photodetector module 4, and the classification result is finally used as the network output.

[0057] The spatial multiplexing array 5 includes multiple uniformly arranged microlenses;

[0058] The microlenses are arranged in M ​​rows and N columns to form MN channels. The light field after passing through the light processing module 2 is divided into MN channel images after passing through the spatial multiplexing array 5 and input to the optical modulation array 6 for all-optical neural network calculation.

[0059] The optical modulation array 6 is used to perform fully connected optical computation on the input multi-channel image;

[0060] The optical modulation array 6 adopts any one of static memory mask, metasurface array, and liquid crystal cell array. The modulation information is one or both of phase and amplitude. The modulation information is amplitude or phase. Multiple optical modulation arrays are vertically integrated to form a series structure, thereby realizing the multiplication of two or more neural networks.

[0061] Multiple different filters or polarizers are set between the spatial multiplexing array 5 and the optical modulation array 6 to perform wavelength-selective transmission or polarization selection on each channel image, thereby realizing frequency division multiplexing or polarization multiplexing.

[0062] The optical detection module 4 is either a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) sensor.

[0063] Example 1

[0064] In this embodiment, the optical input module 1 adopts a combination of a laser and a digital baffle.

[0065] First, the input image is obtained using the light input module 1. The light input module 1 mainly consists of a laser, a lens group, and a digital baffle. The laser has a wavelength of 670nm red light and an output power of 5mW. The lens group consists of lens 1 with a focal length of 6mm and lens 2 with a focal length of 150mm. It converts the laser output light into parallel light and then incident it onto the digital baffle. At the digital baffle, a light field with uniform intensity distribution and equal incident phase is obtained. The digital baffle is a 1mm thick aluminum plate. The digits in the MNIST dataset are etched by laser processing to obtain the input image. The output light of the light input module 1 is directly input to the light processing module 2.

[0066] In this embodiment, two polarizers are used as the light processing module 2, one of which is placed behind the digital baffle as a polarizer, and the other is placed in front of the light detection module 4 as an analyzer.

[0067] The input image processed by the light processing module 2 is converted into linearly polarized light, and the image is filtered to remove ambient stray light. After being processed by the first polarizer, the light is directly input to the light computing module 3.

[0068] In this embodiment, the spatial multiplexing array 5 of the optical computing module 3 adopts a 7*7 microlens array. After the incident light passes through the microlens array, it is divided into 49 optical transmission channels and then output to the optical modulation array 6. The transmissive spatial light modulator performs parallel all-optical neural network calculations on the images of the 49 channels and outputs the calculated light field.

[0069] In this embodiment, a combination of a CCD camera and an objective lens is used as the light detection module 4. The spatial light modulator of the optical modulation array 6 is placed on the focal plane of the objective lens. The objective lens and the CCD camera are directly connected by threads. The calculated light field of the spatial light modulator is obtained on the CCD camera and used as the network output.

[0070] In this embodiment, the neural network is mainly trained using the MNIST dataset, which contains 10 classes of numbers, 10,000 sample data, and image pixel size of 28*28. The data is divided into groups of 100 data using mini-batch.

[0071] In this embodiment, such as Figure 3 As shown, the detection units in the detection surface of the detection module are distributed in a [3,4,3] unit pattern, and the units are set at the center of the array of the arrayed grid.

[0072] Experiments were conducted based on the scheme described in this embodiment, and the experimental data are shown in the table below:

[0073] Area 0 Area 1 Area 2 Area 3 Area 4 Area 5 Area 6 Area 7 Area 8 Area 9 result Number 0 0.18 0.16 0.15 0.14 0.16 0.04 0.05 0.03 0.04 0.06 Yes Number 1 0.13 0.29 0.19 0.05 0.03 0.03 0.09 0.06 0.03 0.09 Yes Number 2 0.09 0.12 0.55 0.02 0.02 0.03 0.05 0.03 0.04 0.06 Yes Number 3 0.09 0.09 0.11 0.12 0.09 0.09 0.11 0.07 0.11 0.12 Yes Number 4 0.01 0.04 0.20 0.05 0.29 0.05 0.09 0.05 0.05 0.17 Yes Number 5 0.09 0.10 0.12 0.09 0.09 0.13 0.11 0.09 0.08 0.11 Yes Number 6 0.09 0.11 0.16 0.09 0.08 0.07 0.19 0.03 0.03 0.14 Yes Number 7 0.07 0.08 0.11 0.08 0.07 0.08 0.12 0.18 0.08 0.13 Yes Number 8 0.08 0.07 0.13 0.10 0.10 0.08 0.08 0.10 0.13 0.13 No Number 9 0.08 0.09 0.16 0.06 0.07 0.07 0.14 0.08 0.09 0.16 Yes

[0074] As can be seen, 90% of the numbers were successfully identified in the experiment, which is sufficient to demonstrate the effectiveness of this method.

[0075] Example 2

[0076] In this embodiment, the spatial multiplexing array 5 of the optical computing module 3 adopts microlens arrays of 2*2, 3*3, 4*4, 5*5, 6*6 and 7*7 respectively. After the incident light passes through the microlens array 5, it is divided into up to 49 optical transmission channels and then output to the spatial light modulator 6.

[0077] The transmissive spatial light modulator performs parallel all-optical neural network calculations on images of up to 49 channels and outputs the calculated light field.

[0078] In this embodiment, the neural network is mainly trained using the MNIST dataset, which contains 10 classes of numbers, 10,000 sample data, and image pixel size of 28*28. The data is divided into groups of 100 data using mini-batch.

[0079] In this embodiment, the detection units in the detection surface of the optical detection module 4 are distributed in a [3,4,3] unit pattern, and the units are set at the center of the array of the arrayed grid.

[0080] In this embodiment, the optical modulation array 6 of the optical computing module 3 includes three transmissive spatial light modulators, and the distance between each transmissive spatial light modulator is 0 mm.

[0081] In this embodiment, as the number of input arrays increases, the network accuracy is 24.2%, 53.3%, 61.8%, 66.5%, 70.7%, and 73.5%, respectively. Figure 4 As shown, it can be seen that the accuracy of digital recognition increases with the increase of the number of optical multiplexes in the optical multiplexing array.

[0082] Example 3

[0083] In this embodiment, the spatial multiplexing array 5 of the optical computing module 3 adopts 5*5, 6*6 and 7*7 microlens arrays respectively. After the incident light passes through the spatial multiplexing array 5, it is divided into up to 49 optical transmission channels and then output to the spatial light modulator of the optical modulation array 6. The transmissive spatial light modulator performs parallel all-optical neural network calculations on the images of up to 49 channels and outputs the calculated light field.

[0084] In this embodiment, the neural network is mainly trained using the MNIST dataset, which contains 10 classes of numbers, 10,000 sample data, and image pixel size of 28*28. The data is divided into groups of 100 data using mini-batch.

[0085] In this embodiment, the detection units in the detection surface of the optical detection module 4 are distributed in a [3,4,3] unit pattern, and the units are set at the center of the array of the arrayed grid.

[0086] In this embodiment, the optical modulation array 6 of the optical computing module 3 includes three transmissive spatial light modulators, with each transmissive spatial light modulator spaced 2 mm apart.

[0087] In this embodiment, as the number of input arrays increases, the network accuracy is 88.5%, 89.8%, and 91.4%, respectively. Figure 5 As shown, it can be seen that even with a significantly reduced diffraction distance compared to other schemes, the accuracy can still be on par with other existing schemes.

[0088] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image recognition system with arrayed input analog photonic convolutional neural network, characterized in that, It includes an optical input module (1), an optical processing module (2), an optical computing module (3), and an optical detection module (4). The light input module (1) is used to acquire the light field of a single-wavelength spectral image; The light processing module (2) is used to process the light field of the acquired single-wavelength spectral image; The optical computing module (3) is used to perform full neural network calculations on the optical field; The optical detection module (4) is used to classify the calculation results, thereby outputting image recognition results; The optical input module (1) is connected to the optical processing module (2) and the optical computing module (3), and the optical computing module (3) is connected to the optical detection module (4) through the optical processing module (2); The optical computing module (3) includes a spatial multiplexing array (5) and an optical modulation array (6). The spatial multiplexing array (5) receives the light field processed by the light processing module (2), and uniformly splits the incident light field to generate spatial beam arrays at different positions, forming multiple array-type image channels, and then outputs them to the optical modulation array (6). The optical modulation array (6) receives the array-type image channel output by the spatial multiplexing array (5), performs optical fully connected calculations, modulates the amplitude and phase information of the incident light field, and performs multiply-add calculations on the incident light field, simulating the role of neurons in a traditional neural network, realizing convolutional neural network calculations, and outputting the calculated light field. The spatial multiplexing array (5) includes a plurality of uniformly arranged microlenses; The microlenses are arranged in M ​​rows and N columns to form MN channels. The light field of the light processing module (2) is divided into MN channel images after passing through the spatial multiplexing array (5) and input to the optical modulation array (6) for all-optical neural network calculation. The optical modulation array (6) is used to perform fully connected optical computation on the input multi-channel image; The optical modulation array (6) adopts any one of static memory mask, metasurface array, and liquid crystal cell array. The modulation information is one or both of phase and amplitude. The modulation information is amplitude or phase. Multiple optical modulation arrays are vertically integrated to form a series structure, thereby realizing the multiplication of two or more neural networks.

2. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 1, characterized in that, The light input module (1) uses optical elements to obtain a single-wavelength spectral image light field with the same incident phase and a certain light intensity distribution.

3. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 1, characterized in that, The light processing module (2) is used to process the light field of the acquired single-wavelength spectral image, including: Filter out stray ambient light; The light field is polarized to obtain linearly polarized light.

4. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 3, characterized in that, The light processing module (2) includes a first processing unit and a second processing unit; The first processing unit is set as a polarizer between the optical input module (1) and the optical processing module (2), and the second processing unit is set as an analyzer between the optical computing module (3) and the optical detection module (4).

5. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 1, characterized in that, Multiple different filters or polarizers are set between the spatial multiplexing array (5) and the optical modulation array (6) to perform wavelength selection transmission or polarization selection on each channel image, thereby realizing frequency division multiplexing or polarization multiplexing.

6. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 4, characterized in that, The light processing module (2) uses one of a polarizer, a polarizing prism, or a waveplate.

7. The analog photonic convolutional neural network image recognition system with arrayed input according to claim 1, characterized in that, The optical detection module (4) is either a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) detector.

Images