Method for realizing speckle image target classification based on photoelectric convolutional neural network

Abstract

The invention relates to a method for realizing speckle image target classification based on a photoelectric convolutional neural network, which realizes operations such as two-dimensional convolution and global average pooling in digital operation through an optical 4f system consisting of photoelectric devices such as a coherent laser source, a spatial light modulator and a micro-lens array, and realizes nonlinear activation operation through a photorefractive crystal. Therefore, a convolutional neural network of an optical platform is constructed, and digital processing is replaced to realize speckle image target classification. According to the invention, the operation burden of the digital processor is reduced, and the dependence of network operation efficiency on the computing power of the processor is reduced.

Description

Technical Field

[0001] The invention relates to a method for realizing speckle image target classification based on an optoelectronic convolutional neural network, and belongs to the field of optoelectronic collaborative processing. Background Art

[0002] Normally, when a sensor receives a light beam reflected or transmitted from the surface of an object, it will obtain the image information of the object. However, when the light passes through uneven media such as clouds and fog during the transmission path, dispersion will occur. The information carried in the dispersed light beam will change, and the original image cannot be directly obtained from it. Therefore, we need to post-process the dispersed light beam and reconstruct and restore the information carried in the light beam through the technical means of speckle reconstruction. The development of deep learning provides a new idea for speckle reconstruction. Various speckle image recognition methods based on convolutional neural networks have emerged in an endless stream. However, most of the computational operations in the current scheme need to be implemented through digital processing. The scale of the network model and the amount of data processed affect the processing speed and computing efficiency.

[0003] In recent years, using optical methods to construct neural networks has gradually become a new research hotspot in the field of optical computing. In optical neural networks, data processing can be achieved through optical computing, and its processing speed is negligible. In addition, the coherent light beams carrying information propagate at the speed of light between network layers, are highly parallel, and cross in space without crosstalk, which can achieve high-speed parallel processing of large-scale data. Therefore, an optical neural network architecture for speckle image target classification can be studied to reduce the computational burden of digital processors and reduce the dependence of network computing efficiency on processor computing power. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for realizing speckle image target classification based on an optoelectronic convolutional neural network, thereby reducing the computational burden of a digital processor and reducing the dependence of network computing efficiency on processor computing power.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for classifying speckle image targets based on an optoelectronic convolutional neural network is disclosed. An optical 4f system composed of optoelectronic devices such as a coherent laser source, a spatial light modulator, and a microlens array is used to implement operations such as two-dimensional convolution and global average pooling in digital operations. Nonlinear activation operations are implemented through photorefractive crystals, thereby constructing a convolutional neural network of an optical platform to implement speckle image target classification instead of digital processing.

[0007] The specific steps are as follows:

[0008] Step 1: Construct an optical convolutional neural network, including:

[0009] (1) Constructing the input layer: Based on the coherent laser source and the Fourier lens, parallel and evenly diffused coherent laser light is generated, and the input image is modulated on the spatial light modulator. When the coherent laser light is irradiated on the surface of the spatial light modulator, the reflected light beam will carry the image information and transmit it backward;

[0010] (2) Constructing an optical convolution layer: Based on the spatial light modulator and the microlens array, an optical 4f system is formed to realize the parallel two-dimensional convolution operation of multi-channel input images and convolution kernels. The relationship between the number of convolution channels N and the number of target categories C is N = k*C, where k is a non-negative integer;

[0011] (3) Constructing an optical nonlinear modulation module: A nonlinear activation layer is constructed based on a photorefractive crystal. When a coherent light beam is irradiated on the photorefractive crystal, the light beam output from the crystal will generate an additional phase factor according to the light intensity information, thereby nonlinearly modulating the information carried in the light beam.

[0012] (4) cascading multiple groups of optical convolutional layers and optical nonlinear modulation modules to form a hidden layer of the optoelectronic convolutional neural network;

[0013] (5) Construct a global average pooling module: Perform global average pooling on the output of each channel of the hidden layer based on the microlens array, vectorize the image matrix, and the number of output channels is the same as the number of target categories C. The category of the input image is identified based on the results of each output channel.

[0014] Step 2: Build an optical system and obtain a speckle image dataset for optical neural network training.

[0015] Step 3: Perform model training on the optical convolutional neural network on a computer. When training the digital model, the training set and the test set use the speckle images actually collected and annotated by the optical system in step 2, and the convolution kernel matrix of the optical convolution layer as the parameters to be trained.

[0016] Step 4: Build the optical system of the optoelectronic convolutional neural network, power on the laser light source, spatial light modulator, nonlinear crystal, camera and other optoelectronic components, and load the input image and pre-trained convolution kernel parameters on the spatial light modulator of the optical convolution layer.

[0017] Step 5: The output of each channel after global average pooling (i.e., Airy disk intensity) is obtained through the sCMOS camera in the output layer, where the target category corresponding to the channel with the largest Airy disk intensity is the target category predicted by the optoelectronic convolutional neural network.

[0018] A computer program product stored on a non-transitory computer-readable medium, the computer program product comprising program codes for implementing the above-mentioned method for classifying speckle images based on an optoelectronic convolutional neural network.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) Optical neural networks can achieve low-latency, low-power target detection and recognition;

[0021] (2) Optical neural networks can reduce the reliance on high-computing data processors;

[0022] (3) By using a microlens array to construct an optical convolution layer, multi-channel independent and parallel two-dimensional convolution can be achieved;

[0023] (4) Optical nonlinear modulation can be achieved by using photorefractive crystals to avoid photoelectric conversion. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the optical architecture of the optoelectronic neural network.

[0025] Figure 2 Schematic diagram of the optical neural network model architecture.

[0026] Figure 3 Schematic diagram of the optical system for obtaining speckle images

[0027] Figure 4 Schematic diagram of the input image in the example.

[0028] Figure 5 This is a schematic diagram of the output image in the example. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] A method for classifying speckle images based on an optoelectronic convolutional neural network, the specific steps are as follows:

[0031] Step 1: Construct an optical convolutional neural network, including:

[0032] (1) Constructing the input layer

[0033] A laser transmitter with a wavelength of 532nm is selected as the laser light source of the optoelectronic neural network. After being collimated by a pinhole and a lens, a uniform parallel laser beam is obtained. An amplitude-type spatial light modulator (SLM) is placed on the backward propagation path of the laser beam to load the input image.

[0034] In the present invention, the quantization bit depth of the input image is 8 bits (grayscale value range: 0-255), and the model of the amplitude type SLM is selected as the Hes6001 liquid crystal SLM produced by Holoeye of Germany, with a pixel number of 1080*1920, a quantization bit depth of 8 bits, and a pixel spacing of 8 μm. Therefore, the input image can be directly loaded on the amplitude type SLM through software drive. When the laser beam is irradiated on the liquid crystal surface, the amplitude information of the input image will be modulated in the reflected laser beam. Because the number of channels of the subsequent optical convolution layer is set to 10, it is necessary to copy the input image 10 times and tile it on the amplitude type SLM. The size of each image is 256*256 pixels, and the position of each image on the liquid crystal surface corresponds to the position of the microlens in the optical convolution layer.

[0035] (2) Constructing the optical convolution layer

[0036] According to computational optics theory, Fourier lens can perform two-dimensional Fourier transform. After converting the convolution operation into two Fourier transforms and one frequency domain dot multiplication operation, the two-dimensional convolution operation can be realized through an optical 4f (f is the focal length of the lens) system. In order to realize multi-channel parallel convolution, a microlens array is used instead of a single lens to construct an optical 4f system. The microlens array is purchased from Edmund Company. The physical size is a focal length f = 38mm, and the length and width of each microlens are 3mm. The input layer SLM is located at the front focal plane of the first microlens array, and a phase-type SLM is placed at the back focal plane. The model is Pluto-2 liquid crystal SLM produced by Holoeye Company in Germany. The quantization bit depth is 8bit and the pixel interval is 8μm. The phase-type SLM is used to load the convolution kernel image matrix. The size of the convolution kernel image matrix is ​​consistent with the input image size, both of which are 256*256 pixels, and the matrix elements are randomly generated phases (the phase range is 0-2π, and the phase is quantized to 0-255 grayscale values ​​after random generation, and π is pi). After the phase-type SLM is powered on, the phase matrix The modulation is placed on the liquid crystal surface (exp[·] is an exponential operation, j is an imaginary unit, is the phase matrix). The laser beam carrying the input image information passes through the microlens array for Fourier transform and then irradiates on the phase SLM liquid crystal surface to complete the dot product of the input image spectrum and the convolution kernel matrix. The result of the dot product continues to propagate with the reflected coherent beam and passes through the microlens array for Fourier transform again to obtain the multi-channel parallel two-dimensional convolution of the input image and the convolution kernel.

[0037] (3) Constructing an optical nonlinear modulation module

[0038] In order to nonlinearly modulate the output image of the optical convolution layer, a photorefractive crystal SBN:60 is added to the optical architecture to construct an optical nonlinear modulation module to enhance the representation ability of the neural network. Photorefractive crystal is a nonlinear optical material that exhibits nonlinear characteristics when an external voltage is applied. When a laser beam is irradiated on a photorefractive crystal, the output crystal beam will load an additional phase, and the phase size is related to the intensity distribution of the input beam. Under an external voltage of 100V, the additional phase generated when the laser beam passes through a 1mm thick SBN:60 photorefractive crystal can be expressed as:

[0039]

[0040] in represents the additional phase, represents the intensity disturbance of the light beam in space, and π represents the circumference of a circle.

[0041] (4) Constructing the hidden layer of the optoelectronic convolutional neural network

[0042] The "optical convolution layer + optical nonlinear modulation module" is used as a submodule, and multiple submodules are cascaded to construct the hidden layer of the optoelectronic convolutional neural network. The output of the optical convolution layer is used as the input of the optical nonlinear modulation module, and the modulated output image is used as the input of the next optical convolution layer. This scalable architecture facilitates the construction of subsequent complex network models. This patent designs a network model that includes two layers of optical convolution layers and two layers of optical nonlinear modulation. In the hidden layer, the number of phase-type SLM, microlens array and photorefractive crystal SBN:60 used is 2.

[0043] (5) Constructing a global average pooling module

[0044] After the input images of 10 channels are operated by the hidden layer, the information of the feature space needs to be mapped to the sample space in order to predict the target category. This patent implements this mapping operation by constructing a global average pooling module through a microlens array. In geometric optics, the light beam located in front of a spherical lens forms a bright spot on the rear focal plane after being converged by the spherical lens, which is the Airy disk. Combined with the principle of Fourier optics, it can be seen that when the input image is located at the front focal plane of the spherical lens, the Airy disk in the rear focal plane is located at the zero point of the spectrum, and its light intensity is proportional to the modulus sum of each matrix element in the input image. The correspondence between input and output is similar to the global average pooling operation, so a microlens array is selected for multi-channel global average pooling. The average light intensity information of the Airy disk in 10 channels is obtained through the sCMOS camera at the output layer. The target category corresponding to the channel where the Airy disk with the largest light intensity is located is the category prediction result of the optoelectronic convolutional neural network for the input target.

[0045] Step 2: Build an optical system and obtain a speckle image dataset for optical neural network training.

[0046] according to Figure 3 The optical system is constructed according to the optical path diagram shown in the figure. A thin glass diffuser (Thorlabs, DG10-220) is selected as the scattering medium and placed between two objective lenses (f = 40 mm, f = 60 mm). 60,000 data images from the handwritten digit dataset MNIST are selected as input images. The size of each image is 256*256 pixels, and it is expanded to 1080*1920 pixels by zero padding. They are loaded on the amplitude type SLM (Holoeye, Hes6001) one by one. At the output end, the image is collected by an sCMOS camera, and the 256*256 pixel range in the center of the camera is cropped as the actual acquired speckle image. Among the 60,000 speckle images, 50,000 images are selected as the training set of the optoelectronic convolutional neural network, and 10,000 images are selected as the test set, and the labels of each image are annotated.

[0047] Step 3: Train the optical convolutional neural network model on a computer.

[0048] according to Figure 1 The optical path diagram shown, Figure 2 The optoelectronic convolutional neural network architecture shown in the figure is simulated and modeled under the Tensorflow framework. The physical data of all optoelectronic devices will be used as parameters to simulate the construction of the digital model, which is consistent with the optical path structure in the real space, and the angular spectrum propagation method is used to simulate the propagation of the light beam in free space. Each optical convolution layer contains 10 convolution channels, and the size of each convolution channel is 375*375 pixels (the image size of the training set and the test set is 256*256 pixels, which needs to be padded to 375*375 pixels). The convolution kernel phase in the convolution channel is a parameter to be trained. Before training, a phase matrix of 375*375 size (the phase range is 0-2π) needs to be randomly generated. The Python version used for the training code is 3.5.0, using NVIDIA TitanX GPU, and the Adam gradient optimization algorithm is used under the Tensorflow framework. The learning rate is set to 0.001, and 20 rounds of training are performed on all samples, which takes a total of 3 hours, and the classification accuracy reaches 92.6%.

[0049] Table 1 Distribution of training set and test set samples

[0050] category 0 1 2 3 4 5 6 7 8 9 Training set 5923 6742 5958 6131 5842 5421 5918 6265 5851 5949 Test Set 980 1135 1032 1010 982 892 958 1028 974 1009

[0051] Step 4: Follow Figure 1 Build an optical system for the optoelectronic convolutional neural network. Power on the optoelectronic components such as the laser light source, spatial light modulator, nonlinear crystal, and camera. After quantizing the input image and the pre-trained convolution kernel phase to a range of 0-255, load them on the corresponding spatial light modulator. Here, a speckle image of a handwritten number "5" is selected as the input end (see Figure 4 ).

[0052] Step 5: The output of each channel after global average pooling (i.e., the average light intensity of the Airy disk) is obtained through the sCMOS camera in the output layer. The target category corresponding to the channel with the largest Airy disk intensity is the target category predicted by the optoelectronic convolutional neural network. The Airy disk intensity of the 10 channels obtained in the output layer is as follows: Figure 5 The corresponding average light intensity values ​​are shown in the line graph. The average light intensity of the Airy disk of the first channel is the largest, which means that the optoelectronic neural network predicts the category of the input target as "5".

[0053] The contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

[0054] Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications to the technical solution of the present invention by using the methods and technical contents disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for implementing speckle image target classification based on optoelectronic convolutional neural network, characterized in that: include: Constructing an optical convolutional neural network; Construct an optical system for acquiring speckle images and obtain a speckle image data set for optical neural network training; Training the optical convolutional neural network model on a computer; Build the optical system of the optoelectronic convolutional neural network, power on the optoelectronic components such as the laser light source, spatial light modulator, nonlinear crystal, and camera, and load the input image and pre-trained convolution kernel parameters on the spatial light modulator of the optical convolution layer; The sCMOS camera in the output layer obtains the output of each channel after global average pooling to determine the target category.

2. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 1, characterized in that: Construct an optical convolutional neural network, including: (1) Constructing the input layer: Based on the coherent laser source and the Fourier lens, parallel and evenly diffused coherent laser light is generated, and the input image is modulated on the spatial light modulator. When the coherent laser light is irradiated on the surface of the spatial light modulator, the reflected light beam will carry the image information and transmit it backward; (2) Constructing an optical convolution layer: Based on the spatial light modulator and the microlens array, an optical 4f system is formed to realize the parallel two-dimensional convolution operation of multi-channel input images and convolution kernels. The relationship between the number of convolution channels N and the number of target categories C is N = k*C, where k is a non-negative integer; (3) Constructing an optical nonlinear modulation module: A nonlinear activation layer is constructed based on a photorefractive crystal. When a coherent light beam is irradiated on the photorefractive crystal, the light beam output from the crystal will generate an additional phase factor according to the light intensity information, thereby nonlinearly modulating the information carried in the light beam. (4) cascading multiple groups of optical convolutional layers and optical nonlinear modulation modules to form a hidden layer of the optoelectronic convolutional neural network; (5) Construct a global average pooling module: Perform global average pooling on the output of each channel of the hidden layer based on the microlens array, vectorize the image matrix, and make the number of output channels the same as the number of target categories C; identify the category of the input image based on the results of each output channel.

3. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 2 is characterized in that: When constructing the input layer, a laser transmitter is selected as the laser light source of the optoelectronic neural network. After being expanded and collimated by a pinhole and a lens, a uniform and parallel laser beam is obtained. An amplitude-type spatial light modulator is placed on the backward propagation path of the laser beam to load the input image.

4. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 3 is characterized in that: The input image can be directly loaded onto the amplitude-type SLM through software drive. When the laser beam is irradiated on the liquid crystal surface, the reflected laser beam will modulate the amplitude information of the input image. The number of channels of the optical convolution layer is the same as the number of copies of the input image. The input image is tiled on the amplitude-type SLM, and the position of each image on the liquid crystal surface corresponds to the position of the microlens in the optical convolution layer.

5. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 2, characterized in that: When constructing the optical convolution layer, the convolution operation is converted into two Fourier transforms and one frequency domain dot multiplication operation, and then a two-dimensional convolution operation is implemented through an optical system. In order to realize multi-channel parallel convolution, a microlens array is used instead of a single lens to construct the optical system.

6. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 5, characterized in that: The input layer SLM is located at the front focal plane of the first microlens array, and a phase-type SLM is placed at the back focal plane. The phase-type SLM is used to load the convolution kernel image matrix. The size of the convolution kernel image matrix is ​​consistent with the size of the input image, and the matrix elements are randomly generated phases. After the phase-type SLM is powered on, the phase matrix will be modulated on the liquid crystal surface according to the correspondence between the grayscale value and the phase. After the laser beam carrying the input image information undergoes Fourier transform through the microlens array, it is irradiated on the liquid crystal surface of the phase SLM to complete the dot product of the frequency spectrum of the input image and the convolution kernel matrix. The result after the dot product continues to propagate with the reflected coherent light beam, and is Fourier transformed through the microlens array again to obtain a multi-channel parallel two-dimensional convolution of the input image and the convolution kernel.

7. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 2, characterized in that: When constructing the optical nonlinear modulation module, in order to nonlinearly modulate the output image of the optical convolution layer, a photorefractive crystal SBN:60 is added to the optical architecture to construct the optical nonlinear modulation module to enhance the representation capability of the neural network.

8. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 2, characterized in that: When constructing the hidden layer of the optoelectronic convolutional neural network, the optical convolutional layer and the optical nonlinear modulation module are used as a submodule, and the hidden layer of the optoelectronic convolutional neural network is constructed in a cascade manner with multiple submodules.

9. The method for implementing speckle image target classification based on optoelectronic convolutional neural network according to claim 2, characterized in that: When constructing the global average pooling module, the global average pooling module is constructed through the microlens array to map the information in the feature space into the sample space to predict the target category.

10. A computer program product stored on a non-transitory computer readable medium, the computer program product comprising a program code for carrying out the method according to any one of claims 1 to 9.

Images