Single-pixel target identification method based on photoelectric hybrid neural network

Abstract

The invention relates to the cross technical field of artificial intelligence and optical calculation, and discloses a single-pixel target recognition method based on a photoelectric hybrid neural network, which comprises the following steps: S1, an optical coding stage: carrying out dynamic structured illumination on a target scene through a spatial light modulator, the single-pixel detector receives the modulated optical signal and outputs a time sequence voltage signal, and optical end feature compression coding is completed; s2, a photoelectric hybrid calculation stage: inputting the time sequence voltage signal into a photoelectric hybrid neural network; and S3, a dynamic feedback stage. According to the single-pixel target recognition method based on the photoelectric hybrid neural network, in a CIFAR-10 data set test, a closed-loop feedback mechanism enables a system to reach 85.3% recognition accuracy under a single-pixel sampling condition, and the recognition accuracy is improved by 32% compared with a traditional compressed sensing method (such as a single pixel and a support vector machine); the differential adjustment of the optical calculation layer enables the network to be trained from end to end, and supports the online learning of new target categories (the actual measurement incremental learning accuracy attenuation is less than 5%).

Description

Technical Field

[0001] The present invention relates to the field of intersection of artificial intelligence and optical computing, and specifically to a single-pixel target recognition method based on an optoelectronic hybrid neural network. Background Art

[0002] The single-pixel target recognition of the optoelectronic hybrid neural network mainly utilizes the advantages of optical information processing, reduces the amount of calculation and improves the processing speed through optoelectronic hybrid, and obtains infrared image data: first, an infrared detector is used to obtain the target image, and it is decomposed into two steps: obtaining the infrared image and generating motion blur.

[0003] Acquire infrared image data through template parameter design; generate motion blurred image: input infrared image data into the blur generation algorithm to generate motion blurred image data; this step simulates the blur effect in the actual scene by simulating infrared blurred image data; target grayscale value acquisition: obtain the grayscale value of the target, and output the infrared blurred image data as infrared radiation signal time series data; hybrid neural network model processing: input the infrared radiation sequence data set into the 1D-CNN (one-dimensional convolutional neural network) + Bi-LSTM (bidirectional long short-term memory network) hybrid neural network model; the model improves the precision and accuracy of target recognition through iterative training; recognition result output: the network model outputs the recognition accuracy curve and loss curve according to the number of iterations, and finally obtains the target recognition result.

[0004] According to the method, device and equipment for constructing a point cloud map mentioned in the invention patent with Chinese patent application number 202011154289.4, the method, device and equipment for constructing a point cloud map, when used, address the technical problem of low efficiency in constructing a point cloud map in the rotating drum technology. However, the method, device and equipment for constructing a point cloud map rely on electronic calculations for feature extraction when used, and the optical part is only used for forward propagation, which fails to give full play to the advantages of optoelectronic synergy.

[0005] In addition, the current mainstream target recognition methods mainly rely on pure digital neural networks (such as CNN) on electronic computing platforms, which require multi-dimensional pixel data to be obtained through high-resolution image sensors. This architecture has two inherent defects: data redundancy problem: more than 99% of the sampled pixels have no substantial contribution to the recognition task, resulting in a huge waste of computing resources; real-time bottleneck: the serial process of optical imaging → analog-to-digital conversion → digital processing introduces at least millisecond-level delays, which makes it difficult to meet the microsecond-level response requirements of scenarios such as autonomous driving. Therefore, it is necessary to propose a single-pixel target recognition method based on an optoelectronic hybrid neural network to solve the above-mentioned problems. Summary of the Invention

[0006] (1) Technical problems solved

[0007] In response to the shortcomings of the existing technology, the present invention provides a single-pixel target recognition method based on an optoelectronic hybrid neural network, which has the advantages of improving recognition accuracy and adaptive learning ability, and solves the problems of limited recognition accuracy and adaptive learning ability of comparative documents in the background technology.

[0008] (2) Technical solution

[0009] To achieve the above object, the present invention provides the following technical solution: a single-pixel target recognition method based on an optoelectronic hybrid neural network, comprising the following steps:

[0010] S1. Optical encoding stage: Dynamically structured illumination of the target scene is performed using a spatial light modulator. A single-pixel detector receives the modulated light signal and outputs a time-series voltage signal, completing optical feature compression encoding.

[0011] S2. Photoelectric hybrid calculation stage: Input the time series voltage signal into the photoelectric hybrid neural network, where:

[0012] The optical computing layer performs tunable matrix multiplication operations based on Mach-Zehnder interferometers to achieve optical domain feature extraction;

[0013] The electronic computing layer uses a pulse neural network to perform spatiotemporal adaptive learning of optical domain features;

[0014] S3. Dynamic feedback stage: According to the classification results of the electronic computing layer, the coding mode of the spatial light modulator and the phase modulation parameters of the optical computing layer are reversely adjusted to form a closed-loop optimization system.

[0015] Preferably, the dynamic structured lighting in step S1 adopts any of the following coding strategies:

[0016] Adaptive Hadamard matrix coding based on target prior knowledge;

[0017] Random projection coding based on convolution kernel sparsification.

[0018] Preferably, the updating frequency of the adaptive Hadamard matrix coding is negatively correlated with the confidence of the electronic computing layer, and the coding mode reconfiguration is triggered when the classification confidence is lower than a threshold.

[0019] Preferably, the optical computing layer comprises a cascade of:

[0020] a first interferometer array for performing a differentiable optical convolution operation;

[0021] The second interferometer array is used to realize optical simulation of the nonlinear activation function.

[0022] Preferably, the weight matrix of the optical convolution operation is dynamically adjusted by the electro-optical effect of the lithium niobate waveguide, and the adjustment gradient is generated by the error back-propagation signal of the electronic computing layer.

[0023] Preferably, the electronic computing layer adopts a pulse timing dependent plasticity (STDP) learning rule, and its input pulse sequence is generated by a light intensity-pulse frequency conversion module.

[0024] Preferably, the closed-loop optimization in step S3 specifically includes:

[0025] The spatial light modulator coding pattern and the optical computing layer parameters are jointly optimized by the gradient descent method;

[0026] The optimization objective function is the Pareto front solution of recognition accuracy and light energy consumption.

[0027] Preferably, the optical encoding stage, the optoelectronic hybrid calculation stage and the dynamic feedback stage respectively include a dynamic illumination module, an optical calculation module and an electronic calculation module.

[0028] Preferably, the dynamic illumination module comprises a digital micromirror device (DMD) and a tunable laser source;

[0029] The optical computing module is composed of a programmable interferometer array implemented by a silicon-based photonic integrated circuit; the electronic computing module integrates a pulse neural network chip based on a memristor.

[0030] (3) Beneficial effects

[0031] Compared with the existing technology, the present invention provides a single-pixel target recognition method based on an optoelectronic hybrid neural network, which has the following beneficial effects:

[0032] 1. This single-pixel target recognition method based on an optoelectronic hybrid neural network compresses traditional megapixel-level data acquisition to less than one thousandth through the collaborative encoding of single-pixel detectors and dynamic structured lighting (actually only 256 samples were required on the MNIST dataset), reducing the optical front-end computing energy consumption by two orders of magnitude. In addition, the optical convolution layer in the optoelectronic hybrid neural network only requires 0.1 nanojoules to perform 4×4 matrix multiplication (Nature Photonics benchmark comparison), which is 100 times more energy efficient than equivalent electronic computing.

[0033] 2. This single-pixel target recognition method based on an optoelectronic hybrid neural network was tested on the CIFAR-10 dataset. The closed-loop feedback mechanism enabled the system to achieve an 85.3% recognition accuracy under single-pixel sampling conditions, a 32% improvement over traditional compressed sensing methods (such as single pixel + support vector machine). The differentiable adjustment of the optical computing layer enables the network to be trained end-to-end and supports online learning of new target categories (the measured incremental learning accuracy attenuation is <5%).

[0034] 3. This single-pixel target recognition method based on an optoelectronic hybrid neural network is compatible with existing fiber-optic communication infrastructure by adopting an optical computing module (1550nm band) using standard silicon-based photonics technology. It is particularly suitable for imaging in scarce bands such as infrared / terahertz; the dynamic encoding strategy supports multi-modal recognition switching. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural flow chart of the single-pixel target recognition method of the present invention;

[0036] Figure 2 Schematic diagram of the optical computing layer structure of the present invention. DETAILED DESCRIPTION

[0037] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0038] See also Figure 1-2 , a single-pixel target recognition method based on optoelectronic hybrid neural network, comprising the following steps:

[0039] S1. Optical encoding stage: Dynamically structured illumination of the target scene is performed using a spatial light modulator. A single-pixel detector receives the modulated light signal and outputs a time-series voltage signal, completing optical feature compression encoding.

[0040] S2. Photoelectric hybrid calculation stage: Input the time series voltage signal into the photoelectric hybrid neural network, where:

[0041] The optical computing layer performs tunable matrix multiplication operations based on Mach-Zehnder interferometers to achieve optical domain feature extraction;

[0042] The electronic computing layer uses a pulse neural network to perform spatiotemporal adaptive learning of optical domain features;

[0043] S3. Dynamic feedback stage: According to the classification results of the electronic computing layer, the coding mode of the spatial light modulator and the phase modulation parameters of the optical computing layer are reversely adjusted to form a closed-loop optimization system.

[0044] In case example 1: dynamic Hadamard coding and optoelectronic joint training:

[0045] (1) Optical encoder configuration: Texas Instruments DLP6500 DMD is used to implement adaptive Hadamard encoding, with an initial pattern of 64×64 basis matrix; Thorlabs PDA100A2 silicon photodiode is used as the single-pixel detector with a sampling frequency of 1MHz; (2) Optical-electrical hybrid network construction; (3) Closed-loop training process: Forward propagation: optical encoding → single-pixel sampling → optical matrix multiplication → electronic pulse classification; Loss calculation: Cross-entropy loss is used to superimpose the light energy consumption penalty term (formula: L = L_CE + λ·Σ|E_optical|); Backward propagation: The optical layer gradient is calculated through a differentiable MZI model, and the DMD encoding pattern and interferometer weights are jointly updated. Finally, under the condition of using only 0.5% pixel data, the aircraft target recognition accuracy of 1024×1024 pixels reaches 89.7%, which is 12 times faster than that of a pure electronic network.

[0046] In case example 2: multi-band migration identification system:

[0047] Hardware configuration: tunable laser source (wavelength range 800-1600nm), lithium niobate waveguide integrated 4×4 MZI array (3dB insertion loss).

[0048] The multi-band migration workflow is as follows: visible light mode: using 532nm laser illumination, storing the optimal encoding matrix during the training phase; infrared switching: when switching to 1064nm wavelength: loading the pre-stored encoding template;

[0049] Wavelength dispersion is compensated for by the optical layer automatic calibration module (Δβ compensation algorithm); recognition verification: infrared and visible light images of the same target maintain >92% feature consistency.

[0050] In the third case study, through the synergy of spatial dynamic modulation and single-pixel sampling in the optical encoding stage (S1), only 0.3% of the original pixel data volume is required on the MNIST dataset (traditional methods require 100%), achieving sampling-computation integration and increasing data throughput by 50 times (actual comparison between the TIDLP development board and the OV5640 sensor); and in the optoelectronic hybrid computing stage (S2), the optical interferometer matrix multiplication takes only 1.2ns / time, while the electronic pulse neural network classification takes 8μs, forming a layered processing architecture of "light-speed computing + electronic decision-making", and the system latency is reduced by 76% compared with the purely electronic solution; at the same time In the closed-loop feedback stage (S3), recognition robustness is improved by 41% in target deformation scenarios (such as vehicle turning) by adjusting the DMD coding mode and MZI phase parameters; the adaptive Hadamard coding achieves a signal-to-noise ratio of 28dB in low-light environments (<10lux), which is 15dB higher than random projection coding, and the confidence-triggered coding reconfiguration reduces energy consumption by 37%; the response speed of lithium niobate waveguide electro-optical modulation reaches 10GHz, supporting per-frame updates of optical convolution kernel weights (30fps video stream processing); the STDP learning rule enables the pulse neural network to have a 19% higher recognition accuracy than the traditional BP network in single-sample learning scenarios.

[0051] In summary, this single-pixel target recognition method based on optoelectronic hybrid neural network compresses traditional megapixel-level data acquisition to less than one thousandth through the collaborative encoding of single-pixel detectors and dynamic structured lighting (only 256 samples are required on the MNIST dataset), reducing the optical front-end computing energy consumption by two orders of magnitude; and the optical convolution layer in the optoelectronic hybrid neural network only requires 0.1 nanojoules to implement 4×4 matrix multiplication (Nature Photonics benchmark comparison), which is 100 times more energy efficient than equivalent electronic computing.

[0052] Moreover, in the CIFAR-10 dataset test, the closed-loop feedback mechanism enabled the system to achieve an 85.3% recognition accuracy under single-pixel sampling conditions, an improvement of 32% over traditional compressed sensing methods (such as single pixel + support vector machine); the differentiable adjustment of the optical computing layer enables the network to be trained end-to-end and supports online learning of new target categories (the measured incremental learning accuracy attenuation is <5%).

[0053] Moreover, the optical computing module (1550nm band) using standard silicon-based photonics technology is compatible with existing fiber-optic communication infrastructure and is particularly suitable for imaging in scarce bands such as infrared / terahertz; the dynamic coding strategy supports multi-modal recognition switching.

[0054] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0055] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A single-pixel target recognition method based on an optoelectronic hybrid neural network is characterized in that: The following steps are involved: S1. Optical encoding stage: Dynamically structured illumination of the target scene is performed using a spatial light modulator. A single-pixel detector receives the modulated light signal and outputs a time-series voltage signal, completing optical feature compression encoding. S2. Photoelectric hybrid calculation stage: Input the time series voltage signal into the photoelectric hybrid neural network, where: The optical computing layer performs tunable matrix multiplication operations based on Mach-Zehnder interferometers to achieve optical domain feature extraction; The electronic computing layer uses a pulse neural network to perform spatiotemporal adaptive learning of optical domain features; S3. Dynamic feedback stage: According to the classification results of the electronic computing layer, the coding mode of the spatial light modulator and the phase modulation parameters of the optical computing layer are reversely adjusted to form a closed-loop optimization system.

2. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 1 is characterized in that: The dynamic structured lighting in step S1 adopts any of the following encoding strategies: Adaptive Hadamard matrix coding based on target prior knowledge; Random projection coding based on convolution kernel sparsification.

3. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 2, characterized in that: The updating frequency of the adaptive Hadamard matrix coding is negatively correlated with the confidence of the electronic computing layer, and the coding mode reconfiguration is triggered when the classification confidence is lower than a threshold.

4. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 2, characterized in that: The optical computing layer comprises a cascade of: a first interferometer array for performing a differentiable optical convolution operation; The second interferometer array is used to realize optical simulation of the nonlinear activation function.

5. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 4 is characterized in that: The weight matrix of the optical convolution operation is dynamically adjusted through the electro-optical effect of the lithium niobate waveguide, and the adjustment gradient is generated by the error back-propagation signal of the electronic computing layer.

6. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 1, characterized in that: The electronic computing layer adopts a pulse timing dependent plasticity (STDP) learning rule, and its input pulse sequence is generated by a light intensity-pulse frequency conversion module.

7. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 1, characterized in that: The closed-loop optimization in step S3 specifically includes: The spatial light modulator coding pattern and the optical computing layer parameters are jointly optimized by the gradient descent method; The optimization objective function is the Pareto front solution of recognition accuracy and light energy consumption.

8. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 1, characterized in that: The optical encoding stage, the optoelectronic hybrid calculation stage and the dynamic feedback stage respectively include a dynamic lighting module, an optical calculation module and an electronic calculation module.

9. The single-pixel target recognition method based on optoelectronic hybrid neural network according to claim 8, characterized in that: The dynamic lighting module includes a digital micromirror device (DMD) and a tunable laser source; the optical computing module is composed of a programmable interferometer array implemented by a silicon-based photonic integrated circuit; and the electronic computing module integrates a pulse neural network chip based on a memristor.

Images