Pre-sensory computing device and method based on multilayer optical neural network

By using multi-layer optical neural network in visual image processing, convolution calculation and nonlinear activation using optical mask and quantum dot thin film layers, the problem of high power consumption and slow speed of visual image processing in the prior art is solved, and the problem of high and slow power consumption in the prior art is realized. Low-power, high-efficiency visual image processing, suitable for high-speed dynamic scenes.

CN119005276APending Publication Date: 2024-11-22TSINGHUA UNIVERSITY

Patent Information

Application Number
CN202410953463.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the prior art, visual image processing has problems of high power consumption and slow speed, especially in high-speed dynamic scenarios, it is difficult to meet the response speed requirements of milliseconds or even microseconds.

Method used

Adopting a pre-sensory computing device based on multi-layer optical neural network, including a pre-sensory optical neural network module, a board-level photosensitive chip and a back-end electrical neural network module, convolutional calculation and nonlinear activation are performed through optical mask plate and quantum dot film layer. , reduce system power consumption and delay, and realize low-power consumption and high-efficiency visual image processing.

Benefits of technology

The calculations originally performed on electricity are transferred to light, reducing system power consumption and delay, improving the efficiency and response speed of visual image processing, and are suitable for autonomous driving and other high-demand applications in high-speed dynamic scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005276A_ABST
    Figure CN119005276A_ABST
Patent Text Reader

Abstract

The invention provides a pre-sensory computing device and method based on a multilayer optical neural network. The device comprises a pre-sensory optical neural network module, a board-level photosensitive chip and a rear-end electrical neural network module. The presensory optical neural network module comprises a first optical mask plate, a quantum dot thin film layer and a second optical mask plate; the optical mask plate is used as a convolution layer to carry out pre-sensing convolution calculation; the quantum dot film layer is used as a nonlinear activation layer to perform nonlinear activation between convolution layers; the board-level photosensitive chip is used for receiving a pre-sensing calculation result of the optical neural network; and the rear-end electrical neural network module is used as a linear full connection layer to complete a target classification task according to the pre-sensing calculation result to obtain a final sensing processing result. According to the invention, the multi-layer optical neural network is established based on two layers of mask plates and one layer of quantum dot film, and the calculation originally carried out on electricity is transferred to be carried out on light, so that the power consumption and time delay of the system are reduced, and low-power-consumption and high-efficiency visual image processing is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of optical perception computing technology, and in particular to a pre-perception computing device and method based on a multi-layer optical neural network. Background Art

[0002] In the era of the Internet of Things, countless sensors are embedded in various devices such as mobile communication terminals, smart wearable devices, automobiles and industrial machines. Visual image sensors are key devices for the intelligent society. However, with the continuous expansion of applications, higher requirements are also put forward for the system power consumption, response speed, safety performance and other aspects of sensors. For example, in high-speed dynamic scenarios such as autonomous driving and high-speed motion behavior analysis, millisecond or even microsecond response speeds are usually required. In the traditional "sensing, transmission and computing" link, the three processes of imaging, perception and processing are separated from each other, resulting in a large amount of information redundancy. At the same time, there is a large amount of data movement in the hardware based on the von Neumann architecture, which limits the system power consumption and speed.

[0003] In summary, the existing technology has the problems of high power consumption and slow speed. Summary of the invention

[0004] The present invention provides a pre-sensory computing device and method based on a multi-layer optical neural network, which is used to solve the defects of high power consumption and slow speed of visual image processing in the prior art, and realize low-power and high-efficiency visual image processing.

[0005] The present invention provides a pre-sensing computing device based on a multi-layer optical neural network, comprising: Pre-sensing optical neural network module, board-level photosensitive chip and back-end electrical neural network module; The pre-sensing optical neural network module includes a first optical mask, a quantum dot film layer and a second optical mask; a convolution kernel is arranged on the optical mask, which is used as a convolution layer to perform pre-sensing convolution calculation; the quantum dot film layer is arranged between the first optical mask and the second optical mask, which is used as a nonlinear activation layer to perform nonlinear activation between convolution layers; The board-level photosensitive chip is used to receive the pre-sensing calculation results of the optical neural network; The back-end electrical neural network module is used as a linear fully connected layer to complete the target classification task according to the pre-sensing calculation result to obtain the final perception processing result; the first optical mask, the quantum dot thin film layer, the second optical mask and the board-level photosensitive chip are arranged in sequence according to a preset interlayer distance, and the board-level photosensitive chip is connected to the back-end electrical neural network module.

[0006] According to a pre-sensing computing device based on a multi-layer optical neural network provided by the present invention, the convolution kernel on the optical mask is obtained by processing a glass substrate after cadmium film plating according to the requirements of the target classification task.

[0007] According to a pre-sensing computing device based on a multi-layer optical neural network provided by the present invention, the inter-layer distance is obtained by correction based on the triangle similarity theory and the characteristic size of the convolution kernel, using the relationship between the spacing between the plate-level photosensitive chip, the optical mask, the quantum dot film layer, and the input light source and the size of the input light source; Among them, the inter-layer distance includes a first distance between the input light source and the first optical mask, a second distance between the first optical mask and the quantum dot film layer, a third distance between the quantum dot film layer and the second optical mask, and a fourth distance between the second optical mask and the board-level photosensitive chip.

[0008] According to a pre-sensing computing device based on a multi-layer optical neural network provided by the present invention, the quantum dot film layer is a CdSe quantum dot film.

[0009] According to a pre-sensing computing device based on a multi-layer optical neural network provided by the present invention, the spatial characteristic size of the convolution kernel of the first optical mask is 120μm, and the spatial characteristic size of the convolution kernel of the second optical mask is 50um.

[0010] The present invention also provides a pre-sensing calculation method based on a multi-layer optical neural network, which is applied to any of the above-mentioned devices and comprises the following steps: Performing interlayer distance correction on the pre-sensing computing device based on the multi-layer optical neural network, and fixing the device after the interlayer distance correction; Acquire incident light emitted by an input light source, perform pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and acquire the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; Based on the back-end electrical neural network module, the target classification task is completed according to the pre-sensing calculation results to obtain the final perception processing results.

[0011] According to a pre-sensing computing method based on a multi-layer optical neural network provided by the present invention, inter-layer distance correction is performed on the pre-sensing computing device based on the multi-layer optical neural network, specifically comprising: Placing the second optical mask as close as possible to the board-level photosensitive chip; Fixing the positions of the modules of the pre-sensing computing device based on the multi-layer optical neural network; Turn on a plurality of input point light sources, and calculate the spacing between the input point light sources based on incident light rays of the input point light sources according to triangle similarity theory; According to the pixel size of the input point light source and the characteristic size of the two-layer convolution kernel, the inter-layer distance is calculated using the relationship between the pixel size of the point spread function detected by the board-level photosensitive chip and the size of the input point light source; The pre-sensing computing device based on the multi-layer optical neural network is adjusted according to the inter-layer distance.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a pre-sensing calculation method based on a multilayer optical neural network as described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described pre-sensing calculation methods based on a multilayer optical neural network.

[0014] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described pre-sensing calculation methods based on a multi-layer optical neural network.

[0015] The present invention provides a pre-sensing computing device and method based on a multi-layer optical neural network, including a pre-sensing optical neural network module, a board-level photosensitive chip and a back-end electrical neural network module; the pre-sensing optical neural network module includes a first optical mask, a quantum dot film layer and a second optical mask; a convolution kernel is arranged on the optical mask, which is used as a convolution layer to perform pre-sensing convolution calculation; the quantum dot film layer is arranged between the first optical mask and the second optical mask, and is used as a nonlinear activation layer to perform nonlinear activation between convolution layers; the board-level photosensitive chip is used to receive the pre-sensing calculation results of the optical neural network; the back-end electrical neural network module is used to serve as a linear fully connected layer to complete the target classification task according to the pre-sensing calculation results to obtain the final perception processing results; the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip are arranged in sequence according to a preset inter-layer distance, and the board-level photosensitive chip is connected to the back-end electrical neural network module. The present invention builds a multi-layer optical neural network based on two layers of masks and a layer of quantum dot film, modulates the amplitude of incident light through the optical mask, completes convolution calculation in space, and reduces the system complexity of optical calculation; at the same time, optical nonlinearity is added between layers through the quantum dot film layer, which can increase the fitting ability of the optical neural network to further enrich the pre-sensing calculation function. The processing results of the pre-sensing calculation are obtained through the board-level photosensitive chip, and the subsequent layers of the neural network are processed through the back-end electrical neural network module. The present invention transfers the calculation originally performed on electricity to light, reduces system power consumption and delay, and realizes low-power and high-efficiency visual image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 This is one of the structural schematic diagrams of the pre-sensing computing device based on the multi-layer optical neural network provided by the present invention.

[0018] Figure 2 This is the second structural schematic diagram of the pre-sensing computing device based on the multi-layer optical neural network provided by the present invention.

[0019] Figure 3 It is a schematic diagram of interlayer distance correction of a pre-sensing computing device based on a multi-layer optical neural network provided by the present invention.

[0020] Figure 4It is a schematic diagram of the activation and nonlinear response characteristics of quantum dots in the quantum dot film layer of the pre-sensing computing device based on the multi-layer optical neural network provided by the present invention.

[0021] Figure 5 This is the third structural schematic diagram of the pre-sensing computing device based on the multi-layer optical neural network provided by the present invention.

[0022] Figure 6 This is one of the flow charts of the pre-sensing calculation method based on the multi-layer optical neural network provided by the present invention.

[0023] Figure 7 This is the second flow chart of the pre-sensing calculation method based on the multi-layer optical neural network provided by the present invention.

[0024] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] In order to solve the problems of the prior art, the system needs to have a certain data processing capability at the near-sensing end, transfer part of the computing tasks from the cloud computing center to the edge device, reduce energy consumption and time delay, save communication bandwidth and enhance data security and privacy. According to the location of the computing unit, the near-sensing end computing can be divided into three paradigms: pre-sensing computing, intra-sensing computing and near-sensing computing. Compared with the design and modification of the chip for intra-sensing computing and near-sensing computing, pre-sensing optical computing can transfer the calculation originally performed electrically to light through the design of optical elements. At this time, the optical transmission process completes certain computing tasks instead of imaging, and upgrades the "imaging-centric" solution to "computing-centric", realizing functions such as data compression and feature extraction. Based on this, the present invention provides a pre-sensing computing device and method based on a multi-layer optical neural network.

[0027] Combine the following Figure 1-Figure 5 Describe the pre-sensory computing device based on a multi-layer optical neural network of the present invention, Figure 1 is one of the structural schematic diagrams of the pre-sensing computing device based on the multi-layer optical neural network provided by the present invention, such as Figure 1 As shown, the device includes a front-sensing optical neural network module 110, a board-level photosensitive chip 120 and a back-end electrical neural network module 130.

[0028] The pre-sensing optical neural network module 110 includes a first optical mask, a quantum dot film layer and a second optical mask; the optical mask is provided with a convolution kernel, which is used as a convolution layer to perform pre-sensing convolution calculation; the quantum dot film layer is provided between the first optical mask and the second optical mask, which is used as a nonlinear activation layer to perform nonlinear activation between convolution layers. The board-level photosensitive chip 120 is used to receive the pre-sensing calculation results of the optical neural network.

[0029] It should be noted that the optical mask mentioned in the present invention is formed by an opaque light-shielding film on a transparent substrate to form a mask pattern structure, and 01 convolution calculation is realized by shielding the incident light. The mask manufacturing process includes forming a pattern structure on a glass / quartz substrate, a chrome layer and a photoresist layer. The present invention performs convolution calculation in space through a lensless spatial convolution architecture based on a planar optical element such as an optical mask (Mask), and uses a lensless architecture based on a planar optical element to modulate the amplitude of the incident light.

[0030] It should be noted that the quantum dots (QDs) in the quantum dot film layer are nano-scale crystals. Under light irradiation, the electrons in the quantum dots effectively absorb the energy of the incident photons, resulting in a transition from the conduction band to the valence band. If the electrons in the conduction band transition back to the valence band and emit photons, the phenomenon of quantum dot luminescence will be observed. In the photoluminescence process, the composite luminescence is basically composed of three different forms: direct composite luminescence of electron-hole pairs, luminescence generated by surface defects, and luminescence related to impurity levels. They compete with each other, and due to the presence of surface defect states, the inherent nonlinear properties of quantum dot luminescence are caused. The present invention adds optical nonlinearity through the quantum dot film (QDs film) to realize the nonlinear function between layers. In the actual operation process, the incident light emitted by the input light source is detected by the board-level photosensitive chip 120 after being pre-calculated by the multi-layer optical neural network processor.

[0031] That is to say, the pre-sensing optical neural network module 110 composed of the first optical mask, the quantum dot film layer and the second optical mask is a multi-layer optical neural network processor (MONN), and the output of the multi-layer optical neural network processor is detected by the board-level photosensitive chip 120.

[0032] It should be emphasized that the compact structure based on masks and quantum dots can reduce the system structure complexity of optical computing, and the nonlinear functions between layers can increase the fitting ability of optical neural networks to further enrich the computing functions of the pre-sensing system.

[0033] It should be noted that the pre-sensing optical neural network module 110 and the board-level photosensitive chip 120 are arranged in sequence along the direction of the incident light. In more detail, the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip 120 are arranged in sequence according to a preset interlayer distance.

[0034] In a specific embodiment, the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip 120 are arranged in sequence and arranged perpendicular to the test bench, such as Figure 5 shown.

[0035] In addition, in order to further fix the positions of the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip 120, the device also includes a bracket for fixing the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip 120.

[0036] Further, in some embodiments, the input light source may be an LED array, such as Figure 2 and Figure 5 shown.

[0037] The present invention does not impose any specific restrictions on the board-level photosensitive chip, and a specific sensor can be selected according to actual conditions. In some embodiments, the board-level photosensitive chip can be a CMOS sensor.

[0038] The present invention builds a multi-layer optical neural network based on two layers of optical masks and a layer of quantum dot film. The two layers of masks perform two convolutions respectively, transferring the calculation originally performed on electricity to light. The whole system is compact and passive. Since optical computing has the advantages of high speed, high bandwidth, high efficiency and low power consumption, the present invention can greatly improve power consumption and speed.

[0039] Furthermore, the back-end electrical neural network module 130 is used as a linear fully connected layer to complete the target classification task according to the pre-sensing calculation result to obtain the final perception processing result.

[0040] It should be emphasized that the result detected on the chip is the calculation result of the incident light of the object after passing through the multi-layer convolutional neural network, and then input into the digital neural network in the subsequent back-end electrical neural network module 130 to obtain the final intelligent perception processing result.

[0041] It should be noted that the back-end electrical neural network module 130 is set in the computer to act as a fully connected layer, so that the entire device can complete low-level tasks such as image reconstruction and high-level tasks such as classification, recognition, and detection. Furthermore, the back-end electrical neural network module can be a pre-trained model or other image processing algorithms, and the present invention does not limit this.

[0042] In addition, it should be emphasized that the back-end electrical neural network module 130 is connected to the board-level photosensitive chip 120. In some embodiments, the back-end electrical neural network module 130 is electrically connected to the board-level photosensitive chip 120.

[0043] The following is an explanation of the convolution kernel of the pre-sensing optical neural network module 110. The convolution kernel on the optical mask is obtained by processing a glass substrate after cadmium film plating according to the requirements of the target classification task.

[0044] Specifically, when performing target classification tasks, the convolution kernel needs to be reverse optimized according to the specific content of the target classification task, so as to achieve the functions of object reconstruction, classification, recognition, detection, etc. It can be understood that in digital convolutional neural networks, two key hyperparameters control the convolution operation: the size of the convolution kernel and the convolution step size. The convolution kernel size indicates the length of the convolution filter, while the step size indicates the incremental movement of the convolution process.

[0045] In the specific implementation process, the size of the convolution kernel and the convolution step length are determined according to the target classification task, and a 0.1 binary convolution kernel is produced on a cadmium-coated glass substrate according to the determined size of the convolution kernel and the convolution step length. According to the convolution step length, the position of each device in the system can be appropriately adjusted. Furthermore, the mask based on the coating technology can also be replaced by other modulation devices, such as a liquid crystal spatial light modulator, to modulate the amplitude of the incident light.

[0046] In a specific embodiment, the thickness of the glass substrate is 1 mm.

[0047] Furthermore, in some embodiments, the chip area can be fully utilized to realize space division multiplexing, and a single convolution kernel on a layer of optical mask can be expanded into multiple convolution kernels, that is, multiple different convolution kernels are etched on the mask, thereby realizing parallel calculation of multiple convolution kernels on the light, improving the calculation speed and task recognition accuracy, and further reducing the size and power consumption of the system.

[0048] The following describes an important arrangement principle, i.e., the interlayer distance, for arranging the first optical mask, the quantum dot film layer, the second optical mask, and the board-level photosensitive chip 120. The interlayer distance is obtained by correction based on the triangle similarity theory and the characteristic size of the convolution kernel, using the relationship between the spacing between the board-level photosensitive chip, the optical mask, the quantum dot film layer, and the input light source and the size of the input light source; Among them, the inter-layer distance includes a first distance between the input light source and the first optical mask, a second distance between the first optical mask and the quantum dot film layer, a third distance between the quantum dot film layer and the second optical mask, and a fourth distance between the second optical mask and the board-level photosensitive chip.

[0049] First of all, it should be noted that the spatial size of the convolution kernel on the mask and the distance relationship between the input light source, mask, quantum dot film, and chip can be calibrated to achieve specific convolution calculations and nonlinear mapping. Specifically, the object can be regarded as composed of multiple point light sources. In order to realize the mathematical convolution process, the system needs to be calibrated. The correction content is mainly the spacing between multiple layers. The first distance, second distance, third distance, and fourth distance are represented as d1, d2, d3, and d4 respectively. They must meet specific relationship constraints to achieve the expected convolution calculation operation.

[0050] The correction method of d1, d2, d3 and d4 is: put the second mask and the chip as close as possible, calculate the distance between the light sources according to the triangle similarity theory in a fixed optical system, substitute the characteristic size of the two layers of convolution kernels, and calculate the interlayer distance. On this basis, fine-tune the optical system to achieve it. The calculation process based on the triangle similarity theory is not described in detail in this invention.

[0051] In a specific embodiment, we manually set the kernel size of two specific convolutional layers to 5x5 to correct the inter-layer distance. In this case, for the two point light sources in the lighting image, the pixel size of the spacing between the centers of the point spread function (PSF) detected on the corresponding chip should be 4 times the feature size. If the conditions are met, the correction process is completed.

[0052] The present invention realizes multiple parallel convolution calculations on light through multi-layer cascaded optical masks, realizes all-optical inter-layer nonlinear activation through quantum dot materials, and obtains the processing results of pre-sensing calculations on the board-level photosensitive chip. Different convolution kernels designed through task reverse optimization can realize functions such as object reconstruction, classification, recognition, and detection. This device has a compact structure and can achieve a 1-2 order of magnitude reduction in optical length compared to other optical neural networks based on lens systems. At the same time, it can transfer convolution operations with large computational complexity from electrical to optical, reducing system power consumption and latency.

[0053] In some embodiments, the quantum dot film layer is a CdSe quantum dot film.

[0054] Specifically, Figure 4As shown, this embodiment uses CdSe quantum dot film to achieve all-optical nonlinear activation. The quantum dot material can convert blue light (405 nm) into green light (512 nm) with a light intensity threshold of 0.64 mW / cm².

[0055] It is understandable that in actual use, the luminescence characteristics can only be observed when the input light intensity exceeds the threshold, and its nonlinear response is as follows Figure 4 As shown in Figure 1, the response is similar to the ReLU function in digital neural networks. The response and reset time of quantum dots are both in the nanosecond level, making them very suitable as materials for optical computing, enhancing real-time processing capabilities.

[0056] In a specific embodiment, the spatial feature size of the convolution kernel of the first optical mask is 120 μm, and the spatial feature size of the convolution kernel of the second optical mask is 50 um.

[0057] The present invention provides a pre-sensing computing device based on a multi-layer optical neural network, comprising a pre-sensing optical neural network module, a board-level photosensitive chip and a back-end electrical neural network module; the pre-sensing optical neural network module comprises a first optical mask, a quantum dot film layer and a second optical mask; a convolution kernel is arranged on the optical mask, which is used as a convolution layer to perform pre-sensing convolution calculation; the quantum dot film layer is arranged between the first optical mask and the second optical mask, and is used as a nonlinear activation layer to perform nonlinear activation between convolution layers; the board-level photosensitive chip is used to receive the pre-sensing computing results of the optical neural network; the back-end electrical neural network module is used to serve as a linear fully connected layer to complete the target classification task according to the pre-sensing computing results to obtain the final perception processing results; the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip are arranged in sequence according to a preset inter-layer distance, and the board-level photosensitive chip is connected to the back-end electrical neural network module. The present invention builds a multi-layer optical neural network based on two layers of masks and a layer of quantum dot film, modulates the amplitude of incident light through the optical mask, completes convolution calculation in space, and reduces the system complexity of optical calculation; at the same time, optical nonlinearity is added between layers through the quantum dot film layer, which can increase the fitting ability of the optical neural network to further enrich the pre-sensing calculation function. The processing results of the pre-sensing calculation are obtained through the board-level photosensitive chip, and the subsequent layers of the neural network are processed through the back-end electrical neural network module. The present invention transfers the calculation originally performed on electricity to light, reduces system power consumption and delay, and realizes low-power and high-efficiency visual image processing.

[0058] The following is a description of the pre-sensing computing method based on a multi-layer optical neural network provided by the present invention. The pre-sensing computing method based on a multi-layer optical neural network described below and the pre-sensing computing device based on a multi-layer optical neural network described above can be referred to each other. Figure 6 As shown, the method comprises the following steps: Step 610: performing interlayer distance correction on the pre-sensing computing device based on the multi-layer optical neural network, and fixing the device after the interlayer distance correction; Step 620: Obtain incident light emitted by an input light source, perform pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and obtain the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; Step 630: Based on the back-end electrical neural network module, the target classification task is completed according to the pre-sensing calculation result to obtain the final perception processing result.

[0059] Specifically, in the actual operation process, the convolution layer, nonlinear activation between layers, and an electrically linear fully connected layer of the pre-sensing computing device based on a multi-layer optical neural network are first determined. For example, in a specific embodiment, the spatial feature size of the convolution kernel on the mask is selected to be 120μm (first layer) and 50um (second layer), and the network structure selects two optical single convolution kernel convolution layers and nonlinear activation between layers, as well as an electrically linear fully connected layer, and the convolution kernel is optimized in reverse for the task.

[0060] After that, the device is calibrated for the convolution process. It should be explained that the calibration of the convolution process mainly includes the adjustment of the inter-layer distance. After the calibration is completed, the optical system is fixed, and the convolution results of different images are collected under the pre-trained model, and different tasks are completed through the fully connected layer on the electrical side.

[0061] In a specific embodiment, the collected feature map and the point spread function (PSF) of the system can be used to achieve preliminary reconstruction of the image based on the Wiener filtering method, such as Figure 7 shown.

[0062] According to a pre-sensing computing method based on a multi-layer optical neural network provided by the present invention, inter-layer distance correction is performed on the pre-sensing computing device based on the multi-layer optical neural network, specifically comprising: Place the first optical mask, the quantum dot film layer, the second optical mask and the board-level photosensitive chip as close as possible; Fixing the positions of the modules of the pre-sensing computing device based on the multi-layer optical neural network; Turn on a plurality of input point light sources, and calculate the spacing between the input point light sources based on incident light rays of the input point light sources according to triangle similarity theory; According to the pixel size of the input point light source and the characteristic size of the two-layer convolution kernel, the inter-layer distance is calculated using the relationship between the pixel size of the point spread function detected by the board-level photosensitive chip and the size of the input point light source; The pre-sensing computing device based on the multi-layer optical neural network is adjusted according to the inter-layer distance.

[0063] Specifically, the specific method for adjusting the interlayer distance is the same as the correction method of d1, d2, d3 and d4 mentioned above, and the present invention will not be elaborated here.

[0064] According to a pre-sensing calculation method based on a multi-layer optical neural network provided by the present invention, the convolution kernel on the optical mask is obtained by processing a glass substrate after cadmium film plating according to the requirements of the target classification task.

[0065] According to a pre-sensing calculation method based on a multi-layer optical neural network provided by the present invention, the inter-layer distance is obtained by correction based on the triangle similarity theory and the characteristic size of the convolution kernel, using the relationship between the plate-level photosensitive chip, the optical mask, the quantum dot film layer, the input light source spacing and the size of the input light source; Among them, the inter-layer distance includes a first distance between the input light source and the first optical mask, a second distance between the first optical mask and the quantum dot film layer, a third distance between the quantum dot film layer and the second optical mask, and a fourth distance between the second optical mask and the board-level photosensitive chip.

[0066] According to a pre-sensing computing method based on a multi-layer optical neural network provided by the present invention, the quantum dot film layer is a CdSe quantum dot film.

[0067] According to a pre-sensing calculation method based on a multi-layer optical neural network provided by the present invention, the spatial characteristic size of the convolution kernel of the first optical mask is 120μm, and the spatial characteristic size of the convolution kernel of the second optical mask is 50um.

[0068] The pre-sensing calculation method based on a multi-layer optical neural network provided by the present invention performs inter-layer distance correction on the pre-sensing calculation device based on the multi-layer optical neural network, and fixes the device after inter-layer distance correction; obtains the incident light emitted by the input light source, performs pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and obtains the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; completes the target classification task based on the pre-sensing calculation result based on the back-end electrical neural network module to obtain the final perception processing result. The present invention builds a multi-layer optical neural network based on two layers of mask plates and one layer of quantum dot film, modulates the amplitude of the incident light through the optical mask plate, completes the convolution calculation in space, and reduces the system complexity of optical calculation; at the same time, optical nonlinearity is added between layers through the quantum dot film layer, which can increase the fitting ability of the optical neural network to further enrich the pre-sensing calculation function, and the processing result of the pre-sensing calculation is obtained by the board-level photosensitive chip, and the subsequent layers of the neural network are processed by the back-end electrical neural network module. The present invention transfers the calculation originally performed on electricity to light, reduces system power consumption and delay, and realizes low-power and high-efficiency visual image processing.

[0069] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the pre-sensing calculation method based on the multi-layer optical neural network, the method comprising: performing inter-layer distance correction on the pre-sensing calculation device based on the multi-layer optical neural network, and fixing the device after the inter-layer distance correction; obtaining the incident light emitted by the input light source, performing pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and obtaining the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; completing the target classification task based on the pre-sensing calculation result based on the back-end electrical neural network module to obtain the final perception processing result.

[0070] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the pre-sensing calculation method based on a multi-layer optical neural network provided by the above-mentioned methods. The method includes: performing inter-layer distance correction on the pre-sensing calculation device based on the multi-layer optical neural network, and fixing the device after the inter-layer distance correction; obtaining the incident light emitted by the input light source, performing pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and obtaining the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; completing the target classification task based on the pre-sensing calculation result based on the back-end electrical neural network module to obtain the final perception processing result.

[0072] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the pre-sensing calculation method based on a multi-layer optical neural network provided by the above-mentioned methods. The method includes: performing inter-layer distance correction on the pre-sensing calculation device based on the multi-layer optical neural network, and fixing the device after the inter-layer distance correction; obtaining incident light emitted by an input light source, performing pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and obtaining the pre-sensing calculation result output by the pre-sensing optical neural network module based on a board-level photosensitive chip; completing the target classification task based on the pre-sensing calculation result based on the back-end electrical neural network module to obtain the final perception processing result.

[0073] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0074] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pre-sensing computing device based on a multi-layer optical neural network, characterized in that: include: Pre-sensing optical neural network module, board-level photosensitive chip and back-end electrical neural network module; The pre-sensing optical neural network module includes a first optical mask, a quantum dot film layer and a second optical mask; a convolution kernel is arranged on the optical mask, and is used as a convolution layer to perform pre-sensing convolution calculation; the quantum dot film layer is arranged between the first optical mask and the second optical mask, and is used as a nonlinear activation layer to perform nonlinear activation between convolution layers; The board-level photosensitive chip is used to receive the pre-sensing calculation results of the optical neural network; The back-end electrical neural network module is used as a linear fully connected layer to complete the target classification task according to the pre-sensing calculation result to obtain the final perception processing result; the first optical mask, the quantum dot thin film layer, the second optical mask and the board-level photosensitive chip are arranged in sequence according to a preset interlayer distance, and the board-level photosensitive chip is connected to the back-end electrical neural network module.

2. The pre-sensing computing device based on a multi-layer optical neural network according to claim 1, characterized in that: The convolution kernel on the optical mask is obtained by processing a glass substrate after cadmium film coating according to the requirements of the target classification task.

3. The pre-sensing computing device based on a multi-layer optical neural network according to claim 1, characterized in that: The interlayer distance is obtained by correction based on the triangle similarity theory and the characteristic size of the convolution kernel, using the relationship between the spacing between the plate-level photosensitive chip, the optical mask, the quantum dot film layer, the input light source and the size of the input light source; Among them, the inter-layer distance includes a first distance between the input light source and the first optical mask, a second distance between the first optical mask and the quantum dot film layer, a third distance between the quantum dot film layer and the second optical mask, and a fourth distance between the second optical mask and the board-level photosensitive chip.

4. The pre-sensing computing device based on a multi-layer optical neural network according to claim 1, characterized in that: The quantum dot film layer is a CdSe quantum dot film.

5. The pre-sensing computing device based on a multi-layer optical neural network according to claim 1, characterized in that: The spatial feature size of the convolution kernel of the first optical mask is 120 μm, and the spatial feature size of the convolution kernel of the second optical mask is 50 μm.

6. A pre-sensing calculation method based on a multi-layer optical neural network, applied to the device as claimed in claims 1-5, characterized in that: include: Performing interlayer distance correction on the pre-sensing computing device based on the multi-layer optical neural network, and fixing the device after the interlayer distance correction; Acquire incident light emitted by an input light source, perform pre-sensing calculation on the incident light based on the pre-sensing optical neural network module, and acquire the pre-sensing calculation result output by the pre-sensing optical neural network module based on the board-level photosensitive chip; Based on the back-end electrical neural network module, the target classification task is completed according to the pre-sensing calculation results to obtain the final perception processing results.

7. The pre-sensing computing method based on a multi-layer optical neural network according to claim 6, characterized in that: The interlayer distance correction is performed on the pre-sensing computing device based on the multi-layer optical neural network, specifically comprising: Placing the second optical mask as close as possible to the board-level photosensitive chip; Fixing the positions of the modules of the pre-sensing computing device based on the multi-layer optical neural network; Turn on a plurality of input point light sources, and calculate the spacing between the input point light sources based on incident light rays of the input point light sources according to triangle similarity theory; According to the pixel size of the input point light source and the characteristic size of the two-layer convolution kernel, the inter-layer distance is calculated using the relationship between the pixel size of the point spread function detected by the board-level photosensitive chip and the size of the input point light source; The pre-sensing computing device based on the multi-layer optical neural network is adjusted according to the inter-layer distance.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the pre-sensing calculation method based on the multi-layer optical neural network as described in claim 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the pre-sensing calculation method based on the multi-layer optical neural network as claimed in claim 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the processor, the pre-sensing calculation method based on the multi-layer optical neural network as claimed in claim 7 is implemented.

Citation Information

Patent Citations

  • Large-view-field adaptive optical system point spread function estimation method

    CN111553866A

  • Multilayer optical convolutional neural network method

    CN112699993A

  • Recording method for virtual point spread function of coded aperture related holography

    CN113376993A

  • Encrypted image information acquisition device and method

    CN114491592A

  • Image information processing apparatus and method

    CN116543058A

Cited By

  • All-optical nonlinear layer, all-optical neural network system and calculation method

    CN119828393A

  • Space-time spectrum intelligent all-optical reconfigurable characterization method, framework and system

    CN119918018A

  • Multilayer nonlinear extensible optical neural network computing system based on 2D / 3D composite perovskite thin film

    CN122287743A