Method for improving photon utilization efficiency of optical neural network cell screening device
By introducing image copy layer, amplification layer, compression layer and imaging detection layer into the optical computing neural network, the optical fiber bundle and nonlinear amplification technology are used to improve the photon utilization efficiency, and the problem of low photon utilization efficiency in the optical computing neural network is solved. , achieving higher imaging accuracy and speed.
Patent Information
- Application Number
- CN202411133897.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-29
AI Technical Summary
The existing optical computing neural networks are not efficient in cell screening devices, resulting in low imaging accuracy and slow imaging speed.
Using image copy layer, image amplification layer, image compression layer and imaging detection layer, optical signals are converged, copied, inequivalent linear shrinkage, nonlinear amplification and spatial dimension compression through optical fiber bundle coupling devices to form an efficient optical neural network Structure, using the high conduction efficiency and nonlinear amplification technology of optical fibers to improve photon utilization efficiency.
It significantly improves the photon utilization efficiency of the cell screening device, reduces the system size and energy consumption, improves the imaging accuracy and speed, increases the recognition accuracy by 8% to 20%, and accelerates the training speed by 3-5 times.
Smart Images

Figure CN119048460A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cell screening based on neural networks, and in particular to a method for improving the photon utilization efficiency of an optical neural network cell screening device. Background Art
[0002] With the development of biomedical technology, cell identification and screening play an increasingly important role in early disease prevention, clinical disease diagnosis, drug development and other aspects. Traditional cell identification and screening devices such as fluorescence spectral flow cytometers, imaging flow cytometers, microfluidic chips, current counters, etc. have one or more problems such as large equipment size, slow recognition speed and long time, high energy consumption, complex operation or insufficient recognition accuracy. Among them, the most advanced cell identification and screening device "imaging flow cytometer" has greatly improved the recognition accuracy and recognition speed by combining the latest neural network. However, in general, the imaging flow cytometer combined with the traditional neural network form is limited by the physical limitations of the propagation speed of the circuit and the upper limit of the speed of electron transport, and it is impossible to achieve ultra-fast imaging or real-time high-speed screening. Therefore, replacing the circuit-based neural network with an optical neural network is very helpful in improving the upper limit of the recognition speed of the imaging flow cytometer.
[0003] Generally, there are many types of optical neural networks used for image recognition, including diffraction, microring resonance, Michelson interferometer, Mach-Zehnder interference, optical computing, etc. Among them, the diffraction type of neural network has slow recognition speed and low efficiency, microring resonance is not suitable for most biologically available laser wavelength ranges, and the Michelson interference and Mach-Zehnder interference types of neural networks are large in size. Therefore, the optical neural network that is most suitable for imaging flow cytometry is the optical computing type of optical neural network.
[0004] Generally, optical computing neural networks are sensitive to photon utilization efficiency and have high requirements for photon intensity. The present invention provides a method for improving the photon utilization efficiency of an optical neural network cell screening device, thereby improving the photon utilization efficiency of the system and thus improving the imaging accuracy and imaging speed. Summary of the invention
[0005] The purpose of the present invention is to provide a method for improving the photon utilization efficiency of an optical neural network cell screening device in view of the defects of low imaging accuracy and slow imaging rate caused by insufficient photon utilization efficiency in the existing optical computing neural network technology. First, the optical signal of the detected flow cell is obtained through the image copy layer, and the optical signal is converged, copied multiple times, and non-equivalent linear reduction operations are performed in sequence. Then, the image amplification layer non-linearly amplifies the intensity of the non-equivalent linearly reduced optical signal according to the non-linear amplification function, and the image compression layer compresses the spatial dimension of the non-linearly amplified optical signal. Finally, the compressed optical signal is output as an image through the imaging detection layer. The optical neural network-based imaging flow cytometer introduced in the present invention calculates, compresses, and identifies optical signals for cell recognition, which can effectively reduce the overall size of the system while meeting the requirements of cell recognition and screening. Compared with the imaging cytometer without adding the optical neural network, the recognition accuracy is improved by 8% to 20%. It also greatly reduces the energy (especially photon) consumption of the system. Compared with the traditional electrical image processing method, the present invention utilizes the high energy information of the light beam, evenly copies the optical signal through a microlens array in the "image copy layer", and adopts unequal linear reduction and nonlinear amplification to ensure that each image has sufficient brightness and sufficient image feature information is retained without distorting the image after the final detection.
[0006] The optical neural network cell screening device provided by the embodiment of the present invention is composed of an image copy layer, an image amplification layer, an image compression layer, an imaging detection layer and an optical fiber bundle coupling device. Different layers have different functions, and multiple layers can be directly superimposed to achieve multiple extended functions. The layers are connected by imaging optical fiber bundles to improve the efficiency of photon utilization and remove multiple conventional lenses used for focusing and signal transmission in traditional optical computing. Reduce the volume and weight of the system.
[0007] Fiber optic bundles are an efficient and lossless optical transmission method. Compared with the free space photon propagation used by traditional optical neural networks, the photon loss of the system can be further reduced. The optical neural network cell screening device provided by the embodiment of the present invention utilizes the high transmission efficiency and fixed length characteristics of optical fibers, introduces optical fibers as optical transmission media in free space, and improves the utilization efficiency of photons. At the same time, by folding and flipping the optical fibers, the spacing between the optical neural network layers can be reduced while meeting the focal length of the lens in the original free space, thereby reducing the volume of the system. In addition, pure optical means such as optical signal compression and optical signal intensity nonlinear amplification are used to collect, calculate, and compress optical signals in an integrated manner, effectively improving the conversion efficiency and processing speed of the system, and effectively improving the screening recognition speed and accuracy compared to the electrical calculation method, greatly reducing the energy consumption of the system; the optical fiber beam composed of high light conductivity and low loss optical fibers further reduces the energy loss of the system.
[0008] The present invention overcomes the spherical wave energy loss caused by the wave nature of photons in free space, and transmits the optical image to the next optical neural network layer in a directional manner through optical fibers. For example, in the form of an optical fiber bundle, the photon utilization efficiency of the same type of optical neural network in free space can be increased by 14.8%, which is of great significance for weak light detection, high-speed imaging, etc.
[0009] To achieve the above-mentioned purpose, in a first aspect, an embodiment of the present invention provides a method for improving the photon utilization efficiency of an optical neural network cell screening device, wherein the optical neural network of the optical neural network cell screening device has four layers, the first layer is an image copy layer, the second layer is an image amplification layer, the third layer is an image compression layer, and the fourth layer is an imaging detection layer;
[0010] The method comprises:
[0011] The image copy layer acquires the optical signal, and sequentially converges, copies, and linearly reduces the optical signal in an unequal manner;
[0012] The image amplification layer nonlinearly amplifies the intensity of the unequally linearly reduced light signal;
[0013] The image compression layer performs neural network calculation on the nonlinearly amplified optical signal to generate an optical signal compressed in spatial dimension;
[0014] The imaging detection layer acquires the light signal compressed in the spatial dimension and forms an image signal output;
[0015] The image copy layer, image magnification layer, image compression layer and imaging detection layer are connected through an imaging fiber bundle system; the imaging fiber bundle system includes: a first imaging fiber bundle, a second imaging fiber bundle, a third imaging fiber bundle, a fourth imaging fiber bundle and a fifth imaging fiber bundle.
[0016] Preferably, the elements of the image copy layer include: an imaging lens, a first microlens array and a liquid crystal panel;
[0017] The process of the image copy layer acquiring the optical signal and converging and linearly reducing the optical signal is specifically as follows:
[0018] The imaging lens acquires the optical signal and converges the optical signal;
[0019] The first microlens array acquires the converged light signal through the first imaging optical fiber bundle and copies it;
[0020] The liquid crystal panel obtains the multiple copied optical signals through the second imaging optical fiber bundle, and simultaneously performs unequal linear reduction of the optical intensity of the multiple copied optical signals by changing the grayscale value;
[0021] The unequal linear reduction specifically includes: performing linear reduction of different multiples on each optical signal obtained by copying.
[0022] Preferably, the elements of the image amplification layer include nonlinear amplification devices of optical intensity;
[0023] The process of the image amplification layer performing nonlinear amplification processing on the unequal linearly reduced light signal is specifically as follows: the nonlinear amplification device obtains the unequal linearly reduced light signal through the third imaging optical fiber bundle, and nonlinearly amplifies the intensity of the unequal linearly reduced light signal according to different intrinsic nonlinear intensity amplification functions of the nonlinear amplification device.
[0024] Preferably, the elements of the image compression layer include a second microlens array;
[0025] The image compression layer performs neural network calculation on the nonlinearly amplified optical signal to generate an optical signal compressed in spatial dimension as follows: the second microlens array obtains the nonlinearly amplified optical signal through the fourth imaging optical fiber bundle, and 2 The intensity of the optical signal after nonlinear amplification is superimposed by optical signal through optical fiber bundle merging to generate an optical signal with compressed spatial dimension, wherein n is a non-zero integer.
[0026] Preferably, the elements of the imaging detection layer include image detection equipment;
[0027] The imaging detection layer acquires the compressed optical signal and forms an image signal output as follows: the image detection device acquires the optical signal compressed in the spatial dimension through the fifth imaging optical fiber bundle, and forms an image signal output to a display device or a host computer for processing.
[0028] In a second aspect, an embodiment of the present invention provides a device for executing the method for improving the photon utilization efficiency of an optical neural network cell screening device as described in the first aspect, the device comprising: an optical neural network and an imaging fiber bundle system;
[0029] The optical neural network comprises: an image copying layer, an image amplifying layer, an image compressing layer and an imaging detection layer; the layers of the optical neural network are connected through the imaging optical fiber bundle system.
[0030] Preferably, the imaging fiber bundle system comprises: a first imaging fiber bundle, a second imaging fiber bundle, a third imaging fiber bundle, a fourth imaging fiber bundle and a fifth imaging fiber bundle;
[0031] The components of the image copy layer include: an imaging lens, a first microlens array and a liquid crystal panel;
[0032] The elements of the image amplification layer include nonlinear amplification devices;
[0033] The elements of the image compression layer include a second microlens array;
[0034] The elements of the imaging detection layer include image detection devices;
[0035] The imaging lens is connected to one end of the first microlens array through the first imaging fiber bundle; the other end of the first microlens array is connected to one end of the liquid crystal panel through the second imaging fiber bundle; the other end of the liquid crystal panel is connected to one end of the nonlinear amplifier device through the third imaging fiber bundle; the other end of the nonlinear amplifier device is connected to one end of the second microlens array through the fourth imaging fiber bundle; the other end of the second microlens array is connected to the image detection device through the fifth imaging fiber bundle.
[0036] Preferably, the imaging lens acquires light signals, and converges the light signals and sends them to the first microlens array through the first imaging optical fiber bundle;
[0037] The first microlens array makes multiple copies of the converged optical signal and transmits the copies to the liquid crystal panel through the second imaging optical fiber bundle; the liquid crystal panel performs unequal linear reduction on the intensity of the multiple copies of the optical signal and transmits the copies to the nonlinear amplifier through the third imaging optical fiber bundle;
[0038] The nonlinear amplifier device nonlinearly amplifies the intensity of the optical signal after the unequal linear reduction according to its own intrinsic optical intensity nonlinear amplification function, and transmits it to the second microlens array through the fourth imaging optical fiber bundle;
[0039] The second microlens array compresses the spatial dimension of the optical signal after the nonlinear amplification, and sends the spatial dimension compressed optical signal to the image detection device through the fifth imaging optical fiber bundle to output an image signal;
[0040] The image detection device includes: a weak light detection camera or an imaging detector with a quantum efficiency higher than 40%.
[0041] Further preferably, the first imaging fiber bundle, the second imaging fiber bundle, the third imaging fiber bundle, the fourth imaging fiber bundle and the fifth imaging fiber bundle are all composed of multiple strands of optical fibers;
[0042] The length of the first imaging fiber bundle is the same as the focal length of the imaging lens; every 2-3 optical fibers in the first imaging fiber bundle on the side close to the first microlens array are aligned and coupled with a microlens in the first microlens array;
[0043] The length of the second imaging fiber bundle is the same as the focal length of the first microlens array; the second imaging fiber bundle on a side close to the liquid crystal panel is aligned with the pixels of the liquid crystal panel;
[0044] The third imaging fiber bundle is aligned with the pixels of the nonlinear amplifying device; n 2 The strand of optical fiber is aligned and coupled with a pixel of a sub-microlens of the second microlens array, wherein n is a non-zero integer;
[0045] The number of sub-pixels in the first microlens array is i times the number of sub-pixels in the second microlens array. 2 times, where i is an integer and i≥1;
[0046] The nonlinear amplifier device is a device for optical nonlinear amplification, including: any one of a microchannel plate, an up-conversion material, and a non-saturated material; the intrinsic optical intensity amplification function of the nonlinear amplifier device is a nonlinear function of input light intensity and output light intensity;
[0047] The weak light detection camera includes: any one of an sCMOS camera, an EMCCD camera or an ICCD camera.
[0048] Preferably, the device also includes a main frame; the imaging lens is fixed to one side of the main frame by a bayonet; the first microlens array, the liquid crystal panel, the nonlinear amplifier device, the second microlens array, the image detection device and the imaging fiber optic bundle system are placed in the main frame; an opening is provided on the outside of the main frame for inserting drive lines, signal lines and electrical wires.
[0049] An embodiment of the present invention provides a method for improving the photon utilization efficiency of an optical neural network cell screening device. First, the optical signal of the detected flow cell is acquired through an image copy layer, and the optical signal is converged, copied multiple times, and non-equivalent linear reduction operations are performed in sequence. Then, the image amplification layer non-linearly amplifies the intensity of the non-equivalent linearly reduced optical signal according to a nonlinear amplification function, and the image compression layer further compresses the spatial dimension of the non-linearly amplified optical signal. Finally, the compressed optical signal is output as an image through the imaging detection layer.
[0050] The method provided by the embodiment of the present invention can obtain high-dimensional image data in the form of an optical neural network, and perform computational processing, compression, semantic segmentation, etc. on the image data (optical signal). Thanks to the high-dimensional characteristics of the image data, the neural network can be superimposed in multiple layers, adding different functions to improve the detection accuracy of the system. The present invention improves the signal recognition accuracy to more than 85% through the nonlinear amplification function of the image amplification layer, and utilizes the high speed of the imaging detection layer to increase the training speed of the data set by more than 2 times compared to traditional detectors. Thanks to the improved photon utilization and transmission efficiency, the overall training speed can be 3-5 times faster than the traditional optical neural network imaging flow cytometer while ensuring the recognition accuracy.
[0051] The optical neural network cell screening device provided by the embodiment of the present invention reduces the system volume and the difficulty of system splicing. The components of multiple neural network layers are connected by optical fiber bundles, and the length of the optical fiber is controlled to be the focal length of the relevant lens. At the same time, the foldability and bendability of the optical fiber can be used to fold the system in physical space to reduce the volume, while improving the alignment accuracy between optical components and reducing the difficulty of system splicing.
[0052] The optical neural network cell screening device provided by the embodiment of the present invention has high detection accuracy, detection speed and photon utilization efficiency, and reduces system delay. By linking each optical neural network layer through an imaging fiber bundle, the photon utilization efficiency of the system is greatly improved compared with the spherical wave loss of light transmission in free space, the detection capability is improved, and the detection speed is accelerated.
[0053] The optical neural network cell screening device provided by the embodiment of the present invention is applied to imaging flow cell screening. Different cells (different in size, morphology, transmittance, etc.) in the flow pipeline can be directly identified and features extracted through the nonlinear neural network, and compared with the data in the data set library to distinguish the cell type, thereby achieving the purpose of cell morphology analysis. The current single-layer nonlinear layer recognition accuracy can reach 85%, and the recognition accuracy can reach 93% by superimposing two or more nonlinear layers.
[0054] The optical neural network cell screening device provided by the embodiment of the present invention has the characteristics of high recognition accuracy, fast speed, and low delay compared to traditional flow cytometers; the overall system has a small size and low energy consumption. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 This is a structural block diagram of the optical neural network cell screening device 1 provided in an embodiment of the present invention.
[0056] Figure 2 A flow chart of a method for improving the photon utilization efficiency of the optical neural network cell screening device 1 provided in an embodiment of the present invention.
[0057] Figure 3 This is a structural block diagram of the optical neural network cell screening device 2 provided in an embodiment of the present invention.
[0058] Figure 4 A flow chart of a method for improving the photon utilization efficiency of an optical neural network cell screening device 2 provided in an embodiment of the present invention.
[0059] Figure 5 A method for compressing the intensity of an optical signal after nonlinear amplification provided in Embodiment 2 of the present invention.
[0060] Figure 6 This is a schematic diagram of the structure of the optical neural network cell screening device provided in Example 1 of the present invention.
[0061] Figure 7 This is a schematic diagram of the structure of the main frame of the optical neural network cell screening device provided in Example 1 of the present invention.
[0062] Figure 8 Schematic diagram of a first microlens array and a second microlens array in the device provided in Embodiment 1 of the present invention.
[0063] Fig. 9 This is a graph of the incident light and the amplified light intensity of the nonlinear amplification function of the nonlinear amplifying device in the device of Example 1 provided in Example 2 of the present invention. DETAILED DESCRIPTION
[0064] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0065] An embodiment of the present invention provides a method for improving the photon utilization efficiency of an optical neural network cell screening device. For ease of understanding, the optical neural network cell screening device used to execute the method is first introduced.
[0066] Figure 1 This is a structural block diagram of the optical neural network cell screening device provided in an embodiment of the present invention.
[0067] like Figure 1 As shown, the optical neural network cell screening device 1 includes: an optical neural network 11 and an imaging fiber bundle system 12, and the layers of the optical neural network 11 are connected through the imaging fiber bundle system 12.
[0068] The optical neural network 11 has four layers: the first layer is an image copy layer 13, the second layer is an image magnification layer 14, the third layer is an image compression layer 15, and the fourth layer is an imaging detection layer 16. The layers of the optical neural network 11 are connected through an imaging fiber bundle system 12.
[0069] The embodiment of the present invention provides a method for improving the photon utilization efficiency of the optical neural network cell screening device 1, such as Figure 2 As shown, the specific steps include:
[0070] In step 110 , the image copy layer 13 acquires the optical signal, and sequentially converges, copies, and performs non-equivalent linear reduction on the optical signal.
[0071] Among them, the optical signal refers to the optical signal of the detected cell or the external simulated image; the detected cell is a flow cell; the external simulated image is used to debug the optical neural network cell screening device of the present invention.
[0072] In step 120 , the image magnifying layer 14 non-linearly amplifies the intensity of the unequally linearly reduced light signal.
[0073] Step 130: the image compression layer 15 performs neural network calculation on the optical signal after nonlinear amplification to generate an optical signal after spatial dimension compression.
[0074] In step 140 , the imaging detection layer 16 acquires the light signal after the spatial dimension is compressed and forms an image signal output.
[0075] Figure 3 A structural block diagram of a preferred solution of an optical neural network cell screening device is provided for an embodiment of the present invention.
[0076] like Figure 3 As shown, the imaging fiber optic bundle system in the optical neural network cell screening device 2 includes: a first imaging fiber optic bundle 21, a second imaging fiber optic bundle 22, a third imaging fiber optic bundle 23, a fourth imaging fiber optic bundle 24 and a fifth imaging fiber optic bundle 25.
[0077] The first imaging fiber bundle 21 , the second imaging fiber bundle 22 , the third imaging fiber bundle 23 , the fourth imaging fiber bundle 24 , and the fifth imaging fiber bundle 25 are all composed of a plurality of optical fibers.
[0078] The optical neural network in the optical neural network cell screening device 2 includes: an image copy layer 26, an image amplification layer 27, an image compression layer 28 and an imaging detection layer 29.
[0079] The components included in each layer of the optical neural network of the optical neural network cell screening device 2 are introduced as follows:
[0080] The image copy layer 26 is used to obtain optical signals, and to converge, make multiple copies, and perform unequal linear reduction on the optical signals in sequence. The components included in the image copy layer 26 are: an imaging lens 261 , a first microlens array 262 , and a liquid crystal panel 263 .
[0081] The image amplification layer 27 is used to nonlinearly amplify the intensity of the optical signal after convergence, multiple copies and unequal linear reduction, and the components of the image amplification layer 27 include a nonlinear amplification device 271. The nonlinear amplification device 271 is a device that plays a nonlinear amplification role on the optical signal, including: any one of a microchannel plate, an up-conversion material, and a non-saturated material; the optical amplification function of the nonlinear amplification device 271 is a nonlinear function of the input light intensity and the output light intensity.
[0082] The image compression layer 28 is used to perform neural network calculation on the optical signal after nonlinear amplification, so as to obtain the optical signal after spatial dimension compression; the elements of the image compression layer 28 include a second microlens array 281.
[0083] The imaging detection layer 29 is used to image the compressed light signal and form an image signal output, and the components of the imaging detection layer 29 include an image detection device 291. The image detection device 291 includes: a weak light detection camera or an imaging detector with a quantum efficiency higher than 40%; the weak light detection camera includes: any one of an sCMOS camera, an EMCCD camera or an ICCD camera.
[0084] The optical neural network cell screening device 2 also includes a main frame (not shown in the figure); the imaging lens 261 is fixed to one side of the main frame by a bayonet; the first microlens array 262, the liquid crystal panel 263, the nonlinear amplifier device 271, the second microlens array 281, the image detection device 291, the first imaging fiber optic bundle 21, the second imaging fiber optic bundle 22, the third imaging fiber optic bundle 23, the fourth imaging fiber optic bundle 24 and the fifth imaging fiber optic bundle 25 are placed in the main frame; an opening is provided on the outside of the main frame for inserting drive lines, signal lines and electrical wires.
[0085] The connection relationship of the above-mentioned components is specifically as follows: the imaging lens 261 is connected to one end of the first microlens array 262 through the first imaging fiber optic bundle 21; the other end of the first microlens array 262 is connected to one end of the liquid crystal panel 263 through the second imaging fiber optic bundle 22; the other end of the liquid crystal panel 263 is connected to one end of the nonlinear amplifier device 271 through the third imaging fiber optic bundle 23; the other end of the nonlinear amplifier device 271 is connected to one end of the second microlens array 281 through the fourth imaging fiber optic bundle 24; the other end of the second microlens array 281 is connected to the image detection device 291 through the fifth imaging fiber optic bundle 25.
[0086] The length of the first imaging fiber bundle 21 is the same as the focal length of the imaging lens 261 ; every 2-3 optical fibers in the first imaging fiber bundle 21 on the side close to the first microlens array 262 are aligned and coupled with a microlens in the first microlens array 262 .
[0087] The length of the second imaging fiber bundle 22 is the same as the focal length of the first microlens array 262 ; the second imaging fiber bundle 22 on the side close to the liquid crystal panel 263 is aligned with the pixels of the liquid crystal panel 263 .
[0088] The third imaging fiber bundle 23 is aligned with the pixels of the nonlinear amplifier device 271; each n pixels of the third imaging fiber bundle 23 near the side of the second microlens array 281 are aligned with the pixels of the nonlinear amplifier device 271; 2 The strand of optical fiber is aligned and coupled with a pixel of a microlens in the second microlens array 281, where n is a non-zero integer.
[0089] The number of sub-pixels in the first microlens array 262 is equal to i times the number of sub-pixels in the second microlens array 281. 2 times, where i is an integer and i≥1; for example, the number of sub-pixels of the first microlens array 262 is 4 times, 9 times or 10 times the number of sub-pixels of the second microlens array 281.
[0090] In an optional solution, the first microlens array 262 and the second microlens array 281 can adjust the focal length and magnification according to the actual screening situation; or, according to the detection needs, the imaging lens 261 can be replaced with other devices, such as an endoscope or an endoscopic probe, which can be used for endoscopic imaging to realize interventional detection.
[0091] The embodiment of the present invention provides a method for improving the photon utilization efficiency of the optical neural network cell screening device 2, such as Figure 4 As shown, combined Figure 3 The specific steps are described, including the following steps:
[0092] Step 210, the imaging lens 261 acquires the optical signal and converges the optical signal;
[0093] Among them, the optical signal refers to the optical signal of the detected cell or the external simulated image; the detected cell is a flow cell; the external simulated image is used to debug the optical neural network cell screening device of the present invention.
[0094] In step 220 , the first microlens array 262 acquires the converged optical signal through the first imaging optical fiber bundle 21 and makes a copy.
[0095] In step 230 , the liquid crystal panel 263 obtains the multiple copies of the optical signal through the second imaging optical fiber bundle 22 , and simultaneously performs unequal linear reduction of the optical intensity of the multiple copies of the optical signal by changing the grayscale value.
[0096] The unequal linear reduction specifically refers to: performing linear reduction of different multiples on each optical signal obtained by copying.
[0097] In step 240, the nonlinear amplifier 271 obtains the unequal linearly reduced optical signal through the third imaging optical fiber 23 bundle, and nonlinearly amplifies the intensity of the unequal linearly reduced optical signal according to the intrinsic nonlinear intensity amplification function of different nonlinear amplifier devices.
[0098] Step 250: The second microlens array 281 obtains the nonlinearly amplified optical signal through the fourth imaging fiber bundle 24, and 2 The intensities of the optical signals after nonlinear amplification are superimposed by optical signal combination through optical fiber bundle merging to generate an optical signal with compressed spatial dimension, where n is a non-zero integer.
[0099] Regarding the principle of compressing the spatial dimension of the optical signal after nonlinear amplification in the embodiment of the present invention, the second microlens array 281 is taken as an example of 2×2 microlenses. Figure 5As shown, it is a simplified calculation method of optical neural network in mathematical dimension, which includes the principle of "image compression" and the mathematical operation expression of neural network. The light signal reflected by the cell aligned with the imaging lens 261 of the device 1 is mathematically marked as the variable "X". After being collected by the imaging lens 261, it enters the first microlens array 262 and is copied into multiple images, which are all marked with the variable "X". The total number of copies is determined by the number of microlenses in the second microlens array 281. When the second microlens array 281 is a microlens array composed of 2×2 microlenses, the optical intensity of each copied image is linearly reduced after passing through the liquid crystal panel. For the mathematical representation of the image, it is equivalent to multiplying by the constant "a i ”, the mathematical representation of the image signal becomes “a i X”, and we use “σ” to represent the nonlinear amplification function of the nonlinear amplification device 271. After passing through the nonlinear amplification device 271, the image signal can be mathematically represented as “σ(a i The nonlinear amplification function "σ" of the nonlinear amplification device 271 depends on the nonlinear material properties (such as different nonlinear layer materials) and the control applied to the nonlinear layer (such as voltage, temperature, etc.). The four post-operation images obtained after linear and nonlinear operations on the image are: "σ(a 11 X),σ(a 12 X),σ(a 21 X),σ(a 22 X)” can obtain the characteristic value of the system through addition and subtraction operations, thereby performing image screening.
[0100] This embodiment obtains the characteristic analysis of the image in the form of a nonlinear neural network. For example, it can be used in cell observation: the membrane interface in the cell and the image (light signal) of the background part will produce a certain contrast. Through the nonlinear amplification method described above, the obtained "σ(a 11 X),σ(a 12 X),σ(a 21 X),σ(a 22 The "X)" image signal will be amplified in the low-contrast area in the cell image. The maximum value obtained by adding the four eigenvalues is the boundary part with contrast. The difference of the four eigenvalues will subtract the background part and only retain the part with contrast. Different contrast thresholds can be controlled by adjusting the nonlinear magnification σ and the linear magnification a. For example, the part with low contrast can be regarded as the background or uniform material, and the part with high contrast can be amplified as the outline of the cell (or the observed object), so that feature extraction is achieved.
[0101] In step 260, the image detection device 29 obtains the light signal compressed in the spatial dimension through the fifth imaging optical fiber bundle 25, and forms an image signal to output to the display device or the host computer for processing.
[0102] In order to better understand the technical solution provided by the present invention, the following specific examples are used to illustrate the method of improving the photon utilization efficiency of the optical neural network cell screening device of the present invention.
[0103] Example 1
[0104] This embodiment provides an optical neural network cell screening device, the structural schematic diagram of which is shown in FIG. Figure 5 shown.
[0105] The imaging fiber bundle system of the optical neural network cell screening device 3 of this embodiment includes: a first imaging fiber bundle 31, a second imaging fiber bundle 32, a third imaging fiber bundle 33, a fourth imaging fiber bundle 34 and a fifth imaging fiber bundle 35.
[0106] The optical neural network in the optical neural network cell screening device 3 includes: an image copy layer 36, an image amplification layer 37, an image compression layer 38 and an imaging detection layer 39.
[0107] The components of the image copy layer 36 include: an imaging lens 361 , a first microlens array 362 , and a liquid crystal panel 363 .
[0108] The element of the image amplifying layer 37 is a nonlinear amplifying device 371; the nonlinear amplifying device 371 of this embodiment is a microchannel plate (MCP).
[0109] An element of the image compression layer 38 is a second microlens array 381 .
[0110] The element of the imaging detection layer 39 is an image detection device 391. In this embodiment, the image detection device 391 adopts a weak light detection camera, specifically a sCMOS camera.
[0111] In addition, the optical neural network cell screening device 3 of this embodiment also includes a main frame, such as Figure 6 As shown, the imaging lens 361 is fixed to one side of the main frame through a bayonet; the first microlens array 362, the liquid crystal panel 363, the nonlinear amplifier device 371, the second microlens array 381, the image detection device 391, the first imaging fiber bundle 31, the second imaging fiber bundle 32, the third imaging fiber bundle 33, the fourth imaging fiber bundle 34 and the fifth imaging fiber bundle 35 are all placed in the main frame; an opening (not shown in the figure) is provided on the outside of the main frame for inserting drive lines, signal lines, electrical wires, etc.
[0112] The connection relationship of the above-mentioned components is specifically as follows: the imaging lens 361 is connected to one end of the first microlens array 362 through the first imaging fiber optic bundle 31; the other end of the first microlens array 362 is connected to one end of the liquid crystal panel 363 through the second imaging fiber optic bundle 32; the other end of the liquid crystal panel 363 is connected to one end of the nonlinear amplifier device 371 through the third imaging fiber optic bundle 33; the other end of the nonlinear amplifier device 371 is connected to one end of the second microlens array 381 through the fourth imaging fiber optic bundle 34; the other end of the second microlens array 381 is connected to the image detection device 391 through the fifth imaging fiber optic bundle 35.
[0113] The length of the first imaging fiber bundle 31 is the same as the focal length of the imaging lens 361 ; each three optical fibers in the first imaging fiber bundle 31 close to the first microlens array 362 are aligned and coupled with a microlens in the first microlens array 362 .
[0114] The length of the second imaging fiber bundle 32 is the same as the focal length of the first microlens array 362 ; the second imaging fiber bundle 32 on the side close to the liquid crystal panel 363 is aligned with the pixels of the liquid crystal panel 363 .
[0115] The third imaging fiber bundle 33 is aligned with the pixels of the nonlinear amplifier device 371; each n pixels of the third imaging fiber bundle 33 near the side of the second microlens array 381 are aligned with the pixels of the nonlinear amplifier device 371; 2 The strand of optical fiber (where n is a non-zero integer) is aligned and coupled with a pixel of a microlens in the second microlens array 381.
[0116] In this embodiment, the first microlens array 362 is composed of 9×9 microlenses, and the second microlens array 381 is composed of 3×3 microlenses. Figure 7 That is to say, the number of sub-pixels in the first microlens array 362 is 9 times the number of sub-pixels in the second microlens array 381. The present invention utilizes multiple microlens groups in the first microlens array 362 to equally divide the copies of spatial light, and utilizes multiple microlens groups in the second microlens array 381 to equally fuse and compress the spatial light.
[0117] Example 2
[0118] This embodiment provides a method for improving the photon utilization efficiency of the optical neural network cell screening device of embodiment 1, combining Figure 5 , specifically including the following steps:
[0119] (1) The imaging lens 361 acquires the optical signals of the detected flow cells and converges the optical signals of the flow cells.
[0120] (2) The first microlens array 362 acquires the converged optical signal of the flow cell through the first imaging fiber bundle 31 and makes a copy.
[0121] (3) The liquid crystal panel 363 obtains the multiple copies of the optical signal through the second imaging optical fiber bundle 32, and simultaneously performs unequal linear reduction of the optical intensity of the multiple copies of the optical signal by changing the grayscale value.
[0122] (4) The nonlinear amplifier 371 obtains the copied and non-equivalent linearly reduced optical signal through the third imaging fiber bundle 33 , and non-linearly amplifies the intensity of the non-equivalent linearly reduced optical signal according to the intrinsic nonlinear intensity amplification function of the nonlinear amplifier 371 .
[0123] The nonlinear amplification function of this embodiment is an optical amplification function, such as Figure 8 As shown, it includes a "Sigma-type" amplification function represented by a microchannel plate (MCP) and an "exponential" amplification function represented by an electrical signal amplification. The horizontal axis in the figure represents the light intensity incident on the nonlinear amplifier device 371, and the vertical axis represents the light intensity after amplification.
[0124] (5) The second microlens array 381 obtains the nonlinearly amplified optical signal through the fourth imaging optical fiber bundle 34, and superimposes the intensity of each 9 nonlinearly amplified optical signals by optical fiber bundle merging to generate an optical signal with compressed spatial dimension.
[0125] (6) The image detection device 391 obtains the compressed light signal through the fifth imaging optical fiber bundle 35, forms an optical signal, performs imaging according to the optical signal, and outputs it to an external display device.
[0126] The image detection device 391 of this embodiment uses an sCMOS camera, which can train the data set model at more than 120 images per second, and automatically calculate the optimal σ and a for different observation objects (such as animal cells and plant cells). i The data are combined and recorded to facilitate the subsequent direct use of the corresponding parameters for cell analysis. This embodiment utilizes the high speed of sCMOS to increase the training speed of the data set by more than 2 times.
[0127] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0128] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0129] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A method for improving the photon utilization efficiency of an optical neural network cell screening device, characterized in that: The optical neural network of the optical neural network cell screening device has four layers, the first layer is an image copy layer, the second layer is an image amplification layer, the third layer is an image compression layer, and the fourth layer is an imaging detection layer; The method comprises: The image copy layer acquires the optical signal, and sequentially converges, copies, and linearly reduces the optical signal in an unequal manner; The image amplification layer nonlinearly amplifies the intensity of the unequally linearly reduced light signal; The image compression layer performs neural network calculation on the nonlinearly amplified optical signal to generate an optical signal compressed in spatial dimension; The imaging detection layer acquires the light signal compressed in the spatial dimension and forms an image signal output; The image copy layer, image magnification layer, image compression layer and imaging detection layer are connected through an imaging fiber bundle system; the imaging fiber bundle system includes: a first imaging fiber bundle, a second imaging fiber bundle, a third imaging fiber bundle, a fourth imaging fiber bundle and a fifth imaging fiber bundle.
2. The method for improving the photon utilization efficiency of the optical neural network cell screening device according to claim 1, characterized in that: The components of the image copy layer include: an imaging lens, a first microlens array and a liquid crystal panel; The process of the image copy layer acquiring the optical signal and converging and linearly reducing the optical signal is specifically as follows: The imaging lens acquires the optical signal and converges the optical signal; The first microlens array acquires the converged light signal through the first imaging optical fiber bundle and copies it; The liquid crystal panel obtains the multiple copied optical signals through the second imaging optical fiber bundle, and simultaneously performs unequal linear reduction of the optical intensity of the multiple copied optical signals by changing the grayscale value; The unequal linear reduction specifically includes: performing linear reduction of different multiples on each optical signal obtained by copying.
3. The method for improving the photon utilization efficiency of the optical neural network cell screening device according to claim 1, characterized in that: The elements of the image amplification layer include nonlinear amplification devices of optical intensity; The process of the image amplification layer performing nonlinear amplification processing on the unequal linearly reduced light signal is specifically as follows: the nonlinear amplification device obtains the unequal linearly reduced light signal through the third imaging optical fiber bundle, and nonlinearly amplifies the intensity of the unequal linearly reduced light signal according to different intrinsic nonlinear intensity amplification functions of the nonlinear amplification device.
4. The method for improving the photon utilization efficiency of the optical neural network cell screening device according to claim 1, characterized in that: The elements of the image compression layer include a second microlens array; The image compression layer performs neural network calculation on the nonlinearly amplified optical signal to generate an optical signal compressed in spatial dimension as follows: the second microlens array obtains the nonlinearly amplified optical signal through the fourth imaging optical fiber bundle, and 2 The intensity of the optical signal after nonlinear amplification is superimposed by optical signal through optical fiber bundle merging to generate an optical signal with compressed spatial dimension, wherein n is a non-zero integer.
5. The method for improving the photon utilization efficiency of the optical neural network cell screening device according to claim 1, characterized in that: The elements of the imaging detection layer include image detection devices; The imaging detection layer acquires the compressed optical signal and forms an image signal output as follows: the image detection device acquires the optical signal compressed in the spatial dimension through the fifth imaging optical fiber bundle, and forms an image signal output to a display device or a host computer for processing.
6. A device for executing the method for improving the photon utilization efficiency of an optical neural network cell screening device as described in any one of claims 1 to 5, characterized in that: The device comprises: an optical neural network and an imaging fiber bundle system; The optical neural network comprises: an image copying layer, an image amplifying layer, an image compressing layer and an imaging detection layer; the layers of the optical neural network are connected through the imaging optical fiber bundle system.
7. The device according to claim 6, characterized in that The imaging fiber bundle system comprises: a first imaging fiber bundle, a second imaging fiber bundle, a third imaging fiber bundle, a fourth imaging fiber bundle and a fifth imaging fiber bundle; The components of the image copy layer include: an imaging lens, a first microlens array and a liquid crystal panel; The elements of the image amplification layer include nonlinear amplification devices; The elements of the image compression layer include a second microlens array; The elements of the imaging detection layer include image detection devices; The imaging lens is connected to one end of the first microlens array through the first imaging fiber bundle; the other end of the first microlens array is connected to one end of the liquid crystal panel through the second imaging fiber bundle; the other end of the liquid crystal panel is connected to one end of the nonlinear amplifier device through the third imaging fiber bundle; the other end of the nonlinear amplifier device is connected to one end of the second microlens array through the fourth imaging fiber bundle; the other end of the second microlens array is connected to the image detection device through the fifth imaging fiber bundle.
8. The device according to claim 7, characterized in that The imaging lens acquires a light signal, and converges the light signal and sends it to the first microlens array through the first imaging optical fiber bundle; The first microlens array makes multiple copies of the converged optical signal and transmits the copies to the liquid crystal panel through the second imaging optical fiber bundle; the liquid crystal panel performs unequal linear reduction on the intensity of the multiple copies of the optical signal and transmits the copies to the nonlinear amplifier through the third imaging optical fiber bundle; The nonlinear amplifier device nonlinearly amplifies the intensity of the optical signal after the unequal linear reduction according to its own intrinsic optical intensity nonlinear amplification function, and transmits it to the second microlens array through the fourth imaging optical fiber bundle; The second microlens array compresses the spatial dimension of the nonlinearly amplified optical signal, and sends the spatially compressed optical signal to the image detection device through the fifth imaging optical fiber bundle to output an image signal; The image detection device includes: a weak light detection camera or an imaging detector with a quantum efficiency higher than 40%.
9. The device according to claim 8, characterized in that The first imaging fiber bundle, the second imaging fiber bundle, the third imaging fiber bundle, the fourth imaging fiber bundle and the fifth imaging fiber bundle are all composed of multiple strands of optical fibers; The length of the first imaging fiber bundle is the same as the focal length of the imaging lens; every 2-3 optical fibers in the first imaging fiber bundle on the side close to the first microlens array are aligned and coupled with a microlens in the first microlens array; The length of the second imaging fiber bundle is the same as the focal length of the first microlens array; the second imaging fiber bundle on a side close to the liquid crystal panel is aligned with the pixels of the liquid crystal panel; The third imaging fiber bundle is aligned with pixels of the nonlinear amplification device; Each n in the third imaging fiber bundle close to the side of the second microlens array 2 The strand of optical fiber is aligned and coupled with a pixel of a sub-microlens of the second microlens array, wherein n is a non-zero integer; The number of sub-pixels in the first microlens array is i times the number of sub-pixels in the second microlens array. 2 times, where i is an integer and i≥1; The nonlinear amplifier device is a device for optical nonlinear amplification, including: any one of a microchannel plate, an up-conversion material, and a non-saturated material; the intrinsic optical intensity amplification function of the nonlinear amplifier device is a nonlinear function of input light intensity and output light intensity; The weak light detection camera includes: any one of an sCMOS camera, an EMCCD camera or an ICCD camera.
10. The device according to claim 7, characterized in that The device also includes a main frame; the imaging lens is fixed to one side of the main frame through a bayonet; the first microlens array, the liquid crystal panel, the nonlinear amplifier device, the second microlens array, the image detection device and the imaging fiber bundle system are placed in the main frame; an opening is provided on the outside of the main frame for inserting drive lines, signal lines and electrical wires.
