Deep learning-based multimode optical fiber imaging method, system and device

Abstract

The invention relates to the technical field of optical imaging, in particular to a multimode optical fiber imaging method, system and device based on deep learning. The method comprises the steps that a hologram selected after mask processing is loaded into an optical modulator, so that light beams output by a multimode optical fiber achieve mask scanning imaging, and the hologram is used for achieving phase modulation of the optical modulator; and collecting an image obtained by mask scanning imaging, and restoring the image based on the pre-trained deep learning network model. According to the invention, the light modulator is modulated through the hologram selected after mask processing, so that when the light beam modulated by the light modulator images the sample to be detected through the multimode optical fiber, mask scanning imaging can be realized, that is, the imaging speed is improved and the luminous flux is reduced by reducing the number of sampling points. Meanwhile, a deep learning network model is further adopted for restoration, and a data basis is provided for subsequent accurate analysis of the to-be-tested sample.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and particularly to a multimode fiber imaging method, system and device based on deep learning. Background Art

[0002] With the development of minimally invasive medicine and high-resolution imaging technology, the demand for endoscopic imaging in the fields of medical diagnosis, biological research, etc. is increasing day by day. Due to the advantages of small diameter, good flexibility and low cost, multimode fibers are considered to be an important solution for realizing high-resolution endoscopic imaging. Since multimode fibers have multiple modes, there are intermodal coupling and modal dispersion effects, and the optical field information output by the multimode fiber is highly aliased, and a clear image cannot be directly obtained.

[0003] Existing multimode fiber endoscopic imaging systems use wavefront shaping technology to modulate the speckle output by the fiber and then focus it, and use point scanning imaging. However, using point scanning imaging has the problems of long scanning time and slow imaging. Summary of the Invention

[0004] In view of this, the present invention provides a multimode fiber imaging method and device based on deep learning to solve the problems of long scanning time and slow imaging existing in the use of point scanning imaging in multimode fiber endoscopic imaging in the prior art.

[0005] In a first aspect, the present invention provides a multimode fiber imaging method based on deep learning, which is applied to a multimode fiber imaging system. The system modulates a light beam by a light modulator and then inputs it into a multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging on a sample to be measured. The method includes: loading a selected hologram after mask processing into the light modulator, so that the light beam output by the multimode fiber realizes mask scanning imaging, where the hologram is a hologram for realizing phase modulation of the light modulator; collecting the image obtained by mask scanning imaging, and restoring the image based on a pre-trained deep learning network model.

[0006] In the present invention, the light modulator is modulated by the selected hologram after mask processing. Thus, when the light beam modulated by the light modulator passes through the multimode fiber to image the sample to be measured, mask scanning imaging can be realized, that is, the imaging speed is increased and the light flux is reduced by reducing the number of sampling points. At the same time, further restoration is performed using a deep learning network model, providing a data basis for the subsequent accurate analysis of the sample to be measured.

[0007] In an alternative embodiment, before loading the selected hologram after masking into the optical modulator, the method further includes: inputting a Hadamard matrix pattern into the optical modulator; obtaining the optical field intensity after the interference of the signal light and the reference light signal modulated by the optical modulator, and determining the optical field phase based on the four-step phase-shifting principle; calculating a transmission matrix according to the Hadamard matrix pattern and the optical field phase; and determining a hologram for realizing the phase modulation of the optical modulator based on the transmission matrix by using the Lee holography principle.

[0008] In the present invention, a transmission matrix is determined by using a Hadamard matrix pattern and an optical field phase determined based on the four-step phase-shifting principle. Based on this transmission matrix, a hologram constructed by using the Lee holography principle realizes the phase modulation of light modulation, avoiding the defect of amplitude modulation of the DMD. At the same time, the determination of the hologram also provides a data basis for the subsequent masking process.

[0009] In an alternative embodiment, loading the selected hologram after masking into the optical modulator includes: loading the selected hologram after masking into the optical modulator by using a pre-trained deep learning network model. The selected hologram after masking includes 25% of the total number of holograms.

[0010] In the present invention, a scanning method of 25% of the pixel points is adopted. This scanning method can quadruple the speed of the system for collecting pictures, and greatly reduce the optical flux, reducing the damage to the sample caused by light illumination.

[0011] In an alternative embodiment, loading the selected hologram after masking into the optical modulator further includes: when the sample to be measured is a sparse sample, obtaining the effective information region of the sparse sample; and loading the selected hologram after masking into the optical modulator based on the effective information region.

[0012] In the present invention, for a sparse sample, only the effective information region is masked and scanned, which can further double the speed on the basis of the speed improvement.

[0013] In an alternative embodiment, when the deep learning network model is the Swin MAE model, the deep learning network model is pre-trained in the following manner: dividing the image in the training data into masked small blocks of a preset size by using a window mask and then performing masking to obtain a masked image; extracting the features of the masked image by using an encoder; predicting the masked part of the image by using a decoder based on the features of the image; updating the parameters of the encoder and the decoder according to the difference between the prediction result and the original image, and repeating the above process until a preset condition is reached to obtain a trained deep learning network model.

[0014] In the present invention, the Swin MAE model is adopted for image restoration, reducing the requirements for the data volume of the dataset and training conditions.

[0015] In a second aspect, the present invention provides a multi-mode fiber optic imaging system based on deep learning. The system includes: a control module for loading a selected hologram after mask processing into a light modulator, where the hologram is a hologram for realizing the phase modulation of the light modulator; a laser optical path module including a light source and a light modulator, where the light modulator is used to modulate the light beam output by the light source; a multi-mode optical fiber for transmitting the modulated light beam to irradiate a sample to be measured; a fluorescence detection module for receiving the fluorescence signal reflected by the sample to be measured; and the control module is further used to receive the fluorescence signal to generate a fluorescence imaging map and restore the fluorescence imaging map using a pre-trained deep learning network model.

[0016] In an optional implementation manner, the system further includes: an interference module; the control module is further used to input a Hadamard matrix diagram to the light modulator; the interference module is used to receive the interference signal output after interference inside the multi-mode optical fiber, or interfere the signal output by the multi-mode optical fiber with a reference light to generate an interference signal, and convert the interference signal into the light field intensity of the light beam; the control module is further used to calculate a transmission matrix based on the four-step phase shift principle according to the Hadamard matrix diagram and the phase corresponding to the light field intensity; and based on the transmission matrix, determine the hologram for realizing the phase modulation of the light modulator using the Li holography principle. The hologram includes a plurality of different holograms, and different holograms are loaded into the light modulator so that the light beam output by the multi-mode optical fiber realizes focusing at different positions.

[0017] In an optional implementation manner, the control module is further used to control the light modulator and the fluorescence detection module to work synchronously according to a synchronization pulse.

[0018] In a third aspect, the present invention provides a multi-mode fiber optic imaging device based on deep learning, which is applied to a multi-mode fiber optic imaging system. The system modulates a light beam using a light modulator and then inputs it into a multi-mode optical fiber. The light beam output by the multi-mode optical fiber is used to perform focused scanning imaging on a sample to be measured. The device includes: a mask modulation module for loading a selected hologram after mask processing into the light modulator so that the light beam output by the multi-mode optical fiber realizes mask scanning imaging, where the hologram is a hologram for realizing the phase modulation of the light modulator; and a restoration module for collecting the image obtained by mask scanning imaging and restoring the image based on a pre-trained deep learning network model.

[0019] Fourth aspect, the present invention provides a computer device, comprising: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the multi-mode fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof as described above.

[0020] Fifth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored. The computer instructions are used to cause a computer to execute the multi-mode fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof as described above.

[0021] Sixth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to cause a computer to execute the multi-mode fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof as described above. Description of the Drawings

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required to be used in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a schematic flowchart of the multi-mode fiber imaging method based on deep learning according to an embodiment of the present invention; Figure 2 is a schematic imaging optical path diagram of the multi-mode fiber imaging system based on deep learning according to an embodiment of the present invention; Figure 3 is a schematic working principle diagram of the multi-mode fiber imaging system based on deep learning according to an embodiment of the present invention; Figure 4 is a schematic diagram of the deep learning network model architecture according to an embodiment of the present invention; Figure 5 is a schematic scanning mode diagram with n = 2 as an example according to an embodiment of the present invention; Figure 6 is a point focusing result diagram of the multi-mode fiber imaging system based on deep learning according to an embodiment of the present invention; Figure 7 is a schematic image restoration effect diagram according to an embodiment of the present invention; Figure 8 is a structural block diagram of the multi-mode fiber imaging device based on deep learning according to an embodiment of the present invention; Figure 9It is a schematic diagram of the hardware structure of the computer device according to an embodiment of the present invention. Detailed implementation manners

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0025] According to an embodiment of the present invention, an embodiment of a multimode fiber imaging method based on deep learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0026] In this embodiment, a multimode fiber imaging method based on deep learning is provided, which can be used in electronic devices such as computers, mobile phones, and tablet computers. Figure 1 It is a flowchart of the multimode fiber imaging method based on deep learning according to an embodiment of the present invention. As Figure 1 shown, the process includes the following steps: Step S101: Load the selected hologram after mask processing into the optical modulator, so that the light beam output by the multimode fiber realizes mask scanning imaging, and the hologram is a hologram for realizing the phase modulation of the optical modulator.

[0027] Step S102: Collect the image obtained by mask scanning imaging, and restore the image based on a pre-trained deep learning network model.

[0028] Among them, this imaging method is applied to a multimode fiber imaging system. In this system, an optical modulator is used to modulate the light beam and then input it into the multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging on the sample to be measured. Based on the related technology, it is known that point scanning imaging is mostly used during scanning imaging, resulting in a large number of sampling points and a slow imaging speed. Therefore, in this embodiment, the optical modulator is modulated by using the selected hologram after mask processing. Thus, when the light beam modulated by the optical modulator passes through the multimode fiber to image the sample to be measured, mask scanning imaging can be realized, that is, the imaging speed is increased and the light flux is reduced by reducing the number of sampling points.

[0029] Specifically, in this embodiment, a DMD (Digital Micromirror Device) is used as the optical modulator. When modulating the DMD, it can only perform amplitude modulation by itself and cannot meet the phase modulation requirements needed during regulation. Therefore, a hologram generated using the Lee holographic principle is used to achieve phase modulation of the DMD. Among them, the Lee holographic principle encodes phase information onto the amplitude distribution, generates a special hologram, and loads it onto the DMD to achieve equivalent phase modulation.

[0030] In this embodiment, however, the generated multiple holograms are not directly loaded onto the DMD. Instead, they are first masked, that is, masked processing and selection are performed on the multiple holograms to obtain the selected hologram, and the selected hologram is loaded onto the DMD to achieve phase modulation of the DMD. Thus, when the modulated DMD modulates the input light beam, scanning imaging of the corresponding masked manner of the sample to be measured can be achieved. It should be noted that multiple holograms corresponding to scanning different positions of the sample to be measured are generated using the Lee holographic principle, that is, each hologram corresponds to the position of a point on the sample to be measured. Masked processing and selection of the multiple holograms means removing the holograms corresponding to the positions that do not need to be scanned, that is, removing a partial number of holograms from the multiple holograms, and only inputting the remaining number of holograms into the DMD.

[0031] In addition, since the imaging image is obtained through a masked scanning method, that is, the obtained imaging image is incomplete, this embodiment pre-trains a deep learning network model, which can restore the obtained imaging image to obtain a complete image. Subsequently, analysis and other processing of the sample to be measured can be performed based on this complete image.

[0032] In this embodiment, a multimode fiber imaging method based on deep learning is provided. The process includes the following steps: Step S201, input a Hadamard matrix diagram into the optical modulator.

[0033] Step S202, obtain the optical field intensity after the interference of the signal light and the reference light signal modulated by the optical modulator, and determine the optical field phase based on the four-step phase shift principle.

[0034] Step S203, calculate the transfer matrix according to the Hadamard matrix diagram and the optical field phase.

[0035] Step S204, based on the transfer matrix, use the Lee holographic principle to determine the hologram for realizing the phase modulation of the optical modulator. The hologram includes multiple different holograms. Different holograms are loaded into the optical modulator, so that the light beam output by the multimode fiber can be focused at different positions.

[0036] Specifically, in order to enable the DMD after hologram modulation to modulate the light beam so that the light beam can be focused on any position of the sample to be measured, the corresponding hologram needs to be determined first. Based on this, in this embodiment, the transmission matrix of the optical modulator is determined by the above steps first.

[0037] First, a Hadamard matrix pattern is input into the optical modulator. Specifically, 64×64 groups of 4 Hadamard matrix patterns with a phase interval of π / 2 can be input respectively. The Hadamard matrix is a special orthogonal matrix with elements only +1 and -1. Different Hadamard matrix patterns can modulate the light field differently. Here, 4 matrix patterns with a phase interval of π / 2 in each group are used to introduce different phase information into the light field. After that, after inputting a group of Hadamard matrix patterns, the light field intensity after the interference of the modulated light beam and the reference signal is collected once, and the light intensity information corresponding to 4 different phases (i.e., the phases corresponding to 4 Hadamard matrix patterns with a phase interval of π / 2) is obtained; then, based on the four-step phase shift principle, the light field phase information is calculated. The specific calculation process can be implemented with reference to related technologies.

[0038] When the light field phases corresponding to different Hadamard matrix patterns are determined, the transmission matrix can be calculated. The transmission matrix describes the transformation relationship of the system to the amplitude and phase of the light field during the process of the light field from input to output. By determining the transmission matrix, the characteristics changes of the light field at each position after the light field passes through the DMD and the multimode fiber can be accurately known, so as to realize the control of focusing at different positions at the output end of the multimode fiber. Specifically, after the known input Hadamard matrix pattern (representing the modulation mode of the input light field) and the output light field (including phase and amplitude information) measured by methods such as four-step phase shift are obtained, the transmission matrix can be calculated by using algorithms related to linear algebra and optical propagation theory. Finally, when the transmission matrix is determined, the corresponding hologram can be determined in combination with the Li holography principle.

[0039] Step S205: Load the selected hologram after mask processing into the optical modulator, so that the light beam output by the multimode fiber realizes mask scanning imaging. The hologram is a hologram for realizing the phase modulation of the optical modulator; for details, please refer to Figure 1 Step S101 of the illustrated embodiment, which will not be elaborated here.

[0040] Step S206: Collect the image obtained by mask scanning imaging and restore the image based on a pre-trained deep learning network model. For details, please refer to Figure 1 Step S102 of the illustrated embodiment, which will not be elaborated here.

[0041] In this embodiment, a multimode fiber imaging method based on deep learning is provided. The method includes the following steps: Step S301: The images in the training data are divided into masked patches of a preset size using a window mask and then masked to obtain the masked images.

[0042] Step S302: An encoder is used to extract the features of the masked images.

[0043] Step S303: Based on the features of the images, a decoder is used to predict the occluded parts of the images.

[0044] Step S304: The parameters of the encoder and decoder are updated according to the difference between the prediction results and the original images, and the above process is repeated until a preset condition is met to obtain the trained deep learning network model.

[0045] Specifically, for the deep learning network model used in this embodiment, the Swin MAE (Swin Masked Autoencoders) model can be used, or the GAN (Generative Adversarial Networks) model can be used. Among them, the Swin MAE model uses the Swin Transformer as the backbone network. The Swin Transformer adopts a hierarchical structure and a sliding window mechanism, which can effectively process images of different scales and capture local and global information. Using the Swin MAE model for image restoration reduces the requirements for the amount of data in the dataset and training conditions.

[0046] Specifically, the Swin MAE model is trained in the following way: The original images in the training dataset are divided into n×n masked patches through the window mask process, and 75% of them are occluded, which is similar to sampling 25% of the parts in a specific masking manner in real system acquisition. Subsequently, the images are input into the encoder of the model as input, converted into the sequences required by the encoder, and then feature extraction is performed in the form of windows. The occluded parts are predicted in the decoder, and finally restored to the original image sequence as output. Then, this process is repeated to iteratively update the parameters of the encoder and decoder. When the relevant conditions are met, the iterative process is stopped to obtain the trained model.

[0047] Step S305: Load the selected hologram after masking into the optical modulator so that the beam output from the multimode fiber realizes masked scanning imaging. The hologram is a hologram for realizing the phase modulation of the optical modulator. Specifically, during the training process of the Swin MAE model, when using window mask for masking, a set of masks is first generated, and then the original image is covered with the masks. Thus, a specific set of mask outputs during the model training process can be used to mask and select multiple holograms. Among them, when the mask is 0, no sampling is performed, and when the mask is 1, sampling is performed. The mask represents whether to perform focused scanning at different positions within the field of view. In the Lee holographic method based on the transfer matrix, the focus point at each position corresponds to a pre-calculated phase diagram. Therefore, directly inputting the phase diagram corresponding to the position where focused sampling is required, that is, the hologram after mask processing and selection, into the DMD can realize scanning imaging in a masked manner.

[0048] In an alternative embodiment, when the sample to be measured is a sparse sample, obtain the effective information region of the sparse sample; based on the effective information region, load the selected hologram after masking into the optical modulator. Specifically, in endoscopic imaging, the overall sample contains a large amount of data, but the information valuable for research or diagnosis is not evenly distributed. For example, in biomedical samples, only specific regions such as certain cells and molecular structures may contain information that can reflect the physiological and pathological states. These regions are the effective information regions, while most other regions may be background substances, etc., with low or no valuable information. Thus, only the effective information region of the sample to be measured can be sampled.

[0049] Specifically, before sampling, the effective information region of the sparse sample can be determined first, and then multiple holograms corresponding to sampling the effective information region are determined (the number of these holograms is less than the number of holograms for sampling the entire sample to be measured). Then, mask processing and selection are performed on the multiple holograms, that is, 25% of the holograms among the multiple holograms are retained. For example, if there are 100 holograms, 25 holograms are retained. Thus, in the case of a sparse sample, a mask pattern with a 25% random masked scanning method is set only for the effective information part, thereby further doubling the rate on the basis of the original four times.

[0050] Step S306: Collect the image obtained by masked scanning imaging, and restore the image based on a pre-trained deep learning network model. For details, please refer to Figure 1 Step S102 of the embodiment shown, which will not be elaborated here.

[0051] In this embodiment, a multimode fiber imaging system based on deep learning is further provided. The system includes: a control module for loading a selected hologram after masking processing into a light modulator, where the hologram is a hologram for realizing the phase modulation of the light modulator; a laser optical path module including a light source and a light modulator, where the light modulator is used to modulate the light beam output by the light source; a multimode fiber for transmitting and irradiating the modulated light beam on a sample to be measured; a fluorescence detection module for receiving the fluorescence signal reflected by the sample to be measured; and the control module is further used to receive the fluorescence signal to generate a fluorescence imaging map, and restore the fluorescence imaging map using a pre-trained deep learning network model.

[0052] In an alternative embodiment, the control module is further used to input a Hadamard matrix diagram to the light modulator; the interference module is used to receive the interference signal output after interference inside the multimode fiber, or interfere the signal output by the multimode fiber with a reference light to generate an interference signal, and convert the interference signal into the light field intensity of the light beam; the control module is further used to calculate a transmission matrix based on the four-step phase shift principle according to the Hadamard matrix diagram and the phase corresponding to the light field intensity; based on the transmission matrix, using the principle of Lee holography, determine the hologram for realizing the phase modulation of the light modulator, where the hologram includes a plurality of different holograms, and different holograms are loaded into the light modulator so that the light beam output by the multimode fiber realizes focusing at different positions.

[0053] Specifically, in this imaging system, the optical path used for imaging is as Figure 2 shown: A continuous laser 1 with a wavelength of 532 nm emits laser light, which is expanded by a first 4f system 2 (including a third lens 0201, a small hole 0202, and a fourth lens 0203). The small hole 0202 filters the light beam to eliminate stray waves. A half-wave plate 3 and a polarization beam splitter prism 4 are used to adjust the power of the light beam, and at the same time, a reference light is split off and enters a beam splitter 17 through a seventh mirror 24, an eighth mirror 25, a ninth mirror 26, and a tenth mirror 27 for interference with the light beam after passing through the multimode fiber. The other light beam passes through the polarization beam splitter prism 4 and is further expanded by a second 4f system 5 (including a fifth lens 0501 and a sixth lens 0502), and then irradiates on a DMD 8 after passing through a first mirror 6 and a second mirror 7.

[0054] The light beam modulated by DMD8 enters the third 4f system 9. The seventh lens 0901 performs the Fourier transform of the light beam. The aperture 0902 selects the diffracted-order reflected light of the target light field on the Fourier plane. The eighth lens 0903 performs the inverse Fourier transform of the light beam. The light beam emitted from the third 4f system 9 passes through the third mirror 10 and the fourth mirror 11 and then enters the dichroic mirror 12. Since the light beam is less than the cut-off wavelength after passing through the dichroic mirror 12, it is reflected. Subsequently, the light beam passes through the fifth mirror 13, the first objective lens 14, and is coupled into the multimode fiber section 15 (including the first fiber collimator 1501, the multimode fiber 1502, and the sample stage 1503). The light beam passing through the multimode fiber 1502 enters the second objective lens 16 for collimation. The collimated light beam interferes with the reference light through the beam splitter 17. The interfered light beam is focused by the lens 18 into the camera 19 and the light field is calculated and modulated by the computer terminal 20.

[0055] When imaging the sample, the multimode fiber 1502 is inserted into the sample placed on the sample stage 1503. The light beam output from the multimode fiber hits the sample and excites fluorescence. The fluorescence is reflected back to the multimode fiber 1502 and transmitted back along the original path to the dichroic mirror 12. Since the wavelength of the reflected fluorescence is greater than the cut-off wavelength of the dichroic mirror 12, the fluorescence transmits through the dichroic mirror 12, passes through the sixth mirror 21, enters the second lens 22, and then the light beam is focused into the PMT (Photomultiplier Tube) 23. The PMT 23 converts the received fluorescence signal into an electrical signal and inputs it into the computer terminal 20, where it is converted into fluorescence intensity information at different positions, thus realizing imaging.

[0056] Among them, before obtaining the information collected by the camera 19, the DMD inputs 64×64 groups of Hadamard matrix diagrams with 4 phase intervals of pi / 2 in each group. Based on this matrix diagram, the DMD modulates the received light beam. The modulated light beam is transmitted to the camera 19 in the above manner. The camera 19 collects the light field intensity signal of the received light beam and sends it to the computer terminal 20. Then, the four-step phase-shift principle is used to calculate this light field intensity signal, so as to obtain the phase of the corresponding light field. The transmission matrix of the system is calculated under the condition of known input Hadamard matrix and output light field. Since the DMD can only perform amplitude modulation, the Lee holography principle is needed to make the DMD perform phase modulation. Specifically, based on the transmission matrix, the Lee holography principle is used to determine the hologram for realizing phase modulation. The computer terminal loads this hologram onto the DMD. At this time, the light beam modulated by the DMD irradiates the sample to be measured through the above optical path, and mask scanning imaging can be realized. That is, the fluorescence intensity information obtained by the above computer terminal 20 only contains partial information of the sample to be measured, and a pre-trained deep learning network model needs to be further used to restore the imaging.

[0057] It should be noted that the optical path used in the above imaging is in the form of interference with an external reference light. Further, the seventh to tenth reflectors in the optical path can also be removed, and the reference light and the signal light are transmitted through the same optical path, that is, the reference light also passes through the above optical path and is transmitted into the multimode fiber, and interferes with the internal reference light in the multimode fiber.

[0058] In an alternative embodiment, the control module is further configured to control the optical modulator and the fluorescence detection module to work synchronously according to a synchronization pulse. Specifically, a signal synchronization module can be further set in the system. The control module sends the synchronization pulse to the signal synchronization module, and the signal synchronization module sends synchronization signals to the DMD and the PMT according to the synchronization pulse to achieve the synchronous operation of the DMD and the PMT.

[0059] As a specific application embodiment of the embodiment of the present invention, as Figure 3 shown, the multimode fiber imaging system based on deep learning specifically includes: An excitation optical path module, which emits laser light from a continuous laser 1 with a wavelength of 532 nm as shown in Figure 2 . The laser light is expanded by a first 4f system 2, and the small hole 0202 filters the light beam to eliminate stray light. A half-wave plate 3 and a polarization beam splitter prism 4 are used to adjust the power of the light beam, and at the same time, a reference light is split off for interference with the light beam after passing through the multimode fiber. The other light beam passes through the polarization beam splitter prism 4 and is further expanded by a second 4f system 5 and then irradiated on the DMD.

[0060] A DMD random scanning module. The DMD random scanning module modulates the speckles output from the multimode fiber by using the DMD, so that they are focused and can perform random point scanning. The main principle is that the DMD inputs 64×64 groups of Hadamard matrix diagrams with 4 phase intervals of pi / 2 in each group. Using the four-step phase-shifting principle, the light field intensity obtained by the camera after interference with the internal reference light is calculated, so as to obtain the phase of the corresponding light field. Given the input Hadamard matrix and the output light field, the transmission matrix of the system is calculated, and then the focusing at different positions at the output end of the multimode fiber is achieved. Among them, since the DMD can only perform amplitude modulation, the Li holographic principle needs to be used to make the DMD perform phase modulation. As shown in Figure 2, the computer terminal 20 controls the DMD8 to calculate and input the corresponding hologram. The seventh lens 0901 realizes the Fourier transform of the light beam. The aperture 0902 realizes the selection of the diffracted-order reflected light of the target light field on the Fourier plane. The eighth lens 0903 realizes the inverse Fourier transform of the light beam. After the light beam passes through the dichroic mirror 12, it is reflected because it is less than the cut-off wavelength. Subsequently, the light beam passes through the first objective lens 14 and is coupled into the multimode optical fiber. The light beam passing through the multimode optical fiber enters the second objective lens 16 for collimation. The collimated light beam is interfered with the reference light through the beam splitter 17. The interfered light beam is focused by the lens 18 into the camera 19 and the computer terminal 20 performs light field calculation and modulation.

[0061] The sample module, such as Figure 2 shown, the first fiber collimator 1501 in the sample module is used to couple the collimated light beam into the multimode optical fiber. The multimode optical fiber 1502 is inserted into the sample placed on the sample stage 1503, and the light beam passing through the multimode optical fiber performs imaging by point scanning of the field of view range in the sample.

[0062] The PMT detection module, such as Figure 2 shown, after the focused light beam hits the sample, fluorescence is excited. The fluorescence is reflected back into the multimode optical fiber 1502 and passes through the multimode optical fiber. Since the wavelength of the reflected fluorescence is greater than the cut-off wavelength of the dichroic mirror 12, the fluorescence transmits through the dichroic mirror 12. After entering the second lens 22, the light beam is focused into the PMT23. In the PMT23, the fluorescence signal is converted into an electrical signal and transmitted to the signal processing module, and the signal processing module calculates the fluorescence signal obtained by the PMT.

[0063] The signal processing module mainly includes the light field intensity signal, the fluorescence intensity signal and the synchronization signal. Such as Figure 2 shown, after receiving the light field intensity signal transmitted by the camera 19, the signal processing part calculates the corresponding phase of the light field by using the four-step phase-shifting method, calculates the transmission matrix of the system and further calculates the hologram required for focusing at the corresponding position points. After the fluorescence enters the PMT23, it is converted into an electrical signal and further input into the signal processing module, and the electrical signal is converted into fluorescence intensity information at different positions, thereby realizing imaging. The signal processing module will also send out synchronization pulses to synchronize the DMD modulation scanning and the PMT signal acquisition.

[0064] The model processing module includes the model training process and the model inference process. Such as Figure 4As shown in the figure. During the training process, the original images in the training dataset are divided into n×n masked patches through the window mask process, and 75% of them are masked, which is similar to sampling 25% of the real system acquisition in a specific masking manner. Subsequently, the images are input into the encoder of the model as input, converted into the required sequence of the encoder, and feature extraction is performed in the form of a window. Then, the masked part is predicted in the decoder, and finally restored to the original image sequence as the output. During the inference process, the model receives the fluorescence image scanned and imaged in the form of specific masked patches collected by the system. The fluorescence image is fed into the model as input and restored through the trained model to obtain the complete original image.

[0065] The mask modulation module, as Figure 5 shown, the mask modulation module needs to provide the phase map corresponding to the required mask for the DMD. Figure 5 It is divided into sequential scanning in the normal case and the random scanning method of the present invention. Among them, in the random scanning method, generally n×n pixels (in this embodiment, n = 2) are used as a masked patch (pixel block), and only 25% of the field of view needs to be randomly masked and scanned to generate the corresponding mask map ( Figure 5 as shown below). In the case of sparse samples, it is set to perform a random mask scan of 25% only on the effective information part to generate the mask map, thereby further doubling the rate on the basis of the original four times. The mask modulation module needs to convert the random mask image into the phase map required by the DMD and send it to the DMD for scanning and imaging.

[0066] The signal synchronization module. The signal synchronization module receives the synchronization pulse given by the signal processing module, and then gives synchronization signals to the DMD scanning module and the PMT signal acquisition module, so that the system can realize the acquisition of fluorescence intensity signals at random positions.

[0067] As Figure 6 shown, it is the point focusing result diagram of the above imaging system. In Figure 6 it, the three upper diagrams represent three holograms scanned at different positions input to the DMD, and the three lower diagrams represent the diagrams of different positions of the spots focused by the multi-mode fiber after inputting the hologram and received by the camera 19. As Figure 7 shown, it is the effect diagram restored by using the deep learning network model for the application instance Instance. Among them, INPUT is the diagram input to the model after masking 75%, output is the predicted diagram after outputting from the model, and GT is the original diagram of this diagram, that is, the evaluation criterion for the predicted diagram.

[0068] In the present invention, since a mask block of n×n pixels is used for scanning imaging and only 25% of the pixel points of the entire field of view are scanned, the imaging speed of the multimode fiber imaging system is increased by four times by means of sparse sampling. At the same time, if a mask sampling method that only includes 25% of the effective information part in the sparse sample is used, the speed can be further doubled on the basis of four times. And due to the reduction in the number of scanned points, the optical flux is greatly reduced, thereby reducing photobleaching and protecting the sample activity and biological functions. At the same time, the Swin MAE model is used for image restoration, reducing the requirements for the data volume of the data set and the training conditions, making it applicable to biomedical imaging systems, and the current image restoration effect is quite good.

[0069] In this embodiment, a multimode fiber imaging device based on deep learning is also provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0070] This embodiment provides a multimode fiber imaging device based on deep learning, which is applied to a multimode fiber imaging system. The system modulates a light beam with a light modulator and then inputs it into a multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging on a sample to be measured, as Figure 8 shown, the device includes: A mask modulation module 81, configured to load the selected hologram after mask processing into the light modulator, so that the light beam output by the multimode fiber realizes mask scanning imaging, and the hologram is a hologram for realizing the phase modulation of the light modulator; A restoration module 82, configured to collect the image obtained by mask scanning imaging and restore the image based on a pre-trained deep learning network model.

[0071] The further function descriptions of the above-mentioned respective modules are the same as those in the corresponding above-mentioned embodiments, and will not be repeated here.

[0072] An embodiment of the present invention also provides a computer device having the above-mentioned Figure 8 shown multimode fiber imaging device based on deep learning.

[0073] Please refer to Figure 9 , Figure 9 is a schematic structural diagram of a computer device provided by an alternative embodiment of the present invention, as Figure 9As shown, the computer device includes: one or more processors 100, a memory 200, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some alternative embodiments, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories if needed. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (such as an array of servers, a set of blade servers, or a multi-processor system). Figure 9 In the figure, a processor 100 is taken as an example.

[0074] The processor 100 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 100 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a general array logic, or any combination thereof.

[0075] Among them, the memory 200 stores instructions executable by at least one processor 100, so that at least one processor 100 executes the method shown in the above embodiments.

[0076] The memory 200 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the computer device presented by a kind of landing page of a small program, etc. In addition, the memory 200 can include high-speed random access memory and can also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 200 can optionally include a memory remotely set relative to the processor 100, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0077] The memory 200 can include volatile memory, such as random access memory; the memory can also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive; the memory 200 can also include a combination of the above types of memory.

[0078] The computer device further includes a communication interface 300 for the computer device to communicate with other devices or a communication network.

[0079] Embodiments of the present invention also provide a computer-readable storage medium. The method according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0080] A part of the present invention can be applied as a computer program product, for example, computer program instructions. When executed by a computer, through the operation of the computer, the methods and / or technical solutions according to the present invention can be called or provided. Those skilled in the art should be able to understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0081] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A multimode fiber optic imaging method based on deep learning, characterized in that, Applied to a multimode fiber imaging system, which uses an optical modulator to modulate a light beam and then inputs it into a multimode fiber, and the light beam output by the multimode fiber is used for focusing and scanning imaging of a sample to be measured. The method includes: Loading the selected hologram after mask processing into the optical modulator, so that the light beam output by the multimode fiber realizes mask scanning imaging, and the hologram is a hologram for realizing the phase modulation of the optical modulator; Collecting the image obtained by mask scanning imaging and restoring the image based on a pre-trained deep learning network model.

2. The method according to claim 1, characterized in that, Before loading the selected hologram after mask processing into the optical modulator, the method further includes: Inputting a Hadamard matrix pattern into the optical modulator; Obtaining the optical field intensity after the interference of the signal light and the reference light signal modulated by the optical modulator, and determining the optical field phase based on the four-step phase shift principle; Calculating a transmission matrix according to the Hadamard matrix pattern and the optical field phase; Based on the transmission matrix, using the principle of Lee holography to determine the hologram for realizing the phase modulation of the optical modulator.

3. The method according to claim 1, characterized in that, Loading the selected hologram after mask processing into the optical modulator includes: Using a pre-trained deep learning network model to load the selected hologram after mask processing into the optical modulator, and the selected hologram after mask processing includes 25% of the total number of holograms.

4. The method according to claim 1, characterized in that, Loading the selected hologram after mask processing into the optical modulator further includes: When the sample to be measured is a sparse sample, obtaining the effective information region of the sparse sample; Loading the selected hologram after mask processing into the optical modulator based on the effective information region.

5. The method according to claim 1, wherein When the deep learning network model is the Swin MAE model, the deep learning network model is pre-trained in the following way: Using a window mask to divide the images in the training data into masked small blocks of a preset size and then performing masking to obtain the masked images; Using an encoder to extract the features of the masked images; Based on the features of the images, using a decoder to predict the masked parts of the images; Updating the parameters of the encoder and decoder according to the difference between the prediction result and the original image, and repeating the above process until a preset condition is reached to obtain the trained deep learning network model.

6. A multimode fiber optic imaging system based on deep learning, characterized in that, The system includes: A control module for loading the selected hologram after mask processing into the optical modulator, and the hologram is a hologram for realizing the phase modulation of the optical modulator; A laser optical path module including a light source and an optical modulator, and the optical modulator is used for modulating the light beam output by the light source; A multimode fiber for transmitting and irradiating the modulated light beam on the sample to be measured; A fluorescence detection module for receiving the fluorescence signal reflected by the sample to be measured; The control module is further used for receiving the fluorescence signal to generate a fluorescence imaging map and restoring the fluorescence imaging map using a pre-trained deep learning network model.

7. The system according to claim 6, wherein The system further includes: an interference module; The control module is further used for inputting a Hadamard matrix pattern into the optical modulator; The interference module is used to receive the interference signal output after the interference inside the multimode optical fiber, or to interfere the signal output by the multimode optical fiber with the reference light to generate an interference signal, and convert the interference signal into the optical field intensity of the light beam; The control module is further configured to calculate the transmission matrix based on the four-step phase shift principle according to the Hadamard matrix diagram and the phase corresponding to the optical field intensity; based on the transmission matrix, adopt the Li holography principle to determine the hologram for realizing the phase modulation of the optical modulator, and the hologram includes a plurality of different holograms, and different holograms are loaded into the optical modulator, so that the light beam output by the multimode optical fiber realizes focusing at different positions.

8. The system according to claim 6, wherein The control module is further configured to control the optical modulator and the fluorescence detection module to work synchronously according to the synchronization pulse.

9. A multimode fiber optic imaging device based on deep learning, characterized in that, Applied to a multimode optical fiber imaging system, the system uses an optical modulator to modulate a light beam and then inputs it into a multimode optical fiber, and the light beam output by the multimode optical fiber is used to perform focused scanning imaging on a sample to be measured. The device includes: A mask modulation module, configured to load the selected hologram after mask processing into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging, and the hologram is the hologram for realizing the phase modulation of the optical modulator; A restoration module, configured to collect the image obtained by mask scanning imaging and restore the image based on a pre-trained deep learning network model.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the deep learning-based multimode optical fiber imaging method according to any one of claims 1 to 5.

Images