A multimode optical fiber imaging method, system and device based on deep learning

Abstract

The present invention relates to the field of optical imaging technology, and in particular to a multimode optical fiber imaging method, system and device based on deep learning. The method comprises: loading a hologram selected after mask processing into an optical modulator so that the light beam output by the multimode optical fiber realizes mask scanning imaging, and the hologram is a hologram used to realize phase modulation of the optical modulator; collecting an image obtained by mask scanning imaging, and restoring the image based on a pre-trained deep learning network model. In the present invention, the optical modulator is modulated by the hologram selected after mask processing, so that when the light beam modulated by the optical modulator passes through the multimode optical fiber to image the sample to be tested, mask scanning imaging can be realized, that is, the imaging speed is improved and the light flux is reduced by reducing the number of sampling points. At the same time, a deep learning network model is further used for restoration, which provides a data basis for the subsequent accurate analysis of the sample to be tested.

Description

Technical Field

[0001] The present invention relates to the field of optical imaging technology, and in particular to a multimode optical fiber imaging method, system, and device based on deep learning. Background Art

[0002] With the advancement of minimally invasive medical treatment and high-resolution imaging technologies, the demand for endoscopic imaging is growing in fields such as medical diagnosis and biological research. Multimode fiber, due to its small diameter, flexibility, and low cost, is considered a key solution for achieving high-resolution endoscopic imaging. However, due to the multiple modes of multimode fiber, intermodal coupling and modal dispersion can occur, resulting in a high degree of aliasing of the light field information output through multimode fiber, making it impossible to directly obtain clear images.

[0003] Existing multimode fiber endoscopic imaging systems use wavefront shaping technology to modulate the speckle pattern output by the fiber and then focus it, and then use point scanning to form images. However, point scanning imaging has the problem of long scanning time and slow imaging. Summary of the Invention

[0004] In view of this, the present invention provides a multimode fiber optic imaging method and device based on deep learning to solve the problem of long scanning time and slow imaging in the prior art of point scanning imaging in multimode fiber endoscopic imaging.

[0005] In the first aspect, the present invention provides a multimode fiber imaging method based on deep learning, which is applied to a multimode fiber imaging system. The system uses an optical modulator to modulate the light beam and then input it into the multimode optical fiber. The light beam output by the multimode optical fiber is used to focus and scan the sample to be tested. The method includes: loading a hologram selected after mask processing into the optical modulator so that the light beam output by the multimode optical fiber realizes mask scanning imaging, and the hologram is a hologram used to realize phase modulation of the optical modulator; collecting an image obtained by mask scanning imaging, and restoring the image based on a pre-trained deep learning network model.

[0006] In this invention, a light modulator is modulated using a hologram selected after masking. This allows masked scanning imaging to be achieved when the modulated light beam passes through a multimode fiber to image the sample under test. This reduces the number of sampling points, improving imaging speed and reducing light flux. Furthermore, a deep learning network model is employed for restoration, providing a data foundation for subsequent accurate analysis of the sample under test.

[0007] In an optional embodiment, before loading the hologram selected after mask processing into the optical modulator, the method further includes: inputting a Hadamard code matrix diagram into the optical modulator; obtaining the light field intensity after interference between the signal light modulated by the optical modulator and the reference light signal, and determining the light field phase based on the four-step phase shift principle; calculating a transmission matrix based on the Hadamard code matrix diagram and the light field phase; and determining, based on the transmission matrix and using the Lie holographic principle, a hologram used to implement phase modulation of the optical modulator.

[0008] In this invention, a transmission matrix is ​​determined using a Hadamard matrix diagram and the light field phase determined based on the four-step phase shift principle. Based on this transmission matrix, a hologram constructed using the Lie holographic principle achieves phase modulation of light modulation, avoiding the amplitude modulation drawbacks of DMD. The hologram also provides the data foundation for subsequent mask processing.

[0009] In an optional embodiment, the hologram selected after mask processing is loaded into the optical modulator, including: using a pre-trained deep learning network model to load the hologram selected after mask processing into the optical modulator, and the hologram selected after mask processing includes 25% of the total number of holograms.

[0010] The present invention uses a 25% pixel scanning method, which can quadruple the system's image acquisition speed and significantly reduce light flux, minimizing light damage to samples.

[0011] In an optional embodiment, the hologram selected after mask processing is loaded into the optical modulator, and the method further includes: when the sample to be tested is a sparse sample, obtaining the effective information area of ​​the sparse sample; and loading the hologram selected after mask processing into the optical modulator based on the effective information area.

[0012] In the present invention, for sparse samples, only the effective information area is masked and scanned, which can further increase the speed by several times on the basis of the speed increase.

[0013] In an optional embodiment, when the deep learning network model is a Swin MAE model, the deep learning network model is pre-trained in the following manner: a window mask is used to divide the image in the training data into mask blocks of a preset size and then masked to obtain a masked image; an encoder is used to extract features of the masked image; based on the features of the image, a decoder is used to predict the masked part of the image; the parameters of the encoder and decoder are updated according to the difference between the prediction result and the original image, and the above process is repeated until the preset conditions are met to obtain a trained deep learning network model.

[0014] In the present invention, the Swin MAE model is used for image restoration, which reduces the requirements on the amount of data set and the training conditions.

[0015] In second aspect, the present invention provides a multimode fiber imaging system based on deep learning, the system comprising: a control module, for loading a hologram selected after mask processing into an optical modulator, the hologram being a hologram for realizing phase modulation of the optical modulator; a laser optical path module, comprising a light source and an optical modulator, the optical modulator being used to modulate the light beam output by the light source; a multimode optical fiber, for transmitting the modulated light beam to illuminate a sample to be tested; a fluorescence detection module, for receiving a fluorescence signal reflected by the sample to be tested; a control module, further for receiving the fluorescence signal to generate a fluorescence imaging image, and restoring the fluorescence imaging image using a pre-trained deep learning network model.

[0016] In an optional embodiment, the system also includes: an interference module; the control module is also used to input a Hadamard code matrix diagram into the optical modulator; the interference module is used to receive an interference signal output after internal interference in a multimode optical fiber, or to interfere the signal output by the multimode optical fiber with a reference light to generate an interference signal, and convert the interference signal into the light field intensity of the light beam; the control module is also used to obtain a transmission matrix based on the four-step phase shift principle and the phase calculation corresponding to the Hadamard code matrix diagram and the light field intensity; based on the transmission matrix, the Lie holographic principle is used to determine the hologram used to realize phase modulation of the optical modulator, and the hologram includes multiple different holograms, and different holograms are loaded into the optical modulator so that the light beam output by the multimode optical fiber can be focused at different positions.

[0017] In an optional embodiment, the control module is further configured to control the light modulator and the fluorescence detection module to operate synchronously according to a synchronization pulse.

[0018] In the third aspect, the present invention provides a multimode fiber imaging device based on deep learning, which is applied to a multimode fiber imaging system. The system uses an optical modulator to modulate the light beam and then input it into the multimode optical fiber. The light beam output by the multimode optical fiber is used to focus and scan the sample to be tested. The device includes: a mask modulation module, which is used to load the hologram selected after mask processing into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging, and the hologram is a hologram used to realize phase modulation of the optical modulator; a restoration module, which is used to collect the image obtained by mask scanning imaging and restore the image based on a pre-trained deep learning network model.

[0019] In a fourth aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimode optical fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0020] In a fifth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimode optical fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof.

[0021] In a sixth aspect, the present invention provides a computer program product comprising computer instructions, which are used to enable a computer to execute the multimode optical fiber imaging method based on deep learning according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 is a flowchart of a multimode optical fiber imaging method based on deep learning according to an embodiment of the present invention;

[0024] Figure 2 2. Schematic diagram of the imaging optical path of a multimode optical fiber imaging system based on deep learning according to an embodiment of the present invention;

[0025] Figure 3 2. It is a schematic diagram of the working principle of a multimode optical fiber imaging system based on deep learning according to an embodiment of the present invention;

[0026] Figure 4 is a schematic diagram of a deep learning network model architecture according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of a scanning method taking n=2 as an example according to an embodiment of the present invention;

[0028] Figure 6 2. This is a diagram showing the point focusing results of a multimode optical fiber imaging system based on deep learning according to an embodiment of the present invention;

[0029] Figure 7 is an image restoration effect diagram according to an embodiment of the present invention;

[0030] Figure 8 is a structural block diagram of a multimode optical fiber imaging device based on deep learning according to an embodiment of the present invention;

[0031] Figure 9 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0033] According to an embodiment of the present invention, an embodiment of a multimode optical fiber imaging method based on deep learning is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] In this embodiment, a multimode optical fiber imaging method based on deep learning is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a multimode optical fiber imaging method based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0035] Step S101 , loading the selected hologram after mask processing into the optical modulator so that the light beam outputted by the multimode optical fiber realizes mask scanning imaging, wherein the hologram is a hologram for realizing phase modulation of the optical modulator.

[0036] Step S102: Acquire an image obtained by mask scanning imaging, and restore the image based on a pre-trained deep learning network model.

[0037] This imaging method is applied to a multimode fiber imaging system. In this system, a light beam is modulated by an optical modulator and then input into a multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging of the sample to be measured. Based on related art, point scanning imaging is often used during scanning imaging, resulting in a large number of sampling points and slow imaging speed. Therefore, this embodiment modulates the light modulator using a hologram selected after mask processing. As a result, when the light beam modulated by the optical modulator passes through the multimode fiber to image the sample to be measured, mask scanning imaging can be achieved. This means that the imaging speed is improved by reducing the number of sampling points and the light flux is reduced.

[0038] Specifically, this embodiment uses a DMD (Digital Micromirror Device) as an optical modulator. However, when modulating a DMD, it can only perform amplitude modulation and cannot meet the phase modulation requirements required for control. Therefore, a hologram generated using the Lie holographic principle is used to achieve phase modulation of the DMD. The Lie holographic principle encodes phase information into an amplitude distribution, generating a special hologram that is loaded onto the DMD to achieve equivalent phase modulation.

[0039] In this embodiment, however, the generated multiple holograms are not directly loaded onto the DMD, but are first masked, that is, the multiple holograms are masked and selected to obtain a selected hologram, and the selected hologram is loaded onto the DMD to achieve phase modulation of the DMD. Thus, after the modulated DMD is used to modulate the input light beam, scanning imaging of the sample to be tested in a corresponding masked manner can be achieved. It should be noted that the Lie holographic principle is used to generate multiple holograms corresponding to scanning different positions of the sample to be tested, that is, each hologram corresponds to the position of scanning a point of the sample to be tested, and the masking selection of the multiple holograms refers to removing the holograms corresponding to the positions that do not need to be scanned, that is, removing a portion of the holograms of the multiple holograms and only inputting the remaining holograms into the DMD.

[0040] In addition, since the image is obtained through mask scanning, the acquired image is incomplete. Therefore, this embodiment pre-trains a deep learning network model that can restore the acquired image to obtain a complete image. Subsequent processing such as analysis of the sample to be tested can be performed based on this complete image.

[0041] In this embodiment, a multimode optical fiber imaging method based on deep learning is provided, and the process includes the following steps:

[0042] Step S201: inputting a Hadamard matrix diagram into the optical modulator.

[0043] Step S202 : obtaining the intensity of the light field after the interference between the signal light modulated by the optical modulator and the reference light signal, and determining the light field phase based on the four-step phase shift principle.

[0044] Step S203: Calculate a transmission matrix based on the Hadamard matrix diagram and the light field phase.

[0045] Step S204: Based on the transmission matrix and using the Lie holographic principle, a hologram for realizing phase modulation of the optical modulator is determined. The hologram includes a plurality of different holograms. Different holograms are loaded into the optical modulator so that the light beam output by the multimode optical fiber is focused at different positions.

[0046] Specifically, in order to focus the light beam at any position of the sample after the DMD modulates the light beam, the corresponding hologram needs to be determined first. Based on this, the present embodiment first uses the above steps to determine the transmission matrix of the light modulator.

[0047] First, the optical modulator inputs a Hadamard matrix diagram. Specifically, 64×64 groups of four Hadamard matrices, each with a phase interval of pi / 2, can be input. A Hadamard matrix is ​​a special orthogonal matrix with only +1 and -1 elements. Different Hadamard matrices can modulate the light field in different ways. The four matrices, each with a phase interval of π / 2, are used to introduce different phase information into the light field. After inputting a set of Hadamard matrices, the light field intensity after the interference of the modulated beam and the reference signal is collected to obtain the light intensity information corresponding to the four different phases (i.e., the phases corresponding to the four Hadamard matrices with a phase interval of pi / 2). The light field phase information is then calculated based on the four-step phase shift principle. The specific calculation process can be implemented by referring to related technologies.

[0048] Once the light field phases corresponding to different Hadamard matrix diagrams are determined, the transmission matrix can be solved. The transmission matrix describes the system's transformation relationship between the light field amplitude and phase during the process of light field input to output. By determining the transmission matrix, the changes in the light field characteristics at each position after the light field passes through the DMD and multimode fiber can be accurately known, thereby achieving control of the focusing at different positions of the multimode fiber output end. Specifically, after the input Hadamard matrix diagram (representing the modulation pattern of the input light field) and the output light field (containing phase and amplitude information) measured by methods such as four-step phase shift are known, the transmission matrix can be calculated using algorithms related to linear algebra and optical propagation theory. Finally, after the transmission matrix is ​​determined, the corresponding hologram can be determined by combining the Lie holographic principle.

[0049] Step S205, loading the selected hologram after mask processing into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging, and the hologram is a hologram used to realize phase modulation of the optical modulator; for details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0050] Step S206: Acquire the image obtained by mask scanning imaging, and restore the image based on the pre-trained deep learning network model. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0051] In this embodiment, a multimode optical fiber imaging method based on deep learning is provided, which includes the following steps:

[0052] Step S301 : Using a window mask, the image in the training data is divided into mask blocks of a preset size and then masked to obtain a masked image.

[0053] Step S302: extract features of the masked image using an encoder.

[0054] Step S303: Based on the features of the image, a decoder is used to predict the obscured portion of the image.

[0055] Step S304: Update the parameters of the encoder and decoder according to the difference between the prediction result and the original image, and repeat the above process until the preset conditions are met to obtain the trained deep learning network model.

[0056] Specifically, the deep learning network model used in this embodiment can be a Swin MAE (Swin Masked Autoencoders) model or a GAN (Generative Adversarial Networks) model. The Swin MAE model uses the Swin Transformer as its backbone network. The Swin Transformer employs a layered structure and sliding window mechanism, effectively processing images of varying scales and capturing both local and global information. Using the Swin MAE model for image restoration reduces the dataset size and training requirements.

[0057] Specifically, the Swin MAE model is trained as follows: the original image in the training dataset is divided into n×n masked blocks through a window masking process, with 75% of the image masked. This is similar to sampling 25% of the image using a specific mask in real-world acquisition systems. The image is then fed into the model's encoder, converted into the required sequence, and features are extracted using a windowed format. The decoder then predicts the masked portion, ultimately restoring the original image sequence as output. This process is then repeated, iteratively updating the encoder and decoder parameters. When the relevant conditions are met, the iteration process is terminated, resulting in the trained model.

[0058] In step S305, the hologram selected after mask processing is loaded into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging. The hologram is a hologram used to realize phase modulation of the optical modulator. Specifically, during the training process of the Swin MAE model, when using the window mask for masking, a set of masks is first generated, and then the original image is masked. Therefore, a specific set of mask outputs during the model training process can be used to perform mask processing on multiple holograms, wherein no sampling occurs when the mask is 0, and sampling occurs when the mask is 1. The mask represents whether focus scanning is performed at different positions within the field of view. In the Lie holographic method based on the transmission matrix, the focus point at each position corresponds to a pre-calculated phase map, so the phase map corresponding to the position where the focus sampling is required, that is, the hologram selected after mask processing, is directly input into the DMD to realize mask scanning imaging.

[0059] In an optional embodiment, when the sample to be tested is a sparse sample, the effective information area of ​​the sparse sample is obtained; based on the effective information area, the hologram selected after mask processing is loaded into the light modulator. Specifically, in endoscopic imaging, the sample as a whole contains a large amount of data, but the information valuable for research or diagnosis is not evenly distributed. For example, in biomedical samples, only specific cells, molecular structures and other areas may contain information that can reflect physiological and pathological conditions. These areas are effective information areas, while most other areas may be just background materials, etc., with low information content or no value. Therefore, only the effective information area of ​​the sample to be tested can be sampled.

[0060] Specifically, before sampling, the effective information area of ​​the sparse sample can be determined. Then, the corresponding holograms for sampling the effective information area are determined (this number of holograms is smaller than the number of holograms when sampling the entire sample). Masking is then performed on these multiple holograms, retaining 25% of the holograms. For example, if the holograms consist of 100, only 25 holograms are retained. Thus, in the case of a sparse sample, a mask is set that randomly masks only 25% of the effective information portion, further increasing the speed by a factor of four.

[0061] Step S306: Acquire the image obtained by mask scanning imaging, and restore the image based on the pre-trained deep learning network model. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0062] In this embodiment, a multimode fiber imaging system based on deep learning is also provided, which includes: a control module, which is used to load a hologram selected after mask processing into an optical modulator, and the hologram is a hologram used to realize phase modulation of the optical modulator; a laser optical path module, which includes a light source and an optical modulator, and the optical modulator is used to modulate the light beam output by the light source; a multimode optical fiber, which is used to transmit the modulated light beam to illuminate the sample to be tested; a fluorescence detection module, which is used to receive the fluorescence signal reflected by the sample to be tested; and a control module, which is also used to receive the fluorescence signal to generate a fluorescence imaging image, and restore the fluorescence imaging image using a pre-trained deep learning network model.

[0063] In an optional embodiment, the control module is also used to input a Hadamard code matrix diagram into the optical modulator; the interference module is used to receive an interference signal output after internal interference in the multimode optical fiber, or to interfere the signal output by the multimode optical fiber with a reference light to generate an interference signal, and convert the interference signal into the light field intensity of the light beam; the control module is also used to obtain a transmission matrix based on the four-step phase shift principle and the phase calculation corresponding to the Hadamard code matrix diagram and the light field intensity; based on the transmission matrix, the Lie holographic principle is used to determine a hologram for realizing phase modulation of the optical modulator, and the hologram includes multiple different holograms, and different holograms are loaded into the optical modulator so that the light beam output by the multimode optical fiber can be focused at different positions.

[0064] Specifically, in this imaging system, the optical path used for imaging is as follows: Figure 2As shown: a continuous laser 1 with a wavelength of 532nm emits laser light, which is expanded by the first 4f system 2 (including the third lens 0201, the pinhole 0202 and the fourth lens 0203). The pinhole 0202 filters the light beam to eliminate stray waves. The half-wave plate 3 and the polarization beam splitter prism 4 are used to adjust the light beam power. At the same time, a reference light beam is separated and passes through the seventh reflector 24, the eighth reflector 25, the ninth reflector 26 and the tenth reflector 27 to enter the beam splitter 17 for interference with the light beam after passing through the multimode optical fiber. The other light beam passes through the polarization beam splitter prism 4 and is further expanded by the second 4f system 5 (including the fifth lens 0501 and the sixth lens 0502). It then passes through the first reflector 6 and the second reflector 7 to irradiate the DMD 8.

[0065] The light beam modulated by the DMD 8 enters the third 4f system 9, the seventh lens 0901 realizes the Fourier transform of the light beam, the aperture 0902 realizes the selection of the diffraction order reflected light of the target light field on the Fourier plane, and the eighth lens 0903 realizes the inverse Fourier transform of the light beam. The light beam emitted from the third 4f system 9 passes through the third reflector 10 and the fourth reflector 11 and enters the dichroic plate 12. After passing through the dichroic plate 12, the light beam is reflected because it is less than the cutoff wavelength. Then the light beam passes through the fifth reflector 13 and the first objective lens 14, and is coupled into the multimode fiber part 15 (including the first fiber collimator 1501, the multimode fiber 1502 and the sample stage 1503). The light beam passing through the multimode fiber 1502 enters the second objective lens 16 for collimation. The collimated light beam passes through the beam splitter 17 to interfere with the reference light. The interfered light beam is focused by the lens 18 and enters the camera 19, and the computer terminal 20 performs light field calculation and modulation.

[0066] When imaging a sample, multimode fiber 1502 is inserted into the sample placed on sample stage 1503. The light beam output by the multimode fiber strikes the sample and excites fluorescence. This fluorescence is then reflected back through multimode fiber 1502 and transmitted along its original path to dichroic filter 12. Because the wavelength of the reflected fluorescence exceeds the cutoff wavelength of dichroic filter 12, the fluorescence passes through dichroic filter 12, passes through sixth reflector 21, enters second lens 22, and is focused into PMT (photomultiplier tube) 23. PMT 23 converts the received fluorescence signal into an electrical signal, which is then input into computer terminal 20 and converted into fluorescence intensity information at different locations, thereby achieving imaging.

[0067] Before acquiring the information captured by camera 19, the DMD inputs 64×64 groups of four Hadamard matrices, each with a phase interval of pi / 2. Based on this matrix, the DMD modulates the received light beam, which is then transmitted to camera 19 in the aforementioned manner. Camera 19 then collects the light field intensity signal of the received light beam and transmits it to computer terminal 20. This light field intensity signal is then calculated using the four-step phase shift principle to determine the phase of the corresponding light field. Given the input Hadamard matrix and the output light field, the system's transmission matrix is ​​calculated. Since the DMD can only perform amplitude modulation, the Lie holographic principle is used to enable the DMD to perform phase modulation. Specifically, based on the transmission matrix, the Lie holographic principle is used to determine a hologram that achieves phase modulation. The computer terminal then loads this hologram onto the DMD. The DMD-modulated light beam then passes through the aforementioned optical path and is then irradiated onto the sample to be tested, enabling mask scanning imaging. This means that the fluorescence intensity information acquired by computer terminal 20 only contains partial information about the sample to be tested, requiring further reconstruction of the image using a pre-trained deep learning network model.

[0068] It should be noted that the optical path used for the above-mentioned imaging is a method of interfering with the external reference light. Furthermore, the seventh to tenth reflectors in the optical path can be removed, and the reference light and the signal light can be transmitted using the same optical path. That is, the reference light is also transmitted through the above-mentioned optical path to the multimode optical fiber, and interferes with the internal reference light in the multimode optical fiber.

[0069] In an optional embodiment, the control module is further configured to control the optical modulator and the fluorescence detection module to operate synchronously based on a synchronization pulse. Specifically, a signal synchronization module may be further provided in the system, wherein the control module transmits a synchronization pulse to the signal synchronization module, and the signal synchronization module transmits a synchronization signal to the DMD and PMT based on the synchronization pulse, thereby achieving synchronized operation of the DMD and PMT.

[0070] As a specific application example of the embodiment of the present invention, Figure 3 As shown, the multimode fiber imaging system based on deep learning specifically includes:

[0071] Excitation optical module, consisting of Figure 2 The continuous laser 1 with a wavelength of 532nm emits laser light, which is expanded by the first 4f system 2. The small hole 0202 therein filters the light beam to eliminate stray waves. The half-wave plate 3 and polarization beam splitter prism 4 are used to adjust the beam power. At the same time, a reference light is separated to interfere with the light beam after passing through the multimode fiber. The other light beam passes through the polarization beam splitter prism 4 and is further expanded by the second 4f system 5 before irradiating the DMD.

[0072] DMD random scanning module, DMD random scanning module uses DMD to modulate the speckle of the output multimode optical fiber, so that it can be focused and perform random point scanning. The main principle is that DMD inputs 64×64 groups of 4 Hadamard matrix diagrams with a phase interval of pi / 2, and uses the four-step phase shift principle to calculate the light field intensity obtained by the camera after interference with the internal reference light, thereby obtaining the phase of the corresponding light field. When the input Hadamard matrix and the output light field are known, the transmission matrix of the system is calculated, thereby achieving focusing at different positions of the multimode optical fiber output end. Among them, since DMD can only perform amplitude modulation, it is necessary to use the Lie holographic principle to make DMD perform phase modulation. Figure 2 The computer terminal 20 controls the DMD8 to calculate the corresponding hologram input, the seventh lens 0901 realizes the Fourier transform of the light beam, the aperture 0902 realizes the selection of the diffraction order reflected light of the target light field on the Fourier plane, and the eighth lens 0903 realizes the inverse Fourier transform of the light beam. After passing through the dichroic plate 12, the light beam is reflected because it is less than the cut-off wavelength. Then the light beam is coupled into the multimode optical fiber through the first objective lens 14. The light beam passing through the multimode optical fiber enters the second objective lens 16 for collimation. The collimated light beam passes through the beam splitter 17 to interfere with the reference light. The interfered light beam is focused by the lens 18 and enters the camera 19 and is used for light field calculation and modulation by the computer terminal 20.

[0073] Sample modules such as Figure 2 As shown, the first fiber collimator 1501 in the sample module is used to couple the collimated light beam into the multimode optical fiber, insert the multimode optical fiber 1502 into the sample placed on the sample stage 1503, and use the light beam of the multimode optical fiber to scan the field of view points in the sample for imaging.

[0074] PMT detection module, such as Figure 2 As shown, the focused light beam hits the sample and excites fluorescence, which is then reflected back to the multimode optical fiber 1502 and passes through the multimode optical fiber. Since the wavelength of the reflected fluorescence is greater than the cutoff wavelength of the dichroic plate 12, the fluorescence passes through the dichroic plate 12 and enters the second lens 22. The light beam is focused into the PMT 23, and the fluorescence signal is converted into an electrical signal in the PMT 23 and transmitted to the signal processing module. The signal processing module calculates the fluorescence signal obtained by the PMT.

[0075] Signal processing module, the signal processing module mainly includes light field intensity signal, fluorescence intensity signal and synchronization signal. Figure 2As shown, after receiving the light field intensity signal from camera 19, the signal processing unit uses a four-step phase shift method to calculate the phase of the corresponding light field, the system's transmission matrix, and further calculates the hologram required for point focus at the corresponding position. Fluorescence enters PMT 23 and is converted into an electrical signal. This signal is then fed into the signal processing module, which converts the electrical signal into fluorescence intensity information at different locations, thereby achieving imaging. The signal processing module also generates synchronization pulses to synchronize DMD modulation scanning with PMT signal acquisition.

[0076] Model processing module, the model processing module includes the model training process and the model inference process, such as Figure 4 As shown in the figure, during training, the original image in the training dataset is divided into n×n masked patches through a window masking process, with 75% of the image masked. This is similar to sampling 25% of the image using a specific mask in real-world system acquisition. The image is then fed into the model's encoder as input, converted into the required sequence, and feature extraction is performed in a windowed manner. The decoder then predicts the masked portion, ultimately restoring the original image sequence as output. During inference, the model receives a fluorescent image acquired by the system, scanned using a specific masked patch. The fluorescent image is fed into the model as input, and the trained model performs image restoration to obtain the complete original image.

[0077] Mask control module, such as Figure 5 As shown, the mask control module needs to provide the DMD with a phase diagram corresponding to the required mask. Figure 5 The random scanning method is divided into the conventional sequential scanning method and the random scanning method of the present invention. In the random scanning method, n×n pixels (in this embodiment, n=2) are generally used as a mask block (pixel block), and only 25% of the field of view is randomly masked to generate the corresponding mask map ( Figure 5 For sparse samples, a random mask scan pattern is set to scan only 25% of the effective information, further increasing the rate by a factor of four. The mask control module converts the random mask image into the phase map required by the DMD and transmits it to the DMD for scanning and imaging.

[0078] The signal synchronization module receives the synchronization pulse from the signal processing module, and then gives synchronization signals to the DMD scanning module and the PMT signal acquisition module, so that the system can realize the acquisition of fluorescence intensity signals at random positions.

[0079] like Figure 6 As shown in the figure, it is the point focusing result diagram of the above imaging system. Figure 6 In the figure, the three pictures in the upper part represent three holograms scanned at different positions of the input DMD, and the three pictures in the lower part represent pictures of points at different positions after the input hologram is focused by the multimode optical fiber and received by the camera 19. Figure 7 The following figure shows the restored image of the application instance using a deep learning network model. Here, "INPUT" represents the input model image after masking 75% of the image, "OUTPUT" represents the predicted image after the output model is applied, and "GT" represents the original image, which serves as the evaluation criteria for the predicted image.

[0080] In the present invention, since a mask of n×n pixels is used for scanning imaging, and only 25% of the pixels in the entire field of view are scanned, the imaging speed of the multimode fiber imaging system is quadrupled by using a sparse sampling method. At the same time, if a mask sampling method that only contains 25% of the effective information in the sparse sample is used, the speed can be further increased exponentially on the basis of the quadruple. In addition, due to the reduction in the number of scanning points, the light flux is greatly reduced, thereby reducing photobleaching and protecting the activity and biological function of the sample. At the same time, the Swin MAE model is used for image restoration, which reduces the data volume of the data set and the requirements for training conditions, making it applicable to biomedical imaging systems. At the same time, the image restoration effect achieved so far is already relatively good.

[0081] This embodiment also provides a multimode fiber imaging device based on deep learning, which is used to implement the above-mentioned embodiments and preferred embodiments. Details that have already been described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0082] This embodiment provides a multimode fiber imaging device based on deep learning, which is applied to a multimode fiber imaging system. The system uses an optical modulator to modulate the light beam and then input it into the multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging on the sample to be tested. Figure 8 As shown, the device includes:

[0083] The mask modulation module 81 is used to load the hologram selected after the mask processing into the optical modulator so that the light beam output by the multimode optical fiber realizes mask scanning imaging. The hologram is a hologram used to realize phase modulation of the optical modulator;

[0084] The restoration module 82 is used to collect images obtained by mask scanning imaging and restore the images based on a pre-trained deep learning network model.

[0085] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0086] The embodiment of the present invention also provides a computer device having the above Figure 8 The deep learning-based multimode fiber imaging device shown.

[0087] See also Figure 9 , Figure 9 is a structural diagram of a computer device provided by an optional embodiment of the present invention, such as Figure 9 As shown, the computer device includes: one or more processors 100, memory 200, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components utilize different buses to communicate with each other and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of a GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Equally, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 9 A processor 100 is taken as an example.

[0088] The processor 100 may be a central processing unit (CPU), a network processor (NPU), or a combination thereof. The processor 100 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device (PLD) may be a complex programmable logic device (CPLD), a field programmable gate array (FPGA), a general purpose array logic (GAL), or any combination thereof.

[0089] The memory 200 stores instructions that can be executed by at least one processor 100, so as to enable at least one processor 100 to execute the method shown in the above embodiment.

[0090] The memory 200 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of a computer device based on the presentation of a small program landing page, etc. In addition, the memory 200 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 200 may optionally include a memory remotely located relative to the processor 100, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0091] The memory 200 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 200 may also include a combination of the above types of memory.

[0092] The computer device further includes a communication interface 300 for the computer device to communicate with other devices or a communication network.

[0093] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0094] A portion of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the form in which the computer program instruction exists in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc. Accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium that can be accessed by the computer.

[0095] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multimode optical fiber imaging method based on deep learning, characterized in that: Applied to a multimode fiber imaging system, the system uses an optical modulator to modulate a light beam and then inputs it into a multimode fiber. The light beam output by the multimode fiber is used to perform focused scanning imaging on a sample to be measured. The method includes: The hologram selected after the mask processing is loaded into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging. The hologram is a hologram used to realize phase modulation of the optical modulator. The hologram includes multiple holograms corresponding to different scanning positions of the sample to be tested, that is, each hologram corresponds to the position of scanning a point of the sample to be tested. The mask processing selection of the multiple holograms refers to removing the holograms corresponding to the positions that do not need to be scanned, that is, removing a part of the holograms of the multiple holograms and inputting only the remaining holograms into the optical modulator; Acquire an image obtained by mask scanning imaging, and restore the image based on a pre-trained deep learning network model; The method of loading the selected hologram after mask processing into the light modulator comprises: The pre-trained deep learning network model is used to load the masked holograms selected into the light modulator. The masked holograms selected include 25% of the total number of holograms. Loading the masked hologram into the light modulator also includes: When the sample to be tested is a sparse sample, obtaining the effective information area of ​​the sparse sample; loading the hologram selected after masking based on the effective information area into the light modulator; When the deep learning network model is a Swin MAE model, the deep learning network model is pre-trained in the following way: Use window masking to divide the image in the training data into small mask blocks of preset size and then perform masking to obtain the masked image; An encoder is used to extract features of the masked image; Based on the features of the image, a decoder is used to predict the obscured portion of the image; The parameters of the encoder and decoder are updated according to the difference between the predicted results and the original image, and the above process is repeated until the preset conditions are met to obtain the trained deep learning network model; The Swin MAE model uses the Swin Transformer as the backbone network. The Swin Transformer adopts a hierarchical structure and sliding window mechanism to process images of different scales and capture local and global information.

2. The method according to claim 1, characterized in that Before loading the hologram selected after mask processing into the light modulator, the method further includes: inputting a Hadamard matrix diagram into the optical modulator; Obtain the intensity of the light field after the interference of the signal light modulated by the optical modulator and the reference light signal, and determine the phase of the light field based on the four-step phase shift principle; A transmission matrix is ​​obtained by calculating the Hadamard matrix diagram and the light field phase; Based on the transmission matrix and using the Lie holographic principle, a hologram for realizing phase modulation of the optical modulator is determined.

3. A multimode fiber imaging system based on deep learning, characterized in that: The system comprises: a control module, configured to load the hologram selected after masking into the optical modulator, wherein the hologram is a hologram used to implement phase modulation of the optical modulator, and the hologram includes multiple holograms corresponding to different scanning positions of the sample to be tested, that is, each hologram corresponds to the position of scanning a point of the sample to be tested, and the masking selection of the multiple holograms refers to removing the holograms corresponding to the positions that do not need to be scanned, that is, removing a portion of the multiple holograms and inputting only the remaining holograms into the optical modulator; A laser optical path module, comprising a light source and a light modulator, wherein the light modulator is used to modulate the light beam output by the light source; Multimode optical fiber, used to transmit the modulated light beam to illuminate the sample to be tested; A fluorescence detection module, used to receive the fluorescence signal reflected by the sample to be tested; The control module is further configured to receive the fluorescence signal to generate a fluorescence imaging image, and restore the fluorescence imaging image using a pre-trained deep learning network model; The method of loading the selected hologram after mask processing into the light modulator comprises: The pre-trained deep learning network model is used to load the masked holograms selected into the light modulator. The masked holograms selected include 25% of the total number of holograms. Loading the masked hologram into the light modulator also includes: When the sample to be tested is a sparse sample, obtaining the effective information area of ​​the sparse sample; loading the hologram selected after masking based on the effective information area into the light modulator; When the deep learning network model is a Swin MAE model, the deep learning network model is pre-trained in the following way: Use window masking to divide the image in the training data into small mask blocks of preset size and then perform masking to obtain the masked image; An encoder is used to extract features of the masked image; Based on the features of the image, a decoder is used to predict the obscured portion of the image; The parameters of the encoder and decoder are updated according to the difference between the predicted results and the original image, and the above process is repeated until the preset conditions are met to obtain the trained deep learning network model; The Swin MAE model uses the Swin Transformer as the backbone network. The Swin Transformer adopts a hierarchical structure and sliding window mechanism to process images of different scales and capture local and global information.

4. The system according to claim 3, characterized in that The system further comprises: an interference module; The control module is further configured to input a Hadamard matrix diagram into the optical modulator; The interference module is used to receive the interference signal output after the internal interference of the multimode optical fiber, or to interfere the signal output by the multimode optical fiber with the reference light to generate an interference signal, and convert the interference signal into the light field intensity of the light beam; The control module is also used to calculate a transmission matrix based on the four-step phase shift principle and the phase corresponding to the Hadamard code matrix diagram and the light field intensity; based on the transmission matrix, the Lie holographic principle is used to determine the hologram used to achieve phase modulation of the optical modulator, and the hologram includes multiple different holograms. Different holograms are loaded into the optical modulator so that the light beam output by the multimode optical fiber is focused at different positions.

5. The system according to claim 3, wherein: The control module is further configured to control the light modulator and the fluorescence detection module to operate synchronously according to the synchronization pulse.

6. A multimode optical fiber imaging device based on deep learning, characterized in that: Applied to a multimode fiber imaging system, the system uses an optical modulator to modulate the light beam and then input it into the multimode fiber. The light beam output by the multimode fiber is used to focus, scan and image the sample to be measured. The device includes: A mask modulation module is used to load the hologram selected after mask processing into the optical modulator, so that the light beam output by the multimode optical fiber realizes mask scanning imaging. The hologram is a hologram used to realize phase modulation of the optical modulator. The hologram includes multiple holograms corresponding to different positions of the sample to be tested, that is, each hologram corresponds to the position of scanning a point of the sample to be tested. The mask processing selection of the multiple holograms refers to removing the holograms corresponding to the positions that do not need to be scanned, that is, removing a portion of the holograms of the multiple holograms and inputting only the remaining holograms into the optical modulator; A restoration module, configured to collect images obtained by mask scanning imaging and restore the images based on a pre-trained deep learning network model; The method of loading the selected hologram after mask processing into the light modulator comprises: The pre-trained deep learning network model is used to load the masked holograms selected into the light modulator. The masked holograms selected include 25% of the total number of holograms. Loading the masked hologram into the light modulator also includes: When the sample to be tested is a sparse sample, obtaining the effective information area of ​​the sparse sample; loading the hologram selected after masking based on the effective information area into the light modulator; When the deep learning network model is a Swin MAE model, the deep learning network model is pre-trained in the following way: Use window masking to divide the image in the training data into small mask blocks of preset size and then perform masking to obtain the masked image; An encoder is used to extract features of the masked image; Based on the features of the image, a decoder is used to predict the obscured portion of the image; The parameters of the encoder and decoder are updated according to the difference between the predicted results and the original image, and the above process is repeated until the preset conditions are met to obtain the trained deep learning network model; The Swin MAE model uses the Swin Transformer as the backbone network. The Swin Transformer adopts a hierarchical structure and sliding window mechanism to process images of different scales and capture local and global information.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a computer to execute the multimode optical fiber imaging method based on deep learning according to claim 1 or 2.

Images