A microscope imaging device

Abstract

The application discloses a kind of microscopic imaging devices, device is applied to the three-dimensional microscopic imaging of living cell, device includes structured light generating unit, sample stage, control unit, high-speed image compression acquisition unit and processing unit.Structured light generating unit generates structured light, to excite the fluorescence carrying sample information;Sample stage is provided with piezoelectric ceramic, to control structured light periodic focusing in different longitudinal depth of the sample to be measured;Control unit controls piezoelectric ceramic to complete n times longitudinal depth change of the sample to be measured in single exposure period;High-speed image compression acquisition unit samples the fluorescence pattern of the sample to be measured, and the fluorescence pattern contains n longitudinal depth position fluorescent information, and any longitudinal depth position contains at least three kinds of fluorescent information;Processing unit is reconstructed based on single frame fluorescence pattern obtained in any exposure period, obtains the three-dimensional image of the sample to be measured.The application can obtain the three-dimensional image of clear display living cell sample structure distribution and continuous dynamic behavior.

Description

Technical Field

[0001] The present application relates to the technical field of optical imaging equipment, and in particular to a microscopic imaging device. Background Art

[0002] In modern biomedicine, research on nanoscale biological samples, such as animal and plant organelles, neurons, and bacteria, is steadily advancing. Optical microscopes with super-resolution capabilities, as a crucial observation tool, play a decisive role in related research. Common super-resolution microscopy techniques include structured illumination microscopy, stimulated emission depletion microscopy, stochastic optical reconstruction microscopy, and single-molecule localization microscopy.

[0003] These imaging techniques can cause significant photobleaching and phototoxicity to living cells, rapidly killing them during observation of active biological cells and preventing long-term observation of their vital processes. Furthermore, due to the hardware limitations of CCD / CMOS sensors, exposure times are long, resulting in blurred dynamic microscopic images and making it impossible to reconstruct continuous images of biological activity. Summary of the Invention

[0004] In order to solve the deficiencies of the prior art, this application adopts the following technical solutions:

[0005] The present application provides a microscopic imaging device, which is used for three-dimensional microscopic imaging of living cells. The microscopic imaging device includes:

[0006] a structured light generating unit, wherein the structured light generating unit is configured to periodically generate at least three types of structured light, wherein the structured light converges on the sample to be tested to excite fluorescence carrying sample information;

[0007] A sample stage for holding the sample to be tested, wherein the sample stage is provided with a piezoelectric ceramic, and the piezoelectric ceramic is configured to adjust the sample to be tested so as to control the periodic focusing of the structured light at different longitudinal depths of the sample to be tested;

[0008] A control unit configured to set the number of sampling depths of the microscopic imaging device and control the piezoelectric ceramic to complete n times of longitudinal depth change of the sample to be measured within a single exposure cycle of the high-speed image compression acquisition unit, where n is the set number of sampling depths;

[0009] a high-speed image compression acquisition unit configured to sample fluorescence at different longitudinal depths of the sample to be tested to obtain a fluorescence pattern, and transmit the fluorescence pattern to a processing unit, wherein the fluorescence pattern includes fluorescence information at n longitudinal depths, and any longitudinal depth includes fluorescence information excited by at least three types of the structured light;

[0010] A processing unit is configured to reconstruct a single-frame fluorescence pattern obtained in any exposure cycle of the high-speed image compression acquisition unit to obtain a three-dimensional image of the sample to be tested.

[0011] In summary, the present application provides a microscopic imaging device that utilizes structured light of different directions and phases to excite fluorescence carrying sample information, thereby avoiding photobleaching and phototoxicity to active biological samples, ensuring that the samples are not harmed during observation, and avoiding affecting the life activities of living samples; within a single exposure cycle, fluorescence images of the sample at multiple longitudinal depth positions under different structured light irradiation are collected, and a single-frame super-resolution image within a single exposure cycle is reconstructed through a reconstruction algorithm, and three-dimensional reconstruction is performed based on the single-frame super-resolution image to obtain a three-dimensional image of the measured sample with a resolution of hundreds of nanometers, clearly showing the cell structure distribution and continuous dynamic behavior of the cells of the living sample, and realizing the instantaneous capture of the dynamic life process of the living cell sample within a single exposure cycle.

[0012] Furthermore, the structured light generating unit includes: a laser for emitting laser light;

[0013] a first digital micromirror configured to receive the laser light and form a grating image to modulate the laser light to generate diffracted light of multiple diffraction orders;

[0014] a light focusing unit configured to converge the plurality of diffracted lights generated by the first digital micromirror, causing the diffracted lights to interfere with each other to form structured light, and to focus the structured light on the sample to be measured to excite fluorescence carrying sample information;

[0015] The control unit is further configured to control the grating image conversion frequency of the first digital micromirror, the displacement frequency of the piezoelectric ceramic, and the sampling frequency of the high-speed image compression acquisition unit. The frequencies of the various components controlled by the control unit satisfy: the grating image conversion frequency is 3M times the displacement frequency of the piezoelectric ceramic, and the displacement frequency of the piezoelectric ceramic is n times the sampling frequency of the high-speed image compression acquisition unit, where M is equal to 1, 2, or 3, and n is equal to the number of sampling depths.

[0016] Furthermore, the processing unit reconstructs the single-frame fluorescence pattern obtained within one exposure cycle of the high-speed image compression acquisition unit to obtain a three-dimensional image of the sample to be tested, including:

[0017] Step S11, using a video reconstruction algorithm to reconstruct the fluorescence pattern into 3M*n frames of restored images each carrying fluorescence information at different longitudinal depths;

[0018] Step S12, reconstructing the 3M*n restored images using a deep learning reconstruction algorithm to reconstruct 3M*n optical slice images corresponding to n different longitudinal depths;

[0019] Step S13: reconstruct the 3M*n light slice images using a light slice image reconstruction algorithm to obtain a three-dimensional image of the sample to be tested.

[0020] Furthermore, the processing unit reconstructs the restored image using a deep learning reconstruction algorithm, including:

[0021] Step S121, encoding the 3M*n frames of restored images respectively to perform two-dimensional compression observation to obtain observation values;

[0022] Step S122, calculating, based on the observation values, a common information estimation observation value and a unique information estimation observation value of each frame of the restored image corresponding to the first longitudinal depth;

[0023] Step S123: inputting the common information estimated observation value corresponding to the first longitudinal depth into a common information reconstruction module, and the common information reconstruction module reconstructing the common information estimated observation value corresponding to the first longitudinal depth to obtain 3M frames of reconstructed common information corresponding to the first longitudinal depth;

[0024] Step S124 , performing channel dimension concatenation on the unique information estimation observation value of the first longitudinal depth of the 3M frames and the reconstructed common information of the first longitudinal depth to obtain the optical slice image of the 3M frames of the first longitudinal depth.

[0025] Furthermore, the observed value is expressed by the following formula:

[0026] ;

[0027] Where, represents the observed value, is the kth random mask, is the kth image frame, w and h are the height and width of the image frame, Represents the Hadamard product operation, B=3M*n.

[0028] Furthermore, the shared information estimated observation value is expressed by the following formula:

[0029] ;

[0030] Where, represents the observed value, is the kth random mask, Represents the common information estimated observation value at the nth vertical depth.

[0031] Furthermore, the unique information estimated observation value is expressed by the following formula:

[0032] ;

[0033] Where, represents the common information estimated observation value at the nth vertical depth, represents the k-th unique information estimated observation.

[0034] Furthermore, the processing unit reconstructs 3M*n light slice images using a light slice image reconstruction algorithm, including:

[0035] The light slice images corresponding to the same longitudinal depth in the 3M*n light slice images are reconstructed to obtain super-resolution images of n longitudinal depths, and the super-resolution images of n longitudinal depths are reconstructed into a three-dimensional image of the sample to be tested using three-dimensional image processing technology.

[0036] Furthermore, the first digital micromirror is configured to modulate the laser to generate three diffraction orders of 0, +1, and -1;

[0037] The microscopic imaging device further includes a filter configured to filter out the 0th order diffraction light and allow the +1st order diffraction light and the -1st order diffraction light to pass.

[0038] Furthermore, the microscopic imaging device further includes a dichroic mirror, wherein the dichroic mirror is configured to reflect the +1-order diffraction light and the -1-order diffraction light to the focusing unit;

[0039] The focusing unit converges the +1st order diffraction light and the -1st order diffraction light to generate interference, thereby forming the structured light. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the structural connection of a microscopic imaging device provided in one embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structural connection of a structured light generating unit in a microscopic imaging device provided in one embodiment of the present application;

[0042] Figure 3 A flowchart of the steps of image reconstruction performed by a processing unit in a microscopic imaging device provided in one embodiment of the present application;

[0043] Figure 4 A flowchart of the steps for a processing unit in a microscopic imaging device provided in one embodiment of the present application to reconstruct a restored image using a deep learning reconstruction algorithm. DETAILED DESCRIPTION

[0044] The present application will be described in detail below in conjunction with the specific embodiments shown in the accompanying drawings, but these embodiments do not limit the present application. Structural, methodological, or functional changes made by ordinary technicians in this field based on these embodiments are included in the scope of protection of the present application.

[0045] In order to solve the deficiencies of the prior art, the present application provides a microscopic imaging device 100, which is applied to three-dimensional microscopic imaging of living cells, such as Figure 1 As shown, the microscopic imaging device 100 includes: a structured light generating unit 11 , a sample stage 12 , a control unit 13 , a high-speed image compression acquisition unit 14 and a processing unit 15 .

[0046] The structured light generating unit 11 is configured to periodically generate at least three types of structured light, which converge on the sample to be tested to excite fluorescence carrying sample information. The sample stage 12 is used to hold the sample to be tested. The sample stage 12 is provided with piezoelectric ceramics, which are configured to adjust the sample to be tested to control the periodic focusing of the structured light at different longitudinal depths of the sample to be tested. The control unit 13 is configured to set the number of sampling depths of the microscopic imaging device 100 and control the piezoelectric ceramics to complete n changes in the longitudinal depth of the sample to be tested within a single exposure cycle of the high-speed image compression acquisition unit 14, where n is the set number of sampling depths.

[0047] High-speed image compression acquisition unit 14 is configured to sample fluorescence at different longitudinal depths of the sample to be tested to obtain a fluorescence pattern, and transmit the fluorescence pattern to processing unit 15. The fluorescence pattern includes fluorescence information at n longitudinal depths, with any longitudinal depth containing fluorescence information from at least three types of structured light excitation. Processing unit 15 reconstructs a three-dimensional image of the sample to be tested based on the single-frame fluorescence pattern obtained during any exposure cycle of high-speed image compression acquisition unit 14.

[0048] Specifically, the structured light generating unit 11 periodically generates at least three types of structured light, which are focused at a predetermined longitudinal depth within the sample being measured. Through structured light illumination at different directions or phases, fluorescence carrying sample information is stimulated. The sample's three-dimensional information is encoded into a two-dimensional fluorescence pattern, providing the data foundation for subsequent three-dimensional reconstruction. Structured light does not cause photobleaching or phototoxicity to living biological samples, ensuring that the sample is not harmed during observation and preventing any impact on the vital activities of the living sample.

[0049] The sample stage 12 is used to hold the biological sample and uses integrated piezoelectric ceramics to achieve high-precision longitudinal displacement control. The piezoelectric ceramics can quickly adjust the longitudinal position of the sample. The piezoelectric ceramics move the sample to be measured, periodically focusing the structured light at different longitudinal depths of the sample. This ensures that the complete structured light sequence is captured at each longitudinal depth of the sample. This enables dynamic sampling of multiple longitudinal depths of the sample within a single exposure cycle, avoiding the time delay and light damage associated with traditional layer-by-layer scanning.

[0050] The control unit 13 sets the number of sampling depths of the microscopic imaging device 100 to n. By presetting the value n, the control unit 13 controls the piezoelectric ceramic to complete n longitudinal depth changes within a single exposure cycle of the high-speed image compression acquisition unit 14. The structured light generation unit 11 generates different structured lights, and within a single exposure cycle of the high-speed image compression acquisition unit 14, the structured light is used to illuminate each longitudinal depth position of the sample, thereby stimulating fluorescence information at multiple longitudinal depths of the sample. For example, if the number of sampling depths n is four, the piezoelectric ceramic needs to be moved to four different longitudinal depth positions in a single exposure cycle. At each longitudinal depth position, it is illuminated by at least three types of structured light, thereby obtaining multiple fluorescence images carrying sample information.

[0051] The high-speed image compression acquisition unit 14 collects fluorescence information at different longitudinal depth positions of the sample to be tested, encodes the fluorescence information, integrates the fluorescence information collected within a single exposure cycle into a single frame image, and compresses it into a single frame image, so that the single frame image contains fluorescence information at n longitudinal depth positions, and any longitudinal depth position contains at least three types of fluorescence responses excited by structured light, thereby reducing the amount of data for microscopic imaging, facilitating image data storage, significantly reducing the data storage and data transmission pressure of the microscopic imaging device 100, and reducing the use cost of the microscopic imaging device 100.

[0052] Based on the single-frame fluorescence pattern acquired during any exposure cycle by the high-speed image compression acquisition unit 14, the processing unit 15 can employ a reconstruction algorithm to encode and reconstruct the compressed single-frame fluorescence pattern to obtain a super-resolution image. Based on the super-resolution image, the processing unit 15 can employ a 3D reconstruction algorithm to reconstruct and superimpose the super-resolution images at different longitudinal depths, thereby obtaining a super-resolution 3D image of the sample, clearly displaying the cellular structure distribution and dynamic behavior of the living sample.

[0053] According to the above description, the present application provides a microscopic imaging device 100, which avoids photobleaching and phototoxicity to active biological samples by utilizing structured light of different directions and phases to excite fluorescence carrying sample information, thereby ensuring that the sample is not harmed during observation and avoiding affecting the life activities of the living sample; collects fluorescence images of the sample at multiple longitudinal depth positions under different structured light irradiation within a single exposure cycle, reconstructs a single-frame super-resolution image within a single exposure cycle through a reconstruction algorithm, and performs three-dimensional reconstruction based on the single-frame super-resolution image to obtain a three-dimensional image of the measured sample with a resolution of hundreds of nanometers, clearly showing the cell structure distribution and continuous dynamic behavior of the cells of the living sample, and realizing instantaneous capture of the dynamic life process of the living cell sample within a single exposure cycle.

[0054] As a way to implement Figure 2 As shown, the structured light generating unit 11 includes a laser 111, a first digital micromirror 112, and a focusing unit 113. The laser 111 is used to emit laser light; the first digital micromirror 112 is configured to receive the laser light and form a grating image to modulate the laser light, generating diffracted light of multiple diffraction orders. The focusing unit 113 is configured to converge the multiple diffracted light beams generated by the first digital micromirror 112, causing the diffracted light beams to interfere with each other to form structured light. The structured light beams are then focused on the sample to be tested to excite fluorescence that carries sample information.

[0055] Laser 111 serves as a light source, emitting laser light of a specific wavelength. The laser light emitted by laser 111 is received by first digital micromirror 112 and used to optically excite fluorescently stained proteins within the biological sample, thereby stimulating a fluorescent signal. Optionally, a beam expander 116 and a first reflector 117a are positioned between laser 111 and first digital micromirror 112. After expansion and collimation by beam expander 116, the laser light emitted by laser 111 is reflected by first reflector 117a and reaches first digital micromirror 112.

[0056] After receiving the expanded and collimated laser light, the first digital micromirror 112 spatially modulates the laser light by dynamically applying a grating pattern. The first digital micromirror 112 decomposes the single laser beam into diffracted light of multiple diffraction orders. The diffracted light generated by the first digital micromirror 112 is then reflected by the second reflector 117b to the focusing unit 113. By providing the reflectors 117a and 117b, the optical path of the microscopic imaging device 100 is adjusted to optimize the structural layout of the microscopic imaging device 100 and increase its structural flexibility.

[0057] The condensing unit 113 receives the diffracted light generated by the first digital micromirror 112 through the refraction of the second mirror 117b, and the condensing unit 113 converges the plurality of diffracted light so that the plurality of diffracted light interferes with each other to form a structured light for photoexcitation of the sample. The first digital micromirror 112 generates a corresponding structured light for each grating pattern switched. In the embodiment of the present application, the first digital micromirror 112 includes nine grating patterns combined by three grating directions 0°, 60°, 120° and three phases 0, , The corresponding condensing unit 113 can generate nine structured lights with different directions and phases.

[0058] The condensing unit 113 is located above the sample stage 12, and the structured light converges on the biological sample of the sample stage 12, and the structured light photoexcites the fluorescently labeled protein in the biological sample to generate a fluorescent signal carrying information of the biological sample. Optionally, a lens 118 can be arranged between the second mirror 117b and the condensing unit 113, and the condensing unit 113 receives the diffracted light refracted by the second mirror 117b through the lens 118. The lens 118 can converge the divergent or parallel diffracted light beams to a specific focal position, avoid the dispersion of light energy, and improve the intensity and contrast of the subsequent interference structured light. In the embodiment of the present application, the condensing unit 113 is configured as a microscope objective.

[0059] As an implementation manner, the first digital micromirror 112 is configured to modulate the laser to generate 0, +1, and -1 three diffraction orders; the first digital micromirror 112 forms a dynamic grating pattern by rapid flipping. When the laser irradiates the surface of the first digital micromirror 112, the periodic structure of the grating pattern modulates the laser in space, and the periodic grating modulates the incident light wave in phase or amplitude to form 0, +1, and -1 three diffraction order beams. The 0 order diffracted light propagates in the same direction as the incident light, and the 0 order diffracted light does not carry the spatial frequency information of the grating modulation; the +1 order and -1 order diffracted light are generated by the periodic modulation of the grating, and are deflected to the two sides at symmetrical angles, respectively. The +1 order and -1 order diffracted light carry the spatial frequency information of the grating modulation.

[0060] The microscope imaging device 100 further includes a filter 114 configured to filter out the 0 order diffracted light and pass the +1 order diffracted light and the -1 order diffracted light. Since the 0 order diffracted light does not carry the spatial frequency information of the grating modulation, the illumination of the 0 order diffracted light on the sample surface will cover the interference effect of the structured light. The filter 114 filters out the 0 order diffracted light and passes the +1 order diffracted light and the -1 order diffracted light, so as to avoid the introduction of the 0 order diffracted light affecting the imaging contrast and causing the reconstruction of the super-resolution image to be unable to be realized.

[0061] As an implementation method, the microscopic imaging device 100 also includes a dichroic mirror 115, which is configured to reflect the +1-order diffraction light and the -1-order diffraction light to the focusing unit 113; the focusing unit 113 converges the +1-order diffraction light and the -1-order diffraction light to produce interference and form structured light.

[0062] Specifically, the diffracted light modulated by the first digital micromirror 112 is refracted by the second reflector 117b and then passes through the lens 118 and the filter 114. The filter 114 filters out the 0th-order diffracted light, and the remaining +1st-order diffracted light and -1st-order diffracted light are then irradiated by the dichroic mirror 115. The dichroic mirror 115 is a specially coated lens. The +1st-order diffracted light and -1st-order diffracted light are refracted by the dichroic mirror 115 and refracted to the focusing unit 113. The dichroic mirror 115 is tilted at a specific angle to ensure that the propagation paths of the refracted +1st-order diffracted light and -1st-order diffracted light are aligned with the optical axis of the focusing unit 113, thereby preventing light energy loss or light direction deviation.

[0063] The focusing unit 113 converges the +1st order diffraction light and the -1st order diffraction light to interfere with each other, forming nine types of structured light with different directions and phases. The structured light excites the fluorescent-stained proteins in the biological sample, and the fluorescent signal carrying the biological sample information is received by the high-speed image compression acquisition unit 14 through the dichroic mirror 115.

[0064] Furthermore, the microscopic imaging device 100 also includes a tube lens 119, which is arranged between the dichroic mirror 115 and the high-speed image compression and acquisition unit 14. The fluorescent signal carrying the biological sample information passes through the dichroic mirror 115 and the tube lens 11925 in turn and is received by the high-speed image compression and acquisition unit 14. The tube lens 119 can further avoid light energy loss and light direction deviation.

[0065] As an implementation, the control unit 13 is further configured to control the grating image conversion frequency of the first digital micromirror 112, the displacement frequency of the piezoelectric ceramic, and the sampling frequency of the high-speed image compression acquisition unit 14. The frequencies of the various components controlled by the control unit 13 satisfy the following requirements: the grating image conversion frequency is 3M times the displacement frequency of the piezoelectric ceramic, and the piezoelectric ceramic displacement frequency is n times the sampling frequency of the high-speed image compression acquisition unit 14, where M is equal to 1, 2, or 3, and n is equal to the number of sampling depths.

[0066] Specifically, the control unit 13 is in communication with the first digital micromirror 112 , the piezoelectric ceramics and the high-speed image compression acquisition unit 14 , thereby controlling the grating image conversion frequency of the first digital micromirror 112 , the displacement frequency of the piezoelectric ceramics and the sampling frequency of the high-speed image compression acquisition unit 14 .

[0067] The control unit 13 sets the grating image conversion frequency of the first digital micromirror 112 to 3M times the displacement frequency of the piezoelectric ceramic. The first digital micromirror 112 quickly switches grating patterns of different phases and directions to generate different structured lights. When the first digital micromirror 112 operates according to the frequency, the piezoelectric ceramic synchronously completes a displacement every time it completes 3M grating transformations, ensuring that each longitudinal depth of the sample can be illuminated by 3M different structured lights within a single exposure cycle, providing sufficient information redundancy for subsequent three-dimensional super-resolution reconstruction. For example, when M=1, the first digital micromirror 112 switches the grating pattern three times, the piezoelectric ceramic moves once, and three types of structured lights illuminate a longitudinal depth position of the sample, obtaining fluorescence information of the longitudinal depth position of the sample under the illumination of the three types of structured lights.

[0068] Furthermore, the control unit 13 sets the displacement frequency of the piezoelectric ceramic to n times the sampling frequency of the high-speed image compression acquisition unit 14, where n is the number of sampling depths. The piezoelectric ceramic is configured to adjust the sample to be tested to control the periodic focusing of the structured light at different longitudinal depths of the sample to be tested. By setting the piezoelectric frequency to n times the acquisition frequency of the high-speed image compression acquisition unit 14, the control unit 13 enables the high-speed acquisition unit to complete n displacement cycles of the piezoelectric ceramic within one exposure cycle, so that each acquisition frame can simultaneously record fluorescence signals at n different longitudinal depths, and each longitudinal depth contains 3M types of fluorescence information under the structured light mode.

[0069] As an implementation method, the processing unit 15 reconstructs the single-frame fluorescence pattern obtained by the high-speed image compression acquisition unit 14 within one exposure cycle to obtain a three-dimensional image of the sample to be tested. Figure 3 As shown, the reconstruction process includes the following steps:

[0070] Step S11 : using a video reconstruction algorithm to reconstruct the fluorescence pattern into 3M*n frames of restored images each carrying fluorescence information at different longitudinal depths.

[0071] Step S12: reconstruct the 3M*n frame restoration images using a deep learning reconstruction algorithm to reconstruct 3M*n optical slice images corresponding to n different longitudinal depths.

[0072] Step S13: reconstruct the 3M*n light slice images using a light slice image reconstruction algorithm to obtain a three-dimensional image of the sample to be tested.

[0073] Specifically, a single-frame fluorescence pattern is captured by the high-speed image compression acquisition unit 14 within a single exposure cycle. Encoding techniques are used to compress fluorescence signals at multiple depths and in multiple structured light modes into a single frame. A video reconstruction algorithm is then used to reconstruct the single-frame compressed fluorescence pattern into a 3M*n frame restored image, restoring the original multi-frame sequence. Each restored image corresponds to fluorescence information at a specific longitudinal depth and structured light mode, ultimately outputting 3M*n frames of restored images carrying fluorescence information at different longitudinal depths.

[0074] The deep learning reconstruction algorithm feeds each image frame into a deep neural network encoder. Using multi-layer convolution operations, it extracts high-level features from the image. It also learns the shared and unique information between frames, extracting effective information from the image, suppressing noise, and improving resolution. Subsequently, the algorithm estimates the shared and unique information components of each frame and reconstructs them separately, fusing the shared and unique information to reconstruct 3M*n optical slice images corresponding to n different longitudinal depths.

[0075] Under structured light illumination, the structure of the focal surface of the sample is modulated by the fringes, and the focal surface has fringes; the structure of the defocused background of the sample is not affected by the fringes, and the defocused background has no fringes. The image can be expressed by the following formula:

[0076] ;

[0077] Where, Represents the focal plane portion of the sample, i.e., the optical slice image of the sample; represents the out-of-focus background portion of the sample, represents the plane coordinates, represents the imaging image, represents the mean intensity of the striped squares, represents the modulation depth, represents the fringe spatial frequency, Indicates phase.

[0078] Solving light slice images , shift the initial phase of the stripe structured light 3 times, the phase shift interval is , the phase shift combination is ( , , ), we get the following system of equations:

[0079] ;

[0080] Solving the system of equations yields:

[0081] ;

[0082] ;

[0083] Where, represents a widefield image, The light slice image with the out-of-focus background removed is obtained by solving the above equations through the phase shift of the structured light.

[0084] After obtaining 3M*n optical slice images, the optical slice image reconstruction algorithm performs phase superposition and spectral synthesis on the 3M optical slice images at each longitudinal depth, utilizing the multi-directional fringe information from the structured light illumination to recover a super-resolution planar image at that longitudinal depth. Subsequently, the n super-resolution images at each longitudinal depth are spatially arranged, and 3D reconstruction is performed to synthesize these 2D image sequences into a complete 3D super-resolution image, thereby obtaining a 3D image of the sample under test.

[0085] As an implementation method, the processing unit 15 reconstructs the restored image using a deep learning reconstruction algorithm, such as Figure 4 As shown, the reconstruction process includes the following steps:

[0086] In step S121 , the 3M*n frame restored images are encoded respectively to perform two-dimensional compression observation to obtain observation values.

[0087] Step S122 : Based on the observation values, the common information estimated observation values ​​and the unique information estimated observation values ​​of each restored image frame corresponding to the first vertical depth are calculated.

[0088] In step S123 , the common information estimated observation value corresponding to the first longitudinal depth is input into the common information reconstruction module. The common information reconstruction module reconstructs the common information estimated observation value corresponding to the first longitudinal depth to obtain the common information reconstructed by 3M frames corresponding to the first longitudinal depth.

[0089] Step S124 , performing channel dimension concatenation on the unique information estimated observation value of the first longitudinal depth of the 3M frames and the reconstructed common information of the first longitudinal depth to obtain the 3M frames of optical slice images of the first longitudinal depth.

[0090] Specifically, in the process of reconstructing the restored image using the deep learning reconstruction algorithm, a random mask matrix is ​​used to encode each frame of the image, the 3M*n frame restored images are encoded, and each image is modulated using a different random mask.

[0091] As an implementation, the observation value is expressed as follows:

[0092] ;

[0093] Where, y represents the observed value, and the observed value ;C k represents the kth random mask, and the random mask , X k represents the kth image frame, and the image frame ; w and h Represent the height and width of the image frame respectively, Represents the Hadamard product operation, B=3M*n.

[0094] In the deep learning reconstruction algorithm, 3M*n random masks are used For 3M*n frame images Through two-dimensional compression observation, the spatial information of each image is converted into the observation value y, providing a data basis for subsequent information separation and image reconstruction.

[0095] After obtaining the observations, one of the n longitudinal depths is selected, defined as the first longitudinal depth. The first longitudinal depth corresponds to 3M restored images. The deep learning reconstruction algorithm separates the shared and unique information of the 3M frames. The shared information represents the common fluorescence features across all structured light modes in the first longitudinal depth 3M frames, while the unique information represents the differences between the first longitudinal depth 3M frames in different structured light modes.

[0096] In the deep learning reconstruction algorithm, the common information estimated observation value is obtained by calculating the ratio of the observation value to the sum of the random masks. The sum of the random masks is calculated to avoid the influence of the randomness of the masks. As an implementation method, the common information estimated observation value is expressed by the following formula:

[0097] ;

[0098] Where y represents the observed value, C k represents the kth random mask, Represents the common information estimated observation value at the nth vertical depth.

[0099] In the deep learning reconstruction algorithm, the unique information estimation observation value is obtained by calculating the common information estimation value and multiplying it pixel by pixel by the mask of each frame. As an implementation method, the unique information estimation observation value is expressed by the following formula:

[0100] ;

[0101] Where, represents the common information estimated observation value at the nth vertical depth, represents the kth unique information estimated observation value, Ck represents the kth random mask.

[0102] reconstructs the common information corresponding to the first longitudinal depth from the common information estimation observation value The input common information reconstruction module includes a plurality of residual blocks, each of which is composed of a convolutional layer and an activation function. The input common information estimation observation value extracts features step by step through multi-layer convolution and activation function, avoids gradient disappearance by using residual connection, reduces the common component disturbed by low resolution or noise, and reconstructs the common information estimation observation value of the first longitudinal depth through the common information reconstruction module , and finally outputs the 3M frame reconstructed common information of the first longitudinal depth. For ease of description, the 3M frame reconstructed common information of the first longitudinal depth is denoted as x c .

[0103] Further, the 3M frame specific information estimation observation value of the first longitudinal depth and the reconstructed common information of the first longitudinal depth x c The input common information reconstruction module combines and reconstructs the 3M frame specific information estimation observation value of the first longitudinal depth and the reconstructed common information of the first longitudinal depth x c , and inputs the combined result into the subsequent residual block to generate the 3M frame light slice image of the first longitudinal depth. For ease of description, the 3M frame light slice image of the first longitudinal depth is denoted as .

[0104] As an implementation manner, after obtaining the light slice image, the processing unit 15 uses a light slice image reconstruction algorithm to reconstruct the 3M*n light slice images, including: reconstructing the light slice images corresponding to the same longitudinal depth in the 3M*n light slice images to obtain n longitudinal depth super-resolution images, and using three-dimensional image processing technology to reconstruct the n longitudinal depth super-resolution images into a three-dimensional image of the sample to be measured.

[0105] Specifically, in the super-resolution reconstruction stage, the processing unit 15 synchronously processes the 3M optical section images of the same longitudinal depth, eliminates the phase aliasing phenomenon caused by the structured light interference, and restores the continuous phase distribution. For example, when the structured light irradiates the sample in different directions (such as 0°, 60°, and 120°) and phases (0, 2π / 3, and 4π / 3), each optical section image contains the stripe modulation information of a specific direction. By solving the phase change of the stripe, the light intensity distribution of the sample at the depth is determined. The algorithm evaluates the spatial frequency of the stripe, identifies the high-frequency component and the low-frequency component. By using the spectrum synthesis technology, the high-frequency information of different directions is combined to restore a planar image with higher resolution than the original image, thereby obtaining n super-resolution images of longitudinal depths. Subsequently, the n super-resolution images of longitudinal depths are arranged in spatial order to form a three-dimensional body data, and the spatial offset is corrected through feature matching or elastic registration technology, so as to realize the reconstruction of the three-dimensional image of the sample to be measured from the n super-resolution images of longitudinal depths.

[0106] According to the above description, the microscopic imaging device 100 provided by the present application can avoid causing photo-bleaching and photo-toxicity to the active biological sample by using the structured light of different directions and phases to excite the fluorescence carrying sample information, so as to protect the sample from being damaged during observation and avoid affecting the life activities of the living sample. The fluorescence images of the sample at multiple longitudinal depth positions under the irradiation of different structured light are collected in a single exposure period, a single frame of super-resolution image in a single exposure period is reconstructed through a reconstruction algorithm, and three-dimensional reconstruction is performed based on the single frame of super-resolution image, so as to obtain a three-dimensional image of the measured sample with a hundred nanometer resolution, clearly display the cell structure distribution and continuous dynamic behavior of the living sample, and realize the instantaneous capture of the dynamic life process of the living cell sample in a single exposure period. In addition, the data storage amount is reduced through algorithm optimization, and the cost of the microscopic imaging device 100 is reduced.

[0107] It can be understood that the word "exemplary" used herein means "serving as an example, illustration, or description". Any embodiment described as "exemplary" is not necessarily superior or preferred to other embodiments and / or does not exclude features combined with other embodiments. It should be understood that certain features of the present application described in the context of separate embodiments can also be provided in combination. Conversely, various features of the present application described in the context of a single embodiment can also be provided separately or in any suitable combination or as any other described embodiment of the present application.

[0108] The above disclosed is only the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Those skilled in the art can understand that changes, modifications, substitutions, combinations, simplifications, etc. made without departing from the spirit and scope of the present application and the appended claims should be equivalent replacements and still fall within the scope of the present application.

Claims

1. A microscopic imaging device, characterized in that: The microscopic imaging device is applied to three-dimensional microscopic imaging of living cells, and the microscopic imaging device comprises: a structured light generating unit, wherein the structured light generating unit is configured to periodically generate at least three types of structured light, wherein the structured light converges on the sample to be tested to excite fluorescence carrying sample information; A sample stage for holding the sample to be tested, wherein the sample stage is provided with a piezoelectric ceramic, and the piezoelectric ceramic is configured to adjust the sample to be tested so as to control the periodic focusing of the structured light at different longitudinal depths of the sample to be tested; A control unit configured to set the number of sampling depths of the microscopic imaging device and control the piezoelectric ceramic to complete n times of longitudinal depth change of the sample to be measured within a single exposure cycle of the high-speed image compression acquisition unit, where n is the set number of sampling depths; a high-speed image compression acquisition unit configured to sample fluorescence at different longitudinal depths of the sample to be tested to obtain a fluorescence pattern, and transmit the fluorescence pattern to a processing unit, wherein the fluorescence pattern includes fluorescence information at n longitudinal depths, and any longitudinal depth includes fluorescence information excited by at least three types of the structured light; a processing unit, wherein the processing unit reconstructs a single-frame fluorescence pattern obtained in any exposure cycle of the high-speed image compression acquisition unit to obtain a three-dimensional image of the sample to be tested; The structured light generating unit includes: a laser for emitting laser light; a first digital micromirror configured to receive the laser light and form a grating image to modulate the laser light to generate diffracted light of multiple diffraction orders; a light focusing unit configured to converge the plurality of diffracted lights generated by the first digital micromirror, causing the diffracted lights to interfere with each other to form structured light, and to focus the structured light on the sample to be measured to excite fluorescence carrying sample information; The control unit is further configured to control a grating image conversion frequency of the first digital micromirror, a displacement frequency of the piezoelectric ceramic, and a sampling frequency of the high-speed image compression acquisition unit, wherein the frequencies of the various components controlled by the control unit satisfy: the grating image conversion frequency is 3M times the displacement frequency of the piezoelectric ceramic, and the displacement frequency of the piezoelectric ceramic is n times the sampling frequency of the high-speed image compression acquisition unit, wherein M is equal to 1, 2, or 3, and n is equal to the number of sampling depths; The processing unit reconstructs the single-frame fluorescence pattern obtained within one exposure cycle of the high-speed image compression acquisition unit to obtain a three-dimensional image of the sample to be tested, comprising: Reconstructing the fluorescence pattern into 3M*n frames of restored images each carrying fluorescence information at different longitudinal depths using a video reconstruction algorithm; A deep learning reconstruction algorithm is used to reconstruct the restored images of the 3M*n frames to reconstruct 3M*n optical slice images corresponding to n different longitudinal depths; Reconstructing the 3M*n light slice images using a light slice image reconstruction algorithm to obtain a three-dimensional image of the sample to be tested; The processing unit reconstructing the restored image using a deep learning reconstruction algorithm includes: Encoding the restored images of the 3M*n frames respectively to perform two-dimensional compression observation to obtain observation values; Based on the observation values, calculating the common information estimation observation values ​​and the unique information estimation observation values ​​of the restored image frames corresponding to the first longitudinal depth; Inputting the common information estimated observation value corresponding to the first longitudinal depth into a common information reconstruction module, the common information reconstruction module reconstructing the common information estimated observation value corresponding to the first longitudinal depth to obtain common information reconstructed from 3M frames corresponding to the first longitudinal depth; Performing channel dimension concatenation on the unique information estimation observation value of the first longitudinal depth of the 3M frames and the reconstructed common information of the first longitudinal depth to obtain the optical slice image of the 3M frames of the first longitudinal depth; The processing unit is configured to solve the light slice image using the following formula to remove the defocused background of the light slice image to obtain a light slice image with the defocused background removed. The solving formula of the light slice image is as follows: ； The solution formula of the light slice image is obtained by solving the equation group, which is obtained in the following way: based on the imaging image, the initial phase of the stripe structured light is shifted 3 times, and the phase shift interval is , the phase shift combination is ( , , ), to obtain the system of equations; The imaging images are shown below: ； Where, Represents the focal plane portion of the sample, i.e., the optical slice image of the sample; represents the out-of-focus background portion of the sample, represents the plane coordinates, represents the imaging image, represents the mean intensity of the striped squares, represents the modulation depth, represents the fringe spatial frequency, Indicates phase; The system of equations is expressed as follows: ； Where, represents the mean intensity of the striped squares, represents the modulation depth, represents the out-of-focus background portion of the sample, Represents the focal plane portion of the sample, i.e., the optical slice image of the sample; The processing unit performs phase superposition and spectrum synthesis based on the 3M light slice images with the defocused background removed at each longitudinal depth position, and uses the multi-directional fringe information of structured light illumination to restore the super-resolution planar image of the longitudinal depth. Subsequently, the n super-resolution images of the longitudinal depth are arranged in spatial order, and these two-dimensional image sequences are synthesized into a complete three-dimensional super-resolution image through three-dimensional reconstruction, thereby obtaining a three-dimensional image of the sample to be tested.

2. The microscopic imaging device according to claim 1, characterized in that The observed value is expressed by the following formula: ； Where, represents the observed value, is the kth random mask, is the kth image frame, w and h are the height and width of the image frame, Represents the Hadamard product operation, B=3M*n.

3. The microscopic imaging device according to claim 2, characterized in that The common information estimated observation value is expressed by the following formula: ； Where, represents the observed value, is the kth random mask, Represents the common information estimated observation value at the nth vertical depth.

4. The microscopic imaging device according to claim 3, characterized in that The unique information estimated observation value is expressed by the following formula: ； Where, represents the common information estimated observation value at the nth vertical depth, represents the k-th unique information estimated observation.

5. The microscopic imaging device according to claim 1, characterized in that The processing unit reconstructs 3M*n light slice images using a light slice image reconstruction algorithm, including: The light slice images corresponding to the same longitudinal depth in the 3M*n light slice images are reconstructed to obtain super-resolution images of n longitudinal depths, and the super-resolution images of n longitudinal depths are reconstructed into a three-dimensional image of the sample to be tested using three-dimensional image processing technology.

6. The microscopic imaging device according to claim 1, characterized in that The first digital micromirror is configured to modulate the laser to generate three diffraction orders of 0, +1, and -1; The microscopic imaging device further includes a filter configured to filter out the 0th order diffraction light and allow the +1st order diffraction light and the -1st order diffraction light to pass.

7. The microscopic imaging device according to claim 6, characterized in that The microscopic imaging device further includes a dichroic mirror configured to reflect the +1st order diffraction light and the -1st order diffraction light to the focusing unit; The focusing unit converges the +1st order diffraction light and the -1st order diffraction light to generate interference, thereby forming the structured light.

Images