Imaging device and method for multi-label fluorescence lifetime spectrum

Abstract

The invention relates to an imaging device and method for a multi-label fluorescence lifetime spectrum. The device comprises an excitation light source, a sensor and a microcontroller (MCU). The laser light source is located right above the imaging device and emits exciting light which directly irradiates the image sensor below the laser light source so as to excite the fluorescence sample to enable the fluorescence sample to emit a fluorescence signal. The image sensor is located under the imaging device and used for placing a fluorescence sample and making direct contact with the fluorescence sample. The MCU outputs two paths of control signals, the first control signal controls the on and off of the excitation light source, and the second control signal controls the image sensor to start to collect a fluorescence signal emitted by the fluorescence sample excited by the excitation light source. The image sensor collects a fluorescence signal emitted by the fluorescence sample based on the second control signal. A fluorescence signal collected by the image sensor is reconstructed by the multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

Description

Technical Field

[0001] The present invention relates to the technical field of fluorescence microscopy, and in particular to an imaging device and method for multi-label fluorescence lifetime spectrum. Background Art

[0002] Lensless microscopy typically uses off-the-shelf components such as monochromatic light-emitting diodes (LEDs), complementary metal-oxide semiconductors (CMOS), and charge-coupled devices (CCDs), making imaging systems very inexpensive, compact, and easy to maintain. Furthermore, the sample is placed directly on the surface of the imaging sensor for observation, thus providing a wide field of view for high-throughput applications. Shadow imaging is a common brightfield lensless imaging scheme. Its operating principle is to illuminate the sample and capture two-dimensional (2D) spatial features (shadows) with an electronic image detector. Furthermore, by using a coherent light source (i.e., a monochromatic wave), the spatial features are converted into interference patterns, and holographic information is obtained according to scalar diffraction theory. Finally, using numerical methods such as phase inversion, the sample's shape details on a specific surface can be reconstructed. However, in many cases, brightfield microscopy cannot meet the requirements of specific applications. In these cases, fluorescent probes are necessary for observing cellular structures or detecting pathologies due to their high specificity and sensitivity. Therefore, fluorescence imaging is indispensable in modern clinical diagnostics and biological research.

[0003] Lensless fluorescence microscopy faces two key challenges. First, the incoherent point spread function (PSF) of fluorescence lacks the high-frequency information necessary for high-resolution image reconstruction. Therefore, digital holography and other interferometric measurement schemes cannot be used. Second, fluorescence microscopy requires the use of interference-based or absorption-based filters. Excitation light is typically much more intense than fluorescence light. To obtain a pure fluorescence signal, lensless fluorescence microscopy requires thick filters based on thin-film interferometry, which causes the fluorescence signal to travel far from the sensor. Because lensless systems lack lenses to focus light, the fluorescence signal propagates in all directions, and the signal-to-noise ratio and resolution decrease with distance from the sensor. A first solution to this problem is to add an amplitude modulation mask above the filter to modulate the incoherent PSF, thereby recovering high-frequency spatial information. However, due to the use of an amplitude modulation mask, this approach results in a low signal-to-noise ratio. Another option is to embed a metal grid within the filter to restrict the direction in which the fluorescence signal propagates. While this approach offers a higher signal-to-noise ratio, the processing technology is complex. A third solution is to use total internal reflection to reduce the excitation light intensity. Because the total internal reflection (TIR) ​​process is so robust at suppressing excitation sources, high-end fluorescence filters based on thin-film interferometry are unnecessary. Inexpensive plastic absorption filters can be used to block scattered pump photons that violate the TIR condition, providing a better dark-field background. Although the filter thickness is reduced, the fluorescence signal still scatters, resulting in insufficient spatial resolution.

[0004] In summary, a key issue with lensless fluorescence microscopy is that the presence of filters causes the fluorescence signal to propagate a long distance in free space before being received by the sensor, ultimately resulting in poor spatial resolution. Therefore, an imaging method and device that can reduce the effect of filters on the fluorescence signal is needed. Summary of the Invention

[0005] In order to solve at least part of the above problems, the present application provides a multi-label fluorescence lifetime spectrum imaging device and a corresponding method.

[0006] According to one aspect of the present application, a multi-label fluorescence lifetime spectrum imaging device includes: an excitation light source, an image sensor, and a microcontroller MCU; wherein the laser light source is located directly above the imaging device, and emits excitation light that directly illuminates the image sensor below, and the excitation light is used to excite a fluorescent sample so that the fluorescent sample emits a fluorescent signal; wherein the image sensor is located directly below the imaging device, and is used to place the fluorescent sample and directly contact the fluorescent sample; wherein the MCU outputs two control signals, the first control signal of the two control signals controls the turning on and off of the excitation light source, and the second control signal of the two control signals controls the image sensor to start collecting the fluorescent signal emitted by the fluorescent sample excited by the excitation light source; wherein the image sensor receives the second control signal from the MCU, and collects the fluorescent signal emitted by the fluorescent sample based on the second control signal; and wherein the fluorescence signal collected by the image sensor is reconstructed by a multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

[0007] According to another aspect of the present application, a multi-label fluorescence lifetime spectrum imaging method includes: a microcontroller MCU outputs two control signals, the first control signal of the two control signals is used to control the turning on and off of the excitation light source, and the second control signal of the two control signals is used to control the image sensor to start collecting the fluorescence signal emitted by the fluorescence sample excited by the excitation light source; under the control of the first control signal, the excitation light source is turned on at a predetermined time point and turned off when the excitation light intensity reaches a predetermined value; and under the control of the second control signal, the image sensor collects the fluorescence signal at a specific time point, wherein the specific time point is selected from the time period when the excitation light decays to 0 but the fluorescence signal intensity is not 0, so that the sensor can directly collect the fluorescence signal without the need for a filter to filter the excitation light; the fluorescence signal collected by the image sensor is reconstructed by the multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

[0008] The multi-label fluorescence lifetime spectrum imaging device and method according to the present invention achieves high-resolution reconstruction of fluorescence lifetime spectrum and multi-channel color fluorescence images without the need for filters. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Hereinafter, the above and other purposes, features, and advantages of the present invention will become more apparent by describing in detail embodiments of the present invention in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same elements throughout. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the present invention.

[0010] Figure 1 A schematic diagram of an exemplary imaging device for multi-label fluorescence lifetime spectroscopy according to an embodiment of the present application is shown;

[0011] Figure 2 A flow chart showing a method for imaging multi-label fluorescence lifetime spectroscopy according to an exemplary embodiment of the present application is shown;

[0012] Figure 3 A flowchart showing a method for imaging multi-label fluorescence lifetime spectrum according to another exemplary embodiment of the present application is shown; and

[0013] Figure 4 The figure shows the experimental results of fluorescence imaging performed according to the imaging method for multi-label fluorescence lifetime spectrum proposed in this application.

[0014] Those skilled in the art will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. DETAILED DESCRIPTION

[0015] The following description is provided to enable those skilled in the art to understand, make and use the invention as described herein and as expressly claimed herein. Because the specific combination of each feature will produce a large number of practical embodiments in which the present invention can be practiced, and for the purpose of providing a reasonably clear and concise description, only the preferred embodiment will be presented herein. However, it should be recognized that other embodiments not expressly described herein can be practiced by those of ordinary skill in the art in a similar manner. Thus, any assessment of the scope of the present invention should be made with respect to the claims expressly provided herein and interpreted in light of the present description and the general knowledge and technical level in the art in its broadest reasonable interpretation. Nothing in this description is intended to limit the spirit and scope of the present invention.

[0016] The imaging device and method for multi-label fluorescence lifetime spectrum according to embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely illustrative and should not be construed as limiting the scope of the present invention.

[0017] Figure 1 FIG2 shows a schematic diagram of an exemplary imaging device for multi-label fluorescence lifetime spectrum according to an embodiment of the present application. Figure 1 According to an exemplary embodiment of the present application, an imaging device 100 for multi-label fluorescence lifetime spectroscopy includes: an excitation light source 110, an image sensor 120, and a microcontroller (MCU) 130. Figure 1 As shown, as an example, a laser light source 110 can be located directly above the imaging device 100. The laser light source 110 emits excitation light 140 that directly illuminates the image sensor 120 below. The excitation light 140 is used to excite a fluorescent sample, causing it to emit a fluorescent signal 150. The image sensor 120 can be located directly below the imaging device 100. As an example, the image sensor 120 is used to support the fluorescent sample illuminated by the laser light source 110 and is in direct contact with the fluorescent sample without the need for additional filters. The MCU 130 can control the overall operation of the imaging device 100, such as light source illumination, signal acquisition, and image reconstruction. As an example, the MCU 130 generates and outputs two control signals: a first control signal 160 (i.e., an excitation light source control signal) and a second control signal 170 (i.e., a sensor control signal). The first control signal 160 can control the on / off switching of the excitation light source 110, and the second control signal 170 (i.e., a sensor control signal) can control the image sensor 120 to begin acquiring the fluorescent signal emitted by the fluorescent sample excited by the excitation light source.

[0018] Image sensor 120 receives a second control signal from MCU 130 and, based on the second control signal, collects a fluorescence signal emitted by the fluorescent sample. MCU 130 reconstructs fluorescence signal 150 collected by image sensor 120 using a multi-label fluorescence lifetime spectrum reconstruction network to output a fluorescence lifetime spectrum and a multi-channel color fluorescence image, thereby achieving imaging.

[0019] Compared with traditional fluorescence imaging devices (such as microscopes based on objective lenses), the imaging device 100 for multi-label fluorescence lifetime spectroscopy proposed in this application has the following advantages:

[0020] (1) Since it does not use a complex optical system, it greatly reduces the cost of use, and significantly reduces the weight and volume. At the same time, it is no longer constrained by the field of view and resolution, and its space-bandwidth product can theoretically be infinitely expanded.

[0021] (2) Since the excitation light signal and the fluorescence signal are misaligned in the time dimension, the excitation light can be filtered without using a filter, so that the fluorescence sample can be closely fitted with the image sensor, greatly improving the resolution.

[0022] According to further exemplary embodiments, the excitation light source 110 may include, but is not limited to, an LED light source, a laser light source, or a continuous spectrum light source. The wavelength range of the excitation light source 110 may be from the ultraviolet spectrum to the infrared spectrum.

[0023] According to a further exemplary embodiment, the image sensor 120 may be any sensor capable of acquiring a two-dimensional image, for example, may be a two-dimensional area array sensor.

[0024] According to further exemplary embodiments, the image sensor 120 may employ at least one of a rolling shutter sensor supporting a global reset release (GRR) mode, or a sensor supporting a global shutter.

[0025] According to a further exemplary embodiment, the two control signals output by the MCU 130 may be transistor-transistor logic (TTL) control signals for respectively controlling the excitation light source 110 and the image sensor 120. As an example, the TTL signal validity may be one or more of: high-level validity, low-level validity, rising-edge validity, and falling-edge validity.

[0026] According to a further exemplary embodiment, the MCU 130 may be, for example, but not limited to, an MCU of the model STM32F103C8T6.

[0027] According to a further exemplary embodiment, the first control signal 160 can be configured to control the excitation light source 110 to turn on at a predetermined time point and to turn off when the excitation light intensity reaches a predetermined value. The second control signal 170 can be configured to control the image sensor 120 to collect the fluorescence signal 150 at a specific time point, wherein the specific time point is selected from a time period when the excitation light decays to 0 but the fluorescence signal intensity is not 0, so that the image sensor 120 can directly collect the fluorescence signal 150 without the need for a filter to filter the excitation light first. Turning on the excitation light source to irradiate the fluorescent substance for a period of time can generate a fluorescence signal. After turning off the excitation light source, the excitation light source intensity is 0 at this time, but because the intensity decay rate of the fluorescence signal is less than the intensity decay rate of the excitation light source, the fluorescence signal intensity is not 0. Therefore, during this time period, the control sensor can directly perform image acquisition without using a filter to filter the excitation light. In other words, by setting the two control signals output by the MCU in the above manner, the sensor can directly collect the fluorescence signal 150 without the need for a filter to filter the excitation light.

[0028] According to a further exemplary embodiment, M grayscale images (i.e., true degraded fluorescence images) can be obtained based on the collected fluorescence signal 150, and a fluorescence lifetime spectrum and a multi-channel color fluorescence image can be obtained therefrom. For example, a multi-label fluorescence lifetime spectrum reconstruction network can be used to obtain multiple frames of grayscale images from the fluorescence signal 150 collected by the image sensor 120, and the intensity information differences between the multiple frames of grayscale images can be used to reconstruct a shape of The fluorescence lifetime spectrum T, and the shape of the fluorescence lifetime spectrum obtained by digital filtering is The multi-channel color fluorescence image I, where a is the magnification factor ( ), N is the number of channels, H and W are the height and width of the image respectively. In addition, each channel represents a fluorescence lifetime, and for a coordinate in space The shape of its lifetime spectrum is .

[0029] According to a further exemplary embodiment, for any pixel of the image sensor 120 , the signal received comes from fluorescent molecules with different fluorescence lifetimes, and the fluorescence lifetime spectrum represents the ratio of fluorescent molecules with different lifetimes in the signal received by each pixel.

[0030] According to a further exemplary embodiment, the two control signals output by the MCU can be obtained by a multi-label fluorescence lifetime spectrum reconstruction network framework.

[0031] According to a further exemplary embodiment, obtaining the control signal through a multi-label fluorescence lifetime spectrum reconstruction network framework may include:

[0032] A lifetime distribution of fluorescence lifetimes to be reconstructed is determined, where the number of fluorescence lifetimes is N and N is a positive integer.

[0033] Select a dataset that contains multiple channels, for example, the number of channels of the images in the dataset is C, where C is a positive integer and As an example, data from N channels can be selected from a dataset containing C channels as training data. As an example, the acquired dataset can be divided into a training set and a test set according to a pre-set appropriate ratio (e.g., an 8:2 ratio). Each channel represents a fluorescence lifetime, and its intensity represents the ratio of the current lifetime to all fluorescence lifetimes that need to be reconstructed.

[0034] The selected training data is fed into the degradation model of the neural network, and the degradation model includes a fluorescence attenuation model, a fuzzy model, and a noise model.

[0035] The fluorescence decay model is expressed as follows: for each lifetime The sum of the intensities of the fluorescence signals of all different fluorescence lifetimes at different times in a fluorescent sample is:

[0036]

[0037] in, Representation Space The sum of the fluorescence signal intensities of all different fluorescence lifetimes at coordinate t and time t, Representation Space The intensity of the fluorescence signal at the coordinate and with a fluorescence lifetime of τ, Represents the acquisition parameters, which represent the moment of collecting fluorescence signals and can be set as optimizable parameters. The T on the right side of the above formula, namely the fluorescence lifetime spectrum, is a matrix related to the space (x, y) and the fluorescence lifetime. Each value represents the initial intensity of the fluorescence signal with a fluorescence lifetime of τ at the space (x, y). Because the intensity of the fluorescence signal decays over time, at time t, the integral of the fluorescence signals with different fluorescence lifetimes will have different values. .

[0038] Among them, the fuzzy model and noise model are expressed as:

[0039]

[0040] in, is the fluorescence signal after blur degradation and noise degradation.

[0041] in represents the blur function and represents the optical and electrical crosstalk of the sensor. Assuming that the crosstalk effect is uniform and linear, the blur effect can be modeled as follows:

[0042]

[0043] in represents the blur kernel, Represents the spatial convolution operator.

[0044] The fluorescence signal degraded by the fuzzy model is expressed as .

[0045] Typically, imaging noise consists of shot noise , dark noise , and readout noise (Gaussian noise) It should be noted that for the sake of convenience, only simple 、 and To represent shot noise, dark noise and Gaussian noise respectively, the detailed description is given below and will not be repeated here. Therefore, the formula of the noise function is as follows:

[0046]

[0047] Specifically, shot noise originates from the randomness of photon incidence and photoelectric conversion, and the expected number of electrons in shot noise is . dark noise It is randomly generated by electrons and holes in the depletion region that are independent of the incident signal. Therefore, the probability distribution of dark noise can be expressed as ,in express The readout noise follows a Gaussian distribution. ,in Represents the variance of the readout noise. Without loss of generality, we use Gaussian to approximate Poisson noise, and the noise can be expressed as:

[0048]

[0049] in , .

[0050] Furthermore, as an example, the multi-label fluorescence lifetime spectrum reconstruction network can contain three branches, namely the main Transformation branch, local information branch and global information branch, 、 、 The virtual τ domain filter guided by global and local information is given by The input of the reconstruction network is several Measurements collected under different acquisition parameters. The fluorescence lifetime spectrum T is extracted from the input. At the same time, the spatial local and global features, i.e. 、 Depend on 、 These features are then fed into the guided filter module Used to improve the accuracy of virtual filtering and ultimately reconstruct multi-channel color fluorescence images The process can be described as:

[0051]

[0052] Training the degradation model using training data to obtain a measurement value corresponding to the fluorescence signal in the imaging device;

[0053] The acquisition parameter t in the fluorescence decay model is optimized to construct a lifetime spectrum reconstruction neural network to reconstruct the fluorescence lifetime spectrum from the measured values ​​and obtain the optimal acquisition parameter under the current conditions. ;

[0054] The optimal acquisition parameters Loaded into MCU 130, two control signals are obtained to finally obtain the fluorescence lifetime spectrum and multi-channel color fluorescence images .

[0055] Since the mathematical operations involved in the constructed differentiable imaging model and the fluorescence lifetime spectrum reconstruction neural network are all differentiable operations, they can be implemented in software using the scientific computing library Pytorch. On this basis, the differentiable imaging model and the deep reconstruction neural network are encapsulated into an end-to-end training framework. The framework takes the training set as input and reconstructs the fluorescence lifetime spectrum by training the lifetime spectrum reconstruction neural network. and multi-channel color fluorescence images Relying on the design of this framework, the shared propagation of reverse gradients can be realized, thereby achieving the acquisition parameters In addition, it should be noted that the differentiable degradation model can be used to simulate the measurement of the fluorescent sample signal by the imaging device, that is, the measured value of the image after degradation .

[0056] Reconstructing the neural network based on the fluorescence lifetime spectrum can obtain the corresponding fluorescence lifetime spectrum and multi-channel color fluorescence images The proposed fluorescence lifetime spectrum reconstruction neural network outputs two types of results, namely lifetime spectrum And the multi-channel multi-label image calculated by virtual filtering ; In order to supervise the fluorescence lifetime spectrum reconstruction neural network, we designed a corresponding loss function, which is composed of the lifetime spectrum loss and multi-label intensity map loss Composition; among them, the pre-filtering loss and guided filter loss constitute ;

[0057]

[0058] Where α and β are weights that balance these losses, 、 and represent the estimated fluorescence lifetime spectrum, the intermediate estimation of the multi-channel color fluorescence image by the preliminary filters in the guided filter block, and the final multi-channel color fluorescence image, respectively; and Represents the true value corresponding to the lifetime spectrum and multi-channel color fluorescence image. Lifetime spectrum loss It is used to supervise the lifetime spectrum T, which is defined as:

[0059]

[0060] in represents the sum of squared error loss; represents the weight of the sum of squared errors, represents the weight of the gradient loss, represents the weight of the fast Fourier transform. Denotes the gradient loss:

[0061]

[0062] in represents the gradient operation along the x and y directions. N is the normalization factor, which represents the total number of elements in the variable. and They represent the output of the network (e.g. 、 and ) and the corresponding true value in the loss function. represents the loss in the Fast Fourier Transform (FFT) domain and is calculated as follows:

[0063]

[0064] in, represents the fast Fourier transform.

[0065] Initial filtering loss and guided filter loss They are defined on the preliminary filtered channel color fluorescence image and the final filtered channel color fluorescence image, respectively, and they are both calculated using the same function.

[0066]

[0067] in, and They are the common mean absolute error (MAE) loss and structural similarity index metric (SSIM) loss, Represents the weight of the structural similarity index measurement.

[0068] The t in the decay model is set as an optimizable parameter and optimized training is performed. After the training, the optimal acquisition parameter t under the current conditions and the weight of the life spectrum reconstruction neural network are obtained to construct the life spectrum reconstruction neural network.

[0069] The optimal acquisition parameter t obtained can be loaded into the MCU to generate a control signal for controlling the excitation light source and the image sensor. Under the control of the control signal, M real degraded fluorescence images are acquired. ,in M. Then, Enter the life spectrum reconstruction neural network and finally obtain an image with the number of channels Fluorescence lifetime spectrum of , and the number of channels is Multi-channel color fluorescence images .

[0070] Figure 2 FIG. 4 is a flowchart illustrating an exemplary method 200 for imaging multi-label fluorescence lifetime spectroscopy according to an embodiment of the present application.

[0071] Reference Figure 2 According to an exemplary embodiment, in step S210, the MCU outputs two control signals, wherein the first control signal of the two control signals is used to control the turning on and off of the excitation light source, and the second control signal of the two control signals is used to control the image sensor to start collecting the fluorescence signal emitted by the fluorescence sample excited by the excitation light source.

[0072] In step S220, under the control of the first control signal, the excitation light source is turned on and / or off. For example, the excitation light source may be turned on at a predetermined time to excite the fluorescent sample to emit a fluorescent signal, and turned off when the excitation light intensity reaches a predetermined value.

[0073] In step S230, under the control of the second control signal, the image sensor collects the fluorescence signal at a specific time point. After turning on the excitation light source to irradiate the fluorescent substance and then turning off the excitation light source, the intensity decay rate of the fluorescence signal is less than the intensity decay rate of the excitation light source. Therefore, after turning off the excitation light source, the sensor can be controlled to perform an image acquisition operation. At this time, the intensity of the excitation light source is 0, but the intensity of the fluorescence signal is not 0, so there is no need to use a filter to filter the excitation light. For example, the specific time point for collecting the fluorescence signal can be selected from a time period when the excitation light decays to 0 but the fluorescence signal intensity is not 0, so that the sensor can directly collect the fluorescence signal without the need for a filter to filter the excitation light.

[0074] Furthermore, in step S240 , the fluorescence signal collected by the image sensor is reconstructed by a multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

[0075] According to a further exemplary embodiment, the above step S240 may include: obtaining M grayscale images (i.e., true degraded fluorescence images) based on the collected fluorescence signal 150, and obtaining a fluorescence lifetime spectrum and a multi-channel color fluorescence image based on the M grayscale images. For example, a multi-label fluorescence lifetime spectrum reconstruction network may be used to obtain multiple frames of grayscale images from the fluorescence signal collected by the image sensor, and the intensity information difference between the multiple frames of grayscale images may be used to reconstruct a shape of The fluorescence lifetime spectrum T is obtained, and the fluorescence lifetime spectrum is digitally filtered to obtain a multi-channel color fluorescence image I with a shape of N*H*W, where a is the magnification factor ( ), N is the number of channels, H and W are the height and width of the image respectively. In addition, each channel represents a fluorescence lifetime, and for a coordinate in space The shape of the fluorescence lifetime spectrum is .

[0076] Figure 3 FIG. 3 is a flow chart illustrating an exemplary method 300 for imaging multi-label fluorescence lifetime spectra according to another embodiment of the present application.

[0077] Reference Figure 3 According to another exemplary embodiment of the present application, the two control signals output by the MCU in the above step S210 can be obtained by a multi-label fluorescence lifetime spectrum reconstruction network framework. Figure 3 As shown, as an example, obtaining two control signals by a multi-label fluorescence lifetime spectrum reconstruction network may include the following steps.

[0078] In step 310 , after determining the lifetime distribution of the fluorescence lifetimes that need to be reconstructed, the number of fluorescence lifetimes is N.

[0079] In step 320, a data set containing multiple channels is selected, where the number of channels of the image in the data set is C, and the requirement is , and select N channels of data in the dataset as training data.

[0080] In step 330 , the training data is fed into a degradation model of the multi-label fluorescence lifetime spectrum reconstruction network, where the degradation model may include a fluorescence decay model, a fuzzy model, and a noise model.

[0081] The fluorescence decay model is expressed as:

[0082]

[0083] in, Representation Space The sum of the fluorescence signal intensities of all different fluorescence lifetimes at coordinate t and time t, Representation Space The intensity of the fluorescence signal at the coordinate and with a fluorescence lifetime of τ, represents the acquisition parameters, which represent the moment when the fluorescence signal is acquired and can be set as optimizable parameters.

[0084] The fuzzy model is expressed as:

[0085]

[0086] in represents the blur kernel, represents the spatial convolution operator, represents the fuzzy function, represents the fluorescent signal after blurring;

[0087] Noise models include shot noise, dark noise, Gaussian noise (readout noise),

[0088] Shot noise is expressed as:

[0089]

[0090] The specific meaning is: in pixels The intensity of the fluorescent signal after blurring Probability Following a Poisson distribution, It is expressed as the number of electrons expected to be collected;

[0091] Dark noise is expressed as:

[0092]

[0093] where the expected number of dark current electrons in each pixel is

[0094] Gaussian noise is expressed as:

[0095]

[0096] in, is the variance of the noise, with a mean of 0; It means that given a mean of 0 and a variance of Under the condition of , the probability density function of the random variable x;

[0097] After the above-mentioned simulation degradation model, the corresponding measurement value of the fluorescence signal in the imaging device is obtained.

[0098] In step 340, the acquisition parameter t in the decay model is optimized to construct a lifetime spectrum reconstruction neural network to reconstruct the fluorescence lifetime spectrum from the measured values ​​and obtain the optimal acquisition parameter t under the current conditions. optimal ;

[0099] In step 350, the optimal acquisition parameter t optimal Load it into the MCU to obtain two control signals.

[0100] According to a further exemplary embodiment, the image sensor will collect M grayscale images (i.e., real degraded fluorescence images) based on the second control signal. Then, the M grayscale images collected by the image sensor can be passed into the reconstruction neural network to finally obtain an imaging image. For example, the imaging image can be a channel number of The fluorescence lifetime spectrum and channel number are Multi-channel color fluorescence image.

[0101] Figure 4 The figure shows the experimental results of fluorescence imaging performed according to the imaging method for multi-label fluorescence lifetime spectrum proposed in this application.

[0102] like Figure 4 As shown on the left, the multi-frame grayscale images collected by the sensor as the input of the multi-label fluorescence lifetime spectrum reconstruction network obtained through simulation experiments are given, as well as the multi-channel color fluorescence image as the output of the multi-label fluorescence lifetime spectrum reconstruction network. Figure 4 As shown on the right side of , the fluorescence lifetime spectrum obtained through simulation experiments is given, where the horizontal axis represents the different fluorescence lifetimes, and the vertical axis represents the corresponding signal intensity.

[0103] Depend on Figure 4 As can be seen, the errors in both the fluorescence lifetime spectrum and the multi-channel color fluorescence image output by the method of the present application are very small compared to the true value results. In other words, the fluorescence lifetime spectrum and the multi-channel color fluorescence image can be accurately reconstructed. Therefore, the method of the present application can acquire the required fluorescence signal without filters and accurately reconstruct the fluorescence lifetime spectrum and the multi-channel color fluorescence image based on this, thus achieving filter-free imaging.

[0104] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but should be given a full scope consistent with the language of the claims, where reference to an element in the singular is not intended to mean "one and only one" (unless explicitly stated as such), but rather "one or more." Unless otherwise explicitly stated, the term "some" refers to one or more. All structures and functions of the elements throughout the various aspects described in this disclosure that are known or later will be known to those of ordinary skill in the art, and their equivalents, are covered by the claims.

[0105] It should be understood by those skilled in the art that the scope of protection claimed in this application is not limited to the precise configuration and components shown above. Various modifications, changes and variations may be made to the arrangement, operation and details of the above methods and apparatus without departing from the scope of the claims.

Claims

1. An imaging device for multi-label fluorescence lifetime spectroscopy, comprising: Excitation light source, image sensor, and microcontroller MCU; The laser light source is located directly above the imaging device, and emits excitation light that directly illuminates the image sensor below. The excitation light is used to excite the fluorescent sample so that the fluorescent sample emits a fluorescent signal. The image sensor is located directly below the imaging device and is used for placing the fluorescent sample and directly contacting the fluorescent sample; The MCU outputs two control signals, a first control signal of the two control signals controls turning on and off the excitation light source, and a second control signal of the two control signals controls the image sensor to start collecting a fluorescence signal emitted by a fluorescence sample excited by the excitation light source; The image sensor receives the second control signal from the MCU, and collects the fluorescence signal emitted by the fluorescence sample based on the second control signal; and The fluorescence signal collected by the image sensor is reconstructed by a multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

2. The imaging device according to claim 1, wherein The excitation light source includes at least one of an LED light source, a laser light source or a continuous spectrum light source.

3. The imaging device according to claim 1, wherein The image sensor is a two-dimensional area array sensor.

4. The imaging device according to claim 1, wherein The image sensor adopts at least one of a rolling shutter sensor supporting a global reset release mode or a sensor supporting a global shutter.

5. The imaging device according to claim 1, wherein The two control signals output by the MCU are transistor-transistor logic level control signals.

6. The imaging device according to claim 1, wherein The model of the MCU used is STM32F103C8T6.

7. The imaging device according to claim 1, wherein: The first control signal is configured to control the excitation light source to turn on at a predetermined time point and to turn off when the excitation light intensity reaches a predetermined value; and The second control signal is configured to control the image sensor to collect a fluorescence signal at a specific time point, wherein the specific time point is selected from a time period when the excitation light decays to 0 but the fluorescence signal intensity is not 0, so that the image sensor can directly collect the fluorescence signal without the need for a filter to filter the excitation light.

8. The imaging device according to claim 1, wherein Reconstructing the fluorescence signal collected by the image sensor includes: The multi-label fluorescence lifetime spectrum reconstruction network obtains multiple frames of grayscale images from the fluorescence signals collected by the image sensor, and uses the intensity information differences between the multiple frames of grayscale images to reconstruct the shape of The fluorescence lifetime spectrum is obtained by digitally filtering the fluorescence lifetime spectrum to obtain a shape of Multi-channel color fluorescence image, where a is the magnification factor ( ), N is the number of channels of the multi-channel color fluorescence image, H and W are the height and width of the multi-channel color fluorescence image, and each of the N channels represents a fluorescence lifetime. For a coordinate in the multi-channel color fluorescence image, The shape of the fluorescence lifetime spectrum is .

9. The imaging device according to claim 8, wherein For each pixel of the image sensor, the signal received comes from fluorescent molecules with different fluorescence lifetimes. The fluorescence lifetime spectrum represents the ratio of fluorescent molecules with different fluorescence lifetimes in the signal received by each pixel.

10. The imaging device according to claim 1, wherein The two control signals output by the MCU are obtained by the multi-label fluorescence lifetime spectrum reconstruction network.

11. The imaging device according to claim 10, wherein Obtaining the two control signals by the multi-label fluorescence lifetime spectrum reconstruction network includes: Determining a lifetime distribution of fluorescence lifetimes represented by N channels that need to be reconstructed, where the number of fluorescence lifetimes is N and N is a positive integer; Select a dataset containing C channels, where C is a positive integer and , and select N channels of data from the data set as training data; The training data is fed into a degradation model of a multi-label fluorescence lifetime spectrum reconstruction network, wherein the degradation model includes a fluorescence decay model, a fuzzy model, and a noise model. Wherein, the fluorescence decay model is expressed as: in, Representation Space The sum of the fluorescence signal intensities of all different fluorescence lifetimes at coordinate t and time t, Representation Space The intensity of the fluorescence signal at the coordinate and with a fluorescence lifetime of τ, represents an acquisition parameter, which represents the moment when the fluorescence signal is acquired and is set as an optimizable parameter; and Wherein, the fuzzy model is expressed as: in, represents the blur kernel, represents the spatial convolution operator, represents the fuzzy function, represents the fluorescent signal after blurring; and The noise model includes shot noise, dark noise, and Gaussian noise. Shot noise is expressed as: Among them, in pixels The intensity of the fluorescent signal after blurring Probability Following a Poisson distribution, It is expressed as the number of electrons expected to be collected; Dark noise is expressed as: where the expected number of dark current electrons in each pixel is ; Gaussian noise is expressed as: in, is the variance of the noise, with a mean of 0; It means that given a mean of 0 and a variance of Under the condition of , the probability density function of the random variable x; After training the degradation model using the training data, obtaining a measurement value corresponding to the fluorescence signal in the imaging device; The acquisition parameter t in the fluorescence decay model is optimized to construct a lifetime spectrum reconstruction neural network to reconstruct the fluorescence lifetime spectrum from the measured values ​​and obtain the optimal acquisition parameter ; The optimal acquisition parameters Loaded into the MCU to obtain the two control signals.

12. A method for imaging multi-label fluorescence lifetime spectroscopy, comprising: A microcontroller MCU outputs two control signals, a first control signal of the two control signals being used to control the on and off of an excitation light source, and a second control signal of the two control signals being used to control an image sensor to start collecting a fluorescent signal emitted by a fluorescent sample excited by the excitation light source; under the control of the first control signal, the excitation light source is turned on at a predetermined time point, and is turned off when the excitation light intensity reaches a predetermined value; as well as Under the control of the second control signal, the image sensor collects the fluorescence signal at a specific time point, wherein the specific time point is selected from a time period when the excitation light decays to 0 but the fluorescence signal intensity is not 0, so that the sensor can directly collect the fluorescence signal without the need for a filter to filter the excitation light; The fluorescence signal collected by the image sensor is reconstructed by a multi-label fluorescence lifetime spectrum reconstruction network to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image.

13. The imaging method according to claim 12, wherein the multi-label fluorescence lifetime spectrum reconstruction network reconstructs the fluorescence signal collected by the image sensor to obtain a fluorescence lifetime spectrum and a multi-channel color fluorescence image, comprising: Using a multi-label fluorescence lifetime spectrum reconstruction network, multiple frames of grayscale images are obtained from the fluorescence signals collected by the image sensor, and the intensity information difference between the multiple frames of grayscale images is used to reconstruct the shape of The fluorescence lifetime spectrum is obtained by digitally filtering the fluorescence lifetime spectrum to obtain a multi-channel color fluorescence image with a shape of N*H*W, where a is the magnification factor ( ), N is the number of channels of the multi-channel color fluorescence image, H and W are the height and width of the multi-channel color fluorescence image, and each of the N channels represents a fluorescence lifetime. For a coordinate in the multi-channel color fluorescence image space, The shape of the fluorescence lifetime spectrum is .

14. The imaging method according to claim 12, wherein: The two control signals output by the MCU are obtained by a multi-label fluorescence lifetime spectrum reconstruction network.

15. The imaging method according to claim 14, wherein: Obtaining the two control signals by the multi-label fluorescence lifetime spectrum reconstruction network includes: Determining a lifetime distribution of fluorescence lifetimes represented by N channels that need to be reconstructed, where the number of fluorescence lifetimes is N and N is a positive integer; Select a dataset containing C channels where C is a positive integer and , and select N channels of data from the data set as training data; Select a dataset containing C channels where C is a positive integer and , and select N channels of data from the data set as test data; The ratio of training data to test data is 8:2; The training data is fed into a degradation model of a multi-label fluorescence lifetime spectrum reconstruction network, wherein the degradation model includes a fluorescence decay model, a fuzzy model, and a noise model. Wherein, the fluorescence decay model is expressed as: in, Representation Space The sum of the fluorescence signal intensities of all different fluorescence lifetimes at coordinate t and time t, Representation Space The intensity of the fluorescence signal at the coordinate and with a fluorescence lifetime of τ, represents an acquisition parameter, which represents the moment when the fluorescence signal is acquired and is set as an optimizable parameter; and Wherein, the fuzzy model is expressed as: in, represents the blur kernel, represents the spatial convolution operator, represents the fuzzy function, represents the fluorescent signal after blurring; and The noise model includes shot noise, dark noise, and Gaussian noise. Shot noise is expressed as: Among them, in pixels The intensity of the fluorescent signal after blurring Probability Following a Poisson distribution, It is expressed as the number of electrons expected to be collected; Dark noise is expressed as: where the expected number of dark current electrons in each pixel is Gaussian noise is expressed as: in, is the variance of the noise, with a mean of 0; It means that given a mean of 0 and a variance of Under the condition of , the probability density function of the random variable x; After training the degradation model using the training data, obtaining a measurement value corresponding to the fluorescence signal in the imaging device; The acquisition parameter t in the fluorescence decay model is optimized to construct a lifetime spectrum reconstruction neural network to reconstruct the fluorescence lifetime spectrum from the measured values ​​and obtain the optimal acquisition parameter in the current training stage. ; Repeat the above steps using the test data set, but do not perform optimization operations. When the model loss value of the degradation model no longer decreases, stop network training and obtain the optimal acquisition parameters. ; The optimal acquisition parameters Loaded into the MCU to obtain the two control signals.

Images