TCSPC data deconvolution method and system based on lightweight neural network

Abstract

The application discloses a TCSPC data deconvolution method and system based on a lightweight neural network, and relates to the technical field of photonics and artificial intelligence. The original fluorescence decay curve and the instrument response function of a sample are obtained and preprocessed; a conditional vector is generated from the preprocessed instrument response function by using an encoding network; a multi-scale time sequence feature of the preprocessed fluorescence decay curve is extracted by using a lightweight multi-scale time sequence feature extraction network; a time sequence relationship of the extracted multi-scale time sequence feature is modeled by using a lightweight gated recurrent unit; feature fusion is performed on the conditional vector and the modeled time sequence feature by using a feature layer modulation technology; and result prediction is performed on the conditionally fused features by using a multi-task output head. The application has the advantages of strong real-time performance, high precision, edge deployment and the like, and provides a possibility for the wide application of fluorescence lifetime detection technology in the fields of clinical field diagnosis and rapid drug screening.

Description

Technical Field

[0001] This invention belongs to the field of photonics and artificial intelligence technology, specifically relating to a TCSPC data deconvolution method and system based on a lightweight neural network. Background Technology

[0002] Time-resolved spectroscopy, a type of high-resolution spectroscopy, is widely used to explore transient processes arising from light-matter interactions due to its ultra-high temporal resolution derived from ultrashort pulses. Examples include determining excited-state lifetimes and studying relaxation processes of atoms, molecules, and ions in gas, liquid, and solid phases. Fluorescence lifetime, the average time it takes for a fluorescent molecule to return from an excited state to its ground state after excitation, is an intrinsic physical parameter characterizing the local microenvironment of a molecule (such as ion concentration, pH, temperature, and viscosity). Fluorescence lifetime-based detection techniques, particularly fluorescence lifetime imaging microscopy (FLIM), have shown great potential in cutting-edge fields such as clinical diagnosis (e.g., tumor boundary identification), drug screening, and cellular metabolic state monitoring due to their ultra-high sensitivity to environmental changes and robustness largely unaffected by fluctuations in fluorescent probe concentration and excitation light intensity. The core technological challenge in achieving fluorescence lifetime detection lies in how to rapidly and accurately extract lifetime parameters from the acquired fluorescence decay data. Traditional fluorescence decay curve analysis primarily relies on time-correlated single-photon counting (TCSPC) techniques, employing iterative algorithms (such as the Levenberg-Marquardt algorithm) and nonlinear least squares fitting (NLSF) to fit the decay curve of each pixel, thereby deconvolving to derive the fluorescence lifetime value. However, these iterative algorithms are computationally intensive, often requiring tens of seconds or even minutes to process a single 256×256 pixel image, severely hindering the widespread adoption of fluorescence lifetime technology in applications requiring real-time feedback, such as fluorescence-guided surgery (FSS). Furthermore, under weak signal conditions with low photon counts, iterative fitting algorithms are prone to getting trapped in local optima, leading to significant deviations in lifetime estimation results and failing to meet the demands of high-sensitivity on-site detection.

[0003] In recent years, with the development of deep learning technology, using convolutional neural networks (CNNs) for fluorescence lifetime analysis has become a new research direction. By training offline on a large amount of simulation data, neural networks can learn the complex nonlinear mapping relationship from the original photon count histogram to lifetime parameters, thereby achieving fast computation in the inference stage. However, most existing deep learning-based methods adopt standard two-dimensional (2D) or three-dimensional (3D) CNN architectures (such as Convolutional Networks for Biomedical Image Segmentation, U-Net). These models have a large number of parameters and are computationally intensive, usually requiring high-performance graphics processing units (GPUs), making them difficult to deploy in resource-constrained portable or handheld devices.

[0004] Meanwhile, advancements in hardware technology have driven the miniaturization and cost reduction of fluorescence detection devices. Solid-state detection systems, centered on silicon photomultiplier (SiPM) arrays and on-chip time-to-digital converters (TDCs), have laid the foundation for the development of portable field detection devices due to their advantages of high gain, high temporal resolution, and low power consumption. However, the core bottleneck of these portable devices lies in their limited edge computing capabilities, which cannot support complex traditional algorithms or large neural network models, resulting in a situation where "hardware comes first, but algorithms lag behind." How to design a lightweight intelligent algorithm that deeply collaborates with SiPM and on-chip TDC hardware systems and can achieve real-time, high-precision deconvolution on their limited computing resources (such as field-programmable gate arrays, FPGAs) has become a key technical challenge in bringing advanced fluorescence lifetime detection technology from the laboratory to rapid field detection.

[0005] In other words, existing fluorescence lifetime detection technologies have the following problems: 1) Insufficient real-time performance: Traditional iterative fitting algorithms involve huge amounts of computation, which cannot meet the needs of portable devices for real-time on-site detection; 2) Deployment difficulties: Existing deep learning-based algorithm models are too large and computationally complex, making them difficult to deploy on low-power, low-cost portable hardware platforms with FPGAs or embedded processors as the core.

[0006] 3) Hardware and software disconnect: The algorithm design and the front-end hardware system lack deep collaboration, and the hardware characteristics are not fully utilized to optimize the data processing flow, resulting in a bottleneck in the overall system performance. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides a TCSPC data deconvolution method and system based on a lightweight neural network.

[0008] The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a real-time fluorescence lifetime deconvolution method based on a lightweight neural network, comprising: Obtain the original fluorescence decay curve and instrument response function of the sample; The original fluorescence decay curve and the instrument response function are preprocessed to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function. A condition vector is generated using an encoding network based on the preprocessed instrument response function; A lightweight multi-scale temporal feature extraction network based on one-dimensional convolution is used to extract the multi-scale temporal features of the preprocessed fluorescence decay curve to obtain multi-scale temporal features. A lightweight gated loop unit is used to model the temporal relationship of the multi-scale temporal features to obtain the temporal features; By using feature layer modulation technology, feature fusion is performed on the conditional vector and the temporal features to obtain the conditionally fused features; Using a multi-task output head, predictions are made based on the features fused under the conditions to obtain the prediction results for the sample. The prediction results include: the number of components, the fluorescence lifetime value of each component, and the relative amplitude corresponding to each fluorescence lifetime value.

[0009] The present invention also provides a portable fluorescence lifetime detection system, comprising: a data processing unit, wherein the data processing unit is used to implement the steps of the above-described real-time deconvolution method for fluorescence lifetime based on a lightweight neural network.

[0010] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) Achieved true real-time detection: This invention greatly shortens the single-pixel processing time by replacing computationally intensive iterative fitting with forward inference of an efficient and lightweight neural network model, thus meeting the stringent real-time requirements such as rapid on-site detection.

[0011] 2) Edge deployment on portable devices is achieved: The neural network model architecture proposed in this invention greatly reduces the number of model parameters and computational load through the design of a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution and a lightweight gated recurrent unit. Combined with hardware-aware fixed-point quantization technology, the entire algorithm model can be deployed on low-cost FPGAs or embedded SoCs with extremely low resource consumption and power consumption, making it possible to realize handheld and portable fluorescence detection devices.

[0012] 3) Improved detection accuracy: The neural network model proposed in this invention can learn complex decay patterns from data autonomously through end-to-end training. Under the harsh signal-to-noise ratio conditions of low photon number, its lifetime estimation accuracy and robustness are significantly better than traditional fitting algorithms, effectively reducing phototoxicity to biological samples.

[0013] 4) It can fully utilize hardware characteristics to optimize the data processing flow: The algorithm proposed in this invention can be deployed in FPGA and implemented efficiently using the hardware accelerator in FPGA, thereby further improving detection efficiency, optimizing the data processing flow, and thus improving the detection performance of the fluorescence lifetime detection system.

[0014] 5) Reduced system cost and complexity: The invention eliminates the reliance on expensive external computers or GPUs through the lightweight design of the algorithm, which greatly reduces the cost of the entire high-performance fluorescence lifetime detection system and promotes its widespread application in primary healthcare and on-site rapid detection scenarios.

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a real-time fluorescence lifetime deconvolution method based on a lightweight neural network provided in an embodiment of the present invention. Figure 2 This is another flowchart illustrating the real-time deconvolution method for fluorescence lifetime based on lightweight neural networks provided in this embodiment of the invention. Detailed Implementation

[0017] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0018] Figure 1 This is a flowchart illustrating a real-time fluorescence lifetime deconvolution method based on a lightweight neural network, as provided in an embodiment of the present invention. Figure 1 As shown, the method includes: S101. Obtain the original fluorescence decay curve and instrumental response function (IRF) of the sample.

[0019] Here, the original fluorescence decay curve of the sample. This was measured using the TCSPC method and is a curve showing the decay process of the sample's fluorescence intensity over time. Instrument response function. Representing the response of the measurement system, it is a key calibration parameter in the measurement system, mainly used to eliminate the influence of the measurement system's own characteristics on the detection results, and It is also obtained through measurement.

[0020] S102, Original fluorescence decay curve and instrument response function Preprocessing was performed to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function.

[0021] S103. Using the coding network, generate a condition vector based on the preprocessed instrument response function.

[0022] S104. Using a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution, multi-scale temporal features of the preprocessed fluorescence decay curve are extracted to obtain multi-scale temporal features.

[0023] Here, a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution can efficiently extract deep features of decay curves at different time scales. Furthermore, one-dimensional convolution can reduce computational complexity and the number of parameters.

[0024] S105. Using lightweight gated loop units, time series relationships are modeled for multi-scale time series features to obtain time series features.

[0025] Here, by using lightweight gated loop units, long-term time-dependent relationships in the fluorescence decay process can be captured with low computational overhead.

[0026] S106. Using Feature-wise Linear Modulation (FiLM) technology, feature fusion is performed on the conditional vector and temporal features to obtain the conditionally fused features.

[0027] Here, by using the FiLM technique to fuse conditional vectors and temporal features, the network can adapt to the changes in response functions of different instruments, thereby achieving deconvolution of the IRF.

[0028] S107. Using the multi-task output head, prediction is performed based on the features after conditional fusion to obtain the prediction results of the sample. The prediction results include: number of components, fluorescence lifetime value of each component, and relative amplitude corresponding to each fluorescence lifetime value.

[0029] Specifically, the component classification indicates the number of different components contained in the sample, suggesting whether the sample is single-component, dual-component, or multi-component. Different components have different fluorescence lifetime values; therefore, the prediction result includes at least one fluorescence lifetime value. The relative amplitude corresponding to the fluorescence lifetime value refers to the proportion of fluorescence intensity contributed by the component with that fluorescence lifetime value to the total intensity in the fluorescence decay model.

[0030] In this invention, the aforementioned encoding network, the lightweight multi-scale temporal feature extraction network based on one-dimensional convolution, the lightweight gated recurrent unit, the multi-task output head, and the fusion module for feature fusion using feature layer modulation technology constitute a lightweight fluorescence lifetime real-time deconvolution model (i.e., the neural network model designed in this invention, hereinafter referred to as the neural network model). Steps S101 to S107 are implemented by the trained neural network model. The construction of the training dataset, the composition of the loss function, and the training process of this neural network model will be explained in subsequent sections.

[0031] In some embodiments, the above-mentioned S102 can be implemented by steps S1021 to S1024: S1021, Original fluorescence decay curve and instrument response function Background noise was subtracted separately to obtain the denoised fluorescence attenuation curve and the denoised instrument response function.

[0032] Here, the original fluorescence decay curve can be estimated and subtracted by the constant background noise introduced by dark counting or ambient light. and instrument response function Background noise reduction is based on existing technology, so the specific principles are not described in detail here.

[0033] S1022. The peak values ​​of the denoised fluorescence decay curve and the denoised instrument response function are normalized to obtain the normalized fluorescence decay curve and the normalized instrument response function.

[0034] Here, the peak intensity of the denoised fluorescence decay curve and the denoised instrument response function can be normalized to the [0, 1] interval to eliminate the effects of excitation light intensity fluctuations and detector gain changes.

[0035] S1023. Align the normalized fluorescence decay curve and the normalized instrument response function with the time axis to obtain aligned fluorescence decay curve and aligned instrument response function.

[0036] Here, time axis alignment between the normalized fluorescence decay curve and the peak position of the normalized instrument response function is achieved by aligning the two. Time axis alignment ensures that the two are precisely synchronized in time.

[0037] S1024. Resample the aligned fluorescence decay curve and the aligned instrument response function to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function.

[0038] Here, by resampling the data, high-resolution time channels (e.g., 1024 or 4096 time channels) can be resampled to a fixed, lower dimension, such as 256 time channels, through linear interpolation or local summation, which can significantly reduce the dimensionality of the data.

[0039] It should be noted that in some other embodiments, the above-described S102 can also be implemented by simply using the above-described steps S1021 to S1023.

[0040] The morphology of the IRF (e.g., pulse width, tail) has a convolutional effect on the fluorescence decay curve and may drift with device temperature and aging. To enable the main network of the neural network model designed in this invention to adapt to such changes, this invention employs an encoding network to encode the preprocessed instrument response function, generating a low-dimensional conditional vector. This vector is then used as a "fingerprint" of the IRF pattern to guide the deconvolution process of the main network. Based on this, S103 is specifically implemented as follows: The coefficients in the preprocessed instrument response function are extracted using an encoding network, and these coefficients are encoded into a vector to obtain the conditional vector. It should be noted that the condition vector It is a 1×16 dimensional vector.

[0041] Unlike traditional CNNs that use standard two-dimensional convolutions, this invention employs entirely one-dimensional convolutions because fluorescence decay is essentially a one-dimensional time series problem. Specifically, the lightweight multi-scale temporal feature extraction network based on one-dimensional convolutions includes: M parallel depthwise separable temporal convolutional branches and M corresponding Efficient Channel Attention (ECA) modules; the convolutional kernels in the M parallel depthwise separable temporal convolutional branches have different sizes; the output of each depthwise separable temporal convolutional branch is connected to the input of an Efficient Channel Attention module, and each depthwise separable temporal convolutional branch includes a depthwise convolution and a pointwise convolution, where M is a positive integer greater than 2. Here, the depthwise convolution independently filters each input channel to capture temporal features; while the pointwise convolution handles the linear combination between channels. This decomposition reduces the computational cost and parameter count by an order of magnitude with almost no loss of accuracy. By introducing a non-dimensionality-reduction channel attention module at the output of each branch, the importance of different feature channels can be adaptively learned, and key channels (e.g., fast-decaying feature channels associated with short-lived components) can be weighted to suppress noisy or redundant feature channels.

[0042] For example, a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution includes three parallel depthwise separable temporal convolutional branches and three efficient channel attention modules. In the first depthwise separable temporal convolutional branch, both the depthwise convolution and pointwise convolution kernels are 3×3; in the second branch, they are 7×7; and in the third, they are 15×15. Here, small kernels focus on local details such as the rapid descent edge of the decay curve, while large kernels capture global decay trends such as long tails. This structure allows the network to analyze features at different time scales simultaneously.

[0043] While the standard Gate Recurrent Unit (GRU) can model temporal relationships well, its gating structure is computationally complex. Therefore, this invention improves the GRU into a lightweight GRU (LGRU). The difference between LGRU and the standard GRU is that LGRU can group the one-dimensional feature vector according to the channel dimension of the input one-dimensional feature vector, so that different groups of features can share a set of gating parameters, thereby reducing the amount of computation.

[0044] Based on this, the above S105 can be achieved through steps S1051~S1052: S1051. Flatten the multi-scale temporal features into a one-dimensional feature vector and input it into the LGRU.

[0045] S1052 and LGRU group the one-dimensional feature vector according to the input channel to obtain G group features. The features of all groups are calculated by GRU using the same set of gating parameters. The final calculation results of the G group features are fused to obtain the time series features, where G is a positive integer greater than 1.

[0046] It should be noted that GRU computation refers to standard GRU computation; that is, at each time step, LGRU independently performs a complete and standard GRU gating computation for the features of each group, but the gating parameters for features of different groups are different, and features of all groups share the same set of gating parameters. By adding such a grouping mechanism, the number of GRU parameters and computational cost can be reduced to 1 / G of the original while maintaining the ability to model time series.

[0047] In some embodiments, the above-mentioned S106 can be implemented by steps S1061 to S1062: S1061, Utilizing a fully connected layer based on condition vectors Two modulation parameters are generated.

[0048] Here, the condition vector Inputting a fully connected layer yields a scaling factor. and bias factor .

[0049] S1062. Perform an affine transformation on the time-series features using two modulation parameters to obtain the conditionally fused features. .

[0050] For example, The expression is: ; in, This represents the features after conditional fusion. Representing temporal characteristics, , These represent the scaling factor and the bias factor, respectively.

[0051] This fusion method is equivalent to teaching the neural network model "if the IRF is like this (by...") (Definition), then I should adjust the attenuation features I extracted in this way, so as to dynamically and non-linearly "offset" the convolution effect of IRF.

[0052] In some embodiments, the multi-task output head of the neural network model proposed in this invention includes multiple parallel prediction heads for different tasks. Exemplarily, the multi-task output head includes a shared feature layer for unifying feature scales and multiple parallel prediction heads. The shared feature layer can be a one-dimensional convolution or a fully connected layer. The multiple parallel prediction heads include at least prediction heads for predicting the number of components, the fluorescence lifetime values ​​of the components, and the relative amplitude corresponding to each fluorescence lifetime value. In some embodiments, the multiple parallel prediction heads further include: a prediction head for predicting the confidence level of the number of components, a detection head for predicting the confidence level of the fluorescence lifetime values, and a prediction head for predicting the confidence level of the relative amplitude corresponding to the fluorescence lifetime values. The structure of each prediction head can be selected according to actual needs.

[0053] For example, a prediction head used to predict fluorescence lifetime values ​​can consist of a feature transformation layer, a linear layer without an activation function, and a ReLU activation function. The feature transformation layer (e.g., a fully connected layer or a 1-2 layer convolutional layer with a kernel size of 1×1) is used to adjust the number of feature channels to a fixed value. Linear layers are used for output. A value, for example, when the sample is a single component. =1, meaning it outputs a fluorescence lifetime value when the sample is a two-component sample. =2, meaning two fluorescence lifetime values ​​are output; the ReLU activation function ensures the output value is positive. For example, the prediction head for predicting the confidence level of a fluorescence lifetime value or the confidence level of the relative amplitude corresponding to the fluorescence lifetime value can be a linear layer plus a sigmoid activation function. For example, the prediction head for predicting the relative amplitude corresponding to the fluorescence lifetime value can consist of a feature transformation layer, a linear layer without an activation function, and a sigmoid or softmax activation function to constrain the output value within the (0, 1) interval, representing probability or proportion. For example, the structure of the prediction head for predicting the confidence level of the relative amplitude corresponding to the fluorescence lifetime value is the same as the structure of the prediction head for predicting the confidence level of the fluorescence lifetime value. For example, a prediction head used for classification tasks can be chosen as the prediction head for predicting the group number. For example, the structure of the prediction head for predicting the confidence level of the group number can be the same as the structure of the prediction head for predicting the confidence level of the fluorescence lifetime value, or other structures can be chosen.

[0054] The present invention also provides a portable fluorescence lifetime detection system, comprising: a data processing unit, which implements the steps of the real-time fluorescence lifetime deconvolution method based on a lightweight neural network proposed in this invention. The data processing unit may be a computing unit based on an FPGA or an embedded processor.

[0055] For example, the data processing unit can be an FPGA or a System-on-Chip (SoC) containing an FPGA, and the FPGA is used to implement the real-time fluorescence lifetime deconvolution method based on lightweight neural networks proposed in this invention. For instance, the FPGA includes a hardware accelerator for executing the real-time fluorescence lifetime deconvolution method based on lightweight neural networks proposed in this invention, and the hardware accelerator employs fixed-point quantization (such as INT8 or INT16) and a parallel pipeline architecture, thereby enabling low-power, high-throughput computation of the neural network model proposed in this invention. For example, Figure 2 This is a flowchart illustrating the process of a portable fluorescence lifetime detection system executing a real-time fluorescence lifetime deconvolution method based on a lightweight neural network. (Example) Figure 2 As shown, in the portable fluorescence lifetime detection system, an FPGA pre-deployed with the neural network model proposed in this application serves as a lightweight intelligent operator engine. The hardware sensing front-end of the portable fluorescence lifetime detection system obtains the sample's data through processes such as SiPM and TDC data acquisition and hardware histogram generation. and Afterwards, and The data is sent to the lightweight intelligent operator engine. The lightweight intelligent operator engine performs operations such as background subtraction, peak normalization, and resampling. and After preprocessing, the preprocessed material is first processed using an IRF coding network (i.e., the coding network mentioned above). Encoding yields a 1×16 dimensional condition vector. And a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution is used to extract preprocessed features. The multi-scale temporal features are then analyzed. Next, the lightweight intelligent operator engine utilizes the LGRU with grouped shared parameters to perform long-term temporal dependency modeling on the extracted multi-scale temporal features, obtaining the temporal features. Then, the lightweight intelligent operator engine employs FiLM technology and conditional vector... The temporal features are subjected to affine transformation to adaptively and dynamically cancel the IRF convolution effect, resulting in conditionally fused features. Finally, the conditionally fused features are input into a multi-task output head to predict one or more of the following tasks: fluorescence lifetime, relative amplitude, number of components, and confidence level. The prediction results can be applied to scenarios such as clinical emergency monitoring, ecological water quality early warning, or industrial process traceability. The response time of the entire execution process is less than or equal to 50ms, and the power consumption is less than 5W.

[0056] The following details the construction process of the training dataset, the composition of the loss function, and the training process of the neural network model designed in this invention. For example, when the prediction results of the neural network model designed in this invention include: fluorescence lifetime value, the relative amplitude corresponding to the fluorescence lifetime value, and the number of components, the specific principles of the training dataset construction process, the composition of the loss function, and the training process of the model are as follows.

[0057] Regarding the training dataset, since obtaining a large amount of real experimental data with precise fluorescence lifetime "ground truth" labels is extremely difficult, this invention employs a simulation method based on a physical model to generate a large-scale, diverse training dataset. This method can simulate the real TCSPC measurement process, generating training samples that are highly consistent with the distribution of experimental data. The training dataset contains multiple training samples, each consisting of a fluorescence decay curve F(t) and the corresponding instrument response function IRF(t). A training sample can be represented as {F(t), IRF(t)}. Specifically, the method for generating a single training sample is as follows: S10. Randomly generate a set of intrinsic physical parameters for fluorescence attenuation. Specifically, for a sample containing N fluorescent components, randomly generate a set of intrinsic physical parameters for fluorescence attenuation, where the physical parameters include: the number of components N, and N relative amplitudes {A1, A2, ..., A...} corresponding one-to-one with the N fluorescent components. N}, and N fluorescence lifetime values ​​{τ1, τ2, ..., τ} corresponding one-to-one with the N fluorescent components. N The value of N can be chosen randomly; for example, N can be 1, 2, or 3. N relative amplitudes {A1, A2, ..., A...} N} is normalized so that ΣA i = 1, where i is a positive integer, and the value of i ranges from 1 to N. Fluorescence lifetime values ​​{τ1, τ2, ..., τ...} N} represents N lifetime values ​​{τ1, τ2, ..., τ} randomly sampled from a pre-defined range that conforms to common fluorescent probe distributions. N}, for example, τ i ∈[0.1 ns, 20 ns], where ns represents nanoseconds.

[0058] S11. Generate an ideal multi-exponential fluorescence decay function D(t) that does not contain instrument effects based on the physical parameters in step S10, where D(t) = Σ i (A i / τ i )×exp(-t / τ i ), where t represents time.

[0059] S12. Simulate an instrument response function IRF(t) that conforms to the hardware characteristics of a portable device. Typically, a Gaussian function or a Gaussian-exponential convolution can be used for modeling. By randomly adjusting the peak position, full width at half maximum (FWHM), and tailing parameters of the IRF, IRFs of various shapes can be generated to improve the model's generalization ability and robustness to hardware drift.

[0060] S13. Based on the ideal multi-exponential fluorescence decay function D(t) and the simulated instrument response function IRF(t), generate a simulated and noise-free fluorescence decay curve F_clean(t), F_clean(t) = IRF(t)×D(t).

[0061] S14. Add Poisson noise to the simulated and noise-free fluorescence decay curve F_clean(t) generated in step S13 to obtain a fluorescence decay curve F(t). By adding Poisson noise, the statistical fluctuations in photon counts during the TCSPC measurement process can be simulated. By setting a total number of photons (e.g., randomly sampling between 50 and 10000), training data with different signal-to-noise ratio levels can be generated.

[0062] S15, the fluorescence decay curve F(t) obtained in step S14, and the instrument response function IRF(t) simulated in step S12 constitute a training sample {F(t), IRF(t)}, and the label of this training sample is {{τ1, τ2, ..., τ...}. N},{A1, A2,..., A N}, N}.

[0063] It should be noted that by executing the above steps S10 to S15 multiple times, for example hundreds of thousands of times, a training dataset for training the neural network model designed in this invention can be constructed.

[0064] The loss function of the neural network model designed in this invention comprises four parts: reconstruction loss L_recon, component loss L_comp, amplitude loss L_amp, and lifetime loss L_life. The weighted sum of these four losses constitutes the total loss of the neural network model designed in this invention. Therefore, the expression for the loss function L_total of the neural network model designed in this invention is as follows: L_total = w1×L_life + w2×L_amp + w3×L_comp + w4×L_recon; Here, w1, w2, w3, and w4 are the weights of the lifetime loss L_life, amplitude loss L_amp, component loss L_comp, and reconstruction loss L_recon, respectively. This composite loss function enables the neural network model to simultaneously and accurately learn multiple tasks, including fluorescence lifetime value, relative amplitude, and component number.

[0065] Reconstruction loss is a key physical model-based regularization term. It represents the mean-square error between the reconstructed fluorescence decay curve and the fluorescence decay curve in the training sample. The reconstructed fluorescence decay curve is obtained by convolving the prediction result of the neural network model based on the training sample with the instrument response function in the training sample. Specifically, for a training sample, the reconstruction loss is calculated as follows: the fluorescence lifetime value and relative amplitude predicted by the neural network model based on the input training sample are convolved with the instrument response function in the training sample to obtain a reconstructed fluorescence decay curve F'(t). The mean-square error (MSE) between the reconstructed fluorescence decay curve F'(t) and the fluorescence decay curve F(t) in the training sample is then calculated, yielding the reconstruction loss for that training sample. The reconstruction loss forces the neural network model's prediction results to conform to the physical laws of fluorescence decay, significantly improving the model's accuracy and robustness.

[0066] Group score loss is the cross-entropy between the predicted group score from the neural network model based on the training samples and the true group score in the labels of the training samples. Treating group score prediction as a classification problem, for a training sample, cross-entropy loss is used to measure the difference between the predicted group score N' and the true group score N in the labels of that training sample, thus obtaining the group score loss for that training sample.

[0067] Amplitude loss is the mean squared error between the relative amplitude predicted by the neural network model based on the training samples and the true relative amplitude in the labels of the training samples. Specifically, for a training sample, the MSE loss is used to measure the difference between the relative amplitude A' predicted by the neural network model and the true relative amplitude A in the labels of that training sample, thus obtaining the amplitude loss corresponding to that training sample.

[0068] In some embodiments, lifetime loss is the mean squared error between the fluorescence lifetime value predicted by the neural network model based on the training samples and the true fluorescence lifetime value in the labels of the training samples. Specifically, for a training sample, MSE loss or smoothed L1 loss can be used to measure the difference between the fluorescence lifetime τ' predicted by the neural network model and the true fluorescence lifetime τ in the labels of that training sample. It should be noted that for multi-component cases, the predicted relative amplitudes and the true relative amplitudes need to be sorted and matched first before calculating the loss. Similarly, for fluorescence lifetime, the predicted fluorescence lifetimes and the true fluorescence lifetimes also need to be sorted and matched first before calculating the loss.

[0069] After constructing the training dataset and loss function, the neural network model can be trained. It should be noted that the model training method used in this application is a common model training method in deep learning. For example, the training process is completed offline on a server equipped with a high-performance GPU, and the specific process is as follows: Step 20: Model Initialization. Initialize the weight parameters of the neural network model in this application.

[0070] Step 21, Data Loading. Load data from the generated training dataset in mini-batches.

[0071] Step 22, Forward Propagation. Input a batch of {F(t), IRF(t)} data into the initialized neural network model, and the model outputs the prediction results.

[0072] Step 23: Calculate the loss. Using the prediction results and the corresponding training sample labels, calculate the value of the loss function L_total.

[0073] Step 24: Backpropagation and Optimization. Based on the loss value, the gradient of the loss function with respect to each parameter of the neural network model is calculated using the backpropagation algorithm, and the weights of the neural network model are updated using optimizers such as Adam.

[0074] Step 25, Iteration: Repeat steps S21 to S24 until the neural network model converges on the independent validation set (e.g., the loss on multiple consecutive validation sets no longer decreases), thus obtaining the trained neural network model.

[0075] The present invention has the following beneficial effects: 1) Achieved true real-time detection: This invention significantly shortens the single-pixel processing time by replacing computationally intensive iterative fitting with forward inference from a highly efficient, lightweight neural network model, thus meeting stringent real-time requirements such as rapid on-site detection. By embedding this lightweight neural network model into an FPGA hardware accelerator, the single-pixel processing time can be reduced from seconds to sub-milliseconds, making the entire frame image processing time less than 50 ms.

[0076] 2) Edge deployment on portable devices is achieved: The neural network model architecture proposed in this invention, through the design of a lightweight multi-scale temporal feature extraction network based on one-dimensional convolution and a lightweight gated recurrent unit, greatly compresses the number of model parameters and computational load, making the number of model parameters less than 100 K. Combined with hardware-aware fixed-point quantization technology, the entire algorithm model can be deployed on low-cost FPGAs or embedded SoCs with extremely low resource consumption and power consumption, making it possible to realize handheld and portable fluorescence detection devices, with resource consumption less than 200 KB and power consumption less than 5W.

[0077] 3) Improved detection accuracy: The neural network model proposed in this invention can learn complex decay patterns from data autonomously through end-to-end training. Under the harsh signal-to-noise ratio conditions of low photon count (less than 50 photons / pixel), its lifetime estimation accuracy and robustness are significantly better than traditional fitting algorithms, effectively reducing phototoxicity to biological samples.

[0078] 4) It can fully utilize hardware characteristics to optimize the data processing flow: The algorithm proposed in this invention can be deployed in FPGA and implemented efficiently using the hardware accelerator in FPGA, thereby further improving detection efficiency, optimizing the data processing flow, and thus improving the detection performance of the fluorescence lifetime detection system.

[0079] 5) Reduced system cost and complexity: The invention eliminates the reliance on expensive external computers or GPUs through the lightweight design of the algorithm, which greatly reduces the cost of the entire high-performance fluorescence lifetime detection system and promotes its widespread application in primary healthcare and on-site rapid detection scenarios.

[0080] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0081] In this specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple instances. While different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0082] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A real-time fluorescence lifetime deconvolution method based on lightweight neural networks, characterized in that, include: Obtain the original fluorescence decay curve and instrument response function of the sample; The original fluorescence decay curve and the instrument response function are preprocessed to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function. A condition vector is generated using an encoding network based on the preprocessed instrument response function; A lightweight multi-scale temporal feature extraction network based on one-dimensional convolution is used to extract the multi-scale temporal features of the preprocessed fluorescence decay curve to obtain multi-scale temporal features. A lightweight gated loop unit is used to model the temporal relationship of the multi-scale temporal features to obtain the temporal features; By using feature layer modulation technology, feature fusion is performed on the conditional vector and the temporal features to obtain the conditionally fused features; Using a multi-task output head, predictions are made based on the features fused under the conditions to obtain the prediction results for the sample. The prediction results include: the number of components, the fluorescence lifetime value of each component, and the relative amplitude corresponding to each fluorescence lifetime value.

2. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The preprocessing of the original fluorescence decay curve and the instrument response function to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function includes: Background noise is subtracted from the original fluorescence decay curve and the instrument response function to obtain the denoised fluorescence decay curve and the denoised instrument response function. Peak normalization is performed on the denoised fluorescence decay curve and the denoised instrument response function to obtain the normalized fluorescence decay curve and the normalized instrument response function. The normalized fluorescence decay curve and the normalized instrument response function are aligned on the time axis to obtain aligned fluorescence decay curve and aligned instrument response function. The aligned fluorescence decay curve and the aligned instrument response function are resampled to obtain the preprocessed fluorescence decay curve and the preprocessed instrument response function.

3. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The step of generating a condition vector using an encoding network based on the preprocessed instrument response function includes: The coefficients in the preprocessed instrument response function are extracted using the coding network, and the coefficients are encoded into a vector to obtain the condition vector.

4. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The lightweight multi-scale temporal feature extraction network based on one-dimensional convolution includes: M parallel depthwise separable temporal convolutional branches and M corresponding efficient channel attention modules; the convolutional kernels of the convolutional layers in the M parallel depthwise separable temporal convolutional branches have different sizes; the output of each depthwise separable temporal convolutional branch is connected to the input of an efficient channel attention module, and each depthwise separable temporal convolutional branch includes a depthwise convolution and a pointwise convolution, where M is a positive integer greater than 2.

5. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 4, characterized in that, When M is 3, the kernel size of both the depthwise convolution and the pointwise convolution in the first depthwise separable temporal convolution branch is 3×3, the kernel size of both the depthwise convolution and the pointwise convolution in the second depthwise separable temporal convolution branch is 7×7, and the kernel size of both the depthwise convolution and the pointwise convolution in the third depthwise separable temporal convolution branch is 15×15.

6. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The process of using lightweight gated loop units to model the temporal relationships of the multi-scale temporal features yields temporal features, including: The multi-scale temporal features are flattened into a one-dimensional feature vector and input into the lightweight gated loop unit; The lightweight gated loop unit groups the one-dimensional feature vector according to the input channel to obtain G group features. All group features are calculated using the same set of gating parameters using GRU. The final calculation results of the G group features are fused to obtain the time series feature, where G is a positive integer greater than 1.

7. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The step of using feature layer modulation technology to fuse the conditional vector and the temporal features to obtain the conditionally fused features includes: Two modulation parameters are generated based on the condition vector using a fully connected layer; The temporal features are subjected to an affine transformation using the two modulation parameters to obtain the conditionally fused features.

8. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 7, characterized in that, The two modulation parameters include: scaling factor and bias factor The expression for the feature after conditional fusion is: ； in, This represents the features resulting from the fusion of the conditions. This represents the temporal characteristics.

9. The real-time fluorescence lifetime deconvolution method based on lightweight neural networks according to claim 1, characterized in that, The encoding network, the lightweight multi-scale temporal feature extraction network based on one-dimensional convolution, the lightweight gated recurrent unit, the fusion module for feature fusion using feature layer modulation technology, and the multi-task output head constitute a lightweight fluorescence lifetime real-time deconvolution model. The training dataset of the lightweight fluorescence lifetime real-time deconvolution model includes multiple training samples. Each training sample consists of a fluorescence decay curve and an instrument response function. Furthermore, the labels of each training sample include: the number of true components, the true fluorescence lifetime value of each component, and the true relative amplitude corresponding to each fluorescence lifetime value. The loss function of the lightweight fluorescence lifetime real-time deconvolution model is a composite loss function, which includes reconstruction loss, component loss, amplitude loss and lifetime loss. The reconstruction loss is the mean square error between the reconstructed fluorescence decay curve and the fluorescence decay curve in the training sample. The reconstructed fluorescence decay curve is obtained by convolving the prediction result output by the lightweight fluorescence lifetime real-time deconvolution model with the instrument response function in the training sample. The group fraction loss is the cross-entropy between the group fraction in the prediction result output by the lightweight fluorescence lifetime real-time deconvolution model based on the training samples and the true group fraction in the labels of the training samples. The amplitude loss is the mean square error between the relative amplitude in the prediction result output by the lightweight fluorescence lifetime real-time deconvolution model based on the training samples and the true relative amplitude in the labels of the training samples. The lifetime loss is the mean squared error between the fluorescence lifetime value in the predicted result output by the lightweight fluorescence lifetime real-time deconvolution model based on the training samples and the true fluorescence lifetime value in the labels of the training samples.

10. A portable fluorescence lifetime detection system, characterized in that, include: A data processing unit, wherein the data processing unit is used to implement the steps of the method according to any one of claims 1 to 9.

Images