A method and system for unmixing multicolor fluorescence spectra
By combining a linear spectral mixing model and sparse regularization, and employing the alternating direction multiplier method and a multilayer three-dimensional convolutional neural network, the spectral unmixing problem in a multicolor fluorescence imaging system was solved, achieving efficient and interpretable spectral unmixing results.
Patent Information
- Application Number
- CN202211694486.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-28
- Publication Date
- 2025-10-28
- Estimated Expiration
- 2042-12-28
AI Technical Summary
Existing multicolor fluorescence imaging systems suffer from spectral crosstalk during spectral unmixing. Traditional sparse regression methods lead to modeling errors and ill-conditioned inverse problems. While deep learning methods improve unmixing performance, they require significant computational resources and lack interpretability.
By combining a linear spectral mixing model and sparse regularization, an objective function is constructed using the alternating direction multiplier method. A hierarchical neural network with non-shared parameters is established, and spectral unmixing is performed through a multi-layer three-dimensional convolutional neural network. An end-to-end neural network mapping is constructed using sparse regularization and total variation regularization constraints.
With fewer iterations, better spectral unmixing was achieved, artifacts were reduced, image smoothness and detail were improved, and the interpretability of the neural network was enhanced.
Smart Images
Figure CN115829007B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multicolor fluorescence imaging, and in particular to a method and system for demixing multicolor fluorescence spectra. Background Art
[0002] Optical microscopy, with its high resolution at the micrometer level, can non-invasively detect dynamic processes at the cellular level within living tissues, offering advantages such as being non-destructive, non-ionizing, highly sensitive, and highly specific. In studying simultaneous imaging of the whole-brain input-output loop, it is necessary to monitor the distribution and interactions of input and output neurons and building blocks simultaneously. Multicolor fluorescence imaging is an important tool for achieving this research goal. Multicolor fluorescence imaging systems require multiple filters in the probe optical path to detect fluorescence signals at different wavelengths. However, when there are many types of fluorescent probes, emission spectral overlap is prone to occur, which cannot be removed by optical means, resulting in spectral crosstalk, necessitating spectral demixing. Traditional sparse regression methods for spectral unmixing are based on the establishment of linear spectral mixture models. These models are characterized by clear physical meaning and strong interpretability. However, due to the complexity of biological tissue structures and the instability of fluorescent probe expression, forward modeling based on linear spectral mixture models can lead to modeling errors. Furthermore, the high correlation between the emission spectra of different fluorescent probes makes the inverse problem somewhat pathological. When there is some noise, it results in many artifacts in the unmixed image and the image is not smooth enough. Therefore, following the traditional image restoration research approach makes it difficult to achieve substantial breakthroughs in the current spectral unmixing effect.
[0003] In recent years, deep learning methods have been gradually introduced into the field of biomedical imaging, such as CT and MRI, and have made significant progress, providing new ideas for the problem of spectral unmixing.
[0004] Deep learning can perform spectral unmixing by establishing end-to-end neural networks, which can abandon the traditional paradigm of forward modeling and inverse solving and directly establish the mapping relationship between crosstalk data distribution and the distribution of each fluorescent probe. Therefore, it completely avoids the errors of forward modeling and the ill-conditioned nature of inverse solving, and is expected to improve the spectral unmixing effect.
[0005] However, since the original mathematical method paradigm has been abandoned, network architecture design is often based on existing network structures. These networks suffer from the black box effect, the learning parameters are not interpretable, and the generalization ability of the network can only be improved by increasing the training set and increasing the network depth. This will lead to a significant increase in the computing resources required for training. At the same time, how to obtain a large dataset is also a problem that cannot be ignored.
[0006] Therefore, how to improve the unmixing effect while using deep learning methods, and at the same time introduce the corresponding physical principles to improve the interpretability of neural networks so as to reduce the required training samples and network depth, has become an urgent problem to be solved. Summary of the Invention
[0007] The purpose of this invention is to provide a method and system for demixing multicolor fluorescence spectra, which can improve the demixing effect of multicolor fluorescence spectra.
[0008] To achieve the above objectives, the present invention provides the following solution:
[0009] A method for unmixing multicolor fluorescence spectra, comprising:
[0010] Based on the linear spectral mixing model, an objective function is constructed according to the detected crosstalk signal distribution, the spectral matrix determined by the optical system parameters and the emission spectra of each fluorescent probe, and the corresponding compensation matrix, combined with sparse regularization and total variation regularization constraints.
[0011] Based on the objective function, the Alternating Direction Method of Multipliers (ADMM) is used to determine the computation graph; the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping;
[0012] The compensation matrix and inverse matrix in the computation graph are parameterized based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and the end-to-end neural network is constrained by a loss function.
[0013] A multicolor fluorescence crosstalk simulation dataset is constructed; and an end-to-end neural network is trained and tested using the multicolor fluorescence crosstalk simulation dataset; the multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
[0014] Optionally, the objective function is:
[0015]
[0016] Where Ω is the objective function to be minimized in spectral unmixing. To minimize the objective function Ω in spectral unmixing, the optimization variable is x, A is the spectral matrix, ΔA is the compensation matrix for the spectral matrix A, x is the true distribution of each fluorescent probe in the target organism, Y is the distribution of the detected crosstalk signal, and λ, λ TV Here, |x|1 is the regularization coefficient, |x|2 is the l1 norm of x, ensuring the sparsity of the solution, and TV(x) is the total variation of the anisotropy of the true distribution x of each fluorescent probe. H ix is the discrete gradient extracted from image x at pixel i, and H is the discrete difference operator.
[0017] Optionally, the computational graph is determined using the alternating direction multiplier method based on the objective function; the computational graph adopts a hierarchical network structure with non-shared parameters, establishing an end-to-end neural network mapping, specifically including:
[0018] Based on the objective function, the computational graph is determined using the alternating direction multiplier method.
[0019] Periodic boundary conditions are applied to the computation graph, and the inverse of the Hessian matrix is diagonalized using a two-dimensional discrete Fourier transform.
[0020] The computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping.
[0021] Optionally, the parameterization of the compensation matrix and inverse matrix in the computation graph based on multiple multi-layer three-dimensional convolutional neural network structures yields an end-to-end neural network; and the end-to-end neural network is constrained by a loss function, specifically including:
[0022] The residual block structure based on multi-layer three-dimensional convolutional neural networks parameterizes the compensation transformation in each layer of the network.
[0023] The skip connection structure based on two multi-layer three-dimensional convolutional neural networks parameterizes the inverse transformations and diagonal matrices in each layer of the network.
[0024] Optionally, the loss function is:
[0025]
[0026]
[0027] loss = loss1 + β × loss2
[0028] Where loss1 is the distribution of each fluorescent probe after unmixing, U N The mean square error between the true distribution x of each fluorescent probe and the mean square error between the two probes, loss2 is the inverse constraint loss function used to constrain the inverse relationship between the forward and inverse transformation networks, β is the regularization parameter, m is the number of detection channels, n is the number of fluorescent probe types, and n is the mean square error between the two probes. 2 The image size to be detected is given by M, the total number of samples in the training set is given by N, and r is given by r. k The input to the skip connection network is the intermediate variable of the k-th layer network. These are inverse transformations.
[0029] A multicolor fluorescence spectral unmixing system, comprising:
[0030] The objective function construction module is used to construct an objective function based on a linear spectral mixing model, according to the distribution of detected crosstalk signals, the spectral matrix determined by the emission spectra of each fluorescent probe and the optical system parameters, and the corresponding compensation matrix, combined with sparse regularization and total variation regularization constraints.
[0031] The network mapping establishment module is used to determine the computation graph based on the objective function using the alternating direction multiplier method; the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping.
[0032] The neural network determination module is used to parameterize the compensation matrix and inverse matrix in the computation graph based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and to constrain the end-to-end neural network through a loss function.
[0033] The network training and testing module is used to construct a multicolor fluorescence crosstalk simulation dataset; the multicolor fluorescence crosstalk simulation dataset is used to train and test the end-to-end neural network; the multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
[0034] Optionally, the network mapping establishment module specifically includes:
[0035] The computational graph determination unit is used to determine the computational graph based on the objective function using the alternating direction multiplier method.
[0036] The matrix transformation unit is used to apply periodic boundary conditions to the computation graph and diagonalize the inverse of the Hessian matrix through a two-dimensional discrete Fourier transform.
[0037] The network mapping establishment unit is used to establish an end-to-end neural network mapping by adopting a hierarchical network structure with non-shared parameters in the computation graph.
[0038] Optionally, the neural network determination module specifically includes:
[0039] The first parameterization unit is used to parameterize the compensation transformation in each layer of the network based on the residual block structure of a multi-layer three-dimensional convolutional neural network.
[0040] The second parameterization unit is used to parameterize the inverse transformations and diagonal matrices in each layer of the network based on the skip connection structure of two multi-layer three-dimensional convolutional neural networks.
[0041] A multicolor fluorescence spectral unmixing system includes: at least one processor, at least one memory, and computer program instructions stored in the memory, which implement the multicolor fluorescence spectral unmixing method when executed by the processor.
[0042] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:
[0043] This invention provides a multicolor fluorescence spectral unmixing method and system. Based on a linear spectral mixing model, it obtains the objective function for spectral unmixing by combining optimizations in the spectral and spatial domains and compensating the spectral matrix. The objective function is then expanded into a computational graph using the ADMM algorithm. An end-to-end neural network mapping is obtained by employing a hierarchical network structure with non-shared parameters on the computational graph. Multiple multi-layer three-dimensional convolutional neural network structures are used to parameterize the compensation and inverse matrices, and a loss function is used for constraint. Finally, a simulation dataset is established to complete the training and testing of the end-to-end neural network. Compared with direct spectral unmixing using the ADMM algorithm, this method achieves better unmixing results with fewer iterations, fewer artifacts, smoother images, and better detail information. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0045] Figure 1 This is a schematic diagram of a multicolor fluorescence spectroscopy unmixing method provided by the present invention;
[0046] Figure 2 This is a schematic diagram of an end-to-end neural network mapping structure;
[0047] Figure 3 This is a schematic diagram of the residual block structure of a multi-layer three-dimensional convolutional neural network.
[0048] Figure 4 This is a schematic diagram of the skip connection structure of two multi-layer three-dimensional convolutional neural networks.
[0049] Figure 5 Here are schematic diagrams of the emission spectra of three fluorescent probes;
[0050] Figure 6 This is a schematic diagram illustrating the solution results for the sample. Detailed Implementation
[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0052] The purpose of this invention is to provide a method and system for demixing multicolor fluorescence spectra, which can improve the demixing effect of multicolor fluorescence spectra.
[0053] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0054] like Figure 1 As shown, the multicolor fluorescence spectroscopy unmixing method provided by the present invention includes:
[0055] S101, based on the linear spectral mixing model, constructs an objective function by combining the distribution of detected crosstalk signals, the spectral matrix determined by the emission spectra of each fluorescent probe and the optical system parameters, and the corresponding compensation matrix, along with sparse regularization and total variation regularization constraints.
[0056] The objective function is:
[0057]
[0058] Where Ω is the objective function to be minimized in spectral unmixing. To minimize the objective function Ω in spectral unmixing, the optimization variable is x, A is the spectral matrix, ΔA is the compensation matrix for the spectral matrix A, x is the true distribution of each fluorescent probe in the target organism, Y is the distribution of the detected crosstalk signal, and λ, λ TV Here, |x|1 is the regularization coefficient, |x|2 is the l1 norm of x, ensuring the sparsity of the solution, and TV(x) is the total variation of the anisotropy of the true distribution x of each fluorescent probe. H i x is the discrete gradient extracted from image x at pixel i, and H is the discrete difference operator.
[0059] S102, Based on the objective function, the alternating direction multiplier method is used to determine the computation graph; and the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping;
[0060] S102 determines the computational graph based on the objective function using the alternating direction multiplier method;
[0061] The computational graph is shown below:
[0062] First, the ADMM method is used to obtain the equivalent form of the optimization problem:
[0063]
[0064] U is the main variable in the optimization problem, which is the distribution of each fluorescent probe to be determined; V1 and V2 are auxiliary variables.
[0065] The augmented Lagrange form of the optimization problem is then obtained as follows:
[0066]
[0067] μ1 and μ2 are penalty parameters; μ1D1 and μ2D2 are Lagrange multipliers of the constraints.
[0068] The complex optimization problem can be broken down into a series of simpler subproblems as follows:
[0069]
[0070] The computational graph obtained by solving each subproblem is as follows:
[0071]
[0072] soft(x,θ) is the soft threshold shrinkage function, which is mathematically represented as soft(x,θ)=sign(x)·max(|x|-θ,0).
[0073] Periodic boundary conditions are applied to the computation graph, and the inverse of the Hessian matrix is diagonalized using a two-dimensional discrete Fourier transform.
[0074] Applying periodic boundary conditions to an image can... and The result of dividing the data into blocks is as follows:
[0075]
[0076] a i,j ∈R n×n ,h i,j ∈R n×n The matrix is a cyclic matrix; n×n is the image size of the detector; m is the number of detector channels (number of fluorescent probe types); any cyclic matrix can be diagonalized by a Fourier transform matrix: C = F×Λ×F H ;
[0077] C is the circulant matrix; F is the discrete Fourier transform matrix, which is a unitary matrix, i.e., F×F H =I; Λ is a diagonal matrix, and the diagonal elements are the eigenvalues of the cyclic matrix C;
[0078] Therefore, we have:
[0079]
[0080] It is a unitary matrix; It is a diagonal matrix with elements a i,j ∈R n×n ,h i,j ∈R n×n eigenvalues;
[0081] Through matrix operations, the inverse of the Hessian matrix can be written as:
[0082]
[0083] It is a diagonal matrix;
[0084] Block matrix Each element block is a diagonal matrix, which can be solved directly. The eigenvalue calculation is too complex, so it can be decomposed into n small matrices S. k ∈R mn×mn By solving for the eigenvalues, the inverse of the Hessian matrix is transformed into:
[0085] [(A+ΔA) T (A+ΔA)+μI+μH T H] -1 =LΛL -1 ;
[0086] It is an invertible matrix when the inverse of the Hessian matrix is diagonalized; The eigenvalues of the inverse of the Hessian matrix are obtained by solving n small matrices S. k ∈R mn×mn The characteristic value is obtained;
[0087] The simplified calculation diagram is as follows:
[0088]
[0089] The computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping.
[0090] The design steps for a non-parameter-shared hierarchical network structure for the computation graph include:
[0091] 1) To improve spectral unmixing performance and increase network capacity, this network employs a general nonlinear transformation function. Replace the compensation matrix ΔA respectively T Inverse matrices L, L -1 ,in, It is the identity operator, resulting in the following computation graph:
[0092]
[0093] Compensation Transformation Inverse transformation diagonal matrix Λ k Penalty parameters soft threshold Set as a learnable parameter;
[0094] 2) Based on the iterative computation graph of each round, design the corresponding network structure layer by layer, establish the mapping relationship between layers, and the input of the network structure corresponding to the k-th layer is... The output is The mapping relationship satisfies the iterative calculation steps of the kth round described above;
[0095] 3) Employing a non-shared parameter approach, each layer of the network undergoes parameter compensation transformation. Inverse transformation diagonal matrix Λ k Penalty parameters soft threshold They are all different;
[0096] 4) Fix the total number of network layers N to 10 layers, and combine the network structure of each layer to obtain the end-to-end neural network mapping.
[0097] The overall network mapping structure is as follows Figure 2 As shown.
[0098] S103, based on multiple multi-layer three-dimensional convolutional neural network structures, parameterizes the compensation matrix and inverse matrix in the computation graph to obtain an end-to-end neural network; and constrains the end-to-end neural network through a loss function;
[0099] S103 specifically includes:
[0100] The residual block structure based on a multi-layer 3D convolutional neural network performs compensation transformations in each layer of the network. Parameterization;
[0101] The specific mathematical representation is as follows:
[0102]
[0103] The compensation transformation in the k-th layer network is represented by the residual block structure of a multi-layer three-dimensional convolutional neural network, where the input of the residual block structure of the multi-layer three-dimensional convolutional neural network is the crosstalk signal distribution Y; Compensation transformation in the k-th layer network The convolution kernel parameters of the t-th layer of the 3D convolutional neural network, where the convolution kernel W is the value of the convolution kernel as t ranges from 1 to T-1. t k The size is 3×3×3, and the T-th layer convolution kernel The size is 1×1×1, where T is the total number of layers in the 3D convolutional neural network, which is set to 4; Compensation transformation in the k-th layer network The bias parameters of the t-th layer of the 3D convolutional neural network, where t ranges from 1 to T, are given by the bias parameters. All are scalars; * represents convolution operation; ReLU(·) is a linear rectified function, which acts as a non-linear activation function in the network structure; Compensation transformation in the k-th layer network The crosstalk signal distribution Y in the image is the output of the t-th layer of the three-dimensional convolutional neural network, where t ranges from 0 to T-1 and does not contain residual block structures.
[0104] The residual block structure of the 3D convolutional neural network with compensation transformation is as follows: Figure 3 As shown.
[0105] The skip connection structure based on two multi-layer 3D convolutional neural networks enables the inverse transformations in each layer of the network. and diagonal matrix Λ k Parameterization.
[0106] The specific mathematical representation is as follows:
[0107]
[0108] Inverse transformation in the k-th layer network The convolution kernel parameters of the t-th layer of the 3D convolutional neural network, where t ranges from 1 to T, and the convolution kernel... The size is 3×3×3, where T is the total number of layers in the 3D convolutional neural network, set to 2; ReLU(·) is the linear rectified function, which acts as a non-linear activation function in the network structure; f k For the positive transform in the k-th layer network Input; Forward transformation in the k-th layer network The convolution kernel parameters of the t-th layer of the 3D convolutional neural network, where t ranges from 1 to T, and the convolution kernel... The size is 3×3×3, where T is the total number of layers in the 3D convolutional neural network, set to 2; U k For the k-th layer network after inverse transformation and diagonal matrix Λ k The output is represented by a skip connection structure of two multi-layer 3D convolutional neural networks, where the input of the skip connection network is the intermediate variable r of the k-th layer network. k .
[0109] The skip connection structure of a 3D convolutional neural network with inverse transformations and diagonal matrices is as follows: Figure 4 As shown.
[0110] The loss function is:
[0111]
[0112]
[0113] loss = loss1 + β × loss2
[0114] Where loss1 is the distribution of each fluorescent probe after unmixing, U N The mean square error between the forward and inverse transformation networks and the true distribution x of each fluorescent probe, loss2 is the inverse constraint loss function used to constrain the inverse relationship between the forward and inverse transformation networks, β is the regularization parameter, m is the number of detection channels (number of fluorescent probe types), and n is the mean square error between the forward and inverse transformation networks and the true distribution x of each fluorescent probe. 2 The image size to be detected is given by M, the total number of samples in the training set is given by N, and r is given by r. k The input to the skip connection network is the intermediate variable of the k-th layer network. These are inverse transformations.
[0115] S104, construct a multicolor fluorescence crosstalk simulation dataset; and use the multicolor fluorescence crosstalk simulation dataset to train and test the end-to-end neural network; the multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
[0116] The invention is further illustrated by examples. This embodiment uses three fluorescent probes and a three-channel detector as a simulation case. The fluorescent probes are DAPI, GFP, and YFP, and their emission spectra are as follows: Figure 5 As shown, the good performance of this method in ill-conditioned problems is demonstrated by selecting highly correlated spectral bands. The detection bands of the three channels are selected as 497-509nm, 510-524nm, and 525-539nm, respectively. The spectral matrix is calculated based on the three fluorescent probes and the three detection bands according to the linear spectral mixing model, and the parameters are shown in Table 1. The dataset is constructed using actual mouse brain data detected by the optical system as the real distribution of the three fluorescent probes. The calculated spectral matrix is used to generate the three-channel fluorescence crosstalk signal distribution of the corresponding bands through the forward process of the linear spectral mixing model, and 10% Gaussian noise is added to it, generating a total of 1000 samples. 600 samples are used as the training set to train the proposed neural network, and 400 samples are used as the test set to test the network performance.
[0117] Table 1. Relative fluorescence intensity of each fluorescent probe in each channel within the simulated vivo system.
[0118]
[0119] The fluorescence crosstalk signals from the test set are input into the trained ADMM algorithm expanded network to obtain the spectral unmixing results. The solution effect for one sample is shown below. Figure 6 As shown, compared with the true distribution of each fluorescent probe, the unmixing result of this network is more consistent with the true distribution, with very small error, very few artifacts, smooth image with good detail information, and high image unmixing quality.
[0120] As a specific embodiment, the present invention also provides a multicolor fluorescence spectral demixing system, comprising:
[0121] The objective function construction module is used to construct an objective function based on a linear spectral mixing model, according to the distribution of detected crosstalk signals, the spectral matrix determined by the emission spectra of each fluorescent probe and the optical system parameters, and the corresponding compensation matrix, combined with sparse regularization and total variation regularization constraints.
[0122] The network mapping establishment module is used to determine the computation graph based on the objective function using the alternating direction multiplier method; and the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping.
[0123] The neural network determination module is used to parameterize the compensation matrix and inverse matrix in the computation graph based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and to constrain the end-to-end neural network through a loss function.
[0124] The network training and testing module is used to construct a multicolor fluorescence crosstalk simulation dataset; and to train and test the end-to-end neural network using the multicolor fluorescence crosstalk simulation dataset; the multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
[0125] The network mapping establishment module specifically includes:
[0126] The computational graph determination unit is used to determine the computational graph based on the objective function using the alternating direction multiplier method.
[0127] The matrix transformation unit is used to apply periodic boundary conditions to the computation graph and diagonalize the inverse of the Hessian matrix through a two-dimensional discrete Fourier transform.
[0128] The network mapping establishment unit is used to establish an end-to-end neural network mapping by adopting a hierarchical network structure with non-shared parameters in the computation graph.
[0129] The neural network determination module specifically includes:
[0130] The first parameterization unit is used to parameterize the compensation transformation in each layer of the network based on the residual block structure of a multi-layer three-dimensional convolutional neural network.
[0131] The second parameterization unit is used to parameterize the inverse transformations and diagonal matrices in each layer of the network based on the skip connection structure of two multi-layer three-dimensional convolutional neural networks.
[0132] In order to execute the method corresponding to Embodiment 1 above and achieve the corresponding functions and technical effects, the present invention also provides a multicolor fluorescence spectral demixing system, including: at least one processor, at least one memory, and computer program instructions stored in the memory, wherein when the computer program instructions are executed by the processor, the multicolor fluorescence spectral demixing method described above is implemented.
[0133] The embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.
[0134] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for unmixing multicolor fluorescence spectra, characterized in that, include: Based on the linear spectral mixing model, an objective function is constructed according to the detected crosstalk signal distribution, the spectral matrix determined by the optical system parameters and the emission spectra of each fluorescent probe, and the corresponding compensation matrix, combined with sparse regularization and total variation regularization constraints. Based on the objective function, the alternating direction multiplier method is used to determine the computation graph; the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping; The compensation matrix and inverse matrix in the computation graph are parameterized based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and the end-to-end neural network is constrained by a loss function. Construct a multicolor fluorescence crosstalk simulation dataset; The end-to-end neural network was trained and tested using a multicolor fluorescence crosstalk simulation dataset. The multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
2. The method for unmixing multicolor fluorescence spectra according to claim 1, characterized in that, The objective function is: Where Ω is the objective function to be minimized in spectral unmixing. To minimize the objective function Ω in spectral unmixing, the optimization variable is x, A is the spectral matrix, ΔA is the compensation matrix for the spectral matrix A, x is the true distribution of each fluorescent probe in the target organism, Y is the distribution of the detected crosstalk signal, and λ, λ TV Here, |x|1 is the regularization coefficient, |x|2 is the l1 norm of x, ensuring the sparsity of the solution, and TV(x) is the total variation of the anisotropy of the true distribution x of each fluorescent probe. H i x is the discrete gradient extracted from image x at pixel i, and H is the discrete difference operator.
3. The method for unmixing multicolor fluorescence spectra according to claim 1, characterized in that, The process involves determining the computational graph based on the objective function using the alternating direction multiplier method; and establishing an end-to-end neural network mapping by adopting a non-shared parameter hierarchical network structure for the computational graph. Specifically, this includes: Based on the objective function, the computational graph is determined using the alternating direction multiplier method. Periodic boundary conditions are applied to the computation graph, and the inverse of the Hessian matrix is diagonalized using a two-dimensional discrete Fourier transform. The computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping.
4. The method for unmixing multicolor fluorescence spectra according to claim 1, characterized in that, The method parameterizes the compensation matrix and inverse matrix in the computation graph based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and constrains the end-to-end neural network through a loss function, specifically including: The residual block structure based on multi-layer three-dimensional convolutional neural networks parameterizes the compensation transformation in each layer of the network. The skip connection structure based on two multi-layer three-dimensional convolutional neural networks parameterizes the inverse transformations and diagonal matrices in each layer of the network.
5. The multicolor fluorescence spectroscopy unmixing method according to claim 4, characterized in that, The loss function is: loss = loss1 + β × loss2 Where loss1 is the distribution of each fluorescent probe after unmixing, U N The mean square error between the true distribution x of each fluorescent probe and the mean square error between the two probes, loss2 is the inverse constraint loss function used to constrain the inverse relationship between the forward and inverse transformation networks, β is the regularization parameter, m is the number of detection channels, n is the number of fluorescent probe types, and n is the mean square error between the two probes. 2 The image size to be detected is given by M, the total number of samples in the training set is given by N, and r is given by r. k The input to the skip connection network is the intermediate variable of the k-th layer network. These are inverse transformations.
6. A multicolor fluorescence spectral demixing system, characterized in that, include: The objective function construction module is used to construct an objective function based on a linear spectral mixing model, according to the distribution of detected crosstalk signals, the spectral matrix determined by the optical system parameters and the emission spectra of each fluorescent probe, and the corresponding compensation matrix, combined with sparse regularization and total variation regularization constraints. The network mapping establishment module is used to determine the computation graph based on the objective function using the alternating direction multiplier method; and the computation graph adopts a hierarchical network structure with non-shared parameters to establish an end-to-end neural network mapping. The neural network determination module is used to parameterize the compensation matrix and inverse matrix in the computation graph based on multiple multi-layer 3D convolutional neural network structures to obtain an end-to-end neural network; and to constrain the end-to-end neural network through a loss function. The network training and testing module is used to construct a multicolor fluorescence crosstalk simulation dataset; The end-to-end neural network was trained and tested using a multicolor fluorescence crosstalk simulation dataset. The multicolor fluorescence crosstalk simulation dataset includes: crosstalk signal distribution and the actual distribution of each fluorescent probe.
7. The multicolor fluorescence spectral demixing system according to claim 6, characterized in that, The network mapping establishment module specifically includes: The computational graph determination unit is used to determine the computational graph based on the objective function using the alternating direction multiplier method. The matrix transformation unit is used to apply periodic boundary conditions to the computation graph and diagonalize the inverse of the Hessian matrix through a two-dimensional discrete Fourier transform. The network mapping establishment unit is used to establish an end-to-end neural network mapping by adopting a hierarchical network structure with non-shared parameters in the computation graph.
8. The multicolor fluorescence spectral demixing system according to claim 6, characterized in that, The neural network determination module specifically includes: The first parameterization unit is used to parameterize the compensation transformation in each layer of the network based on the residual block structure of a multi-layer three-dimensional convolutional neural network. The second parameterization unit is used to parameterize the inverse transformations and diagonal matrices in each layer of the network based on the skip connection structure of two multi-layer three-dimensional convolutional neural networks.
9. A multicolor fluorescence spectral unmixing system, characterized in that, include: The method comprises at least one processor, at least one memory, and computer program instructions stored in the memory, which, when executed by the processor, implement a multicolor fluorescence spectral unmixing method as described in any one of claims 1-5.