Three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling

Abstract

The invention provides a three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling. The method comprises the following steps: acquiring an original three-dimensional fluorescence spectrum of a sample; carrying out independent component analysis on the three-dimensional fluorescence spectrum, and reconstructing the spectrum by utilizing an analysis result; and carrying out modeling on the reconstructed spectrum by using a multivariate curve resolution-alternating least square algorithm to obtain an analysis report. According to the method, interference signals such as scattering in an original three-dimensional fluorescence spectrum are automatically removed through combination of two two-dimensional unsupervised modeling algorithms, then modeling is carried out on a reconstructed spectrum, dependence of three-dimensional fluorescence spectrum analysis on data, tools, platforms and personnel is reduced, good fault tolerance and accuracy are guaranteed while flexibility is improved, and the method is suitable for popularization and application. And reliable technical support is provided for water quality monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of qualitative analysis of three-dimensional fluorescence spectra, and in particular to a three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling. Background Art

[0002] Water quality monitoring is of great significance for preventing pollution and protecting the environment. Existing water quality monitoring technologies, especially fluorescence spectroscopy, are widely used for the detection of dissolved organic matter (DOM) and other fluorescent compounds due to their high sensitivity, non-destructiveness, rapid detection and low cost. Among them, since specific fluorescent substances have unique EEM characteristics corresponding to them (called fluorescence fingerprints), the three-dimensional fluorescence method has high selectivity and is suitable as a conventional water quality monitoring method. However, these technologies still face some challenges in practical applications. Especially when dealing with complex environmental samples, the fluorescence signal often becomes complicated due to the mixture of multiple pollutants, making it difficult to resolve the signal source, and the interference of scattered light also makes it difficult to correctly identify and quantify pollutants.

[0003] Traditional methods mainly rely on simple multivariate analysis methods or graphical analysis to extract characteristic signals and concentration distributions of pollutants from three-dimensional fluorescence data, such as publication number CN115236048A, entitled: A method for monitoring ammonia nitrogen concentration in water based on three-dimensional fluorescence spectroscopy, and publication number CN111198165A, entitled: A method for determining water quality parameters based on spectral data standardization. Among them, the parallel factor analysis (PARAFAC) method is considered to be a standard algorithm for processing EEMs data because it can directly extract information with chemical significance. However, these methods are sensitive to the degree of mixing of noise and signal, and are not effective when dealing with strong scattered light interference. In this case, it is usually necessary to perform preprocessing such as scattering removal manually according to data characteristics, which requires high chemometric knowledge, programming skills and practical operation experience of the analyst, and is heavily dependent on analysis software such as Matlab, which limits the widespread application of three-dimensional fluorescence technology.

[0004] With the rise of artificial intelligence, machine learning algorithms have been successfully applied in various qualitative and quantitative tasks in analytical chemistry. Compared with traditional chemical data processing, machine learning does not assume any predetermined chemical model, but seeks to develop rules in the form of mathematical models to convert chemical measurements into useful information. Among them, convolutional neural networks based on supervised deep learning are the most widely used, such as publication number CN116720110A, entitled: A method for analyzing three-dimensional fluorescence spectrum data. Deep learning models are less sensitive to preprocessing and faster. However, its capabilities come from a large amount of high-quality training data and a computing platform with high computing power. The model lacks consistency and interpretability, and manual preprocessing is still required in the case of strong interference.

[0005] Independent component analysis (ICA) is a numerical iterative method that originated from neural networks and is based on a blind source separation algorithm to deconvolve mixed signals by maximizing the independence of each of the mixed signals. Since the algorithm uses an orthogonal negative entropy approximation to maximize the contribution of independent components, we call the data analysis method based on this algorithm contribution analysis, which is used to decompose fluorescence signals into independent components. However, since ICA is not subject to any restrictions, the solution of ICA must be carefully evaluated before converting it into chemical information. Summary of the invention

[0006] In view of the above problems, the object of the present invention is to provide a three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling.

[0007] The technical solution of the present invention is as follows:

[0008] A three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling comprises the following steps:

[0009] S1: Obtain the original three-dimensional fluorescence spectrum of the sample; the specific process is as follows:

[0010] S1.1: Obtain data from measurement instruments or existing work and organize them into multiple EEM (EEMs) files and metadata files as data sets; the EEM file is the EEM data of the sample, and the metadata file includes information such as excitation wavelength and emission wavelength.

[0011] S1.2: Record the dataset information in the yml file, including name, nEx, nEm, and number of samples.

[0012] S2: Perform independent component analysis (ICA) on the three-dimensional fluorescence spectrum and use the analysis results to reconstruct the spectrum; the specific process is as follows:

[0013] S2.1: Select an appropriate number of independent contributions, input EEMs into the ICA algorithm for modeling, and output the signal source matrix and concentration distribution matrix.

[0014] Furthermore, the selection of the number of independent contributions may be performed using the following method: gradually increasing the number of independent contributions until the algorithm is able to separate the interference signal as an independent component, and taking the number of independent contributions at this time as the optimal value.

[0015] Furthermore, the mathematical principle of the ICA algorithm is X=S·A+V, where X is a two-dimensional matrix (a vector obtained by expanding the sample×EEM), S is a signal source matrix, A is a concentration distribution matrix, and V is a residual.

[0016] S2.2: Collapse the signal source matrix into signal source EEMs.

[0017] Furthermore, each signal source EEM actually corresponds to an independent component in the ICA algorithm result.

[0018] S2.3: In the signal source EEMs, separate the interference signal and leave the fluorescence fingerprint.

[0019] Furthermore, the interference signal includes scattering and noise, wherein the scattering includes Rayleigh scattering and Raman scattering.

[0020] S2.4: Merge fluorescence fingerprints with similar peak positions.

[0021] Furthermore, the peak position is a coordinate and can be expressed as (excitation wavelength, emission wavelength).

[0022] Furthermore, the close peak positions mean that the excitation wavelengths are close or the emission wavelengths are close.

[0023] Furthermore, the criterion for judging whether the peak positions are similar is: if the similarity of the excitation wavelength intervals of the two fluorescence peaks reaches more than 80%, or the similarity of the emission wavelength intervals of the two fluorescence peaks reaches more than 80%, then the peak positions of the two fluorescence peaks are considered to be similar; in addition, the two fluorescence peaks cannot overlap after being merged.

[0024] Furthermore, the specific method of merging fluorescent fingerprints is: select a main fingerprint and a secondary fingerprint, superimpose the secondary fingerprint on the main fingerprint, or negate the secondary fingerprint (multiply by negative one) and superimpose it on the main fingerprint, and then mark the secondary fingerprint as an interference signal. Since the ICA algorithm has no non-negative restriction and the data is centered around 0 during implementation, the results are both positive and negative, and there will be positive and negative fingerprints accordingly. When superimposing, it is necessary to consider whether to negate it as appropriate to ensure the positive and negative consistency of the main fingerprint and the secondary fingerprint.

[0025] S2.5: Crop the concentration distribution matrix and delete the rows or columns corresponding to the interference signal.

[0026] S2.6: Expand the fluorescence fingerprint into a fluorescence signal source matrix, multiply it with the concentration distribution matrix to obtain the reconstructed spectrum.

[0027] S3: Use the MCR-ALS (Multivariate Curve Resolution-Alternating Least Squares) algorithm to model the reconstructed spectrum and obtain an analysis report; the specific process is as follows:

[0028] S3.1: Select an appropriate number of components, input the reconstructed spectrum into the MCR-ALS algorithm for modeling, and output the signal source matrix and concentration distribution matrix.

[0029] Furthermore, the mathematical principle of the MCR-ALS algorithm is d=C·S T , where d is a two-dimensional matrix (the vector obtained by sample × EEM expansion), S is the signal source matrix, and C is the concentration distribution matrix. The default constraint on the result is non-negative.

[0030] S3.2: Collapse the signal source matrix into a sample fluorescence fingerprint.

[0031] S3.3: Use the reference library to identify the analyte corresponding to the sample's fluorescence fingerprint. The identification is based on the fluorescence fingerprint of known analytes, which usually come from an internal or external database, called the reference library. Since the fluorescence fingerprint of each analyte is specific, the analyte contained in the sample corresponding to a given three-dimensional fluorescence spectrum is determined by comparison, thereby achieving identification.

[0032] Further, the analyte is a substance or chemical component of interest in a chemical analysis procedure. The analyte can be a specific pure substance (inorganic, organic), or a class of substances (such as humus-like substances and fulvic acid-like substances in DOM analysis).

[0033] The beneficial effects of the present invention are as follows:

[0034] 1) Compared with other methods, the present invention supports direct input of original three-dimensional fluorescence spectra for analysis, is easy to operate, and has low requirements on data quality.

[0035] 2) Compared with conventional preprocessing methods, the present invention uses the ICA algorithm to automatically remove scattering and noise in the spectrum, getting rid of the dependence on specific programming platforms and preprocessing tools, thereby effectively reducing the risk of analysis failure due to human factors such as lack of experience and operational errors.

[0036] 3) Compared with the three-dimensional algorithm PARAFAC, the two-dimensional algorithms (ICA and MCR-ALS) used in the present invention are not constrained by the spectral bilinear assumption and are more flexible. Among them, the MCR-ALS algorithm supports non-negative constraints and can well remove noise in the reconstructed spectrum while ensuring good fault tolerance and accuracy.

[0037] 4) Compared with supervised deep learning and other artificial intelligence methods, this method uses unsupervised algorithms, which have lower requirements on the amount of data and computing power of the computing platform, and can effectively save costs. In addition, the algorithm has clear mathematical theoretical support and good interpretability.

[0038] 5) In summary, the present invention is particularly suitable for processing three-dimensional fluorescence spectra of environmental water samples with complex components, providing more reliable technical support for water quality monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a schematic diagram of the method of the present invention;

[0040] Figure 2 is a flow chart of the method of the present invention;

[0041] Figure 3 This is a visualization result diagram of the signal source EEMs obtained in step S2.2 of the embodiment of the present application;

[0042] Figure 4 This is a schematic diagram of the fluorescent fingerprint obtained in step S2.4 of the embodiment of the present application;

[0043] Figure 5 This is a schematic diagram for explaining the similarity of peak positions in step S2.4 of the embodiment of the present application;

[0044] Figure 6 This is a schematic diagram of the fluorescent fingerprint obtained in step S3.2 of the embodiment of the present application;

[0045] Figure 7 This is a schematic diagram of the reference library fluorescence fingerprint used in step S3.3 of this embodiment of the application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0048] The present invention uses a public dataset named dorrit as the dataset used in the embodiments. Due to the different public versions from different channels, there may be slight differences in the data format and content, and the experimental results may also be different, but this does not affect the effectiveness of the present invention.

[0049] A three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling comprises the following steps:

[0050] S1: Obtain the original three-dimensional fluorescence spectrum of the sample. Specifically including:

[0051] S1.1: Write a Matlab script to export data from dorrit.mat. The output is several csv files, which can be divided into EEM files and metadata files. The metadata files include Ex.csv (excitation wavelength, only one row), Em.csv (emission wavelength, only one row), and conc_labels.csv (concentration table, the number of rows is the number of samples, and the number of columns is the number of analytes). The EEM file is the EEM data of the sample, the number of rows is the number of emission wavelengths (nEm), and the number of columns is the number of excitation wavelengths (nEx).

[0052] S1.2: Record the dataset parameters in the yml file, including name, nEx, nEm, and number of samples.

[0053] S2: Perform independent component analysis (ICA) on the three-dimensional fluorescence spectrum and use the analysis results to reconstruct the spectrum. Specifically include:

[0054] S2.1: Select an appropriate number of independent contributions, input the EEMs into the ICA algorithm for modeling, and output a signal source matrix and a concentration distribution matrix. For the embodiment, the number of independent contributions is selected as 7.

[0055] S2.2: Fold the signal source matrix into signal source EEMs. For the example, the results are shown in the attached Figure 3 .

[0056] S2.3: In the signal source EEMs, separate the interference signal and leave the fluorescence fingerprint. For the embodiment, it is easy to obtain that EEM plot 1 is the interference signal and EEM plots 2-7 are the fluorescence fingerprint.

[0057] S2.4: Merge fluorescence fingerprints with similar peak positions. In this embodiment, the peak positions of EEM plot 2 and EEM plot 6 are similar, and the peak positions of EEM plot 3 and EEM plot 4 are similar, which can be denoted as E[2] ∽ E[6], E[3] ∽ E[4]. Superimpose EEM plot 6 on EEM plot 2, invert EEM plot 3 and then superimpose it on EEM plot 4, and then mark EEM plot 6 and EEM plot 3 as interference signals. The results are shown in the appendix Figure 4 .

[0058] Among them, for the judgment method of similar peak positions, simple visual judgment can be adopted, or quantitative calculation can be adopted. For visual judgment, please see the appendix Figure 3 . It is easy to see with the naked eye that the excitation wavelength positions of the fluorescence fingerprints in EEM plot 2 are similar to those in EEM plot 6, and the excitation wavelength positions of the fluorescence fingerprints in EEM plot 3 are similar to those in EEM plot 4, which can be denoted as PeakEx[2] ∽ PeakEx[6], PeakEx[3] ∽ PeakEx[4]. At the same time, the emission wavelength positions of the fluorescence fingerprints in the two groups of EEM plots are not similar, so they will not overlap after merging, which can be denoted as For quantitative calculation, please see the appendix Figure 5 . The appendix Figure 5 Based on the appendix Figure 3 , the excitation wavelength ranges of the fluorescence fingerprints of EEM plots 2, 3, 4, and 6 are marked with black lines. After obtaining the interval coordinates, the interval similarity calculation can be carried out. Here we use the Jaccard similarity coefficient as the similarity index. Let interval A = [a1, a2] and interval B = [b1, b2], where a1 < a2 and b1 < b2. Then the formula is as follows:

[0059]

[0060] The calculation results of the above formula are shown in Table 1.

[0061] Table 1

[0062] EEMplot Number Similarity of the excitation wavelength range of the fluorescence fingerprint 2 and 6 93％ 3 and 4 83％

[0063] Both of these similarities exceed 80%, so E[2] ∽ E[6], E[3] ∽ E[4] can be obtained.

[0064] S2.5: Crop the concentration distribution matrix and delete the columns corresponding to the interference signals. For the embodiment, delete columns 1, 3, and 6.

[0065] S2.6: Expand the fluorescence fingerprint into a fluorescence signal source matrix, multiply it with the concentration distribution matrix to obtain the reconstructed spectrum.

[0066] S3: Use the MCR-ALS (Multivariate Curve Resolution-Alternating Least Squares) algorithm to model the reconstructed spectrum and obtain an analysis report.

[0067] S3.1: Select an appropriate number of components, input the EEMs into the MCR-ALS algorithm for modeling, and output a signal source matrix and a concentration distribution matrix. For the embodiment, the number of components is selected as 4, (the modeling in step S2 obtains 7 signal source EEMs, removes the first EEM (interference signal), merges EEMs with similar peak positions, and finally leaves 4 EEMs (fluorescence fingerprints)).

[0068] S3.2: Fold the signal source matrix into the sample fluorescence fingerprint. For the example, the results are shown in the attached Figure 6 .

[0069] S3.3: Using the reference library, identify the analyte corresponding to the sample fluorescence fingerprint. For the embodiment, the PARAFAC modeling results of the sample (see Appendix Figure 7 ) as a benchmark, and compared with the results of the present invention, the results are shown in Table 2. Among them, the similarity calculation index is the Pearson determination coefficient, and the formula is:

[0070]

[0071] Table 2

[0072] Standard algorithm fluorescence fingerprint numbering Fluorescent fingerprint numbering of the present invention Similarity 1 2 88.4％ 2 4 89.4％ 3 3 89.7％ 4 1 84.9％

[0073] It can be seen from Table 2 that the matching results of the present invention are consistent with the facts, and the average similarity reaches 88.1%, indicating that the present invention is effective.

[0074] The present invention automatically removes interference signals such as scattering in the original three-dimensional fluorescence spectrum through a combination of two two-dimensional unsupervised modeling algorithms, and then models the reconstructed spectrum, thereby reducing the dependence of three-dimensional fluorescence spectrum analysis on data, tools, platforms and personnel, while improving flexibility and ensuring good fault tolerance and accuracy, providing reliable technical support for water quality monitoring.

[0075] The above embodiments are only preferred embodiments of the present invention and are not limitations of the technical solutions of the present invention. Any technical solution that can be implemented on the basis of the above embodiments without creative work should be deemed to fall within the scope of protection of the patent of the present invention.

Claims

1. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling, characterized in that: The method comprises the following steps: S1: Get the original three-dimensional fluorescence spectrum of the sample; S2: Perform independent component analysis (ICA) on the three-dimensional fluorescence spectrum and use the analysis results to reconstruct the spectrum; S3: Use the multivariate curve resolution-alternating least squares algorithm MCR-ALS to model the reconstructed spectrum and obtain an analysis report.

2. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 1, characterized in that: The three-dimensional fluorescence spectrum in step S1 includes multiple excitation emission matrices EEM and corresponding metadata, wherein the multiple excitation emission matrices EEM are recorded as EEMs; the metadata is the data output by the measuring instrument other than EEMs, including excitation wavelength, emission wavelength, date and time and other preset scanning parameters; the three-dimensional fluorescence spectrum comes from actual measurement or from artificial simulation.

3. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 1, characterized in that: The step S2 comprises the following steps: S2.1: Select the optimal number of independent contributions, convert the original EEMs into matrices and input them into the ICA algorithm for modeling, and output the signal source matrix and concentration distribution matrix. S2.2: Convert the signal source matrix into signal source EEMs; S2.3: In the signal source EEMs, the interference signal is separated, leaving the fluorescence fingerprint; S2.4: Merge fluorescence fingerprints with similar peak positions; S2.5: trim the concentration distribution matrix and delete the row or column corresponding to the interference signal; S2.6: Convert the fluorescence fingerprint into a fluorescence signal source matrix, multiply it with the concentration distribution matrix to obtain the reconstructed spectrum.

4. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 3, characterized in that: The judgment criterion for the similarity of peak positions in step S2.4 is: if the similarity of the excitation wavelength ranges of the two fluorescence peaks reaches more than 80%, or the similarity of the emission wavelength ranges of the two fluorescence peaks reaches more than 80%, then the peak positions of the two fluorescence peaks are considered to be similar; in addition, the two fluorescence peaks cannot overlap after being merged.

5. The three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 3, characterized in that: The step S3 comprises the following steps: S3.1: Select the number of fluorescence fingerprints remaining after step 2.4 as the number of components, input the reconstructed spectrum into the MCR-ALS algorithm for modeling, and output the signal source matrix and concentration distribution matrix; S3.2: Convert the signal source matrix into sample fluorescence fingerprint; S3.3: Using the reference library, identify the analyte corresponding to the sample fluorescence fingerprint.

6. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 5, characterized in that: The step S3.3 needs to be based on the fluorescence fingerprint of the known analyte during identification; the fluorescence fingerprint of the known analyte usually comes from an internal or external database, called a reference library; since the fluorescence fingerprint of each analyte is specific, the analyte contained in the sample corresponding to a given three-dimensional fluorescence spectrum is determined by comparison, thereby achieving identification.

Images