A three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling

Abstract

The application provides a three-dimensional fluorescence spectrum analysis method based on unsupervised contribution modeling, and the method comprises the following steps: obtaining original three-dimensional fluorescence spectrum of a sample; performing independent component analysis on the three-dimensional fluorescence spectrum, and reconstructing the spectrum by using the analysis result; and modeling the reconstructed spectrum by using a multivariate curve resolution-alternating least squares algorithm to obtain an analysis report. By combining two two-dimensional unsupervised modeling algorithms, the application can automatically remove interference signals such as scattering in the original three-dimensional fluorescence spectrum, then model the reconstructed spectrum, reduce the dependence of three-dimensional fluorescence spectrum analysis on data, tools, platforms and personnel, improve flexibility, ensure good fault tolerance and accuracy, and provide reliable technical support for water quality monitoring.

Description

Technical Field

[0001] This invention relates to the field of qualitative analysis technology of three-dimensional fluorescence spectroscopy, and specifically to a three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling. Background Technology

[0002] Water quality monitoring is crucial for pollution prevention and environmental protection. Existing water quality monitoring technologies, especially fluorescence spectroscopy, are widely used for the detection of dissolved organic matter (DOM) and other fluorescent compounds due to their high sensitivity, non-destructive nature, rapid detection, and low cost. Among these, three-dimensional fluorescence spectroscopy exhibits high selectivity and is suitable as a routine water quality monitoring method because specific fluorescent substances possess unique corresponding EEM characteristics (called fluorescence fingerprints). However, these technologies still face some challenges in practical applications. Especially when dealing with complex environmental samples, fluorescence signals often become complex due to the mixing of multiple pollutants, making signal source resolution difficult. Furthermore, interference from scattered light also makes accurate identification and quantification of pollutants challenging.

[0003] Traditional methods primarily rely on simple multivariate analysis or graphical analysis to extract characteristic signals and concentration distributions of pollutants from three-dimensional fluorescence data, such as publications CN115236048A ("A Method for Monitoring Ammonia Nitrogen Concentration in Water Based on Three-Dimensional Fluorescence Spectroscopy") and CN111198165A ("A Method for Determining Water Quality Parameters Based on Spectral Data Standardization"). Parallel factor analysis (PARAFAC) is considered the standard algorithm for processing EEMs data because it can directly extract chemically meaningful information. However, these methods are sensitive to noise and signal mixing, and perform poorly when dealing with strong scattered light interference. In such cases, manual preprocessing such as scatter cutoff is usually required based on data characteristics, demanding high levels of chemometrics knowledge, programming skills, and practical experience from the analyst, and heavily relying on analytical software such as Matlab, thus limiting the widespread application of three-dimensional fluorescence technology.

[0004] With the rise of artificial intelligence, machine learning algorithms have been successfully applied in various qualitative and quantitative tasks in analytical chemistry. Compared with traditional chemical data processing, machine learning does not assume any predetermined chemical models, but seeks to develop rules in the form of mathematical models to transform chemical measurements into useful information. Among these, convolutional neural networks based on supervised deep learning are the most widely used, such as in publication number CN116720110A, entitled: "An Analysis Method for Three-Dimensional Fluorescence Spectroscopy Data." Deep learning models have low sensitivity to preprocessing and are fast. However, their capabilities come from large amounts of high-quality training data and high-performance computing platforms; the models lack consistency and interpretability, and manual preprocessing is still required under conditions of strong interference.

[0005] Independent Component Analysis (ICA) is a numerical iterative method originating from neural networks. Based on blind source separation algorithms, it deconvolvees a mixed signal by maximizing the independence of each component. Because this algorithm uses an orthogonal negative entropy approximation to maximize the contribution of each independent component, we refer to the data analysis method based on this algorithm as contribution analysis, used to decompose fluorescence signals into independent components. However, since ICA is unrestricted, its solutions must be carefully evaluated before being converted into chemical information. Summary of the Invention

[0006] To address the aforementioned problems, the present invention aims to provide a three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling.

[0007] The technical solution of the present invention is as follows:

[0008] A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling includes the following steps:

[0009] S1: Obtain the original three-dimensional fluorescence spectrum of the sample; the specific process is as follows:

[0010] S1.1: Obtain data from measuring instruments or existing work and organize it into multiple EEM (EEMs) files and metadata files as a dataset; the EEM file contains the EEM data of the sample, and the metadata file includes information such as excitation wavelength and emission wavelength.

[0011] S1.2: Record dataset information in the yml file, including name, nEx, nEm, and number of samples.

[0012] S2: Perform independent component analysis (ICA) on the three-dimensional fluorescence spectrum, and reconstruct the spectrum using the analysis results; the specific process is as follows:

[0013] S2.1: Select an appropriate number of independent contributions, input the EEMs into the ICA algorithm for modeling, and output the signal source matrix and concentration distribution matrix.

[0014] Furthermore, the number of independent contributions can be selected using the following method: gradually increase the number of independent contributions until the algorithm can separate the interference signal as an independent component, and take the number of independent contributions at this point as the optimal value.

[0015] Furthermore, the mathematical principle of the ICA algorithm is X = S·A + V, where X is a two-dimensional matrix (a vector obtained by expanding the sample × EEM), S is the signal source matrix, A is the concentration distribution matrix, and V is the residual.

[0016] S2.2: Fold the signal source matrix into signal source EEMs.

[0017] Furthermore, each signal source EEM actually corresponds to an independent component in the ICA algorithm result.

[0018] S2.3: In the signal source EEMs, separate the interference signal and leave a fluorescent fingerprint.

[0019] Furthermore, the interference signal includes scattering and noise, wherein the scattering includes Rayleigh scattering and Raman scattering.

[0020] S2.4: Merge fluorescent fingerprints with similar peak positions.

[0021] Furthermore, the peak position is a coordinate, which can be represented as (excitation wavelength, emission wavelength).

[0022] Furthermore, the phrase "similar peak positions" means "similar excitation wavelengths" or "similar emission wavelengths".

[0023] Furthermore, the criteria for judging whether two fluorescence peaks are similar are as follows: if the similarity of the excitation wavelength ranges of the two fluorescence peaks is greater than 80%, or the similarity of the emission wavelength ranges of the two fluorescence peaks is greater than 80%, then the peak positions of the two fluorescence peaks are considered to be similar; otherwise, the two fluorescence peaks cannot overlap after merging.

[0024] Furthermore, the specific method for merging fluorescent fingerprints is as follows: select a primary fingerprint and a secondary fingerprint, superimpose the secondary fingerprint onto the primary fingerprint, or invert the secondary fingerprint (multiply by negative one) and superimpose it onto the primary fingerprint, then mark the secondary fingerprint as an interference signal. Since the ICA algorithm does not have a non-negativity constraint and centers the data around 0 during implementation, the results can be positive or negative, resulting in positive and negative fingerprints. Therefore, inversion must be considered during superposition to ensure the consistency of the positive and negative signs between the primary and secondary fingerprints.

[0025] S2.5: Prune the concentration distribution matrix and delete rows or columns corresponding to interference signals.

[0026] S2.6: Expand the fluorescent fingerprint into a fluorescent signal source matrix, multiply it by the concentration distribution matrix to obtain the reconstructed spectrum.

[0027] S3: The reconstructed spectrum is modeled using the MCR-ALS (Multivariate Curve Resolution-Alternating Least Squares) algorithm to obtain an analysis report; the specific process is as follows:

[0028] S3.1: Select appropriate components, input the reconstructed spectrum into the MCR-ALS algorithm for modeling, and output the signal source matrix and concentration distribution matrix.

[0029] Furthermore, the mathematical principle of the MCR-ALS algorithm is d = C·S T , where d is a two-dimensional matrix (a vector obtained by expanding the sample × EEM), S is the signal source matrix, and C is the concentration distribution matrix. The default constraint for the result is non-negativity.

[0030] S3.2: Fold the signal source matrix into a sample fluorescent fingerprint.

[0031] S3.3: Using a reference library, identify the analytes corresponding to the fluorescent fingerprints of samples. Identification requires relying on the fluorescent fingerprints of known analytes, which are usually from internal or external databases and are called reference libraries. Since the fluorescent fingerprints of each analyte are specific, the analytes contained in the sample corresponding to a given three-dimensional fluorescence spectrum are determined by comparison, thereby achieving identification.

[0032] Furthermore, the analyte is a substance or chemical component of interest in the chemical analysis procedure. The analyte can be a specific pure substance (inorganic or organic), or it can be a class of substances (such as humic substances or fulvic acid-like substances in DOM analysis).

[0033] The beneficial effects of this invention are as follows:

[0034] 1) Compared with other methods, this invention supports direct input of raw three-dimensional fluorescence spectra for analysis, which is convenient to operate and has low requirements for data quality.

[0035] 2) Compared with conventional preprocessing methods, this invention uses the ICA algorithm to automatically remove scattering and noise from the spectrum, eliminating the dependence on specific programming platforms and preprocessing tools, thereby effectively reducing the risk of analysis failure due to human factors such as insufficient personnel experience and operational errors.

[0036] 3) Compared with the three-dimensional algorithm PARAFAC, the two-dimensional algorithms (ICA and MCR-ALS) used in this invention are not constrained by the bilinearity assumption of the spectrum, and are more flexible. Among them, the MCR-ALS algorithm supports non-negativity constraints, which can effectively remove noise in the reconstructed spectrum, while ensuring good fault tolerance and accuracy.

[0037] 4) Compared with supervised deep learning and other artificial intelligence methods, this method uses an unsupervised algorithm, which has lower requirements for the amount of data and the computing power of the computing platform, thus effectively saving costs. In addition, the algorithm has clear mathematical theoretical support and good interpretability.

[0038] 5) In summary, the present invention is particularly suitable for processing three-dimensional fluorescence spectra of environmental water samples with complex composition, providing more reliable technical support for water quality monitoring. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the method of the present invention;

[0040] Figure 2 This is a flowchart of the method of the present invention;

[0041] Figure 3 This is a visualization result of the signal source EEMs obtained in step S2.2 of the embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the fluorescent fingerprint obtained in step S2.4 of an embodiment of this application;

[0043] Figure 5 This is a schematic diagram illustrating the similarity of peak positions in step S2.4 of the embodiment of this application;

[0044] Figure 6 This is a schematic diagram of the fluorescent fingerprint obtained in step S3.2 of an embodiment of this application;

[0045] Figure 7 This is a schematic diagram of the reference library fluorescent fingerprint used in step S3.3 of this application embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0047] Conversely, this invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of the invention as defined in the claims. Furthermore, to provide a better understanding of the invention, certain specific details are described in detail below. However, those skilled in the art will fully understand the invention even without these detailed descriptions.

[0048] This invention uses a publicly available dataset called Dorrit as the dataset used in the embodiments. Due to the different publicly available versions from different channels, there may be slight differences in the data format and content, and the experimental results may also differ, but this does not affect the effectiveness of this invention.

[0049] A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling includes the following steps:

[0050] S1: Obtain the raw three-dimensional fluorescence spectrum of the sample. Specifically, this includes:

[0051] S1.1: Write a Matlab script to export data from dorrit.mat. The output will consist of several CSV files, which can be divided into EEM files and metadata files. The metadata files include Ex.csv (excitation wavelength, only one row), Em.csv (emission wavelength, only one row), and conc_labels.csv (concentration table, rows represent the number of samples, columns represent the number of analytes). The EEM files contain the EEM data of the samples, with rows representing the number of emission wavelengths (nEm) and columns representing the number of excitation wavelengths (nEx).

[0052] S1.2: Record the dataset parameters in the yml file, including name, nEx, nEm, and number of samples.

[0053] S2: Independent component analysis (ICA) of the three-dimensional fluorescence spectrum is performed, and the spectrum is reconstructed using the analysis results. Specifically, this includes:

[0054] S2.1: Select an appropriate number of independent contributions, input the EEMs into the ICA algorithm for modeling, and output the signal source matrix and concentration distribution matrix. For this example, the number of independent contributions is selected as 7.

[0055] S2.2: Fold the signal source matrix into signal source EEMs. For the example, the results are shown in the appendix. Figure 3 .

[0056] S2.3: In the signal source EEMs, the interference signal is separated and a fluorescent fingerprint is left. For the example, it is easy to see that EEM plot 1 is the interference signal and EEM plots 2-7 are the fluorescent fingerprints.

[0057] S2.4: Merge fluorescence fingerprints with similar peak positions. In this embodiment, the peak positions of EEM plot 2 and EEM plot 6 are similar, and the peak positions of EEM plot 3 and EEM plot 4 are similar, which can be denoted as E[2] ∽ E[6], E[3] ∽ E[4]. Superimpose EEM plot 6 on EEM plot 2, reverse EEM plot 3 and then superimpose it on EEM plot 4, and then mark EEM plot 6 and EEM plot 3 as interference signals. The results are shown in the appendix Figure 4 .

[0058] Among them, for the judgment method of similar peak positions, simple visual judgment can be adopted, or quantitative calculation can be adopted. For visual judgment, please see the appendix Figure 3 . It is easy to see with the naked eye that the excitation wavelength positions of the fluorescence fingerprints in EEM plot 2 are similar to those of the fluorescence fingerprints in EEM plot 6, and the excitation wavelength positions of the fluorescence fingerprints in EEM plot 3 are similar to those of the fluorescence fingerprints in EEM plot 4, which can be denoted as PeakEx[2] ∽ PeakEx[6], PeakEx[3] ∽ PeakEx[4]. At the same time, the emission wavelength positions of the fluorescence fingerprints in the two groups of EEM plots are not similar, so they will not overlap after merging, which can be denoted as For quantitative calculation, please see the appendix Figure 5 . The appendix Figure 5 In the appendix Figure 3 , the excitation wavelength ranges of the fluorescence fingerprints of EEM plots 2, 3, 4, and 6 are marked with black lines. After obtaining the interval coordinates, the interval similarity calculation can be carried out. Here we use the Jaccard similarity coefficient as the similarity index. Let interval A = [a1, a2] and interval B = [b1, b2], where a1 < a2 and b1 < b2. Then the formula is as follows:

[0059]

[0060] The calculation results of the above formula are shown in Table 1.

[0061] Table 1

[0062] EEMplot number Similarity of excitation wavelength range of fluorescent fingerprints 2 and 6 93％ 3 and 4 83％

[0063] Both of these similarities exceed 80%, so we can get E[2] ∽ E[6], E[3] ∽ E[4].

[0064] S2.5: Crop the concentration distribution matrix and delete the columns corresponding to the interference signals. For this embodiment, delete columns 1, 3, and 6.

[0065] S2.6: Expand the fluorescent fingerprint into a fluorescent signal source matrix, multiply it by the concentration distribution matrix to obtain the reconstructed spectrum.

[0066] S3: The reconstructed spectrum is modeled using the MCR-ALS (Multivariate Curve Resolution-Alternating Least Squares) algorithm to obtain an analysis report.

[0067] S3.1: Select an appropriate number of components, input the EEMs into the MCR-ALS algorithm for modeling, and output the signal source matrix and concentration distribution matrix. For the example, select 4 components (step S2 modeling yields 7 signal source EEMs, remove the first EEM (interference signal), merge EEMs with similar peak positions, and finally leave 4 EEMs (fluorescent fingerprints)).

[0068] S3.2: Fold the signal source matrix into a sample fluorescent fingerprint. For the example, the results are shown in the appendix. Figure 6 .

[0069] S3.3: Using a benchmark library, identify the analytes corresponding to the sample's fluorescent fingerprint. For the example, the PARAFAC modeling results of the sample are used (see Appendix). Figure 7 Using [a specific method / mechanism] as a benchmark, the results were compared with those of the present invention, and the results are shown in Table 2. The similarity calculation index is the Pearson coefficient of determination, and the formula is:

[0070]

[0071] Table 2

[0072] Standard algorithm fluorescent fingerprint numbering Fluorescent fingerprint number of this invention Similarity 1 2 88.4％ 2 4 89.4％ 3 3 89.7％ 4 1 84.9％

[0073] As shown in Table 2, the matching results of the present invention are consistent with the facts, and the average similarity reaches 88.1%, indicating that the present invention is effective.

[0074] This invention combines two two-dimensional unsupervised modeling algorithms to automatically remove interference signals such as scattering from the original three-dimensional fluorescence spectrum, and then models the reconstructed spectrum. This reduces the dependence of three-dimensional fluorescence spectroscopy analysis on data, tools, platforms, and personnel, while improving flexibility and ensuring good fault tolerance and accuracy, providing reliable technical support for water quality monitoring.

[0075] The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the technical solutions of the present invention. Any technical solution that can be implemented based on the above embodiments without creative effort should be considered to fall within the scope of protection of the patent of the present invention.

Claims

1. A three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling, characterized in that, The method includes the following steps: S1: Obtain the raw three-dimensional fluorescence spectrum of the sample; S2: Perform independent component analysis (ICA) on the three-dimensional fluorescence spectrum and reconstruct the spectrum using the analysis results; Step S2 includes the following steps: S2.1: Select the optimal number of independent contributions, convert the original EEMs into matrices, input them into the ICA algorithm for modeling, and output the signal source matrix and concentration distribution matrix; S2.2: Convert the signal source matrix into signal source EEMs; S2.3: In the signal source EEMs, the interference signal is separated and a fluorescent fingerprint is left; S2.4: Merge fluorescent fingerprints with similar peak positions; S2.5: Prune the concentration distribution matrix and delete rows or columns corresponding to interference signals; S2.6: Convert the fluorescent fingerprint into a fluorescent signal source matrix, multiply it by the concentration distribution matrix to obtain the reconstructed spectrum; S3: The reconstructed spectrum is modeled using the Multivariate Curve Resolution-Alternating Least Squares (MCR-ALS) algorithm, and an analysis report is obtained.

2. The three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 1, characterized in that, In step S1, the three-dimensional fluorescence spectrum includes multiple excitation-emission matrices (EEMs) and corresponding metadata. The multiple excitation-emission matrices (EEMs) are denoted as EEMs. The metadata is the data output by the measuring instrument other than EEMs, including excitation wavelength, emission wavelength, date and time, and other preset scanning parameters. The three-dimensional fluorescence spectrum comes from actual measurement or artificial simulation.

3. The three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 1, characterized in that, The criterion for judging whether the peak positions are similar in step S2.4 is as follows: if the similarity of the excitation wavelength ranges of the two fluorescence peaks is 80% or more, or the similarity of the emission wavelength ranges of the two fluorescence peaks is 80% or more, then the peak positions of the two fluorescence peaks are considered to be similar; otherwise, the two fluorescence peaks cannot overlap after merging.

4. The three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 1, characterized in that, Step S3 includes the following steps: S3.1: Select the number of fluorescent fingerprints remaining after step S2.4 as the number of components, input the reconstructed spectrum into the MCR-ALS algorithm for modeling, and output the signal source matrix and concentration distribution matrix; S3.2: Convert the signal source matrix into a sample fluorescent fingerprint; S3.3: Using a benchmark library, identify the analytes corresponding to the fluorescent fingerprints of the samples.

5. The three-dimensional fluorescence spectroscopy analysis method based on unsupervised contribution modeling according to claim 4, characterized in that, Step S3.3 requires identification based on the fluorescence fingerprints of known analytes. The fluorescence fingerprints of known analytes come from internal or external databases, called benchmark databases. Since the fluorescence fingerprints of each analyte are specific, the analytes contained in the sample corresponding to a given three-dimensional fluorescence spectrum are determined by comparison, thereby achieving identification.

Images