A Raman Spectroscopy Denoising Method Based on Adaptive Sparse Decomposition

Abstract

This invention discloses a Raman spectroscopy denoising method based on adaptive sparse decomposition, comprising the following steps: constructing a static dictionary based on real Raman spectra; acquiring the Raman spectrum to be processed, and traversing the spectra of each substance in the static dictionary to perform similarity calculations, and querying the first characteristic peak with the highest similarity that meets preset conditions; when the similarity with the spectra of each substance does not meet the preset conditions, constructing a dynamic dictionary based on the Raman spectrum to be processed, and generating a second characteristic peak; performing sparse decomposition on the Raman spectrum to be processed, and by querying the static or dynamic dictionary, selecting the matching first or second characteristic peak in the Raman spectrum to be processed, and correcting the Raman spectrum to be processed to obtain a denoised spectrum; this invention uses the construction of a dictionary combining dynamic and static methods to correct the spectral peaks in the Raman spectrum to achieve denoising, thus improving the denoising effect.

Description

Technical Field

[0001] This invention relates to the field of spectral data analysis technology, and more specifically to a Raman spectral denoising method based on adaptive sparse decomposition. Background Technology

[0002] Raman spectroscopy is a type of scattering spectroscopy where characteristic peaks are related to the vibrational or rotational energy levels of molecules in a material and contain information about these vibrations or rotations. Raman characteristic peaks can reflect molecular information within a substance and can be used in fields such as food analysis, life sciences, and customs monitoring. Analysis of existing technologies reveals at least the following problems: Chemometric Raman spectroscopy preprocessing algorithms suffer from poor noise reduction robustness, poor feature retention, and poor peak shape fitting; neural network-based methods for Raman spectroscopy preprocessing face challenges such as difficulty in database construction and slow processing speed.

[0003] Since Raman spectrometers mostly use CCD elements for measurement, they are often accompanied by various noise interferences, such as shot noise, dark current noise, and emission noise, collectively referred to as random noise. Furthermore, due to interference from material background fluorescence, significant fluorescence background noise will exist in the spectrum. These noises will affect the Raman spectrum and distort the spectral characteristic peaks. Currently, the commonly used Raman spectroscopy preprocessing approaches are mainly the following two:

[0004] Chemometrics-based spectral preprocessing methods first remove random noise from the spectrum using smoothing filters. These smoothing filters primarily include the sliding window averaging method, the sliding window median method, and the Savitzky-Golay (SG) filter. Then, the spectral baseline is estimated using least squares and polynomial fitting to remove background noise. This traditional preprocessing algorithm is simple, fast, and easy to understand. However, it only performs statistical analysis on the original spectral data to remove background noise, resulting in poor robustness. Furthermore, the removed noise data contains many characteristic peaks, leading to the loss of Raman characteristic peak intensity and morphology.

[0005] Machine learning-based spectral preprocessing methods primarily train a network by acquiring standard spectra, resulting in a network with excellent denoising performance. However, its preprocessing effectiveness is heavily influenced by the training data set. More accurate and comprehensive the training data, the better the network performs in removing background baselines. However, constructing the training set requires significant time for data acquisition, and ideal Raman spectral training data is difficult to obtain using ordinary experimental instruments, placing high demands on the equipment. Therefore, the effectiveness of machine learning-based Raman spectral preprocessing methods remains limited most of the time, failing to effectively remove all background baselines and random noise.

[0006] In recent years, sparse decomposition, a widely used algorithm in image and signal processing, has also gradually emerged in the field of spectroscopy. Given a complete spectral dictionary, sparse decomposition can effectively extract spectral features and eliminate various noises in the original signal through adaptive linear combination. However, many current sparse decomposition dictionaries use fixed Gaussian or other wavelet dictionaries, which are difficult to use for denoising based on the characteristics of Raman spectra.

[0007] Therefore, how to improve the denoising effect and make up for the shortcomings of dictionary construction are problems that urgently need to be solved by those skilled in the art. Summary of the Invention

[0008] In view of this, the present invention provides a Raman spectral denoising method based on adaptive sparse decomposition, which uses a dictionary combining dynamic and static methods to correct the spectral peaks in the Raman spectrum to achieve denoising, thereby improving the denoising effect.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] A Raman spectral denoising method based on adaptive sparse decomposition includes the following steps:

[0011] A static dictionary is constructed based on real Raman spectra.

[0012] The Raman spectrum to be processed is obtained, and the spectra of each substance in the static dictionary are traversed to calculate the similarity. The first characteristic peak with the highest similarity and that meets the preset conditions is then searched.

[0013] When the similarity with the spectra of each substance does not meet the preset conditions, a dynamic dictionary is constructed based on the Raman spectrum to be processed, and a second characteristic peak is generated.

[0014] The orthogonal matching pursuit algorithm is used to perform sparse decomposition on the Raman spectrum to be processed. By querying a static or dynamic dictionary, the matching first or second characteristic peak is selected in the Raman spectrum to be processed. The Raman spectrum to be processed is corrected according to the matching degree, and the denoised spectrum is obtained.

[0015] Furthermore, a static dictionary is constructed based on real historical Raman spectra, including the following steps:

[0016] Obtain the actual Raman spectrum from the spectral database;

[0017] The actual Raman spectrum is preprocessed;

[0018] Identify characteristic peaks in the preprocessed spectrum;

[0019] The first characteristic peaks corresponding to each substance are obtained by screening the identified characteristic peaks based on the standard Raman spectrum peaks.

[0020] Furthermore, the preprocessing steps include:

[0021] Calculate the average spectrum of the actual Raman spectrum to obtain the mean data;

[0022] The fluorescence subtraction algorithm was used to perform baseline correction on the mean data.

[0023] Furthermore, after identifying the characteristic peaks, the steps also include:

[0024] The Voigt function is used to fit the peak shape of the identified characteristic peaks.

[0025] Furthermore, characteristic peaks are identified in the preprocessed spectrum, specifically:

[0026] An iterative peak-filling algorithm is used to calculate the spectral residuals and fill in the peaks at the positions of the highest residual values.

[0027] Calculate the loss and perform iterative parameter optimization;

[0028] The process is iterated until the preset convergence condition is met, and the characteristic peak parameters are obtained.

[0029] Furthermore, the process of filtering the identified characteristic peaks based on standard Raman spectral peaks specifically involves:

[0030] Based on the position of the standard Raman spectrum peak, characteristic peaks with corresponding positional relationships are selected from the identified characteristic peaks as the first characteristic peak.

[0031] Furthermore, a dynamic dictionary is constructed based on the Raman spectrum to be processed, the steps of which include:

[0032] Baseline correction is performed on the Raman spectrum to be processed;

[0033] An iterative peak-compensation algorithm is used to identify and optimize characteristic peaks.

[0034] The optimized feature peaks are fitted using the Voigt function to obtain the fitted peak shape, which is used as the second feature peak in the dynamic dictionary.

[0035] Furthermore, the steps of constructing a static dictionary based on real historical Raman spectra also include:

[0036] After identifying the characteristic peaks, the absolute intensity of each spectral characteristic peak is recorded and normalized, and the resulting weight is used as the upper limit of the weight.

[0037] During sparse decomposition, after selecting the matching first characteristic peak in the Raman spectrum to be processed and calculating the weight, the weight is constrained by the upper limit of the weight.

[0038] A substance identification method based on Raman spectroscopy includes the following steps:

[0039] Obtain Raman spectra that carry information about the type of substance;

[0040] The Raman spectrum is denoised using the aforementioned Raman spectroscopy denoising method based on adaptive sparse decomposition.

[0041] The denoised Raman spectrum is input into the trained neural network model, which outputs the substance category.

[0042] A method for determining substance concentration based on Raman spectroscopy includes the following steps:

[0043] Obtain Raman spectra that carry information about the concentration of the mixture;

[0044] The Raman spectrum is denoised using one of the adaptive sparse decomposition-based Raman spectral denoising methods described above.

[0045] The denoised Raman spectrum is input into the trained neural network model, which outputs the concentration of each substance in the mixture.

[0046] The beneficial effects of this invention are:

[0047] As can be seen from the above technical solution, compared with the prior art, this invention discloses a Raman spectral denoising method based on adaptive sparse decomposition. It achieves denoising by correcting spectral peaks in the Raman spectrum using a dictionary that combines dynamic and static elements, thus improving the denoising effect. It minimizes the data requirements for dictionary training, thereby avoiding the need to build a database for machine learning algorithms. By utilizing the characteristics of actual Raman feature peaks, it combines static and dynamic dictionaries, thereby overcoming the limitations of dictionary construction in existing sparse decomposition algorithms. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0049] Figure 1 The attached figure is a schematic diagram of a Raman spectroscopy denoising method based on adaptive sparse decomposition provided by the present invention;

[0050] Figure 2 The attached figure is a schematic diagram of the sparse decomposition method in this invention;

[0051] Figure 3 The attached figure is a schematic diagram of the static dictionary construction method in this invention;

[0052] Figure 4 The attached figure is a schematic diagram of the dynamic dictionary construction method in this invention;

[0053] Figure 5 The attached figure is a schematic diagram of the dictionary construction method in this invention;

[0054] Figure 6 The attached figure shows a comparison of the noise reduction effects of simulated sulfur Raman spectra;

[0055] Figure 7 The attached figure shows a comparison of the noise reduction effects of Raman spectroscopy for acetaminophen. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] Example 1

[0058] This invention discloses a Raman spectroscopy denoising method based on adaptive sparse decomposition, comprising the following steps:

[0059] S1: Constructing a static dictionary based on real Raman spectra; Traditional sparse decomposition algorithms often use fixed dictionaries, which are unsuitable for fitting actual Raman signals, resulting in suboptimal denoising results. Therefore, this invention utilizes the characteristic peaks identified from real Raman spectra to construct a static dictionary.

[0060] In one embodiment, the steps for constructing a static dictionary are as follows:

[0061] S11: Obtain the actual Raman spectrum from the spectral database.

[0062] S12: Preprocess the real Raman spectrum; calculate the average spectrum of the real Raman spectrum to obtain the mean data, remove randomness, and then perform baseline correction on the mean data. The correction can be achieved by using the intelligent fluorescence subtraction algorithm (baselineWavelet). This algorithm is based on continuous wavelet transform and penalized least squares, which can subtract as much background baseline as possible while minimizing the influence on the characteristic peaks.

[0063] S13: Identify characteristic peaks in the preprocessed spectrum; use an iterative peak-filling algorithm to detect characteristic peaks in the spectrum according to the set requirements, such as: peaks with less than a certain intensity are not considered peaks, and peaks that are too flat are considered background noise. During the detection process, automatically find the highest point of the spectral residual and add a new peak at the highest point. Then optimize the baseline parameters, the new peak and the parameters of all added peaks at the same time. The boundary conditions are judged based on the derivative of the spectrum. Iterate until the convergence condition is met.

[0064] In one embodiment, due to limitations in acquisition conditions, the acquired spectral characteristic peaks may not conform to the shape of standard characteristic peaks. Therefore, it is necessary to perform peak shape fitting on the original characteristic peaks, and use the fitted peak shape as the final characteristic peak identification result. Specifically, the Voigt function is used to perform peak shape fitting on the identified characteristic peaks.

[0065] S14: Based on the standard Raman spectral peaks, the identified characteristic peaks are screened to obtain the first characteristic peak corresponding to each substance. Specifically, by consulting relevant literature, the positions of the standard Raman spectral peaks for each substance are obtained. Based on this, impurity peaks that do not belong to that substance are removed, and the remaining peaks are the first characteristic peaks of that substance, which are then added to the standard Raman spectral dictionary.

[0066] S2: Obtain the Raman spectrum to be processed, and traverse the spectra of each substance in the static dictionary to perform similarity calculations, and search for the first characteristic peak with the highest similarity that meets the preset conditions.

[0067] S3: When the similarity with the spectra of each substance does not meet the preset conditions, a dynamic dictionary is constructed based on the Raman spectrum to be processed, and a second characteristic peak is generated.

[0068] In one embodiment, before constructing the dictionary, existing spectral data is first used to build a database and create an ID table for subsequent determination of whether a sample belongs to a substance in the static dictionary. During the traversal, after sampling the Raman spectrum to be processed, similarity calculations are performed with the spectral data of each ID to detect whether it belongs to a substance in the static dictionary. If it does, matching characteristic peaks from the static dictionary are selected for sparse decomposition; otherwise, a dynamic dictionary is constructed, characteristic peaks are generated, and sparse decomposition is performed.

[0069] S4: Perform sparse decomposition on the Raman spectrum to be processed. By querying a static or dynamic dictionary, select the matching first or second characteristic peak in the Raman spectrum to be processed, and correct the Raman spectrum to be processed to obtain a denoised spectrum.

[0070] In one embodiment, the Orthogonal Matching Pursuit (OMP) algorithm is used for sparse decomposition to reconstruct the Raman spectrum. The peak dictionary D required for the reconstruction process is composed of both a static dictionary and a dynamic dictionary.

[0071] Specifically, the OMP algorithm decomposes an input signal into a sparse signal, i.e., A = D·x, where the input signal A and the spectral peak dictionary D are known, and the sparse vector x is calculated. From linear algebra, we know that A is a linear combination of the column vectors of D, meaning that x is the weighted sum of the column vectors of D. The algorithm aims to find the column vector that contributes the most to A, then the column vectors with the next largest contribution, and so on, until it finds the column vector with the smallest contribution. However, the final sparse vector x is not the ideal result calculated by the formula, but rather an approximation. Based on this objective, the main flow of the algorithm is as follows:

[0072] The algorithm takes a dictionary matrix D, an input signal A, and a sparsity K (not greater than the number of peaks in the spectrum) as input; the algorithm outputs a value that approximates the ideal sparse vector x.

[0073] First, initialize the algorithm by setting the residual f0 = A and the index set. Counter t = 0, where t represents the iteration number, and then steps S41-S45 are executed repeatedly:

[0074] S41: In the t-th iteration, the residual changes from f0 to f t Find the residual f t and columns of dictionary matrix D The subscript r corresponding to the maximum value in the inner product t ,Right now

[0075]

[0076] S42: Update index set θ t =θ t-1 Ur t Establish a spectral reconstruction set

[0077] S43: Obtained by least squares method x t This refers to the vector that approximates the ideal sparse vector calculated in round t. Furthermore, when using a static dictionary, x... t Each weight in the dictionary must be less than the upper limit of the weights output by the previous static dictionary.

[0078] S44: Update residuals t = t + 1

[0079] S45: Determine if the iteration termination condition is met: t > K, i.e., the number of iterations is greater than the preset sparsity. If met, the iteration ends; output the approximate result of the ideal sparse vector calculated in the last round. Conversely, the loop continues. Finally, the resulting sparse vector... The reconstructed spectrum can be obtained by performing matrix multiplication with dictionary D.

[0080] In another embodiment, S13 further includes:

[0081] After identifying the characteristic peaks, the absolute intensity of each spectral characteristic peak is recorded and normalized, and the resulting weight is used as the upper limit of the weight. In S4, during sparse decomposition, after selecting the matching first characteristic peak in the Raman spectrum to be processed and calculating its weight, the upper limit of the weight is used for constraint. Through this process, OMP can ignore the background noise introduced by the least squares weight calculation process when calculating the weight, thus obtaining a more accurate Raman spectrum.

[0082] The effects of the embodiments are explained using experimental data:

[0083] First, the denoising effect achieved in this embodiment is compared with other existing industry algorithms.

[0084] In particular, traditional polynomial fitting (PF) denoising methods, machine learning denoising methods utilizing artificial neural networks (ANNs), and sparse decomposition methods based on Gaussian dictionaries were used as comparison methods. Four algorithms were implemented to preprocess simulated sulfur Raman spectra. Cosine similarity was used to evaluate and compare the denoising effects of the proposed methods on Raman spectrum preprocessing.

[0085] Simulated sulfur signal processing: The original Raman peak is constructed using the acquired sulfur data. The acquired sulfur data undergoes baseline removal and smoothing filtering, and the signal between 70 cm⁻¹ and 1100 cm⁻¹ is extracted as the original Raman peak signal. The background noise signal is used to construct a smooth background baseline using trigonometric functions, and the random noise signal is generated using GNN training. Finally, the simulated sulfur Raman spectrum is synthesized.

[0086] Five signal processing methods were then selected: preprocessing based on static dictionary construction, preprocessing based on dynamic dictionary construction, polynomial fitting preprocessing, ANN preprocessing, and preprocessing based on a fixed Gaussian dictionary. These methods are then presented in [the document / platform / etc.]. Figure 5 middle.

[0087] analyze Figure 5It can be seen that the static dictionary performs better than the dynamic dictionary. The peak shape produced by the static dictionary is perfect because a lot of random noise is artificially added to the synthesized spectrum. The dynamic dictionary is constructed based on the sample spectrum, so if the sample spectrum contains a lot of noise, it will inevitably affect the construction of the dynamic dictionary. On the other hand, the static dictionary is constructed based on the acquired offline data, which contains much less noise than the real-time acquired samples, so the constructed dictionary is more accurate. In other words, when the random noise is relatively small, the processing effect of the dynamic dictionary and the static dictionary is not significantly different.

[0088] The other three comparison algorithms, including polynomial fitting, ANN preprocessing (two Raman denoising algorithms), and the fixed dictionary method commonly used in sparse decomposition, are significantly worse than static and dynamic dictionary methods. Polynomial fitting relies on chemometrics principles; although simple, fast, and highly interpretable, its performance is greatly affected by high noise levels. The processed spectrum in the figure not only contains a lot of noise but also loses several spectral peaks.

[0089] ANN-based preprocessing algorithms are overly reliant on the amount of data. While they perform well if the training data encompasses various real-world scenarios, it's difficult to capture noise and other influencing factors when building the training dataset. This makes them particularly vulnerable when processing spectra that haven't been previously trained on. Figure 5 It also retains a lot of random noise.

[0090] The idea behind the sparse decomposition method based on a fixed Gaussian dictionary is similar to that of the algorithm in this case. However, a Gaussian dictionary with a fixed position is difficult to retain all the spectral peaks in the spectrum, resulting in peak loss. Moreover, the peak shape is also greatly affected, and the peak half-width in the figure is significantly narrowed.

[0091] The original data and the synthesized analog signal were subjected to parameter calculations to compare the processing effects of five algorithms. Specific parameters included: signal-to-noise ratio (SNR), root mean square error (RMSE), cosine similarity, Pearson correlation coefficient, and average X-axis offset. Lower RMSE and average X-axis offset were preferred, while higher values ​​for the other three parameters were preferred. The parameters of the original data and the results after processing by the five preprocessing algorithms were calculated and compared with the original parameters to illustrate the processing effects, as shown in Table 1.

[0092] Table 1 Preprocessing parameters for simulated sulfur Raman spectra

[0093]

[0094]

[0095] As can be seen from the five processing parameters in Table 1, the denoising effects of preprocessing based on static dictionary construction and preprocessing based on dynamic dictionary construction are quite similar. The processing effect of dynamic dictionary is weaker because the addition of relatively large random noise to the spectrum affects the construction of dynamic dictionary. Static dictionary construction, on the other hand, does not depend on the current signal, so even if the random noise in the current detected signal is large, it will not affect the processing effect of static dictionary. When the random noise is relatively small, the processing effect of dynamic dictionary is also greatly improved. The processing effects of the other algorithms are much worse. However, in summary, the processing effects of static dictionary OMP algorithm and dynamic dictionary OMP algorithm are very similar. Therefore, for the sake of simplifying the experiment, the static dictionary OMP algorithm will be used as the representative algorithm, referred to as the OMP algorithm.

[0096] II. Comparison of Actual Spectral Processing Effects

[0097] The four methods described above were used to process acetaminophen data obtained using a portable Raman instrument (Portman 785). The preprocessed spectra are shown, highlighting the method's ability to denoise Raman spectra containing multiple background noises, multiple random noises, multiple characteristic peaks, and overlapping peaks compared to other methods.

[0098] The Raman spectrum of acetaminophen is used as an example because it has the most peaks and the most complex spectral form. The processing effect is as follows: Figure 7 As shown.

[0099] The preprocessing effect based on artificial neural networks is mainly affected by the training set data. The more accurate and extensive the training data, the better the network will be in removing the background baseline. However, the construction of the training set requires a lot of time to collect data, and ideal Raman spectroscopy training data is difficult to obtain through ordinary experimental instruments, which places high demands on the instruments. Therefore, the effect of machine learning Raman spectroscopy preprocessing is still limited most of the time and it is difficult to effectively remove all background baselines and random noise.

[0100] The traditional preprocessing algorithm based on chemometrics uses a polynomial fitting method to estimate the background baseline. This traditional preprocessing algorithm only performs statistical analysis on the original spectral data, fits a regression curve, and considers it as the background baseline of the spectrum for subtraction. It is an open-loop process with poor robustness. The removed noise data contains many characteristic peaks, and the feature preservation effect is not good.

[0101] Then, in Figure 7In previous experiments, the denoising results of the sparse decomposition method based on Gaussian dictionaries showed some distortion, which is due to the same reasons observed in experiments processing analog signals. Furthermore, the signal reconstruction results deviated significantly from the actual spectral peaks, which is inferior to the algorithm proposed in this paper. The dictionary construction method of the algorithm proposed in this paper perfectly adapts to the actual signal and does not remove useful signals during the denoising process.

[0102] The sparse decomposition algorithm based on dynamic and static dictionaries proposed in this case completely removes background baselines and random noise from the processed spectrum, leaving only standard Raman characteristic peaks conforming to the Voigt function distribution. The algorithm achieves near-perfect preprocessing results by using only a portion of the data to form a training set and generate a peak dictionary. The removed noise data is primarily background baselines, random noise, and clutter peaks, while retaining useful characteristic peaks. The algorithm demonstrates good baseline removal and random noise removal performance, effectively removing almost all background baselines and random noise. In contrast, the sparse decomposition algorithm based on dynamic dictionaries, lacking specific peak positions, does not remove all clutter peaks.

[0103] Example 2

[0104] This invention provides a substance identification method based on Raman spectroscopy, comprising the following steps:

[0105] Obtain Raman spectra carrying substance category information; denoise the Raman spectra using the denoising method described in Example 1 above; input the denoised Raman spectra into a trained neural network model to output the substance category.

[0106] The primary applications of Raman spectroscopy are substance identification and concentration estimation of mixture components. Raman spectroscopy is closely related to the interaction between light and the chemical bonds within a substance. Each Raman characteristic peak in the spectrum represents the wavelength and intensity of Raman scattered light, which is directly related to the chemical bonds present in the substance. Therefore, unknown sample substances can be identified by searching for matching spectra in a Raman spectroscopy database. The intensity of a Raman peak is positively correlated with the concentration of the corresponding substance in a mixture. Therefore, the concentration of components in a mixture can be estimated by searching for matching spectra in a mixture database.

[0107] This study uses methanol, ethanol, and propanol as classification data, totaling 180 data points. Since these are all alcohols, their chemical bonds are mostly similar, meaning their Raman spectra are very similar, differing only in the number of carbon-hydrogen bonds, resulting in slight variations in the number of Raman peaks. This poses a significant challenge to the classification problem; therefore, this data was used to demonstrate the classification effect. Furthermore, this study employs Random Forest (RF) and Support Vector Machine (SVM) networks to classify substances and predict the concentration of mixtures. Finally, the network parameters were adjusted multiple times, and multiple classification results were calculated to avoid randomness. In this part, only the sparse decomposition method based on a dynamic dictionary, abbreviated as ASDD, was used. This is because the two experiments above show that static dictionary processing performs better than dynamic dictionary processing; therefore, for ease of comparison of algorithm capabilities, only the sparse decomposition method based on a dynamic dictionary is used for comparison.

[0108] The raw spectra, comprising 2048 data points, were fed into a trained convolutional neural network for feature extraction. Each spectrum was converted into an array with 64 feature values ​​to reduce computation. A small amount of data was selected to train a classification network, using material categories as labels. The resulting two trained classification networks were then capable of efficiently classifying materials.

[0109] In the experimental results, the present invention achieved the highest classification accuracy, with both networks achieving over 95%. The second highest accuracy was achieved by the traditional preprocessing method, approaching 93%, but its stability was significantly lower, and its accuracy decreased further with more complex Raman spectra. The lowest accuracy was achieved by the machine learning method based on artificial neural networks, which was unstable and inaccurate, with both networks achieving below 80%. The lower accuracy compared to the traditional preprocessing method, due to the limited training data, indicates that a good denoising artificial neural network requires substantial training data. Furthermore, the sparse decomposition method based on Gaussian dictionaries is very similar in principle to the algorithm presented in this paper, both incorporating the idea of ​​sparse decomposition. However, in the classification experiments, using a fixed Gaussian dictionary proved difficult to adapt to the spectral reconstruction of three substances. Consequently, its classification accuracy was significantly lower than the algorithm presented in this paper, with both networks achieving below 85%.

[0110] Example 3

[0111] This invention provides a method for determining substance concentration based on Raman spectroscopy, comprising the following steps:

[0112] Obtain Raman spectra carrying mixture concentration information; denoise the Raman spectra using the denoising method described in Example 1;

[0113] The denoised Raman spectrum is input into the trained neural network model, which outputs the concentration of each substance in the mixture.

[0114] In this study, an experiment involving the prediction of mixture concentrations was conducted using the same three methods. Specifically, the mixture consisted of C1 and C7 compounds in a specific ratio, and its spectra were collected using scientific Raman spectroscopy. The experiment involved the collection of 959 sets of spectra, obtained by varying the mixing ratio and the machine acquisition power. Of these, 899 sets were used as training data, and the remaining 60 sets were used as test data.

[0115] In the first step, the raw spectral data is preprocessed using three different methods before being fed into a convolutional neural network specifically trained for classification experiments, thereby facilitating feature extraction and minimizing computational requirements. Following this, the dataset is divided into 899 samples for training and 60 samples for testing, with the C1 compound concentration designated as the label for training the random forest, resulting in a trained prediction network. Finally, the test data is fed into the prediction network to generate predictions of the C1 compound concentration.

[0116] In the experimental results, the prediction accuracy between the predicted results and the actual data was calculated. Both traditional preprocessing algorithms and machine learning-based preprocessing algorithms produced considerable errors. While some predictions were accurate, most showed significant discrepancies with the actual data, highlighting the challenge of stable and effective preprocessing of the original mixture spectra. Notably, the relationship between substance concentration and characteristic peak intensity in the Raman spectra could not be observed from the preprocessed results. Therefore, it is difficult to apply this method to predict the concentration of mixtures.

[0117] In 60 experiments, it was observed that the sparse decomposition method using Gaussian dictionaries yielded significantly worse results, exhibiting alarmingly low prediction accuracy and substantial prediction bias. The concentration prediction accuracy of this invention is highly satisfactory. This invention preserves effective characteristic peaks while retaining the intensity of characteristic peaks positively correlated with substance concentration.

[0118] Experimental results clearly demonstrate that the present invention exhibits very promising denoising effects and robustness, indicating its suitability for concentration prediction and material classification systems in portable Raman spectrometers.

[0119] Compared with other methods in the industry, the dictionary construction method of this invention has the following innovative optimizations:

[0120] First, by applying a stoichiometric feature extraction method, a dictionary including spectral characteristic peaks is built from the input spectrum. Compared with traditional dictionaries, this method is better aligned with the original data.

[0121] The industry often uses fixed dictionaries for construction, such as Gaussian signal dictionaries or other wavelet signal dictionaries. The signals in these dictionaries are often generated in a fixed manner, focusing only on the generated signals without a deep understanding of the characteristics of Raman spectral signals. Therefore, the spectral peaks of these signals often differ significantly from those of Raman spectra, and the shape of the spectral peaks is different from that of the real Raman signals. This will cause signal loss to the spectral peaks and change the parameters of the spectral peaks, such as peak intensity and peak half width.

[0122] The static and dynamic dictionaries in this invention are constructed based on real Raman signals, rapidly adapting to new substances using sample Raman spectral data. First, the Baselinewavelet algorithm, a chemometrics-based algorithm, is used for preliminary denoising, removing background and random noise. Using chemometrics for preliminary denoising leverages the algorithm's strong interpretability, allowing for immediate handling of denoising anomalies. However, the initially denoised spectrum still retains many scattered noises. Next, an iterative peak-filling algorithm is employed to detect characteristic peaks in the spectrum according to set requirements. The Voigt function is then used to fit these characteristic peaks, ensuring the peak shapes conform to the ideal Raman spectrum. This process results in highly standardized peaks in the resulting dictionary with low signal loss, allowing for significant preservation of characteristic peak information after subsequent denoising. Finally, an orthogonal matching pursuit algorithm is used to sparsely decompose the Raman spectrum into the constructed dictionary, effectively eliminating various random and background noises in the Raman spectrum.

[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0124] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A Raman spectral denoising method based on adaptive sparse decomposition, characterized in that, Includes the following steps: A static dictionary is constructed based on real Raman spectra. The Raman spectrum to be processed is obtained, and the spectra of each substance in the static dictionary are traversed to calculate the similarity. The first characteristic peak with the highest similarity and that meets the preset conditions is then searched. When the similarity with the spectra of each substance does not meet the preset conditions, a dynamic dictionary is constructed based on the Raman spectrum to be processed, and a second characteristic peak is generated. The orthogonal matching pursuit algorithm is used to perform sparse decomposition on the Raman spectrum to be processed. By querying a static or dynamic dictionary, the matching first or second characteristic peak is selected in the Raman spectrum to be processed. The Raman spectrum to be processed is corrected according to the matching degree, and the denoised spectrum is obtained.

2. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 1, characterized in that, The steps for constructing a static dictionary based on real historical Raman spectra include: Obtain the actual Raman spectrum from the spectral database; The actual Raman spectrum is preprocessed; Identify characteristic peaks in the preprocessed spectrum; The first characteristic peaks corresponding to each substance are obtained by screening the identified characteristic peaks based on the standard Raman spectrum peaks.

3. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 2, characterized in that, The preprocessing steps include: Calculate the average spectrum of the actual Raman spectrum to obtain the mean data; The fluorescence subtraction algorithm was used to perform baseline correction on the mean data.

4. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 2, characterized in that, After identifying the characteristic peaks, the steps also include: The Voigt function is used to fit the peak shape of the identified characteristic peaks.

5. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 2, characterized in that, Characteristic peaks are identified in the preprocessed spectrum, specifically as follows: An iterative peak-filling algorithm is used to calculate the spectral residual and fill the peak at the position of the highest residual value; Calculate the loss and perform iterative parameter optimization; The process is iterated until the preset convergence condition is met, and the characteristic peak parameters are obtained.

6. A Raman spectral denoising method based on adaptive sparse decomposition according to claim 2 or 4, characterized in that, The process of filtering the identified characteristic peaks based on standard Raman spectral peaks specifically involves: Based on the position of the standard Raman spectrum peak, characteristic peaks with corresponding positional relationships are selected from the identified characteristic peaks as the first characteristic peak.

7. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 1, characterized in that, Constructing a dynamic dictionary based on the Raman spectrum to be processed includes the following steps: Baseline correction is performed on the Raman spectrum to be processed; An iterative peak-compensation algorithm is used to identify and optimize characteristic peaks. The optimized feature peaks are fitted using the Voigt function to obtain the fitted peak shape, which is used as the second feature peak in the dynamic dictionary.

8. The Raman spectral denoising method based on adaptive sparse decomposition according to claim 2, characterized in that, The steps for constructing a static dictionary based on real historical Raman spectra also include: After identifying the characteristic peaks, the absolute intensity of each spectral characteristic peak is recorded and normalized, and the resulting weight is used as the upper limit of the weight. During sparse decomposition, after selecting the matching first characteristic peak in the Raman spectrum to be processed and calculating the weight, the weight is constrained by the upper limit of the weight.

9. A substance identification method based on Raman spectroscopy, characterized in that, Includes the following steps: Obtain Raman spectra that carry information about the type of substance; The Raman spectrum is denoised using any of the adaptive sparse decomposition-based Raman spectral denoising methods described in claims 1-8; The denoised Raman spectrum is input into the trained neural network model, which outputs the substance category.

10. A method for determining substance concentration based on Raman spectroscopy, characterized in that, Includes the following steps: Obtain Raman spectra that carry information about the concentration of the mixture; The Raman spectrum is denoised using any of the adaptive sparse decomposition-based Raman spectroscopy denoising methods described in claims 1-8; The denoised Raman spectrum is input into the trained neural network model, which outputs the concentration of each substance in the mixture.

Images