Raman spectrum noise reduction method based on MSC-U-Net algorithm

Abstract

The invention discloses a Raman spectrum noise reduction method based on an MSC-U-Net algorithm, and the method comprises the following steps: S1, measuring Raman spectrums of a plurality of substances with different concentrations under the same measurement condition and environment, and respectively measuring a plurality of spectrums for each concentration of each substance to obtain an original spectrum data set containing noise; s2, performing normalization preprocessing on all the collected spectrums, dividing a data set into a training set, a verification set and a test set, and constructing a data set required by a noise reduction model; s3, averaging the collected original spectrums according to a plurality of spectrums of the same substance and the same concentration to obtain ideal spectrums without noise; and S4, constructing an MSC-U-Net model, inputting the training set into the model to carry out parameter learning, and carrying out iterative optimization according to a loss function to obtain a spectral noise reduction model. Noise signals including singular peaks in the Raman spectrum can be effectively removed, meanwhile, effective signals are not lost, the signal-to-noise ratio of the Raman spectrum is increased, and a foundation is laid for follow-up spectrum quantitative or qualitative analysis.

Description

Technical Field

[0001] The present invention relates to the field of Raman spectrum denoising, and in particular to a Raman spectrum denoising method based on an MSC-U-Net algorithm. Background Art

[0002] Due to interference from factors such as instruments, environment, and cosmic rays, the measured Raman spectra contain certain noise, which will interfere with the subsequent quantitative or qualitative analysis of the Raman spectrum. Therefore, it is necessary to perform noise reduction processing on the spectrum.

[0003] 1. Traditional noise reduction methods

[0004] Commonly used statistical-based spectral denoising methods include SG filtering, WT smoothing, sliding average method, etc. In patent CN117574063A, the spectrum data of the spectrum is obtained by using discrete Fourier transform, and the effective signal is extracted from it, the noise signal is removed, and then the effective signal is restored by Fourier transform to obtain the denoised light. In patent CN111289106, by designing the frequency-amplitude response function PF of the spectral signal S, the PF is transformed to obtain the corresponding filter convolution transfer function H, and the convolution Y of the spectrum and H is calculated, and the data of the same length as the spectrum S is intercepted to obtain the denoised spectrum data. In patent CN110208211A, the EEMD method is used to decompose the signal of the Hongpu, and several IMF components are obtained according to the IMF components. The IMF components containing noise are selected, the selected noise components are smoothed, and the processed signal is reconstructed to obtain the denoised spectrum.

[0005] 2. Machine Learning

[0006] In patent CN111678599A, the Adam algorithm neural network topology model is established, the optimal parameter combination of SG filtering is selected, and the SG filtering algorithm is used according to the optimal parameters to perform adaptive denoising on the spectrum. In patent CN110135320A, a spectrum denoising method and device based on BP neural network and variance compensation is disclosed, and the BP neural network algorithm is used to obtain the most effective filter parameter combination to denoise the spectrum.

[0007] In the process of measuring Raman spectra, due to the influence of environment, instruments, cosmic rays, etc., the noise present in the spectrum includes periodic noise and randomly distributed noise. The traditional spectrum method only has a good noise reduction effect on periodic noise signals, and its performance in removing random noise is average. The noise reduction method based on polynomial fitting is very sensitive to the selection of parameters. The choice of parameters has a great influence on the results, and some effective signals will be lost while reducing the noise. Some network spectrum noise reduction models are trained with artificial synthetic data, which makes a big difference in actual applications.

[0008] Therefore, it is necessary to provide a Raman spectrum denoising method based on the U-Net neural network algorithm, which can effectively remove the noise signals in the Raman spectrum, including the singular peaks that are difficult to remove by traditional methods, so as to improve the signal-to-noise ratio of the Raman spectrum and lay the foundation for subsequent quantitative or qualitative analysis of the spectrum. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a Raman spectrum denoising method based on the MSC-U-Net algorithm, which can effectively remove the noise signal in the Raman spectrum to improve the signal-to-noise ratio of the Raman spectrum.

[0010] To solve the above technical problems, the present invention provides a Raman spectrum denoising method based on the MSC-U-Net algorithm, comprising the following steps: S1: under the same measurement conditions and environment, measuring Raman spectra of different concentrations of multiple substances, measuring multiple spectra for each concentration of each substance, and obtaining an original spectrum data set containing noise; S2: performing normalization preprocessing on all collected spectra, dividing the data set into a training set, a validation set and a test set, and constructing a data set required for a denoising model; S3: averaging the collected original spectra according to multiple spectra of the same substance and the same concentration, to obtain an ideal spectrum without noise; S4: constructing an MSC-U-Net model, inputting the training set into the model for parameter learning, and iteratively optimizing according to a loss function to obtain a spectrum denoising model.

[0011] Furthermore, the step S4 includes the following steps: S41: MSC processing: according to the same substance and the same concentration, the training set and the validation set spectra are subjected to MSC processing; S42: downsampling: convolution layer, pooling layer, and BN layer are alternately formed, wherein the convolution kernel size of the convolution layer is 1×3, padding processing is used, the BN layer standardizes the output data, the pooling layer adopts 1×2 maximum pooling, and the activation function adopts ReLU function; S43: upsampling: composed of convolution layer and deconvolution layer, the upsampling adopts 1×2 size convolution kernel for deconvolution operation, and is connected with the corresponding downsampling feature map; the convolution layer also uses 1×3 convolution kernel for convolution operation, padding processing is used, the BN layer standardizes the output data, and the activation function adopts ReLU function; S44: the output layer is composed of two convolution layers, the convolution kernel size is 1×1, and the activation function is ReLU function.

[0012] Furthermore, the ReLU function in step S4 is as follows:

[0013] f(x)=max(0,x).

[0014] Furthermore, the loss function calculation formula in step S4 is as follows:

[0015] .

[0017] Furthermore, in step S1, Raman spectra of at least 6 different substances at at least 5 different concentrations are measured, and at least 7 spectra are measured for each concentration of each substance.

[0018] Furthermore, in step S2, the ratio of the training set, the validation set and the test set is 8:1:1 or 6:2:2.

[0019] Furthermore, the optimization strategy adopted in step S4 is the Adam algorithm, and the initial learning rate of the parameter is 1×10 -2 -1×10 -3 The number of iterative updates is determined according to the validation set loss function value, and the iteration ends when the validation set loss function value no longer decreases.

[0020] Furthermore, the method further includes step S5: using the trained network model to perform denoising on the test set data that has not participated in the training, and calculating the MSE with the ideal spectrum to verify the denoising ability of the model. The MSE is less than 10 -5 This indicates that the data denoising effect is ideal. If the denoising effect is poor, return to step S1 to increase the data set and continue to optimize the model.

[0021] Compared with the prior art, the present invention has the following advantages: the Raman spectrum denoising method based on the MSC-U-Net algorithm provided by the present invention performs normalization preprocessing on all collected spectra and divides them into training set, verification set and test set, and then averages the collected original spectra according to multiple spectra of the same substance and the same concentration to obtain an ideal spectrum without noise; finally, the MSC-U-Net model is constructed, the training set is input into the model for parameter learning, and the spectrum denoising model is obtained according to iterative optimization of the loss function. The noise signal in the Raman spectrum can be effectively removed, including the removal of singular peaks that are difficult to remove by traditional methods, without losing effective signals, thereby improving the signal-to-noise ratio of the Raman spectrum and laying a foundation for subsequent quantitative or qualitative analysis of the spectrum. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Schematic diagram of the process of the Raman spectrum denoising method based on the MSC-U-Net algorithm of the present invention;

[0023] Figure 2 Schematic diagram of the process of the Raman spectrum denoising method based on the MSC-U-Net algorithm in an embodiment of the present invention;

[0024] Figure 3 It is the original collection spectrum in the embodiment of the present invention;

[0025] Figure 4is the ideal spectrum in the embodiment of the present invention;

[0026] Figure 5 This is a U-Net model diagram in an embodiment of the present invention;

[0027] Figure 6 The spectrum after the original spectrum preprocessing in the embodiment of the present invention;

[0028] Figure 7 The graph after the training set u-net denoising process in the embodiment of the present invention;

[0029] Figure 8 This is the graph after the u-net denoising process of the test set in the embodiment of the present invention;

[0030] Fig. 9 is the original spectrum of the unfamiliar data set in the embodiment of the present invention;

[0031] Fig.10 for Fig. 9 The graph of unfamiliar data after denoising by MSC-U-Net;

[0032] Fig.11 It is a spectrum obtained by SG algorithm parameters with window size 5 and smoothing order 3 after denoising in the prior art;

[0033] Fig.12 This is a spectrum obtained by SG algorithm parameters window size 7 and smoothing order 5 after denoising the spectral data in the prior art. DETAILED DESCRIPTION

[0034] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0035] See also Figure 1 The present invention provides a Raman spectrum denoising method based on a U-Net neural network algorithm, comprising the following steps:

[0036] Step S1: Under the same measurement conditions and environment, measure the Raman spectra of multiple substances at different concentrations, measure multiple spectra for each concentration of each substance, and obtain a raw spectral data set containing noise. The more substances, the better, no less than 6, the concentration range is not limited, each substance has at least 5 different concentrations, and each substance has at least 7 spectra for each concentration.

[0037] Step S2: Perform normalization preprocessing on the collected spectra, divide the data set into a training set, a validation set, and a test set, and construct the data set required for the noise reduction model. Furthermore, the ratio of the training set to the validation set and the test set is preferably 8:1:1, or 6:2:2, and can also be divided according to the specific situation of the data set.

[0038] Step S3: average the collected original spectra according to the same substance and the same concentration to obtain an ideal spectrum without noise.

[0039] Step S4: construct the MSC-U-Net model, input the training set into the model for parameter learning, and iteratively optimize according to the loss function to obtain the spectral denoising model; the optimization strategy adopted is the Adam algorithm, and the initial learning rate of the parameters is 1×10 -2 -1×10 -3 The number of iterations is determined according to the validation set loss function value, and the iteration ends when the validation set loss function value no longer decreases. Further, the following steps are included:

[0040] Step S41: MSC (Multiple Scatter Correction) processing: according to the same substance and the same concentration, the training set spectrum is subjected to MSC processing;

[0041] Step S42: downsampling: convolution layer, pooling layer, and BN layer are alternately formed, wherein the convolution kernel size of the convolution layer is 1×3, padding processing (inner margin) is used, the BN layer standardizes the output data, the pooling layer adopts 1×2 maximum pooling, and the activation function adopts ReLU function (Rectified Linear Unit);

[0042] Step S43: Upsampling: It is composed of convolutional layer, deconvolutional layer and BN layer alternately. The upsampling layer uses 1×2 convolutional kernel for deconvolution operation and is connected with the corresponding downsampling feature map. The convolutional layer also uses 1×3 convolutional kernel for convolution operation and padding processing. The BN layer standardizes the output data and the activation function uses ReLU function.

[0043] Step S44: the output layer consists of two convolutional layers, the convolution kernel size is 1×1, and the activation function is ReLU;

[0044] Furthermore, in steps S42 and S43, the ReLU function formula is as follows:

[0045] f(x)=max(0,x)

[0046] Furthermore, in step S4, the loss function calculation formula is as follows:

[0047]

[0048] Step S5: Use the trained network model to perform denoising on the test set data that has not participated in the training to verify the denoising ability of the model. Calculate the MSE with the ideal spectrum to verify the denoising ability of the model. The MSE is less than 10 -5This indicates that the data denoising effect is ideal. If the MSE is greater than or equal to 10 -5 , indicating that the noise reduction effect is poor, then return to step S1 to increase the data set and continue to optimize the model until the MSE is less than 10 -5 .

[0049] In a specific embodiment, see Figure 2 , the Raman spectrum denoising method based on the U-Net neural network algorithm specifically includes the following steps:

[0050] First, obtain the data set required by the U-Net neural network algorithm. The present invention uses the U-Net algorithm to remove noise signals in the Raman spectrum. It is necessary to obtain Raman spectra of different concentrations of multiple substances. Multiple spectra are measured for each concentration of each substance to obtain the original spectral data set containing noise. In this embodiment, under the same measurement conditions and environment, 10 Raman spectra of 7 different substances with 5 different concentrations were measured, and a total of 350 Raman spectra were measured. The original acquisition spectrum obtained is as follows: Figure 3 shown.

[0051] Next, all collected spectra were subjected to Max-Min normalization preprocessing operations to ensure that model training would not be affected by intensity scale differences. The data were randomly divided into training set, validation set, and test set. The data ratio of training set, validation set, and test set was 8:1:1.

[0052] Then, according to the substance type and concentration, every 10 spectra are averaged to obtain the ideal spectrum, such as Figure 4 shown.

[0053] Then, the MSC-U-Net denoising model is constructed.

[0054] The spectral data are first processed by MSC according to the substances and their concentrations, with the aim of reducing the difference in spectral intensity of the same substance at the same concentration and preventing the influence of the model on the effective Raman signal.

[0055] U-Net is a currently popular convolutional neural network (CNN) structure proposed by Olaf Ronneberger et al. in 2015. It is mainly used for image segmentation tasks.

[0056] U-Net combines downsampling and upsampling to form a U-shaped symmetrical network structure. Downsampling consists of multiple convolutional layers and pooling layers alternating, and upsampling consists of multiple convolutional and transposed convolutional layers, which gradually restore the feature map to the original input dimension and jump-connect with the feature map of the corresponding downsampling layer, so that details and positioning information can be retained in the output.

[0057] In this embodiment, the convolution kernel size of the convolution layer during the U-Net downsampling process is 1×3. After each convolution operation, the BN layer (Batch Normalization) standardizes the data. The pooling layer uses 1×2 maximum pooling, and the activation function uses the ReLU function. The upsampling is composed of a convolution layer, a deconvolution layer, and a BN layer alternately. The upsampling uses a 1×2 convolution kernel for deconvolution and is connected to the corresponding downsampling feature map. Then, a 1×3 convolution kernel is used for convolution. After each convolution operation, the BN layer standardizes the data. The output layer consists of two convolution layers, the convolution kernel size is 1×1, and the activation function is ReLU. In this embodiment, the optimization strategy adopted is the Adam algorithm, and the initial learning rate of the parameter is 1×10 -3 , the number of iterations is 1000. The specific feature graph parameters and other structural information of the network model are as follows: Figure 5 shown.

[0058] Finally, the U-Net convolutional neural network is iterated 1000 times to update the training set denoising results of the parameters, and the validation set loss function value is no longer significantly reduced. In this embodiment, the data participating in the training model is input into the trained model for denoising results, where Figure 7 This is a graph of the training set in an embodiment of the present invention after being processed by the U-Net denoising model. Figure 8 This is a graph of the test set in an embodiment of the present invention after being processed by the U-Net denoising model. Figure 7 , 8 Compared to Figure 6 , it can be clearly seen Figure 6 The noise circled in the middle has been removed, and the denoising effect is obvious. The training set MSE is 2.2814×10 -6 , test set MSE = 1.5772×10 -6 In machine learning, mean squared error (MSE) is a common metric for measuring the difference between the model's predicted value and the true value. The smaller the MSE value, the better the model's predictive performance. The MSE value is very small, ranging from 10 -6 This indicates that the predicted value of the model is very close to the actual value, and the prediction performance of the denoising model is quite good. It can be seen that the loss of effective signals in the denoising process is very low.

[0059] This embodiment also imports new data into the U-Net denoising model for denoising processing. It can be clearly seen that all noise including singular peaks has been significantly reduced. Fig. 9 This is the original spectrum of unfamiliar data. The areas with obvious noise are marked with circles. Fig.10 for Fig. 9The spectrum of unfamiliar data after denoising by the U-Net denoising model shows that the noise has been effectively removed.

[0060] Comparative Example

[0061] The denoising method used in this embodiment is compared with the denoising effect of different parameters of the commonly used SG smoothing algorithm. It is found that the method provided by the present invention is more effective in removing noise. The SG algorithm is not sensitive to singular peaks, and many singular peak signals are still retained in the spectrum.

[0062] Fig.11 The existing SG algorithm is used with the following parameters: window size 5, smoothing order 3, and denoised spectrum. Fig.12 The existing SG algorithm is used with the following parameters: window size 7, smoothing order 5, and the spectrum after noise reduction. Fig.11 and Fig.12 It can be seen that many singular peak signals are still retained in the spectrum. It can be seen that the Raman spectrum denoising method based on the neural network algorithm provided by the present invention can more effectively remove the noise signal in the Raman spectrum.

[0063] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be based on the definition of the claims.

Claims

1. A Raman spectrum denoising method based on MSC-U-Net algorithm, characterized in that: The steps include: S1: Under the same measurement conditions and environment, the Raman spectra of various substances with different concentrations are measured. Multiple spectra are measured for each concentration of each substance to obtain the original spectral data set containing noise. S2: Perform normalization preprocessing on all collected spectra, divide the data set into training set, validation set and test set, and construct the data set required for the denoising model; S3: average the collected original spectra according to the multiple spectra of the same substance and the same concentration to obtain an ideal spectrum without noise; S4: Build the MSC-U-Net model, input the training set into the model for parameter learning, iteratively optimize according to the loss function, and obtain the spectral denoising model.

2. The Raman spectrum noise reduction method according to claim 1, characterized in that: The step S4 comprises the following steps: S41: MSC processing: According to the same substance and concentration, the training set and validation set spectra were subjected to MSC processing; S42: Downsampling: Convolutional layers, pooling layers, and BN layers are formed alternately. The convolutional kernel size of the convolutional layer is 1×3, and padding is used. The BN layer standardizes the output data. The pooling layer uses 1×2 maximum pooling, and the activation function uses the ReLU function. S43: Upsampling: It is composed of convolutional layers, deconvolutional layers, and BN layers alternately. The upsampling uses a 1×2 convolution kernel for deconvolution and is connected to the corresponding downsampled feature map. The convolution layer also uses a 1×3 convolution kernel for convolution operations, uses padding processing, the BN layer normalizes the output data, and the activation function uses the ReLU function; S44: The output layer consists of two convolutional layers, the convolution kernel size is 1×1, and the activation function is the ReLU function.

3. The Raman spectrum noise reduction method according to claim 2, characterized in that: The ReLU function in step S4 is as follows: f(x)=max(0,x).

4. The Raman spectrum noise reduction method according to claim 1, characterized in that: The loss function calculation formula in step S4 is as follows:

5. The Raman spectrum noise reduction method according to claim 1, characterized in that: In the step S1 , Raman spectra of at least 6 different substances at at least 5 different concentrations are measured, and at least 7 spectra are measured for each concentration of each substance.

6. The Raman spectrum noise reduction method according to claim 1, characterized in that: In step S2, the ratio of the training set, the validation set and the test set is 8:1:1 or 6:2:

2.

7. The Raman spectrum noise reduction method according to claim 6, characterized in that: The optimization strategy adopted in step S4 is the Adam algorithm, and the initial learning rate of the parameter is 1×10 -2 -1×10 -3 The number of iterative updates is determined according to the validation set loss function value, and the iteration ends when the validation set loss function value no longer decreases.

8. The Raman spectrum noise reduction method according to claim 7, characterized in that: The method also includes step S5: using the trained network model to perform noise reduction on the test set data that has not participated in the training, and calculating the MSE with the ideal spectrum to verify the noise reduction ability of the model. If the MSE is greater than or equal to 10 -5 , then return to step S1 to increase the data set and continue to optimize the model until the MSE is less than 10 -5 .

Images