A raman spectrum denoising method based on MSC-U-Net algorithm

Abstract

The application discloses a Raman spectrum denoising method based on an MSC-U-Net algorithm, and comprises the following steps: S1, measuring Raman spectra of different concentrations of multiple substances under the same condition, measuring multiple spectra for each concentration of each substance, and obtaining a raw spectrum data set containing noise; S2, performing normalization preprocessing on all the collected spectra, dividing the data set into a training set, a verification set and a test set, and constructing a data set required by a denoising model; S3, performing average processing on the collected raw spectra according to multiple spectra of the same concentration of the same substance, and obtaining ideal spectra without noise; and S4, constructing a Transformer-Attention U-Net model, inputting the training set into the model for parameter learning, iteratively optimizing according to a loss function, and obtaining a spectrum denoising model. The application can effectively remove noise signals in Raman spectra, does not lose effective signals, and improves the signal-to-noise ratio of Raman spectra.

Description

Technical Field

[0001] This invention relates to the field of Raman spectroscopy noise reduction, and more particularly to a Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm. Background Technology

[0002] Due to interference from factors such as instruments, environment, and cosmic rays, the measured Raman spectra all contain a certain amount of noise, which can interfere with subsequent quantitative or qualitative analysis of Raman spectra. Therefore, it is necessary to perform noise reduction processing on the spectra.

[0003] 1. Traditional noise reduction methods Commonly used statistical-based spectral denoising methods include SG filtering, WT smoothing, and moving average methods. In patent CN117574063A, the spectral data is obtained using discrete Fourier transform, and the effective signal is extracted, noise is removed, and then the effective signal is reconstructed using Fourier transform to obtain the denoised light. In patent CN111289106, the frequency-amplitude response function PF of the spectral signal S is designed, and the corresponding filter convolution transfer function H is obtained by transforming PF. The convolution Y of the spectrum and H is calculated, and data of the same length as the spectrum S is truncated to obtain the denoised spectral data. In patent CN110208211A, the EEMD method is used to decompose the Hong Kong spectral signal, obtaining several IMF components. Based on the IMF components, adaptively obtaining the signal, selecting the noisy IMF components, smoothing the selected noise components, and reconstructing the processed signal to obtain the denoised spectrum.

[0004] 2. Machine Learning In patent CN111678599A, an Adam algorithm neural network topology model is established, the optimal parameter combination for SG filtering is selected, and the SG filtering algorithm is then used to adaptively reduce the noise of the spectrum based on the optimal parameters. In patent CN110135320A, a spectral noise reduction method and apparatus based on BP neural network and variance compensation are disclosed, which uses the BP neural network algorithm to obtain the optimal filter parameter combination for spectral noise reduction.

[0005] During Raman spectroscopy measurements, noise in the spectrum can be either periodic or randomly distributed due to environmental factors, instrumentation, and cosmic rays. Traditional spectral methods are only effective at reducing periodic noise signals, with limited performance in removing random noise. Furthermore, denoising methods based on polynomial fitting are highly sensitive to parameter selection, which significantly impacts the results, and some effective signal is lost during denoising. Some network-based spectral denoising models are trained using artificially synthesized data, which leads to significant differences in practical applications.

[0006] Therefore, it is necessary to provide a Raman spectroscopy noise reduction method based on the U-Net neural network algorithm, which can effectively remove noise signals in Raman spectra, including singular peaks that are difficult to remove by traditional methods, so as to improve the signal-to-noise ratio of Raman spectra and lay the foundation for subsequent quantitative or qualitative spectral analysis. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm, which can effectively remove noise signals in Raman spectra and improve the signal-to-noise ratio of Raman spectra.

[0008] To address the aforementioned technical problems, this invention provides a Raman spectroscopy denoising method based on the MSC-U-Net algorithm, comprising the following steps: S1: Under the same measurement conditions and environment, measure the Raman spectra of various substances at different concentrations, measuring multiple spectra for each concentration of each substance to obtain a noisy original spectral dataset; S2: Perform normalization preprocessing on all acquired spectra, dividing the dataset into training, validation, and test sets to construct the dataset required for the denoising model; S3: Average the acquired original spectra according to multiple spectra of the same substance at the same concentration to obtain an ideal spectrum free of noise; S4: Construct a Transformer-Attention U-Net model, input the training set into the model for parameter learning, and iteratively optimize according to the loss function to obtain the spectral denoising model.

[0009] Further, step S4 includes the following steps: S41: MSC processing: Perform multiple scattering correction preprocessing on the spectra of the training set and the validation set according to the same substance and the same concentration to eliminate scattering / baseline differences and enhance spectral consistency; S42: Downsampling module construction: The encoder consists of several convolutional layers, batch normalization layers, activation layers, and pooling layers alternately, wherein the convolutional layers use 1×3 one-dimensional convolutional kernels and use appropriate padding to maintain local alignment, the batch normalization layers normalize the output channels one by one, and the pooling layers use 1×2 Max pooling is used, and the activation function is ReLU; S43: Upsampling and skip fusion: The decoder is composed of several upsampling-concatenating corresponding downsampling feature maps-convolutional layers-batch normalization layers-activation layers in an alternating manner. The upsampling output and the corresponding encoder features are fused through skip connections, and a cross-attention fusion mechanism based on CBAM weighted features is used at each skip connection; and a Transformer module is set at the bottleneck of the intermediate layer between the encoder and decoder; S44: Output layer: It consists of two convolutional layers, both with a kernel size of 1×1 and the activation function is ReLU.

[0010] Furthermore, the ReLU function in step S42 is as follows: .

[0011] Furthermore, in step S43, a cross-attention fusion mechanism based on CBAM weighted features is adopted at each hop connection as follows: the decoder upsampled features are used as queries, and the CBAM-weighted high-resolution encoder features are used as keys / values. The weighted response of the query to the encoder features is calculated through multi-head cross-attention. The multi-head cross-attention output is fused with the original decoder features in the form of residuals after projection. A learnable one-dimensional relative position bias is introduced in the attention calculation to enhance the modeling ability of spectral position relative relationships.

[0012] Further, step S43 specifically includes: (1) Projection and Transformation: ; Where Q represents the query vector, K represents the key vector, and V represents the value vector. This represents the projection operation, B is the batch size, U is the upsampled feature, and E is the encoder feature after CBAM. Represents the set of real numbers. , These represent the sequence lengths of the Query vector and the Key vector, respectively. The feature dimensions representing the Q, K, and V vectors; (2) Attention: ; in Represents the output of the attention mechanism. For relative position offset, This represents the dimension of the Q, K, and V vectors processed by each head; (3) Integration: ; in Represents the final output after fusion. The output projection layer is used to project the output of the attention mechanism back to a dimension that matches U. It is a learnable gating system.

[0013] Further, step S43 also includes setting a ConvToToken adapter between the encoder's convolutional layer and the Transformer module, setting a TokenToConv adapter between the decoder's convolutional layer and the Transformer module, and using LayerNorm / residual bridging to achieve stability matching between the batch normalization layer and LayerNorm; the ConvToToken adapter transposes and linearly projects the 1D convolutional feature map output by the encoder, normalizes it using LayerNorm, and then outputs it to the Transformer module; the TokenToConv adapter maps the output of the Transformer module back to the original channel through inverse linear projection and performs 1×1 convolution. The global context features output by the Transformer are reintegrated into the feature maps of the convolutional neural network, making... The features are smoothly fed into the next stage of the U-Net decoder; the Transformer reconstruction features are dynamically adjusted via a learnable gating mechanism. The injection intensity is then added to the original base feature as a residual. This is to achieve intelligent fusion of global and local information: ; ; Where g is the gate weight vector, used to control the intensity of Transformer feature injection, and σ represents Sigmoid activation. Y represents the basic features, i.e., the features before they are input to the ConvToToken adapter, and Y represents the final output features.

[0014] Furthermore, the loss function in step S4 adopts a weighted combination of MSE and peak retention, as shown in the formula: L= α ·MSE( Y pred, Y ideal)+ β ·SSIM( Y pred, Y ideal); in This is the total loss value, α is the weighting coefficient of the MSE term, and β is the weighting coefficient of the SSIM term. SSIM is the structural similarity index, used to ensure the morphological fidelity of the spectrum after noise reduction. This represents the model's predicted output. This represents the actual target value.

[0015] Furthermore, in step S2, the ratio of the training set, validation set, and test set is 8:1:1 or 6:2:2.

[0016] Furthermore, the optimization strategy employed in step S4 is the Adam algorithm, with an initial learning rate of 1×10⁻⁶. -2 -1×10 -3 The number of iterations is determined based on the value of the validation set loss function. The iteration ends when the value of the validation set loss function no longer decreases.

[0017] Furthermore, it also includes step S5: using the trained network model to denoise the test set data that was not used in training, and calculating the MSE with the ideal spectrum to verify the model's denoising ability. If the MSE is greater than or equal to 10... -5 Then return to step S1 to increase the dataset and continue optimizing the model until the MSE is less than 10. -5 .

[0018] Compared with existing technologies, this invention offers the following advantages: The Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm provided by this invention preprocesses all acquired spectra by normalization and dividing them into training, validation, and test sets. Then, it averages the acquired raw spectra according to multiple spectra of the same substance and concentration to obtain noise-free ideal spectra. Finally, it constructs an MSC-U-Net model, inputs the training set into the model for parameter learning, and iteratively optimizes the model based on the loss function to obtain the spectral noise reduction model. This effectively removes noise signals from Raman spectra, including singular peaks that are difficult to remove using traditional methods, without losing effective signals, thus improving the signal-to-noise ratio of Raman spectra and laying the foundation for subsequent quantitative or qualitative spectral analysis. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm of the present invention; Figure 2 This is a flowchart illustrating the Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm in an embodiment of the present invention. Figure 3 This is the original data acquisition map in the embodiment of the present invention; Figure 4 This is an ideal spectrum in the embodiments of the present invention; Figure 5 This is a diagram of the U-Net model in an embodiment of the present invention; Figure 6 This is the spectrum after preprocessing the original spectrum in an embodiment of the present invention; Figure 7 This is the graph of the training set after denoising using u-net in this embodiment of the invention; Figure 8This is the spectrum after denoising using u-net in the test set of this embodiment of the invention; Figure 9 This refers to the original spectrum of an unfamiliar dataset in this embodiment of the invention; Figure 10 for Figure 9 A map of unfamiliar data after noise reduction using MSC-U-Net; Figure 11 The image is a spectrum of spectral data after noise reduction processing using the SG algorithm with parameters of window size 5 and smoothing order 3, as described in the prior art. Figure 12 The image is a spectrum of spectral data after noise reduction processing using the SG algorithm with parameters of window size 7 and smoothing order 5, based on existing technology. Detailed Implementation

[0020] The present invention will now be further described with reference to the accompanying drawings and embodiments.

[0021] Please see Figure 1 The present invention provides a Raman spectroscopy noise reduction method based on the MSC-U-Net neural network algorithm, comprising the following steps: Step S1: Under the same measurement conditions and environment, measure the Raman spectra of multiple substances at different concentrations. For each concentration of each substance, measure multiple spectra to obtain a raw spectral dataset containing noise. The more substances, the better, but no less than 6. The concentration range is not limited, but each substance must have at least 5 different concentrations, and at least 7 spectra must be measured for each concentration of each substance.

[0022] Step S2: Normalize the collected spectra and preprocess them to divide the dataset into training, validation, and test sets to construct the dataset required for the noise reduction model. Furthermore, the preferred ratio of the training set to the validation and test sets is 8:1:1 or 6:2:2, but can be adjusted according to the specific dataset.

[0023] Step S3: Average the collected raw spectra according to the same substance and the same concentration to obtain an ideal spectrum without noise.

[0024] Step S4: Construct a Transformer-Attention U-Net model, input the training set into the model for parameter learning, and iteratively optimize according to the loss function to obtain the spectral denoising model; the optimization strategy used is the Adam algorithm, and the initial parameter learning rate is 1×10. -2 -1×10 -3 The number of iterations is determined based on the value of the validation set loss function. The iteration ends when the value of the validation set loss function no longer decreases.

[0025] Furthermore, it includes the following steps: Step S41: MSC processing: Perform multi-scale preprocessing such as MSC (Multiplicative Scatter Correction) on the spectra of the training set and the validation set according to "same substance, same concentration" to eliminate scattering / baseline differences and enhance spectral consistency. Step S42: Downsampling module construction: The encoder consists of several alternating layers of "convolutional layer - batch normalization layer (BN) - activation layer - pooling layer". The convolutional layer uses a 1×3 one-dimensional convolutional kernel and uses appropriate padding to maintain local alignment. The BN layer normalizes the output channel one by one. The pooling layer uses 1×2 max pooling (MaxPool1d). The activation function is ReLU. Step S43: Upsampling and Skip Fusion: The decoder consists of several alternating modules: "upsampling (implemented by deconvolution / transposed convolution, kernel size 1×2) — concatenating corresponding downsampling feature maps — convolutional layer — BN layer — activation layer". The upsampling output and the corresponding encoder features are fused through skip connections. At each skip connection, a channel-spatial attention module (CBAM) is embedded to adaptively weight the encoder features, thereby highlighting spectral peak-related channels and suppressing background noise. The bottleneck is a lightweight Transformer module integrated after the convolutional layer. This 1D Transformer, based on multi-head self-attention, is used to model long-range dependencies of spectral lines and enhance global context awareness. Step S44: Output layer: consists of two convolutional layers, each with a kernel size of 1×1 and the activation function is ReLU.

[0026] Furthermore, in steps S42 and S43, the ReLU function formula is as follows: ; Furthermore, in step S4, the loss function adopts a weighted combination of MSE and peak retention, as shown in the formula: L= α ·MSE( Y pred, Y ideal)+ β ·SSIM( Y pred, Y ideal); in This is the total loss value, α is the weighting coefficient of the MSE term, and β is the weighting coefficient of the SSIM term. SSIM (Structural Similarity Index) ensures the morphological fidelity of the spectrum after noise reduction. This represents the model's predicted output. This represents the actual target value.

[0027] Furthermore, in step S43, a cross-attention fusion mechanism based on CBAM weighted features is employed at the skip connections: the decoder upsampled features are used as the query, and the CBAM-weighted high-resolution encoder features are used as the key / value pair. Multi-head cross-attention is used to calculate the weighted response of the query to the encoder features. The cross-attention output is projected and fused with the original decoder features in the form of residuals. A learnable one-dimensional relative position bias is introduced into the attention calculation to enhance the modeling ability of spectral position relationships. This cross-attention fusion enables the decoder to actively select and absorb useful local information from the encoder, thereby effectively avoiding global context overwriting of high-resolution local details and improving spectral peak preservation. (1) Projection and Transformation: .

[0028] Where Q represents the query vector, K represents the key vector, and V represents the value vector. This represents the projection operation, B is the batch size, U is the upsampled feature, and E is the encoder feature after CBAM. Represents the set of real numbers. , These represent the sequence lengths of the Query vector and the Key vector, respectively. The feature dimensions represent the Q, K, and V vectors.

[0029] (2) Attention: .

[0030] in Represents the output of the attention mechanism. For relative position offset, This represents the dimension of the Q, K, and V vectors processed by each head.

[0031] (3) Integration: ;

[0032] in Represents the final output after fusion. The output projection layer is used to project the output of the attention mechanism back to a dimension that matches U. It is a learnable gating system.

[0033] Furthermore, in step S43, when the output features of a CNN module using Batch Normalization (BN) are directly fed into a Transformer module using Latch Normalization (LN), the two modules make completely different assumptions about data distribution and statistical properties, which can lead to gradient conflicts and drastic changes in data distribution during training. This can result in slow model convergence or even training collapse (divergence), i.e., "training instability". Therefore, this invention sets up ConvToToken and TokenToConv adapters between the convolutional layers of the encoder and decoder and the Transformer module, respectively, and uses LayerNorm / residual bridging (PreNorm structure) to solve the mismatch between the BN layer and LayerNorm, thus ensuring training stability. (1) ConvToToken Adapter - Tokenization (patch) and Linear Projection: When standard Transformers (such as Vision Transformer, ViT) process images, they first roughly segment the 2D image into a series of non-overlapping 2D "patches" and then perform linear projection. If this method is directly applied to 1D spectra, it is equivalent to "segmenting" the spectrum, which will destroy the continuity of the local spectral peak features just extracted by the convolutional layer (encoder).

[0034] This invention does not simply "cut", but rather transforms the 1D convolutional feature map output by the encoder... pass ( The "linear projection" and "transformation" of the tokens into a "token sequence" (Z) that the Transformer can understand, where the "patch rearrangement" operation is to... The shape (B, C, L) is converted to the Transformer-friendly (B, N, D) format, where the feature map length (L) is considered as the sequence length (N) and the number of channels (C) is considered as the feature dimension (D) of each sequence element. Then, the last two dimensions of (B, D, N) are transposed to obtain the (B, N, D) sequence.

[0035] ; ; in These are the spectral features after channel transformation, where T is the sequence tensor after patch rearrangement. and is the parameter of the linear projection layer, and Z is the sequence of tokens after projection.

[0036] (2) Normalization (PreNorm) and Transformer: ; in The normalized sequence, This represents the output sequence of the Transformer module.

[0037] (3) TokenToConv adapter - backprojection restoration: This invention designs a TokenToConv adapter symmetrical to (1). It achieves this through inverse linear projection. , Restore spectral form And a key 1×1 convolution To complete, among which Is in (1) The reverse operation, The tensor shape is restored to (B, C, L).

[0038] 1×1 convolution Crucially, it not only maps back to the original channels but also reassembles the "global context features" output by the Transformer into a "CNN-friendly" feature map. This makes... It can be smoothly fed into the next stage of the U-Net decoder (which will again use BN) without causing conflicts.

[0039] ; ; ; in and These are the parameters of the inverse linear projection layer. This indicates that the Transformer output is back-projected back onto the patch representation. This indicates that the signal has been restored to its spectral form. This represents the sequence mapped back to the original channel.

[0040] (4) Residual / Gated Fusion: ; ; Where g is the gate weight vector, used to control the intensity of Transformer feature injection, and σ represents Sigmoid activation. Y represents the basic features, i.e., the features before they are input to the ConvToToken adapter, and Y represents the final output features.

[0041] Step S5: Denoise the untrained test set data using the trained network model to verify the model's denoising ability. Calculate the MSE (Mean Sequence Equation) compared to the ideal spectrum to verify the model's denoising ability; an MSE less than 10 is acceptable. -5 This indicates that the data noise reduction effect is relatively ideal, if MSE is greater than or equal to 10. -5 If the noise reduction effect is poor, return to step S1 to increase the dataset and continue optimizing the model until the MSE is less than 10. -5 .

[0042] In one specific embodiment, please refer to Figure 2 The Raman spectroscopy noise reduction method based on the MSC-U-Net neural network algorithm specifically includes the following steps: First, the dataset required for the MSC-U-Net neural network algorithm is obtained. This invention uses the MSC-U-Net algorithm to remove noise signals from Raman spectra, requiring the acquisition of Raman spectra of various substances at different concentrations. Multiple spectra are measured for each concentration of each substance, resulting in a raw spectral dataset containing noise. In this embodiment, under the same measurement conditions and environment, 10 Raman spectra of each of 7 different substances at 5 different concentrations were measured, for a total of 350 Raman spectra. If there are fewer than 7 Raman spectra, the average performance will be poor. Random noise (especially singular peaks) may not be effectively canceled, resulting in a less than ideal spectrum generated in step S3, still containing noise. Regarding the types and concentrations of substances, too few will lead to overfitting; the model may only be effective for specific substances or concentrations trained on, but when encountering unfamiliar datasets not used in training, its noise reduction effect will significantly decrease, failing to effectively remove singular peaks or retain valid signals. The acquired raw spectra are shown below. Figure 3 As shown.

[0043] Next, all collected spectra are preprocessed using Max-Min normalization to ensure that the model training is not affected by intensity scale differences. The data is then randomly divided into training, validation, and test sets with a ratio of 8:1:1.

[0044] Next, based on the type and concentration of the substance, the ideal spectrum is obtained by averaging every 10 spectra, such as... Figure 4 As shown.

[0045] Then, the MSC-U-Net noise reduction model is constructed.

[0046] The spectral data are first processed by MSC based on the substance and its concentration. The purpose is to reduce the difference in spectral intensity of the same substance at the same concentration and to prevent the model from affecting the effective Raman signal.

[0047] U-Net is a popular Convolutional Neural Network (CNN) architecture, proposed by Olaf Ronneberger et al. in 2015. It is mainly used for image segmentation tasks.

[0048] U-Net employs a combination of downsampling and upsampling to form a U-shaped symmetrical network structure. Downsampling consists of multiple convolutional and pooling layers alternating, while upsampling consists of multiple convolutional and transposed convolutional layers, gradually restoring the feature maps to the original input dimension and skipping connections with the feature maps of the corresponding downsampling layers. This preserves details and localization information in the output.

[0049] In this embodiment, the model used is Transformer-Attention U-Net (a variant of U-Net based on Transformer and attention mechanisms). While maintaining the symmetric encoder-decoder skeleton of U-Net, the network structure is improved as follows: a cross-attention fusion mechanism based on CBAM weighted features is used at the skip connection between each encoder layer and the corresponding decoder to adaptively weight the features from the encoder to highlight the spectral peak correlation channel and suppress background noise; a lightweight 1D Transformer module (based on multi-head self-attention mechanism) is additionally integrated at the network bottleneck to model the long-range dependencies of the spectral sequence and global contextual information, thereby enhancing the ability to model periodic noise and baseline drift. In terms of implementation, both the encoder and decoder use one-dimensional convolutions with a kernel size of 1×3, and padding is used to maintain local alignment. After each convolution, a Batch Normalization (BN) layer is applied to normalize the channels. The pooling layer uses 1×2 max pooling (MaxPool1d), and the activation function is ReLU. In the upsampling stage, scale recovery is achieved through 1×2 transposed convolution (ConvTranspose1d). The upsampled output (as the query) is fused with the attention-weighted encoder features (as the key / value) through a cross-attention fusion mechanism. This fused output is projected and fused with the original decoder features in residual form, and then processed by 1×3 convolution and BN. The output layer consists of two 1×1 pointwise convolutions (BN and activation can be concatenated between the two layers), ultimately outputting a single-channel denoised spectrum.

[0050] In this embodiment, the optimization strategy used is the Adam algorithm, with an initial learning rate of 1×10⁻⁶. -3The number of iterations is determined by the value of the validation set loss function. The iteration ends when the validation set loss function value no longer decreases. Specific feature map parameters and other structural information of the network model are as follows: Figure 5 As shown.

[0051] This embodiment inputs the data used to train the model into the trained model to perform noise reduction on the results. Figure 7 This is the graph of the training set after processing by the MSC-U-Net denoising model in this embodiment of the invention. Figure 8 This is the graph of the test set processed by the MSC-U-Net noise reduction model in this embodiment of the invention. Figure 7 , 8 Compared to Figure 6 It is obvious Figure 6 The noise portion circled in the middle was removed, and the denoising effect was significant. The training set MSE = 2.0114 × 10⁻⁶. -6 The test set MSE is 1.3825 × 10⁻⁶. -6 In machine learning, Mean Squared Error (MSE) is a commonly used metric to measure the difference between model predictions and actual values. A smaller MSE value generally indicates better predictive performance. MSE values ​​are very small, typically around 10. -6 This is on the order of magnitude, indicating that the model's predictions are very close to the actual values, and the denoising model has excellent predictive performance. This demonstrates that the loss of effective signal during denoising is very low.

[0052] In this embodiment, the new data is also imported into the MSC-U-Net noise reduction model for noise reduction processing, and it can be clearly seen that all noise, including singular peaks, has been significantly reduced. Figure 9 This is the raw spectrum of unfamiliar data. Areas with significant noise have been marked with circles in the figure. Figure 10 for Figure 9 The spectrum of unfamiliar data after denoising using the U-Net denoising model shows that noise has been effectively removed.

[0053] Comparative Example By comparing the noise reduction method used in this embodiment with the noise reduction effect of the commonly used SG smoothing algorithm with different parameters, it was found that the method provided by the present invention has a better noise removal effect. The SG algorithm is not sensitive to singular peaks, and many singular peak signals are still preserved in the spectrum.

[0054] Figure 11 The spectrum after noise reduction using the existing SG algorithm with parameters of window size 5 and smoothing order 3. Figure 12 The spectrum after noise reduction using the existing SG algorithm with parameters of window size 7 and smoothing order 5. From Figure 11 and Figure 12 As can be seen, many unusual peak signals are still preserved in the spectrum. This demonstrates that the Raman spectroscopy noise reduction method based on a neural network algorithm provided in this invention is more effective at removing noise signals from Raman spectra.

[0055] In summary, this invention is the first to combine the Transformer with an attention mechanism within the U-Net framework, achieving a balance between noise removal and signal preservation. Through multi-scale data and real-time processing design, it overcomes the limitations of traditional Raman spectroscopy denoising methods in removing random noise and singular peaks. Specific improvements are as follows: Multi-scale attention fusion: A CBAM module is embedded at the skip connections of U-Net to enhance spectral peak features and suppress background noise through adaptive channel-spatial weights. Simultaneously, a cross-attention mechanism is introduced, enabling the decoder to actively select encoder features and avoid global context overshadowing local details.

[0056] Long-range dependency modeling: A lightweight 1D Transformer is integrated at the bottleneck to capture the global dependencies of spectral sequences through multi-head self-attention. The newly added ConvToToken adapter converts convolutional features into token sequences, solving the mismatch problem between BN layers and LayerNorm and improving training stability.

[0057] Loss function optimization: The loss function adopts a weighted combination of MSE and peak preservation, as shown in the formula: L= α ·MSE( Y pred, Y ideal)+ β ·SSIM( Y pred, Y ideal); SSIM (Structural Similarity Index) ensures the morphological fidelity of the spectrum after noise reduction.

[0058] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be defined by the claims.

Claims

1. A Raman spectroscopy noise reduction method based on the MSC-U-Net algorithm, characterized in that, Includes the following steps: S1: Under the same measurement conditions and environment, measure the Raman spectra of various substances at different concentrations. For each concentration of each substance, measure multiple spectra to obtain a raw spectral dataset containing noise. S2: Normalize and preprocess all collected spectra, divide the dataset into training, validation and test sets, and construct the dataset required for the noise reduction model; S3: The collected raw spectra are averaged separately according to multiple spectra of the same substance and the same concentration to obtain an ideal spectrum without noise; S4: Construct a Transformer-Attention U-Net model for MSC processing. Input the training set into the model for parameter learning, and iteratively optimize according to the loss function to obtain the spectral denoising model. The encoder consists of several alternating convolutional layers, batch normalization layers, activation layers, and pooling layers. The decoder consists of several alternating upsampling, concatenating corresponding downsampling feature maps, convolutional layers, batch normalization layers, and activation layers. The upsampling output and the corresponding encoder features are fused through skip connections. At each skip connection, a cross-attention fusion mechanism based on CBAM weighted features is used. A Transformer module is set at the bottleneck of the intermediate layer between the encoder and decoder. The decoder upsampling features are used as queries, and the CBAM-weighted features are then processed. Weighted high-resolution encoder features are used as keys / value pairs. Multi-head cross-attention is used to compute the weighted response of the query to the encoder features. The multi-head cross-attention output is projected and fused with the original decoder features as a residual. A learnable one-dimensional relative position bias is introduced into the attention computation to enhance the modeling ability of spectral position relationships. A ConvToToken adapter is set between the encoder's convolutional layer and the Transformer module, and a TokenToConv adapter is set between the decoder's convolutional layer and the Transformer module. LayerNorm / residual bridging is used to implement batch normalization layers and LayerNorm. To ensure stability matching, the ConvToToken adapter transposes and linearly projects the 1D convolutional feature map output by the encoder, normalizes it using LayerNorm, and then outputs it to the Transformer module. The TokenToConv adapter maps the output of the Transformer module back to the original channel through inverse linear projection, and reassembles the global context features output by the Transformer into a convolutional neural network feature map through a 1×1 convolution X_(rec_proj), so that X_(rec_proj) is smoothly fed into the next stage of the U-Net decoder. A learnable gating mechanism is used to dynamically adjust the Transformer's reconstructed features. The injection intensity is then added to the original base feature as a residual. This is to achieve intelligent fusion of global and local information: ； ； Where g is the gate weight vector, used to control the intensity of Transformer feature injection, and σ represents Sigmoid activation. Y represents the basic features, i.e., the features before they are input to the ConvToToken adapter, and Y represents the final output features.

2. The Raman spectroscopy noise reduction method as described in claim 1, characterized in that, Step S4 includes the following steps: S41: MSC processing: Perform multiple scattering correction preprocessing on the spectra of the training set and the validation set with the same substance and the same concentration to eliminate scattering / baseline differences and enhance spectral consistency; S42: Downsampling module construction: The encoder consists of several convolutional layers, batch normalization layers, activation layers, and pooling layers in an alternating manner. The convolutional layers use 1×3 one-dimensional convolutional kernels and appropriate padding to maintain local alignment. The batch normalization layers normalize the output channels one by one. The pooling layers use 1×2 max pooling. The activation function is ReLU. S43: Upsampling and skip fusion: The decoder is composed of several upsampling-splitting corresponding downsampling feature maps-convolutional layers-batch normalization layers-activation layers in an alternating manner. The upsampling output and the corresponding encoder features are fused through skip connections, and a cross-attention fusion mechanism based on CBAM weighted features is adopted at each skip connection. A Transformer module is set at the bottleneck of the intermediate layer between the encoder and the decoder. S44: Output layer: consists of two convolutional layers, each with a kernel size of 1×1 and the activation function is ReLU.

3. The Raman spectroscopy noise reduction method as described in claim 2, characterized in that, The ReLU function in step S42 is as follows: 。 4. The Raman spectroscopy noise reduction method as described in claim 2, characterized in that, Step S43 specifically includes: (1) Projection and Transformation: ； Where Q represents the query vector, K represents the key vector, and V represents the value vector. This represents the projection operation, B is the batch size, U is the upsampled feature, and E is the encoder feature after CBAM. Represents the set of real numbers. , These represent the sequence lengths of the Query vector and the Key vector, respectively. The feature dimensions representing the Q, K, and V vectors; (2) Attention: ； in Represents the output of the attention mechanism. For relative position offset, This represents the dimension of the Q, K, and V vectors processed by each head; (3) Integration: ； in Represents the final output after fusion. The output projection layer is used to project the output of the attention mechanism back to a dimension that matches U. It is a learnable gating system.

5. The Raman spectroscopy noise reduction method as described in claim 1, characterized in that, The loss function in step S4 uses a weighted combination of MSE and peak retention, as shown in the formula: L= α ·MSE( Y pred, Y ideal)+ β ·SSIM( Y pred, Y ideal); in This is the total loss value, α is the weighting coefficient of the MSE term, and β is the weighting coefficient of the SSIM term. SSIM is the structural similarity index, used to ensure the morphological fidelity of the spectrum after noise reduction. This represents the model's predicted output. This represents the actual target value.

6. The Raman spectroscopy noise reduction method as described in claim 1, characterized in that, In step S2, the ratio of the training set, validation set, and test set is 8:1:1 or 6:2:

2.

7. The Raman spectroscopy noise reduction method as described in claim 1, characterized in that, The optimization strategy used in step S4 is the Adam algorithm, with an initial learning rate of 1×10⁻⁶. -2 -1×10 -3 The number of iterations is determined based on the value of the validation set loss function. The iteration ends when the value of the validation set loss function no longer decreases.

8. The Raman spectroscopy noise reduction method as described in claim 7, characterized in that, The process also includes step S5: Denoising the untrained test set data using the trained network model, calculating the MSE (Mean Separation Equation) relative to the ideal spectrum to verify the model's denoising capability. If the MSE is greater than or equal to 10... -5 Then return to step S1 to increase the dataset and continue optimizing the model until the MSE is less than 10. -5 .

Images