A raman spectrum processing method based on adaptive baseline correction

Abstract

The application relates to the technical field of spectral data processing, and discloses a Raman spectrum processing method based on adaptive baseline correction, which comprises the following steps: S1: acquiring the Raman spectrum of a sample to be measured; S2: performing adaptive Kalman filtering estimation on the Raman spectrum to obtain a preliminary baseline, and adopting a real-time constraint mechanism to make the baseline in the estimation process close to the lower side of the signal; S3: performing multi-stage extreme smoothing processing based on the preliminary baseline, combining the real-time constraint mechanism, extracting a baseline trend, and obtaining an extreme smoothing baseline; S4: performing independent SG fine smoothing based on the extreme smoothing baseline, and confirming a final baseline; S5: performing baseline correction according to the final baseline, and obtaining a corrected spectrum; S6: performing denoising based on the corrected spectrum; S7: extracting key characteristic peaks based on the denoised spectrum, and calculating a coefficient of variation according to the peak area of the key characteristic peaks; judging the data consistency according to the coefficient of variation, and triggering a feedback mechanism when the consistency is abnormal; and the application can significantly improve the signal-to-noise ratio and characteristic fidelity of the Raman spectrum.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a Raman spectroscopy processing method based on adaptive baseline correction. Background Technology

[0002] Raman spectroscopy is a non-destructive testing technique widely used in chemistry, biology, and agricultural products. It provides molecular fingerprint information of substances and is particularly suitable for the qualitative and quantitative analysis of active ingredients in agricultural products such as wolfberry, including carotenoids, polysaccharides, and phenolic compounds. However, due to the biological complexity of wolfberry samples and the presence of inherent fluorescent substances, their Raman spectra are easily affected by strong fluorescence background interference and severe baseline drift during actual acquisition. This baseline distortion caused by high-intensity fluorescence background and instrument noise can raise or distort the overall spectral signal, severely masking weak Raman characteristic peaks related to the active ingredients (such as the C=C vibration peak of carotenoids (~1158 cm⁻¹)). - ¹), CO vibration peak of polysaccharides (~1080 cm⁻¹) - ¹) and characteristic signals of some phenolic compounds, thus directly affecting the accuracy and reliability of the spectroscopic component analysis and origin identification model of wolfberry. Therefore, developing a Raman spectroscopy acquisition and preprocessing method that can suppress interference at the source, adaptively correct the baseline, and systematically optimize data quality is key to promoting the application of this technology in rapid detection and quality control of agricultural products.

[0003] In existing technologies, common methods for baseline correction in Raman spectroscopy include polynomial fitting, wavelet transform, and morphological filtering. While these methods are effective in general scenarios, they still have significant limitations when processing spectra like those of wolfberry, which have high fluorescence background and weak characteristic peaks: polynomial fitting is sensitive to parameter selection and easily affected by local noise and sharp peaks, leading to baseline overfitting or underfitting; wavelet transform, although capable of separating different frequency components, has a high computational complexity due to its highly empirically dependent decomposition layer and threshold selection, and it is difficult to achieve a balance between strong noise suppression and weak feature preservation; morphological filtering can roughly extract baseline trends, but it is highly dependent on window size parameters, making it difficult to achieve a balance between smoothing noise and preserving effective features, especially broad and gentle peaks, and its use alone often fails to effectively handle the complex and nonlinear fluorescence background commonly found in wolfberry spectra.

[0004] Furthermore, most existing studies treat preprocessing steps such as baseline correction and denoising as independent, sequential "pipeline" operations, lacking a systematic coordination and feedback mechanism. This step-by-step processing mode leads to isolated parameter configurations, accumulated information loss, and an inability to adaptively adjust subsequent parameters based on the results of preceding processing. It also makes it difficult to provide feedback optimization for the quality of the data acquisition stage (such as the representativeness of sampling points and the number of measurements). Especially in highly variable samples like wolfberry, traditional methods often struggle to completely remove high-intensity, nonlinear fluorescence background while retaining key weak characteristic peaks, limiting the accuracy and robustness of subsequent analyses.

[0005] Therefore, how to systematically improve the quality of Raman spectroscopy data and the reliability of subsequent analysis is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of the above problems, the present invention is proposed to provide a Raman spectroscopy processing method based on adaptive baseline correction that overcomes or at least partially solves the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A Raman spectral processing method based on adaptive baseline correction includes the following steps:

[0009] S1: Obtain the Raman spectrum of the sample to be tested;

[0010] S2: Perform adaptive Kalman filtering on the Raman spectrum to estimate the initial baseline, and use a real-time constraint mechanism to ensure that the baseline closely follows the lower edge of the signal during the estimation process;

[0011] S3: Perform multi-stage extreme smoothing processing based on the initial baseline, and combine it with the real-time constraint mechanism to extract the baseline trend and obtain the extreme smoothed baseline;

[0012] S4: Perform independent SG fine smoothing based on the extremely smooth baseline to confirm the final baseline;

[0013] S5: Perform baseline correction based on the final baseline to obtain the corrected spectrum.

[0014] Preferably, S2 includes:

[0015] Multiple rounds of state prediction and updating are performed based on the intensity sequence of the Raman spectrum;

[0016] After each update, the baseline constraint is applied so that the updated baseline is flush with the lower edge of the signal.

[0017] Preferably, the baseline constraint is:

[0018]

[0019] in, Ymax and Ymin are the maximum and minimum intensities of the original spectrum, respectively, and γ is the offset coefficient. The small positive offset δ is calculated in the baseline constraint to control how closely the baseline follows the lower edge of the signal.

[0020] Preferably, S3 includes:

[0021] S31: Large Window SG Smoothing:

[0022]

[0023] in, This is the result after large-window SG smoothing; j is the position index relative to the center point i. This represents the number of points extending from the center point i to the left and right. The weights c of each point within the convolution kernel j algebraic sum (i.e. This is used to ensure that the mean intensity of the signal reconstructed by polynomial fitting does not undergo systematic shift within the local window. The convolution coefficients are determined by local polynomial least squares fitting; This serves as a preliminary baseline.

[0024] S32: Gaussian filter:

[0025]

[0026]

[0027] Where G(j;σ) is the discrete Gaussian kernel function, which itself contains a normalization factor. This ensures that the weighted sum of the discrete kernels approaches 1, so there is no need to introduce an additional normalization coefficient in the formula. σ is the standard deviation, and L is the half-window width of the Gaussian kernel, which is usually taken as an integer part of 3σ to cover the main energy of the kernel.

[0028] S33: Moving Average

[0029]

[0030] in, The number of discrete points participating in the arithmetic mean is specifically defined as follows: In the calculation process, it is directly used as the denominator for normalization to eliminate random fluctuations. The width of the window in the moving average smoothing is half the width. This is the baseline after extreme smoothing.

[0031] Preferably, S4 includes:

[0032]

[0033] Where i is the index of the current data point being processed. The output sequence represents the final baseline intensity value at index i; is the normalization coefficient; j is the relative position index within the window, ranging from arrive ; These are the Savitzky-Golay convolution coefficients, representing the contribution weights of each point within the window to the fitted value of the center point; The input sequence represents the intensity value of the baseline at index i+j after extreme smoothing.

[0034] Preferably, the step further includes: S6: Denoising based on the corrected spectrum; specifically including:

[0035] S61: Normalize the corrected spectrum to obtain a normalized signal;

[0036] S62: Confirm the optimal VMD decomposition parameters, perform VMD decomposition on the normalized signal, and extract the IMF component;

[0037] S63: Through correlation analysis, select components that meet the preset threshold from multiple IMF components as valid signal components;

[0038] S64: The selected effective IMF components are processed using an improved piecewise semi-soft threshold function to perform partitioned noise reduction, resulting in the noise-reduced components.

[0039] S65: Perform a weighted summation on each of the denoised components to reconstruct the normalized denoised signal.

[0040] S66: Perform inverse normalization processing on the normalized denoised signal to obtain the denoised spectrum.

[0041] Preferably, confirming the optimal decomposition parameters of VMD specifically includes: using the RIME optimization algorithm, with the balance between envelope entropy and reconstruction fidelity as the fitness function, and adaptively determining the optimal decomposition level K and penalty factor α of VMD within a preset search range.

[0042] Preferably, the steps further include: S7: extracting key feature peaks based on the denoised spectrum, and calculating the coefficient of variation based on the peak area of ​​the key feature peaks; judging data consistency based on the coefficient of variation, and triggering a feedback mechanism when the consistency is abnormal, and updating the baseline correction parameters based on historical data.

[0043] Preferably, the steps further include: calculating the overall signal-to-noise ratio and the average intensity of the key characteristic peak regions based on the corrected spectrum, adjusting the search range of the VMD decomposition parameters during the optimization process based on the overall signal-to-noise ratio, and fine-tuning the threshold scaling factor in the piecewise semi-soft thresholding function based on the average intensity.

[0044] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a Raman spectroscopy processing method based on adaptive baseline correction. The three-layer adaptive baseline correction architecture can completely remove the high-intensity nonlinear fluorescence background while fully preserving carotenoids (1158 cm⁻¹). - ¹、1525 cm - ¹) and polysaccharides (1080 cm - ¹) The morphology and intensity of key weak feature peaks. In the denoising stage, the RIME algorithm is introduced to adaptively optimize the VMD decomposition parameters, and a piecewise semi-soft threshold function is designed to efficiently filter out high-frequency noise while maximizing the protection of feature peak morphology. On this basis, by establishing a cross-step parameter linkage mechanism and a terminal quality feedback closed loop, the limitations of traditional pipeline processing are broken, and global coordination and self-optimization from data acquisition to signal purification are realized. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of a Raman spectroscopy processing method based on adaptive baseline correction provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the processing procedure of Kalman filtering combined with real-time constraint mechanism provided in the embodiments of the present invention;

[0048] Figure 3 This is a schematic diagram of the multi-stage extreme smoothing processing method provided in the embodiments of the present invention;

[0049] Figure 4 This is a schematic diagram of the adaptive denoising process based on VMD-RIME optimization and segmented semi-soft thresholding provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the feedback mechanism for updating spectral consistency anomaly trigger parameters provided in an embodiment of the present invention;

[0051] Figure 6 This is a baseline correction effect diagram using the traditional morphological filtering method in the embodiments of the present invention;

[0052] Figure 7 This is a diagram showing the effect of using the baseline correction method of the present invention in an embodiment of the present invention;

[0053] Figure 8 This is a comparison of the spectra before and after noise removal using the baseline correction method of the present invention in an embodiment of the present invention. Detailed Implementation

[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] For samples with high fluorescence backgrounds, such as wolfberry, traditional baseline correction methods, such as polynomial fitting and wavelet transform, are sensitive to parameter selection and struggle to achieve a balance between strong noise suppression and weak feature peak preservation. Single algorithms are prone to overfitting, underfitting, or feature peak distortion when dealing with complex, nonlinear fluorescence baselines. Therefore, this invention provides Example 1 to address this problem.

[0056] Example 1

[0057] like Figure 1 This invention discloses a Raman spectroscopy processing method based on adaptive baseline correction, the steps of which include: S1: acquiring the Raman spectrum of the sample to be tested;

[0058] S2: Perform adaptive Kalman filtering on the Raman spectrum to estimate the initial baseline, and use a real-time constraint mechanism to ensure that the baseline closely follows the lower edge of the signal during the estimation process;

[0059] S3: Perform multi-stage extreme smoothing based on the initial baseline, and combine it with a real-time constraint mechanism to extract the baseline trend and obtain the extreme smoothed baseline;

[0060] S4: Perform independent SG fine smoothing based on the extremely smooth baseline to confirm the final baseline;

[0061] S5: Perform baseline correction based on the final baseline to obtain the corrected spectrum.

[0062] In this embodiment, the present invention proposes a three-stage cascaded adaptive baseline correction architecture of Kalman filtering, multi-stage extreme smoothing, and independent SG smoothing. This architecture systematically solves the problem that traditional methods are unable to balance strong fluorescence subtraction and weak characteristic peak retention in Raman spectra with high fluorescence backgrounds such as wolfberry by means of progressive and collaborative division of labor.

[0063] Before adaptive baseline correction, this embodiment systematically acquires the Raman spectra of the samples to be tested as raw spectra through systematic sampling. Three dried wolfberry samples from different origins are randomly selected from each origin. For each sample, three measurement segments (front, middle, and back) are equally spaced along the long axis of the sample. Using a 785nm micro Raman spectrometer, 50 raw Raman spectra are continuously and automatically acquired within each segment, and all raw spectral data (Yraw) and corresponding sampling location information are saved.

[0064] like Figure 2 In this embodiment, S2 is the first-level processing, which adopts a strongly constrained Kalman filter. By setting a minimum process noise and a large measurement noise, and combining the real-time constraint of "forced baseline at the bottom", the filter can perform preliminary baseline estimation close to the lower edge of the signal under a high-intensity nonlinear fluorescence background. This effectively overcomes the defects of traditional polynomial fitting or wavelet transform, which are prone to deviating from the true baseline due to noise interference, and provides an accurate initial baseline for subsequent processing.

[0065] S2 specifically includes:

[0066] A discrete Kalman filter with a constant velocity (CV) model is used to perform preliminary baseline estimation on the original spectral intensity sequence. The state-space model is defined as follows:

[0067] State prediction equation:

[0068]

[0069]

[0070] State update equation:

[0071]

[0072]

[0073]

[0074] in, Let k be the state vector at time k. This represents the prior estimate of the state vector. This indicates the location of the baseline, i.e., the intensity value. This represents the prior estimate of the state estimation error covariance matrix; This indicates the rate of change of the baseline. Here is the state transition matrix. The step size of the Raman shift (usually 1 cm) -1 ). Let be the state estimation error covariance matrix. The process noise covariance matrix is ​​set to a minimum value in this invention (e.g., =10 -12 (on a magnitude scale) to force the baseline evolution to be extremely smooth. Yraw[k] is the observed value at time k, i.e., the original spectral intensity. Let R be the observation matrix. R is the measurement noise variance, which is set to a large value in this invention to reduce the impact of single-point noise on state estimation and enhance robustness. This is the Kalman gain matrix.

[0075] After each policy update, constraint checks and corrections are performed based on the obtained baseline estimate; that is, the real-time constraint mechanism ensures that the baseline during the estimation process closely follows the lower edge of the signal. The baseline constraint conditions are:

[0076]

[0077] in, Ymax and Ymin are the maximum and minimum intensities of the original spectrum, respectively, and γ is the offset coefficient used to calculate the small positive offset δ in the baseline constraint, controlling how closely the baseline follows the lower edge of the signal.

[0078] like Figure 3 In this embodiment, S3 is the second-level processing stage, which consists of three stages in series: large-window Savitzky-Golay smoothing, Gaussian filtering, and moving average, and is executed iteratively multiple times. This stage aims to thoroughly remove the global and stable fluorescence background trend in the spectrum, thereby specifically solving the problem of residual mid-to-high frequency fluctuations in the initial baseline and creating a clean background environment for the preservation of weak characteristic peaks.

[0079] S3 specifically includes:

[0080] 1. Large window Savitzky-Golay smoothing:

[0081]

[0082] The total width of the window is , This represents the number of points extending to the left and right sides from the center point i. It determines the range of data participating in the local polynomial fitting. The weights c of each point within the convolution kernel j algebraic sum (i.e. This is used to ensure that the mean intensity of the signal reconstructed by polynomial fitting does not undergo systematic shift within the local window. The convolution coefficients (polynomial order 3) are determined by local polynomial least squares fitting.

[0083] 2. Gaussian filtering:

[0084]

[0085]

[0086] Where G(j;σ) is the discrete Gaussian kernel function, which itself contains a normalization factor. This ensures that the weighted sum of the discrete kernels approaches 1, thus eliminating the need for an additional normalization coefficient in the formula. σ represents the standard deviation, and L is the half-window width of the Gaussian kernel, typically taken as an integer part of 3σ to cover the main energies of the kernel. This step further suppresses mid-frequency fluctuations.

[0087] 3. Moving Average:

[0088]

[0089] Among them, the total width of the window . The width of the window used in the second level of "moving average" smoothing, which outputs the extremely smoothed baseline. The above three levels of smooth, iterative execution can be performed N times. iter This is to further enhance the trend extraction effect. N iter This is the number of iterations performed in the second-level multi-stage extreme smoothing process (which includes three sub-steps: Savitzky-Golay, Gaussian filtering, and moving average), used to enhance the extraction effect of the baseline trend.

[0090] In this embodiment, S3 is a process of multiple iterative corrections, and after each smoothing, the real-time reduction mechanism in S2 is used again to check the constraints.

[0091] In this embodiment, S4 is the third-level processing stage. Based on the fact that the macroscopic trend of the spectral baseline has been basically determined by the first two stages of processing, this stage introduces an independent Savitzky-Golay smoothing as the final optimization step. Compared with the smoothing in the second stage, the smoothing operation in the third stage is completely free from the physical constraint of forcing the baseline to the lower edge in the previous stage. It no longer aims to fit the lower edge of the signal, but focuses purely on optimizing the local smoothness and global continuity of the baseline from the perspective of signal processing.

[0092] Specifically, conventional Savitzky-Golay smoothing is typically used as an embedded module in baseline correction. Its output is often still constrained by limitations such as not being lower than the original signal, making it difficult to independently optimize between smoothness and trend fidelity. In contrast, the independent Savitzky-Golay smoothing in this stage introduces a separate, unconstrained Savitzky-Golay filter after the first two stages of processing have essentially locked in the baseline trend. Its sole objective is to eliminate, to the greatest extent possible, any minor unnatural fluctuations that may remain from the previous stages, ensuring that the final output baseline B... final It can faithfully reflect the overall gradual trend of fluorescence background and has a sufficiently high smoothness, thus effectively avoiding the introduction of false peaks or distortion of characteristic peaks due to local oscillations during subsequent baseline subtraction.

[0093] Extremely smoothed baseline for the second-stage output Apply the following Savitzky-Golay filter:

[0094]

[0095] Where i is the index of the current data point being processed (i=0,1,...,N-1), and N is the total length of the signal. The output sequence represents the final baseline intensity value at index i after independent Savitzky-Golay smoothing. These are normalization coefficients, typically convolution coefficients. The sum of these values ​​is used to ensure that smoothing does not alter the DC component (overall mean) of the signal. j is the relative position index within the window, rounded to an integer, ranging from... arrive . The Savitzky-Golay convolution coefficients are determined by the window width W. final The order of the polynomial (3rd order in this invention) is pre-calculated using the least squares method, representing the contribution weight of each point within the window to the fitted value of the center point. The input sequence represents the intensity value of the baseline at index i+j after the second-level multi-stage extreme smoothing.

[0096] This step uses a separate Savitzky-Golay filter with a window width of The polynomial order is 3. The width is half the window width (a positive integer). The key innovation is that this smoothing operation is not constrained by the "baseline-below" principle; its purpose is purely to optimize from a signal processing perspective. The smoothness and continuity of the output correct for any minor unnatural fluctuations that may remain in the preceding stage, resulting in the final baseline B. final Finally, the corrected spectrum Obtained through the following formula:

[0097]

[0098] Where k is the index of the data point (k=0,1,...,N-1). The output sequence represents the baseline-corrected Raman spectral intensity value at index k. The input sequence represents the original Raman spectral intensity value obtained at index k. Let be the input sequence, representing the final baseline estimate obtained at index k by the aforementioned independent Savitzky-Golay smoothing.

[0099] The results are then zeroed out.

[0100]

[0101] in To determine the larger of the two values, the function here will... If the result is negative when compared with 0, it is set to 0 to ensure that the spectral intensity is non-negative after correction.

[0102] Through the cascaded processing chain defined in the above formula, this scheme systematically combines adaptive filtering, strong trend extraction, and fine smoothing, ensuring that while completely removing the high-intensity, nonlinear fluorescence background in the wolfberry spectrum, it also fully preserves the fluorescence at 1158 cm⁻¹. -1 (Carotenoid C=C stretching vibration) and 1080cm -1 The morphology and intensity information of key weak characteristic peaks such as (polysaccharide CO vibration).

[0103] Example 2

[0104] After baseline correction in Example 1, high-frequency random noise, instrument noise, and residual fluorescence fluctuations still exist in the Raman spectrum of wolfberry. Traditional denoising methods (such as wavelet thresholding, moving average, and SG smoothing) often struggle to achieve an ideal balance between noise suppression and feature preservation. Commonly used hard thresholding functions can easily lead to signal discontinuities, while soft thresholding, although smoothing, may weaken characteristic peaks. Furthermore, traditional methods have fixed parameters, making them unsuitable for the noise characteristics of different samples and measurement conditions, easily causing characteristic peak distortion or over-smoothing, especially for carotenoids (1152 cm⁻¹). - ¹、1525 cm - ¹) The weak peak regions of polysaccharides and phenolic compounds are significantly affected.

[0105] Therefore, this embodiment performs S6 denoising processing on the basis of embodiment 1 to further improve the signal-to-noise ratio and feature fidelity of the Raman spectrum.

[0106] like Figure 4In this embodiment, the decomposition layer K and penalty factor α of VMD are adaptively optimized by the RIME algorithm to achieve efficient separation of signal and noise. Then, a piecewise semi-soft threshold function is used to differentiate the high-frequency IMF components, which are forced to zero in the noise region, smoothly shrunk in the transition region, and approximately preserved in the feature region, so as to preserve the feature peak shape and intensity to the maximum extent while removing noise.

[0107] S6 specifically includes:

[0108] S61: Normalize the corrected spectrum to obtain a normalized signal.

[0109]

[0110] in, This is the baseline-corrected spectral intensity sequence. and These are the maximum and minimum values ​​of the sequence, respectively. This is the normalized signal.

[0111] S62: Confirm the optimal VMD decomposition parameters, perform VMD decomposition on the normalized signal, and extract the IMF component;

[0112] The RIME optimization algorithm is employed, using the balance between envelope entropy and reconstruction fidelity as the fitness function. Within a preset search range, the optimal decomposition level K and penalty factor α for VMD are adaptively determined. The initial search range for the penalty factor α is set to [α...]. min ,α max The search range for the decomposition level K is set to [K]. min ,K max The fitness function is defined as follows:

[0113]

[0114] in, Let be the average envelope entropy of each IMF component, C be the correlation coefficient between the reconstructed signal and the original signal, and ε be a minimal constant to prevent division by zero. Envelope Entropy The calculation formula is:

[0115]

[0116] in:

[0117]

[0118] N is the signal length of the intrinsic mode function (IMF) component, i.e., the total number of data points. In both equations, N represents the same length, ensuring the probability distribution p... j The normalization range is consistent with the summation range of entropy, envj Let env be the envelope signal at the j-th data point obtained through Hilbert transform. m Let p be the envelope signal at the m-th data point obtained through Hilbert transform. j The probability distribution of the envelope of the IMF component is formed by the proportion of the j-th envelope amplitude to the sum of all envelope amplitudes, where δ is a very small positive number (e.g., 10). 10 (), used to avoid cases where the logarithm is zero or negative, ensuring numerical stability.

[0119] The location update mechanism of the RIME optimization algorithm is as follows:

[0120]

[0121] in, Let be the position vector of the i-th individual in the t-th generation (inclusive of α and K). Let r be the current optimal individual position, and r be a random number within the interval [-1, 1]. Through iterative optimization, the optimal parameter combination is finally obtained. , ).

[0122] Using optimized parameters ( , For normalized signals Perform VMD decomposition to obtain One eigenmode function component:

[0123]

[0124] S63: Through correlation analysis, select components that meet the preset threshold from multiple IMF components as valid signal components.

[0125] Calculate each IMF component Compared with the original normalized signal correlation coefficient and its kurtosis value :

[0126]

[0127]

[0128] Set a threshold for filtering: If >0.12 or If the value is greater than 3.0, the IMF component is determined to be a valid signal component and proceeds to the next step of noise reduction processing; otherwise, it is determined to be a noise-dominant component and discarded.

[0129] S64: The selected effective IMF components are processed using an improved piecewise semi-soft threshold function to perform partitioned noise reduction, resulting in the noise-reduced components.

[0130] Let a valid IMF component selected by S63 be... , where c k Let c be the coefficient value of this component at the k-th wavenumber point (which can be positive or negative). To suppress residual noise while preserving the characteristic peak shape, an improved piecewise semi-soft thresholding function is constructed as follows: for each coefficient c... k Handle independently.

[0131] Define coefficient magnitude The threshold function output is Its expression is divided into three regions for processing based on the relationship between x and the adaptive threshold T:

[0132] In the noisy area Forced reset to zero, i.e. To completely eliminate low-amplitude noise;

[0133] In the transition zone An exponential contraction function is used to achieve a smooth transition and avoid distortion;

[0134]

[0135] In the feature region Using approximate identity mapping To preserve the shape and intensity of characteristic peaks to the greatest extent possible.

[0136] The formula for calculating the adaptive threshold T is as follows:

[0137]

[0138] In the formula, β is the threshold scaling factor, which is used to adjust the overall threshold level, and the default value is 1.2; This is a robust estimate of the noise level of the IMF component, where N is the signal length.

[0139] Robust estimation of IMF component noise levels:

[0140]

[0141] After applying this threshold function, the denoised IMF components are obtained. .

[0142] S65: Perform a weighted summation on each of the denoised components to reconstruct the normalized denoised signal.

[0143]

[0144] in, For the reconstructed normalized denoised signal, S is the set of indices of the retained effective components. These are the weighting coefficients, and the weights are... The default value is 1.0. For the highest frequency IMF component (i=Kopt), if its correlation coefficient... If the value is less than 0.2, then a lower weight is assigned (e.g., ...). =0.5), to suppress residual high-frequency noise.

[0145] S66: Perform inverse normalization processing on the normalized denoised signal to obtain the denoised spectrum.

[0146] Example 3

[0147] Building upon Example 2, this example constructs a systematic collaborative optimization framework encompassing sampling, adaptive baseline correction, and optimized denoising. This framework achieves global collaborative optimization from data acquisition to signal purification by establishing an adaptive parameter transfer mechanism, online processing quality evaluation and feedback loop, and a three-stage integrated process. Its core is breaking down information barriers between steps, enabling subsequent processing to "sense" the results of preceding steps and dynamically adjust, while simultaneously feeding back the terminal signal quality to the source, forming a self-optimizing processing loop, thereby systematically improving the overall performance and robustness of the processing flow.

[0148] like Figure 5 This embodiment specifically includes:

[0149] S7: Extract key feature peaks based on denoised spectra and calculate the coefficient of variation based on the peak area of ​​the key feature peaks; determine data consistency based on the coefficient of variation and trigger a feedback mechanism when consistency is abnormal to update the baseline correction parameters based on historical data.

[0150] After the noise reduction step is completed, the preset key quality feature peaks are accurately extracted and the terminal quality index is calculated.

[0151] Let the denoised spectrum be For specific characteristic peaks (Central position is) Half-height and width are ), define its extraction window as The peak area is calculated as follows:

[0152]

[0153] The local signal-to-noise ratio is calculated as follows:

[0154]

[0155] in, For the outer neighborhood (e.g.) The variance of the intensity.

[0156] Terminal quality indicators are used for feedback evaluation. For M spectra in a certain region of the same wolfberry sample, their peak areas are calculated. Coefficient of variation for (j=1,2,...,M):

[0157]

[0158] in, .

[0159] like If τCV > 15 (e.g., τCV = 15), the measurement data consistency of that section is deemed abnormal, triggering the feedback mechanism. The Kalman filter parameters in the baseline correction stage S2 are then dynamically optimized.

[0160] The system records the process noise parameter q used by S2 in each processing step. 11 (i.e., the baseline state noise term in the state estimation error covariance matrix) and its corresponding terminal signal-to-noise ratio (SNR) i Each group (q) in the historical data 11 SNR i ) are considered as sample points, and q is fitted through nonlinear regression analysis. 11 Regarding SNR i Functional relationship:

[0161]

[0162] Here, α, β, and γ are model parameters, obtained by fitting historical data points using the least squares method. This model expresses that, for a given signal-to-noise ratio (SNR)... i Theoretically, the optimal process noise parameter q should be used. opt (i.e., q) 11 (Recommended value).

[0163] If the SNR of the current spectrum i SNR lower than the best value of similar samples in history opt If CVi exceeds 90% of the limit, or if CVi exceeds the limit in three consecutive processing iterations, a parameter update is triggered. The system calculates the theoretical optimal value q based on the current SNRi and substitutes it into the above model. opt and q in S2 11 Updated to:

[0164]

[0165] The clip function will q opt Limited to a preset stable operating range

[10] 1410 10 [Inside], to prevent the filter from diverging.

[0166] Update q 11 Then, the system re-executes the baseline correction procedures S2~S5 on the original spectra of all segments of the sample to ensure that the same optimized parameters are used for the same batch of samples. The newly generated q 11 and the corresponding SNR i CV i It will be stored in the historical database for the next round of model iteration and optimization.

[0167] Through the aforementioned closed-loop mechanism, the system achieves the goal of improving terminal quality indicators (CV). i From the front-end core parameters (q) 11 The adaptive guidance of the baseline correction process enables dynamic adjustment based on sample characteristics, effectively improving the robustness of processing samples with high fluorescence background.

[0168] S8: Calculate the overall signal-to-noise ratio and the average intensity of the key characteristic peak regions based on the corrected spectrum, adjust the search range of the VMD decomposition parameters in the optimization process based on the overall signal-to-noise ratio, and fine-tune the threshold scaling factor in the piecewise semi-soft threshold function based on the average intensity.

[0169] After the baseline correction step is completed, the overall signal-to-noise ratio and the intensity of key characteristic peaks in the output spectrum are calculated in real time, and the parameters of the denoising step are dynamically adjusted accordingly.

[0170] Let the baseline-corrected spectral signal be... , where k = 0,1,...,N-1, and N is the total number of spectral data points.

[0171] First, calculate the overall signal-to-noise ratio:

[0172]

[0173] in, for The mean.

[0174] Next, the key characteristic peak regions (such as...) are calculated. = [1150, 1550]cm -1 Average strength:

[0175]

[0176] Then, the denoising parameters are dynamically adjusted based on these metrics. The search range of the penalty factor α in the RIME optimizer is based on... Adaptive scaling, with the following scaling rules:

[0177]

[0178] in, The adjusted upper bound. The upper bound is the default value, and ρ is the adjustment coefficient (which can be 0.3). For reference signal-to-noise ratio.

[0179] The threshold scaling factor in the piecewise semi-soft threshold function is based on Fine-tuning will be performed, and the fine-tuning rules are as follows:

[0180]

[0181] Where β' is the adjusted threshold scaling factor, used for threshold calculation in the next round of denoising. base The default value is 1.2. This is a reference strength value.

[0182] In this embodiment, to achieve full-process collaborative optimization from sampling to signal purification, the present invention constructs a three-stage collaborative processing system, which integrates the three stages of systematic sampling (M), adaptive baseline correction (B), and optimized denoising (D) into one through a unified data interface, dynamic parameter transfer mechanism, and automated execution engine.

[0183] A unified data interface and status markers are added, that is, each raw spectral data is accompanied by a data status label (sample ID, sampling segment, processing stage). Each module identifies the data status through the label to ensure that the data flows in the correct direction.

[0184] Mean value of fluorescence background intensity of the segment output during sampling phase (M) Used to dynamically set the process noise parameter q of the Kalman filter. 11 The initial range of q. 11 The sensitivity of the control baseline to signal fluctuations needs to be moderately increased under strong fluorescence backgrounds to allow for some baseline flexibility, while it should be kept to a minimum under weak backgrounds to protect weak characteristic peaks. Specifically, the system... q is determined by the following formula 11 Initial value:

[0185]

[0186] Where q base It is a minimum value (e.g., 10). -12 ), where 'a' is the adjustment coefficient. The reference fluorescence intensity is used. This mechanism adapts the initial parameters for baseline correction to the actual fluorescence characteristics of the sample, avoiding blind setting.

[0187] The baseline correction module (B) outputs the corrected spectrum while simultaneously calculating the overall signal-to-noise ratio (SNR).global and the average intensity of the key characteristic peak region I pr The result is passed to the denoising module (D) to guide the scaling of the penalty factor α search range in the RIME optimizer and the fine-tuning of the threshold scaling factor β in the piecewise semi-soft threshold function, thereby achieving adaptive matching of subsequent denoising parameters.

[0188] The terminal quality index (peak area coefficient of variation) calculated by the denoising module (D) i Signal-to-noise ratio (SNR) i If an exception is triggered (such as CV) i (>15%), the system initiates a two-level feedback mechanism:

[0189] Based on the degree of deviation of the coefficient of variation, the laser power P, single exposure time T, and signal accumulation times N during the acquisition phase are adjusted in a coordinated manner. If the deviation is only slightly exceeded (15%), the adjustment will be adjusted accordingly. <CV i If the sample density is ≤20%, then the number of accumulations N should be increased first to suppress random noise using the averaging effect without increasing the risk of sample damage. The new number of accumulations is set as follows:

[0190]

[0191] Where η is the adjustment coefficient, N max This represents the hardware limit; if it is severely exceeded (CV) i If the fluorescence intensity is >20% and the effect is limited after increasing the number of accumulations, then, while ensuring sample safety, gradually increase the laser power P or exposure time T (e.g., in 5% increments) to enhance signal intensity and suppress relative fluorescence fluctuations. Reacquire the spectrum for this region using the optimized parameter combination and recalculate the CV. i If the indicators return to normal, then the parameter combination is marked as the recommended collection parameters for this type of sample; if it is still abnormal, then the operator is prompted to check the optical path or the sample.

[0192] Meanwhile, the system adjusts the overall signal-to-noise ratio (SNR) of the current spectrum. global An empirical model fitted using historical data is called. Calculate the theoretically optimal process noise parameter q opt And combined with the average fluorescence intensity during the sampling phase For the q of the Kalman filter 11 Update:

[0193]

[0194] in As a fluorescence intensity-based fine-tuning function, the clip function limits the update value to a stable operating range (e.g., 10). -14 10 -10This prevents filter divergence. The updated parameters will be used to reprocess all sections of the sample to ensure consistent parameters across the same batch of samples.

[0195] To systematically verify the effectiveness of the proposed "systematic sampling-adaptive baseline correction-optimized denoising" collaborative preprocessing method, dried wolfberry fruit samples from three different producing areas were selected and subjected to comparative analysis of spectral acquisition and processing under the same experimental conditions. The experimental instrument was a 785nm micro Raman spectrometer (laser power 150mW, spectral width 200-3000cm). - ¹, Exposure time 5s, accumulating 2 times), 3 samples were randomly selected from each origin.

[0196] Figure 6 This demonstrates the performance of traditional morphological filtering methods for baseline removal in processing the spectra of Lycium barbarum with high fluorescence background. Traditional methods are prone to underfitting in areas of strong fluorescence, leading to lower baseline estimates (e.g., ...). Figure 6 (As shown by the solid blue line in the middle), the C=C vibration peak of carotenoids in the spectrum after subtraction (solid red line) is at 1158 cm⁻¹. - ¹) Severely weakened.

[0197] The method of this invention uses Kalman filtering and multi-stage extreme smoothing cascade processing to estimate a baseline (blue dashed line) that more closely matches the true lower boundary of the signal, and the spectrum after subtraction (red solid line) performs excellently in preserving weak characteristic peaks. Figure 7 The method of this invention further demonstrates its effectiveness in subtracting nonlinear fluorescence background. The original spectrum (black solid line) was overexposed by the overall rise in fluorescence background, masking numerous weak Raman peaks; after processing using the method of this invention, the estimated baseline (blue solid line) is smooth and continuous, and the corrected spectrum (red solid line) is within 1080 cm⁻¹. - ¹(polysaccharide CO vibration) and 1158 cm⁻¹ - The characteristic peak at point ¹ is clearly visible, the baseline is flat, and there is no residual trend.

[0198] Figure 8 This demonstrates the complete process and effectiveness of the proposed method for further denoising based on baseline correction. The spectrum after baseline correction (solid black line) still contains high-frequency instrument noise and residual fluorescence fluctuations; after VMD-RIME adaptive denoising (solid red line), the noise level is significantly reduced, while key characteristic peaks (such as 1080 cm⁻¹) are significantly denoised. - ¹, 1158 cm - ¹、1525 cm - ¹) The shape is intact and the intensity is fully preserved, without the “step effect” or feature peak weakening phenomenon commonly seen in traditional threshold denoising.

[0199] To objectively evaluate the processing effect of the method of the present invention, the following four quantitative indicators were selected for analysis:

[0200] Coefficient of determination (R²): The goodness of fit between the processed spectrum and the characteristic peak regions in the original spectrum is calculated. The method of this invention achieves 0.9764, which indicates that the processing process preserves the characteristic structure of the original signal well and has high fitting accuracy.

[0201] Mean Square Error (MSE): Measures the deviation of the spectrum in the characteristic peak region before and after processing. The MSE obtained by the method of this invention is 1.1680 × 10³. Combined with the original spectral intensity range, this error value is at a low level, indicating that the processing did not introduce significant distortion.

[0202] Average peak retention rate: with carotenoids at 1158 cm⁻¹ - Taking the characteristic peak as an example, the retention ratio of the processed peak height relative to the original peak height is calculated. The average peak retention rate of the method of this invention is 0.0400 ± 0.0329. Note: This value is the relative retention rate (processed / original), and 0.04 indicates that 4% of the original peak height is retained. This result needs to be understood in context—if there is a high-intensity fluorescence background in the original spectrum, the peak height will naturally decrease after baseline subtraction, so a lower retention rate is normal. The key is whether the characteristic peak is clearly distinguishable. In practical applications, more attention is paid to the relative intensity and shape of the characteristic peak, rather than its absolute value.

[0203] Segmented Signal-to-Noise Ratio (SSNR): Based on the characteristic peak region (1150-1200 cm⁻¹) - ¹) and adjacent noise areas (1800-1900 cm) - ¹) Calculations show that the SSNR after processing by the method of the present invention reaches 13.87 dB, which is about 92.4% higher than the original spectrum of 7.21 dB, proving that the noise is effectively suppressed while the signal strength is maintained.

[0204] Experimental results demonstrate that the Raman spectra processed by the method of this invention exhibit excellent performance across multiple quantitative indicators: the coefficient of determination (R²) reaches 0.9764, indicating that the processing method effectively preserves the characteristic structure of the original signal; the mean square error (MSE) is as low as 1.1680 × 10³, indicating minimal introduced distortion; the segmented signal-to-noise ratio (SSNR) is improved to 13.87 dB, approximately 92% higher than the original spectrum, demonstrating significant noise suppression; key characteristic peaks (such as 1158 cm⁻¹) are effectively suppressed. - ¹) The average peak retention rate remained at a reasonable level after deducting the high-intensity fluorescence background, and the characteristic morphology was complete and clear, providing a high-quality data foundation for subsequent qualitative, quantitative and origin identification analysis.

[0205] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0206] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A Raman spectral processing method based on adaptive baseline correction, characterized in that, Includes the following steps: S1: Obtain the Raman spectrum of the sample to be tested; S2: Perform adaptive Kalman filtering on the Raman spectrum to estimate the initial baseline, and use a real-time constraint mechanism to ensure that the baseline during the estimation process is close to the lower edge of the signal; S2 includes: Multiple rounds of state prediction and updating are performed based on the intensity sequence of the Raman spectrum; After each update, a baseline constraint is applied so that the updated baseline is flush with the lower edge of the signal. The baseline constraint is: ； ； in, Here is the state estimation error covariance matrix. Let k be the observation value at time k. It is a small positive offset. and These represent the maximum and minimum intensities of the original spectrum, respectively, and γ is the offset coefficient used to control how closely the baseline follows the lower edge of the signal; S3: Perform multi-stage extreme smoothing based on the initial baseline, and combine it with a real-time constraint mechanism to extract the baseline trend and obtain the extreme smoothed baseline; S4: Perform independent SG fine smoothing based on the extremely smooth baseline to confirm the final baseline; S5: Perform baseline correction based on the final baseline to obtain the corrected spectrum; S6: Denoising based on the corrected spectrum; specifically including: S61: Normalize the corrected spectrum to obtain a normalized signal; S62: Confirm the optimal VMD decomposition parameters, perform VMD decomposition on the normalized signal, and extract the IMF component; S63: Through correlation analysis, select components that meet the preset threshold from multiple IMF components as valid signal components; S64: The selected effective IMF components are processed using an improved piecewise semi-soft threshold function to perform partitioned noise reduction, resulting in the noise-reduced components. The partitioned noise reduction includes: comparing the amplitude of the effective IMF component with an adaptive threshold, and dividing it into a noise region, a transition region, and a feature region in order from low to high according to the comparison relationship; by controlling the output of the threshold function in each partition, the threshold is forced to zero in the noise region, a shrinkage function is used in the transition region, and an approximate identity mapping is used in the feature region. The adaptive threshold is: ； Where T is the adaptive threshold. This is the threshold scaling factor. A robust estimate of the noise level of the IMF component, where N is the signal length; S65: Perform a weighted summation on each of the denoised components to reconstruct the normalized denoised signal; S66: Perform inverse normalization processing on the normalized denoised signal to obtain the denoised spectrum.

2. The Raman spectroscopy processing method based on adaptive baseline correction according to claim 1, characterized in that, S3 includes: S31: Large Window SG Smoothing: ； in, This is the result after large-window SG smoothing; j is the position index relative to the center point i. This represents the number of points extending from the center point i to the left and right. The convolution coefficients at each point within the convolution kernel algebraic sum, The convolution coefficients are determined by local polynomial least squares fitting; This serves as a preliminary baseline. S32: Gaussian filter: ； ； Where G(j;σ) is the discrete Gaussian kernel function, which itself contains a normalization factor. This ensures that the weighted sum of the discrete kernels approaches 1, so there is no need to introduce an additional normalization coefficient in the formula. σ is the standard deviation, and L is the half-window width of the Gaussian kernel. S33: Moving Average ； in, The number of discrete points participating in the arithmetic mean. The width of the window in the moving average smoothing is half the width. This is the baseline after extreme smoothing.

3. The Raman spectroscopy processing method based on adaptive baseline correction according to claim 1, characterized in that, S4 includes: ； Where i is the index of the current data point being processed. The output sequence represents the final baseline intensity value at index i; is the normalization coefficient; j is the relative position index within the window, ranging from arrive ; represents the Savitzky-Golay convolution coefficients, indicating the contribution weight of each point within the window to the fitted value of the center point; Bextreme[i+j] is the input sequence, representing the intensity value of the baseline at index i+j after extreme smoothing.

4. The Raman spectral processing method based on adaptive baseline correction according to claim 1, characterized in that, The steps also include: S7: Extract key feature peaks based on the denoised spectrum, and calculate the coefficient of variation based on the peak area of ​​the key feature peaks; determine data consistency based on the coefficient of variation, and trigger a feedback mechanism when the consistency is abnormal, and update the baseline correction parameters based on historical data.

5. The Raman spectral processing method based on adaptive baseline correction according to claim 4, characterized in that, The steps also include: calculating the average intensity of the key characteristic peak region based on the corrected spectrum, and fine-tuning the threshold scaling factor based on the average intensity.

6. The Raman spectral processing method based on adaptive baseline correction according to claim 4, characterized in that, The specific steps to confirm the optimal decomposition parameters of VMD include: using the RIME optimization algorithm, with the balance between envelope entropy and reconstruction fidelity as the fitness function, and adaptively determining the optimal decomposition level K and penalty factor α of VMD within a preset search range.

7. The Raman spectral processing method based on adaptive baseline correction according to claim 6, characterized in that, The steps also include: calculating the overall signal-to-noise ratio based on the corrected spectrum, and adjusting the search range of the VMD decomposition parameters during the optimization process based on the overall signal-to-noise ratio.

Images