A distortion tolerance based baseline removal method

Abstract

A baseline elimination method based on distortion tolerance is used to balance the consistency and flatness of the calculated output and the real signal. The direct effect is to overcome the need to define multiple parameter inputs in the existing method, and the output effect needs to be adjusted repeatedly by visual perception, which is difficult to judge the excessive deduction caused by the overlapping peak valley of the multi-peak and the actual baseline difference, and further leads to a series of problems such as peak distortion. In different application scenarios of Raman and infrared spectrum, accurate baseline elimination effect can be realized. The present application solves two key problems that have plagued the current baseline elimination method: one is that almost no parameters need to be set and adjusted by man (usually set tolerance t>4), and the other is that it has good adaptability to noise or other random interference.

Description

Technical Field

[0001] This invention relates to the field of baseline interference elimination in analytical measurements, and particularly to a baseline elimination method based on distortion tolerance. Background Technology

[0002] Baseline interference is a long-standing key technical problem in analytical measurements, particularly prominent in molecular vibrational spectroscopy detection such as Raman and infrared spectroscopy. A raised baseline reduces the proportion of the true signal, interfering with the correctness and accuracy of qualitative and quantitative detection, and in severe cases, even rendering the analysis results invalid. Numerous baseline elimination methods and techniques have been proposed to address this issue, ranging from manual correction relying on operator's subjective visual judgment to machine recognition and iterative search optimization algorithms based on empirical parameters, and even artificial intelligence (AI)-assisted technologies that have emerged in recent years. However, most methods require setting multiple hyperparameters (such as smoothing windows and iteration counts), and parameter optimization relies on human experience, making the operation cumbersome and difficult to standardize. When faced with complex spectra such as strong noise, overlapping peaks, and severe baseline drift, baseline fitting bias is prone to occur, or effective spectral peaks may be mistakenly identified as baselines and excessively subtracted, leading to a series of problems such as spectral distortion. Some algorithms based on iteration or regularization have high computational complexity, making them difficult to adapt to real-time detection scenarios, while lightweight methods often sacrifice the accuracy of baseline fitting.

[0003] Previous methods have not explicitly considered the determinism of the output. In actual measurements, only the total measurement signal, including the baseline, the true spectrum, and noise, is known; the specific contributions of these three components to the total measurement signal are unknown. For baseline elimination in practical measurement scenarios, a solution to the inverse problem is needed to separate the baseline, but this solution process is ill-posed.

[0004] The core characteristic of ill-posed problems is insufficient constraints, which can lead to situations where there are no solutions, multiple solutions, or unstable solutions. Therefore, it is necessary to introduce reasonable constraints on ill-posed problems, that is, to achieve a balance between the stability, accuracy, and physical rationality of the solution by defining regularization conditions.

[0005] This invention proposes a method to balance the consistency and flatness of the calculated output with the real signal by defining a distortion tolerance for baseline subtraction. The direct effect is that it overcomes the problems of existing methods, which require defining multiple input parameters, manually adjusting the output based on visual perception, and struggling to judge excessive subtraction caused by overlapping peaks and valleys and differences in the actual baseline, leading to peak distortion and other related issues. Summary of the Invention

[0006] To address the problems existing in the prior art, the present invention aims to provide a baseline elimination method based on distortion tolerance. To achieve the above objectives, the technical solution of the present invention is as follows: A baseline elimination method based on distortion tolerance consists of the following steps: Step 1. Define distortion tolerance Directly measured signals ( S The information includes spectral peak information. P ),noise( N ) and baseline ( B ): (1) In the formula S It is known that the other three ( P , N , B The unknown, where any one of them will constitute an ill-posed inverse problem; From the perspective of peak width P , N , B right S Contribution N It has the smallest peak width (highest frequency). P Secondly, B Maximum; taking Raman or infrared as examples, the actual peak shape of the spectrum is the Voigt peak, but for simplicity, a Gaussian peak is usually used as an approximation; assuming 3000 data points are collected, P , N , B The peak widths of the three types of signals are expressed in terms of Gaussian peak equivalents. N =1, P The typical range is [1, 20], while B Usually greater than 500; If yes S Find the second derivative (d S ), then, d S middle N and B The strength ratio will be greater than 10 5 , P and B Strength ratio greater than 10 4 Order of magnitude; that is, the second or higher order derivative can suppress the baseline signal to a very weak level. Based on this fact, if the baseline is not excessively eliminated, it will not cause... P Distortion; therefore, the measured signal S Second derivatives before and after baseline removal ( dS 0, dS 1) It will maintain a very high correlation (using the correlation coefficient). R describe); This invention defines tThe tolerance for distortion eliminated for the baseline, or simply the tolerance; Using Gaussian peak equivalent calculations, the reasonable tolerance is ( t () is greater than 4, that is: (2) Step 2. Convolution trial calculation to eliminate initial baseline The baseline is a signal with a frequency significantly lower than the spectral peaks and noise; adjusting the frequency range of the convolution filter based on a tolerance metric can achieve good low-pass filtering while preserving most low-frequency components; this invention uses a super-Gaussian function to construct the convolution kernel, through direct measurement (… S Convolution with a convolution kernel to achieve low-pass filtering; A kernel function with a very narrow peak width will not cause baseline overfitting; therefore, gradually increasing the peak width of the kernel function can improve the degree of baseline elimination. However, as the peak width of the kernel function increases, it may lead to… P and N Low-frequency components remain in B In the middle; if a tolerance is set, the direct measurement value ( S Low-frequency output () S f In this process, it can be guaranteed that... P and N The component is significantly smaller than B , S and S f The difference S d ( S d = S - S f If the pre-set tolerance level is met, it can be considered that... S d Distortion-free preservation P and N , S d It has been eliminated S Most of the baselines in the middle ( B )contribute; Step 3. b -Spline interpolation flattening eliminates the convolution baseline of the spectral lines. In principle, further... S d Convolution filtering can continue to approximate the pair B The elimination, but due to B The contribution is already very small, which will result in low computational efficiency; because B The contribution was minimal, requiring only minor adjustments. S dThis invention proposes that, under tolerance constraints, by... (The sentence is incomplete and requires more context to translate accurately.) b - spline fit S d The lower edge is subtracted, and the resulting baseline-reduced flat spectral line is output.

[0007] Furthermore, the above method is specifically as follows: Step 1 uses the superGaussian function ( G Construct low-pass filter convolution kernels ( K ) The simplified expression for the superGaussian function is: (3) in, x As the independent variable, w For peak width, when k =2 is the ordinary Gaussian function. k >2 is a super-Gaussian function, which has a more sensitive frequency cutoff effect; Low-pass convolution kernel K It is obtained by inverse Fourier transforming the frequency response function in the frequency domain after filtering out noise and high-frequency signals to the time domain; (4) in, G It is a superGaussian function. n It is a constant reflecting the noise intensity. iFFT It is the inverse Fourier transform operation; The fixed initial convolution kernel parameters are as follows: w 0=1, k =4, n =0.05; in subsequent steps, only the following needs to be changed. w The other parameters remain unchanged; Step 2: Obtain the direct signal ( S The second derivative benchmark (d) S 0) According to S1, choose the wider one. w The 0-structure can filter out most noise. N )of G The conversion yields the convolution kernel. K 0, i.e., the zeroth-order convolution kernel; calculation K Second difference of 0 K d That is, a 2-order convolution kernel, obtained through convolution (conv) dS 0; (5) Step 3: Eliminate the initial baseline through convolution trial calculation (1) Gradually increase in step size of 0.01. w Obtaining 0th-order kernels of different widths K i ;Depend on S and K i Convolution output S f And thus obtain S d ; (6) (2) Calculation S d The second derivative d S 1 (7) (3) Calculate d S 0 and d S Correlation coefficient of 1 ; (4) According to the definition t ,judge Is it greater than or equal to 1-10? -t If so, repeat steps 3(1)-(3) until the result is less than 1-10. -t Then, the corresponding low-frequency signal is output. S f ), that is, the initial baseline obtained by convolution calculation. B c Corresponding S d The sequence is the measurement signal after baseline elimination via convolution; Step 4 b -Spline interpolation flattening eliminates the convolution baseline of the spectral lines. (1) According to S d The number of data points M in the sequence defines the sequential search range. m (in m Sizes can be selected from M / 3 to M / 40. Find out m Within range S d The minimum values ​​constitute a discrete set M of interval minimum values. min Points within this set can be obtained using b-spline interpolation. ; (2) From S deduct and ,get S real ; (8) (3) CalculationS real The second derivative d S 2 (9) (4) Calculate d S 0 and d S Correlation coefficient of 2 ; (5) Based on the defined distortion tolerance t ,judge Is it greater than 1-10? -t If so, repeat steps 4(2)-(4) until the result is less than 1-10. -t Then, the corresponding low-frequency signal is output. ), corresponding S real The sequence is the measurement signal after the baseline has been finally eliminated.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a baseline elimination method based on distortion tolerance, which demonstrates significant technical advantages in practical applications compared to the most commonly used airPLS algorithm. This method achieves accurate baseline elimination in various practical application scenarios of Raman and infrared spectroscopy. This invention solves two key problems currently plaguing baseline elimination methods: first, it only requires setting a universal tolerance threshold (usually t>4), significantly reducing the reliance on operator expertise and improving the ease of use and consistency of results; second, it has excellent adaptability to noise or other random interference, accurately distinguishing true signal peaks from baseline interference even in complex spectral backgrounds, effectively avoiding over-correction or under-correction and ensuring the integrity of spectral features.

[0009] This method, through strict constraints of second derivative characteristics, ensures that the true spectral peaks are not distorted during baseline elimination, further improving the accuracy and reliability of the analysis results and providing a more efficient and reliable technical solution for the spectral analysis of complex samples. Attached Figure Description

[0010] Figure 1 A stable signal simulated using Gaussian peak superposition; Figure 2 The baseline is simulated using superimposed sine curves; Figure 3 Analog mixed signals; Figure 4 The processing results of the method of the present invention on analog signals; Figure 5 Changes in the second derivative during the calculation process; Figure 6 Comparison of baseline correction results in low-noise scenarios; Figure 7 Comparison of baseline correction results in high-noise scenarios; Figure 8 Application effects on high background Raman signals; Figure 9 The application effect on Raman spectral signals with weak baselines but many fluctuations; Figure 10 The effects of applying infrared spectroscopy. Detailed Implementation

[0011] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: like Figure 1-10 As shown, Example 1: Baseline Elimination of Analog Signals 1) Constructing analog signals (1) Nine Gaussian peaks were superimposed to simulate a stationary signal. P ,like Figure 1 .

[0012] (2) Simulate baseline fluctuations by superimposing four sine curves of different frequencies. B ,like Figure 2 .

[0013] (3) Representing the stable overlapping peaks that are true spectral features P Overlay the baseline B Additionally, add white noise with an intensity of one-thousandth. N The resulting mixed signal, containing baseline fluctuations, is obtained. S ,like Figure 3 Baseline drift and noise interference partially mask the true peak characteristics.

[0014] (4) Set the tolerance t=5, and proceed according to the implementation steps of this invention. S The process was performed, and the result was as follows: Figure 4 As shown, where S Mixed signals including baseline S d Subtracting the low-pass filtered signal, B c The output of the low-pass filter, S real for b- The signal after spline interpolation, spectral line convolution, baseline elimination, and correction is corrected. S f + The baseline is obtained by low-pass filtering and b-spline correction.

[0015] The output results show that the obtained baseline almost perfectly matches the actual baseline. The baseline components initially separated by low-pass filtering... B c The signal exhibits a trend highly consistent with the true baseline; the signal after initial baseline removal... S d The outline of the true peak has begun to emerge, but some baseline fluctuations still remain; b- The final signal obtained after spline interpolation, spectral line convolution, and baseline elimination S real Baseline interference was precisely eliminated, revealing the true signal. P The position, intensity, and contour are all fully preserved, and noise is effectively suppressed; the fitted baseline S f + The result almost perfectly overlaps with the real baseline, verifying the accurate baseline fitting capability of the method of this invention.

[0016] The method of this invention can accurately separate and eliminate baseline interference from noisy, baseline-containing mixed signals, while fully preserving the characteristics of the true spectral peaks, providing highly reliable preprocessing results for subsequent spectral analysis.

[0017] Under strict tolerance constraints, the baseline elimination process hardly alters the second derivative characteristics of the original signal, thus ensuring the preservation of true spectral peaks. P No distortion occurs; only the baseline is precisely removed. B The interference. The change of the second derivative during the calculation process is as follows: Figure 5 As shown in the figure, this figure visually verifies the rigor of the baseline correction method. Throughout the calculation process, the corrected second derivative is almost identical to the original second derivative, which is precisely because a very strict tolerance was set. t ( t =5), baseline correction only eliminates baseline interference and does not affect the second derivative characteristics of the true signal peak.

[0018] In a low-noise interference test scenario, based on the error with the true value, the airPLS algorithm is optimized by searching and determining a set of optimal parameters: regularization coefficient. =10 4 The polynomial order is 2, the weight coefficient wep is 0.16, the penalty factor p is 0.3, and the number of iterations is 20. Under these parameters, the result calculated by airPLS is consistent with the result of one parameter definition in the method of this invention ( t The baseline correction results of (=5) show almost no difference in output quality. Figure 6 This is a comparison between the baseline correction results of the airPLS algorithm after parameter optimization and the method of this invention. Figure 6As can be seen, under low-noise conditions, the baseline correction effects of the two methods are almost identical, both effectively eliminating baseline interference and clearly revealing the characteristics of the true spectral peaks. This indicates that in an ideal low-noise environment, the method of this invention (requiring only a single tolerance parameter t=5) can achieve correction accuracy comparable to that of the parameter-optimized airPLS.

[0019] However, by scaling the noise intensity to one percent, the spectral interference of more realistic complex samples was simulated. Using the same parameters, the output results of the two methods showed significant differences. Figure 7 The comparison shows the baseline correction results under high noise conditions. The signal corrected by the airPLS algorithm exhibits significant noise residue and baseline fluctuations, with the intensity and contour of some true peaks being masked by noise interference, failing to accurately reflect the characteristics of the original signal. Even under high noise interference, the method of this invention can still accurately separate the baseline from the noise, and the corrected signal closely matches the true signal, with the position, intensity, and contour of the true spectral peaks fully preserved, demonstrating excellent anti-interference capabilities.

[0020] The results of the simulation test cases with completely known data show that the method of the present invention is significantly better than the most commonly used airPLS algorithm in terms of parameter robustness and anti-interference ability, and is more suitable for processing spectral data of complex real samples.

[0021] Example 2: Application effect of the method of the present invention on actual measured signals (1) Application effect of high background Raman signal Raman spectra of steviol glycosides were acquired using a Raman spectrometer (excitation wavelength 532 nm). The original spectra were then baseline-corrected using the method of this invention. The results are as follows: Figure 8 As shown in the figure, the overall measurement signal is elevated, the baseline background is high and contains fluorescence interference, and the true characteristic peaks are masked by the high background, making them difficult to identify directly. After processing by the baseline correction method of this invention, the high background and fluorescence interference are accurately eliminated, the baseline is flattened, and the characteristic peaks of steviol glycosides are clearly revealed, achieving the effect of extracting effective Raman signals from high background.

[0022] (2) Application effect of Raman spectroscopy with weak baseline but many fluctuations Data was collected using a Raman spectrometer (excitation wavelength 532 nm). L The Raman spectrum of arabinose was obtained by baseline correction of the original spectrum using the method of this invention, and the results are as follows: Figure 9As shown in the figure, although the baseline is not high overall, it exhibits frequent ups and downs. This irregular baseline fluctuation can interfere with the judgment of true characteristic peaks, easily misinterpreting the baseline as a signal peak, and also masking some weaker characteristic peaks. After baseline correction, the originally fluctuating baseline is corrected to a stable baseline, and the characteristic peaks of L-arabinose, including the weak peaks masked by the baseline fluctuations, are clearly preserved and highlighted. The intensity and position information of the peaks are accurately restored, indicating that this method also has a good correction ability for weak but highly fluctuating baselines.

[0023] (3) Application effects of infrared spectroscopy A sample of chili powder of a certain variety was mixed with potassium bromide at a ratio of 1:100, ground, pressed into tablets, and infrared spectral data were collected (500~4000 cm⁻¹). -1 The resolution is 4 cm. -1 (3 scans) Figure 10 This demonstrates the application effect of the method of this invention on the infrared spectrum of chili powder. As can be seen from the figure, there is a significant baseline drift and irregular fluctuations in different wavenumber ranges, causing the infrared absorption peaks of the sample to be interfered with by the baseline, making it difficult to accurately analyze the relative intensity and true position of the peaks. After baseline correction, the baseline was adjusted to a stable level, and the outline and intensity of the characteristic infrared absorption peaks of chili powder were clearly presented, eliminating the influence of baseline tilt and fluctuations on the signal.

[0024] The baseline correction method of this invention can accurately eliminate baseline drift, tilt and fluctuation interference in both Raman spectroscopy (high background and weak baseline fluctuation scenarios) and infrared spectroscopy, allowing the true spectral characteristic peaks of the sample to be clearly displayed, providing a reliable data foundation for subsequent qualitative and quantitative analysis.

[0025] This invention solves two key problems that currently plague baseline elimination methods: first, it requires almost no manual setting and adjustment of parameters (usually a tolerance t>4 is sufficient); second, it has good adaptability to noise or other random interference.

[0026] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. A baseline elimination method based on distortion tolerance, characterized in that, It consists of the following steps: Step 1. Define distortion tolerance: Directly measured signals ( S The information includes spectral peak information. P ),noise( N ) and baseline ( B ): (1)； In the formula S It is known that the other three ( P , N , B The unknown, where any one of them will constitute an ill-posed inverse problem; From the perspective of peak width P , N , B right S Contribution N It has the smallest peak width (highest frequency). P Secondly, B Maximum; taking Raman or infrared as examples, the actual peak shape of the spectrum is the Voigt peak, but for simplicity, a Gaussian peak is usually used as an approximation; assuming 3000 data points are collected, P , N , B The peak widths of the three types of signals are expressed in terms of Gaussian peak equivalents. N =1, P The typical range is [1, 20], while B Usually greater than 500; If yes S Find the second derivative (d S ), then, d S middle N and B The strength ratio will be greater than 10 5 , P and B Strength ratio greater than 10 4 Order of magnitude; that is, second-order or higher derivatives can suppress the baseline to a very weak degree. Based on this fact, if the baseline is not excessively eliminated, it will not cause... P Distortion; therefore, the measured signal S Second derivatives before and after baseline removal ( dS 0, dS 1) It will maintain a very high correlation (using the correlation coefficient). R describe); This invention defines t The tolerance for distortion eliminated for the baseline, or simply the tolerance; Using Gaussian peak equivalent calculations, the reasonable tolerance is ( t () is greater than 4, that is: (2)； Step 2. Convolution trial calculation to obtain preliminary baseline elimination: The baseline is a signal with a frequency significantly lower than the spectral peaks and noise; adjusting the frequency range of the convolution filter based on a tolerance metric can achieve good low-pass filtering while preserving most low-frequency components; this invention uses a super-Gaussian function to construct the convolution kernel, through direct measurement (… S Convolution with a convolution kernel to achieve low-pass filtering; A kernel function with a very narrow peak width will not cause baseline overfitting, and gradually increasing the peak width of the kernel function can improve the elimination effect; however, as the peak width of the kernel function increases, it may lead to... P and N Low-frequency components remain in B In the middle; if a tolerance is set, the direct measurement value ( S Low-frequency output () S f In ), guarantee P and N The component is significantly smaller than B , S and S f The difference S d ( S d = S - S f If the tolerance requirement is met, then it can be considered that... S d Distortion-free preservation P and N , S d It has been eliminated S Most of the baselines in B )contribute; Step 3. b -Elimination of convolution baseline of spectral lines after spline interpolation flattening: In principle, further S d Convolution filtering can continue to approximate the pair B The elimination, but due to B The contribution is already very small, which will result in low computational efficiency; because B The contribution was minimal, requiring only minor adjustments. S d This invention proposes that, under tolerance constraints, by... (The sentence is incomplete and requires more context to translate accurately.) b - spline fit S d The lower edge is subtracted, and the resulting baseline-reduced flat spectral line is output.

2. The method according to claim 1, characterized in that, The method is as follows: Step 1 uses the superGaussian function ( G Construct low-pass filter convolution kernel ( K ): The simplified expression for the superGaussian function is: (3)； in, x As the independent variable, w For peak width, when k =2 is the ordinary Gaussian function. k >2 is a super-Gaussian function, which has a more sensitive frequency cutoff effect; Low-pass convolution kernel K It is obtained by inverse Fourier transforming the frequency response function in the frequency domain after filtering out noise and high-frequency signals to the time domain; (4)； in, G It is a superGaussian function. n It is a constant reflecting the noise intensity. iFFT It is the inverse Fourier transform operation; The fixed initial convolution kernel parameters are as follows: w 0=1, k =4, n =0.05; in subsequent steps, only the following needs to be changed. w The other parameters remain unchanged; Step 2: Obtain the direct signal ( S The second derivative benchmark (d) S 0): First, choose the wider one. w The 0-structure can filter out most noise. N )of G The conversion yields the convolution kernel. K 0, i.e., the zeroth-order convolution kernel; calculation K Second difference of 0 K d That is, a 2-order convolution kernel, obtained through convolution (conv) dS 0; (5)； Step 3: Eliminate the initial baseline through convolution trial calculation: (1) Gradually increase in step size of 0.

01. w Obtaining 0th-order kernels of different widths K i ;Depend on S and K i Convolution output S f And thus obtain S d ; (6)； (2) Calculation S d The second derivative d S 1： (7)； (3) Calculate d S 0 and d S Correlation coefficient of 1 ; (4) According to the definition t ,judge Is it greater than or equal to 1-10? -t If so, repeat steps 3(1)-(3) until the result is less than 1-10. -t Then, the corresponding low-frequency signal is output. S f ), that is, the initial baseline obtained by convolution calculation. B c Corresponding S d The sequence is the measurement signal after baseline elimination via convolution; Step 4 b -Elimination of convolution baseline of spectral lines after spline interpolation flattening: (1) According to S d The number of data points M in the sequence defines the sequential search range. m (in m Sizes can be selected from M / 3 to M / 40. Find out m Within range S d The minimum values ​​constitute a discrete set M of interval minimum values. min Points within this set can be obtained using b-spline interpolation. ; (2) From S deduct and ,get S real： (8)； (3) Calculation S real The second derivative d S 2： (9)； (4) Calculate d S 0 and d S Correlation coefficient of 2 ; (5) Based on the defined distortion tolerance t ,judge Is it greater than 1-10? -t If so, repeat steps 4(2)-(4) until the result is less than 1-10. -t Then, the corresponding low-frequency signal is output. ), corresponding S real The sequence is the measurement signal after the baseline has been finally eliminated.

Images