An adaptive spectral filter

Abstract

An adaptive digital filter is described below. The filter is implemented as follows: Input the spectral measurement signal RS to be processed; perform a Fourier transform on RS and obtain the absolute value AS of its real part; find the upper boundary EP of AS and obtain its logarithmic value LEP; use hierarchical clustering to cluster LEP into 2 major categories and 4 minor categories; the first three minor categories (C1-C3) form one major category, corresponding to spectral signal clusters containing different levels of noise, and the last minor category (C4) forms another major category, representing the noise-dominated region; based on the LEP clustering results, EP is clustered into EP1 and EP2 according to the major category. EP1 includes the contributions of both signal and noise, and the mean value MEP2 of EP2 is the estimated noise intensity; the sequence value (EP1-MEP2) represents the signal components at different frequencies, thus the frequency response value FRV = (EP1-MEP2) / EP1 at different frequencies can be obtained.

Description

Technical Field

[0001] This invention relates to the field of digital filters, and particularly to adaptive digital filters. Background Technology

[0002] In the field of instrumental analysis, spectral data is one of the most commonly acquired types of data. However, spectral data acquired directly by instruments usually requires processing before it can be output as usable measurement results. Currently, digital methods have become the mainstream of instrument development, with digital filters playing a crucial role.

[0003] As is well known, spectral data obtained from different measuring instruments and test objects vary. Existing filters require different parameters to be defined based on the characteristics of the data during data processing. Furthermore, the selection and determination of filter parameters demand a high level of technical skill and experience from the user; finding parameters suitable for the characteristics of the data being processed often requires prior knowledge and a degree of trial and error, resulting in a significant workload.

[0004] Spectral filters utilize the frequency distribution differences between effective information and noise to filter out high-frequency dominant noise, thereby improving the output signal-to-noise ratio. There are two main types of filters: FIR (Finite Impulse Response) and IIR (Infinite Impulse Response). Both roughly define signal throughput as three regions based on frequency range: bandpass (complete throughput), bandstop (complete blocking), and transition region (partial throughput). Certain parameters are artificially set to define the frequency response function (FRF) into these three regions. The main problems with existing filters are: first, it is difficult to accurately distinguish the specific filtering degree of a real signal according to its actual characteristics, limiting accuracy; second, when predefined parameters are not met, repeated adjustments based on output feedback are required until an acceptable level is reached, failing to achieve self-adaptation to the input signal. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide an adaptive digital filter for spectral signal processing that, without requiring parameter definition, can perform data processing functions such as spectral noise reduction and calculus calculations. This filter can be integrated into the hardware and software of a spectrometer, enabling adaptive processing of each acquired raw data, significantly reducing the workload required for user technical expertise and instrument adjustments.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] An adaptive digital filter; the implementation steps of this filter are as follows:

[0008] Step 1: Input the spectral measurement signal RS to be processed;

[0009] Step 2: Perform a Fourier transform on RS and take the absolute value of its real part AS;

[0010] Step 3: Locate the upper boundary EP of AS and take its logarithmic value LEP;

[0011] Step 4: Using hierarchical clustering, LEP is clustered into 2 major categories and 4 minor categories; the first three minor categories (C1-C3) form one major category, corresponding to the spectral signal clustering regions with different levels of noise, and the last minor category (C4) forms one major category, which is the region where noise is absolutely dominant.

[0012] Step 5: Based on the LEP clustering results, EP is divided into two major clusters: EP1 and EP2. EP1 includes the contributions of both signal and noise, and the mean MEP2 of EP2 is the estimated noise intensity.

[0013] Step 6: Since the sequence values ​​(EP1-MEP2) are signal components at different frequencies, the frequency response values ​​FRV = (EP1-MEP2) / EP1 at different frequencies can be obtained.

[0014] Step 7: Apply the hyperbolic cotangent function Sech((aω) to the discrete FRV. b The frequency response function (FRF) is derived from the regression; this FRF already reflects the frequency distribution of the input RS, eliminating the need for further searching and feedback adjustments.

[0015] Step 8: Perform an inverse Fourier transform of the FRF into a convolution kernel function KF;

[0016] Step 9: The direct convolution of RS and KF results in a smoothed filtered output after signal denoising.

[0017] Furthermore, the CKF function, after differentiating or integrating KF, is convolved with RS to obtain the output of the differential or integral processing corresponding to the original input.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0019] This invention proposes to pre-identify multiple pass intervals and band-stop regions with different degrees of pass through hierarchical clustering based on the frequency characteristics of the input signal. By regressing these multiple intervals, a specific frequency response function that conforms to the actual input is calculated. This can effectively solve the key problems existing in current filters, determine a more accurate filtering function with actual input adaptive characteristics, eliminate the need for repeated adjustments after prior input-output feedback, significantly reduce the difficulty of use for users, and achieve accurate and convenient spectral signal filtering processing. Attached Figure Description

[0020] Figure 1The original spectral data and the absolute value of the real part of the FFT transform;

[0021] Figure 2 The upper boundary of the FFT transform data and its magnitude logarithmic value;

[0022] Figure 3 Hierarchical clustering results;

[0023] Figure 4 Classification results of data after FFT transformation;

[0024] Figure 5 Using Sech((aω)) b The frequency response function (FRF) is obtained through regression.

[0025] Figure 6 The FRF is transformed into a convolution kernel function KF by inverse Fourier transform;

[0026] Figure 7 Processing results of full-band data;

[0027] Figure 8 The processing effect of the sharpest local peaks;

[0028] Figure 9 Comparison of noise reduction results for direct Raman spectroscopy measurements using ibuprofen;

[0029] Figure 10 Compare the difference between the original value and the filtered result. Detailed Implementation

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0031] like Figure 1-10 As shown, Example 1

[0032] An adaptive digital filter; the implementation steps of this filter are as follows:

[0033] Step 1: Input the raw data RS to be processed, such as... Figure 1 (a).

[0034] Step 2: Perform an FFT (Fast Fourier Transform) on RS. Figure 1 (b) is the absolute value of its real part AS.

[0035] Step 3: Locate the upper boundary EP of AS, such as Figure 2 (a). Calculate its logarithm, LEP, as follows: Figure 2 (b)

[0036] Step 4: Through hierarchical clustering, LEP is divided into 2 major categories and 4 minor categories. For example... Figure 3 .

[0037] Step 5: Designate the last subcategory as EP2, and the first 3 subcategories as EP1, such as... Figure 4 EP1 includes contributions from both signal and noise, while the mean of EP2, MEP2, is the estimated noise intensity (amplitude).

[0038] Step 6: Since the sequence values ​​(EP1-MEP2) are signal components at different frequencies, the frequency response values ​​(FRV) at different frequencies can be obtained, FRV = (EP1-MEP2) / EP1.

[0039] Step 7: Apply the hyperbolic cotangent function Sech((aω) to the discrete FRV. b The frequency response function (FRF) is regressed, and the regressed FRF is as follows: Figure 5 This FRF can specifically reflect the frequency distribution of the directly input spectral signal RS without the need for further searching and feedback adjustment.

[0040] Step 8: Perform an inverse Fourier transform of the FRF into a convolution kernel function KF, such as... Figure 6 .

[0041] Step 9: Directly convolve RS and KF; the output is the smoothed, denoised output of the original signal. For example... Figure 7 This is the result of processing the original spectrum with a filter. Figure 7 (a) is the smoothed result. Figure 7 (b) is the result of the second derivative calculation. Figure 8 The processing effect of the sharpest local peaks, Figure 8 (a) shows the smoothing effect. Figure 8 (b) is the result of the second derivative calculation.

[0042] from Figure 7 and Figure 8 As can be seen, compared with the results of the most commonly used SG filter (with parameters repeatedly optimized, selected as 3rd order and data window width 15), the results of the adaptive filter of this invention, which is directly calculated, not only ensure the noise reduction effect but also have higher accuracy.

[0043] Furthermore, if the original spectrum differential or integral processing output is required, the differential or integral of KF can be obtained to get the CKF function. The convolution of CKF with RS can then yield the differential or integral processing output corresponding to the original input.

[0044] Example 2

[0045] Collect the raw Raman spectrum of ibuprofen;

[0046] The original spectrum is processed using the filter of this invention. Figure 9The comparison shows the processing results of directly measuring the ibuprofen Raman spectrum using the filter of this invention and SG. From top to bottom, the results are: adaptive direct noise reduction of this invention, noise reduction after SG parameter tuning, and the original measured spectrum. Figure 10 The values ​​represent the difference between the original value and the filtered result. From top to bottom, these are the difference between the original value and the result of this invention, and the difference between the original value and the SG tuning result.

[0047] It can be seen that the adaptive filter of the present invention has a smaller error in the output without human intervention than the output after SG parameter tuning.

[0048] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions conceived without inventive effort should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope defined in the claims.

Claims

1. An adaptive digital filter adjustment method; the steps are as follows: Step 1: Input the spectral measurement signal RS to be processed; Step 2: Perform a Fourier transform on RS and take the absolute value of its real part AS; Step 3: Locate the upper boundary EP of AS and take its logarithmic value LEP; Step 4: Using hierarchical clustering, LEP is clustered into 2 major categories and 4 minor categories; the first three minor categories (C1-C3) form one major category, corresponding to the spectral signal clustering regions with different levels of noise, and the last minor category (C4) forms one major category, which is the region where noise is absolutely dominant. Step 5: Based on the LEP clustering results, EP is divided into two major clusters: EP1 and EP2. EP1 includes the contributions of both signal and noise, and the mean MEP2 of EP2 is the estimated noise intensity. Step 6: Since the sequence values ​​(EP1-MEP2) are signal components at different frequencies, the frequency response values ​​FRV at different frequencies can be obtained as FRV=(EP1-MEP2) / EP1; Step 7: Apply the hyperbolic secant function Sech((aω)) to the discrete FRV. b The frequency response function (FRF) is derived from the regression; this FRF already reflects the frequency distribution of the input RS, eliminating the need for further searching and feedback adjustments. Step 8: Perform an inverse Fourier transform of the FRF into a convolution kernel function KF; Step 9: The direct convolution of RS and KF results in a smoothed filtered output after signal denoising.

Images