Signal filtering method and device based on MFDE abnormal point elimination

Abstract

The invention provides a signal filtering method and device based on MFDE abnormal point elimination, and the method comprises the steps: receiving a to-be-processed signal, and carrying out the short-time Fourier transform of the to-be-processed signal, and obtaining an initial time-frequency spectrum of the to-be-processed signal; determining a target abnormal point in the initial time-frequency spectrum by using an MFDE multi-scale fluctuation dispersion entropy algorithm, an LOF local abnormal factor algorithm and an interior point method; and filtering the target abnormal point from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and performing inverse short-time Fourier transform on the target time-frequency spectrum to obtain a target signal. According to the scheme, abnormal points and real signals are effectively distinguished, and the filtering effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of signal processing, and in particular to a signal filtering method and device based on MFDE outlier elimination. Background Art

[0002] In the field of signal processing, filtering is a crucial step, which aims to remove noise and interference from the original signal to extract useful information. With the continuous development of science and technology, the complexity of signal processing and the requirements for accuracy are increasing.

[0003] Traditional filtering methods, such as arithmetic mean filtering and first-order lag filtering, can smooth signals and reduce noise to a certain extent. However, when processing signals containing a large number of outliers (such as pulse noise, mutations, etc.), the appearance of outliers is often irregular and difficult to predict, which makes it difficult to effectively distinguish outliers from real signals, resulting in poor filtering effect. Summary of the invention

[0004] In view of this, the object of the present invention is to provide a signal filtering method and device based on MFDE outlier elimination, so as to effectively distinguish outliers from real signals and improve the filtering effect.

[0005] In a first aspect, an embodiment of the present application provides a signal filtering method based on MFDE outlier elimination, the method comprising:

[0006] Receiving a signal to be processed, and performing short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed;

[0007] The target abnormal point in the initial time-frequency spectrum is determined by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local abnormal factor algorithm and the interior point method;

[0008] The target abnormal point is filtered out from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and an inverse short-time Fourier transform is performed on the target time-frequency spectrum to obtain a target signal.

[0009] Optionally, the method of determining the target abnormal point in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation spread entropy algorithm, the LOF local abnormal factor algorithm and the interior point method includes:

[0010] According to the initial time-frequency spectrum, an initial MFDE scatter plot is generated using the MFDE multi-scale fluctuation scatter entropy algorithm;

[0011] Determine the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method;

[0012] Based on the optimal anomaly ratio parameter, the target anomaly point is determined using the LOF local anomaly factor algorithm.

[0013] Optionally, generating an initial MFDE scatter plot using the MFDE multi-scale fluctuation scatter entropy algorithm according to the initial time-frequency spectrum includes:

[0014] Segmenting the initial time-frequency spectrum to obtain subsequences under different scale factors;

[0015] Calculate the FDE fluctuation spread entropy of each subsequence, and determine the MFDE multi-scale fluctuation spread entropy based on the FDE fluctuation spread entropy of each subsequence;

[0016] The initial MFDE scatter plot is generated based on the MFDE multi-scale fluctuation scatter entropy.

[0017] Optionally, the determining the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method includes:

[0018] Based on the initial anomaly ratio parameter, the initial anomaly points in the initial MFDE scatter plot are determined using the LOF local anomaly factor algorithm;

[0019] Removing the initial outliers from the initial MFDE scatter plot to obtain an optimized MFDE scatter plot;

[0020] According to the optimized MFDE scatter plot, the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm is determined using the interior point method algorithm.

[0021] Optionally, the step of determining the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm using an interior point algorithm according to the optimized MFDE scatter plot includes:

[0022] Calculating an initial convex hull area in the initial MFDE scatter plot and an optimized convex hull area in the optimized MFDE scatter plot;

[0023] Constructing the objective function of the interior point method algorithm based on the initial convex hull area and the optimized convex hull area;

[0024] Setting a barrier function of the interior point method algorithm, wherein the barrier function includes an anomaly ratio parameter in the LOF local anomaly factor algorithm;

[0025] Based on the obstacle function, an abnormality ratio parameter that can minimize the optimized convex hull area when satisfying the objective function is determined as the optimal abnormality ratio parameter.

[0026] In a second aspect, an embodiment of the present application provides a signal filtering device based on MFDE outlier elimination, the device comprising:

[0027] A signal receiving module, used for receiving a signal to be processed, and performing a short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed;

[0028] An outlier determination module is used to determine the target outlier points in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local outlier factor algorithm and the interior point method;

[0029] The signal optimization module is used to filter out the target abnormal point from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and perform an inverse short-time Fourier transform on the target time-frequency spectrum to obtain a target signal.

[0030] The technical solution provided by this application includes but is not limited to the following beneficial effects:

[0031] This application achieves high-precision identification of abnormal points in the signal by comprehensively using the MFDE multi-scale fluctuation spread entropy algorithm, the LOF local anomaly factor algorithm and the interior point method. This multi-algorithm collaborative approach can capture the characteristics of abnormal points more comprehensively and reduce the probability of misjudgment and missed judgment compared to a single algorithm, thereby ensuring the accuracy and reliability of the filtering process. Furthermore, due to the ability to accurately identify and exclude abnormal points, this method has significantly improved the filtering effect compared to traditional methods, making the filtered signal smoother and closer to the true characteristics of the original signal, thereby improving the quality and availability of the signal. In addition, due to the use of multiple algorithms working together, this design enables the algorithm to maintain a high robustness when facing different signal environments and abnormal point distributions. Whether it is a complex signal or a simple signal, this method can achieve good filtering effects, thereby improving the adaptability and application scope of the filtering method.

[0032] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 A flow chart of a signal filtering method based on MFDE outlier elimination provided in the first embodiment of the present invention is shown;

[0035] Figure 2 A flow chart of a method for determining a target abnormal point provided by the first embodiment of the present invention is shown;

[0036] Figure 3 A flow chart of a MFDE scatter plot generation method provided in Embodiment 1 of the present invention is shown;

[0037] Figure 4 A flow chart of a method for determining an optimal abnormality ratio parameter provided by the first embodiment of the present invention is shown;

[0038] Figure 5 A flowchart of a specific optimal abnormality ratio parameter method provided by the first embodiment of the present invention is shown;

[0039] Figure 6 A flowchart of a specific signal filtering method based on MFDE outlier elimination provided in the first embodiment of the present invention is shown;

[0040] Figure 7 A schematic diagram of a source signal frequency domain MFDE scatter plot provided by the first embodiment of the present invention is shown;

[0041] Figure 8 A schematic diagram of a mixed signal frequency domain MFDE scatter plot provided by the first embodiment of the present invention is shown;

[0042] Fig. 9 A schematic diagram of a scatter plot of a filtered frequency domain MFDE provided in the first embodiment of the present invention is shown;

[0043] Fig.10 A schematic diagram of a source signal spectrum provided by Embodiment 1 of the present invention is shown;

[0044] Fig.11 A schematic diagram of a mixed signal spectrum provided by the first embodiment of the present invention is shown;

[0045] Fig.12 A schematic diagram of a filtered signal spectrum provided by the first embodiment of the present invention is shown;

[0046] Fig.13 A schematic diagram of a source signal time domain diagram provided by the first embodiment of the present invention is shown;

[0047] Fig.14 A schematic diagram of a mixed signal time domain diagram provided by the first embodiment of the present invention is shown;

[0048] Fig.15 A schematic diagram of a time domain diagram of a filtered signal provided by the first embodiment of the present invention is shown;

[0049] Fig.16 A schematic diagram of the structure of a signal filtering device based on MFDE outlier elimination provided in the second embodiment of the present invention is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0051] Embodiment 1

[0052] To facilitate understanding of this application, Figure 1 The flowchart of a signal filtering method based on MFDE outlier elimination provided in the first embodiment of the present invention is shown to describe the content of the first embodiment of the present application in detail.

[0053] See also Figure 1 As shown, Figure 1 A flow chart of a signal filtering method based on MFDE outlier elimination provided in the first embodiment of the present invention is shown, wherein the method comprises steps S101 to S103:

[0054] S101: receiving a signal to be processed, and performing short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed.

[0055] Specifically, a signal to be processed is received, which can come from various sources, such as sensor data, audio signals, communication signals, etc., and then the signal is subjected to STFT short-time Fourier transform to generate an initial time spectrum of the signal, which shows the frequency components of the signal at different time points.

[0056] Specifically, the process of short-time Fourier transform of the input signal is as follows:

[0057] The mathematical expression of STFT can be written as:

[0058]

[0059] Where: X(tt,f) is the STFT result at time index tt and frequency f; u[t+ttL] is the discrete time signal after the shift; ttL is the position of the window function on the time axis; w[t] is the window function; t is time; L is the number of overlapping samples; M is the step size of the window function sliding on the time axis, generally Nw-L; Nw is the length of the window function; K is equal to the number of FFT points; F s is the sampling frequency; fk is the frequency index, fk=0,1,2,...,M-1; the relationship between the actual frequency f and the frequency index fk is Here, f ranges from 0 to (Nyquist frequency); N is the index of the last sample point in each time frame, and n is the sequence number of the sample points in each time frame.

[0060] S102: Determine the target abnormal point in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local abnormal factor algorithm and the interior point method.

[0061] Specifically, the MFDE multi-scale fluctuation dispersion entropy algorithm can quantify the fluctuation dispersion characteristics of signals at different scales; the LOF local anomaly factor algorithm identifies anomalies based on density calculation of local density ratios; and the interior point method iteratively optimizes the optimal local anomaly factor algorithm parameters. Combining these three, the target anomaly points in the initial time-frequency spectrum can be determined.

[0062] S103: filtering out the target abnormal point from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and performing an inverse short-time Fourier transform on the target time-frequency spectrum to obtain a target signal.

[0063] Specifically, the target outlier points determined in step S102 are filtered out from the initial time-frequency spectrum to obtain the target time-frequency spectrum. The purpose of filtering out the outliers is to reduce noise and interference and improve the signal-to-noise ratio and clarity of the signal. Next, the target time-frequency spectrum is subjected to an inverse short-time Fourier transform (ISTFT) to convert the time-frequency spectrum back to the time domain representation of the original signal to reconstruct the target signal. This step obtains the target signal with outliers filtered out and reconstructed, which is usually superior to the original signal to be processed in terms of quality and clarity.

[0064] Specifically, ISTFT is used to reconstruct the original discrete-time signal from the STFT coefficients. The mathematical expression of ISTFT is:

[0065]

[0066] Where: xafter[t] is the reconstructed discrete-time signal, and Xflrf is the time-frequency spectrum after time domain filtering.

[0067] Through the above steps, the quality and clarity of the signal can be improved through short-time Fourier transform, outlier detection and inverse short-time Fourier transform.

[0068] In an alternative embodiment, see Figure 2 As shown, Figure 2 A flow chart of a method for determining a target outlier point provided in the first embodiment of the present invention is shown, wherein the method for determining the target outlier point in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation spread entropy algorithm, the LOF local outlier factor algorithm and the interior point method comprises steps S201 to S203:

[0069] S201: Generate an initial MFDE scatter plot using the MFDE multi-scale fluctuation scatter entropy algorithm according to the initial time-frequency spectrum.

[0070] Specifically, first, according to the initial time-frequency spectrum, the MFDE multi-scale fluctuation scatter entropy algorithm is used to calculate the MFDE value of each point in the time-frequency spectrum. After calculating the MFDE value of each point, these values ​​are expressed in the form of a scatter plot to obtain an initial MFDE scatter plot. In this scatter plot, each point represents a point in the time-frequency spectrum, its horizontal axis can be time or frequency, and the vertical axis is the MFDE value of the point.

[0071] S202: Determine the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using an interior point method.

[0072] Specifically, the interior point method is used to analyze the initial MFDE scatter plot to determine the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm. By analyzing the distribution characteristics of the points in the scatter plot, the interior point method can estimate the distribution range of normal points and determine a reasonable anomaly ratio parameter based on this. This parameter will be used in the LOF algorithm to distinguish normal points from anomalies.

[0073] S203: Based on the optimal anomaly ratio parameter, the target anomaly point is determined using the LOF local anomaly factor algorithm.

[0074] Specifically, based on the optimal anomaly ratio parameter determined in step S202, the LOF algorithm is used to identify anomalies by calculating the local density ratio of each point to its neighboring points. In this process, the algorithm considers factors such as the number of neighbors of each point, the distance between neighbors, and the density distribution between neighbors. Based on this information, the algorithm assigns a LOF value to each point, which reflects the degree of abnormality of the point relative to its neighboring points. Finally, according to a preset LOF threshold (usually related to the optimal anomaly ratio parameter), it is determined which points are target anomalies. These points usually have higher LOF values, indicating that their distribution characteristics in the time-frequency spectrum are significantly different from those of normal points.

[0075] In summary, these three steps constitute a complete outlier detection process, which aims to accurately identify the target outliers in the initial time-frequency spectrum by combining the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local anomaly factor algorithm and the interior point method.

[0076] In an alternative embodiment, see Figure 3 As shown, Figure 3 A flow chart of a MFDE scatter plot generation method provided by the first embodiment of the present invention is shown, wherein the initial MFDE scatter plot is generated according to the initial time-frequency spectrum using the MFDE multi-scale fluctuation scatter entropy algorithm, including steps S301 to S303:

[0077] S301: Segment the initial time-frequency spectrum to obtain subsequences under different scale factors.

[0078] Specifically, the initial time-frequency spectrum is segmented. The purpose of segmentation is to divide the time-frequency spectrum into subsequences under different scale factors for subsequent multi-scale analysis. The scale factor can be a preset numerical range used to control the coarseness of the segmentation. By adjusting the scale factor, time-frequency subsequences of different granularities can be obtained.

[0079] Specifically, take the data of a single frequency band in the STFT time-frequency spectrum as an example x=X(:,1).

[0080] Given the original signal {x i}, where i = 1, 2, ..., N, and use the floor function to divide it into non-overlapping subsequences. For each scale factor τ, calculate the mean of each subsequence and generate a new sequence:

[0081]

[0082] where τ is the scale factor and τ = 1, 2, …, s, flo(·) represents the floor function, N is the signal length, s is the maximum number of scale factors, and x b represents the bth data point in the original signal.

[0083] S302: Calculate the FDE fluctuation spread entropy of each subsequence, and determine the MFDE multi-scale fluctuation spread entropy based on the FDE fluctuation spread entropy of each subsequence.

[0084] Specifically, for each subsequence, the FDE value of the subsequence is calculated, and then these FDE values ​​are combined to obtain the MFDE multi-scale fluctuation dispersion entropy. The MFDE value of each time-frequency spectrum point at different scales forms a multidimensional vector, which is used to describe the fluctuation dispersion characteristics of the point at different scales.

[0085] Specifically, under each scale factor τ, the fluctuation dispersion entropy (FDE) is calculated for each subsequence. The time series x is mapped to y according to the normal distribution function:

[0086]

[0087] y o is the oth data point in the mapped time series. o The value of is mapped to the interval [1,c], in, represents the oth value of the classification time series. round means rounding the numbers, and c is the category. Calculate the embedding vector

[0088]

[0089] Will Mapping to wave patterns Compute the relative frequency of each wave mode:

[0090]

[0091] Calculate the FDE of each subsequence with the scaling factor τ and obtain the multi-scale fluctuation dispersion entropy (MFDE):

[0092]

[0093] MFDE(x,n,c,d,τ)=FDE(x (τ) ,n,c,d),forτ=1,…s

[0094] s is the number of maximum scale factors. Finally, the multi-scale fluctuation dispersion entropy of each frequency band is obtained:

[0095] xfre(f,τ)=MFDE(X(:,f))

[0096] S303: Generate the initial MFDE scatter plot based on the MFDE multi-scale fluctuation scatter entropy.

[0097] Specifically, the MFDE multi-scale fluctuation scatter entropy calculated in step S302 is used to generate an initial MFDE scatter plot. The horizontal axis of the scatter plot can be time or frequency (depending on the representation of the time-frequency spectrum), and the vertical axis is the MFDE value of the corresponding point at different scales (or a certain statistic of the MFDE value, such as the mean, median, etc.).

[0098] Specifically, let tx = xfre(:, 1) ty = xfre(:, 2) and construct a scatter plot with the x coordinate of the point set Z as tx and the y coordinate as ty, where Z = {xs i},xs i =(tx i ,ty i ),i=1,2,...,fmax; fmax is the maximum value of the frequency domain length in the time-frequency domain. xsi is the coordinate point in the scatter plot.

[0099] The initial MFDE scatter plot can intuitively display the fluctuation dispersion characteristics of each point in the time-frequency spectrum at different scales, which is helpful to identify outliers using the LOF local anomaly factor algorithm and interior point method in subsequent steps.

[0100] In an alternative embodiment, see Figure 4 As shown, Figure 4 A flow chart of a method for determining an optimal anomaly ratio parameter provided by the first embodiment of the present invention is shown, wherein the method of determining the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter diagram using the interior point method includes S401 to S403:

[0101] S401: Based on the initial anomaly ratio parameter, the initial anomaly points in the initial MFDE scatter plot are determined using the LOF local anomaly factor algorithm.

[0102] Specifically, first set an initial anomaly ratio parameter to indicate what proportion of data points are considered potential outliers in the LOF algorithm. Then, the LOF local anomaly factor algorithm is used to calculate the LOF value of each data point based on this initial anomaly ratio parameter and the data points in the initial MFDE scatter plot. The LOF value reflects the degree of deviation of the local density of the data point relative to its neighboring points. Data points with higher LOF values ​​are considered potential outliers because they have significant differences in density distribution from the surrounding data points. Then, the LOF value and a preset threshold (usually related to the initial anomaly ratio parameter) are used to determine which data points are initial outliers.

[0103] S402: removing the initial outliers from the initial MFDE scatter plot to obtain an optimized MFDE scatter plot.

[0104] Specifically, the initial outliers determined in step S401 are removed from the initial MFDE scatter plot to reduce the impact of the outliers on subsequent analysis, thereby obtaining an optimized MFDE scatter plot. The optimized MFDE scatter plot only contains data that are considered to be normal points, which are more likely to represent the main distribution characteristics of the data set.

[0105] S403: According to the optimized MFDE scatter plot, an optimal anomaly ratio parameter in the LOF local anomaly factor algorithm is determined using an interior point method algorithm.

[0106] Specifically, the interior point method is used to analyze and optimize the data points in the MFDE scatter plot, and the convex hull area of ​​the data points is calculated. By comparing the evaluation results under different parameters, the system can determine an optimal anomaly ratio parameter that can best balance outlier detection and false alarm rate. This optimal anomaly ratio parameter will be used in the subsequent LOF algorithm to more accurately identify outliers in the data set.

[0107] In an alternative embodiment, see Figure 5 As shown, Figure 5 A flowchart of a specific optimal anomaly ratio parameter method provided by the first embodiment of the present invention is shown, wherein the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm is determined by using an interior point method algorithm according to the optimized MFDE scatter plot, including S501 to S504:

[0108] S501: Calculate an initial convex hull area in the initial MFDE scatter plot and an optimized convex hull area in the optimized MFDE scatter plot.

[0109] Specifically, the convex hull area of ​​all data points in the initial MFDE scatter plot is first calculated. This area represents the surface area of ​​the smallest convex polygon or convex polyhedron that contains all data points. Next, the convex hull area of ​​the remaining data points in the optimized MFDE scatter plot (i.e., the scatter plot after removing the initial outliers) is calculated. This area is called the optimized convex hull area. Since possible outliers are removed, the optimized convex hull area is usually smaller than the initial convex hull area and closer to the main distribution of data points.

[0110] Specifically, the initial convex hull area in the initial MFDE scatter plot is calculated by the following method:

[0111] Find the point P with the smallest x coordinate from the point set Z l =(tx min ,ty min ) and the point with the largest x coordinate Pr=(txmax,tymax), these points must be on the convex hull. l P r The positions are divided into upper and lower groups.

[0112] Use the following formula to determine point P i (tx i ,ty i ), i is the i-th point in the scatter plot, relative to the straight line P l P rThe position Pos = (tyr-tyl)txi + (txl-txr)yi + (txrtyl-txltyr). If Pos>0, the point is on the left side of the line (above); if Pos<0, the point is on the right side of the line (below); for the point sets in the upper and lower groups, use the following steps to recursively solve the convex hull:

[0113] Find the point farthest from the line: For each point in the group, find the distance from the line P l P r The farthest point P d . Calculate point P i To the straight line P l P r The distance d i :

[0114]

[0115] Select the one with the largest d i Value point P d As the farthest point. Divide the current line segment into two parts and process recursively: Use the farthest point P found d Set the current line segment P l P r Divided into two parts: l P d and P d P r For the upper part, find the concentration from P l P d The farthest point. For the lower part, find the point that is concentrated away from P d P r The farthest point. Repeat this process recursively until no point is above the current line segment. This is the desired convex hull point set ZS.

[0116] Use the shoelace formula to calculate the area of ​​a polygon for the convex hull point set ZS for each vertex Ps of the polygon i =(txs i ,tys i ), calculate the sum of the following products:

[0117]

[0118] Finally, the area A of the polygon is given by: Here, m0 is the number of convex hull vertices, (txs m0 ,tys m0 ) is the coordinate of the last vertex, and (txs1,tys1) is the first vertex. Then the area of ​​the unfiltered initial convex hull is S start =area(Z).

[0119] The optimized convex hull area in the optimized MFDE scatter plot is calculated by the following method:

[0120] Each data point Z i , find its k nearest neighbors, k = kmean*zmax, kmean is the ratio of neighbor points, zmax is the number of points in the collection Z, and in this frequency domain outlier exclusion it is fmax. The k-neighbor distance kd(x) refers to the Euclidean distance between the data point xs and its kth nearest neighbor: Among them, x k is the kth nearest neighbor of data point x. The k-neighborhood Ns of data point xs k (xs) contains all points whose distance from x is no more than k-neighbors away.

[0121] Calculate the local reachability density (LRD) of point x:

[0122]

[0123] rd k The reachable density rd for each data point x k (xs,xs i )=max(kd(xs i ),dist(xs,xs i )), the LOF of point x is the average of the ratios of the LRDs of all points in its neighborhood to the point’s own LRD:

[0124]

[0125] If LOF k (xs)≈1, then point x is in a normal density region; if LOF k (xs)>>1, then point x is an outlier.

[0126] According to the outlier factor LOF k The values ​​of (x) are sorted from high to low, and the number of outliers is determined based on the input outlier ratio m. The first m×xn points are selected as outliers, where xn is the total number of data sets. Let the set of corresponding positions of these outliers in the time-frequency spectrum be loc. m is the input outlier ratio. Then the non-outlier points are used as the new data point set Zlof, Z lof = Z(loc), the convex hull area S of the MFDE scatter plot after filtering filter =area(Z lof ).

[0127] At this point, the calculation of the convex hull area of ​​MFDE before and after filtering is completed.

[0128] S502: Constructing an objective function of the interior point method algorithm based on the initial convex hull area and the optimized convex hull area.

[0129] Specifically, an objective function is constructed using the initial convex hull area and the optimized convex hull area. The purpose of this objective function is to minimize the optimized convex hull area while ensuring that the number of outliers removed is within a reasonable range. The objective function may involve factors such as the difference in convex hull area, the number of outliers, and anomaly ratio parameters.

[0130] Specifically, the objective function of the interior point algorithm is set so that the convex hull area of ​​the non-abnormal points of the objective function can be minimized while the abnormal point ratio m is increased to the minimum extent, that is, the clustering of the scatter plot is enhanced.

[0131] The objective function is set to f(m) = S filter +S start *m*NS, Sfilter is the convex hull area of ​​the MFDE scatter plot of non-outlier points, S start is the convex hull area of ​​the MFDE scatter plot without filtering. When the same number of outliers at the same position are removed, the convex hull area of ​​the scatter plot is reduced by NS times when the scatter plot is evenly distributed, the optimization algorithm is effective. NS is a self-set value that can adjust the sensitivity to outliers. Compared with the initial scatter plot, the separated scatter plot shows stronger aggregation under uniform conditions.

[0132] S503: Setting a barrier function of the interior point method algorithm, wherein the barrier function includes an anomaly ratio parameter in the LOF local anomaly factor algorithm.

[0133] Specifically, a barrier function is set for the interior point algorithm, which is a function used to constrain the optimization problem and is used together with the objective function to ensure that the optimization process meets specific constraints. In this scenario, the barrier function may include the outlier ratio parameter in the LOF local outlier factor algorithm, as well as constraints on the number of outliers or the change in the convex hull area.

[0134] Specifically, when setting the barrier function, consider the standard form of the optimization problem:

[0135] minf(m)stAm=b,m≥0

[0136] Among them, f(m) is the objective function, A is the constraint matrix, b is the constraint value vector, and m is the variable vector. At the same time, the dual problem is considered as: Among them, λ is the dual variable vector, μ is the m in m i The corresponding Lagrange multiplier vector μ i A collection of Represents the gradient of the objective function f(m), which is a vector containing the gradient of f(m) for each variable m i The partial derivative of .

[0137] Integrate the constraints into the objective function and introduce the barrier function B(m): Where h is the barrier parameter and n is the dimension of the variable m.

[0138] The barrier function is designed to maintain the robustness of the data during the optimization process and prevent significant changes in the data distribution characteristics due to misjudgment of outliers.

[0139] S504: Determine, based on the barrier function, an abnormality ratio parameter that can minimize the optimized convex hull area when the objective function is satisfied, as the optimal abnormality ratio parameter.

[0140] Specifically, the objective function is solved by the interior point algorithm under the constraint of the barrier function to find the anomaly ratio parameter that minimizes the optimized convex hull area. This parameter is considered to be the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm because it can most accurately identify anomalies while maintaining data robustness. Specifically, the system may continuously adjust the value of the anomaly ratio parameter through an iterative optimization process and calculate the corresponding optimized convex hull area and objective function value. When the anomaly ratio parameter that satisfies the objective function and minimizes the optimized convex hull area is found, the iterative process ends and the parameter is determined to be the optimal anomaly ratio parameter.

[0141] Specifically, the goal of the interior point method is to find the central path and satisfy the following conditions (m,λ,μ):

[0142]

[0143] Where τ>0 is a fixed parameter that controls the distance between the path and the boundary of the feasible region.

[0144] At each iteration, the interior point method computes the Newton directions (Δm, Δλ, Δμ) by solving the following linear system:

[0145]

[0146] Where Λ is the diagonal matrix of the dual variables, S is a diagonal matrix whose diagonal elements are s i =x i μ i / τ, the vector a is usually a vector of all 1s, and its dimension is the same as the number of inequality constraints in the optimization problem. In is a Kronecker product operation, which is used mathematically to expand the dimension of the matrix. σ is a parameter between 0 and 1, which is used to control the speed at which the iteration point moves along the center path.

[0147] Determine the step size α to update the current point (m, λ, μ): (m, λ, μ) ← (m, λ, μ) + α (Δm, Δλ, Δμ)

[0148] When the condition k≥maxtir is met, the iteration terminates. maxtir is the set maximum number of iterations, and k is the current number of iterations. The interior point method gradually approaches the optimal solution in this way and outputs the optimal outlier ratio m.

[0149] Furthermore, based on the optimal anomaly ratio parameter, the target anomaly point is determined using the LOF local anomaly factor algorithm, and the optimal anomaly ratio obtained after iteration is brought into the LOF algorithm to obtain the anomaly point of MFDE:

[0150] Slof = [xlof i ]

[0151] xlof i =1or0,i=1,2,…,fmax

[0152] if i∈loc,xlof i =0

[0153]

[0154] Slof is the mask vector of frequency domain filtering, fmax is the maximum value in the frequency domain in the time-frequency domain, and loc is the set of corresponding positions of the outliers in the time-frequency spectrum.

[0155] Xflrf(:,f)=X(:,f).*Slof,f=1,2,...,fmax

[0156] Among them, Xflrf is the time-frequency spectrum after time-domain filtering. So far, the MFDE abnormal points are found on the frequency band and the time-frequency spectrum is filtered.

[0157] In general, see Figure 6 As shown, Figure 6A flowchart of a specific signal filtering method based on MFDE outlier exclusion provided by the first embodiment of the present invention is shown, wherein a signal is first input, and the received signal is analyzed in the time-frequency domain using the short-time Fourier transform STFT to obtain the time-frequency representation of the signal. Subsequently, the frequency domain characteristics of the time-frequency spectrum, namely the frequency domain MFDE, are obtained respectively by multi-scale fluctuation spread entropy (MFDE), and then the interior point method in the optimization algorithm is used to optimize the abnormal proportion parameter in the local anomaly factor to obtain the average minimum convex hull area of ​​the MFDE non-abnormal point, and the optimal abnormal proportion parameter is applied to the local anomaly factor LOF algorithm to map the abnormal point position to the time-frequency spectrum position to obtain the abnormal frequency domain segment, and then the corresponding time-frequency spectrum data is filtered out to achieve the frequency domain outlier exclusion. Finally, the processed time-frequency spectrum representation is reconstructed into a time domain signal by the inverse short-time Fourier transform ISTFT to obtain the final filtered output signal.

[0158] The innovation of this algorithm lies in calculating the time-frequency spectrum characteristics by multi-scale fluctuation spread entropy (MFDE) and combining it with the local outlier factor (LOF). The interior point method in the optimization algorithm is used to optimize the outlier ratio parameter in the local outlier factor to obtain the optimal convex hull area of ​​the MFDE non-outlier point. The optimal outlier ratio parameter is applied to the local outlier factor algorithm to obtain the outlier frequency domain segment and filter it out in the time-frequency spectrum to solve the limitations of traditional methods in processing non-stationary signals and outlier detection. It has the following advantages: the algorithm does not rely on prior knowledge of the signal direction or frequency characteristics, can directly extract features from the signal itself, and is suitable for unknown or changing signal environments. The LOF algorithm is used to detect local anomalies of signal components, effectively identify and remove outliers, and improve the accuracy of signal processing. The signal is converted to the time-frequency domain through STFT, so that the algorithm can consider time and frequency information at the same time and process non-stationary signals more effectively. The algorithm structure combining MFDE and LOF improves the robustness of the algorithm to different types of noise and outliers, and ensures stability in complex signal environments.

[0159] For the signal filtering method based on MFDE outlier elimination provided in the present application, the filtering effect thereof can be evaluated by the following evaluation function.

[0160] Using cosine similarity RMS Error The filtering effect is evaluated. is the estimated signal, y is the actual signal, and m is the number of data points. and ||y|| are modulo y.

[0161] After that, Matlab can be used to simulate the processed signal and the target signal filtered by the signal filtering method based on MFDE outlier elimination provided in this application, and the parameters are as follows:

[0162] The number of overlapping samples L of the window function is defined; the number of FFT points K is defined; the sampling frequency f0 of the signal is defined; the window function w is defined; the maximum number of iterations maxtir of the optimization algorithm is defined; and the neighbor point ratio kmean in the LOF algorithm is defined.

[0163] See also Figure 7 As shown, Figure 7 FIG. 1 is a schematic diagram showing a source signal frequency domain MFDE scatter plot provided by the first embodiment of the present invention, see Figure 8 As shown, Figure 8 FIG. 1 is a schematic diagram showing a mixed signal frequency domain MFDE scatter plot provided by the first embodiment of the present invention, see FIG. Fig. 9 As shown, Fig. 9 A schematic diagram of a filtered frequency domain MFDE scatter plot provided in Embodiment 1 of the present invention is shown. The horizontal axis FIRMFDE (:, 1) is the MFDE data with a scale factor of 1, the vertical axis FIRMFDE (:, 2) is the MFDE data with a scale factor of 1, "" represents a non-abnormal point, "×" represents an abnormal point, and "-" represents a convex hull boundary.

[0164] First, use STFT to get the signal time-frequency spectrum, then calculate MFDE for each frequency band, and calculate the initial convex hull area with MFDE scale one and two as axes. Use the interior point method to optimize the anomaly ratio parameter in LOF to reduce the convex hull area of ​​non-abnormal points. Apply the optimal anomaly ratio parameter to the local anomaly factor algorithm to obtain the abnormal point, such as Figure 8 As shown in the figure, the points in the scatter plot are classified into abnormal points and non-abnormal points through the local abnormal factor algorithm. The abnormal points are mapped to the corresponding time-frequency domain and filtered out. The MFDE after filtering out the abnormal points is as follows Fig. 9 As shown, Figure 8 By comparison, it can be seen that the convex hull area of ​​the scatter plot, i.e. the black frame area, is reduced. Finally, the processed time-frequency domain representation is reconstructed into a time domain signal through inverse STFT to obtain the final filtered output signal. The filtered time domain signal is as follows: Fig. 9 .

[0165] When evaluating the filtering effect, refer to Fig.10 As shown, Fig.10 A schematic diagram of a source signal spectrum provided by the first embodiment of the present invention is shown. Fig.11 As shown, Fig.11 A schematic diagram of a mixed signal spectrum provided by the first embodiment of the present invention is shown. Fig.12 As shown, Fig.12 FIG. 1 is a schematic diagram of a signal spectrum after filtering provided by the first embodiment of the present invention. The horizontal axis is the frequency in Hertz (Hz), and the vertical axis is the amplitude. Fig.13 As shown, Fig.13 See Fig.14 As shown, Fig.14 A schematic diagram of a mixed signal time domain diagram provided by the first embodiment of the present invention is shown, see Fig.15 As shown, Fig.15 FIG. 1 is a schematic diagram of a time domain diagram of a filtered signal provided by the first embodiment of the present invention. The horizontal axis is time in seconds, and the vertical axis is amplitude. Fig.10 It can be seen that the power of the source signal in the frequency range of 0-1000 Hz is mainly concentrated at 200 Hz. Fig.11 and Fig.12 It can be seen that the algorithm filters out the 400hz and 600hz components and retains the 200hz component. Fig.10 and Fig.11 The data obtains the cosine similarity of the spectrum before filtering; Fig.10 and Fig.12 The cosine similarity of the filtered spectrum is obtained by Fig.13 and Fig.14 The cosine similarity and root mean square error of the data in the time domain before filtering are obtained; Fig.13 and Fig.15 The data is filtered to obtain the cosine similarity in the time domain. The values ​​of the specific evaluation index table are shown in the table below:

[0166]

[0167] According to the above evaluation indicators, the root mean square error RMSE decreased by 0.4943, the time domain cosine similarity increased by 0.5333, and the cosine similarity of the spectrum increased to 0.6098. This proves that the signal filtering method based on MFDE outlier elimination provided in this application can effectively reduce noise interference and significantly improve signal quality.

[0168] Embodiment 2

[0169] Embodiment 2 of the present invention provides an integral data processing device, see Fig.16 As shown, Fig.16 A schematic diagram of the structure of a signal filtering device based on MFDE outlier elimination provided by Embodiment 2 of the present invention is shown, wherein the device comprises:

[0170] The signal receiving module 1601 is used to receive a signal to be processed, and perform short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed;

[0171] An outlier determination module 1602 is used to determine the target outlier points in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local outlier factor algorithm and the interior point method;

[0172] The signal optimization module 1603 is used to filter out the target abnormal point from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and perform an inverse short-time Fourier transform on the target time-frequency spectrum to obtain a target signal.

[0173] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0174] The signal filtering device based on MFDE outlier elimination provided in the embodiment of the present invention can be specific hardware on the device or software or firmware installed on the device. The device provided in the embodiment of the present invention has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brief description, the parts not mentioned in the device embodiment can refer to the corresponding contents in the aforementioned method embodiment. Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can all refer to the corresponding processes in the aforementioned method embodiment, and will not be repeated here.

[0175] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, each functional unit in the embodiment provided by the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0178] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0179] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0180] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can still modify the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or perform equivalent replacements on some of the technical features thereof; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. They should all be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A signal filtering method based on MFDE outlier elimination, characterized in that: The method comprises: Receiving a signal to be processed, and performing short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed; The target abnormal point in the initial time-frequency spectrum is determined by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local abnormal factor algorithm and the interior point method; The target abnormal point is filtered out from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and an inverse short-time Fourier transform is performed on the target time-frequency spectrum to obtain a target signal.

2. The method according to claim 1, characterized in that The method of using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local anomaly factor algorithm and the interior point method to determine the target anomaly point in the initial time-frequency spectrum includes: According to the initial time-frequency spectrum, an initial MFDE scatter plot is generated using the MFDE multi-scale fluctuation scatter entropy algorithm; Determine the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method; Based on the optimal anomaly ratio parameter, the target anomaly point is determined using the LOF local anomaly factor algorithm.

3. The method according to claim 2, characterized in that The step of generating an initial MFDE scatter plot using the MFDE multi-scale fluctuation scatter entropy algorithm according to the initial time-frequency spectrum includes: Segmenting the initial time-frequency spectrum to obtain subsequences under different scale factors; Calculate the FDE fluctuation spread entropy of each subsequence, and determine the MFDE multi-scale fluctuation spread entropy based on the FDE fluctuation spread entropy of each subsequence; The initial MFDE scatter plot is generated based on the MFDE multi-scale fluctuation scatter entropy.

4. The method according to claim 2, characterized in that: The method of determining the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method includes: Based on the initial anomaly ratio parameter, the initial anomaly points in the initial MFDE scatter plot are determined using the LOF local anomaly factor algorithm; Removing the initial outliers from the initial MFDE scatter plot to obtain an optimized MFDE scatter plot; According to the optimized MFDE scatter plot, the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm is determined using the interior point method algorithm.

5. The method according to claim 4, characterized in that Determining the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm using an interior point algorithm according to the optimized MFDE scatter plot includes: Calculating an initial convex hull area in the initial MFDE scatter plot and an optimized convex hull area in the optimized MFDE scatter plot; Constructing the objective function of the interior point method algorithm based on the initial convex hull area and the optimized convex hull area; Setting a barrier function of the interior point method algorithm, wherein the barrier function includes an anomaly ratio parameter in the LOF local anomaly factor algorithm; Based on the obstacle function, an abnormality ratio parameter that can minimize the optimized convex hull area when satisfying the objective function is determined as the optimal abnormality ratio parameter.

6. A signal filtering device based on MFDE outlier elimination, characterized in that: The device comprises: A signal receiving module, used for receiving a signal to be processed, and performing a short-time Fourier transform on the signal to be processed to obtain an initial time-frequency spectrum of the signal to be processed; An outlier determination module is used to determine the target outlier points in the initial time-frequency spectrum by using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local outlier factor algorithm and the interior point method; The signal optimization module is used to filter out the target abnormal point from the initial time-frequency spectrum to obtain a target time-frequency spectrum, and perform an inverse short-time Fourier transform on the target time-frequency spectrum to obtain a target signal.

7. The device according to claim 6, characterized in that The method of using the MFDE multi-scale fluctuation dispersion entropy algorithm, the LOF local anomaly factor algorithm and the interior point method to determine the target anomaly point in the initial time-frequency spectrum includes: According to the initial time-frequency spectrum, an initial MFDE scatter plot is generated using the MFDE multi-scale fluctuation scatter entropy algorithm; Determine the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method; Based on the optimal anomaly ratio parameter, the target anomaly point is determined using the LOF local anomaly factor algorithm.

8. The device according to claim 7, characterized in that The step of generating an initial MFDE scatter plot using the MFDE multi-scale fluctuation scatter entropy algorithm according to the initial time-frequency spectrum includes: Segmenting the initial time-frequency spectrum to obtain subsequences under different scale factors; Calculate the FDE fluctuation spread entropy of each subsequence, and determine the MFDE multi-scale fluctuation spread entropy based on the FDE fluctuation spread entropy of each subsequence; The initial MFDE scatter plot is generated based on the MFDE multi-scale fluctuation scatter entropy.

9. The device according to claim 7, characterized in that The method of determining the optimal anomaly ratio parameter of the LOF local anomaly factor algorithm according to the initial MFDE scatter plot using the interior point method includes: Based on the initial anomaly ratio parameter, the initial anomaly points in the initial MFDE scatter plot are determined using the LOF local anomaly factor algorithm; Removing the initial outliers from the initial MFDE scatter plot to obtain an optimized MFDE scatter plot; According to the optimized MFDE scatter plot, the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm is determined using the interior point method algorithm.

10. The device according to claim 9, characterized in that Determining the optimal anomaly ratio parameter in the LOF local anomaly factor algorithm using an interior point algorithm according to the optimized MFDE scatter plot includes: Calculating an initial convex hull area in the initial MFDE scatter plot and an optimized convex hull area in the optimized MFDE scatter plot; Constructing the objective function of the interior point method algorithm based on the initial convex hull area and the optimized convex hull area; Setting a barrier function of the interior point method algorithm, wherein the barrier function includes an anomaly ratio parameter in the LOF local anomaly factor algorithm; Based on the obstacle function, an abnormality ratio parameter that can minimize the optimized convex hull area when satisfying the objective function is determined as the optimal abnormality ratio parameter.

Images