Data filtering algorithm

Abstract

The invention discloses a filtering algorithm for data processing, which is particularly suitable for smoothing or noise reduction processing of sequence data so as to better display data trend or mode characteristics. According to the method, upper and lower boundary lines are generated by setting width parameters, a filtering value of data is calculated through linear interpolation, noise is reduced, and the main trend of the data is reserved. Compared with a traditional filtering method, the method is low in calculation complexity, can process a big data set in real time, and is suitable for the fields of finance, engineering, Internet of Things, signal processing and the like. According to the method, the data visualization effect can be improved, and the accuracy of trend identification is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data processing, analysis and visualization, and in particular to a system and method for generating data processing and visualization through a filtering algorithm. The method calculates data filtering values ​​by setting width parameters to generate more accurate and clear data trend charts, which can be widely used in data processing, analysis and chart generation in fields including finance, signal processing, engineering analysis, image and video noise reduction, and data compression. Background Art

[0002] With the advent of the big data era, data analysis and visualization have become indispensable tools in many industries. Filtering algorithms also play a wide and important role in the early processing of deep learning and artificial intelligence models, especially in the data preprocessing and feature extraction stages. In data analysis, a large amount of data usually needs to be processed to extract useful trends and patterns. These trends are usually displayed intuitively in the form of charts to help users better understand the data. However, many existing data visualization methods, such as simple smoothing techniques such as moving averages, may not be able to fully capture the subtle fluctuations and complex trend changes in the data.

[0003] Traditional data filtering techniques, such as Fourier transform, wavelet transform and other signal processing methods, although they play an important role in data filtering and analysis, often require a lot of computing resources and have a complex processing process. In addition, when processing some non-stationary data, these methods may not be able to clearly reflect the data trend, and are prone to over-smoothing or failure to effectively remove noise, resulting in insufficient reflection of the true trend of the data by the chart results.

[0004] In some scenarios in the field of data processing and visualization, a new data filtering method is needed. This method can effectively eliminate noise while maintaining the main trend of the data, with low computational complexity, and can provide users with more intuitive and clear visualization charts. Summary of the invention

[0005] The present invention relates to a method and system for generating data filtering data, which is particularly suitable for smoothing a large amount of time series data to better display the trend or pattern characteristics of the data. Although existing filtering technologies such as moving average method and Fourier transform can be used for data filtering and smoothing, they still have limitations on nonlinear data, data with large noise, and real-time processing requirements. To solve this problem, the present invention proposes a filtering method based on upper and lower boundary line calculation, which can flexibly adapt to data fluctuations, reduce calculation complexity, and provide clear trend display for data visualization.

[0006] The method proposed by the present invention comprises the following steps: Get the latest time series data from the data source or data provider and store it in the data list for subsequent processing.

[0007] Parameter initialization, initialize the width parameter and number of data points of the filtering algorithm. The width parameter is used to set the filtering interval range to ensure that the calculated upper and lower boundary lines can contain the main fluctuation range of the data. The width parameter can be given based on experience or other algorithms or adjusted after the results are verified. The number of data points is determined based on the length of the acquired data, usually the data length minus 1, to accommodate indexing operations.

[0008] Filter data calculation, use linear interpolation method to calculate the upper and lower boundaries of the data. In this step, the algorithm iterates each segment of the data list. For each data segment, first calculate the starting point P1 and the end point P n The upper and lower boundary values ​​are calculated based on the set width parameter w, and the starting point of the upper boundary line is P u1 The end point is P un , the starting point of the lower boundary line is P d1 The end point is P dn The upper boundary line is the linear interpolation generated by adding the start and end data values ​​to the width value, while the lower boundary line is the linear interpolation generated by subtracting the width value.

[0009] After generating the upper and lower boundary lines, the algorithm checks all data points in the data segment to determine whether they are between the upper and lower boundary lines. If not all data points are within the upper and lower boundary lines, the calculation starting point P1 remains unchanged and the end point becomes P n-1 The starting point of the upper boundary remains unchanged at Pu1 and the end point is P un-1 , the starting point of the lower boundary line is P d1 The end point is P dn-1 Repeat this process until the starting point P1 and the end point P i The starting point of the upper boundary is Pu1 and the end point is P ui , the starting point of the lower boundary line is P d1 The end point is P di When all data points are within the upper and lower boundary lines, the data segment is considered to meet the filtering conditions and the filtering value of the data segment is generated. The filtering value is obtained by calculating the mean of the upper and lower boundary lines, that is, for each data point, its corresponding filtering value is the average of the upper and lower boundary lines. The calculation result is stored in the filtering data list. The next processing is performed, and the calculation starting point becomes P i and the end point becomes P n , the starting point of the upper boundary becomes P ui The end point is P un , the starting point of the lower boundary line is P di The end point is P dnThis process is repeated until all data are processed.

[0010] The filtering algorithm of the present invention generates upper and lower boundary lines through adaptive width parameters, dynamically adjusts the filtering effect of the data, and has the following advantages: strong real-time performance, by quickly calculating the upper and lower boundary lines and judging whether the data is within the boundary lines, the algorithm can maintain high efficiency when processing large-scale data. Low computational complexity, compared with computationally complex filtering methods such as wavelet transform or Fourier transform, the algorithm of the present invention uses linear interpolation and boundary line checking to reduce unnecessary complex calculations, and is particularly suitable for processing large data sets. Accurate trend identification, by accurately calculating the upper and lower boundary lines and generating filtered data, the present invention can effectively reduce noise, improve the clarity of data visualization and trend accuracy, and is suitable for fields with high requirements for trend analysis, such as engineering data, Internet of Things data, and meteorological data. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0012] Figure 1 is the original data curve; Figure 2 is the upper and lower boundary lines from the initial point to the end point; Figure 3 The upper and lower boundaries from the initial point to the secondary end point; Figure 4 The upper and lower boundaries from the initial point to the secondary starting point; Figure 5 The upper and lower boundaries from the starting point to the end point; Figure 6 It is the superposition of the filtered data curve and the original data curve. DETAILED DESCRIPTION

[0013] This embodiment applies the present invention to sensor data monitoring in an Internet of Things environment, especially sensor data for monitoring the temperature of a machine's operating environment.

[0014] Data acquisition: The sensor collects temperature data once every hour and transmits the data to the server through the application program interface. The original data curve of the temperature data used in this embodiment is as follows: Figure 1 As shown: Parameter initialization: Set the width parameter of the filtering algorithm, Figure 2 The data point 1 plus filter width is 11 points, and the data point 1 minus filter width is 21 points.

[0015] Filter data calculation: Calculate linear interpolation for each segment of data to generate upper and lower boundary lines. Figure 2The first data segment (from the first data point 1 to the last data point 9) is processed, and the upper and lower boundary lines are calculated as follows: Upper boundary line: Figure 2 11-19 Lower boundary line: Figure 2 21-29 Confirm whether all data points are between the upper and lower boundary lines. If they do not meet the filtering conditions: right Figure 3 The second data segment (from the first data point 1 to the second to last data point 8) is processed, and the upper and lower boundary lines are calculated as follows: Upper boundary line: Figure 3 Middle 11-18 Lower boundary line: Figure 3 Middle 21-28 Confirm whether all data points are between the upper and lower boundary lines. If they do not meet the filtering conditions, repeat this operation until Figure 4 As shown, all data points lie between the upper and lower boundary lines: Upper boundary line: Figure 4 Middle 11-15 Lower boundary line: Figure 3 Middle 21-25 but Figure 4 The straight line from 1 to 5 is the data result after the original data is filtered from 1 to 5. The data on the straight line from 1 to 5 can be calculated based on the average value of the corresponding upper and lower boundary lines.

[0016] Again Figure 5 Repeat this algorithm with point 5 as the next starting point. Figure 5 The first data segment (from the first starting data point 5 to the last data point 9) is processed, and the upper and lower boundary lines are calculated as follows: Upper boundary line: Figure 3 15-19 Lower boundary line: Figure 3 Middle 25-29 Repeat the above operation until all the results are found that satisfy all data points are between the upper and lower boundary lines. The filtered data curve is as follows Figure 6 , the dotted line is the filtered data curve, and the solid line is the original data curve. The data has been simplified, and the local extreme points of the original data have been retained, without the lag phenomenon in other filtering algorithms.

[0017] Through this embodiment, the application of the present invention is demonstrated, proving that it can provide an effective solution for the identification and visualization of data trends.

Claims

1. A data filtering processing algorithm, characterized in that: The method comprises the following steps: (a) Obtain the data sequence and store it in a data list; (b) Initialize the width parameter w of the filtering algorithm and the starting point P1 and end point P1 of the data segment n ; (c) Use linear interpolation to calculate the upper and lower boundaries of the data segment, where: The starting point of the upper boundary line is P u1 , the end point is P un , and is generated by adding a width parameter w to the data value; The starting point of the lower boundary line is P d1 , the end point is P dn , and is generated by subtracting the width parameter w from the data value; (d) Check all data points of the data segment to determine whether they are located between the upper and lower boundary lines; (e) If all data points are not located between the upper and lower boundary lines, move the end point from P n Adjust to P n-1 , and recalculate the upper and lower boundary lines until all data points are within the upper and lower boundary lines; (f) When the conditions are met, a filtering value of the data segment is generated, where the filtering value is obtained by calculating the average of the upper boundary line and the lower boundary line; (g) Store the filtered value of this segment, and repeat the above steps to process the next segment of data until all the data are processed.

2. The method according to claim 1, characterized in that The upper boundary line generated by the linear interpolation calculation is the starting point data value P u1 and the endpoint data value P un Generate by adding the width value w for linear interpolation, with the lower boundary line being the starting point data value P d1 and the endpoint data value P dn Generated by subtracting the width value w.

3. The method according to claim 1, characterized in that The process of checking whether the data point is located between the upper and lower boundary lines in step d is to determine whether each data point in the data segment falls between the corresponding upper boundary line and lower boundary line.

4. The method according to claim 1, characterized in that: When the condition is not met, the end point is from P n Adjust to P n-1 , and repeat the process of calculating the upper and lower boundary lines until the end point is adjusted to P i And all data points are between the upper and lower boundary lines.

5. The method according to claim 1, characterized in that The filtering value is calculated by the mean of the upper and lower boundary lines, that is, the filtering value of each data point is the average of the corresponding upper and lower boundary lines.

6. The method according to claim 1, characterized in that When processing the next segment of data, the starting point of the data segment is adjusted to P i , the end point is adjusted to P n , and repeat the linear interpolation to calculate the upper and lower boundary lines until all data segments are processed.

Images