Data filtering method and device, equipment and storage medium
By obtaining the standard deviation of the data stream in real time, establishing a dynamic window function, calculating the window size, and filtering the original data stream, solving the problem of low adaptability caused by changes in noise intensity in the prior art, and improving the accuracy of filtering.
Patent Information
- Application Number
- CN202510498362.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-21
- Publication Date
- 2025-07-25
AI Technical Summary
The existing noise cancellation methods mainly smooth the data by fixed window size sliding average, but the noise intensity is in a changing state, resulting in poor adaptability and affecting the filtering accuracy.
By obtaining the standard deviation of the original data stream in real time, establishing a dynamic window function, and calculating the window size based on the standard deviation, filtering the original data stream to obtain the target data stream.
It improves the adaptability and accuracy of filtering, and can effectively filter according to changes in noise intensity, solving the problem of insufficient adaptability caused by fixed window size.
Smart Images
Figure CN120377867A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of data filtering, and in particular, to a data filtering method, apparatus, device, and storage medium. Background Art
[0002] In the fields of data processing and signal analysis, instantaneous noise (short-term data anomalies caused by sudden interference or system fluctuations) is a major problem affecting data accuracy. For example, in microservice monitoring, sensor signal acquisition, or real-time trading systems, instantaneous noise can cause distortion of key metrics (such as CPU usage, response time, transaction volume, etc.), thereby affecting system decision-making and stability.
[0003] Existing noise cancellation methods mainly smooth data through the sliding average of a fixed window size. However, the noise intensity is in a changing state, and the method of processing with a fixed window size has poor adaptability and cannot effectively filter under the real-time change of noise intensity, affecting the filtering accuracy. Summary of the Invention
[0004] In view of this, this application provides a data filtering method, apparatus, device, and storage medium, which are used to solve the problem that existing noise cancellation methods mainly smooth data through the sliding average of a fixed window size. However, the noise intensity is in a changing state, and the method of processing with a fixed window size has poor adaptability and cannot effectively filter under the real-time change of noise intensity, affecting the filtering accuracy.
[0005] To achieve the above objectives, the following solutions are proposed:
[0006] In a first aspect, a data filtering method includes:
[0007] Obtain the original data stream in real time and calculate the standard deviation of the original data stream;
[0008] Compare the standard deviation with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream;
[0009] Establish a dynamic window function based on the standard deviation;
[0010] Calculate the window size corresponding to the original data stream based on the dynamic window function;
[0011] Filter the instantaneous noise of the original data stream with the window size as a constraint to obtain the target data stream.
[0012] Preferably, the formula of the dynamic window function is:
[0013] ;
[0014] Among them, represents the window size, represents the minimum value of the window size, represents the maximum value of the window size, represents the standard deviation of the original data stream, represents the adjustment coefficient.
[0015] Preferably, filtering the instantaneous noise of the original data stream with the window size as a constraint to obtain a target data stream includes:
[0016] Determine the number of data in the original data stream and the data order in the data stream;
[0017] According to the number of data, starting from the last data in the data order, select the corresponding number of data as each window data in turn;
[0018] Perform weighted median filtering on each window data to obtain a target data stream.
[0019] Preferably, performing weighted median filtering on each window data to obtain a target data stream includes:
[0020] Calculate the weights of each window data;
[0021] Determine the data values of each window data;
[0022] Sort each window data in ascending order according to the data value;
[0023] For each window data in the sorting, determine whether the window data is the first window data in the sorting;
[0024] If so, use the weight of the window data as its own cumulative weight;
[0025] If not, in the sorting, sum the weights of the window data and all the window data before it to obtain the cumulative weight of the window data;
[0026] Filter each window data according to the cumulative weight to obtain a target data stream.
[0027] Preferably, filtering each window data according to the cumulative weight to obtain a target data stream includes:
[0028] Determine the cumulative weight of the last window data in the sorting;
[0029] Multiplying the accumulated weight by a preset ratio to obtain a median weight;
[0030] According to the sorting, starting from the first window data, it is determined in turn whether the cumulative weight of each window data is greater than the median weight;
[0031] In the judgment process, the window data whose cumulative weight is greater than the median weight for the first time is taken as the first data to be selected;
[0032] A target data stream is determined according to the first data to be selected.
[0033] Preferably, determining the target data stream according to the first data to be selected includes:
[0034] Determine the number of the window data;
[0035] If the number of data in the original data stream is greater than the number of window data, each data in the original data stream except the window data is used as each second data to be selected;
[0036] The first data to be selected and each of the second data to be selected are aggregated to obtain a target data stream.
[0037] Preferably, the weight calculation formula is:
[0038] ;
[0039] in, Indicates The weight of the window data, represents the data value of the first data in the original data stream, Indicates The data value of the window data.
[0040] In a second aspect, a data filtering device includes:
[0041] A standard deviation calculation module, used to obtain the original data stream in real time and calculate the standard deviation of the original data stream;
[0042] An instantaneous noise determination module is used to compare the standard deviation with a preset standard deviation threshold, and if the standard deviation is not greater than the standard deviation threshold, determine that instantaneous noise exists;
[0043] Dynamic window function establishment module, used to establish dynamic window function according to standard deviation;
[0044] A window size calculation module, used to calculate a window size corresponding to the original data stream based on the dynamic window function;
[0045] A filtering processing module, configured to perform filtering processing on the instantaneous noise of the original data stream with the window size as a constraint, so as to obtain a target data stream.
[0046] In a third aspect, a data filtering device includes a memory and a processor;
[0047] The memory is used to store programs;
[0048] The processor is configured to execute the program to implement the steps of the data filtering method according to any one of the first aspects.
[0049] In a fourth aspect, a storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data filtering method according to any one of the first aspects are implemented.
[0050] It can be seen from the above technical solutions that in this application, the original data stream is obtained in real time, and the standard deviation of the original data stream is calculated; the standard deviation is compared with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream; a dynamic window function is established according to the standard deviation; the window size corresponding to the original data stream is calculated based on the dynamic window function; the instantaneous noise of the original data stream is filtered with the window size as a constraint to obtain a target data stream. This application takes into account dynamicity and real-time nature. It obtains the original data stream in real time, then calculates the standard deviation, and uses the size of the standard deviation to determine whether there is instantaneous noise in the original data stream. If there is, dynamic filtering processing is required. A dynamic window function is established according to the standard deviation, and the window size corresponding to the standard deviation is calculated based on this dynamic window function, so that the original data stream corresponds to the window size and is dynamic. Thus, the instantaneous noise of the original data stream is filtered with the window size as a constraint, which can solve the problem of poor adaptability caused by processing with a fixed window size in the prior art, improve adaptability, perform effective filtering following the change of noise intensity, and improve the accuracy of filtering. Description of the Drawings
[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.
[0052] Figure 1 It is an optional flowchart of a data filtering method provided by an embodiment of this application;
[0053] Figure 2Schematic diagram of a data filtering device provided by an embodiment of the present application;
[0054] Figure 3 Schematic diagram of a data filtering device provided by an embodiment of the present application. Detailed implementation manners
[0055] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.
[0056] The present invention can be used in many general or special computing device environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, and so on.
[0057] An embodiment of the present invention provides a data filtering method, which can be applied to various computer terminals or intelligent terminals. The execution subject thereof can be a processor or a server of a computer terminal or an intelligent terminal. The method flow chart of the method is as Figure 1 shown and specifically includes:
[0058] S1: Continuously obtain the original data stream and calculate the standard deviation of the original data stream.
[0059] The present application can be adapted to different scenarios, and different scenarios correspond to different types of data, such as CPU usage rate, CPU response time. In one example, if you want to master the relevant situation of the CPU usage rate and filter the relevant data, then it is necessary to continuously obtain the original data stream of the CPU usage rate type and then calculate the standard deviation.
[0060] By continuously obtaining the original data stream in this step, it is possible to respond to dynamic data changes, avoid the delay of offline processing, and also obtain instantaneous data characteristics, improving the accuracy of subsequent processing. The present application uses the calculated standard deviation to introduce subsequent filtering processing, which can adapt to real-time scenarios, reduce complexity, and the standard deviation is a key indicator for measuring the dispersion degree of the original data stream and can reflect the overall noise level or abnormal fluctuation of the original data stream.
[0061] S2: Compare the standard deviation with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream.
[0062] The preset standard deviation threshold can be set according to specific scenarios, historical scenario data, or characteristics in a vertical domain. For example, calculate the standard deviation distribution of historical data and take the value corresponding to a certain percentile (such as 95%) as the standard deviation threshold.
[0063] Compare the standard deviation of the original data stream with the standard deviation threshold. If the standard deviation of the original data stream is not greater than the standard deviation threshold, it indicates that there is instantaneous noise in the original data stream, such as impulse noise, sensor failure, etc. By setting the standard deviation threshold independently, the normal fluctuations and significant noise in the data can be dynamically distinguished, avoiding over-sensitivity and overcorrection.
[0064] In addition, if the standard deviation of the original data stream is less than the standard deviation threshold, it is considered that there is no instantaneous noise in the original data stream. Therefore, this step only starts subsequent processing when the standard deviation of the original data stream exceeds the standard deviation threshold, which can save computing resources.
[0065] S3: Establish a dynamic window function based on the standard deviation.
[0066] In this step, a dynamic window function is established from the standard deviation of the original data stream. It can be understood that the window size is positively correlated with the standard deviation. To improve the feasibility of calculation, the dynamic window function can be designed to be lightweight and can also ensure real-time performance. Among them, to prevent extreme situations, a maximum window limit and a minimum window limit can be defined.
[0067] S4: Calculate the window size corresponding to the original data stream based on the dynamic window function.
[0068] This step constructs a dynamic window function based on the idea that the window size is positively correlated with the standard deviation, and can calculate the window size according to the dynamic window function, so that the window size changes dynamically with the noise level, thereby avoiding the over-smoothing or under-smoothing problems caused by a fixed window.
[0069] S5: Filter the instantaneous noise of the original data stream with the window size as a constraint to obtain a target data stream.
[0070] It can be understood that the greater the noise, the wider the window, which can enhance the smoothing effect; the smaller the noise, the narrower the window, which can retain the details of the data features, achieve dynamic filtering processing, adapt to different instantaneous noise scenarios, and improve the applicability and the accuracy of the filtered data.
[0071] As can be seen from the above technical solution, the present application obtains the original data stream in real time and calculates the standard deviation of the original data stream; compares the standard deviation with a preset standard deviation threshold, and if the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream; establishes a dynamic window function based on the standard deviation; calculates the window size corresponding to the original data stream based on the dynamic window function; filters the instantaneous noise of the original data stream with the window size as a constraint to obtain the target data stream. Considering the dynamics and real-time nature, the present application obtains the original data stream in real time, then calculates the standard deviation, and uses the size of the standard deviation to determine whether there is instantaneous noise in the original data stream. If so, dynamic filtering processing is required. A dynamic window function is established based on the standard deviation, and the window size corresponding to the standard deviation is calculated based on this dynamic window function, so that the original data stream corresponds to the window size and is dynamic. Therefore, filtering the instantaneous noise of the original data stream with the window size as a constraint can solve the problem of poor adaptability caused by processing with a fixed window size in the prior art, improve adaptability, effectively filter following the change of noise intensity, and improve the accuracy of filtering.
[0072] Optionally, the formula of the dynamic window function is:
[0073] ;
[0074] Wherein, represents the window size, represents the minimum value of the window size, represents the maximum value of the window size, represents the standard deviation of the original data stream, represents the adjustment coefficient.
[0075] Specifically, the minimum value of the window size can be set to the total data value of the original data stream obtained in real time for each round, so as to ensure the basic data volume; the maximum value of the window size can be determined according to the specific scenario and the characteristics of the data type, which can prevent the window size from being too large and causing unnecessary calculation overhead; the adjustment coefficient controls the sensitivity of the window size to instantaneous noise and can also be set or adjusted according to the specific situation.
[0076] Among them, the starting point of the window can be set to the last data in the original data stream. The advantage of the dynamic window function is that it can dynamically adjust the window size according to the size or intensity of the instantaneous noise to adapt to different scenarios. For example, a small window is used when the instantaneous noise is small to reduce the calculation overhead.
[0077] In the method provided by the embodiment of the present invention, the process of filtering the instantaneous noise of the original data stream with the window size as a constraint to obtain the target data stream is specifically described as follows:
[0078] Determine the number of data in the original data stream and the data order in the data stream;
[0079] According to the number of data, starting from the last data in the data order, select the corresponding number of data as each window data in turn;
[0080] Perform weighted median filtering on each window data to obtain the target data stream.
[0081] Specifically, the number of data and the data order can be determined from the original data stream. The determination of the data order is very important for the filtering process. Maintaining this data order can avoid distortion in the filtering process caused by out-of-order data. Since the window size has been determined, it is necessary to perform preliminary screening and filtering on the original data stream according to the window size, that is, starting from the last data in the data order, select data in turn and count in real time. Stop when the corresponding number of data as the window size is selected, and use the selected data as each window data.
[0082] In an example, the original data stream has 5 data, namely a1, a2, a3, a4, a5, the data order is a1, a2, a3, a4, a5, and the window size is 4. Then the number of data is 5. Therefore, it is necessary to start from a5 and select 4 data in turn. That is, the finally selected window data is a1, a2, a3, a4. If arranged in the selected order, it is: a4, a3, a2, a1.
[0083] Then perform secondary filtering on each window data, that is, weighted median filtering, to obtain the target data stream.
[0084] Among them, the process of performing weighted median filtering on each window data to obtain the target data stream is described in detail below.
[0085] Calculate the weights of each window data;
[0086] Determine the data values of each window data;
[0087] Sort each window data in ascending order according to the data value;
[0088] For each window data in the sorting, determine whether the window data is the first window data in the sorting;
[0089] If so, use the weight of the window data as its own cumulative weight;
[0090] If not, in the said sorting, sum up the weights of this window data and all the window data before it to obtain the cumulative weight of this window data;
[0091] Filter each of the window data according to the cumulative weight to obtain the target data stream.
[0092] Among them, the calculation formula for the weight is: ;
[0093] represents the weight of the th window data, represents the data value of the first data in the said original data stream, represents the th window data's data value. To prevent the denominator in the weight calculation formula from being zero, which would affect the calculation, it is possible to first determine whether the denominator is zero. If it is zero, do not calculate according to this formula, and directly set the weight of the corresponding window data to 1.
[0094] Specifically, introducing the setting of weights can assign importance to the data, for example, giving higher influence to key window data can improve the filtering accuracy. Since this application calculates and filters the standard deviation for a type of data in a scenario, the data values of each window data can be determined, and the size relationship between them can also be determined. Therefore, it is possible to sort them in ascending order of the data value. Sorting is the prerequisite for the weighted median filtering calculation, ensuring that the statistical center of the window data can be quickly located.
[0095] Among them, when calculating the cumulative weight, it is accumulated in sequence. Therefore, only the cumulative weight of the first window data is its own weight, and the cumulative weight of the subsequent window data is its own weight plus the weights of all the windows before it. This can achieve noise-robust filtering. Compared with the existing mean filtering, it can resist outliers and retain the signal edge at the same time. Finally, the final filtering is achieved according to the cumulative weight to obtain the target data stream.
[0096] For the step of filtering each of the window data according to the cumulative weight to obtain the target data stream in the above process, it may include:
[0097] Determine the cumulative weight of the last window data in the said sorting;
[0098] Multiply the cumulative weight by a preset ratio to obtain the median weight;
[0099] According to the sorting, starting from the first window data, it is determined in turn whether the cumulative weight of each window data is greater than the median weight;
[0100] In the judgment process, the window data whose cumulative weight is greater than the median weight for the first time is taken as the first data to be selected;
[0101] A target data stream is determined according to the first data to be selected.
[0102] Specifically, it can be understood that the cumulative weight of the last window data is the sum of the weights of all window data, representing the global weight scale. The preset ratio can be set to 0.5, which is the standard median, and can also be adjusted to achieve skewed filtering, such as 0.3. In this way, different filtering strategies can be supported through the preset ratio to improve the flexibility of filtering. This embodiment does not impose any restrictions on this.
[0103] Next, sorting is used to locate the median through a single linear scan, and an early termination strategy is set to stop subsequent judgments when the conditions are met for the first time, reducing unnecessary calculations and achieving an optimal balance between computing efficiency and filtering effects. This is especially suitable for edge computing scenarios with strict requirements on real-time and robustness.
[0104] Further, the process of determining the target data stream according to the first data to be selected may include the following steps:
[0105] Determine the number of the window data;
[0106] If the number of data in the original data stream is greater than the number of window data, each data in the original data stream except the window data is used as each second data to be selected;
[0107] The first data to be selected and each of the second data to be selected are aggregated to obtain a target data stream.
[0108] Specifically, since only the window data is subjected to weighted median filtering according to the window size, the other data outside the window (i.e., each second data to be selected) is still retained. Therefore, the window data finally remaining, i.e., the first data to be selected and each second data to be selected are aggregated to obtain the final target data stream.
[0109] For example, continuing with the above example, the original data stream has a total of 5 data, namely a1, a2, a3, a4, and a5, in the order of a1, a2, a3, a4, a5, the window size is 4, and the data values of these 5 data are 11, 12, 11, 13, and 50 respectively. Then, according to the above window data selection method, the selected window data is a2, a3, a4, a5, that is, the data values are 12, 11, 13, and 50. The weights of each window data are determined as shown in Table 1 below:
[0110] Table 1
[0111]
[0112] The cumulative weights are shown in Table 2 below:
[0113] Table 2
[0114]
[0115] Calculated at a preset ratio of 0.5, the median weight is 1.2628. Then, in the order of sorting, starting from the first window data, it is successively determined whether the cumulative weights of a2, a3, a4, and a5 are greater than 1.2628. It can be first determined that the weight 2 of a3 is greater than 1.2628. Therefore, a3 is determined as the first data to be selected.
[0116] Since there is still one second data to be selected, a1, then a1 and a3 are aggregated, that is, the final target data stream obtained is a1, a3, and in terms of data values, it is: 11, 12.
[0117] Corresponding to Figure 1 the method described above, an embodiment of the present invention further provides a data filtering device for Figure 1 the specific implementation of the method in Figure 2 , the data filtering device provided by the embodiment of the present invention can be in a computer terminal or various mobile devices, and in combination with Figure 2 as shown in
[0118] a standard deviation calculation module 10, configured to obtain the original data stream in real time and calculate the standard deviation of the original data stream;
[0119] an instantaneous noise determination module 20, configured to compare the standard deviation with a preset standard deviation threshold, and if the standard deviation is not greater than the standard deviation threshold, determine that there is instantaneous noise;
[0120] a dynamic window function establishment module 30, configured to establish a dynamic window function according to the standard deviation;
[0121] A window size calculation module 40 is configured to calculate a window size corresponding to the original data stream based on the dynamic window function;
[0122] A filtering processing module 50 is configured to perform filtering processing on the instantaneous noise of the original data stream with the window size as a constraint to obtain a target data stream.
[0123] As can be seen from the above technical solution, in this application, the original data stream is obtained in real time, and the standard deviation of the original data stream is calculated; the standard deviation is compared with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream; a dynamic window function is established according to the standard deviation; the window size corresponding to the original data stream is calculated based on the dynamic window function; the instantaneous noise of the original data stream is filtered with the window size as a constraint to obtain a target data stream. This application takes into account the dynamics and real-time nature. The original data stream is obtained in real time, and then the standard deviation is calculated. The size of the standard deviation is used to determine whether there is instantaneous noise in the original data stream. If so, dynamic filtering processing is required. A dynamic window function is established according to the standard deviation, and the window size corresponding to the standard deviation is calculated based on this dynamic window function, so that the original data stream corresponds to the window size and is dynamic. Thus, the instantaneous noise of the original data stream is filtered with the window size as a constraint, which can solve the problem of poor adaptability caused by processing with a fixed window size in the prior art, improve the adaptability, effectively filter following the change of the noise intensity, and improve the accuracy of filtering.
[0124] Furthermore, an embodiment of this application provides a data filtering device. Optionally, Figure 3 shows a hardware structure block diagram of the data filtering device. Referring to Figure 3 , the hardware structure of the data filtering device may include: at least one processor 01, at least one communication interface 02, at least one memory 03, and at least one communication bus 04.
[0125] In the embodiment of this application, the number of the processor 01, the communication interface 02, the memory 03, and the communication bus 04 is at least one, and the processor 01, the communication interface 02, and the memory 03 complete mutual communication through the communication bus 04.
[0126] The processor 01 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.
[0127] The memory 03 may include a high-speed RAM memory and may also include a non-volatile memory, such as at least one disk memory.
[0128] Among them, the memory stores a program, and the processor can call the program stored in the memory. The program is used to execute the following data filtering method, including:
[0129] Obtain the original data stream in real time and calculate the standard deviation of the original data stream;
[0130] Compare the standard deviation with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream;
[0131] Establish a dynamic window function according to the standard deviation;
[0132] Calculate the window size corresponding to the original data stream based on the dynamic window function;
[0133] Filter the instantaneous noise of the original data stream with the window size as a constraint to obtain a target data stream.
[0134] Optionally, the refinement function and expansion function of the program can refer to the description of the data filtering method in the method embodiment.
[0135] The embodiment of the present application also provides a storage medium. The storage medium can store a program suitable for a processor to execute. When the program runs, it controls the device where the storage medium is located to execute the following data filtering method, including:
[0136] Obtain the original data stream in real time and calculate the standard deviation of the original data stream;
[0137] Compare the standard deviation with a preset standard deviation threshold. If the standard deviation is not greater than the standard deviation threshold, it is determined that there is instantaneous noise in the original data stream;
[0138] Establish a dynamic window function according to the standard deviation;
[0139] Calculate the window size corresponding to the original data stream based on the dynamic window function;
[0140] Filter the instantaneous noise of the original data stream with the window size as a constraint to obtain a target data stream.
[0141] Specifically, the storage medium can be a computer-readable storage medium, and the computer-readable storage medium can be an electronic memory such as a flash memory, EEPROM (electrically erasable programmable read-only memory), EPROM, hard disk, or ROM.
[0142] Optionally, the refinement function and expansion function of the program may refer to the description of the data filtering method in the method embodiments.
[0143] In addition, in each of the embodiments of the present disclosure, the functional modules may be integrated together to form an independent part, or each module may exist alone, or two or more modules may be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a live broadcast device, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present disclosure.
[0144] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the said element.
[0145] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other.
[0146] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data filtering method, characterized in that include: Acquire the original data stream in real time, and calculate the standard deviation of the original data stream; Comparing the standard deviation with a preset standard deviation threshold, and if the standard deviation is not greater than the standard deviation threshold, determining that instantaneous noise exists in the original data stream; Establish a dynamic window function based on standard deviation; Calculate a window size corresponding to the original data stream based on the dynamic window function; The instantaneous noise of the original data stream is filtered with the window size as a constraint to obtain a target data stream.
2. The method according to claim 1, characterized in that, The formula of the dynamic window function is: ； Among them, represents the window size, represents the minimum value of the window size, represents the maximum value of the window size, represents the standard deviation of the original data stream, represents the adjustment coefficient.
3. The method according to claim 1, characterized in that, The filtering process of the instantaneous noise of the original data stream with the window size as a constraint to obtain the target data stream includes: Determine the number of data in the original data stream and the order of data in the data stream; According to the number of data, starting from the last data in the data sequence, sequentially select data corresponding to the window size as each window data; A weighted median filter process is performed on each of the window data to obtain a target data stream.
4. The method according to claim 3, wherein The step of performing weighted median filtering on each of the window data to obtain a target data stream includes: Calculate the weight of each window data; Determine the data value of each of the window data; Sort the window data in ascending order of the data value; For each of the window data in the sorting, determining whether the window data is the first window data in the sorting; If so, the weight of the window data is used as its own cumulative weight; If not, in the sorting, the weights of the window data and all window data before the window data are summed up to obtain the cumulative weight of the window data; The data in each window is filtered according to the accumulated weight to obtain a target data stream.
5. The method according to claim 4, wherein The filtering of each of the window data according to the accumulated weight to obtain a target data stream comprises: Determine the cumulative weight of the last window data in the sorting; Multiplying the accumulated weight by a preset ratio to obtain a median weight; According to the sorting, starting from the first window data, it is determined in turn whether the cumulative weight of each window data is greater than the median weight; In the judgment process, the window data whose cumulative weight is greater than the median weight for the first time is taken as the first data to be selected; A target data stream is determined according to the first data to be selected.
6. The method according to claim 5, characterized in that, The step of determining the target data stream according to the first data to be selected includes: Determine the number of the window data; If the number of data in the original data stream is greater than the number of window data, each data in the original data stream except the window data is used as each second data to be selected; The first data to be selected and each of the second data to be selected are aggregated to obtain a target data stream.
7. The method according to any one of claims 4 to 6, characterized in that The calculation formula of the weight is: ； Among them, represents the weight of the th window data, represents the data value of the first data in the original data stream, represents the th window data value.
8. A data filtering device, characterized in that, include: A standard deviation calculation module, used to obtain the original data stream in real time and calculate the standard deviation of the original data stream; An instantaneous noise determination module is used to compare the standard deviation with a preset standard deviation threshold, and if the standard deviation is not greater than the standard deviation threshold, determine that instantaneous noise exists; A dynamic window function establishment module for establishing a dynamic window function according to the standard deviation; A window size calculation module for calculating a window size corresponding to the original data stream based on the dynamic window function; A filtering processing module for performing filtering processing of instantaneous noise on the original data stream with the window size as a constraint to obtain a target data stream.
9. A data filtering device, characterized in that, It includes a memory and a processor; The memory is used for storing programs; The processor is used for executing the program to implement each step of the data filtering method according to any one of claims 1-7.
10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, each step of the data filtering method according to any one of claims 1-7 is implemented.