Abnormal data screening method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an abnormal data screening method and device, electronic equipment and storage medium, belonging to the technical field of data processing. The method comprises: acquiring an aviation time series data sequence, and dividing the aviation time series data sequence into a plurality of data groups; acquiring an i-th element in one of the data groups; calculating error values of the i-th element and all elements after the i-th element in the data group in time sequence; if the error value is greater than a preset error threshold, marking the error value as an abnormal value, and acquiring a number of the abnormal values; if the number of the abnormal values corresponding to the i-th element is greater than a preset abnormal threshold, determining the i-th element as abnormal data; and acquiring the abnormal data in all the data groups. The abnormal data screening method provided by the embodiments of the present application is simple to calculate, easy to operate, and does not depend on the distribution form of data, and has high data screening precision.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to an abnormal data screening method and device, electronic equipment and storage medium. BACKGROUND

[0002] Aerial time series data has the characteristics of large quantity, multiple types, no obvious distribution rule, and low value density. How to effectively screen out abnormal data in aerial data while retaining valuable data has become an important research direction. The existing common data screening methods include hash calculation preprocessing method, Lillie criterion method, Z-score method, DBSCAN method, etc. Among them, the hash calculation preprocessing method focuses on the integrity of the data, and needs to input non-abnormal data to compare with abnormal data; the Lillie criterion method has strong limitations, and can only be used for processing normally or approximately normally distributed data, and can only eliminate gross error data, and the accuracy is slightly insufficient; the Z-score method must assume that the data satisfies Gaussian distribution; the DBSCAN method uses clustering algorithm, and is not suitable for large data amount. At present, there is no method suitable for screening abnormal data of aerial time series data. SUMMARY

[0003] In order to solve the above technical problems, the embodiments of the present application provide an abnormal data screening method, device, electronic equipment and storage medium.

[0004] In a first aspect, the embodiments of the present application provide an abnormal data screening method, which comprises:

[0005] Obtaining an aerial time series data sequence, and dividing the aerial time series data sequence into a plurality of data groups;

[0006] Obtaining an i-th element in a data group;

[0007] Calculating an error value of the i-th element and all elements after the i-th element in the data group in time sequence;

[0008] If the error value is greater than a preset error threshold, marking the error value as an abnormal value, and obtaining the number of abnormal values corresponding to the i-th element;

[0009] If the number of abnormal values corresponding to the i-th element is greater than a preset abnormal threshold, determining the i-th element as abnormal data;

[0010] Obtaining the abnormal data in all data groups.

[0011] In an embodiment, the dividing of the aerial time series data sequence into a plurality of data groups comprises:

[0012] The aviation time series data is divided into multiple data groups according to a preset time interval, and each data group is sorted in time sequence, wherein elements in each data group are sorted in time sequence.

[0013] In an embodiment, the error value of the ith element and all elements after the ith element in the data group is calculated in time sequence, including:

[0014] The error value of the ith element and the (i+1)th element is calculated by the formula

[0015]

[0016] The mean square value of the ith element and the (i+1)th element is calculated;

[0017] Wherein, A i is the mean square value, X i is the ith element, and X i+1 is the (i+1)th element.

[0018] The error value of the ith element and the (i+1)th element is calculated by the formula

[0019]

[0020] The error value of the ith element and the (i+1)th element is calculated by the formula

[0021] Wherein, n i is the error value.

[0022] The error value of the ith element and all elements after the ith element in the data group is calculated by the above formula.

[0023] In an embodiment, the method further includes:

[0024] All the abnormal data is deleted;

[0025] The gaps in each data group are interpolated.

[0026] In an embodiment, the gaps in each data group are interpolated, including:

[0027] A sliding window with a preset width is used to obtain the element time interval of adjacent two elements in the data group;

[0028] If the element time interval is less than twice the sampling period, no interpolation is performed;

[0029] If the element time interval is greater than or equal to twice the sampling period, interpolation is performed between the adjacent two elements.

[0030] In one embodiment, the step of interpolating between two adjacent elements if the time interval between the elements is greater than or equal to twice the sampling period includes:

[0031] If the time interval between the elements is equal to twice the sampling period, then the average value of all elements in the data set or the average value of two adjacent elements is used for interpolation.

[0032] If the time interval between the elements is greater than twice the sampling period, then a multiple interpolation algorithm is used to interpolate between the two adjacent elements.

[0033] Secondly, embodiments of this application provide an abnormal data filtering device, the device comprising:

[0034] A segmentation module is used to acquire aviation time-series data sequences and segment the aviation time-series data sequences into multiple data groups;

[0035] The first acquisition module is used to acquire the i-th element in one of the data groups;

[0036] The calculation module is used to calculate the error value between the i-th element and all elements after the i-th element in the data group in chronological order.

[0037] The marking module is used to mark the error value as an outlier if the error value is greater than a preset error threshold, and to obtain the number of outliers.

[0038] The determination module is used to determine the i-th element as abnormal data if the number of abnormal values ​​corresponding to the i-th element is greater than a preset abnormal threshold.

[0039] The second acquisition module is used to acquire the abnormal data in all of the data groups.

[0040] In one embodiment, the apparatus further includes:

[0041] The deletion module is used to delete all the aforementioned abnormal data;

[0042] The interpolation module is used to interpolate empty spaces in each of the data groups.

[0043] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the computer program executes the abnormal data filtering method provided in the first aspect when the processor is running.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a processor, executes the abnormal data filtering method provided in the first aspect.

[0045] The abnormal data screening method provided in the present application comprises the following steps: obtaining an aviation time series data sequence, dividing the aviation time series data sequence into a plurality of data groups, obtaining an i-th element in one of the data groups, calculating an error value of the i-th element and all elements after the i-th element in the data group in time sequence, marking the error value as an abnormal value if the error value is greater than a preset error threshold, obtaining a number of the abnormal values, determining the i-th element as abnormal data if the number of the abnormal values corresponding to the i-th element is greater than a preset abnormal threshold, and obtaining the abnormal data in all the data groups. The abnormal data screening method provided in the present application is simple to calculate, easy to operate, and does not depend on the distribution form of data, and has high data screening precision. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as limiting the scope of protection of the present application. In each drawing, similar components are denoted by similar reference numerals.

[0047] Figure 1 A flowchart of the abnormal data screening method provided in the present application is shown;

[0048] Figure 2 A structural diagram of the abnormal data screening device provided in the present application is shown. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application.

[0050] The components of the embodiments of the present application generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0051] Hereinafter, the terms "include", "has", and their conjugations, which can be used in various embodiments of the present application, are merely intended to denote a specific characteristic, number, step, operation, element, component, or a combination thereof, and are not intended to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0052] In addition, the terms "first", "second", "third", and the like are used only to distinguish descriptions, and are not to be understood as indicating or implying relative importance.

[0053] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terms (such as terms defined in a generally used dictionary) will be interpreted to have the same meaning as the contextual meaning in the relevant technical field and will not be interpreted to have idealized or overly formal meanings, unless clearly defined in various embodiments of the present application.

[0054] Embodiment 1

[0055] In the big data environment, data will accumulate quickly. It is very important to analyze the value contained in the massive data and to classify and filter out the valuable data. The existing data analysis and filtering algorithms have their applicable environment, and the characteristics of high data complexity and large data volume of big data increase the difficulty of selecting a suitable data filtering algorithm.

[0056] The time series data in the aviation data computer, such as airspeed, azimuth, heading, distance, etc., has the characteristics of large data volume, multiple types, no obvious distribution rule, and low value density. How to more fully and effectively mine the potential data value of aviation time series data, reduce unnecessary calculation loss, and achieve the purpose of optimizing productivity is an important problem.

[0057] Based on this, the embodiment of the present disclosure provides an abnormal data filtering method, which can classify and store according to different types of data on the basis of retaining the original data features to the maximum extent.

[0058] Specifically, referring to Figure 1 , the abnormal data filtering method comprises:

[0059] Step S110, acquiring an aviation time series data sequence, and dividing the aviation time series data sequence into a plurality of data groups;

[0060] The aviation time-series data sequence can be a set of data obtained at each time point in the aviation process, such as airspeed, azimuth, heading, distance, etc. These data are arranged in time sequence from front to back in the aviation time-series data sequence.

[0061] Because the data amount of the aviation time-series data is large, the aviation time-series data sequence is split. The aviation time-series data sequence is segmented into a plurality of data groups according to a preset time interval, and the obtained data groups are also arranged in time sequence.

[0062] Step S120, obtaining an i-th element in a data group;

[0063] Taking the first data X1 in a data group as an example when calculating for the first time, the process of calculating the error value is described in combination with step S130.

[0064] Step S130, calculating an error value of the i-th element and all elements after the i-th element in the data group in time sequence;

[0065] Comparing the first data X1 with its next data X2, the error value of X1 and X2 is calculated, if the error value meets a preset error threshold, the error value of X1 and X3 is calculated, and so on. Assuming that there are y data in a group, until the error value of X1 and the last element Xy in the group is calculated. y The calculation is completed. At this time, y error values are obtained.

[0066] When the data obtained in the same data group is X i and X i+1 , according to the above idea, the mean square value of X i and X i+1 can be calculated using formula 1, formula 1:

[0067]

[0068] Wherein, A i is the mean square value, X i is the i-th element, and X i+1 is the i+1-th element.

[0069] Then the error value n i = |a-b| can be calculated, where a = |A i -X i 2 |, b = |A i -X i+1 2 |.

[0070] In addition, if the whole set of data of a certain group of data deviates too much, the group of data can be directly discarded. Specifically, because the standard deviation is sensitive to abnormal values in data, the standard deviation formula is used to determine in each group of data corresponding to each time period. All data in the group is compared by formula 2 using the standard deviation, and the standard deviation of each group of data is used to determine the selection of the group of data.

[0071] Formula 2:

[0072]

[0073] where X g represents the mean of the group of data.

[0074] By comparing the e values of multiple normal groups of data, if the e value of an abnormal group of data deviates by more than 8 units from the e value of a normal group of data, the group of data can be discarded without further repair processing or selection for other analysis.

[0075] In step S140, if the error value is greater than the preset error threshold, the error value is marked as an abnormal value, and the number of abnormal values is obtained.

[0076] The error values greater than the preset error threshold are counted and grouped separately, and the number of data in the group is obtained.

[0077] The preset error threshold m is specified according to different needs. In an embodiment, m ∈ (-f, f), where f = 3 * AVERAGE(X t ,X t+1 ), AVERAGE(X t ,X t+1 ) represents the average of the data group. When ni is within the preset error threshold m, it can be determined that the error is small. In actual application, f is generally 1-3 times the average of the group of data, and too large or too small is not representative. The smaller the threshold range, the higher the accuracy of the screening. Continue to compare X i , X i+2 , X i+3 , and so on. When the number of items within the preset error threshold is greater than the preset abnormal threshold (for example, two-thirds of the number of data in the group), the current data is determined to be normal data, and is determined to be the first type of data. Otherwise, it is abnormal data, and is determined to be the second type of data.

[0078] In step S150, if the number of abnormal values corresponding to the ith element is greater than the preset abnormal threshold, the ith element is determined to be abnormal data.

[0079] If the number is greater than a certain index, it means that the i-th element obtained has a problem, at this time, the data group with the problem is screened out as abnormal data, and the comparison continues from the i+1-th data until all the data in a data group are compared with or compared by other data.

[0080] In step S160, the abnormal data in all the data groups are obtained.

[0081] In addition, in an embodiment, the abnormal data screening method provided by the embodiment further comprises:

[0082] All the abnormal data are deleted, and the empty positions in each data group are interpolated.

[0083] After the abnormal data are screened out, empty positions are generated in the original data group, in order to avoid the influence of the empty positions on the whole data, interpolation can be performed under the condition that the accuracy of the data is not affected.

[0084] The embodiment adopts an improved multiple interpolation algorithm (MI) to screen, classify and repair data, specifically, a sliding window model based on a fixed window width is adopted.

[0085] The window width can be set as N, the data missing threshold value can be set as M, and the sampling frequency of the data can be set as T. First, the time interval Δt of adjacent two data X i and X i+1 in the sliding window is calculated, and Δt and M*T are compared.

[0086] When Δt<2*T, the data normally arrives in the sliding window, there is no data missing, and no interpolation repair processing is needed.

[0087] When 2*T≤Δt≤M*T, there is data missing between the data points X i and X i+1 , but the number of missing data is less than the threshold value, if M=2, the average value of the data or the median value of the data before and after is used for interpolation.

[0088] When Δt≥M*T, the number of missing data exceeds the threshold value, and the multiple interpolation algorithm is used for interpolation repair.

[0089] The multiple interpolation algorithm generates multiple possible filling values for each missing item in the data group, and these filling values reflect the uncertainty of the non-response model: first, each generated possible value can be used to fill the blank value in the data column, and the generated values do not consider the combination between each other to generate data. Second, each interpolation data set is statistically analyzed by using statistical methods for complete data sets. Third, according to the score function, the results from each interpolation data set are selected to generate the final interpolation value.

[0090] Suppose a period of data, containing three missing values X a , X b , X c , process this set of data into three groups, group A keeps the original data unchanged, group B only missing values X c , group C missing values X a and X b . When multiple value interpolation is performed, no processing will be performed on group A, a set of estimated values of X c is generated for group B, for example, regression of X c about X a , X b ; a set of paired estimated values of X a and X b is generated for group C, for example, regression of X a , X b about X c .

[0091] In this way, the original data can be screened and classified to better interpolate and repair abnormal data in the original data.

[0092] The abnormal data screening method provided in this embodiment divides data according to time periods, and calculates the data in the time periods. The long data is split into multiple short data to avoid the situation that the calculation power is not enough due to the import of complete data. The modulus value of the difference between the mean square of the front and rear data and the specified threshold value is compared, the difference value between the two data is amplified and highlighted. This operation method greatly increases the screening accuracy on the basis of effectively reducing the calculation difficulty. The standard deviation is used to associate the data between the test groups and the groups, which can effectively reflect the correlation between the groups and the data. That is, whether the group data needs to be retained is determined by the error value. After the original data is screened and classified, the abnormal data in the original data can be better interpolated and repaired. In summary, the abnormal data screening method provided in this embodiment is simple to calculate, easy to operate, and does not depend on the distribution form of the data, and the screening data has high precision.

[0093] Embodiment 2

[0094] In addition, the disclosure embodiment provides an abnormal data screening device.

[0095] Specifically, as shown in Figure 2 , the abnormal data screening device 200 comprises:

[0096] The segmentation module 210 is configured to obtain an aviation time series data sequence, and segment the aviation time series data sequence into a plurality of data groups.

[0097] The first acquisition module 220 is configured to acquire an i-th element in one of the data groups.

[0098] The computing module 230 is configured to calculate error values of the i-th element and all elements after the i-th element in the data set in time sequence.

[0099] The marking module 240 is configured to mark the error value as an abnormal value if the error value is greater than a preset error threshold, and obtain a number of the abnormal values.

[0100] The determining module 250 is configured to determine the i-th element as abnormal data if the number of the abnormal values corresponding to the i-th element is greater than a preset abnormal threshold.

[0101] The second obtaining module 260 is configured to obtain the abnormal data in all the data sets.

[0102] The abnormal data screening device 200 further includes:

[0103] The deleting module is configured to delete all the abnormal data.

[0104] The interpolation module is configured to interpolate the empty positions in each data set.

[0105] The abnormal data screening device 200 provided in the embodiment can implement the abnormal data screening method provided in Embodiment 1, and details are not repeated here to avoid repetition.

[0106] The abnormal data screening device provided in the embodiment is simple to calculate, easy to operate, and does not depend on the distribution form of data, and has high screening data precision.

[0107] Embodiment 3

[0108] In addition, the disclosure provides an electronic device including a memory and a processor, the memory stores a computer program, and the computer program executes the abnormal data screening method provided in Embodiment 1 when running on the processor.

[0109] The electronic device provided in the embodiment can implement the abnormal data screening method provided in Embodiment 1, and details are not repeated here to avoid repetition.

[0110] The electronic device provided in the embodiment is simple to calculate, easy to operate, and does not depend on the distribution form of data, and has high screening data precision.

[0111] Embodiment 4

[0112] The present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program implements the abnormal data screening method provided in Embodiment 1 when executed by a processor.

[0113] In the embodiment, the computer readable storage medium can be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, etc.

[0114] The computer readable storage medium provided in the embodiment can implement the abnormal data screening method provided in the embodiment 1, and details are not described herein again to avoid repetition.

[0115] It should be noted that, in this document, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or terminal. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of another identical element in the process, method, article or terminal including the element.

[0116] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for making a terminal (which can be a mobile phone, computer, server, air conditioner or network device, etc.) execute the methods described in various embodiments of the present application.

[0117] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-mentioned specific embodiments, and the above-mentioned specific embodiments are only illustrative, not restrictive. Those skilled in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the protection scope of the claims.

Claims

1. A method for filtering outomas, characterized in that, The method includes: Acquire an aviation time-series data sequence and divide the aviation time-series data sequence into multiple data groups; Retrieve the i-th element from the data set; Calculate the error value between the i-th element and all elements after the i-th element in the data group in chronological order; If the error value is greater than a preset error threshold, the error value is marked as an outlier, and the number of outliers is obtained. If the number of outliers corresponding to the i-th element is greater than a preset outlier threshold, then the i-th element is determined to be outlier data. Retrieve the abnormal data from all of the data groups; The step of calculating the error value between the i-th element and all elements after the i-th element in the data group according to the time sequence includes: Through formula Calculate the mean square value of the i-th element and the (i+1)-th element; in, A i The mean square value, X i For the i-th element, X i+1 This refers to the (i+1)th element; Through formula Calculate the error value between the i-th element and the (i+1)-th element; in, n i The error value is... The above formula is used to calculate the error value between the i-th element and all elements after the (i+1)-th element in the data set.

2. The abnormal data filtering method according to claim 1, characterized in that, The step of dividing the aeronautical time-series data sequence into multiple data groups includes: The aviation time series data is divided into multiple data groups according to a preset time interval, and each data group is sorted according to time order, wherein the elements in each data group are sorted according to time order.

3. The abnormal data filtering method according to claim 1, characterized in that, The method further includes: Delete all of the aforementioned abnormal data; The empty spaces in each of the data groups are interpolated.

4. The abnormal data filtering method according to claim 3, characterized in that, The process of interpolating empty spaces in each of the data groups includes: The element time interval between two adjacent elements in the data group is obtained using a sliding window of preset width; If the time interval between the elements is less than twice the sampling period, no interpolation is performed; If the time interval between the elements is greater than or equal to twice the sampling period, interpolation is performed between the two adjacent elements.

5. The abnormal data filtering method according to claim 4, characterized in that, If the time interval between the elements is greater than or equal to twice the sampling period, interpolation is performed between the two adjacent elements, including: If the time interval between the elements is equal to twice the sampling period, then the average value of all elements in the data set or the average value of two adjacent elements is used for interpolation. If the time interval between the elements is greater than twice the sampling period, then a multiple interpolation algorithm is used to interpolate between the two adjacent elements.

6. An abnormal data filtering device, characterized in that, The device includes: A segmentation module is used to acquire aviation time-series data sequences and segment the aviation time-series data sequences into multiple data groups; The first acquisition module is used to acquire the i-th element in one of the data groups; The calculation module is used to calculate the error value between the i-th element and all elements after the i-th element in the data group in chronological order. The marking module is used to mark the error value as an outlier if the error value is greater than a preset error threshold, and to obtain the number of outliers. The determination module is used to determine the i-th element as abnormal data if the number of abnormal values ​​corresponding to the i-th element is greater than a preset abnormal threshold. The second acquisition module is used to acquire the abnormal data in all of the data groups; The step of calculating the error value between the i-th element and all elements after the i-th element in the data group according to the time sequence includes: Through formula Calculate the mean square value of the i-th element and the (i+1)-th element; in, A i The mean square value, X i For the i-th element, X i+1 This refers to the (i+1)th element; Through formula Calculate the error value between the i-th element and the (i+1)-th element; in, n i The error value is... The above formula is used to calculate the error value between the i-th element and all elements after the (i+1)-th element in the data set.

7. The abnormal data filtering device according to claim 6, characterized in that, The device further includes: The deletion module is used to delete all the aforementioned abnormal data; The interpolation module is used to interpolate empty spaces in each of the data groups.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that executes the abnormal data filtering method according to any one of claims 1 to 5 when the processor is running.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when run on a processor, executes the abnormal data filtering method according to any one of claims 1 to 5.

Images