A method and device for determining abnormal data

Abstract

The present specification discloses a method and device for determining abnormal data. The data is cleaned to obtain the data to be identified, and the distance between the data to be identified and other data in the current time window is determined. According to the preset number and the determined distance, the neighborhood of the data to be identified is determined, and the adjacent data is determined. Then, according to the time difference between the adjacent data and the data to be identified, the time factor is determined, the first distance is determined according to the distance between the data to be identified and the adjacent data, and the second distance is determined according to the distance between the adjacent data and each data in its own neighborhood. Then, the density factor is determined according to the first distance and the second distance, and the abnormality of the data to be identified is determined according to the time factor and the density factor to determine the abnormal data. The time factor and the density factor can be determined according to the time difference of each data and other data in the neighborhood where the data to be identified is located, the abnormality of the data to be identified is calculated in real time, and problems with the tasks corresponding to the abnormal data are discovered in time.

Description

Technical Field

[0001] The present invention relates to the field of Internet, and more particularly to a method and device for determining abnormal data. Background Art

[0002] With the development and widespread application of Internet technology, various Internet-based platforms and systems have become increasingly abundant, such as game systems, financial systems, etc. The operation of these systems will generate a large amount of data. By determining abnormal data, it is possible to detect whether there are abnormalities in the tasks corresponding to the abnormal data, so as to discover and solve problems in a timely manner.

[0003] At present, when determining abnormal data, the data is usually processed in stages. In the prior art, after collecting the data to be identified, the data must first be cleaned, and then the timing curves of the data must be classified, and the types of timing curves must be divided into two types: strong periodicity and weak periodicity. After obtaining the classification results, different algorithms are used to determine whether the data is abnormal for different timing curve classification results. However, the results obtained by the methods for determining abnormal data in the prior art are not precise and accurate enough, and the efficiency is low. The feedback of the determined results is slow, resulting in the inability to detect abnormal data in a timely manner, and even more so, the inability to take timely measures to solve the problem of tasks corresponding to abnormal data. Summary of the invention

[0004] This specification provides a method and device for determining abnormal data to partially solve the above-mentioned problems existing in the prior art.

[0005] This manual adopts the following technical solutions:

[0006] This specification provides a method for determining abnormal data, specifically including:

[0007] Clean the acquired data to obtain the data to be identified;

[0008] For each piece of data to be identified, determine the distance between the data to be identified and other data in the current time window;

[0009] According to the preset number and the determined distance between the data to be identified and other data in the current time window, a neighborhood of the data to be identified is determined, and each data in the neighborhood is determined to be adjacent data;

[0010] Determine a time difference between the adjacent data and the data to be identified in the current time window, and determine a time factor according to the time difference, wherein the time difference is negatively correlated with the time factor;

[0011] Determine a first distance according to the distance between the data to be identified and the adjacent data, and determine a second distance according to the distance between the adjacent data and each data in the neighborhood of the adjacent data;

[0012] determining a density factor according to the first distance and the second distance;

[0013] According to the time factor and the density factor, the abnormality of the data to be identified is determined through a real-time calculation framework, and according to the abnormality of the data to be identified, it is determined whether the data to be identified is abnormal data.

[0014] Optionally, determining the distance between the data to be identified and other data in the current time window specifically includes:

[0015] Determine the information entropy of the data to be identified in each preset feature dimension;

[0016] According to the determined information entropy difference of the data to be identified in each feature dimension, the distance between the data to be identified and other data in the current time window is determined.

[0017] Optionally, determining the information entropy difference of the data to be identified in each feature dimension specifically includes:

[0018] For each feature dimension, according to the feature value of the data to be identified in the feature dimension, determine the information entropy corresponding to the data to be identified in the feature dimension;

[0019] Determine the residual dimension information entropy corresponding to the feature dimension according to the information entropies corresponding to the other feature dimensions excluding the feature dimension;

[0020] Determine the information entropy difference of the data to be identified in the feature dimension according to the difference between the total information entropy of the data to be identified and the codimensional information entropy;

[0021] The total information entropy is determined according to the information entropy corresponding to each feature dimension of the data to be identified.

[0022] Optionally, determining the first distance according to the distance between the data to be identified and the adjacent data specifically includes:

[0023] According to the information entropy difference of the data to be identified in each feature dimension, the corresponding weight of the data to be identified in each feature dimension is determined respectively;

[0024] For each adjacent data of the data to be identified, determine the weighted distance between the data to be identified and the adjacent data according to the weights corresponding to the data to be identified in each feature dimension and the feature values ​​of the data to be identified and the adjacent data in each feature dimension;

[0025] A first distance is determined according to the weighted distance between the data to be identified and each adjacent data.

[0026] Optionally, according to the time factor and the density factor, determining the abnormality of the data to be identified through a real-time calculation framework specifically includes:

[0027] Determining the outliers of the data to be identified according to the time factor and the density factor;

[0028] Determine a preset abnormality degree range within which the abnormal value falls, and determine the abnormality degree of the data to be identified according to the preset abnormality degree range within which the abnormal value falls.

[0029] Optionally, before determining the abnormality of the data to be identified, the method further includes:

[0030] Judging whether the data to be identified is normal according to the abnormality of the data to be identified in the previous time window;

[0031] If yes, the abnormality degree of the data to be identified in the previous time window is used as the abnormality degree of the data to be identified in the current time window;

[0032] If not, the abnormality of the data to be identified in the current time window is re-determined.

[0033] Optionally, before determining the abnormality of the data to be identified, the method further includes:

[0034] Determine whether the data to be identified is included in the previous time window;

[0035] If yes, there is no need to update the abnormality of the data to be identified;

[0036] If not, the abnormality of the data to be identified is determined.

[0037] This specification provides a device for determining abnormal data, the device specifically comprising:

[0038] The module for acquiring data to be identified is used to clean the acquired data to obtain the data to be identified;

[0039] A data distance determination module is used to determine the distance between each piece of data to be identified and other data in the time window;

[0040] A neighborhood determination module, used to determine the neighborhood of the data to be identified according to a preset number and the determined distance between the data to be identified and other data in the time window, and determine each data in the neighborhood as adjacent data;

[0041] A time factor determination module, used to determine the time difference between the adjacent data and the data to be identified in the time window, and determine the time factor according to the time difference, wherein the time difference is negatively correlated with the time factor;

[0042] A neighborhood distance determination module, configured to determine a first distance according to a distance between the data to be identified and the adjacent data, and to determine a second distance according to a distance between the adjacent data and each data in a neighborhood of the adjacent data;

[0043] A density factor determination module, configured to determine a density factor according to the first distance and the second distance;

[0044] The abnormal data determination module is used to determine the abnormality of the data to be identified through a real-time calculation framework according to the time factor and the density factor, and determine whether the data to be identified is abnormal data according to the abnormality of the data to be identified.

[0045] This specification provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the method for determining abnormal data is implemented.

[0046] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for determining abnormal data when executing the program.

[0047] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0048] In the method for determining abnormal data provided in this specification, the data is cleaned to obtain the data to be identified, and the distance between the data to be identified and other data in the current time window is determined. According to the preset number and the determined distance, the neighborhood of the data to be identified is determined, and the adjacent data is determined. Then, according to the time difference between the adjacent data and the data to be identified, the time factor is determined, the first distance is determined according to the distance between the data to be identified and the adjacent data, and the second distance is determined according to the distance between the adjacent data and each data in its own neighborhood. Then, the density factor is determined according to the first distance and the second distance, and the abnormality of the data to be identified is determined according to the time factor and the density factor to determine the abnormal data.

[0049] It can be seen from the above method that this method can determine the time factor and density factor according to the time difference of each data in the time window and other data in the neighborhood of the data to be identified, and calculate the abnormality of the data to be identified in real time according to the obtained time factor and density factor, so as to obtain a precise and accurate result and promptly discover the problems existing in the task corresponding to the abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0051] Figure 1 A flowchart of a method for determining abnormal data in this specification;

[0052] Figure 2 A schematic diagram of adjacent data of data to be identified provided in this specification;

[0053] Figure 3 A schematic diagram of a data to be identified and its neighborhood provided in this specification;

[0054] Figure 4 A schematic diagram of a time window provided in this specification;

[0055] Figure 5 A schematic diagram of a device for determining abnormal data provided in this specification;

[0056] Figure 6 The corresponding Figure 1 Schematic diagram of the structure of an electronic device. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.

[0058] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0059] At present, when determining abnormal data, it is usually necessary to determine the fluctuation value of the time series curve of each data, and cluster the fluctuation value of each time series curve to obtain multiple initial classifications, and then from the initial classifications, according to the size of the fluctuation value of the time series curve, the time series curve is finally divided into two categories: a curve with sudden increase and decrease and weak periodicity, and a curve with gentle fluctuation and strong periodicity. In the prior art, the two categories of time series curves correspond to two algorithms for determining abnormal data. After obtaining two different classification results, it is necessary to determine the predicted data of the data corresponding to the time series curve by different prediction algorithms for different time series curve classification results, and finally take the difference between the true value of the data corresponding to the time series curve at each moment and the value of the predicted data corresponding to each moment, and determine the data with a difference greater than a threshold as abnormal data.

[0060] Although the existing technology can determine the abnormality of data, the method adopted by the existing technology requires classification first, and then adopts different algorithms to determine whether the data is abnormal, and the efficiency of judging abnormal data is low.

[0061] In order to solve the problem in the prior art that the judgment efficiency of abnormal data is low and abnormal data cannot be discovered in time, the present application provides a method for determining abnormal data.

[0062] Figure 1 The following is a flow chart of a method for determining abnormal data in this specification, which specifically includes the following steps:

[0063] S100: Clean the acquired data to obtain data to be identified.

[0064] In this specification, the method for determining abnormal data can be used to determine the abnormality of time series data. Time series data, that is, time series data, is a data column recorded in chronological order. The system will generate a large amount of time series data in the process of executing various tasks. By determining the abnormality of the data at each moment, it can be detected whether there is an abnormality in the task corresponding to the data. However, there are still data that do not meet the requirements in the data generated by the system at each moment, such as incomplete data, erroneous data, duplicate data, etc. This type of data is usually not used for data processing due to its own errors, so it is generally not necessary to determine the abnormality. Therefore, after obtaining the data generated at each moment, the data can be cleaned first to filter out the data that does not meet the requirements. After the acquired data is cleaned and filtered, the data obtained is the data for which the abnormality needs to be determined, that is, the data to be identified.

[0065] In one or more embodiments of the present specification, the method for cleaning the acquired data may be to use data warehouse technology (Extract-Transform-Load, ETL). In the process of using ETL to clean data, duplicate and erroneous data may be deleted, and incomplete data may be deleted or its missing values ​​may be added to make it complete.

[0066] In one or more embodiments of this specification, the method for determining abnormal data can be specifically executed by a server of a data warehouse to determine whether there is an abnormality in the data stored in the data warehouse. Alternatively, it can also be executed by a server of a system that generates data to determine the business in which the abnormality occurs by identifying the abnormality of the data. Alternatively, the method for determining abnormal data can also be executed by a third-party server, and this specification does not limit it. Of course, for the convenience of description, the method for determining abnormal data executed by a server will be described below.

[0067] S102: For each piece of data to be identified, determine the distance between the data to be identified and other data in the current time window.

[0068] In one or more embodiments of the present specification, after determining each piece of data to be identified, the server can determine the distance between the data to be identified and other data in the current time window for each piece of data to be identified. The current time window can be determined after cleaning the acquired data and obtaining each piece of data to be identified, and then the current time window is determined to determine each piece of data to be identified that falls within the current time window. Alternatively, the data that falls within the current time window can be determined from the acquired data first, and then the data that falls within the current time window is cleaned to obtain each piece of data to be identified. When determining the time window, the length of the time window can be set as needed, for example, 1 month, 1 week, etc., and this specification does not impose any restrictions.

[0069] Specifically, the server can determine the feature vectors of each piece of data to be identified according to the preset feature dimensions and the feature values ​​of each piece of data to be identified in each feature dimension. Then, for each piece of data to be identified, the distance between the piece of data to be identified and other pieces of data to be identified is determined according to the feature vector of the piece of data to be identified and the feature vectors of other pieces of data to be identified. That is, for any two pieces of data to be identified in each piece of data to be identified, the distance between the two pieces of data to be identified can be determined.

[0070] Furthermore, the data of different feature dimensions have different effects on distinguishing whether the data to be identified is abnormal. For example, assuming that for large household appliances, the number of units purchased by household users is usually one, then the feature value of the dimension of the quantity of goods plays a more important role in distinguishing whether the data is abnormal. However, since there are many different sizes of similar products on the market, such as televisions of different sizes, the feature value of the dimension of the size of the household appliance has a lower effect on distinguishing whether the data is abnormal than the previous feature dimension.

[0071] Therefore, in this specification, since the ultimate purpose of determining the distance between each data to be identified is to determine the abnormality of the data to be identified, when determining the distance between each data to be identified, the influence of the contribution of the feature values ​​of different feature dimensions on the distance between the data can also be considered, that is, the weighted distance can be determined to make the abnormality determined according to the weighted distance more accurate.

[0072] Specifically, for each data to be identified, the server can determine the characteristic value of the data to be identified on the characteristic dimension for each characteristic dimension of the data to be identified, and determine the information entropy corresponding to the data to be identified on the characteristic dimension. Then, according to the information entropies corresponding to the other characteristic dimensions except the characteristic dimension, the residual dimension information entropy corresponding to the characteristic dimension is determined, and according to the difference between the total information entropy of the data to be identified and the residual dimension information entropy, the information entropy difference of the data to be identified on the characteristic dimension is determined. Then, according to the determined information entropy difference of the data to be identified on each characteristic dimension, the distance between the data to be identified and other data on the current time window can be determined. Among them, the total information entropy is determined by summing the information entropies corresponding to the data to be identified on each characteristic dimension. The distance between the data to be identified and other data on the current time window can be a cosine distance, which can be set as needed, and this specification does not limit it here.

[0073] The formula for determining the total information entropy is:

[0074]

[0075] Among them, s is the number of feature dimensions included in the data to be identified, x i represents the feature value of the i-th feature dimension of the data to be identified, P(x i ) is x i The probability function of .

[0076] Furthermore, after determining the total information entropy of the data to be identified, the server may also remove the data of the feature dimension in the data to be identified for each feature dimension of the data to be identified, and then calculate the information entropy of each feature dimension of the data to be identified after removing the feature dimension, that is, the residual dimension information entropy corresponding to the feature dimension. The difference between the total information entropy and the residual dimension information entropy corresponding to the feature dimension is taken to obtain the information entropy difference of the data to be identified on the feature dimension, and the information entropy difference represents the degree of influence (or contribution) of the eigenvalue of the feature dimension on the total information entropy. The larger the information entropy difference, the greater the degree of influence of the eigenvalue of the feature dimension on the total information entropy, and vice versa.

[0077] After determining the information entropy differences of the data to be identified in each feature dimension, each information entropy difference can also be normalized, and the normalized value is used as the corresponding weight of the data to be identified in each feature dimension.

[0078] Therefore, in one or more embodiments of the present specification, when determining the distance between the data to be identified and other data, the weighted distance between the data to be identified and other data may also be determined based on the weights corresponding to each feature dimension of the data to be identified.

[0079] Specifically, when determining the weighted distance between the data to be identified and other data, the server can respectively determine the weights corresponding to the data to be identified in each feature dimension, and then for each other data of the data to be identified, determine the weighted distance between the data to be identified and the other data according to the weights corresponding to the data to be identified in each feature dimension and the feature values ​​of the data to be identified and the other data in each feature dimension.

[0080] The formula for determining the weighted distance between the data to be identified and other data is as follows:

[0081]

[0082] Among them, s is the number of feature dimensions included in the data to be identified, x represents the data to be identified, y represents other data in the current time window, and x i is the eigenvalue of the data to be identified in the i-th feature dimension, y i is the eigenvalue of the i-th feature dimension of the other data, ω i is the weight corresponding to the i-th feature dimension of the data to be identified, and 0≤ω i ≤1.

[0083] S104: Determine the neighborhood of the data to be identified according to the preset number and the determined distance between the data to be identified and other data in the current time window, and determine each data in the neighborhood as adjacent data.

[0084] After determining the distance between the data to be identified and other data in the current time window, the server can determine the neighborhood of the data to be identified based on the determined distances and a preset number, and determine each data in the neighborhood as adjacent data. The preset number refers to the number of data included in the neighborhood of the data to be identified, that is, the number of adjacent data.

[0085] In one or more embodiments of the present specification, the neighborhood of the data to be identified may be a k-neighborhood. Then k represents the number of data contained in the neighborhood of the data to be identified. For example, assuming that k is 5, that is, the preset number is 5, when determining the neighborhood of the data to be identified, the distance between the data to be identified and other data in the current time window can be determined, and the first 5 data can be sorted from small to large, and the first 5 data can be used as the neighboring data of the data to be identified.

[0086] S106: Determine the time difference between the adjacent data and the data to be identified in the current time window, and determine the time factor according to the time difference.

[0087] In one or more embodiments of the present specification, after determining the adjacent data in the neighborhood of the data to be identified, the server may determine the time difference between each adjacent data and the data to be identified in the current time window, and determine the time factor corresponding to each adjacent data according to the determined time difference, wherein the time difference is negatively correlated with the time factor.

[0088] Since each adjacent data in the neighborhood of the data to be identified corresponds to different time points when determining the abnormality of the data to be identified, the adjacent data that is closer to the time point corresponding to the data to be identified in the time window has a greater impact on the abnormality of the data to be identified. Therefore, the server can determine the time factor based on the time difference between each adjacent data and the data to be identified in the current time window to represent the weight corresponding to each adjacent data in time when determining the abnormality of the data to be identified.

[0089] Figure 2A schematic diagram of adjacent data of a data to be identified provided in this specification. As shown in the figure, X represents the data to be identified, and its corresponding time is t4, A, B, C, D, E are 5 adjacent data in the neighborhood of the data to be identified X, corresponding to t1, t2, t3, t5, t6, 5 times respectively. As can be seen from the figure, on the time axis, the t3 moment where the adjacent data C is located and the t5 moment where the adjacent data D is located are closer in time to the t4 moment where the data to be identified X is located, so, compared to data A, B, E, data C and data D have a greater impact on the abnormality of the data to be identified X. Similarly, compared to the t1 moment where data A is located, since the t2 moment where data B is located is closer to the t4 moment where the data to be identified X is located, the impact of data B on the abnormality of the data to be identified X is greater than that of data A.

[0090] The formula for determining the time factor is:

[0091]

[0092] Wherein, e is a natural constant, |t1-t2| represents the time difference between the data to be identified and a data in its neighborhood, and λ is a constant, which can be set as needed and is not limited in this specification.

[0093] S108: Determine a first distance according to the distance between the data to be identified and the adjacent data, and determine a second distance according to the distance between the adjacent data and each data in a neighborhood of the adjacent data.

[0094] After determining the weighted distance between the data to be identified and each adjacent data in its neighborhood, the server may determine the first distance according to the determined weighted distance between the data to be identified and each adjacent data.

[0095] In one or more embodiments of the present specification, the first distance may be an average distance obtained by averaging the weighted distances between the data to be identified and each adjacent data, or may be the median of the weighted distances between the data to be identified and each adjacent data, and so on. The specific setting may be based on actual needs and the present specification does not impose any limitation thereto.

[0096] In one or more embodiments of the present specification, after determining the adjacent data of the data to be identified, the server may also determine, for each adjacent data, a weighted distance between the adjacent data and other data, and determine a neighborhood of the adjacent data through the same process as steps S102 to S104, and then determine a second distance based on the weighted distance between each data in the neighborhood of the adjacent data and the adjacent data.

[0097] Figure 3A schematic diagram of a data to be identified and its neighborhood provided in this specification. As shown in the figure, X represents a data to be identified. Taking the number of data in the preset neighborhood of the data to be identified as 6 as an example, A, B, C, D, E, and F are six adjacent data in the neighborhood of the data to be identified X, and the first distance is determined based on the weighted distance between the data to be identified X and the six adjacent data A, B, C, D, E, and F. G, H, I, J, E, and X are data in the neighborhood of the adjacent data F of the data to be identified X, and the second distance is determined based on the weighted distance between the adjacent data F and the six data G, H, I, J, E, and X.

[0098] S110: Determine a density factor according to the first distance and the second distance.

[0099] In various Internet-based platforms and systems, when there is a problem with a task in the system, there may be a situation where the amount of data is relatively dense for a period of time. For example, taking the system as a takeaway platform system, when a merchant in the platform is doing an event, a large amount of order data may be generated during the peak period. When the merchant is not doing an event, if a large amount of order data is generated within a certain period of time, it means that there may be a problem with the corresponding task in the system. Then, it can be further determined whether the order data is abnormal based on the relationship between the quantity of goods in the order data and the corresponding amount. If the quantity of goods in the order data is large, but the corresponding amount is small, then the order data is abnormal. Therefore, the density of data within a period of time can reflect the abnormality of the data within that period of time.

[0100] Therefore, in one or more embodiments of the present specification, after determining the first distance and the second distance, the server may determine a density factor according to the determined first distance and the second distance. Specifically, the first distance may be compared with the second distance to obtain the density factor. In subsequent steps, the abnormality of the data to be identified may be determined according to the density factor.

[0101] S112: Determine the abnormality of the data to be identified through a real-time calculation framework according to the time factor and the density factor, and determine whether the data to be identified is abnormal data according to the abnormality of the data to be identified.

[0102] In one or more embodiments of the present specification, after determining the time factor and the density factor, the server can determine the abnormality of the data to be identified through a real-time computing framework. The real-time computing framework is a Flink framework, which can determine the abnormality of the data to be identified in real time.

[0103] Specifically, the server can determine the abnormal value of the data to be identified based on the time factor and the density factor, and determine the preset abnormality range within which the abnormal value falls, and then determine the abnormality of the data to be identified based on the preset abnormality range within which the abnormal value falls, and determine whether the data to be identified is abnormal data based on the abnormality of the data to be identified. The preset abnormality range can be set as needed.

[0104] For example, assuming that the preset abnormality ranges are 0-25, 25-50, 50-75 and 75-100, when the abnormality value of the data to be identified falls within the abnormality range of 0-25, the data to be identified is determined to be normal, when the abnormality value of the data to be identified falls within the abnormality range of 25-50, the data to be identified is determined to be slightly abnormal, when the abnormality value of the data to be identified falls within the abnormality range of 50-75, the data to be identified is determined to be relatively abnormal, and when the abnormality value of the data to be identified falls within the abnormality range of 75-100, the data to be identified is determined to be extremely abnormal. It can be seen that the method for determining abnormal data provided in this specification can make a more detailed division of the abnormality degree of the data, so as to solve the problem that in the prior art, the data can only be divided into abnormal and normal categories according to the determined results, and the determination results of the abnormal data are not accurate enough, and improve the problem that the prior art cannot find the task corresponding to the abnormal data in a timely and accurate manner.

[0105] The formula for determining the outlier value of the data to be identified is:

[0106]

[0107] Among them, x represents the data to be identified, k is the number of adjacent data in the neighborhood of the data to be identified, and β m Indicates the time factor corresponding to the mth adjacent data of the data to be identified, RD m Indicates the density factor corresponding to the mth adjacent data of the data to be identified.

[0108] based on Figure 1 The method for determining abnormal data shown in the figure cleans the data to obtain the data to be identified, and determines the distance between the data to be identified and other data in the current time window. According to the preset number and the determined distance, the neighborhood of the data to be identified is determined, and the adjacent data is determined. Then, according to the time difference between the adjacent data and the data to be identified, the time factor is determined, the first distance is determined according to the distance between the data to be identified and the adjacent data, and the second distance is determined according to the distance between the adjacent data and each data in its own neighborhood. Then, the density factor is determined according to the first distance and the second distance, and the abnormality of the data to be identified is determined according to the time factor and the density factor to determine the abnormal data.

[0109] It can be seen from the above method that this method can determine the time factor and density factor according to the time difference of each data in the time window and other data in the neighborhood of the data to be identified, and calculate the abnormality of the data to be identified in real time according to the obtained time factor and density factor, so as to obtain a precise and accurate determination result and promptly discover problems with the tasks corresponding to the abnormal data.

[0110] Figure 4 A schematic diagram of a time window provided in this specification. As shown in the figure, the figure includes two time windows, the time window formed from M to O represents the previous time window, and the time window formed from N to P represents the time window after sliding, that is, the current time window. The data generated at t1, t2, t3, and t4 fall into the previous time window formed from M to O, and the data generated at t2, t3, t4, and t5 fall into the current time window formed from N to P. It can be seen that the data contained in the previous time window is not exactly the same as the data contained in the current time window, so the data in the time window may change before and after the time window slides.

[0111] In one or more embodiments provided in this specification, it is considered that the abnormality of the data to be identified is affected by the adjacent data in its neighborhood, and before and after the time window slides, the neighborhood of each data to be identified may change, and its abnormality may also change after the time window slides. Therefore, before determining the abnormality of the data to be identified in step S112, the server can also determine whether it is necessary to re-determine the abnormality of the data to be identified based on whether the judgment result of the abnormality of the data to be identified in the previous time window is normal. If the data to be identified is normal data in the previous time window, the abnormality of the data to be identified in the previous time window is used as the abnormality of the data to be identified in the current time window. If the data to be identified is not normal data in the previous time window, the server can re-determine the abnormality of the data to be identified in the current time window.

[0112] Of course, since the abnormality identification results of each data included in the previous time window have been determined, the server can also determine whether it is necessary to re-determine the abnormality of the data to be identified after determining each data to be identified in step S100. At the latest, the server can make a judgment before determining the abnormality of the data to be identified in step S112, and this manual does not limit the specific setting.

[0113] Furthermore, due to the sliding of the time window, the data to be identified contained in the current time window may be in the following three situations: included in the previous time window and determined to be not normal data, included in the previous time window and determined to be normal data, or not included in the previous time window. In this specification, in order to reduce the operating pressure of the server each time the abnormality is determined, the server can also determine the abnormality of the data to be identified only for the data to be identified that is not included in the previous time window when determining the abnormality of the data.

[0114] Specifically, in one or more embodiments provided in this specification, before determining the abnormality of the data to be identified in step S112, the server may also first determine whether the data to be identified is included in the previous time window. If so, there is no need to update the abnormality of the data to be identified. If not, the time factor and density factor corresponding to the data to be identified are determined, and the abnormality of the data to be identified is determined based on the determined time factor and density factor.

[0115] Furthermore, in one or more embodiments provided in this specification, data determined to be abnormal in a historical time window may be determined to be normal in a current time window. That is, due to the fluctuation of data, the abnormality of some data to be identified may change and fall into the abnormality range corresponding to normal data. Therefore, in order to reduce the service computing pressure and determine the abnormality of each data to be identified as more accurately as possible, before determining the abnormality of the data to be identified in step S112, the server can also determine whether the data to be identified is included in the previous time window, and then re-determine the abnormality of the data determined to be abnormal in the previous time window. For the data to be detected that is not in the previous time window, the time factor and density factor corresponding to the data to be identified are determined, and the abnormality of the data to be identified is determined based on the determined time factor and density factor.

[0116] That is to say, for the above three types of data to be identified: the data to be identified that is included in the previous time window and determined to be not normal data, and the data to be identified that is not included in the previous time window, the method for determining abnormal data provided in this specification is executed to determine the abnormality of the data.

[0117] The above is a method for determining abnormal data provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding device for determining abnormal data, such as Figure 5 shown.

[0118] Figure 5A schematic diagram of a device for determining abnormal data provided in this specification, the device includes: a to-be-identified data acquisition module, a data distance determination module, a neighborhood determination module, a time factor determination module, a neighborhood distance determination module, a density factor determination module, and an abnormal data determination module, wherein:

[0119] The module 200 for obtaining data to be identified is used to clean the acquired data to obtain the data to be identified;

[0120] The data distance determination module 201 is used to determine the distance between each piece of data to be identified and other data in the time window;

[0121] A neighborhood determination module 202 is used to determine the neighborhood of the data to be identified according to a preset number and the determined distance between the data to be identified and other data in the time window, and to determine each data in the neighborhood as adjacent data;

[0122] A time factor determination module 203 is used to determine a time difference between the adjacent data and the data to be identified in a time window, and determine a time factor according to the time difference, wherein the time difference is negatively correlated with the time factor;

[0123] A neighborhood distance determination module 204, configured to determine a first distance according to a distance between the data to be identified and the neighboring data, and to determine a second distance according to a distance between the neighboring data and each data in a neighborhood of the neighboring data;

[0124] A density factor determination module 205, configured to determine a density factor according to the first distance and the second distance;

[0125] The abnormal data determination module 206 is used to determine the abnormality of the data to be identified through a real-time calculation framework according to the time factor and the density factor, and determine whether the data to be identified is abnormal data according to the abnormality of the data to be identified.

[0126] Optionally, the data distance determination module 201 determines the information entropy of the data to be identified in each preset feature dimension, and determines the distance between the data to be identified and other data in the current time window based on the determined information entropy difference of the data to be identified in each feature dimension.

[0127] Optionally, for each feature dimension, the information entropy of the data to be identified in the feature dimension is determined according to the eigenvalue of the data to be identified in the feature dimension, and the residual dimension information entropy corresponding to the feature dimension is determined according to the information entropies corresponding to other feature dimensions excluding the feature dimension, and the information entropy difference of the data to be identified in the feature dimension is determined according to the difference between the total information entropy of the data to be identified and the residual dimension information entropy, wherein the total information entropy is determined according to the information entropies corresponding to the data to be identified in each feature dimension.

[0128] Optionally, the neighborhood distance determination module 204 determines the weights corresponding to the data to be identified in each feature dimension according to the information entropy differences of the data to be identified in each feature dimension, and for each adjacent data of the data to be identified, determines the weighted distance between the data to be identified and the adjacent data according to the weights corresponding to the data to be identified in each feature dimension and the eigenvalues ​​of the data to be identified and the adjacent data in each feature dimension, and determines the first distance according to the weighted distances between the data to be identified and each adjacent data.

[0129] Optionally, the abnormal data determination module 206 determines the abnormal value of the data to be identified based on the time factor and the density factor, determines a preset abnormality range within which the abnormal value falls, and determines the abnormality of the data to be identified based on the preset abnormality range within which the abnormal value falls.

[0130] Optionally, whether the data to be identified is normal is judged based on the abnormality degree of the data to be identified in the previous time window. If so, the abnormality degree of the data to be identified in the previous time window is used as the abnormality degree of the data to be identified in the current time window. If not, the abnormality degree of the data to be identified in the current time window is re-determined.

[0131] Optionally, it is determined whether the data to be identified is included in the previous time window. If so, there is no need to update the abnormality of the data to be identified. If not, the abnormality of the data to be identified is determined.

[0132] This specification also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provides methods for identifying anomalous data.

[0133] This manual also provides Figure 6 The structural diagram of the electronic device shown in FIG. Figure 6 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1Provides methods for identifying anomalous data.

[0134] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the executor of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0135] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0136] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0137] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0138] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0139] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0144] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0145] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0146] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0147] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0148] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0149] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0150] The above description is only an embodiment of the present specification and is not intended to limit the present specification. For those skilled in the art, the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present specification shall be included in the scope of the claims of the present specification.

Claims

1. A method for determining abnormal data, characterized in that: The method specifically comprises: Clean the acquired data to obtain the data to be identified; For each piece of data to be identified, determine the distance between the data to be identified and other data in the current time window; According to the preset number and the determined distance between the data to be identified and other data in the current time window, a neighborhood of the data to be identified is determined, and each data in the neighborhood is determined to be adjacent data; Determine a time difference between the adjacent data and the data to be identified in the current time window, and determine a time factor according to the time difference, wherein the time difference is negatively correlated with the time factor; Determine a first distance according to the distance between the data to be identified and the adjacent data, and determine a second distance according to the distance between the adjacent data and each data in the neighborhood of the adjacent data; determining a density factor according to the first distance and the second distance; According to the time factor and the density factor, the abnormality of the data to be identified is determined through a real-time calculation framework, and according to the abnormality of the data to be identified, it is determined whether the data to be identified is abnormal data.

2. The method according to claim 1, characterized in that Determine the distance between the data to be identified and other data in the current time window, specifically including: Determine the information entropy of the data to be identified in each preset feature dimension; According to the determined information entropy difference of the data to be identified in each feature dimension, the distance between the data to be identified and other data in the current time window is determined.

3. The method according to claim 2, characterized in that Determine the information entropy difference of the data to be identified in each feature dimension, specifically including: For each feature dimension, according to the feature value of the data to be identified in the feature dimension, determine the information entropy corresponding to the data to be identified in the feature dimension; Determine the residual dimension information entropy corresponding to the feature dimension according to the information entropies corresponding to the other feature dimensions excluding the feature dimension; Determine the information entropy difference of the data to be identified in the feature dimension according to the difference between the total information entropy of the data to be identified and the codimensional information entropy; The total information entropy is determined according to the information entropy corresponding to each feature dimension of the data to be identified.

4. The method according to claim 2, characterized in that Determining the first distance according to the distance between the data to be identified and the adjacent data specifically includes: According to the information entropy difference of the data to be identified in each feature dimension, the corresponding weight of the data to be identified in each feature dimension is determined respectively; For each adjacent data of the data to be identified, determine the weighted distance between the data to be identified and the adjacent data according to the weights corresponding to the data to be identified in each feature dimension and the feature values ​​of the data to be identified and the adjacent data in each feature dimension; A first distance is determined according to the weighted distance between the data to be identified and each adjacent data.

5. The method according to claim 1, characterized in that According to the time factor and the density factor, the abnormality of the data to be identified is determined through a real-time calculation framework, specifically including: Determining the outliers of the data to be identified according to the time factor and the density factor; Determine a preset abnormality degree range within which the abnormal value falls, and determine the abnormality degree of the data to be identified according to the preset abnormality degree range within which the abnormal value falls.

6. The method according to claim 1, characterized in that Before determining the abnormality of the data to be identified, the method further includes: Judging whether the data to be identified is normal according to the abnormality of the data to be identified in the previous time window; If yes, the abnormality degree of the data to be identified in the previous time window is used as the abnormality degree of the data to be identified in the current time window; If not, the abnormality of the data to be identified in the current time window is re-determined.

7. The method according to claim 1, characterized in that Before determining the abnormality of the data to be identified, the method further includes: Determine whether the data to be identified is included in the previous time window; If yes, there is no need to update the abnormality of the data to be identified; If not, the abnormality of the data to be identified is determined.

8. A device for determining abnormal data, characterized in that: The device specifically comprises: The module for acquiring data to be identified is used to clean the acquired data to obtain the data to be identified; A data distance determination module is used to determine the distance between each piece of data to be identified and other data in the time window; A neighborhood determination module, used to determine the neighborhood of the data to be identified according to a preset number and the determined distance between the data to be identified and other data in the time window, and determine each data in the neighborhood as adjacent data; A time factor determination module, used to determine the time difference between the adjacent data and the data to be identified in the time window, and determine the time factor according to the time difference, wherein the time difference is negatively correlated with the time factor; A neighborhood distance determination module, configured to determine a first distance according to a distance between the data to be identified and the adjacent data, and to determine a second distance according to a distance between the adjacent data and each data in a neighborhood of the adjacent data; A density factor determination module, configured to determine a density factor according to the first distance and the second distance; The abnormal data determination module is used to determine the abnormality of the data to be identified through a real-time calculation framework according to the time factor and the density factor, and determine whether the data to be identified is abnormal data according to the abnormality of the data to be identified.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images