Data processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a data processing method and device, electronic equipment and a storage medium, and the data processing method comprises the steps: responding to an obtained first data set, segmenting the first data set based on a preset time interval, and obtaining a plurality of second data sets; and determining whether data exception exists or not based on the data in the second data set. Therefore, for the data set acquired in a long period, the data set can be segmented into the data set with smaller time domain span through time domain segmentation for anomaly analysis, so that the data anomaly can be more accurately determined, and the situation that complex anomaly conditions are dispersed in the long period and are difficult to detect is avoided.

Description

Technical Field

[0001] The present application relates to the field of data monitoring technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] In processes such as system and equipment operation and maintenance, long-term data monitoring and anomaly detection are important means to ensure stability and performance. However, when the time domain span is large, the sensitivity of data anomaly detection is affected, making it difficult to effectively identify complex anomaly patterns over a long period of time. Summary of the invention

[0003] The present application provides a data processing method, device, electronic device and storage medium to solve the technical problem of poor sensitivity of data anomaly detection.

[0004] In a first aspect, the present application provides a data processing method, comprising: in response to acquiring a first data set, segmenting the first data set based on a predetermined time interval to obtain multiple second data sets; and determining whether there is a data anomaly based on the data in the second data set.

[0005] In a possible implementation, determining whether there is a data anomaly based on the data in the second data set includes: determining a deviation parameter of the second data set based on the mean and / or standard deviation of the data in the second data set; and determining whether there is a data anomaly based on the deviation parameter.

[0006] In a possible implementation, determining the deviation parameter of the second data set based on the mean and / or standard deviation of the data in the second data set includes: determining the mean and standard deviation of the data in the second data set based on the value and quantity of the data in the second data set; summing up the cube of the difference between the value of each data and the mean to obtain a first parameter; determining the second parameter based on the cube of the standard deviation; and determining the deviation parameter of the second data set based on the ratio of the first parameter to the second parameter.

[0007] In a possible implementation, determining the second parameter based on the cube of the standard deviation includes: determining a deviation adjustment factor based on the amount of data in the second data set; and multiplying the deviation adjustment factor by the cube of the standard deviation to obtain the second parameter.

[0008] In one possible implementation, determining whether there is a data anomaly based on the deviation parameter includes: determining a deviation interval based on at least one of the mean, standard deviation, type, time domain length corresponding to the first data set, and the predetermined time interval of the data; and determining whether there is a data anomaly based on a comparison result of the deviation parameter and the deviation interval.

[0009] In one possible implementation, after determining whether data anomalies exist based on the data in the second data set, the method further includes: if it is determined that data anomalies exist, determining the type of the data anomaly based on at least one of the mean, standard deviation, type, peak value, time domain length corresponding to the first data set, and the predetermined time interval of the data.

[0010] In one possible implementation, in response to acquiring the first data set, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets includes: in response to acquiring the first data set, determining whether a time domain length corresponding to the first data set reaches a predetermined threshold; if the time domain length reaches the predetermined threshold, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets.

[0011] In a second aspect, the present application provides a data processing device, comprising: a processing unit, for, in response to acquiring a first data set, dividing the first data set based on a predetermined time interval to obtain multiple second data sets; a determination unit, for determining whether there is a data anomaly based on the data in the second data set.

[0012] In a third aspect, the present application provides an electronic device comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store a computer program; and the processor is used to implement any of the methods described in the first aspect when executing the computer program.

[0013] In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described in the first aspect.

[0014] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, in response to obtaining a first data set, divides the first data set based on a predetermined time interval to obtain multiple second data sets; determines whether there is a data anomaly based on the data in the second data set. In this way, the data set is divided according to the predetermined time interval in the time domain. When the time domain span of the first data set is large, it can be divided into data sets with a small time domain span for anomaly detection and analysis, thereby avoiding complex abnormal situations from being affected by a large time domain span, and facilitating the capture of subtle changes and anomalies in data distribution within a finer-grained time domain range, thereby improving the sensitivity to data anomalies and the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of a data processing method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of an anomaly detection system provided in an embodiment of the present application;

[0022] Figure 5 A flowchart of an anomaly detection method provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a data processing device provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0026] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0027] In order to solve the technical problem of low sensitivity of data anomaly detection in the prior art, the present application provides a data processing method, device, electronic device and storage medium, which divide the data set based on a predetermined time interval, thereby refining the detection time granularity, making it easier to detect more subtle data anomalies and improving the sensitivity of anomaly detection.

[0028] Figure 1 A flow chart of a data processing method provided in this embodiment is as follows: Figure 1 As shown, the method may include:

[0029] S10: In response to acquiring the first data set, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets;

[0030] S20: Determine whether there is data anomaly based on the data in the second data set.

[0031] The data processing method provided in this embodiment can be applied to a device that requires data monitoring, or can also be applied to a server, cloud platform or other terminal that has the ability to control the device.

[0032] In one embodiment, the first data set may be a data set acquired according to a certain time domain length, wherein the data included in the first data set may be data acquired within the time domain length. For example, the data included in the first data set may be data collected at multiple time points within the time domain length.

[0033] For example, the time domain length corresponding to the first data set is 1 day, and data collected at multiple time points within 1 day constitute the first data set.

[0034] In one embodiment, the first data set may be obtained through a sensor, a logger or other data source, and the data in the first data set may be monitoring data, such as disk read and write speed, input / output operations per second (IOPS), or latency data.

[0035] In one embodiment, the predetermined time interval may be a fixed value, such as 1 hour, 12 hours or 1 day, etc. Alternatively, the predetermined time interval may also be determined according to the time domain length corresponding to the first data set, for example, the longer the time domain length, the larger the predetermined time interval.

[0036] In one embodiment, the multiple second data sets are obtained by segmenting the first data set based on the predetermined time interval, and the time domain length corresponding to each second data set is equal to or less than the predetermined time interval.

[0037] In one embodiment, segmenting the first data set may refer to segmenting the first data set by time domain. For example, segmenting the first data set may include: segmenting the first data set by time domain.

[0038] In one embodiment, segmenting the first data set based on a predetermined time interval may refer to segmenting the first data set based on a predetermined time interval and a time when data in the first data set is acquired.

[0039] In one embodiment, step S20 may include: determining whether there is data anomaly in the first data set based on data in the second data set.

[0040] In one embodiment, after step S20, the method may further include: if there is a data anomaly, outputting anomaly prompt information and / or anomaly handling suggestion. Outputting anomaly prompt information may refer to sending anomaly prompt information via email, SMS or other means. Exemplarily, the anomaly prompt information may be used to indicate possible transaction anomalies and / or disk problems.

[0041] In one embodiment, outputting an exception handling suggestion may refer to outputting a corresponding exception handling suggestion according to the exception type and the severity of the exception, wherein the exception handling suggestion may include at least one of disk checking, data backup, and system tuning.

[0042] In one embodiment, step S10 may include: in response to obtaining the first data set, preprocessing the first data set; dividing the preprocessed first data set based on a predetermined time interval to obtain multiple second data sets. Here, preprocessing may include: performing data clarification processing and / or missing value processing.

[0043] In this way, the data set is segmented in the time domain according to predetermined time intervals. When the time domain span of the first data set is large, it can be segmented into data sets with a small time domain span for anomaly detection analysis, thereby avoiding complex abnormal situations from being affected by a larger time domain span, and facilitating the capture of subtle changes and anomalies in data distribution within a finer-grained time domain range, thereby improving the sensitivity to data anomalies and detection accuracy.

[0044] Anomaly detection in related technologies often detects anomalies by simply performing a threshold judgment on the data directly, which results in poor detection accuracy. Therefore, in order to improve the accuracy and sensitivity of anomaly detection, in some embodiments, Figure 2 As shown, step S20 may include:

[0045] S21: determining a deviation parameter of the second data set based on a mean and / or a standard deviation of the data in the second data set;

[0046] S22: Determine whether there is data anomaly based on the deviation parameter.

[0047] In one embodiment, the deviation parameter may also be referred to as a skewness value, which may be used to describe the symmetry or asymmetry of data distribution, or to describe the degree of skewness of data.

[0048] Here, the skewness of the data may refer to the skewness of the data distribution relative to the mean of the data in the second data set. The deviation parameter may be positive, negative or zero, indicating different characteristics of the data distribution.

[0049] In one embodiment, determining whether there is a data anomaly may refer to determining whether there is a data anomaly in the first data set.

[0050] In one embodiment, step S22 may include: determining whether there is data anomaly based on a comparison result between the deviation parameter and the deviation interval, wherein the deviation interval may be determined based on historical data.

[0051] In one embodiment, when the deviation parameter is a positive value, it can indicate that the data is concentrated in the low value interval, that is, the data is more distributed in the interval with lower values. When the deviation parameter is a negative value, it can indicate that the data is concentrated in the high value interval, that is, the data is more distributed in the interval with higher values.

[0052] In one embodiment, a larger deviation parameter, or a larger absolute value of the deviation parameter, indicates that the data distribution is more asymmetric, that is, the possibility of data anomaly is greater.

[0053] In one embodiment, determining whether there is a data anomaly based on the deviation parameter may include: determining whether there is a data anomaly based on whether the deviation parameter is a positive value or a negative value, and the absolute value of the deviation parameter.

[0054] In one embodiment, determining whether there is a data anomaly based on whether the deviation parameter is a positive or negative value and the absolute value of the deviation parameter may include: determining whether there is a data anomaly based on the type of data in the second data set, whether the deviation parameter is a positive or negative value, and the absolute value of the deviation parameter.

[0055] For example, for financial data, if the deviation parameter is negative and the absolute value is greater than a predetermined value, it is determined that there is data anomaly.

[0056] In this way, the distribution of data in the data set can be more accurately described based on the mean and / or standard deviation, so that the deviation parameter determined can more accurately characterize the distribution deviation of the data in the second data set, and can more accurately discover hidden and complex data anomalies, which is conducive to improving the sensitivity and accuracy of anomaly detection.

[0057] In some embodiments, the above step S21 may include:

[0058] Determining a mean and a standard deviation of the data in the second data set based on the values ​​and quantities of the data in the second data set;

[0059] The cube of the difference between the value of each data and the mean is summed up to obtain a first parameter;

[0060] determining a second parameter based on the cube of the standard deviation;

[0061] Based on the ratio of the first parameter to the second parameter, a deviation parameter of the second data set is determined.

[0062] In one embodiment, summing up the cube of the difference between the value of each data and the mean may refer to calculating the difference between the value of each data and the mean, and calculating the cube of the difference; and summing up the cubes of the differences corresponding to all data.

[0063] In one embodiment, determining the second parameter based on the cube of the standard deviation may include: determining the second parameter based on the cube of the standard deviation and a bias adjustment factor, wherein the bias adjustment factor may be used to correct an estimated value of sample skewness.

[0064] In one embodiment, determining the deviation parameter of the second data set based on the ratio of the first parameter to the second parameter may mean that the deviation parameter of the second data set is equal to the ratio of the first parameter to the second parameter.

[0065] In this way, by determining the deviation parameter through the ratio of the cube of the difference between the data value and the mean to the cube of the standard deviation, we can expand the calculation influence of the fluctuation of the data value compared with the mean and the degree of dispersion represented by the standard deviation, and improve the accuracy of the deviation parameter in representing the data distribution.

[0066] In some embodiments, determining the second parameter based on the cube of the standard deviation may include:

[0067] determining a bias adjustment factor based on the amount of data in the second data set;

[0068] The deviation adjustment factor is multiplied by the cube of the standard deviation to obtain a second parameter.

[0069] In one embodiment, determining the deviation adjustment factor based on the number of data in the second data set may mean that the deviation adjustment factor is equal to the product of the number minus 1 and the number minus 2. That is, if the number of data in the second data set is n, the deviation adjustment factor is (n-1)(n-2).

[0070] In one embodiment, the second parameter corresponding to a second data set is the product of the cube of the standard deviation of the data in the second data set and (n-1)(n-2).

[0071] In one embodiment, a bias adjustment factor is used to correct an estimate of sample skewness.

[0072] In this way, by introducing the second parameter through the deviation adjustment factor related to the number of data, it can be more accurate in the case of finite samples, and the standard deviation can be normalized based on the cube of the standard deviation, further improving the accuracy of the deviation parameter.

[0073] In some embodiments, determining whether there is a data anomaly based on the deviation parameter may include:

[0074] Determine a deviation interval based on at least one of a mean value, a standard deviation, a type of the data, a time domain length corresponding to the first data set, and the predetermined time interval;

[0075] Based on the comparison result of the deviation parameter and the deviation interval, determine whether there is data anomaly.

[0076] In one embodiment, determining the deviation interval based on the mean, standard deviation, type of the data, the time domain length corresponding to the first data set, and at least one of the predetermined time interval may include: determining the deviation interval based on the mean, standard deviation, and confidence level coefficient of the data.

[0077] For example, the lower limit of the deviation interval can be the mean minus the standard deviation and the confidence level coefficient, and the upper limit of the interval can be the mean plus the standard deviation and the confidence level coefficient.

[0078] That is, when the mean is t, the standard deviation is s, and the confidence level coefficient is k, the deviation interval can be [ts*k, t+s*k].

[0079] In one embodiment, the above steps may include: determining the deviation interval based on the mean, standard deviation and confidence level coefficient of the data; and adjusting the deviation interval based on the type of data, the time domain length corresponding to the first data set and at least one of the predetermined time intervals.

[0080] Accordingly, determining whether there is a data anomaly based on a comparison result between the deviation parameter and the deviation interval may include: determining whether there is a data anomaly based on a comparison result between the deviation parameter and the adjusted deviation interval.

[0081] Here, different application scenarios and data types may correspond to different adjustment methods. For example, for financial data, a stricter range may be required, and adjusting the deviation interval may mean narrowing the range of the deviation interval; while for network traffic data, a looser range may be required, and adjusting the deviation interval may mean expanding the range of the deviation interval.

[0082] In one embodiment, adjusting the deviation interval based on the time domain length corresponding to the first data set may mean that the larger the time domain length is, the smaller the range of the deviation interval is. For example, when the time domain length reaches the first threshold, adjusting the deviation interval may mean reducing the range of the deviation interval. Thus, the deviation interval may be reduced when the time domain length is large, thereby improving the sensitivity of the deviation parameter and avoiding the difficulty in identifying complex anomalies due to the large time domain span.

[0083] For example, multiple first threshold ranges may be set, and adjusting the deviation interval based on the time domain length corresponding to the first data set may include: determining the amplitude of reducing the deviation interval range according to the first threshold range in which the time domain length is located, and reducing the range of the deviation interval based on the amplitude.

[0084] In one embodiment, adjusting the deviation interval based on the predetermined time interval may mean that the larger the predetermined time interval is, the smaller the range of the deviation interval is. For example, when the predetermined time interval reaches the second threshold, adjusting the deviation interval may mean reducing the range of the deviation interval. Therefore, when the predetermined time interval corresponding to the second data set is large, the deviation interval may be reduced, thereby improving the sensitivity of the deviation parameter and avoiding the difficulty in identifying complex anomalies due to the large time domain span.

[0085] For example, multiple second threshold ranges may be set, and adjusting the deviation interval based on a predetermined time interval may include: determining an amplitude for reducing the deviation interval according to the second threshold range in which the predetermined time interval is located, and reducing the deviation interval based on the amplitude.

[0086] In this way, the deviation interval corresponding to the second data set can be comprehensively corrected through the distribution level, data type and time domain span of the data in the second data set, so that the interval range compared with the deviation parameters of each second data set can be determined more flexibly and accurately, thereby improving the accuracy and sensitivity of anomaly identification.

[0087] In order to further improve the accuracy of abnormality identification and processing, in some embodiments, such as Figure 3 As shown, after step S20, the method may further include:

[0088] S30: If it is determined that there is data anomaly, determine the type of the data anomaly based on at least one of the mean, standard deviation, type, peak value, time domain length corresponding to the first data set, and a predetermined time interval of the data.

[0089] In one embodiment, if it is determined in step S30 that there is a data anomaly, it may refer to if it is determined that there is a data anomaly in the first data set.

[0090] In one embodiment, after step S30, the method may further include: outputting abnormal prompt information and / or abnormal handling suggestions. Outputting abnormal prompt information may refer to sending abnormal prompt information via email, SMS or other means. The abnormal prompt information may include the type of the data abnormality.

[0091] Exemplarily, the abnormal prompt information can be used to indicate possible transaction abnormalities and / or disk problems, etc.

[0092] In one embodiment, outputting an exception handling suggestion may refer to outputting a corresponding exception handling suggestion according to the exception type and the severity of the exception, wherein the exception handling suggestion may include at least one of disk checking, data backup, and system tuning.

[0093] In one embodiment, the peak value may refer to a peak value of data in the second data set, or may also refer to a peak value of data in the first data set.

[0094] In one embodiment, the type of data anomaly may include the location of the data anomaly, the anomaly object, the severity of the anomaly, etc. Here, the location of occurrence may refer to the time point of occurrence, such as the time period corresponding to the second data set where the data anomaly is located.

[0095] In this way, the type of detected data anomaly can be accurately located by combining the data distribution level, data type and time domain span, thereby providing an accurate basis for anomaly identification processing.

[0096] In some embodiments, step S10 may include:

[0097] In response to acquiring a first data set, determining whether a time domain length corresponding to the first data set reaches a predetermined threshold;

[0098] If the time domain length reaches a predetermined threshold, the first data set is segmented based on a predetermined time interval to obtain a plurality of second data sets.

[0099] Here, the predetermined threshold may be 1 day, 1 week, or 10 days, etc. When the time domain length reaches the predetermined threshold, it indicates that the first data set belongs to a long-period data set.

[0100] In one embodiment, dividing the first data set based on a predetermined time interval to obtain multiple second data sets may include: determining the predetermined time interval based on the time domain length; dividing the first data set based on the predetermined time interval to obtain multiple second data sets.

[0101] Here, determining the predetermined time interval based on the time domain length may include determining the predetermined time interval based on a difference between the time domain length and a predetermined threshold.

[0102] For example, the larger the time domain length, the smaller the predetermined time interval, thereby reducing the influence of long periods on the sensitivity of abnormal identification. Alternatively, the larger the difference between the time domain length and the predetermined threshold, the smaller the predetermined time interval.

[0103] In this way, before the anomaly detection, it is determined whether the first data set belongs to a data set with a large time domain span in combination with a predetermined threshold, thereby avoiding segmentation detection of all data sets and reducing unnecessary resource occupation.

[0104] As a possible implementation method, a long-period anomaly detection method and system based on time domain segmentation and skewness calculation is provided. Figure 4 As shown, the system divides the long-period data, i.e., the first data set, into time domain, and calculates the skewness value of each segment, i.e., the second data set. When the skewness value is in a preset deviation range, the system automatically detects and identifies potential transaction anomalies or disk problems.

[0105] like Figure 5 As shown, the method may specifically include the following steps:

[0106] 1. Data collection module: This module continuously collects monitoring data over a long period of time through sensors, loggers or other data sources, such as disk read and write speed, IOPS, latency, etc.

[0107] 2. Time domain segmentation module: The collected long-period data is segmented according to preset time intervals (such as hours, days, weeks, etc.) to form multiple time domain data segments, namely the second data set.

[0108] 3. Skewness calculation module: Perform statistical analysis on each time domain data segment and calculate its skewness value. The skewness value is used to describe the symmetry of data distribution and reflect the degree of skewness of the data. The skewness calculation algorithm process is as follows:

[0109] Collect all data points of the time domain data segment, recorded as (x_1, x_2, \ldots, x_n), which means: within a specific time period (time domain data segment), the system will collect multiple data points. These data points can be the observation values ​​of a certain monitoring indicator at different time points. These observation values ​​are recorded as (x_1, x_2, \ldots, x_n) in sequence, where (x_i) represents the (i)th observation value, and there are a total of (n) observation values. Specifically: Time domain data segment: refers to data within a specific time range. For example, it can be an hour, a day, a week, etc. Data point: The specific value collected by the system within this time range. For example, the value of a certain indicator (such as CPU utilization, network traffic, disk IO, etc.) at different time points. Recorded as (x_1, x_2, \ldots, x_n): These data points are numbered and recorded in chronological order. (x_1) is the first data point in the time period, (x_2) is the second data point, and so on, until (x_n), which is the last data point in the time period.

[0110] The formula for calculating the mean of a data point is: [\bar{x}=\frac{1}{n}\sum_{i=1}^nx_i] This formula means that in a set of data, the mean (or average) is the sum of all data points divided by the number of data points. The formula is calculated by the following steps: Sum of data points: Add the values ​​of all data points (x_i) to get the sum (\sum_{i=1}^nx_i). Number of data points: Determine the number of data points (n). Calculate the average: Divide the sum of data points by the number of data points to get the mean (\bar{x}). In the formula, (\bar{x}): represents the mean. (n): represents the number of data points. (x_i): represents the value of the (i)th data point. (\sum_{i=1}^nx_i): represents the sum of all values ​​from the 1st data point to the (n)th data point.

[0111] The formula for calculating the standard deviation (StandardDeviation) is: [\sigma=\sqrt{\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2}]. The meaning of this formula is: Standard deviation is a statistic that measures the degree of dispersion of a set of data points relative to their mean (Mean). It reflects the degree of dispersion of data points around the mean. The larger the standard deviation, the more dispersed the distribution of data points; the smaller the standard deviation, the more concentrated the distribution of data points. In the formula, (\sigma): represents the standard deviation (StandardDeviation). (n): represents the number of data points. (x_i): represents the value of the (i)th data point. (\bar{x}): represents the mean (Mean) of the data points, and its calculation formula is (\bar{x}=\frac{1}{n}\sum_{i=1}^nx_i). ((x_i-\bar{x})^2): represents the square of the difference between the (i)th data point and the mean. (\sum_{i=1}^n(x_i-\bar{x})^2): represents the sum of the squares of the differences between all data points and the mean. (\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2): represents the sum of the squares of the differences between all data points and the mean divided by the number of data points, i.e., the variance. (\sqrt{\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2}): represents the square root of the variance, i.e., the standard deviation.

[0112] The formula for calculating the skewness value is: [\text{Skewness}=\frac{n\sum_{i=1}^n(x_i-\bar{x})^3}{(n-1)(n-2)\sigma^3}] Skewness is a statistic that describes the asymmetry of data distribution. It reflects the degree of skewness of the data distribution relative to the mean. The skewness value can be positive, negative, or close to zero, indicating different characteristics of the data distribution. In the formula, (\text{Skewness}): represents the skewness value. (n): represents the number of data points. (x_i): represents the value of the (i)th data point. (\bar{x}): represents the mean of the data points, and its calculation formula is (\bar{x}=\frac{1}{n}\sum_{i=1}^nx_i). (\sigma): represents the standard deviation of the data point (StandardDeviation), and its calculation formula is (\sigma=\sqrt{\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})^2}). ((x_i-\bar{x})^3): represents the cube of the difference between the (i)th data point and the mean. (\sum_{i=1}^n(x_i-\bar{x})^3): represents the sum of the cubes of the differences between all data points and the mean. (n\sum_{i=1}^n(x_i-\bar{x})^3): represents the result of the previous step multiplied by the number of data points (n). ((n-1)(n-2)): is an adjustment factor used to correct the estimated value of the sample skewness to make it more accurate in the case of finite samples. (\sigma^3): represents the cube of the standard deviation, which is used to normalize the skewness value. When the skewness value is positive skewness (positive skewness or right skewness): A positive skewness value indicates that the right tail of the data distribution is longer and the data is more concentrated on the left side of the distribution.

[0115] 4. Preset deviation interval: The normal range of skewness values ​​can usually be determined through historical data analysis and statistical methods. A common practice is to set a confidence interval of the skewness value, i.e., the deviation interval, based on the normal distribution assumption, such as [-1, 1]. The specific deviation interval can be adjusted according to different application scenarios and data characteristics. For example, for financial data, a stricter range may be required, while for network traffic data, a looser range may be required. The preset deviation interval can be calculated by the following formula: $$\text{ConfidenceInterval}=[\text{mean}-k\cdot\sigma,\text{mean}+k\cdot\sigma]$$where k is the coefficient of the confidence level, usually 1.96 (95% confidence level) or 2.58 (99% confidence level).

[0116] 5. Anomaly detection module: Compare the calculated skewness value with the preset deviation interval. When the skewness value exceeds the normal range, it is judged as a potential anomaly. The basis for judging the type of anomaly includes: The direction of the skewness value: positive skewness (skewness value greater than 0) may indicate that the data is concentrated in the low value interval, while negative skewness (skewness value less than 0) may indicate that the data is concentrated in the high value interval. The amplitude of the skewness value: The larger the skewness value (absolute value), the more asymmetric the data distribution is and the greater the possibility of anomaly. Combined with other statistical indicators: such as mean, standard deviation, kurtosis, etc., the type of anomaly is comprehensively judged.

[0117] 6. Early warning module: After detecting an abnormality, the system sends an early warning message to the administrator via email, SMS or other means to prompt possible transaction abnormalities or disk problems.

[0118] 7. Processing suggestion module: Provide corresponding processing suggestions based on the type and severity of the exception, such as disk inspection, data backup, system tuning, etc.

[0119] like Figure 6 As shown, an embodiment of the present application provides a data processing device, which may include:

[0120] The processing unit 100 is configured to, in response to acquiring a first data set, segment the first data set based on a predetermined time interval to obtain a plurality of second data sets;

[0121] The determination unit 200 is configured to determine whether there is a data anomaly based on the data in the second data set.

[0122] In a possible implementation, the determination unit 200 is specifically configured to: determine a deviation parameter of the second data set based on a mean and / or standard deviation of data in the second data set; and determine whether data anomalies exist based on the deviation parameter.

[0123] In a possible implementation, the determination unit 200 is specifically used to: determine the mean and standard deviation of the data in the second data set based on the value and quantity of the data in the second data set; sum the cube of the difference between the value of each data and the mean to obtain a first parameter; determine the second parameter based on the cube of the standard deviation; determine the deviation parameter of the second data set based on the ratio of the first parameter to the second parameter.

[0124] In a possible implementation manner, the determination unit 200 is specifically configured to: determine a deviation adjustment factor based on the amount of data in the second data set; and multiply the deviation adjustment factor by the cube of the standard deviation to obtain a second parameter.

[0125] In one possible implementation, the determination unit 200 is specifically used to: determine the deviation interval based on the mean, standard deviation, type, time domain length corresponding to the first data set, and at least one of the predetermined time intervals of the data; and determine whether there is a data anomaly based on a comparison result between the deviation parameter and the deviation interval.

[0126] In a possible implementation, the determination unit 200 is further used to: if it is determined that data anomaly exists, determine the type of the data anomaly based on the mean, standard deviation, type, peak value, time domain length corresponding to the first data set, and at least one of the predetermined time interval.

[0127] In one possible implementation, the processing unit 100 is specifically used to: in response to acquiring a first data set, determine whether a time domain length corresponding to the first data set reaches a predetermined threshold; if the time domain length reaches the predetermined threshold, segment the first data set based on a predetermined time interval to obtain multiple second data sets.

[0128] like Figure 7 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0129] Memory 113, used for storing computer programs;

[0130] In one embodiment of the present application, the processor 111, when executing the program stored in the memory 113, implements the data processing method provided by any one or more of the aforementioned method embodiments, which at least includes: in response to obtaining a first data set, dividing the first data set based on a predetermined time interval to obtain multiple second data sets; and determining whether there is a data anomaly based on the data in the second data set.

[0131] In a possible implementation, determining whether there is a data anomaly based on the data in the second data set includes: determining a deviation parameter of the second data set based on the mean and / or standard deviation of the data in the second data set; and determining whether there is a data anomaly based on the deviation parameter.

[0132] In a possible implementation, determining the deviation parameter of the second data set based on the mean and / or standard deviation of the data in the second data set includes: determining the mean and standard deviation of the data in the second data set based on the value and quantity of the data in the second data set; summing up the cube of the difference between the value of each data and the mean to obtain a first parameter; determining the second parameter based on the cube of the standard deviation; and determining the deviation parameter of the second data set based on the ratio of the first parameter to the second parameter.

[0133] In a possible implementation, determining the second parameter based on the cube of the standard deviation includes: determining a deviation adjustment factor based on the amount of data in the second data set; and multiplying the deviation adjustment factor by the cube of the standard deviation to obtain the second parameter.

[0134] In one possible implementation, determining whether there is a data anomaly based on the deviation parameter includes: determining a deviation interval based on at least one of the mean, standard deviation, type, time domain length corresponding to the first data set, and the predetermined time interval of the data; and determining whether there is a data anomaly based on a comparison result of the deviation parameter and the deviation interval.

[0135] In one possible implementation, after determining whether data anomalies exist based on the data in the second data set, the method further includes: if it is determined that data anomalies exist, determining the type of the data anomaly based on at least one of the mean, standard deviation, type, peak value, time domain length corresponding to the first data set, and the predetermined time interval of the data.

[0136] In one possible implementation, in response to acquiring the first data set, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets includes: in response to acquiring the first data set, determining whether a time domain length corresponding to the first data set reaches a predetermined threshold; if the time domain length reaches the predetermined threshold, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets.

[0137] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method provided by any one or more of the aforementioned method embodiments are implemented.

[0138] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0140] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0141] In the absence of contradiction, each step in a certain implementation mode or example can be implemented as an independent example, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain implementation mode or example can also be implemented as an independent example, and the order of the steps in a certain implementation mode or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain implementation mode or example can be arbitrarily combined; in addition, the various implementation modes or examples can be arbitrarily combined. For example, some or all steps of different implementation modes or examples can be arbitrarily combined, and a certain implementation mode or example can be arbitrarily combined with the optional methods or optional examples of other implementation modes or examples.

[0142] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A data processing method, characterized in that: The method comprises: In response to acquiring the first data set, segmenting the first data set based on a predetermined time interval to obtain a plurality of second data sets; Determine whether there is a data anomaly based on the data in the second data set.

2. The method according to claim 1, characterized in that The determining whether there is a data anomaly based on the data in the second data set includes: Determining a deviation parameter of the second data set based on a mean and / or a standard deviation of the data in the second data set; A determination is made as to whether a data anomaly exists based on the deviation parameter.

3. The method according to claim 2, characterized in that The determining the deviation parameter of the second data set based on the mean and / or standard deviation of the data in the second data set includes: Determining a mean and a standard deviation of the data in the second data set based on the values ​​and quantities of the data in the second data set; The cube of the difference between the value of each data and the mean is summed up to obtain a first parameter; determining a second parameter based on the cube of the standard deviation; Based on the ratio of the first parameter to the second parameter, a deviation parameter of the second data set is determined.

4. The method according to claim 3, characterized in that The determining of the second parameter based on the cube of the standard deviation comprises: determining a bias adjustment factor based on the amount of data in the second data set; The deviation adjustment factor is multiplied by the cube of the standard deviation to obtain a second parameter.

5. The method according to claim 2, characterized in that: The determining whether there is data anomaly based on the deviation parameter includes: Determine a deviation interval based on at least one of a mean value, a standard deviation, a type of the data, a time domain length corresponding to the first data set, and the predetermined time interval; Based on the comparison result of the deviation parameter and the deviation interval, determine whether there is data anomaly.

6. The method according to claim 1, characterized in that After determining whether there is data anomaly based on the data in the second data set, the method further includes: If it is determined that there is a data anomaly, the type of the data anomaly is determined based on at least one of the mean, standard deviation, type, peak value, time domain length corresponding to the first data set, and the predetermined time interval of the data.

7. The method according to claim 1, characterized in that In response to acquiring the first data set, the first data set is segmented based on a predetermined time interval to obtain a plurality of second data sets, including: In response to acquiring a first data set, determining whether a time domain length corresponding to the first data set reaches a predetermined threshold; If the time domain length reaches a predetermined threshold, the first data set is segmented based on a predetermined time interval to obtain a plurality of second data sets.

8. A data processing device, characterized in that: The device comprises: A processing unit, configured to, in response to acquiring the first data set, segment the first data set based on a predetermined time interval to obtain a plurality of second data sets; A determination unit is used to determine whether there is a data anomaly based on the data in the second data set.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store a computer program; and the processor is used to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images