Abnormal data analysis method, abnormal data analysis system and storage medium

Abstract

The present invention provides an abnormal data analysis method, an abnormal data analysis system and a computer-readable storage medium, comprising: obtaining a sample data set; mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix comprises a plurality of first data rows, the first data rows correspond one-to-one to the sample data in the sample data set, the first data rows comprise distance values ​​between the sample data corresponding to the first data row in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to numerical values; obtaining the clustering radius and density threshold of the DBSCAN algorithm according to the distance matrix; and determining the abnormal data in the sample data set using the clustering radius and density threshold. The present invention can automatically determine the clustering radius and density threshold without manually checking the network data or manually setting the clustering radius and density threshold, and can automatically detect abnormal data, and can perform efficient and accurate detection and analysis on massive data.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of information processing technology, and in particular, to an abnormal data analysis method, an abnormal data analysis system, and a computer-readable storage medium. Background Art

[0002] With the development of the information age, network users and devices are growing rapidly, the network is developing rapidly and diversified, the scope of application is expanding, and the data transmitted by the network is gradually increasing. In order to ensure the security and stability of the network environment, the supervision of network data in the big data era has become very important, and it is necessary to efficiently and comprehensively identify abnormal behaviors of the network. Due to the explosive growth of data traffic, the network has the characteristics of large data volume, strong randomness, and complex relationships between indicators. Traditional network anomaly detection methods are mostly manual. The monitoring method of network operation data is mainly for operation and maintenance personnel to manually check network data or monitor indicators by setting fixed thresholds. It cannot meet the requirements of efficient and accurate detection of massive data, and it is easy to omit and misjudge in the investigation of large amounts of data. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present invention provide an abnormal data analysis method, an abnormal data analysis system and a computer-readable storage medium, which can not only eliminate manual intervention and dependence on annotation parameters, but also meet the requirements of efficient and accurate detection of massive data.

[0005] In a first aspect, an embodiment of the present invention provides a method for analyzing abnormal data, comprising:

[0006] Get a sample dataset;

[0007] Mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix includes a plurality of first data rows, the first data rows correspond to the sample data in the sample data set one by one, the first data rows include distance values ​​between the sample data corresponding to the first data rows in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to numerical values; N is a positive integer;

[0008] Obtaining the clustering radius and density threshold of the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm according to the distance matrix;

[0009] Abnormal data in the sample data set is determined using the cluster radius and the density threshold.

[0010] In a second aspect, an embodiment of the present invention further provides a method for analyzing abnormal data, including:

[0011] Acquire an abnormal sample set, wherein the abnormal samples in the abnormal sample set correspond to multiple network indicators;

[0012] Obtaining multiple network indicator combinations according to the multiple network indicators;

[0013] Processing the plurality of network indicator combinations to obtain a root cause probability score of each of the network indicators;

[0014] The network indicator that has the greatest contribution to generating the abnormal sample set is determined according to the root cause likelihood score.

[0015] In a third aspect, an embodiment of the present invention further provides an abnormal data analysis system, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the abnormal data analysis method as described in the first aspect above is implemented, or, when the processor executes the computer program, the abnormal data analysis method as described in the second aspect above is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the abnormal data analysis method as described in the first aspect above, or the computer-executable instructions are used to execute the abnormal data analysis method as described in the second aspect above.

[0017] The embodiment of the present invention includes: obtaining a sample data set; then mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix includes a plurality of first data rows, the first data rows correspond one-to-one to the sample data in the sample data set, the first data rows include the distance values ​​between the sample data corresponding to the first data rows in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to the numerical value; N is a positive integer; then the clustering radius and density threshold of the DBSCAN algorithm are obtained according to the distance matrix; finally, the abnormal data in the sample data set is determined using the clustering radius and the density threshold. For the technical solution of the embodiment of the present invention, the sample data in the sample data set can be mapped to an N-dimensional space to obtain a distance matrix, and the clustering radius and density threshold in the DBSCAN algorithm can be automatically determined according to the distance matrix, and finally, the sample data set is detected according to the clustering radius and the density threshold to obtain abnormal data. Therefore, the embodiments of the present invention can automatically determine the two parameters of clustering radius and density threshold, without the need to manually check network data or manually set fixed thresholds for the two parameters of clustering radius and density threshold, and abandon manual intervention and dependence on annotation parameters. It can also automatically detect abnormal data in sample data sets, realize autonomous detection of the system, and improve the flexibility of the abnormal data analysis system, thereby enabling efficient and accurate detection and analysis of massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0019] Figure 1 is a schematic diagram of a system architecture platform for executing an abnormal data analysis method provided by an embodiment of the present invention;

[0020] Figure 2 is a flow chart of obtaining abnormal data through an abnormal data analysis method provided by an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a DBSCAN algorithm provided by an embodiment of the present invention;

[0022] Figure 4 A flowchart of sorting an initial matrix to obtain a distance matrix is ​​provided by an embodiment of the present invention;

[0023] Figure 5 is a flow chart of calculating cluster radius and density threshold provided by an embodiment of the present invention;

[0024] Figure 6 is a detailed schematic diagram of an abnormal data analysis method provided by an embodiment of the present invention;

[0025] Figure 7 is a flow chart of performing root cause analysis on an abnormal sample set by using an abnormal data analysis method provided by an embodiment of the present invention;

[0026] Figure 8 It is a flow chart of a method for calculating the root cause probability score of each network indicator provided by an embodiment of the present invention;

[0027] Fig. 9 It is an overall schematic diagram of an abnormal data analysis method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The present invention provides an abnormal data analysis method, an abnormal data analysis system and a computer-readable storage medium, wherein the abnormal data analysis method comprises: obtaining a sample data set; then mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix comprises a plurality of first data rows, the first data rows correspond one-to-one to the sample data in the sample data set, the first data rows comprise distance values ​​between the sample data corresponding to the first data row in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to numerical values; N is a positive integer; then the clustering radius and density threshold of the DBSCAN algorithm are obtained according to the distance matrix; finally, the abnormal data in the sample data set are determined using the clustering radius and the density threshold. Therefore, the embodiment of the present invention can automatically determine the two parameters of the clustering radius and the density threshold, without manually checking the network data or manually setting the fixed thresholds of the two parameters of the clustering radius and the density threshold, abandoning manual intervention and dependence on annotation parameters, and can automatically detect the abnormal data in the sample data set, realize the autonomous detection of the system, improve the flexibility of the abnormal data analysis system, and thus can perform efficient and accurate detection and analysis on massive data.

[0031] Secondly, the abnormal data analysis method also includes: obtaining an abnormal sample set, wherein the abnormal samples in the abnormal sample set correspond to multiple network indicators; then obtaining multiple network indicator combinations based on the multiple network indicators; then processing the multiple network indicator combinations to obtain the root cause possibility scores of each network indicator; and determining the network indicator with the greatest contribution to the generation of the abnormal sample set based on the root cause possibility scores. Therefore, the embodiment of the present invention can also calculate the root cause possibility scores of each network indicator in the abnormal sample set, and through root cause analysis, dig up the abnormal samples to find the contribution of each network indicator to the abnormal results, so that the source network indicator of the abnormal sample can be quickly tracked, thereby improving the accuracy of the abnormal data analysis system.

[0032] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0033] like Figure 1 As shown, Figure 1 It is a schematic diagram of a system architecture platform for an abnormal data analysis method provided by an embodiment of the present invention.

[0034] exist Figure 1 In the example of , the system architecture platform includes an abnormal data analysis system 100, wherein the abnormal data analysis system 100 is provided with a processor 110 and a memory 120, wherein the processor 110 and the memory 120 can be connected via a bus or other means, Figure 1 The example of connecting through bus is taken in the following.

[0035] The memory 120, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely disposed relative to the processor 110, and these remote memories may be connected to the system architecture platform via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] Those skilled in the art will appreciate that the system architecture platform can be applied to 3G communication network systems, LTE communication network systems, 5G communication network systems, and subsequently evolved mobile communication network systems, and this embodiment does not specifically limit this.

[0037] It can be understood by those skilled in the art that Figure 1The system architecture platform shown in the figure does not constitute a limitation on the embodiments of the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0038] exist Figure 1 In the system architecture platform shown, the processor 110 can call the abnormal data analysis program stored in the memory 120 to execute the abnormal data analysis method.

[0039] Based on the above system architecture platform, various embodiments of the abnormal data analysis method of the present invention are proposed below.

[0040] like Figure 2 As shown, Figure 2 It is a flowchart of an abnormal data analysis method provided by an embodiment of the present invention. The method includes but is not limited to step S100, step S200, step S300 and step S400.

[0041] Step S100, obtaining a sample data set;

[0042] Step S200, mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix includes a plurality of first data rows, the first data rows correspond to the sample data in the sample data set one by one, the first data rows include distance values ​​between the sample data corresponding to the first data row in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to numerical values; N is a positive integer;

[0043] Step S300, obtaining the clustering radius and density threshold of the DBSCAN algorithm according to the distance matrix;

[0044] Step S400: determine abnormal data in the sample data set using the cluster radius and density threshold.

[0045] In one embodiment, the embodiment of the present invention can map the sample data in the sample data set to the N-dimensional space, and obtain a distance matrix according to the mapping of the sample data in the N-dimensional space, the obtained distance matrix includes a plurality of first data rows, different first data rows correspond to the distance values ​​between different sample data and the remaining sample data in the N-dimensional space, and the distance values ​​in each first data row are sorted according to the numerical value, and then the embodiment of the present invention calculates the clustering radius and density threshold in the DBSCAN algorithm according to the distance matrix, and detects the sample data set according to the calculated clustering radius and density threshold, and screens out part of the sample data as abnormal data. Therefore, the embodiment of the present invention can automatically determine the two parameters of the clustering radius and density threshold in the DBSCAN algorithm, without manually checking the sample data or manually setting the fixed thresholds of the two parameters of the clustering radius and density threshold, abandoning manual intervention and dependence on annotation parameters, and can automatically detect abnormal data in the sample data set, realize autonomous detection of the abnormal data analysis system, improve the flexibility of the system, so as to be able to perform efficient and accurate detection and analysis of massive data.

[0046] It should be noted that, in order to ensure the security and stability of the network environment, the sample data in the above sample data set and the sample data in the embodiment of the present invention correspond to network sample data.

[0047] In addition, it can be understood that the sample data in the above sample data set can correspond to a network indicator of a single dimension or a network indicator of multiple dimensions. Exemplarily, when the sample data corresponds to a network indicator of a single dimension, the above N-dimensional space corresponds to a one-dimensional coordinate system; when the sample data corresponds to a network indicator of two dimensions, the above N-dimensional space corresponds to a two-dimensional coordinate system; and so on, when the sample data corresponds to a network indicator of N dimensions, the above N-dimensional space corresponds to an N-dimensional coordinate system. Therefore, the embodiment of the present invention can map the sample data into the N-dimensional space, obtain the corresponding mapping points, and obtain the distance matrix after sorting according to the distance value between each two mapping points.

[0048] It is worth noting that for massive network data, in order to accurately obtain abnormal data, the embodiment of the present invention can perform cluster analysis on the sample data set and extract abnormal data by excluding data within the category. Among them, cluster analysis, also known as group analysis, is a statistical analysis method used to study sample or indicator classification problems. Cluster analysis originated from taxonomy, but clustering is not equal to classification. The difference between clustering and classification is that the classes required for clustering are unknown. The algorithms for unsupervised learning clustering are very diverse. The DBSCAN algorithm is a density-based clustering algorithm that can find clusters of any shape, can effectively find noise points and outliers, and is very suitable for processing irregular data samples.

[0049] Specifically, in an embodiment of the present invention, regarding the above-mentioned DBSCAN algorithm, it is mainly embodied as follows: taking each mapping point as the center of the circle, calculating the number of mapping points contained in the circle with the above-mentioned clustering radius Eps (E field) as the radius as the point density value; then judging whether the current center point of the circle is a high-density point or a low-density point according to the above-mentioned MinPts (Minim Points, density threshold), when the center point with the number of points in the circle is less than MinPts is a low-density point, and the center point with the number of points greater than or equal to MinPts is a high-density point. If there is a high-density point in the circle of another high-density point, connect the two points, thereby continuously connecting the center points in series. If there is a low-density point in the circle of a high-density point, connect the low-density point to the nearest high-density point as a boundary point. All connected points form a cluster, and low-density points that are not in any cluster are marked as outliers. For example, if Figure 3 As shown in the figure, data points A, B, C, D, E, and F are all high-density points, P and Q are boundary points, and M is an outlier point. When the value of MinPts is 3, the number of data points contained in the circle with the high-density point as the center and Eps as the radius is greater than or equal to 3.

[0050] Therefore, the abnormal points obtained by the embodiment of the present invention include: data points that are low-density points and do not fall within the clustering radius of high-density points; and data points that are high-density points and do not fall within the clustering radius of other high-density points. After the abnormal points are obtained, the sample data corresponding to the abnormal points are the abnormal data. Therefore, the embodiment of the present invention can determine the abnormal data in the sample data set by using the clustering radius and the density threshold.

[0051] like Figure 4 As shown, Figure 4 is a flow chart of an abnormal data analysis method provided by another embodiment of the present invention. In one embodiment, the above step S200 of mapping the sample data in the sample data set to the N-dimensional space to obtain a distance matrix includes but is not limited to step S510 and step S520.

[0052] Step S510, mapping the sample data in the sample data set to an N-dimensional space to obtain an initial matrix; wherein the initial matrix includes a plurality of second data rows, the second data rows correspond one-to-one to the sample data in the sample data set, and the second data rows include distance values ​​between the sample data corresponding to the second data rows in the N-dimensional space and the remaining sample data in the sample data set;

[0053] Step S520: sort the distance values ​​in each second data row according to their numerical values ​​to obtain a distance matrix.

[0054] In one embodiment, the above-mentioned distance matrix can be obtained in the following manner: map the sample data in the sample data set into an N-dimensional space, and obtain an initial matrix based on the mapping of the sample data in the N-dimensional space, wherein the obtained initial matrix includes multiple second data rows, and different second data rows correspond to distance values ​​between different sample data and the remaining sample data in the N-dimensional space. Since the distance values ​​in the second data rows are unordered, the embodiment of the present invention also needs to sort the distance values ​​in each second data row according to the numerical value, thereby obtaining multiple sorted first data rows, and then obtaining the above-mentioned distance matrix.

[0055] It should be noted that the distance values ​​may be sorted in order of numerical values: from small to large, or from large to small.

[0056] like Figure 5 As shown, Figure 5 1 is a flow chart of an abnormal data analysis method provided by an embodiment of the present invention. In one embodiment, the above step S300 includes but is not limited to step S710, step S720, step S730 and step S740.

[0057] Step S710, traversing the first data rows in the distance matrix, in each first data row, taking the distance value with the smallest value as the starting data point for calculation, sequentially calculating the slope of the previous data point relative to the next data point, and obtaining a slope list corresponding to each first data row;

[0058] Step S720, calculating the slope average and slope standard deviation according to all non-zero slopes in all slope lists;

[0059] Step S730, traverse all slope lists, and obtain the first slope whose value is greater than the sum of the slope average and the slope standard deviation as the target slope;

[0060] Step S740, determining the clustering radius and density threshold of the DBSCAN algorithm according to the target slope.

[0061] In one embodiment, the above-mentioned clustering radius and density threshold can be obtained in the following manner: for a first data row, the embodiment of the present invention can draw a curve according to the distance value in the first data row, and calculate the slope of the data point corresponding to the previous distance value in the first data row relative to the data point corresponding to the next distance value in turn, thereby obtaining a slope list for the first data row; then a slope list is obtained for each first data row, and non-zero slopes with slope values ​​that are not zero are screened out from all slope lists, and the slope average and slope standard deviation of all non-zero slopes are calculated, and then the first slope in the slope list whose value is greater than the sum of the slope average and slope standard deviation is found as the target slope, and the target slope corresponds to the inflection point of the above-mentioned curve. At this time, the clustering radius is the largest and the clustering effect is the best, so the distance value corresponding to this point is the clustering radius, and the number of data points corresponding to this clustering radius is the density threshold.

[0062] like Figure 6 As shown, Figure 6 FIG. 1 is a detailed schematic diagram of an abnormal data analysis method provided by another embodiment of the present invention. The specific steps of the abnormal data analysis method are as follows:

[0063] First, a multidimensional network indicator data set D = {x1, x2, ... xn} is input, where x = {α1, α2, ... αN}, αi is the network indicator value of the i-th dimension.

[0064] Next, the two parameters EPs and MinPts in the DBSCAN algorithm are automatically determined by the DBSCAN parameter adaptive method, as follows: the input multidimensional network indicator data set D is mapped to the N-dimensional space, and the initial matrix DISTn*n={d(i,j),1≤i≤n,1≤j≤n} is obtained according to the mapping situation, where d represents the distance value from the i-th mapping point to the j-th mapping point. Then the obtained initial matrix DIST is reordered, and the distance value of each second data row in the initial matrix DIST is sorted from small to large, so that the second data row will be converted into the first data row after sorting, thereby obtaining the distance matrix Xn*n={x(i,j),1≤i≤n,1≤j≤n}, where x(i,j) represents the distance value of the j-th mapping point closest to the i-th mapping point after sorting. Then draw a graph of Xi (Xi represents the first data row of the i-th distance matrix Xn*n, that is, the set of sorted distance values ​​from the i-th mapping point, 1≤i≤n), and calculate the slope of each data point in Xi relative to the next data point. The slope at j is ||x(i,j)-x(i,j+1)||, 1≤i≤n,1≤j≤n. Then calculate the average slope and standard deviation of all non-zero slopes, and find the first slope that is greater than the sum of the average slope and the standard deviation of the slope as the target slope. The target slope corresponds to the inflection point of the curve. At this time, the clustering radius is the largest and the clustering effect is the best. Therefore, the corresponding distance value at this point is Eps. When the distance value is EPs, the corresponding j is MinPts.

[0065] Then, mark all objects in the multidimensional network indicator dataset D as unread, take any mapping point Di∈D (i=1,2,3...) from D, and mark Di as read; judge any mapping point through EPs and MinPts, if it is a core object (high-density point), find all density-reachable data objects within its radius and mark them as read, where density-reachable means that the points within the radius are density-reachable points with the high-density point as the core. If it is not a high-density point and no object is density-reachable, mark it as an outlier; the outlier identified in this step is not a high-density point itself, and is not within the radius of any high-density point.

[0066] Finally, the embodiment of the present invention will also use one of the core objects as a seed, classify all the density-reachable points of the object into one category, and form a larger range of data object sets, also known as clusters; after all core objects are traversed, the remaining points that are not classified into one category are abnormal points. The abnormal points identified in this step are high-density points themselves, but they are not reachable from other high-density points.

[0067] like Figure 7 As shown, Figure 7It is a flowchart of an abnormal data analysis method provided by an embodiment of the present invention. The method includes but is not limited to step S800, step S900, step S1000 and step S1100.

[0068] Step S800, obtaining an abnormal sample set, wherein the abnormal samples in the abnormal sample set correspond to multiple network indicators;

[0069] Step S900, obtaining multiple network indicator combinations according to multiple network indicators;

[0070] Step S1000, processing multiple network indicator combinations to obtain root cause possibility scores of each network indicator;

[0071] Step S1100, determining the network indicator that has the greatest contribution to generating the abnormal sample set according to the root cause possibility score.

[0072] In one embodiment, since there are many network indicators corresponding to the abnormal samples in the input abnormal sample set, the influence of each network indicator on the abnormal result cannot be directly judged. In order to effectively locate the contribution of each dimensional network indicator to the final abnormal result and realize a more specific and intuitive abnormal analysis of multi-dimensional network indicators, the embodiment of the present invention can perform root cause analysis on the abnormal samples in the abnormal sample set. Due to the large amount of data and the complex association of network indicators, it is necessary to search the root cause layer by layer from the obtained abnormal samples to locate the abnormal dimension set. The embodiment of the present invention uses PS (Potential Score, possibility score) calculation to perform root cause analysis on the contribution of all dimensional indicators to find out the influence of each dimensional network indicator on the abnormal result. Therefore, based on the calculation of the root cause possibility score, the embodiment of the present invention obtains multiple network indicator combinations according to the multiple network indicators corresponding to the abnormal samples, and performs root cause analysis on the multiple network indicator combinations to obtain the root cause possibility score of each network indicator. Finally, according to the value of the root cause possibility score, the contribution of each network indicator to the abnormal result can be obtained, thereby obtaining the network indicator with the greatest contribution to the generation of the abnormal sample set. Therefore, the embodiment of the present invention can calculate the root cause possibility score for each network indicator in the abnormal sample set, and through root cause analysis, dig up the abnormal samples to find the contribution of each network indicator to the abnormal results, so as to quickly trace the source network indicators of the abnormal samples and improve the accuracy of the abnormal data analysis system.

[0073] It is worth noting that, regarding the above-mentioned abnormal samples, the abnormal samples of the embodiment of the present invention can be the above-mentioned Figures 2 to 6 It can be obtained by the steps or methods of any embodiment, or it can also be directly obtained by other methods.

[0074] It should be noted that the above-mentioned network indicator combination may be one network indicator or multiple network indicators.

[0075] It can be understood that the above-mentioned root cause possibility score is specifically expressed as a numerical value. When the numerical value is larger, it indicates that the network indicator of this dimension contributes more to the abnormal result; when the numerical value is smaller, it indicates that the network indicator of this dimension contributes less to the abnormal result.

[0076] like Figure 8 As shown, Figure 8 2 is a flow chart of an abnormal data analysis method provided by another embodiment of the present invention. In one embodiment, the above step S1000 includes but is not limited to step S1210, step S1220, step S1230, step S1240 and step S1250.

[0077] Step S1210, determining a minimum element and a non-minimum element according to a plurality of network indicator combinations, wherein the minimum element is a single network indicator in the plurality of network indicator combinations, and the non-minimum element is a combination of at least two network indicators in the plurality of network indicator combinations;

[0078] Step S1220, obtaining the value of the minimum element and the value of the non-minimum element according to the abnormal sample set;

[0079] Step S1230, obtaining a first offset of the minimum element, where the first offset is a difference between the value of the minimum element and a normal value of the network indicator corresponding to the minimum element;

[0080] Step S1240, obtaining a second offset of the non-minimum element, where the second offset is a difference between the value of the non-minimum element and a normal value of the network indicator combination corresponding to the non-minimum element;

[0081] Step S1250, obtaining the root cause possibility score of each network indicator according to the first offset and the second offset.

[0082] In one embodiment, the embodiment of the present invention screens the dimension combination layer by layer by calculating the PS value to obtain the PS value of each dimensional network indicator. The larger the PS value, the greater the contribution. Since the influence of each dimensional network indicator on the abnormal result is not simply added, but will have an impact on each other, it is not possible to simply add the influence of a certain dimensional network indicator directly, but use the possibility score for calculation. The specific method is as follows: Used to calculate the second offset of non-minimum elements, that is, the offset between outliers and normal values; Used to calculate the first offset of the minimum element. Where v represents the abnormal value of a specific element combination, f represents the normal value of an element in a certain dimension, a represents the normal value of a specific element combination, i represents the data dimension, and 1≤i≤N.

[0083] It is worth noting that the first offset mentioned above can be obtained by the following formula:

[0084]

[0085] in, is the first offset, v is the value of the non-minimum element, f is the normal value of the network indicator corresponding to the minimum element, i is the number of network indicators, and i is a positive integer.

[0086] Secondly, the second offset mentioned above can be obtained by the following formula:

[0087]

[0088] in, is the second offset, and a is the normal value of the network indicator combination corresponding to the non-minimum element.

[0089] In addition, the root cause probability score is obtained by the following formula:

[0090]

[0091] Among them, PS is the root cause probability score.

[0092] like Fig. 9 As shown, Fig. 9 It is an overall schematic diagram of an abnormal data analysis method provided by another embodiment of the present invention.

[0093] For example, a set of network indicators including Q (quality, service quality), P (power, optical power), L (packet loss rate, packet loss rate) and T (delay time) are input as a multidimensional sample set D = {d1, d2, ..., d12}, Q takes the value of Q1, Q2; P takes the value of P1, P2, P3; L takes the value of L1, L2; T takes the value of T1, T2; d1 = (Q1, P1, L1, T1), d2 = (Q1, P1, L1, T2) and so on; the distance matrix D12*12 = {d(i, j), 1≤i≤12, 1≤j≤12}, and each row of elements in D is sorted from small to large to obtain X12*12; EPs and MinPts are obtained by calculating the steep increase in slope. All elements are traversed and marked to obtain an abnormal sample set X = {x1, x2, ..., x16}.

[0094] For this four-dimensional indicator sample, the possible values ​​of Q, P, L and T are 2, 3, 2 and 2 respectively, so we can use Fig. 9 The two three-dimensional cubes shown in the figure represent that the cube on the left is the original sample and the cube on the right is the abnormal sample. Traverse all possible combinations of dimensions and assume that the changes in L and T have a zero impact on the final abnormal result based on the values ​​in the root cause combination of the abnormal sample. The values ​​of the remaining two dimensions, P and Q, have changed. According to the indicator combination in this example, its data structure can be represented. The first layer has P values ​​of P1, P2, and P3; Q values ​​of Q1 and Q2. When searching upward, searching the first layer, there are two indicators that affect the abnormal result. For indicator P, P1 is offset by 30 from the normal value of 30, and P2 is offset by 30 from the normal value of 40. For indicator Q, Q1 is offset by 35 from the normal value of 45, and P2 is offset by 25 from the normal value of 55. P3 has no offset compared to the normal value and will not affect the abnormal result. Therefore, when searching the second layer, there is no need to calculate the influence of P3. The calculation is performed using the possibility score, and the specific method is shown in the following formula:

[0095]

[0096] The impact score of the network indicator of each dimension is calculated, where v represents the abnormal value of a specific element combination, f represents the normal value of an element of a certain dimension, and a represents the normal value of a specific element combination. For the P dimension, P3 does not produce an offset and its impact is not considered, so it is calculated as follows: where △ represents the offset of the unit data.

[0097]

[0098] Since △(P1, Q2), △(P2, Q1) are zero, we can calculate:

[0099]

[0100] It follows that:

[0101]

[0102] The impact probability score of dimension P is 0.5. Similarly, the impact probability score of dimension Q is:

[0103]

[0104] Therefore, the Effect is ranked as Q, P, L and T, and the output is a set of abnormal samples ranked by indicator influence, that is, the final result is X = {X1, X2, ..., X16} among the 16 abnormal points, and Q>P>L=T in each data point.

[0105] It is worth noting that the specific implementation manner and corresponding technical effects of the abnormal data analysis method of the embodiment of the present invention can refer to the embodiment of the abnormal data analysis method described above.

[0106] Based on the above abnormal data analysis method, various embodiments of the abnormal data analysis system and computer-readable storage medium of the present invention are respectively proposed below.

[0107] In addition, an embodiment of the present invention provides an abnormal data analysis system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0108] The processor and the memory may be connected via a bus or other means.

[0109] It should be noted that the abnormal data analysis system in this embodiment can correspond to the following: Figure 1 The abnormal data analysis system in the system architecture platform in the embodiment shown can constitute Figure 1 Part of the system architecture platform in the illustrated embodiment, both belong to the same inventive concept, so both have the same implementation principles and beneficial effects, and will not be described in detail here.

[0110] The non-transient software program and instructions required to implement the abnormal data analysis method of the above embodiment are stored in the memory. When executed by the processor, the abnormal data analysis method of the above embodiment is executed, for example, the above described abnormal data analysis method is executed. Figure 2 The method steps S100 to S400, Figure 4 Steps S510 to S520 of the method, Figure 5 Steps S710 to S740 of the method, Figure 7 Steps S800 to S1100 of the method, Figure 8 Method steps S1210 to S1250.

[0111] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to execute the above-mentioned abnormal data analysis method. For example, when executed by a processor in the above-mentioned abnormal data analysis system embodiment, the above-mentioned processor can execute the abnormal data analysis method in the above-mentioned embodiment, for example, executing the above-mentioned Figure 2The method steps S100 to S400, Figure 4 Steps S510 to S520 of the method, Figure 5 Steps S710 to S740 of the method, Figure 7 Steps S800 to S1100 of the method, Figure 8 Method steps S1210 to S1250.

[0113] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0114] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions under the shared conditions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for analyzing abnormal data, comprising: Get a sample dataset; Mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix, wherein the distance matrix includes a plurality of first data rows, the first data rows correspond to the sample data in the sample data set one by one, the first data rows include distance values ​​between the sample data corresponding to the first data rows in the N-dimensional space and the remaining sample data in the sample data set, and the distance values ​​in the first data rows are sorted according to numerical values; N is a positive integer; Obtaining the clustering radius and density threshold of the density-based clustering DBSCAN algorithm according to the distance matrix; Determine abnormal data in the sample data set using the cluster radius and the density threshold; Wherein, the clustering radius and density threshold of the DBSCAN algorithm are obtained according to the distance matrix, including: Traversing the first data rows in the distance matrix, in each of the first data rows, taking the distance value with the smallest value as the data point for starting calculation, sequentially calculating the slope of the previous data point relative to the next data point, and respectively obtaining a slope list corresponding to each of the first data rows; According to all non-zero slopes in all the slope lists, the slope average and slope standard deviation are calculated; Traversing all the slope lists, obtaining the first slope whose value is greater than the sum of the slope average value and the slope standard deviation as the target slope; The clustering radius and density threshold of the DBSCAN algorithm are determined according to the target slope.

2. The method according to claim 1, characterized in that Mapping the sample data in the sample data set to an N-dimensional space to obtain a distance matrix includes: Mapping the sample data in the sample data set to an N-dimensional space to obtain an initial matrix; wherein the initial matrix includes a plurality of second data rows, each of the second data rows corresponds to the sample data in the sample data set one by one, and each of the second data rows includes distance values ​​between the sample data corresponding to the second data row in the N-dimensional space and the remaining sample data in the sample data set; The distance values ​​in each of the second data rows are sorted according to numerical values ​​to obtain the distance matrix.

3. A method for analyzing abnormal data, comprising: Acquire an abnormal sample set, wherein the abnormal samples in the abnormal sample set correspond to multiple network indicators; Obtaining multiple network indicator combinations according to the multiple network indicators; Processing the plurality of network indicator combinations to obtain a root cause probability score of each of the network indicators; Determining the network indicator that has the greatest contribution to generating the abnormal sample set according to the root cause possibility score; The processing of the plurality of network indicator combinations to obtain the root cause possibility score of each of the network indicators includes: Determine a minimum element and a non-minimum element according to the multiple network indicator combinations, wherein the minimum element is a single network indicator in the multiple network indicator combinations, and the non-minimum element is a combination of at least two network indicators in the multiple network indicator combinations; Acquire the value of the minimum element and the value of the non-minimum element according to the abnormal sample set; Obtaining a first offset of the minimum element, where the first offset is a difference between a value of the minimum element and a normal value of a network indicator corresponding to the minimum element; Obtaining a second offset of the non-minimum element, where the second offset is a difference between a value of the non-minimum element and a normal value of a network indicator combination corresponding to the non-minimum element; A root cause probability score of each of the network indicators is obtained according to the first offset and the second offset.

4. The method according to claim 3, characterized in that The first offset is obtained by the following formula: Among them, the is the first offset, v is the value of the non-minimum element, f is the normal value of the network indicator corresponding to the minimum element, i is the number of the network indicators, and i is a positive integer.

5. The method according to claim 4, characterized in that The second offset is obtained by the following formula: Among them, the is the second offset, and a is the normal value of the network indicator combination corresponding to the non-minimum element.

6. The method according to claim 5, characterized in that The root cause probability score is obtained by the following formula: Wherein, the PS is the root cause probability score.

7. An abnormal data analysis system, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the abnormal data analysis method as described in any one of claims 1 to 2 when executing the computer program, or the processor implements the abnormal data analysis method as described in any one of claims 3 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are used to execute the abnormal data analysis method as described in any one of claims 1 to 2, or the computer executable instructions are used to execute the abnormal data analysis method as described in any one of claims 3 to 6.

Images