Data anomaly detection method and device and storage medium

Abstract

The invention provides a data exception detection method and device and a storage medium. The method comprises the steps of obtaining a first data set for exception detection; performing scene classification on the data in the first data set, and obtaining a second data set corresponding to each scene type based on a scene classification result; performing denoising processing on the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type; based on the third data set corresponding to each scene type, determining a first target threshold interval of each field in each scene type through a box plot quartile algorithm; and performing anomaly detection on the to-be-detected data based on the first target threshold interval of each field in each scene type to obtain an anomaly detection result. Therefore, the threshold interval does not need to be set manually, so that the first threshold interval of each field is objective and reasonable, and the anomaly detection result obtained through the first target threshold interval is more accurate.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and IT support technology, and in particular to a data anomaly detection method, device and storage medium. Background Art

[0002] With the development of 5G private network services, more and more scenarios are beginning to deploy 5G private network applications. Among them, the activation of 5G private network services requires business personnel to submit a service activation form and fill in DNN and slice-related business information.

[0003] At present, the relevant data of 5G private network service activation has irregular data filling and unreasonable field value filling, which affects the service activation process and the subsequent 5G private network operation and maintenance to a certain extent. Based on this, how to detect data anomalies in 5G private network resource data and discover data problems in a timely manner is an issue that needs to be solved urgently.

[0004] In the prior art, for the numerical field values ​​in the 5G private network resource data, the reasonable value range of the minimum and maximum values ​​of the field can be manually set to determine whether the field value is within the range. If it is not within the range, it is judged as abnormal data. However, the values ​​of the same field in the 5G private network resource data of different scenarios vary greatly. For example, the number of terminal IPV4 addresses required in the 5G private network project of the Internet of Vehicles reaches more than 100,000, and the number of terminal IPV4 addresses required in the 5G private network project of the power industry is only a few hundred. If the fixed and single threshold range manually set in the above scheme is used, it may not be possible to accurately detect that the field values ​​filled in for some projects are abnormal. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides a data anomaly detection method, device and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for detecting anomalies in data is provided, the method comprising:

[0007] Acquire a first data set for anomaly detection;

[0008] Classifying the data in the first data set into scenes, and obtaining a second data set corresponding to each scene type based on the scene classification result;

[0009] Performing denoising processing on the second data sets corresponding to the respective scene types to obtain third data sets corresponding to the respective scene types;

[0010] Based on the third data set corresponding to each scenario type, determining a first target threshold interval for each field in each scenario type by using a box plot quartile algorithm;

[0011] Anomaly detection is performed on the data to be detected based on the first target threshold interval of each field in each scenario type to obtain anomaly detection results.

[0012] Optionally, in the embodiment of the present disclosure, the step of obtaining a first data set for anomaly detection includes:

[0013] Get the project information table in the resource data, as well as the DNN data used for anomaly detection and the numerical field data in the slice business data;

[0014] The project name in the project information table is extracted, and the field data in the DNN data and the slice business data are associated to obtain a first data set for anomaly detection.

[0015] Optionally, in the embodiment of the present disclosure, the performing denoising processing on the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type includes:

[0016] Performing first data preprocessing on the second data sets corresponding to the respective scene types to obtain fourth data sets corresponding to the respective scene types;

[0017] Obtaining a data score value of each data of the fourth data set through the trained isolation forest algorithm model;

[0018] The data having a data score value less than a preset threshold is removed from the fourth data set to obtain a third data set corresponding to each scene type.

[0019] Optionally, in an embodiment of the present disclosure, determining the first target threshold interval of each field in each scene type by a box plot quartile algorithm based on the third data set corresponding to each scene type includes:

[0020] Performing second data preprocessing on the third data sets corresponding to the respective scene types to obtain fifth data sets corresponding to the respective scene types;

[0021] Calculate the first quartile, the third quartile and the interquartile range (IQR) of each field in the fifth data set by using the box plot quartile algorithm;

[0022] Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, a first target threshold interval for each field in each scenario type is determined.

[0023] Optionally, in the embodiment of the present disclosure, performing second data preprocessing on the third data set corresponding to each scene type to obtain a fifth data set corresponding to each scene type includes:

[0024] Based on the data of each field in the third data set corresponding to each scenario type, obtaining a first data group for each field;

[0025] Deduplication processing is performed on the data in the first data group of each field to obtain a second data group of each field;

[0026] Sort the second data groups of the fields in ascending order to obtain third data groups of the fields;

[0027] The third data group of each field in each scene type is determined as a fifth data set corresponding to each scene type.

[0028] Optionally, in the embodiment of the present disclosure, determining the first target threshold interval of each field in each scenario type based on the first quartile, the third quartile and the interquartile range IQR of each field includes:

[0029] Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the upper threshold limit of each field in each scenario type is determined by the upper threshold limit calculation formula;

[0030] Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the lower limit of the threshold of each field in each scenario type is determined by a lower limit calculation formula;

[0031] Based on the upper threshold and the lower threshold of each field, a first target threshold interval of each field in each scenario type is determined.

[0032] Optionally, in the embodiment of the present disclosure, performing anomaly detection on the data to be detected based on the first target threshold interval of each field in each scenario type to obtain anomaly detection results includes:

[0033] Determine the target scene type to which the data to be detected belongs;

[0034] Determining a second target threshold interval corresponding to the data to be detected based on a first target threshold interval of the target scene type;

[0035] If the data to be detected is not within the second target threshold range, it is determined that the abnormality detection result of the data to be detected is abnormal.

[0036] According to a second aspect of an embodiment of the present disclosure, a data anomaly detection device is provided, the device comprising:

[0037] An acquisition module, used for acquiring a first data set for anomaly detection;

[0038] A classification module, used to classify the data in the first data set into scene categories, and obtain a second data set corresponding to each scene type based on the scene classification result;

[0039] A data processing module, used for performing denoising processing on the second data sets corresponding to the respective scene types to obtain third data sets corresponding to the respective scene types;

[0040] A determination module, configured to determine a first target threshold interval for each scene type by using a box plot quartile algorithm based on a third data set corresponding to each scene type;

[0041] The anomaly detection module is used to perform anomaly detection on the data to be detected based on the first target threshold interval of each scenario type to obtain anomaly detection results.

[0042] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0043] processor;

[0044] a memory for storing processor-executable instructions;

[0045] The processor is configured to implement the method described in the first aspect above.

[0046] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the method described in the first aspect above can be implemented.

[0047] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0048] The present disclosure proposes a data anomaly detection method, device and storage medium, the method comprising obtaining a first data set for anomaly detection; classifying the data in the first data set by scene, and obtaining a second data set corresponding to each scene type based on the scene classification result; performing denoising on the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type; determining the first target threshold interval of each field in each scene type by a box plot quartile algorithm based on the third data set corresponding to each scene type; performing anomaly detection on the data to be detected based on the first target threshold interval of each field in each scene type to obtain anomaly detection results. Therefore, the present disclosure can perform reasonable threshold calculation on the denoised third data set by a box plot quartile algorithm to obtain the first target threshold interval of each field in each scene type without manual setting, so that the first threshold interval of each field is objective and reasonable, and thus the anomaly detection result obtained by the first target threshold interval is more accurate.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] Figure 1 is a flowchart of a data anomaly detection method according to some embodiments of the present disclosure;

[0052] Figure 2 is a structural schematic diagram of a data anomaly detection device according to some embodiments of the present disclosure;

[0053] Figure 3 It is a block diagram of an electronic device suitable for executing a data anomaly detection method according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0054] Some embodiments of the present disclosure will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. Various changes, modifications and equivalents of the methods, devices and / or devices described herein will become apparent after understanding the present disclosure. For example, the order of operations described herein is merely an example and is not limited to those orders set forth herein, but can be changed as becomes apparent after understanding the present disclosure, except for operations that must be performed in a specific order. In addition, for clarity and brevity, descriptions of features known in the art may be omitted.

[0055] The embodiments described in some embodiments of the present disclosure below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] Figure 1 The flowchart of a data anomaly detection method according to some embodiments of the present disclosure is shown in the figure. As shown in the figure, the method may include the following steps:

[0057] Step 101: Obtain a first data set for anomaly detection.

[0058] It should be noted that the execution subject of the above-mentioned data anomaly detection method is a data anomaly detection device, which can be implemented by software and / or hardware. The data anomaly detection device in this embodiment can be configured in an electronic device.

[0059] In this example embodiment, the electronic device may include devices such as terminal devices and servers, and this embodiment does not limit the electronic device.

[0060] In an embodiment of the present disclosure, the above-mentioned first data set for anomaly detection can be acquired based on 5G private network resource data.

[0061] Specifically, in an embodiment of the present disclosure, the method for obtaining a first data set for anomaly detection may include the following steps:

[0062] Step 1011: Obtain the project information table in the resource data, as well as the DNN data for anomaly detection and the numerical field data in the slice business data;

[0063] In the disclosed embodiment, the project information table in the resource data and the numerical field data in the DNN data and slice business data used for anomaly detection can be obtained by extracting the project information table in the 5G private network resource data and the numerical field data in the DNN data and slice business data used for anomaly detection.

[0064] Step 1012: extract the project name in the project information table, and associate the field data in the DNN data and the slice business data to obtain a first data set for anomaly detection.

[0065] In the embodiment of the present disclosure, the project name and project ID in the project information table can be extracted, and the fields of DNN identifier, terminal IP address requirement number, user uplink maximum rate and user downlink maximum rate in the DNN data, and the field data of slice service identifier, number of card activations, number of slice registered users and end-to-end delay in the slice service data can be associated to obtain a first data set for anomaly detection.

[0066] Furthermore, in the embodiment of the present disclosure, the first data set may also include a company name as auxiliary information for subsequent scene classification.

[0067] Step 102: Classify the data in the first data set by scene, and obtain a second data set corresponding to each scene type based on the scene classification result.

[0068] In the embodiment of the present disclosure, after the first data set is obtained through the above steps, the items in the first data set can be classified into scenes, and a second data set corresponding to each scene type can be obtained based on the scene classification results. In particular, in the embodiment of the present disclosure, before the items in the first data set are classified into scenes, commonly used scene types can be manually limited, including but not limited to electricity, Internet of Vehicles, smart parks, smart industries, video surveillance, enterprise private networks, and government affairs.

[0069] And, in the embodiment of the present disclosure, before the scene classification is performed on the items in the first data set, it is necessary to preprocess the first data set, and perform scene classification based on the preprocessed data. In the embodiment of the present disclosure, the method for preprocessing the first data set may include: reading the project names in the first data set, and deleting the data in which the project name is empty and / or the project name is less than 2 characters, to obtain the preprocessed data.

[0070] Furthermore, in the disclosed embodiment, after obtaining the preprocessed data through the above steps, the scene type to which the preprocessed data belongs can be obtained through the large language model, and the second data set corresponding to each scene type can be obtained based on the scene classification result of the preprocessed data. Among them, in the disclosed embodiment, the above large language model can be a public large language model that has been trained. And, by using the above large language model to identify the scene type, the workload of manual project scene classification is reduced, and the recognition efficiency is improved.

[0071] Furthermore, in the embodiment of the present disclosure, for the items in the first data set of the unrecognized scene types, the names of the items of the unrecognized scene types can be manually analyzed and judged to supplement the existing scene types.

[0072] Further, in the embodiment of the present disclosure, after the scene types are supplemented through the above steps, the project name and company name in the first data set of the unrecognized scene type can be used as parameters to continue calling the above large language model for scene classification until all the data in the above first data set are successfully classified. For example, Table 1 is a scene classification result data table provided in the embodiment of the present disclosure.

[0073] Table 1

[0074] Project ID Project Name Scene Type 1 Project Name 1 Internet of Vehicles 2 Project Name 2 Internet of Vehicles 101 Project Name 101 electricity ... ... ...

[0075] As shown in Table 1, each project name and project ID has a corresponding scene type. For example, the scene types corresponding to project name 1 and project name 2 are Internet of Vehicles, and the scene type corresponding to project name 101 is electricity.

[0076] Furthermore, in the embodiment of the present disclosure, after the scene classification results of all the data in the first data set are obtained through the above steps, the associated data sets of each scene type in the first data set can be determined as the second data sets corresponding to each scene type, and the second data sets corresponding to each scene type need to contain each scene type and the data fields to be detected. Table 2 is a relationship correspondence table between a vehicle networking scene type and a field in the second data set provided in the embodiment of the present disclosure. Table 3 is a relationship correspondence table between a power scene type and a field in the second data set provided in the embodiment of the present disclosure.

[0077] Table 2

[0078]

[0079] Table 3

[0080]

[0081] As shown in Tables 2 and 3, the numerical ranges of the number of terminal IPV4 addresses required for the Internet of Vehicles scenario and the number of terminal IPV4 addresses required for the power scenario are different.

[0082] Step 103: De-noise the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type.

[0083] In the disclosed embodiment, after the second data set corresponding to each scene type is obtained through the above steps, the second data set corresponding to each scene type may be denoised to obtain a third data set corresponding to each scene type.

[0084] Specifically, in the embodiment of the present disclosure, the method of performing denoising on the second data set corresponding to each scene type to obtain the third data set corresponding to each scene type may include the following steps:

[0085] Step 1031: perform first data preprocessing on the second data set corresponding to each scene type to obtain a fourth data set corresponding to each scene type;

[0086] Step 1032: Obtain a data score value of each data of the fourth data set through the trained isolation forest algorithm model;

[0087] Step 1033: remove data with a data score value less than a preset threshold from the fourth data set to obtain a third data set corresponding to each scene type.

[0088] In the disclosed embodiment, some data in the second data set corresponding to each of the above-mentioned scene types contain null values. Based on this, it is necessary to perform first data preprocessing on the second data set corresponding to each of the scene types so that subsequent data analysis results are more accurate.

[0089] In one embodiment of the present disclosure, the method of performing first data preprocessing on the second data set corresponding to each scene type to obtain a fourth data set corresponding to each scene type may include: filling the fields with empty values ​​or null values ​​in the second data set with 0 to obtain a fourth data set corresponding to each scene type.

[0090] And, in another embodiment of the present disclosure, the above-mentioned method of performing first data preprocessing on the second data set corresponding to each scene type to obtain a fourth data set corresponding to each scene type may include: eliminating data with empty or null data fields in the second data set to obtain a fourth data set corresponding to each scene type.

[0091] Furthermore, in the embodiment of the present disclosure, before obtaining the data score value of each data of the fourth data set through the trained isolation forest algorithm model, it is necessary to train the initial isolation forest algorithm model to obtain the trained isolation forest algorithm model.

[0092] Specifically, in an embodiment of the present disclosure, the method of training an initial isolation forest algorithm model to obtain a trained isolation forest algorithm model may include: constructing an initial isolation forest algorithm model, and training the initial isolation forest algorithm model through training data to obtain a trained isolation forest algorithm model.

[0093] Among them, in the embodiment of the present disclosure, the initial parameters in the above-mentioned initial isolation forest algorithm model include: the number of random trees generated in the forest n_estimators = 100; the number of samples of the subtree max_samples = 256, wherein, if the data set is less than 256 samples, the whole amount is used as samples; the number of features of each subtree max_features = 3; the proportion of abnormal data in the given data set contamination = 0.1. And, in the embodiment of the present disclosure, the above-mentioned initial isolation forest algorithm model can be implemented by Python or R language.

[0094] Further, in the disclosed embodiment, after the initial isolation forest algorithm model is constructed through the above steps, the training data can be trained through the fit_predict function to obtain a trained isolation forest algorithm model. In the disclosed embodiment, the output of the trained isolation forest algorithm model can be a data score value for each piece of training data.

[0095] And, in the embodiment of the present disclosure, after the trained isolation forest algorithm model is obtained through the above steps, the data scoring value of each data in the fourth data set can be obtained through the trained isolation forest algorithm model. Specifically, in the embodiment of the present disclosure, each numerical field in the fourth data set is used as a dimension, and the obtained multidimensional data set is input into the trained isolation forest algorithm model to obtain the data scoring value of each data in the fourth data set. Table 4 is an abnormality scoring table corresponding to the fourth data set of a type of Internet of Vehicles scenario proposed in the embodiment of the present disclosure.

[0096] Table 4

[0097]

[0098] Further, in the embodiment of the present disclosure, after obtaining the data score value of each data in the fourth data set through the above steps, the data with a data score value less than a preset threshold can be removed from the fourth data set to obtain a third data set corresponding to each scene type. In the embodiment of the present disclosure, the above preset threshold can be set as needed, for example, the preset threshold can be 0.

[0099] For example, in the embodiment of the present disclosure, if the preset threshold is 0, as shown in Table 4, the anomaly score corresponding to the data with project ID 8 is -0.05658, which is less than 0, then the data with project ID 8 is removed from the fourth data set.

[0100] Among them, in the embodiment of the present disclosure, there are very few abnormal data in the 5G private network resource data, and the proportion of abnormal data in the data is very low. In addition, the difference between abnormal points and normal data points is large, and the distribution between them is relatively sparse. Based on this, the isolation forest model trained above is used to make the analysis of the abnormal data of the fourth data set more accurate, so that the first target threshold interval of each field determined subsequently is more accurate.

[0101] Step 104: Based on the third data set corresponding to each scene type, determine the first target threshold interval of each field in each scene type by using a box plot quartile algorithm.

[0102] In the disclosed embodiment, after determining the third data set corresponding to each scene type through the above steps, the first target threshold interval of each field in each scene type can be determined through a box plot quartile algorithm based on the third data set corresponding to each scene type.

[0103] Specifically, in the embodiment of the present disclosure, the method for determining the first target threshold interval of each field in each scene type by using a box plot quartile algorithm based on the third data set corresponding to each scene type may include the following steps:

[0104] Step 1041, performing second data preprocessing on the third data set corresponding to each scene type to obtain a fifth data set corresponding to each scene type;

[0105] Step 1042: Calculate the first quartile, the third quartile and the interquartile range (IQR) of each field in the fifth data set by using the box plot quartile algorithm;

[0106] Step 1043: Determine the first target threshold interval for each field in each scenario type based on the first quartile, the third quartile, and the interquartile range (IQR) of each field.

[0107] In the embodiment of the present disclosure, the method of performing second data preprocessing on the third data set corresponding to each scene type to obtain a fifth data set corresponding to each scene type may include the following steps:

[0108] Step a, based on the data of each field in the third data set corresponding to each scene type, obtaining a first data group of each field;

[0109] Step b, performing deduplication processing on the data in the first data group of each field to obtain a second data group of each field;

[0110] Step c, sorting the second data group of each field in ascending order to obtain a third data group of each field;

[0111] Step d: determine the third data set of each field in each scene type as the fifth data set corresponding to each scene type.

[0112] For example, in an embodiment of the present disclosure, Table 5 is a third data group of each field corresponding to each scenario type proposed in an embodiment of the present disclosure.

[0113] Table 5

[0114] Fields Scenario The third data set IPV4_add_quantity Internet of Vehicles [200000,300000,524288,824288,...,1000000,1048576,2000000,2097150] IPV4_add_quantity electricity [10,32,100,128,154,200,...,254,600,800,1000,1400] ... ... ... Downlink_rate Internet of Vehicles [1,5,10,15,20,50,80,...,100,150,200,300,400,450,500] Downlink_rate electricity [100,200,500,800,900,...,1000,1200,1500,2000,2500] ... ... ...

[0115] In addition, in an embodiment of the present disclosure, according to the fifth data set corresponding to each scene type, the quartiles are calculated by a box plot quartile algorithm, which are the first quartile (Q1), the second quartile (Q2) and the third quartile (Q3), and the IQR (i.e., interquartile range, which reflects the dispersion of the data) is calculated using the calculated first quartile and third quartile.

[0116] Among them, in the embodiment of the present disclosure, the algorithm for calculating the mth quartile and IQR is as follows:

[0117] k=(n+1)*m / 4

[0118] Q m =A[k]

[0119] IQR=Q3-Q1

[0120] Where n is the number of data in the third data group A corresponding to the field, k is the integer part of (n+1)*m / 4, f is the decimal part of the formula, A[k] is the kth value of the third data group A, Qm is the mth quartile (m is 1, 2, 3). And, if the above k is a non-integer, the algorithm for the mth quartile and IQR is as follows:

[0121] k = TRUNC ((n+1)*m / 4)

[0122] f=(n+1)*m / 4-TRUNC((n+1)*m / 4)

[0123] Qm=A[k]+(f*(A[k+1]-A[k]))

[0124] IQR=Q3-Q1

[0125] Among them, TRUNC is a function for removing the decimal part of a number.

[0126] And, in the embodiment of the present disclosure, based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the method for determining the first target threshold interval of each field in each scenario type may include the following steps:

[0127] Step 10431: Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the upper threshold limit of each field in each scenario type is determined by using the upper threshold limit calculation formula;

[0128] Step 10432: Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the lower limit of the threshold of each field in each scenario type is determined by the lower limit calculation formula;

[0129] Step 10433: Determine the first target threshold interval for each field in each scenario type based on the upper threshold and the lower threshold of each field.

[0130] In the embodiment of the present disclosure, the calculation formula for the upper threshold value is:

[0131] The upper threshold value Yu=Q3+1.5*IQR.

[0132] And, in the embodiment of the present disclosure, the calculation formula of the lower limit of the above threshold is:

[0133] The lower threshold value Yd=Q1-1.5*IQR.

[0134] For example, Table 6 is a first target threshold interval for each field in each scenario type proposed in an embodiment of the present disclosure.

[0135] Table 6

[0136]

[0137] Further, in the embodiment of the present disclosure, after the upper and lower thresholds of each field are determined through the above steps, the interval formed by the upper and lower thresholds of each field can be used to determine the first target threshold interval of each field in each scenario type.

[0138] Furthermore, in the disclosed embodiment, the first target threshold interval of each field in each of the above-mentioned scenario types may be periodically updated so that the first target threshold interval is a dynamic and reasonable threshold interval, thereby making subsequent data anomaly detection results more accurate.

[0139] Step 105 : Perform anomaly detection on the data to be detected based on the first target threshold interval of each field in each scenario type to obtain anomaly detection results.

[0140] Among them, in the embodiment of the present disclosure, after obtaining the first target threshold interval of each field in each scene type through the above steps, anomaly detection can be performed on the data to be detected based on the first target threshold interval of each field in each scene type to obtain anomaly detection results.

[0141] Specifically, in the embodiment of the present disclosure, the method for performing anomaly detection on the data to be detected based on the first target threshold interval of each field in each scene type to obtain anomaly detection results may include the following steps:

[0142] Step 1051, determining the target scene type to which the data to be detected belongs;

[0143] Step 1052: Determine a second target threshold interval corresponding to the data to be detected based on the first target threshold interval of the target scene type;

[0144] Step 1053: If the data to be detected is not within the second target threshold range, determine that the abnormality detection result of the data to be detected is abnormal.

[0145] In the embodiment of the present disclosure, if the data to be detected is within the second target threshold interval, it is determined that the abnormality detection result of the data to be detected is normal.

[0146] Among them, in the embodiment of the present disclosure, the reasonable threshold range of each field in each scenario type in the 5G private network resource data can be obtained through the above steps. Based on this, when business personnel enter new 5G private network resource data, the scenario type to which the project belongs can be analyzed in real time according to the project name entered by the business personnel, and the dynamic reasonable threshold range of the relevant numerical fields under the corresponding scenario type can be provided in real time as a reference, so as to perform anomaly detection on the entered data through the threshold range.

[0147] The present disclosure proposes a data anomaly detection method, which obtains a first data set for anomaly detection; classifies the data in the first data set into scenes, and obtains a second data set corresponding to each scene type based on the scene classification result; performs denoising on the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type; based on the third data set corresponding to each scene type, determines the first target threshold interval of each field in each scene type through a box plot quartile algorithm; performs anomaly detection on the data to be detected based on the first target threshold interval of each field in each scene type to obtain anomaly detection results. Therefore, the present disclosure can perform reasonable threshold calculation on the denoised third data set through a box plot quartile algorithm to obtain the first target threshold interval of each field in each scene type without manual setting, thereby making the first threshold interval of each field objective and reasonable, and further making the anomaly detection result obtained through the first target threshold interval more accurate.

[0148] Figure 2 A data anomaly detection device according to some embodiments of the present disclosure may include:

[0149] An acquisition module 201 is used to acquire a first data set for anomaly detection;

[0150] The classification module 202 is used to classify the data in the first data set into scene types, and obtain a second data set corresponding to each scene type based on the scene classification result;

[0151] A data processing module 203 is used to perform denoising on the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type;

[0152] A determination module 204 is used to determine a first target threshold interval for each scene type by using a box plot quartile algorithm based on a third data set corresponding to each scene type;

[0153] The anomaly detection module 205 is used to perform anomaly detection on the data to be detected based on the first target threshold interval of each scene type to obtain anomaly detection results.

[0154] In the embodiment of the present disclosure, the acquisition module 201 is specifically used for:

[0155] Get the project information table in the resource data, as well as the DNN data used for anomaly detection and the numerical field data in the slice business data;

[0156] The project name in the project information table is extracted, and the field data in the DNN data and the slice business data are associated to obtain the first data set for anomaly detection.

[0157] Optionally, in the embodiment of the present disclosure, the data processing module 203 is further used to:

[0158] Performing first data preprocessing on the second data sets corresponding to each scene type to obtain fourth data sets corresponding to each scene type;

[0159] The data score value of each data of the fourth data set is obtained by using the trained isolation forest algorithm model;

[0160] Data with a data score value less than a preset threshold is removed from the fourth data set to obtain a third data set corresponding to each scene type.

[0161] Optionally, in the embodiment of the present disclosure, the determination module 204 is specifically configured to:

[0162] Performing second data preprocessing on the third data sets corresponding to each scene type to obtain fifth data sets corresponding to each scene type;

[0163] The first quartile, third quartile and interquartile range (IQR) of each field in the fifth data set are calculated using the box plot quartile algorithm;

[0164] Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the first target threshold interval of each field in each scenario type is determined.

[0165] Optionally, in the embodiment of the present disclosure, the determination module 204 is further configured to:

[0166] Based on the data of each field in the third data set corresponding to each scenario type, a first data group of each field is obtained;

[0167] Deduplication processing is performed on the data in the first data group of each field to obtain a second data group of each field;

[0168] Sort the second data group of each field in ascending order to obtain a third data group of each field;

[0169] The third data group of each field in each scene type is determined as the fifth data set corresponding to each scene type.

[0170] Optionally, in the embodiment of the present disclosure, the determination module 204 is further configured to:

[0171] Based on the first quartile, third quartile and interquartile range (IQR) of each field, the upper threshold limit of each field in each scenario type is determined by the upper threshold limit calculation formula;

[0172] Based on the first quartile, third quartile and interquartile range (IQR) of each field, the lower threshold limit of each field in each scenario type is determined by the lower threshold limit calculation formula;

[0173] Based on the upper and lower thresholds of each field, a first target threshold interval for each field in each scenario type is determined.

[0174] Optionally, in the embodiment of the present disclosure, the above-mentioned abnormality detection module 205 is specifically used to:

[0175] Determine the target scene type to which the data to be detected belongs;

[0176] Determine a second target threshold interval corresponding to the data to be detected based on the first target threshold interval of the target scene type;

[0177] If the data to be detected is not within the second target threshold range, it is determined that the abnormality detection result of the data to be detected is abnormal.

[0178] In one or more embodiments of the present disclosure, it includes obtaining a first data set for anomaly detection; classifying the data in the first data set by scene, and obtaining a second data set corresponding to each scene type based on the scene classification result; denoising the second data set corresponding to each scene type to obtain a third data set corresponding to each scene type; based on the third data set corresponding to each scene type, determining the first target threshold interval for each field in each scene type by a box plot quartile algorithm; performing anomaly detection on the data to be detected based on the first target threshold interval for each field in each scene type to obtain anomaly detection results. Therefore, the present disclosure can perform reasonable threshold calculation on the denoised third data set by a box plot quartile algorithm to obtain the first target threshold interval for each field in each scene type without manual setting, so that the first threshold interval for each field is objective and reasonable, and the anomaly detection result obtained by the first target threshold interval is more accurate.

[0179] like Figure 3 As shown, the electronic device 300 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 can also be stored. The computing unit 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0180] A number of components in the electronic device 300 are connected to the I / O interface 305, including: an input unit 308, such as a keyboard, a mouse, etc.; an output unit 307, such as various types of displays, speakers, etc.; a storage unit 308, such as a disk, an optical disk, etc.; and a communication unit 309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 309 allows the electronic device 300 to exchange information / data with other electronic devices through a computer network such as the Internet and / or various telecommunication networks.

[0181] The computing unit 301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 301 performs the various methods and processes described above, such as the data anomaly detection method. For example, in some embodiments, the data anomaly detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via the ROM 302 and / or the communication unit 309. When the computer program is loaded into the RAM 303 and executed by the computing unit 301, one or more steps of the data anomaly detection method described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform the data anomaly detection method in any other appropriate manner (e.g., by means of firmware).

[0182] Various embodiments of the devices and techniques described above herein can be implemented in digital electronic circuit devices, integrated circuit devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), devices on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable device including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage device, at least one input device, and at least one output device, and transmit data and instructions to the storage device, the at least one input device, and the at least one output device.

[0183] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution device, device, or electronic device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor devices, devices, or electronic devices, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage electronic device, a magnetic storage electronic device, or any suitable combination of the foregoing.

[0185] To provide interaction with a user, the devices and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0186] The apparatus and techniques described herein may be implemented in a computing device including backend components (e.g., as a data server), or a computing device including middleware components (e.g., an application server), or a computing device including frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the apparatus and techniques described herein), or a computing device including any combination of such backend components, middleware components, or frontend components. The components of the apparatus may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0187] A computer device may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed device, or a server combined with a blockchain.

[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data anomaly detection method, characterized in that: The method comprises: Acquire a first data set for anomaly detection; Classifying the data in the first data set into scenes, and obtaining a second data set corresponding to each scene type based on the scene classification result; Performing denoising processing on the second data sets corresponding to the respective scene types to obtain third data sets corresponding to the respective scene types; Based on the third data set corresponding to each scenario type, determining a first target threshold interval for each field in each scenario type by using a box plot quartile algorithm; Anomaly detection is performed on the data to be detected based on the first target threshold interval of each field in each scenario type to obtain anomaly detection results.

2. The method according to claim 1, characterized in that The step of acquiring a first data set for anomaly detection includes: Get the project information table in the resource data, as well as the DNN data used for anomaly detection and the numerical field data in the slice business data; The project name in the project information table is extracted, and the field data in the DNN data and the slice business data are associated to obtain a first data set for anomaly detection.

3. The method according to claim 1, characterized in that The performing denoising on the second data sets corresponding to the respective scene types to obtain third data sets corresponding to the respective scene types includes: Performing first data preprocessing on the second data sets corresponding to the respective scene types to obtain fourth data sets corresponding to the respective scene types; Obtaining a data score value of each data of the fourth data set through the trained isolation forest algorithm model; The data having a data score value less than a preset threshold is removed from the fourth data set to obtain a third data set corresponding to each scene type.

4. The method according to claim 1, characterized in that The determining, based on the third data set corresponding to each scene type, the first target threshold interval of each field in each scene type by using a box plot quartile algorithm includes: Performing second data preprocessing on the third data sets corresponding to the respective scene types to obtain fifth data sets corresponding to the respective scene types; Calculate the first quartile, the third quartile and the interquartile range (IQR) of each field in the fifth data set by using the box plot quartile algorithm; Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, a first target threshold interval for each field in each scenario type is determined.

5. The method according to claim 4, characterized in that The performing second data preprocessing on the third data sets corresponding to the respective scene types to obtain fifth data sets corresponding to the respective scene types includes: Based on the data of each field in the third data set corresponding to each scenario type, obtaining a first data group for each field; Deduplication processing is performed on the data in the first data group of each field to obtain a second data group of each field; Sort the second data groups of the fields in ascending order to obtain third data groups of the fields; The third data group of each field in each scene type is determined as a fifth data set corresponding to each scene type.

6. The method according to claim 4, characterized in that The determining, based on the first quartile, the third quartile and the interquartile range (IQR) of each field, a first target threshold interval of each field in each scenario type includes: Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the upper threshold limit of each field in each scenario type is determined by the upper threshold limit calculation formula; Based on the first quartile, the third quartile and the interquartile range (IQR) of each field, the lower limit of the threshold of each field in each scenario type is determined by a lower limit calculation formula; Based on the upper threshold and the lower threshold of each field, a first target threshold interval of each field in each scenario type is determined.

7. The method according to claim 1, characterized in that The performing anomaly detection on the data to be detected based on the first target threshold interval of each field in each scenario type to obtain an anomaly detection result includes: Determine the target scene type to which the data to be detected belongs; Determining a second target threshold interval corresponding to the data to be detected based on a first target threshold interval of the target scene type; If the data to be detected is not within the second target threshold range, it is determined that the abnormality detection result of the data to be detected is abnormal.

8. A data anomaly detection device, characterized in that: The device comprises: An acquisition module, used for acquiring a first data set for anomaly detection; A classification module, used to classify the data in the first data set into scene categories, and obtain a second data set corresponding to each scene type based on the scene classification result; A data processing module, used for performing denoising processing on the second data sets corresponding to the respective scene types to obtain third data sets corresponding to the respective scene types; A determination module, configured to determine a first target threshold interval for each scene type by using a box plot quartile algorithm based on a third data set corresponding to each scene type; The anomaly detection module is used to perform anomaly detection on the data to be detected based on the first target threshold interval of each scenario type to obtain anomaly detection results.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory in communication with the at least one processor, wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein the computer storage medium stores computer-executable instructions; after the computer-executable instructions are executed by a processor, the method according to any one of claims 1 to 7 can be implemented.

Images