Abnormality detection method, electronic device, medium and computer program product

Abstract

The embodiment of the invention discloses an anomaly detection method, electronic equipment, a medium and a computer program product, and the anomaly detection method comprises the steps: determining at least two isolated trees corresponding to a to-be-detected data set; based on the first path length in each isolated tree in the at least two isolated trees, determining a first weight of each isolated tree; determining a first abnormal score of first to-be-detected data through a second path length of the first to-be-detected data in each isolated tree and a first weight of each isolated tree; the first to-be-detected data is any data in the to-be-detected data set; and determining a first anomaly detection result of the first to-be-detected data based on the first anomaly score.

Description

Technical Field

[0001] This application belongs to the field of big data processing technology, and in particular relates to an anomaly detection method, electronic device, medium and computer program product. Background Technology

[0002] In the era of big data, with the dramatic increase in data volume, anomaly detection, as a crucial means of ensuring data quality, is widely applied across various fields. The core objective of anomaly detection is to identify data points that significantly deviate from normal patterns to prevent erroneous decisions or system failures. Currently, anomaly detection methods based on isolated forests are widely used. When performing anomaly detection based on the isolated forest algorithm, the average path length of the data sample within the isolated forest is typically determined based on the path length of each isolated tree in the forest. The presence of anomalies in the data sample is then determined based on this average path length. However, the accuracy of anomaly detection using this method needs improvement. Summary of the Invention

[0003] This application provides an anomaly detection method, electronic device, medium, and computer program product. By considering the differences in anomaly detection capabilities among various isolated trees during the anomaly detection process, the accuracy of anomaly detection is effectively improved.

[0004] This application provides an anomaly detection method, the method comprising: Identify at least two isolated trees corresponding to the dataset to be detected; Based on the first path length in each of the at least two isolated trees, a first weight for each isolated tree is determined; A first anomaly score is determined by the second path length of the first data to be detected in each isolated tree and the first weight of each isolated tree; the first data to be detected is any data in the dataset to be detected. The first anomaly detection result of the first data to be detected is determined based on the first anomaly score.

[0005] This application provides an electronic device, which includes a processor and a memory for storing computer programs capable of running on the processor; wherein, The processor is used to run the computer program to perform any of the above-described anomaly detection methods.

[0006] This application provides a computer storage medium storing a computer program that, when executed by a processor, implements any of the above-described anomaly detection methods.

[0007] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described anomaly detection methods.

[0008] This application provides an anomaly detection method, electronic device, medium, and computer program product. First, at least two isolated trees are constructed from the dataset to be detected to form an isolated forest, ensuring good generalization ability of anomaly detection. Second, the first path length of each isolated tree is analyzed to determine the first weight of each isolated tree, which helps to reflect the anomaly detection capability of each isolated tree and realize differentiated processing for different isolated trees. Finally, the second path length of the first data to be detected and the first weight of each isolated tree are combined to determine the first anomaly score, and the first anomaly score is used to determine whether the first data to be detected is an anomaly point, which not only ensures the robustness of anomaly detection but also improves the accuracy of anomaly detection. Attached Figure Description

[0009] Figure 1 This is a flowchart of an anomaly detection method provided in an embodiment of this application; Figure 2 This is a flowchart of an isolated tree construction method provided in an embodiment of this application; Figure 3 This is a flowchart of a first anomaly score calculation method provided in an embodiment of this application; Figure 4 This is a flowchart of another anomaly detection method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an anomaly detection device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] The era of big data has brought about massive data volumes and enormous data value. Abundant data assets can provide strong information support for daily analysis and decision-making. However, during the data collection and processing process, various data quality problems may occur due to reasons such as data entry errors, human tampering, and mechanical failures, resulting in missing data values, anomalies, lost or redundant data attributes. These problems may seriously harm the subsequent use of the data. Therefore, it is of great significance to know how to reasonably detect abnormal data.

[0012] Isolation forest is a commonly used anomaly detection algorithm, characterized by high execution efficiency and good detection accuracy. In traditional isolation forest algorithms, the average path length of a data sample within the isolation forest is calculated when identifying outliers, ignoring the differences in anomaly detection capabilities between individual isolated trees (also known as isolated binary trees). During the training phase, the isolation forest algorithm randomly selects features to partition the data to be detected, with each feature having an equal probability of being selected. However, in reality, different features have varying degrees of influence on whether a data point is an anomaly. Furthermore, anomaly detection based on isolation forests typically requires building a large number of isolated trees on large datasets, which usually consumes significant computational resources, resulting in long anomaly detection times.

[0013] To overcome the problems existing in related technologies and improve the accuracy of anomaly detection, embodiments of this application provide an anomaly detection method, electronic device, medium, and computer program product. The anomaly detection method provided in this application can improve the accuracy of anomaly detection by assigning a first weight associated with the first path length to each isolated tree, thereby achieving differentiated processing of the detection results for each isolated tree.

[0014] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0015] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other related elements in the method or apparatus that includes that element (e.g., steps in the method or units / modules in the apparatus; for example, units / modules in the apparatus may be portions of circuitry, processors, programs, or software, etc.).

[0016] The anomaly detection method provided in this application includes a series of steps, but the anomaly detection method provided in this application is not limited to the steps described. Similarly, the anomaly detection device provided in this application includes a series of modules, but the device provided in this application is not limited to the modules explicitly described, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0017] Figure 1 A flowchart of an anomaly detection method is shown. Figure 1 The anomaly detection methods shown include: Step 101: Identify at least two isolated trees corresponding to the dataset to be detected.

[0018] An isolation tree is a binary tree structure used for anomaly detection. It isolates data points by randomly selecting features and split values ​​from the data to be detected. The shorter the path length of the isolation tree, i.e., the first path length in step 102, the more likely the data point is to be an anomaly. The core idea of ​​an isolation tree is to recursively partition the data space, making it easier to separate out anomalies. The first path length represents the number of edges traversed by a data point (the data to be detected) from the root node to a leaf node.

[0019] To improve the reliability of anomaly detection, at least two isolated trees corresponding to the dataset to be detected can be generated. These at least two isolated trees together constitute an isolated forest model. Each isolated tree independently isolates the data to be detected or a portion of the data to be detected in the dataset. Each isolated tree typically includes one or more first path lengths. When performing anomaly detection on multiple datasets, each isolated tree typically includes at least two path lengths. The first path length in each isolated tree can be recorded to determine the anomaly detection result of each isolated tree. By combining the anomaly detection results of all isolated trees, the anomaly detection result for each dataset to be detected can be obtained, such as the first anomaly detection result in step 104.

[0020] Step 102: Determine the first weight of each isolated tree based on the first path length in each of the at least two isolated trees.

[0021] The first path length refers to the number of edges traversed from the root node to a leaf node of a given piece of data during the construction of an isolation tree. It can be seen that each isolation tree corresponding to the dataset to be detected can contain at least two first path lengths. The first path length reflects the ease with which a piece of data to be detected is isolated within the isolation tree; the shorter the first path length, the more likely the corresponding data point is to be an outlier.

[0022] Taking any one of the at least two isolation trees as an example, after the first isolation tree is constructed, the average path length of all the data to be detected in the first isolation tree can be calculated based on the first path length of each first isolation tree. This average path length can be used as an indicator to measure the overall performance of the first isolation tree, and the first weight of the first isolation tree can be determined based on the average path length of the first isolation tree.

[0023] Because different isolation trees differ in their anomaly detection capabilities, some isolation trees may be better at identifying certain types of anomalies, while others may be weaker. Therefore, the first weight of each isolation tree can be determined by calculating the standard deviation of the first path length in each isolation tree.

[0024] Step 103: Determine the first anomaly score of the first data to be detected by using the second path length of the first data to be detected in each isolated tree and the first weight of each isolated tree.

[0025] The first data to be detected is any data point in the dataset to be detected.

[0026] The second path length reflects how quickly the first data to be detected is isolated in each isolated tree, indirectly reflecting the possibility that the first data to be detected is abnormal.

[0027] The first anomaly score is calculated by comprehensively considering the second path length of each isolated tree and its corresponding first weight. For example, the first anomaly score can be determined by a weighted sum of the second path length and the first weight of each isolated tree. This can be achieved by multiplying the second path length of each isolated tree by its first weight, and then averaging the results. A higher first anomaly score indicates that the first data point to be detected is more likely to be an anomaly.

[0028] In practical applications, a subset of the dataset to be detected is usually randomly selected to construct the isolated trees in the isolated forest. Therefore, there may be one or more isolated trees that do not contain the first data to be detected, meaning that the first data to be detected was not randomly selected to construct some isolated trees. In this case, the second path length of the isolated tree that does not contain the first data to be detected can be set to 0 or set to the average path length of the isolated forest.

[0029] In practical applications, by weighting the second path length of each isolated tree, the impact of noisy data on anomaly detection results can be effectively reduced, improving the accuracy and stability of anomaly detection. Furthermore, the first weight of each isolated tree can be dynamically adjusted based on its performance, giving the isolated forest algorithm good adaptability and generalization ability.

[0030] Step 104: Determine the first anomaly detection result of the first data to be detected based on the first anomaly score.

[0031] The first anomaly detection result can be determined by the first anomaly score. For example, a first threshold can be set; if the first anomaly score is greater than the first threshold, the first data to be detected is determined to be anomaly-free; if the first anomaly score is less than or equal to the first threshold, the first data to be detected is determined to be non-anomaly-free. Alternatively, the first anomaly scores of each data to be detected in the dataset can be sorted in descending order, and the data to be detected that are ranked before a sorting threshold are considered anomaly-free.

[0032] In summary, the anomaly detection method provided in this application first constructs at least two isolated trees to form an isolated forest from the data to be detected, ensuring good generalization ability of anomaly detection; secondly, it determines the first weight of each isolated tree by analyzing the first path length of each isolated tree, which helps to reflect the differences in anomaly detection capabilities of each isolated tree and realize differentiated processing for different isolated trees; finally, it determines the first anomaly score by combining the second path length of the first data to be detected and the first weight of each isolated tree, and judges whether the first data to be detected is an anomaly based on the first anomaly score, which not only ensures the robustness of anomaly detection but also improves the accuracy of anomaly detection.

[0033] In practical applications, steps 101 to 104 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor.

[0034] In order for the isolation tree to accurately separate outlier data, in some embodiments, the above-mentioned determination of at least two isolation trees corresponding to the dataset to be detected includes: determining the information entropy of each of the at least two features corresponding to the dataset to be detected; determining a second weight of each feature based on the information entropy of each feature; wherein; the second weight of the first feature is negatively correlated with the information entropy of the first feature; the first feature is any one of the at least two features; and determining at least two isolation trees corresponding to the dataset to be detected through the at least two features and the second weight of each feature.

[0035] Each piece of data to be detected in the dataset can include at least two features. For example, data related to computational performance could include CPU utilization, memory usage, etc. For each feature, the information entropy can be used to quantify its uncertainty or randomness. A smaller information entropy indicates a greater amount of information carried by the feature in the dataset. The second weight of each feature can be determined based on its information entropy.

[0036] In practical applications, the information entropy of each feature can be calculated using the entropy weighting method, and then converted into a corresponding second weight. For example, in the dataset to be detected, if the information entropy of the temperature feature is the smallest, it indicates that the temperature feature is most effective in distinguishing between normal and abnormal data. Therefore, a higher second weight can be assigned to the temperature feature, making it account for a larger proportion of the final anomaly score.

[0037] Isolation trees can be generated based on each feature and its corresponding second weight. By quantifying the importance of features, the generated isolation trees become more targeted, thereby improving the overall performance of anomaly detection.

[0038] In practical applications, the construction process of the isolation tree can be dynamically adjusted based on the second weight of the features. For example, for features with high second weights, these features can be prioritized as splitting nodes when constructing the isolation tree to ensure that they play a dominant role in anomaly detection. Alternatively, features with second weights less than a certain threshold can be deleted based on their second weights to obtain updated features, and the isolation tree can then be constructed based on these updated features.

[0039] Alternatively, an isolation tree can be constructed by randomly selecting features from at least two features based on the second weight of each feature. For example, in some embodiments, determining at least two isolation trees corresponding to the dataset to be detected using at least two features and the second weight of each feature includes: randomly selecting a target feature from at least two features based on the second weight of each feature; wherein the probability of the target feature being selected is positively correlated with the second weight of the target feature; and determining at least two isolation trees corresponding to the dataset to be detected based on the target feature.

[0040] Currently, in the process of constructing an isolation tree, features are usually randomly selected from at least two features. In this embodiment, in order to more effectively distinguish abnormal data based on the selected features in the constructed isolation tree, the second weight of each feature can be used as the probability basis for feature selection when constructing the isolation tree, thereby more effectively identifying features that have a significant impact on anomaly detection.

[0041] Specifically, a larger second weight indicates a smaller information entropy for the feature, resulting in a stronger impact on anomaly detection. Therefore, it increases the probability that features with higher second weights will be selected to build an isolation tree. Based on the second weight of each feature, the probability of each feature being selected is determined. Based on the probability of each feature being selected, a target feature is chosen from at least two features, making features with higher second weights more likely to be selected as target features to build an isolation tree.

[0042] For example, in a dataset to be detected that includes features such as time, location, and speed, if the second weight of the location feature is high, the location feature will be used more frequently to partition the dataset when building the isolation tree, thereby improving the ability to identify abnormal data.

[0043] By determining the target features using a second weight, at least two isolated trees can be efficiently constructed, ultimately forming an isolated forest. Since each isolated tree is generated based on a different combination of features, the entire isolated forest can cover more potential anomaly patterns, improving the overall performance of anomaly detection.

[0044] For example, when constructing the first isolation tree, the first target feature and the segmentation value of the first target feature can be randomly selected from at least two features to divide the dataset to be detected (or the second subset of the dataset to be detected) into two subsets, left and right. Then the algorithm recursively executes the above operation until each subset has only one data point or reaches the preset maximum depth of the isolation tree.

[0045] In summary, this embodiment optimizes the construction process of the isolation tree by employing a feature-based random selection strategy with second weights. By adopting this strategy, abnormal data in the dataset to be detected can be captured more accurately, thereby improving the accuracy of anomaly detection and providing stronger support for real-time monitoring and risk warning in practical applications.

[0046] Based on the method for constructing an isolation tree given in the above embodiments, for example, suppose the dataset to be detected includes n data points to be detected, and the dataset to be detected corresponds to k features. Where, for the i-th feature, the attribute values ​​corresponding to the n data to be detected are Normalize the attribute value of the i-th feature for the j-th data to be detected: (1) (2) in, This represents the attribute value of the j-th data to be detected in the i-th feature. This represents the minimum value of the attribute of the i-th feature among all the data to be detected. This represents the maximum value of the attribute of the i-th feature in all the data to be detected. If the attribute value of the i-th feature in the dataset to be detected is negative, normalization can be performed based on formula (2); otherwise, normalization can be performed using formula (1).

[0047] Then, the information entropy of the i-th feature can be calculated based on formula (3): (3) in, ,when At that time, .

[0048] The second weight of the i-th feature can be determined based on formula (4). : (4) Where k represents the total number of features, Let g represent the information entropy of the g-th feature.

[0049] After obtaining the second weight of the features based on the above formula, probability sampling can be performed based on the proportion of the second weight of each feature. That is, the larger the second weight, the higher the probability that the feature will be selected as the target feature. For example, suppose there are two features: feature one is daily water consumption with a corresponding second weight of 0.65, and feature two is daily water consumption change with a corresponding second weight of 0.35. Then, feature one has a 65% probability of being selected as the target feature, and feature two has a 35% probability of being selected as the target feature. When splitting nodes in the isolated tree, "daily water consumption" can be selected as the target feature with a 65% probability, and "daily water consumption change" can be selected as the target feature with a 35% probability. This ensures that features more important for anomaly detection have a higher probability of being selected for data splitting.

[0050] Because in the Isolation Forest algorithm, the shorter the path length, the more likely the corresponding data to be detected is abnormal, and the larger the second weight of the feature corresponding to a node in an isolated tree, the shorter its corresponding path length. To improve the efficiency of determining the first path length, in some embodiments, before determining the first weight of each isolated tree based on the first path length in each of at least two isolated trees, the method further includes: determining the information entropy of each of the at least two features corresponding to the dataset to be detected; determining the second weight of each feature based on the information entropy of each feature; wherein the second weight of the first feature is negatively correlated with the information entropy of the first feature; the first feature is any one of the at least two features; and determining the first path length in each isolated tree based on the second weight of each feature; wherein the first path length is negatively correlated with the second weight of the feature corresponding to the first path length.

[0051] The first path length refers to the path length required for the data to be detected to reach a leaf node in each isolated tree. In the traditional isolated forest algorithm, the path length depends only on randomly selected features and split values. However, in this embodiment, a second weight is introduced as an adjustment factor, making features with high weights more likely to be used to split data at shorter path positions, thereby improving the accuracy of anomaly detection.

[0052] For example, in a network traffic monitoring system, if a certain feature (such as packet size) has a high second weight, then during the process of building an isolation tree in the network traffic monitoring system, this feature will be used first to divide the data space, so that data with abnormal behavior can be more easily and quickly isolated, thereby shortening the second path length of the data to be detected in the isolation tree, shortening the first path length in the isolation tree, and improving the efficiency of anomaly detection.

[0053] Based on the method given in this embodiment, and combined with the above formula, the first path length of the i-th leaf node can be determined based on formula (5): (5) in, Indicates the first The second weight of the features of each leaf node. hour The second weight representing the feature of the i-th leaf node can be determined based on formula (4). It can be seen that the first path length corresponding to any leaf node in the isolated tree, such as the first leaf node, can be determined based on the second weight of the feature of the first leaf node and the second weight of the feature used to construct the isolated tree before the feature of the first leaf node.

[0054] The method for determining the second weight of features based on information entropy in this embodiment can be implemented with reference to the above embodiment, and will not be described again in this embodiment.

[0055] In summary, in this embodiment, by introducing the information entropy and second weight of features when constructing the isolation tree, the construction process of the isolation tree can be optimized. This method can improve the accuracy of anomaly detection and meet the needs of efficient anomaly detection in large-scale data environments.

[0056] After determining the first path length of each leaf node in each isolated tree, taking the first isolated tree as an example, in some embodiments, the above-mentioned determination of the first weight of each isolated tree based on the first path length in each of at least two isolated trees includes: determining the first degree of dispersion corresponding to the first isolated tree based on the first path length of each leaf node in the first isolated tree; wherein, the first isolated tree is any one of at least two isolated trees; determining the isolated tree weight of the first isolated tree based on the first degree of dispersion; and determining the first weight of each isolated tree through the isolated tree weight of the first isolated tree.

[0057] Leaf nodes represent nodes in an isolated tree that have no more child nodes, typically indicating that the data to be detected has been completely isolated. The shorter the first path length, the easier it is to isolate the leaf node, and the closer it is to the abnormal data.

[0058] The first degree of dispersion can be obtained by calculating the standard deviation of the first path lengths of all leaf nodes in the first isolated tree. This first degree of dispersion reflects the distribution of the first path lengths of each leaf node in the first isolated tree. When the standard deviation is large, it indicates that the differences between the various first path lengths are large, which may indicate that the first isolated tree has a strong ability to distinguish the data; conversely, when the standard deviation is small, it may indicate that the first isolated tree has a weak ability to distinguish the data.

[0059] In practical applications, such as in an isolated forest model, each isolated tree partitions the dataset to be detected based on the target features and generates several leaf nodes. By calculating the first path length of these leaf nodes and the first degree of dispersion, the isolation effect of the first isolated tree on the data can be evaluated, and this isolation effect can be used as the basis for subsequent weighted calculations.

[0060] The isolation tree weight of the first isolated tree is a quantitative indicator of its contribution to the entire isolated forest model. The isolation tree weight can be proportional to the first degree of dispersion; that is, the greater the first degree of dispersion, the greater the contribution of the first isolated tree to anomaly detection, and therefore the higher its isolation tree weight. Based on the isolation tree weight of the first isolated tree, the isolation tree weight of each isolated tree can be determined. From the isolation tree weight of each isolated tree, the first weight of each isolated tree in at least two isolated trees can be obtained. Specifically, the isolation tree weight of the first isolated tree can be used as the first weight corresponding to the first isolated tree in at least two isolated trees.

[0061] For example, if the first isolation tree has a significantly higher degree of dispersion than other isolation trees, it indicates that the first isolation tree performs well in isolating outlier data. Therefore, the first isolation tree can be given a higher isolation tree weight, so that the first isolation tree can play a greater role in the final outlier score calculation.

[0062] For example, suppose the set of leaf nodes of the first isolated tree is The set of first path lengths of the leaf nodes of the first isolated tree is Then the standard deviation of the first path length of each leaf node in the first isolated tree can be determined based on formula (6): (6) Where t represents the total number of leaf nodes, This represents the mean of the first path lengths for all leaf nodes in the first isolated tree. This represents the first path length of the i-th leaf node.

[0063] Then, the set of discretenesses of the first path lengths of all isolated trees in the isolated forest is obtained as follows: The maximum value in the set of discreteness is The minimum value is The standard deviation in formula (6) can be normalized to obtain the normalized result as shown in formula (7): (7) Based on the above formula, assume that the set of second path lengths of the data to be detected x in each isolated tree in the dataset to be detected is: The first weight set of at least two isolated trees is Where s represents the total number of isolated trees, the first weight can be determined based on formula (7). Then the first anomaly score corresponding to the data to be detected x can be determined based on formula (8): (8) (9) Where m represents the amount of data in the second subset of the dataset to be detected, which contains the data to be detected x. Let m be the average length of the second path of the m data to be detected. , is Euler's constant. This represents the first weight of the i-th isolated tree. This represents the second path length of the i-th isolated tree of the data to be detected, x.

[0064] In this embodiment, by introducing the dispersion of the first path length as the basis for calculating the first weight of the isolated tree, the contribution of each isolated tree to anomaly detection can be evaluated more effectively, thereby optimizing the overall performance of the isolated forest model and improving the accuracy and real-time performance of data anomaly detection.

[0065] Based on the method given in the above embodiments, Figure 2 A flowchart of an isolated tree construction method is shown, including: Step 201: Begin.

[0066] Step 202: Obtain the dataset to be detected.

[0067] Step 203: Randomly select m data points from the dataset to be detected and place them into the root node of an isolated tree.

[0068] For example, from a dataset containing n data points to be detected and k features... In the process, m data points are randomly selected to form the second subset of the dataset. Based on the second subset of datasets Construct an isolated tree.

[0069] Step 204: Calculate the second weight set of the features using the entropy weight method.

[0070] The second weight of each feature is determined based on information entropy, and the second subset of data is calculated. The corresponding second weight set.

[0071] Step 205: Select features based on the second weight set of the features, and randomly generate a segmentation value p within the range of the attribute values ​​of the current node.

[0072] Based on the second weight of each feature, select the target feature from the k features, for example, obtain the target feature q, and based on the range of attribute values ​​of the target feature q, randomly generate a segmentation value p within the range of attribute values.

[0073] Step 206: Place the data to be detected that are less than p under the currently selected feature into the left branch of the current node, and place the data to be detected that are greater than or equal to p into the right branch of the current node.

[0074] The second subset can be obtained using feature q and attribute segmentation value p. The data is divided into two sub-data spaces. Data to be detected with attribute values ​​less than p in feature q is placed in the left branch of the current node, and data to be detected with attribute values ​​greater than or equal to p is placed in the right branch of the current node.

[0075] Step 207: Determine whether there is only one data point on the left or right leaf node or whether the isolated tree has grown to the preset height.

[0076] If there is only one data (the data to be detected) in the left or right leaf node or the isolated tree has grown to the preset height, proceed to step 208; otherwise, proceed to step 205 to continue building the isolated tree.

[0077] Step 208: Construction of the first isolated tree is complete.

[0078] Step 209: Calculate the standard deviation of the first path length.

[0079] The standard deviation of the first path length of each leaf node in the first isolated tree can be calculated based on the above formula (6).

[0080] Step 210: Determine whether the set number of isolated trees t has been reached.

[0081] If the number of isolated trees constructed reaches the set number t, proceed to step 211; otherwise, proceed to step 203, randomly select the second subset of data, and construct isolated trees again.

[0082] Step 211: All isolated trees are now constructed.

[0083] Step 212: End.

[0084] Based on the method given in the above embodiments, Figure 3 A flowchart of a method for calculating a first anomaly score is shown, including: Step 301: Begin.

[0085] Step 302: Determine the data to be tested.

[0086] Step 303: Construct the isolation tree, determine the second path length and the first weight of each isolation tree.

[0087] like Figure 3 As shown, combined with Figure 2 The isolated tree construction method shown constructs s ​​isolated trees for the data to be detected. Based on the method given in the above embodiment, the second path length of the data to be detected in each isolated tree and the first weight of each isolated tree are determined.

[0088] Step 304: Determine the first abnormal score.

[0089] The first anomaly score of the data to be detected is determined based on the second path length of the data in each isolated tree and the first weight of each isolated tree.

[0090] Step 305: End.

[0091] Based on the anomaly detection using the improved random forest algorithm described in the above embodiments, in order to further improve the anomaly detection efficiency, in some embodiments, the method further includes: determining a first subset of the dataset to be detected; determining a second anomaly score of the first dataset to be detected based on the average value of the first subset; determining a second anomaly detection result of the first dataset to be detected based on the second anomaly score; determining a third anomaly detection result of the first dataset to be detected based on either the first anomaly detection result or the second anomaly detection result; or, determining a third anomaly detection result of the first dataset to be detected based on the first anomaly detection result and the second anomaly detection result.

[0092] By extracting a first subset from the original dataset to be detected, a more refined second anomaly score can be calculated. The selection of the first subset can be defined based on factors such as time windows, spatial proximity, or feature similarity. For example, in time series data analysis, the first subset can be constructed by selecting all data within a fixed-length time window centered on the first data to be detected, or it can be constructed based on all data within a preset time window preceding the first data to be detected. This approach helps improve sensitivity to local anomalies and reduces the impact of global statistical bias. In practical applications, at least two first subsets can be obtained, and the second anomaly score of the first data to be detected can be determined by averaging the values ​​of each of the at least two first subsets, thus improving the accuracy of the second anomaly score. Specifically, the first subset can be a dataset that includes the first data to be detected, or it can be a dataset that is geographically close to the first data to be detected.

[0093] By limiting the scope of the first subset of data, it is possible to focus on local regions closely related to the first dataset to be detected, thereby improving the accuracy of anomaly detection. For example, in some embodiments, determining the first subset of data in the dataset to be detected includes: determining the first subset of data in the dataset to be detected based on a sliding window.

[0094] A sliding window, by setting a fixed-size window, slides progressively across the data stream (the dataset to be detected), processing the data within the window at each step. The sliding window dynamically captures the changing trends and local features of the data to be detected. In this embodiment, the sliding window is used to extract consecutive time window segments as the first subset of the dataset to be detected. The size of the sliding window can be adjusted according to the actual application scenario; for example, it can be set to contain the 10 most recent consecutive subsets of data to be detected, moving forward one data point at a time. Using the sliding window method can divide a large-scale dataset into multiple manageable small subsets without losing time-series information, thereby reducing computational complexity and improving detection efficiency.

[0095] After selecting the first subset of data, the deviation of the first data to be detected from the normal distribution can be evaluated based on the average value of each data to be detected in the first subset. Specifically, the average value of all data to be detected in the first subset is first calculated, and then the first data to be detected is compared with the average value of the first subset. If the difference exceeds a second threshold, the second anomaly score of the first data to be detected can be determined based on the magnitude of the difference.

[0096] After determining the score of the second anomaly, the detection result of the second anomaly can be determined based on the score of the second anomaly.

[0097] For example, a sliding window can be used to calculate the average value of the data to be detected within the sliding window. If the absolute value of the difference between the first data to be detected and the average value is greater than a second threshold, a second anomaly score is obtained. This detection method is low-cost, fast, and has high real-time performance. For example, for the time series of data j to be detected... The detection function can be determined as follows: (10) in, (11) (12) in, Indicates that the data to be detected j is in The value at time, This represents the value of the data j to be detected at time r. Indicates that the data to be detected j is in The value at time, Indicates that the data to be detected j is in The value at time t, where b represents the size of the sliding window used to calculate the mean and variance, and can be the number of data points to be tested in the first subset of the dataset. Indicates the sliding window is in The mean within the range, i.e., the mean of the first subset of data. The standard deviation of the calculated sliding window can be represented by the second threshold mentioned above; based on The second anomaly score can be obtained. In formula (12) It can be determined based on formula (11).

[0098] Introducing a second anomaly detection result can enhance the efficiency and stability of the detection results.

[0099] In this embodiment, the results of the sliding window detection and the improved isolated forest algorithm can be fused to comprehensively determine the abnormal state of the first data to be detected. For example, if both methods consider the first data to be detected to be abnormal, or if the abnormality score of one method is higher than the abnormality threshold, then the first data to be detected is marked as a high-risk anomaly.

[0100] By fusing multiple anomaly scores, detection accuracy can be improved while maintaining detection speed, avoiding the potential for missed or false detections that may occur when using a single method. When determining the third anomaly detection result based on the first anomaly detection result, the first anomaly detection result can be directly used as the third anomaly detection result; that is, if the first anomaly score is greater than the first threshold, the third anomaly detection result indicates that the first data to be detected is abnormal. Similarly, when determining the third anomaly detection result based on the second anomaly detection result, the second anomaly detection result can be directly used as the third anomaly detection result; that is, if the second anomaly score is greater than the second threshold, the third anomaly detection result indicates that the first data to be detected is abnormal. When the third anomaly detection result is jointly determined based on the first and second anomaly detection results, the third anomaly detection result can be determined as indicating that the first data to be detected is abnormal if the first anomaly score is greater than the third threshold and the second anomaly score is greater than the fourth threshold.

[0101] To further improve anomaly detection efficiency, after obtaining a second anomaly score using the averaging method of at least two sliding windows, an improved isolated forest method can be used to obtain a first anomaly score. Then, the anomaly scores from both methods can be combined to determine the third anomaly detection result. (13) Based on formula (13), it can be seen that if both methods consider the first data to be detected to be potentially abnormal, or if one method considers the probability of the first data to be detected to be abnormal to be very high, then the fusion method marks the first data to be detected as abnormal. Through numerous experiments and continuous parameter adjustments, when the abnormal score of one of the anomaly detection algorithms is greater than 0.8, or the abnormal scores of both anomaly detection algorithms are greater than 0.5, the first data to be detected can be set to an abnormal state. In this case, the accuracy of anomaly detection is the highest. In practical applications, the above thresholds can also be adjusted according to the actual application scenario.

[0102] Based on the method given in the above embodiments, Figure 4 A flowchart of another anomaly detection method is shown, including: Step 401: Obtain the dataset to be detected.

[0103] Then, steps 402 and 404 can be executed simultaneously.

[0104] Step 402: Determine the average value based on the sliding window.

[0105] Based on the method described in the above embodiments, a first subset of data can be selected based on a sliding window, and the average value of the data to be detected within the first subset of data can be determined based on the first subset of data.

[0106] Step 403: Obtain the second anomaly score.

[0107] Based on the method described in the above embodiments, a second anomaly score can be obtained based on the difference between the average value determined by the first subset of data and the first data to be detected.

[0108] Then proceed to step 406.

[0109] Step 404: Construct an isolated tree based on the improved isolated forest algorithm.

[0110] Based on the method described in the above embodiments, a second subset of data can be determined from the data to be detected, and at least two isolated trees can be constructed based on the second subset of data to obtain an isolated forest.

[0111] Step 405: Obtain the first anomaly score.

[0112] The second path length of the first data to be detected in each isolated tree is calculated, and the first anomaly score of the first data to be detected can be calculated using formula (8).

[0113] Step 406: Perform anomaly detection based on the first anomaly score and the second anomaly score.

[0114] A first anomaly detection result can be obtained based on a first anomaly score, a second anomaly detection result can be obtained based on a second anomaly score, and a third anomaly detection result can be obtained based on either the first or second anomaly detection result; alternatively, a third anomaly detection result can be obtained based on both the first and second anomaly detection results. Specifically, the anomaly scores of all data to be detected (such as the sum of the first and second anomaly scores) can be sorted, and the anomaly data can be determined based on the sorting results.

[0115] based on Figure 4 The method shown allows for anomaly detection using an improved isolated forest algorithm after calculating the average of at least two sliding window data points. Finally, the anomaly scores from both methods are combined to determine the anomaly status of the current data to be detected.

[0116] This application provides an anomaly detection method that combines a sliding window average anomaly detection method (used multiple times) with a weighted isolated forest anomaly detection method (used in small combinations) using a fusion model-based computational framework. This solves the problem of high computational latency in anomaly detection. By using a hybrid model-based computational framework to combine the sliding window average anomaly detection method with the weighted isolated forest anomaly detection method (used in small combinations), low time cost is maintained while maintaining high accuracy. The anomaly detection method provided in this application improves the overall accuracy and effectiveness of anomaly detection through an improved isolated forest algorithm. By using an improved isolated forest algorithm weighted by feature weights and path lengths to detect anomalies, it can intelligently detect abnormal data in the system, significantly improving detection accuracy.

[0117] Those skilled in the art will understand that the order in which the steps are written in the above-described method of the specific implementation does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0118] Based on the anomaly detection method proposed in the foregoing embodiments, this application also provides an anomaly detection device. Figure 5 A schematic diagram of an anomaly detection device is shown, such as... Figure 5 As shown, the anomaly detection device includes: The first processing module 501 is used to determine at least two isolated trees corresponding to the dataset to be detected; and to determine the first weight of each isolated tree based on the first path length in each of the at least two isolated trees.

[0119] The second processing module 502 is used to determine the first anomaly score of the first data to be detected by using the second path length of the first data to be detected in each isolated tree and the first weight of each isolated tree; the first data to be detected is any data in the dataset to be detected; and the first anomaly detection result of the first data to be detected is determined based on the first anomaly score.

[0120] In practical applications, the first processing module 501 and the second processing module 502 can be implemented based on a processor and a communication device.

[0121] In some embodiments, the first processing module 501 is specifically configured to: determine the information entropy of each of the at least two features corresponding to the dataset to be detected; determine the second weight of each feature based on the information entropy of each feature; wherein the second weight of the first feature is negatively correlated with the information entropy of the first feature; the first feature is any one of the at least two features; and determine at least two isolated trees corresponding to the dataset to be detected through the at least two features and the second weight of each feature.

[0122] In some embodiments, the first processing module 501 is specifically used to: randomly select a target feature from at least two features based on the second weight of each feature; wherein the probability of the target feature being selected is positively correlated with the second weight of the target feature; and determine at least two isolated trees corresponding to the dataset to be detected based on the target feature.

[0123] In some embodiments, before determining the first weight of each isolated tree based on the first path length in each of the at least two isolated trees, the first processing module 501 is further configured to determine the information entropy of each of the at least two features corresponding to the dataset to be detected; determine the second weight of each feature based on the information entropy of each feature; wherein the second weight of the first feature is negatively correlated with the information entropy of the first feature; the first feature is any one of the at least two features; and determine the first path length in each isolated tree based on the second weight of each feature; wherein the first path length is negatively correlated with the second weight of the feature corresponding to the first path length.

[0124] In some embodiments, the first processing module 501 is specifically configured to: determine a first degree of dispersion corresponding to the first isolated tree based on the first path length of each leaf node in the first isolated tree; wherein the first isolated tree is any one of at least two isolated trees; determine the isolated tree weight of the first isolated tree based on the first degree of dispersion; and determine the first weight of each isolated tree through the isolated tree weight of the first isolated tree.

[0125] In some embodiments, the second processing module 502 is further configured to: determine a first subset of the dataset to be detected; determine a second anomaly score of the first dataset to be detected based on the average value of the first subset; determine a second anomaly detection result of the first dataset to be detected based on the second anomaly score; determine a third anomaly detection result of the first dataset to be detected based on either the first anomaly detection result or the second anomaly detection result; or determine a third anomaly detection result of the first dataset to be detected based on the first anomaly detection result and the second anomaly detection result.

[0126] In some embodiments, the second processing module 502 is specifically used to determine a first subset of the dataset to be detected based on a sliding window.

[0127] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0128] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0129] This application also provides an electronic device. Figure 6 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application, as shown below. Figure 6 As shown, the electronic device 60 may include: Memory 601 is used to store executable instructions.

[0130] The processor 602 is used to implement any of the above-mentioned anomaly detection methods when executing executable instructions stored in the memory 601.

[0131] The processor 602 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0132] The aforementioned computer-readable storage medium or memory 601 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0133] This application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the anomaly detection methods provided in the above embodiments.

[0134] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the anomaly detection methods provided in the above embodiments.

[0135] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0136] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0137] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0138] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0139] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0141] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of protection of this application, and these forms are all within the protection scope of this application.

Claims

1. An anomaly detection method, characterized in that, The method includes: Identify at least two isolated trees corresponding to the dataset to be detected; Based on the first path length in each of the at least two isolated trees, a first weight for each isolated tree is determined; A first anomaly score is determined by the second path length of the first data to be detected in each isolated tree and the first weight of each isolated tree; the first data to be detected is any data in the dataset to be detected. The first anomaly detection result of the first data to be detected is determined based on the first anomaly score.

2. The method according to claim 1, characterized in that, The determination of at least two isolated trees corresponding to the dataset to be detected includes: Determine the information entropy of each feature in at least two features corresponding to the dataset to be detected; Based on the information entropy of each feature, a second weight for each feature is determined; wherein the second weight of the first feature is negatively correlated with the information entropy of the first feature; and the first feature is any one of the at least two features. At least two isolated trees corresponding to the dataset to be detected are determined using the at least two features and the second weight of each feature.

3. The method according to claim 2, characterized in that, The step of determining at least two isolated trees corresponding to the dataset to be detected using the at least two features and the second weight of each feature includes: Based on the second weight of each feature, a target feature is randomly selected from the at least two features; wherein the probability of the target feature being selected is positively correlated with the second weight of the target feature; Based on the target features, at least two isolated trees corresponding to the dataset to be detected are determined.

4. The method according to claim 1, characterized in that, Before determining the first weight of each isolated tree based on the first path length in each of the at least two isolated trees, the method further includes: Determine the information entropy of each feature in at least two features corresponding to the dataset to be detected; Based on the information entropy of each feature, a second weight for each feature is determined; wherein the second weight of the first feature is negatively correlated with the information entropy of the first feature; and the first feature is any one of the at least two features. Based on the second weight of each feature, the first path length in each isolated tree is determined; wherein the first path length is negatively correlated with the second weight of the feature corresponding to the first path length.

5. The method according to claim 1, characterized in that, The determination of the first weight of each isolated tree based on the first path length in each of the at least two isolated trees includes: Based on the first path length of each leaf node in the first isolated tree, the first degree of dispersion corresponding to the first isolated tree is determined; wherein, the first isolated tree is any one of the at least two isolated trees; Based on the first degree of dispersion, the isolation tree weights of the first isolation tree are determined; The first weight of each isolated tree is determined by the isolated tree weight of the first isolated tree.

6. The method according to claim 1, characterized in that, The method further includes: Determine a first subset of the dataset to be detected; The second anomaly score of the first data to be detected is determined based on the average value of the first subset of data. The second anomaly detection result of the first data to be detected is determined based on the second anomaly score. A third anomaly detection result is determined based on either the first anomaly detection result or the second anomaly detection result; or, a third anomaly detection result is determined based on both the first anomaly detection result and the second anomaly detection result.

7. The method according to claim 6, characterized in that, Determining the first subset of the dataset in the dataset to be detected includes: The first subset of the dataset to be detected is determined based on a sliding window.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory for storing computer programs capable of running on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 7.

9. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images