Streaming anomaly detection method, device, equipment and medium based on non-uniform sampling

Abstract

The present invention discloses a streaming anomaly detection method, device, equipment and medium based on non-uniform sampling. The method includes: generating an anomaly detection model corresponding to a model generation rule according to a historical sampling database, and detecting target detection data in real-time detection streaming data through the anomaly detection model to obtain a detection result, judging whether the target detection data can be used to sample and update a target decision tree in the anomaly detection model according to the non-uniform sampling rule, and if so, updating the nodes in the target decision tree according to the detection result of the target detection data to obtain an updated anomaly detection model. The present invention belongs to the field of intelligent decision technology, and can dynamically sample and update anomaly detection models based on non-uniform sampling rules to continuously optimize the nodes of the decision tree in the anomaly detection model, thereby improving the sensitivity of the anomaly detection model to data offset, and can greatly improve the accuracy of detecting anomaly data in massive streaming data.

Description

Technical Field

[0001] The present invention relates to the field of intelligent decision-making technology, and in particular to a streaming anomaly detection method, device, equipment and medium based on non-uniform sampling. Background Art

[0002] With the massive increase in real-time data sources in large enterprises, the corresponding streaming time series data has grown massively. Operation and maintenance personnel usually need to obtain these time series data in a churn manner and determine whether there are any anomalies. Manually judging the anomalies of the acquired massive time series data requires a lot of manpower, and the timeliness of the judgment is poor. It is impossible to timely determine whether there are anomalies for the massive time series data. In the prior art method, anomaly detection models can be used to detect anomalies in massive time series data, but in the prior art method, each data has the same weight in the model, resulting in the model's low sensitivity to data offsets, and it is impossible to accurately detect anomalies. Therefore, the anomaly detection method in the prior art method has the problem of being unable to accurately detect anomalies. Summary of the invention

[0003] The embodiments of the present invention provide a streaming anomaly detection method, apparatus, device and medium based on non-uniform sampling, aiming to solve the problem that the anomaly detection method in the prior art cannot accurately perform anomaly detection.

[0004] In a first aspect, an embodiment of the present invention provides a streaming anomaly detection method based on non-uniform sampling, the method comprising:

[0005] If the input model generation rule is received, an anomaly detection model corresponding to the model generation rule is generated according to a pre-stored historical sampling database;

[0006] Obtaining target detection data corresponding to the current time from the input real-time detection streaming data;

[0007] Performing anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists;

[0008] Randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree;

[0009] Determining whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule;

[0010] If the target detection data can be used to sample and update the target decision tree, the nodes in the target decision tree are updated according to the detection results of the target detection data to obtain an updated anomaly detection model, and the step of obtaining the target detection data corresponding to the current time from the input real-time detection streaming data is returned to execute.

[0011] In a second aspect, an embodiment of the present invention provides a streaming anomaly detection device based on non-uniform sampling, which includes:

[0012] An anomaly detection model generation unit is used to generate an anomaly detection model corresponding to the model generation rule according to a pre-stored historical sampling database if an input model generation rule is received;

[0013] The target detection data acquisition unit is used to acquire the target detection data corresponding to the current time from the input real-time detection streaming data;

[0014] A detection result acquisition unit, used to perform anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists;

[0015] A target decision tree determination unit, used to randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree;

[0016] A sampling update judgment unit, used to judge whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule;

[0017] A model updating unit is used to update the nodes in the target decision tree according to the detection results of the target detection data to obtain an updated anomaly detection model if the target detection data can be used to sample and update the target decision tree, and return to execute the step of obtaining the target detection data corresponding to the current time from the input real-time detection streaming data.

[0018] In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the streaming anomaly detection method based on non-uniform sampling described in the first aspect is implemented.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the streaming anomaly detection method based on non-uniform sampling described in the first aspect above.

[0020] The embodiments of the present invention provide a streaming anomaly detection method, device, equipment and medium based on non-uniform sampling. An anomaly detection model corresponding to the model generation rules is generated according to the historical sampling database, and the target detection data in the real-time detection streaming data is detected by the anomaly detection model to obtain the detection result. According to the non-uniform sampling rule, it is judged whether the target detection data can be used to sample and update the target decision tree in the anomaly detection model. If so, the nodes in the target decision tree are updated according to the detection result of the target detection data to obtain the updated anomaly detection model. Through the above method, the anomaly detection model can be dynamically sampled and updated based on the non-uniform sampling rule to continuously optimize the nodes of the decision tree in the anomaly detection model, thereby improving the sensitivity of the anomaly detection model to data offset, and greatly improving the accuracy of detecting anomaly data in massive streaming data. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a sub-process of a streaming anomaly detection method based on non-uniform sampling provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of another sub-process of the streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of another sub-process of the streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of another sub-process of the streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of another sub-process of the streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of another sub-process of the streaming anomaly detection method based on non-uniform sampling provided by an embodiment of the present invention;

[0029] Figure 8A schematic block diagram of a streaming anomaly detection device based on non-uniform sampling provided by an embodiment of the present invention;

[0030] Fig. 9 A schematic block diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0033] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0034] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0035] See also Figure 1 , Figure 1 A flow chart of a streaming anomaly detection method based on non-uniform sampling provided in an embodiment of the present invention; the streaming anomaly detection method based on non-uniform sampling is applied to a terminal device or a management server, and the streaming anomaly detection method based on non-uniform sampling is executed by an application software installed in the terminal device or the management server. The terminal device is a terminal device used to execute streaming anomaly detection based on non-uniform sampling to realize anomaly detection of massive streaming detection data, such as a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart watch, a smart eye, etc. The terminal device can also be an Internet of Things terminal; the management server is a server end used to execute streaming anomaly detection based on non-uniform sampling to realize anomaly detection of massive streaming detection data from the terminal device, such as a server built within an enterprise or a government department. Figure 1As shown, the method includes steps S110 to S160.

[0036] S110: If the input model generation rule is received, an anomaly detection model corresponding to the model generation rule is generated according to a pre-stored historical sampling database.

[0037] If the input model generation rule is received, an anomaly detection model corresponding to the model generation rule is generated according to the pre-stored historical sampling database. The user can input the model generation rule to the terminal device or the management server. The model generation rule is a specific rule for generating an anomaly detection model. The terminal device or the management server pre-stores a historical sampling database. Then, an anomaly detection model corresponding to the model generation rule can be generated according to the pre-stored historical sampling database. The historical sampling database contains multiple sampling data collected at different times. The sampling data may contain one variable or multiple variables. Then, the method in this embodiment can be applied to anomaly detection of single-variable streaming data or multi-variable streaming data.

[0038] For example, when performing anomaly detection on the performance of a terminal device, each piece of sampling data in the historical sampling database may contain multiple variables (multiple indicators) such as CPU utilization, memory occupancy, video memory occupancy, CPU temperature, data read / write rate (I / O rate), etc. In the subsequent anomaly detection process, it is necessary to combine multiple variables to perform multi-indicator comprehensive anomaly detection.

[0039] In one embodiment, if Figure 2 As shown, step S110 includes sub-steps S111, S112 and S113.

[0040] S111 . Filtering target sampling data satisfying the time span from the historical sampling database according to the time span in the model generation rule.

[0041] Each sampling data in the historical sampling database corresponds to a timestamp, which is the specific time information when the sampling data is sampled. The model generation rules include the time span. According to the time span and the timestamp of each sampling data, the target sampling data that meets the time span can be filtered out from the historical sampling database. The time span contains at least one span value.

[0042] For example, the span value included in the time span may be 1 day, 3 days or 7 days (a week). If a time span includes a span value of 3 days, sampling data with a timestamp within 3 days can be obtained from the historical sampling data based on the current time point as the target sampling data.

[0043] S112. Extract corresponding sampling data from the target sampling data according to the configuration quantity of the model generation rule to obtain sampling data partitions.

[0044] The model generation rule also includes the configuration quantity, wherein the configuration quantity includes the decision tree configuration quantity and the node configuration quantity. The configuration quantity is the quantity information for configuring the unit structure contained in the generated model. The anomaly detection model generated in this embodiment is an intelligent detection model based on the random forest algorithm. The decision tree configuration quantity in the configuration quantity can be used to configure the number of decision trees contained in the anomaly detection model, and the node configuration quantity can be used to configure the number of nodes contained in each decision tree. The sample data partition can be extracted from the target sample data through the configuration example. Each sample data partition contains multiple sample data, and each sample data partition can be used to build a decision tree.

[0045] In one embodiment, if Figure 3 As shown, step S112 includes sub-steps S1121 and S1122.

[0046] S1121. Extract a number of sampling data combinations equal to the number of node configurations from the target sampling data to obtain a sampling data partition.

[0047] In the specific implementation process, sampling data equal to the number of node configurations can be extracted from the target sampling data and combined to obtain a corresponding sampling data partition. In the specific implementation process, the sampling data partition can be obtained by random extraction with replacement, that is, a plurality of sampling data equal to the number of node configurations are randomly extracted from the target sampling data and combined into a sampling data partition, and then the extracted plurality of sampling data are replaced and the process of extracting sampling data from the target sampling data is repeated again.

[0048] In one embodiment, if Figure 4 As shown, step S1121 includes sub-steps S1123, S1124, S1125 and S1126.

[0049] In the specific implementation process, a sorting extraction method with replacement can also be used to obtain the sampling data partitions.

[0050] S1123. Randomly generate a random coefficient corresponding to each of the target sampling data.

[0051] Each time when obtaining the sampling data partition, it is necessary to randomly generate a random coefficient corresponding to each target sampling data. Then each target sampling data corresponds to a random coefficient. The random coefficient can be represented by ui, and the value range of ui is (0,1).

[0052] S1124. Calculate the sampling data scoring value corresponding to each target sampling data according to the scoring value calculation formula and the random coefficient.

[0053] According to the scoring value calculation formula and the random coefficient of each target sampling data, the sampling data scoring value corresponding to each target sampling coefficient can be calculated respectively. Then each target sampling data corresponds to a sampling data scoring value. Specifically, the scoring value calculation formula can be expressed by formula (1).

[0054]

[0055] Wherein, e is a natural constant, α is a parameter value preset in the formula, and Δt is the time difference between the timestamp of the target sampling data and the current time.

[0056] S1125, sorting the target sampling data according to the sampling data scoring values ​​to obtain a data sorting result; S1126, extracting a sampling data combination that is ranked first and has the same number as the node configuration according to the data sorting result to obtain a sampling data partition.

[0057] The target sampling data are sorted according to the sampling data score value of each target sampling data. The target sampling data with a larger sampling data score value is sorted higher. The sampling data with a higher sorting value and equal to the number of node configurations can be extracted according to the data sorting result and combined to obtain a corresponding sampling data partition. The number of sampling data contained in each sampling data partition is equal to the number of node configurations.

[0058] By repeating steps S1123 to S1126, multiple sampling data partitions can be obtained respectively. Since the random coefficients corresponding to the target sampling data are randomly generated each time the sampling data partitions are obtained, the random coefficients of the same target sampling data also change each time the sampling data partitions are obtained, resulting in differences in the sampling data contained in the sampling data partitions.

[0059] S1122. Repeat extracting the target sampled data until the number of acquired sampled data partitions is equal to the number of decision tree configurations.

[0060] Repeat step S111 to repeatedly extract sample data partitions from the target sample data until sample data partitions equal to the number of decision tree configurations are obtained, that is, the number of times step S111 is repeatedly executed is equal to the number of decision tree configurations.

[0061] S113. Construct a decision tree corresponding to each sampled data partition according to the classification label of each sampled data in the sampled data partition, so as to generate an anomaly detection model including the decision tree.

[0062] Each sampled data in the sampled data partition also includes a classification label, which is the label information for classifying whether each sampled data has an abnormality, that is, the classification label can be normal or abnormal. An anomaly detection model including a decision tree can be generated according to each sampled data partition, and each sampled data partition can generate a decision tree. The sampled data included in the sampled data partition can generate nodes in the decision tree, and the number of sampled data in each sampled data partition is equal to the number of nodes in the decision tree. For the common branches of two adjacent nodes in the decision tree, any variable can be selected to generate the corresponding binary classification judgment rule, and finally a decision tree composed of a binary tree structure is formed, and the generated node is the leaf node at the end of the decision tree.

[0063] S120: Acquire target detection data corresponding to the current time from the input real-time detection streaming data.

[0064] The target detection data corresponding to the current time is obtained from the input real-time detection streaming data. The real-time detection streaming data can be the streaming data obtained by the terminal device performing real-time detection on the device status, or it can be the streaming data received by the management server from the terminal device performing real-time detection on its own device status. The streaming data can correspond to data values ​​corresponding to one or more variables, and the data values ​​corresponding to multiple variables and the current time can be obtained as the target detection data, that is, the target detection data contains the data values ​​corresponding to the corresponding variables at the current time.

[0065] For example, the specific information contained in a certain target detection data is shown in Table 1.

[0066] variable CPU Utilization Memory usage Video memory usage CPU Temperature I / O rate Data Value 0.65 0.37 0.81 77℃ 265Mb / s

[0067] Table 1

[0068] S130: Perform anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists.

[0069] The target detection data is subjected to anomaly detection according to the anomaly detection model to obtain a detection result of whether an anomaly exists. The target detection data is input into the constructed anomaly detection model, and anomaly detection can be performed on the target detection data using a random forest pruning algorithm to obtain a detection result of whether an anomaly exists.

[0070] In one embodiment, if Figure 5 As shown, step S130 includes sub-steps S131, S132 and S133.

[0071] S131. Inserting the data value of the target detection data as a node to be detected into each decision tree of the anomaly detection model.

[0072] The data value of the target detection data is inserted as the node to be detected into each decision tree of the anomaly detection model. After the node to be detected is inserted, a common branch and corresponding binary classification rules can be generated based on the data value of the node to be detected and the adjacent nodes. After the node to be detected is inserted, a branch structure will be added to the decision tree. Before and after the node to be detected is inserted, the path length from the corresponding conditional branch to the leaf node on the decision tree also changes.

[0073] S132: Calculate the anomaly score of the anomaly detection model after inserting the node to be detected.

[0074] The anomaly score value of each decision tree in the detection model can be calculated after the node to be detected is inserted, and the average value is calculated based on the anomaly score value of each decision tree as the anomaly score of the anomaly detection model. Among them, the anomaly score value can be the Codisp value of the decision tree, and the Codisp value of the decision tree after the node to be detected is the degree to which the unit structure of the decision tree is changed after the node to be detected is inserted into the decision tree. The specific calculation of the Codisp value can be expressed as Codisp(x,T). Codisp(x,T) represents the abnormal score value of node x in decision tree T. The abnormal score value is the degree of model change caused by the insertion of node x into decision tree T. The specific steps for calculating node x in decision tree T include: 1. Traverse all ancestors of node node<-x; 2. If node x is in the left subtree of node, the disp value of the node is the number of nodes in the right subtree of node / the number of nodes in the left subtree; 3. If node x is in the right subtree of node, the disp value of the node is the number of nodes in the left subtree of node / the number of nodes in the right subtree; 4. Find the maximum value of all the above disp values ​​as the Codisp value of node x in decision tree T.

[0075] S133: Determine whether the abnormality score is not greater than a preset score threshold to obtain a detection result of whether an abnormality exists.

[0076] It is judged whether the anomaly score of the anomaly detection model is not greater than the preset score. If the anomaly score is not greater than the score threshold, it means that the unit structure of the decision tree in the anomaly detection model is changed to a small extent after the node to be detected is inserted into each decision tree, and the anomaly detection result of the target detection data is that there is no anomaly; if the anomaly score is greater than the score threshold, it means that the unit structure of the decision tree in the anomaly detection model is changed to a large extent after the node to be detected is inserted into each decision tree, and the anomaly detection result of the target detection data is that there is anomaly. The detection result can be used as the classification label of the target detection data.

[0077] S140: randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree.

[0078] A decision tree is randomly obtained from the anomaly detection model and determined as the target decision tree. While performing anomaly detection on streaming data, the anomaly detection model can also be sampled and updated, that is, the structural unit of the anomaly detection model is updated. Each sampling update only operates on one decision tree in the anomaly detection model. It is necessary to obtain a decision tree from the anomaly detection model and determine it as the target decision tree. The target decision tree can be determined by random acquisition.

[0079] S150: Determine whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule.

[0080] According to the preset non-uniform sampling rules, it is judged whether the target detection data can be used to sample and update the target decision tree. After each detection result of the target detection data is obtained, it is judged whether the target detection data can be used to sample and update the target decision tree according to the non-uniform sampling rules. The non-uniform sampling rules include a scoring value calculation formula.

[0081] In one embodiment, if Figure 6 As shown, step S150 includes sub-steps S151, S152, S153 and S154.

[0082] S151. Randomly generate a random coefficient corresponding to the target detection data.

[0083] The random coefficient corresponding to the target detection data can be randomly generated, and the value range of the random coefficient is (0,1).

[0084] S152. Calculate the detection data score corresponding to the target detection data according to the score calculation formula and the random coefficient.

[0085] The corresponding detection data score is calculated according to the score calculation formula and the random coefficient of the target detection data. In the calculation formula, Δt is the time difference between the timestamp of the target detection data and the current time.

[0086] S153, calculating the corresponding node score value according to the score value calculation formula and the random coefficient of each node in the target decision tree.

[0087] In the above process of generating anomaly detection model, the random coefficients of the target sampling data corresponding to each node in the decision tree have been obtained respectively. Then, the corresponding node score value can be calculated according to the score value calculation formula and the random coefficient of each node in the target decision tree. The calculation process of the node score value is the same as the calculation process of obtaining the sampling data score value. In the calculation formula, Δt is the time difference between the timestamp of the target sampling data corresponding to the node and the current time.

[0088] S154: Determine whether the score of the detection data is greater than a node with the smallest score in the target decision tree, so as to determine whether the target detection data can be used to perform sampling and updating on the target decision tree.

[0089] The nodes in the target decision tree are sorted according to the node score value to obtain a node pile, in which the node score value of the top node of the pile is the smallest, and the node score value of the bottom node of the pile is the largest. It is judged whether the test data score value is greater than the node score value of the smallest node, that is, it is judged whether the test data score value is greater than the node score value of a node at the top of the pile, so as to judge whether the target test data can be used to sample and update the target decision tree. If the test data score value is greater than the node score value of a node at the top of the pile, it is judged that the target test data can be used to sample and update the target decision tree; otherwise, it is judged that the target test data cannot be used to sample and update the target decision tree.

[0090] S160: If the target detection data can be used to sample and update the target decision tree, the nodes in the target decision tree are updated according to the detection results of the target detection data to obtain an updated anomaly detection model, and the step of obtaining the target detection data corresponding to the current time from the input real-time detection streaming data is returned to execute.

[0091] If the target detection data can be used to sample and update the target decision tree, the nodes contained in the target decision tree can be updated according to the detection results of the target detection data to obtain an updated anomaly detection model. Based on the updated anomaly detection model, the target detection data can be obtained from the real-time detection streaming data again and anomaly detection can be performed, that is, returning to execute step S120.

[0092] If the target detection data cannot be used to sample and update the target decision tree, the nodes in the target decision tree are not updated, and the process directly returns to step S120.

[0093] In one embodiment, if Figure 7 As shown, step S160 includes sub-steps S161, S162 and S163.

[0094] S161. Delete a node with the smallest node score in the target decision tree.

[0095] According to the calculated node score value, a node with the smallest node score value is deleted from the target decision tree.

[0096] S162, generating a new node corresponding to the target detection data according to the detection result and adding it to the target decision tree;

[0097] According to the detection results, a new node corresponding to the target detection data is generated, and the new node is added to the target decision tree. The new node can be combined with another node adjacent to the deleted node, and any variable is selected to generate a corresponding binary classification rule to update the common branches between the new node and another node. After updating the common branches, the new node can be added to the target decision tree. At this time, the target decision tree has also completed the update of the structural unit.

[0098] S163, performing node sorting according to the node score value of each node in the target decision tree to obtain a node sorting result.

[0099] After completing the update of the structural units of the target decision tree, the nodes can also be sorted according to the node scoring values ​​of the nodes contained in the updated target structural units. Node sorting is to update the node heap. After the node heap is updated, the node scoring value of the top node in the node heap is the smallest, and the node scoring value of the bottom node is the largest. The sorting of the nodes in the node heap is the obtained node sorting result.

[0100] In the streaming anomaly detection method based on non-uniform sampling provided in the embodiment of the present invention, an anomaly detection model corresponding to the model generation rule is generated according to the historical sampling database, and the target detection data in the real-time detection streaming data is detected by the anomaly detection model to obtain the detection result, and it is determined according to the non-uniform sampling rule whether the target detection data can be used to sample and update the target decision tree in the anomaly detection model. If so, the nodes in the target decision tree are updated according to the detection result of the target detection data to obtain the updated anomaly detection model. Through the above method, the anomaly detection model can be dynamically sampled and updated based on the non-uniform sampling rule to continuously optimize the nodes of the decision tree in the anomaly detection model, thereby improving the sensitivity of the anomaly detection model to data offset, and can greatly improve the accuracy of detecting abnormal data in massive streaming data.

[0101] The embodiment of the present invention further provides a streaming anomaly detection device based on non-uniform sampling, which can be configured in a terminal device or a management server, and is used to execute any embodiment of the aforementioned streaming anomaly detection method based on non-uniform sampling. Figure 8 , Figure 8 A schematic block diagram of a streaming anomaly detection device based on non-uniform sampling provided in an embodiment of the present invention.

[0102] like Figure 8As shown, the streaming anomaly detection device 100 based on non-uniform sampling includes an anomaly detection model generation unit 110, a target detection data acquisition unit 120, a detection result acquisition unit 130, a target decision tree determination unit 140, a sampling update judgment unit 150 and a model update unit 160.

[0103] The anomaly detection model generating unit 110 is used to generate an anomaly detection model corresponding to the model generating rule according to a pre-stored historical sampling database if an input model generating rule is received.

[0104] In a specific embodiment, the anomaly detection model generation unit 110 includes subunits: a target sampling data acquisition unit, which is used to filter out target sampling data that meets the time span from the historical sampling database according to the time span in the model generation rule; a sampling data partition acquisition unit, which is used to extract corresponding sampling data from the target sampling data according to the configuration quantity of the model generation rule to obtain sampling data partitions; a decision tree construction unit, which is used to construct a decision tree corresponding to each sampling data partition according to the classification label of each sampling data in the sampling data partition, so as to generate an anomaly detection model including the decision tree.

[0105] In a specific embodiment, the sampling data partition acquisition unit includes sub-units: a sampling data extraction unit, which is used to extract a sampling data combination equal to the number of node configurations from the target sampling data to obtain a sampling data partition; and a repeated extraction unit, which is used to repeatedly extract the target sampling data until the number of the obtained sampling data partitions is equal to the number of decision tree configurations.

[0106] In a specific embodiment, the sampling data extraction unit includes subunits: a random coefficient generation unit, which is used to randomly generate a random coefficient corresponding to each of the target sampling data; a sampling data scoring value acquisition unit, which is used to calculate the sampling data scoring value corresponding to each of the target sampling data according to the scoring value calculation formula and the random coefficient; a data sorting result acquisition unit, which is used to sort the target sampling data according to the sampling data scoring value to obtain a data sorting result; a data extraction unit, which is used to extract a sampling data combination that is ranked first and equal to the number of node configurations according to the data sorting result to obtain a sampling data partition.

[0107] The target detection data acquisition unit 120 is used to acquire target detection data corresponding to the current time from the input real-time detection streaming data.

[0108] The detection result acquisition unit 130 is used to perform anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists.

[0109] In a specific embodiment, the detection result acquisition unit 130 includes subunits: a node to be detected insertion unit, which is used to insert the data value of the target detection data as a node to be detected into each decision tree of the anomaly detection model; an anomaly score calculation unit, which is used to calculate the anomaly score of the anomaly detection model after inserting the node to be detected; and an anomaly score judgment unit, which is used to judge whether the anomaly score is not greater than a preset score threshold, so as to obtain a detection result of whether there is an abnormality.

[0110] The target decision tree determination unit 140 is used to randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree.

[0111] The sampling update judgment unit 150 is used to judge whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule.

[0112] In a specific embodiment, the sampling update judgment unit 150 includes subunits: a coefficient generation unit, which is used to randomly generate a random coefficient corresponding to the target detection data; a detection data score value acquisition unit, which is used to calculate the detection data score value corresponding to the target detection data according to the score value calculation formula and the random coefficient; a node score value acquisition unit, which is used to calculate the corresponding node score value according to the score value calculation formula and the random coefficient of each node in the target decision tree; a score value judgment unit, which is used to judge whether the detection data score value is greater than a node with the smallest node score value in the target decision tree, so as to determine whether the target detection data can be used to sample and update the target decision tree.

[0113] The model updating unit 160 is used to update the nodes in the target decision tree according to the detection results of the target detection data to obtain an updated anomaly detection model if the target detection data can be used to sample and update the target decision tree, and return to execute the step of obtaining the target detection data corresponding to the current time from the input real-time detection streaming data.

[0114] In a specific embodiment, the model updating unit 160 includes subunits: a node deletion unit, which is used to delete a node with the smallest node score value in the target decision tree; a new node adding unit, which is used to generate a new node corresponding to the target detection data according to the detection result and add it to the target decision tree; a node sorting result acquisition unit, which is used to sort the nodes according to the node score value of each node in the target decision tree to obtain a node sorting result.

[0115] The streaming anomaly detection device based on non-uniform sampling provided in the embodiment of the present invention applies the above-mentioned streaming anomaly detection method based on non-uniform sampling, generates an anomaly detection model corresponding to the model generation rule according to the historical sampling database, and detects the target detection data in the real-time detection streaming data through the anomaly detection model to obtain the detection result, and judges whether the target detection data can be used to sample and update the target decision tree in the anomaly detection model according to the non-uniform sampling rule, and if so, updates the nodes in the target decision tree according to the detection result of the target detection data to obtain the updated anomaly detection model. Through the above method, the anomaly detection model can be dynamically sampled and updated based on the non-uniform sampling rule to continuously optimize the nodes of the decision tree in the anomaly detection model, thereby improving the sensitivity of the anomaly detection model to data offset, and can greatly improve the accuracy of detecting anomaly data in massive streaming data.

[0116] The above-mentioned streaming anomaly detection device based on non-uniform sampling can be implemented in the form of a computer program. The computer program can be used in Fig. 9 Runs on the computer device shown.

[0117] See also Fig. 9 , Fig. 9 1 is a schematic block diagram of a computer device provided by an embodiment of the present invention. The computer device may be a terminal device or a management server for executing a streaming anomaly detection method based on non-uniform sampling to implement anomaly detection on massive streaming detection data.

[0118] See also Fig. 9 The computer device 500 includes a processor 502 , a memory and a network interface 505 connected via a system bus 501 , wherein the memory may include a storage medium 503 and an internal memory 504 .

[0119] The storage medium 503 may store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 may execute a streaming anomaly detection method based on non-uniform sampling, wherein the storage medium 503 may be a volatile storage medium or a non-volatile storage medium.

[0120] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0121] The internal memory 504 provides an environment for the operation of the computer program 5032 in the storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the streaming anomaly detection method based on non-uniform sampling.

[0122] The network interface 505 is used for network communication, such as providing data information transmission, etc. Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device 500 to which the solution of the present invention is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] The processor 502 is used to run a computer program 5032 stored in the memory to implement the corresponding functions in the above-mentioned streaming anomaly detection method based on non-uniform sampling.

[0124] Those skilled in the art will understand that Fig. 9 The embodiments of the computer device shown in the figure do not constitute a limitation on the specific composition of the computer device. In other embodiments, the computer device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, in some embodiments, the computer device may only include a memory and a processor. In such embodiments, the structure and function of the memory and the processor are the same as those of the embodiment of the present invention. Fig. 9 The embodiments shown are consistent and will not be described again here.

[0125] It should be understood that in the embodiment of the present invention, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0126] In another embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium. The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps included in the above-mentioned streaming anomaly detection method based on non-uniform sampling are implemented.

[0127] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the above-described equipment, devices and units can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0128] In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. Units with the same function may also be combined into one unit. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be an electrical, mechanical or other form of connection.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0130] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a magnetic disk or an optical disk.

[0132] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A streaming anomaly detection method based on non-uniform sampling, characterized in that: The method comprises: If the input model generation rule is received, an anomaly detection model corresponding to the model generation rule is generated according to a pre-stored historical sampling database; Obtaining target detection data corresponding to the current time from the input real-time detection streaming data; Performing anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists; Randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree; Determining whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule; If the target detection data can be used to sample and update the target decision tree, the nodes in the target decision tree are updated according to the detection results of the target detection data to obtain an updated anomaly detection model, and the step of obtaining the target detection data corresponding to the current time from the input real-time detection streaming data is returned to be executed; The non-uniform sampling rule includes a scoring value calculation formula, and judging whether the target detection data can be used to perform sampling update on the target decision tree according to the preset non-uniform sampling rule includes: Randomly generating a random coefficient corresponding to the target detection data; Calculate the detection data score corresponding to the target detection data according to the score calculation formula and the random coefficient; Calculate the corresponding node score value according to the score value calculation formula and the random coefficient of each node in the target decision tree; It is determined whether the score of the detection data is greater than a node with the smallest score in the target decision tree to determine whether the target detection data can be used to sample and update the target decision tree.

2. The streaming anomaly detection method based on non-uniform sampling according to claim 1 is characterized in that: The generating of the anomaly detection model corresponding to the model generation rule according to the pre-stored historical sampling database includes: Filtering target sampling data that meets the time span from the historical sampling database according to the time span in the model generation rule; Extracting corresponding sampling data from the target sampling data according to the configuration quantity of the model generation rule to obtain sampling data partitions; A decision tree corresponding to each sampled data partition is constructed according to the classification label of each sampled data in the sampled data partition to generate an anomaly detection model including the decision tree.

3. The streaming anomaly detection method based on non-uniform sampling according to claim 2 is characterized in that: The configuration quantity includes the decision tree configuration quantity and the node configuration quantity, and the configuration quantity according to the model generation rule extracts corresponding sampling data from the target sampling data to obtain the sampling data partition, including: Extracting a sampling data combination equal to the number of the node configurations from the target sampling data to obtain a sampling data partition; The target sampled data is extracted repeatedly until the number of the sampled data partitions obtained is equal to the number of the decision tree configurations.

4. The streaming anomaly detection method based on non-uniform sampling according to claim 1 is characterized in that: The performing anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists includes: Inserting the data value of the target detection data as a node to be detected into each decision tree of the anomaly detection model; Calculating anomaly scores of the anomaly detection model after inserting the node to be detected; It is determined whether the abnormality score is not greater than a preset score threshold to obtain a detection result of whether an abnormality exists.

5. The streaming anomaly detection method based on non-uniform sampling according to claim 3 is characterized in that: The step of extracting a number of sampled data combinations equal to the number of node configurations from the target sampled data to obtain a sampled data partition comprises: Randomly generate a random coefficient corresponding to each of the target sampling data; According to the score value calculation formula and the random coefficient, the sampling data score value corresponding to each target sampling data is calculated respectively; Sorting the target sampled data according to the sampled data scoring values ​​to obtain a data sorting result; According to the data sorting result, a sampling data partition is obtained by extracting a sampling data combination that is ranked first and has the same number as the node configuration.

6. The streaming anomaly detection method based on non-uniform sampling according to claim 1 is characterized in that: The updating of the nodes in the target decision tree according to the detection result of the target detection data to obtain an updated anomaly detection model includes: Deleting a node with the smallest node score in the target decision tree; Generate a new node corresponding to the target detection data according to the detection result and add it to the target decision tree; The nodes are sorted according to the node score value of each node in the target decision tree to obtain a node sorting result.

7. A streaming anomaly detection device based on non-uniform sampling, characterized in that: The device comprises: An anomaly detection model generation unit is used to generate an anomaly detection model corresponding to the model generation rule according to a pre-stored historical sampling database if an input model generation rule is received; The target detection data acquisition unit is used to acquire the target detection data corresponding to the current time from the input real-time detection streaming data; A detection result acquisition unit, used to perform anomaly detection on the target detection data according to the anomaly detection model to obtain a detection result of whether an anomaly exists; A target decision tree determination unit, used to randomly obtain a decision tree from the anomaly detection model and determine it as a target decision tree; A sampling update judgment unit, used to judge whether the target detection data can be used to perform sampling update on the target decision tree according to a preset non-uniform sampling rule; A model updating unit, configured to update the nodes in the target decision tree according to the detection results of the target detection data to obtain an updated anomaly detection model if the target detection data can be used to sample and update the target decision tree, and return to execute the step of acquiring the target detection data corresponding to the current time from the input real-time detection streaming data; The non-uniform sampling rule includes a scoring value calculation formula, and judging whether the target detection data can be used to perform sampling update on the target decision tree according to the preset non-uniform sampling rule includes: Randomly generating a random coefficient corresponding to the target detection data; Calculate the detection data score corresponding to the target detection data according to the score calculation formula and the random coefficient; Calculate the corresponding node score value according to the score value calculation formula and the random coefficient of each node in the target decision tree; It is determined whether the score of the detection data is greater than a node with the smallest score in the target decision tree to determine whether the target detection data can be used to sample and update the target decision tree.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the streaming anomaly detection method based on non-uniform sampling according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the streaming anomaly detection method based on non-uniform sampling according to any one of claims 1 to 6 is implemented.

Images