Abnormal Explanation Method, Abnormal Explanation Device, Computer Device, and Storage Medium

Abstract

In an embodiment of the present application, an anomaly interpretation method, an anomaly interpretation device, a computer device, and a storage medium are provided. The method predicts a sample to be detected using a trained anomaly detection model. In response to the sample to be detected being predicted as an abnormal sample by the anomaly detection model, the outlier degree and local anomaly limit value of the abnormal sample of the predicted result are compared with those of the normal samples of the training result to determine whether the abnormal sample of the predicted result is a global or local anomaly type. The abnormality degree of each feature in the abnormal sample of the predicted result is calculated according to the anomaly type. The feature in the predicted sample of the predicted result whose abnormality degree is higher than a preset feature abnormality threshold is determined as an abnormal feature. The abnormal interpretation is generated based on the determined abnormal feature. The present application makes the interpretation of abnormal samples more accurate by adding a method for distinguishing samples of local abnormal types and quantifying local abnormal features.

Description

Technical Field

[0001] The present application relates to the field of anomaly detection technology, and in particular to an anomaly interpretation method, an anomaly interpretation device, a computer device, and a storage medium. Background Art

[0002] In related fields such as network security, data security, and financial risk control, data analysis will be used to achieve anomaly detection.

[0003] In order to explain the anomaly, that is, to understand the characteristics of the anomaly and the correlation between samples, some anomaly explanation methods will be used.

[0004] However, there is currently no solution that can accurately explain the detected anomalies.

[0005] Invention News

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of this application is to provide an object recognition method, device and storage medium to solve the above-mentioned problems.

[0007] The first aspect of the present application provides an anomaly interpretation method, comprising: dividing each training sample set used to train an anomaly detection model into an abnormal training result sample set and a normal training result sample set based on whether each training sample in the training sample set is abnormal in the prediction result of the anomaly detection model; calculating a local anomaly boundary value based on the outlier degree of each normal training result sample in the normal training result sample set relative to other normal training result samples; predicting a sample to be detected by the trained anomaly detection model; in response to the sample to be detected being predicted as an abnormal prediction result sample by the anomaly detection model, comparing the outlier degree of the abnormal prediction result sample with that of each normal training result sample with the local anomaly boundary value to determine whether the abnormal prediction result sample is a global or local anomaly type; calculating the abnormality degree of each feature in the abnormal prediction result sample according to the anomaly type; determining a feature in the predicted prediction result sample whose abnormality degree is higher than a preset feature abnormality threshold as an abnormal feature; and generating an anomaly interpretation based on the determined abnormal feature.

[0008] In some embodiments of the first aspect, a feature set most relevant to the abnormal prediction result is searched based on the abnormal sample set of the training results and the normal sample set of the training results, and the feature set is divided into a set of strongly correlated features and a set of weakly correlated features, and different abnormality weights are assigned to the strongly correlated features and the weakly correlated features, respectively.

[0009] In some embodiments of the first aspect, the searching for the feature set most relevant to the abnormal prediction result based on the abnormal sample set of the training results and the normal sample set of the training results, and dividing the feature set into a set of strongly correlated features and a set of weakly correlated features, includes: calculating the mutual information coefficient between the feature vector and the label vector of each feature in the feature set corresponding to the training sample set; performing a heuristic search in the feature set based on the maximum mutual information condition to obtain a feature combination as a set of strongly correlated features, and the remaining features constitute the set of weakly correlated features; wherein the feature combination satisfies the probability that the prediction result of the anomaly detection model is abnormal and is higher than a preset threshold.

[0010] In some embodiments of the first aspect, the calculating a local abnormality boundary value based on the outlier degree of each normal training result sample in the normal training result sample set relative to other normal training result samples includes: clustering the normal training result samples in the normal training result sample set to form normal training sample clusters; calculating a first comprehensive result of the correlation between each normal training result sample in each sample cluster and other normal training result samples; and obtaining the maximum value of each of the first comprehensive results as the local abnormality boundary value.

[0011] In some embodiments of the first aspect, the comparing the outlier degree of the predicted result abnormal sample with each of the training result normal samples with the local abnormality boundary value to determine whether the predicted result abnormal sample is a global or local abnormality type includes: calculating a second comprehensive result of the correlation between the predicted result abnormal sample and each of the training result normal samples; comparing the second comprehensive result with the local abnormality boundary value; if the second comprehensive result is higher than the local abnormality boundary value, determining that the predicted result abnormal sample is a global abnormality type; otherwise, determining that the predicted result abnormal sample is a local abnormality type.

[0012] In some embodiments of the first aspect, the global anomaly type and the local anomaly type are respectively used to determine different correlation sample ranges of the prediction result abnormal sample in the training result normal sample set, and each training result normal sample within the correlation sample range is used as a relevant sample for calculating the abnormality of the feature.

[0013] In some embodiments of the first aspect, calculating the abnormality of each feature in the abnormal sample of the predicted result according to the abnormality type includes: combining the abnormality type of the abnormal sample of the predicted result and the abnormality weight of each feature contained in the abnormal sample of the predicted result, and calculating the abnormality of each feature in the abnormal sample.

[0014] In some embodiments of the first aspect, the abnormality type of the abnormal sample of the prediction result and the abnormality weight of each feature contained in the abnormal sample of the prediction result are combined to calculate the abnormality of each feature in the abnormal sample, including: for the t-th feature in the abnormal sample a of the prediction result having a characteristic value v at , calculate the abnormality ρ of the t-th feature according to the following formula anomaly_t :

[0015]

[0016] Among them, α u Indicates the abnormality weight of the strongly correlated feature, α n Represents the abnormality weight of the weakly correlated feature, V St represents the feature value set of the tth feature of the normal sample of the training result on the normal sample set of the training result; C top_i_sim represents the i-th one of the q normal training sample clusters closest to the abnormal sample of the predicted result, It represents the set of feature values of the t-th feature on the i-th normal training sample cluster that is closest to the abnormal sample of the predicted result; mean represents the mean, and Dev represents the deviation.

[0017] In some embodiments of the first aspect, the anomaly interpretation method further includes: if the abnormality degree of the feature not existing in the abnormal sample of the prediction result is higher than a preset feature abnormality threshold, then taking a preset number of features with the largest abnormality degree as abnormal features.

[0018] The second aspect of the present application provides an anomaly interpretation device, comprising: a sample classification module for dividing each training sample set into a training result abnormal sample set and a training result normal sample set based on whether each training sample in the training sample set for training the anomaly detection model is abnormal in the prediction result of the anomaly detection model; a local anomaly boundary calculation module for calculating a local anomaly boundary value based on the outlier degree of each training result normal sample in the training result normal sample set relative to other training result normal samples; an anomaly prediction module for predicting a sample to be detected by using the trained anomaly detection model; an anomaly type determination module for classifying the predicted result abnormal sample as a predicted result abnormal sample in response to the sample to be detected being predicted by the anomaly detection model. The outlier degree of each normal sample of the training result is compared with the local anomaly limit value to determine whether the abnormal sample of the prediction result is a global or local anomaly type; an anomaly degree calculation module is used to calculate the abnormality degree of each feature in the abnormal sample of the prediction result according to the anomaly type; wherein the global anomaly type and the local anomaly type are respectively used to determine different correlation sample ranges of the abnormal sample of the prediction result in the normal sample set of the training result, and each normal sample of the training result within the correlation sample range is used as a relevant sample for calculating the abnormality degree of the feature; an abnormal feature determination module is used to determine a feature in the prediction sample of the prediction result whose abnormality degree is higher than a preset feature abnormality threshold as an abnormal feature; an abnormal explanation module is used to generate an abnormal explanation based on the determined abnormal feature.

[0019] A third aspect of the present application provides a computer device comprising: a communicator, a memory and a processor; the communicator is used to communicate with the outside; the memory is used to store program instructions; and the processor is used to run the program instructions to execute any one of the anomaly interpretation methods described in the first aspect.

[0020] A fourth aspect of the present application provides a computer-readable storage medium storing program instructions, wherein the program instructions are executed to execute the exception interpretation method described in any one of the first aspects.

[0021] As described above, the embodiments of the present application provide an anomaly interpretation method, an anomaly interpretation device, a computer device, and a storage medium, wherein the trained anomaly detection model is used to predict a sample to be detected; in response to the sample to be detected being predicted as an abnormal sample of the predicted result by the anomaly detection model, the outlier degree of the abnormal sample of the predicted result and each normal sample of the training result is compared with the local anomaly limit value to determine whether the abnormal sample of the predicted result is a global or local anomaly type; the abnormality degree of each feature in the abnormal sample of the predicted result is calculated according to the anomaly type; wherein the global anomaly type and the local anomaly type are respectively used to determine the different correlation sample ranges of the abnormal sample of the predicted result in the normal sample set of the training result, and each normal sample of the training result within the correlation sample range is used as a relevant sample for calculating the abnormality degree of the feature; the feature in the predicted sample of the predicted result whose abnormality degree is higher than the preset feature abnormality threshold is determined as an abnormal feature; and an anomaly interpretation is generated based on the determined abnormal feature. The present application makes the interpretation of abnormal samples more accurate by adding a method for sample discrimination of local anomaly types and quantification of local anomaly features. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram showing the structure of a computer device in one embodiment of the present application is shown.

[0023] Figure 2 A flowchart illustrating an abnormality interpretation method in one embodiment of the present application is shown.

[0024] Figure 3 A flowchart showing an exception interpretation method in an application example of the present application is shown.

[0025] Figure 4 A module schematic diagram of a communication anomaly interpretation device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following describes the embodiments of the present application through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present application from the information disclosed in this application. The present application can also be implemented or applied through different specific embodiments. The details in this application can also be modified or changed according to different viewpoints and application systems without departing from the spirit of this application. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless they conflict.

[0027] The following is a detailed description of the embodiments of the present application with reference to the accompanying drawings so that those skilled in the art can easily implement the present application. The present application can be embodied in many different forms and is not limited to the embodiments described herein.

[0028] In the description of this application, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and integrate different embodiments or examples described in this application, as well as features of different embodiments or examples, unless otherwise contradictory.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include at least one such feature. In the context of this application, "plurality" means two or more, unless otherwise specifically defined.

[0030] In order to clearly describe the present application, components not related to the description are omitted, and the same or similar components throughout the specification are given the same reference numerals.

[0031] Throughout this specification, when a device is said to be "connected" to another device, this includes not only "direct connection" but also "indirect connection" with other elements interposed therebetween. Furthermore, when a device is said to "include" a certain component, unless otherwise stated, this does not exclude the inclusion of other components but rather implies that the device may include other components.

[0032] Although the terms first, second, etc. are used in this document to represent various elements in some examples, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, the first interface and the second interface, etc. are represented. Furthermore, as used in this document, the singular forms "one," "an," and "the" are intended to also include the plural forms, unless there is a contrary indication in the context. It should be further understood that the terms "comprise" and "include" indicate the presence of the described features, steps, operations, elements, modules, projects, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, modules, projects, types, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." Exceptions to this definition only occur when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0033] The technical terms used herein are intended only to refer to specific embodiments and are not intended to limit this application. The singular form used herein also includes the plural form unless the statement explicitly indicates otherwise. The term "comprising" as used in this specification is intended to specify specific features, regions, integers, steps, operations, elements, and / or components and does not exclude the presence or addition of other features, regions, integers, steps, operations, elements, and / or components.

[0034] Although not defined differently, all terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art to which this application belongs. Terms defined in commonly used dictionaries are additionally interpreted as having meanings consistent with relevant technical literature and the current information provided. Unless otherwise defined, they should not be overly interpreted as ideal or highly formal meanings.

[0035] Currently, in anomaly detection technology, there is a lack of means to explain anomalies, and anomalies cannot be accurately explained.

[0036] In an embodiment of the present application, a method for explaining anomalies may be provided, which provides global or local type discrimination of detected abnormal samples to obtain a more accurate explanation of the anomaly.

[0037] like Figure 1 FIG. 1 is a schematic diagram showing a circuit structure of a computer device in an embodiment of the present application. The abnormality interpretation method can be implemented by a program running in the computer device.

[0038] In some embodiments, the computer device 100 can be applied to, for example, a server, a desktop computer, or a mobile terminal (such as a smart phone, a laptop computer, a tablet computer, etc.).

[0039] The computer device 100 includes a bus 101, a processor 102, a memory 103, and a communicator 104. The processor 102 and the memory 103 can communicate via the bus 101. The memory 103 can store program instructions (such as system or application software). The processor 102 executes the program instructions in the memory 103 to implement the steps of the object recognition method in the embodiment of the present application.

[0040] The bus 101 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, although Figure 1Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0041] In some embodiments, the processor 102 may be implemented as a central processing unit (CPU), a microprocessor unit (MCU), a system on a chip (SoC), or a field programmable gate array (FPGA). The memory 103 may include volatile memory, such as random access memory (RAM), for temporarily storing data while running programs.

[0042] The memory 103 may also include a non-volatile memory (non-volatile memory) for data storage, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state disk (SSD).

[0043] The communicator 104 is used to communicate with the outside. In a specific example, the communicator 104 may include one or more wired and / or wireless communication circuit modules. For example, the communicator 104 may include one or more of a wired network card, a USB module, a serial interface module, etc. The wireless communication protocols followed by the wireless communication module include, for example, near field communication (NFC) technology, infrared (IR) technology, Global System for Mobile communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), Bluetooth (BT), Global Navigation Satellite System (GNSS), etc. One or more of the above.

[0044] like Figure 2 FIG. 1 is a flow chart showing a method for interpreting anomalies in accordance with an embodiment of the present invention. The method comprises:

[0045] Step S201: Based on whether each training sample in the training sample set used to train the anomaly detection model is abnormal in the prediction result of the anomaly detection model, each training sample set is divided into a training result abnormal sample set and a training result normal sample set.

[0046] In some embodiments, the anomaly detection model can be implemented based on an anomaly detection algorithm, such as Isolation Forest (iForest), Outlier Detection Algorithm (LOF), etc., and anomaly detection is performed based on the characteristics that "abnormal" samples are very different from normal samples but very dense.

[0047] The anomaly detection model is trained using a training sample set. The training sample can be in the form of a feature vector obtained by extracting features from the original sample (e.g., text, image), which contains the eigenvalue of at least one feature. In one possible example, a feature matrix (e.g., behavioral training samples, with eigenvalues listed) can be obtained by stacking the feature vectors of each training sample in the training sample set. Optionally, the feature matrix can be standardized to a format that conforms to the input of the anomaly detection model, and then input into the anomaly detection model for training.

[0048] It is understandable that, in some embodiments, the "experience" regarding abnormal or normal results obtained by inputting the training sample set into the anomaly detection model can be used as a reference for explaining when the sample to be detected is predicted to be abnormal.

[0049] Therefore, in a possible example, the training sample sets used to train the anomaly detection model can be divided into abnormal training sample sets and normal training sample sets based on whether each training sample in the anomaly detection model's prediction results are abnormal. For example, if training sample a is input into the anomaly detection model and the prediction result is abnormal, then a is considered an abnormal training sample and is included in the abnormal training sample set; or if training sample b is input into the anomaly detection model and the prediction result is normal, then b is considered a normal training sample and is included in the normal training sample set. Furthermore, analysis can be performed based on the normal training sample set and the abnormal training sample set to extract "experience."

[0050] Step S202: Calculate a local abnormality limit value based on the outlier degree of each normal training result sample relative to other normal training result samples in the normal training result sample set.

[0051] In some examples, the local abnormality limit value can be determined based on the normal sample set of the training results, and used to determine whether the sample to be detected is a global or local abnormality type when it is an abnormal sample. Specifically, the global abnormality type refers to an abnormality type that is relatively different from the overall normal sample set of the training results (that is, it is relatively "outlier") and is easy to detect; the local abnormality type refers to an abnormality type that has certain differences compared to each normal sample of the training results, but cannot be isolated to the entire normal sample set of the training results; samples of the local abnormality type need to highlight the differences to distinguish them from samples of the global abnormality type, so that the abnormal samples of the local abnormality type can be analyzed and interpreted more accurately. The way to highlight the differences can be a way of locally selecting comparison objects, that is, selecting some normal samples of the training results that are more relevant to the abnormal samples of the local abnormality type in the normal sample set of the training results for comparison.

[0052] Therefore, the global anomaly type and local anomaly type can be used to determine different correlation sample ranges for abnormal prediction result samples within the normal training result sample set. Each normal training result sample within the correlation sample range serves as a relevant sample for calculating the abnormality of the feature. For example, the global anomaly type corresponds to samples in the normal training result sample set as relevant samples, while the local anomaly type corresponds to a portion of samples in the normal training result sample set that are more relevant to it as relevant samples for correlation calculation.

[0053] The following provides an example of calculating the local anomaly limit value.

[0054] Each normal training sample in the training result normal sample set can be clustered to form each normal training sample cluster. Possibly, the clustering can be exemplarily a K-Means method, and the normal training sample cluster uses C i Indicates that for cluster sample c ij There is {c ij |c ij ∈C i ; i=1,2,…,n; j=1,2,…,m i}, where n is the number of clusters, m i is the total number of samples in the i-th cluster.

[0055] A first comprehensive result of the correlation between each normal training sample and other normal training samples in each sample cluster is calculated, and the maximum value among the first comprehensive results is obtained as the local anomaly threshold. Optionally, the correlation can be measured using the Mahalanobis distance, and the first comprehensive result can be the sum of the various Mahalanobis distances. Of course, in other embodiments, the correlation measurement method can also be varied based on needs and the characteristics of different correlation measurement methods, and is not limited to the Mahalanobis distance.

[0056] For example, calculate the local anomaly limit value γ max ,exist:

[0057]

[0058] Where, dist is the calculated Mahalanobis distance. ij represents a normal sample of training results in a normal training sample cluster, c k Indicates c in S ij The training results are normal samples other than .

[0059] It is understandable that the experience of the influence of features in normal training result samples and abnormal training result samples on obtaining abnormal prediction results can also be referenced.

[0060] In some embodiments, based on the abnormal sample set of the training results and the normal sample set of the training results, the feature set that is most relevant to the abnormal prediction result is searched, and the feature set is divided into a set of strongly correlated features and a set of weakly correlated features, and different abnormality weights are assigned to the strongly correlated features and the weakly correlated features, respectively. The abnormality weight can be used as a reference when calculating the abnormality of each feature later. It is understandable that the stronger the correlation between the abnormality weight characterizing the feature and the abnormal prediction result, the greater the contribution to predicting the abnormal prediction result. Specifically, the more relevant it is to the abnormal prediction result, that is, the more "strongly correlated", the greater the weight of the corresponding abnormality may be. Correspondingly, the less relevant it is to the abnormal prediction result, that is, the more "weakly correlated", the smaller the weight of the corresponding abnormality may be.

[0061] In a possible example, a feature set can be obtained based on maximum mutual information, and a set of strongly correlated features in the feature set is determined based on the features that contribute most to the anomaly prediction result, and the remaining features form a set of weakly correlated features. In a specific example, this process may include:

[0062] A) Calculate the mutual information coefficient (MIC) between the feature vector and label vector of each feature in the feature set corresponding to the training sample set.

[0063] Let's use an example to illustrate the feature vector and label vector. A training sample set contains 100 samples, all of which contain feature A. The sample values a1, a2, ..., a100 of feature A in these 100 samples form a 100-dimensional feature vector [a1, a2, ..., a100]. A label vector is a vector composed of the preset label values for each training sample. For example, if a label value of "0" indicates that the training sample is abnormal and a value of "1" indicates that the training sample is normal, then the label vector is a 100-dimensional vector consisting of 0s and 1s. Mutual information can be calculated between this 100-dimensional feature A and the label vector.

[0064] By calculating the mutual information coefficient between the feature vector of each feature and the label vector, the mutual information coefficient is used to show the strength of the correlation between the ideal result to be output by the model corresponding to the feature and the label vector, that is, the contribution of the feature to the ideal result.

[0065] B) performing a heuristic search on the feature set based on the maximum mutual information condition to obtain a feature combination as a set of strongly correlated features, and using the remaining features to form the set of weakly correlated features; wherein the feature combination satisfies that the probability that the prediction result of the anomaly detection model is anomaly is higher than a preset threshold.

[0066] For example, the features in the feature set are sorted according to the size of their mutual information. Feature A with the largest mutual information is first selected and input into the anomaly detection model to obtain the probability that the prediction result is abnormal, which is assumed to be 60%, not higher than the preset threshold, such as 90%; then feature B with the second largest mutual information is selected, and A and B are input into the anomaly detection model together to obtain the probability that the prediction result is abnormal, which is 70%, not higher than the preset threshold 95%; and so on, assuming that feature G with the 7th largest mutual information is obtained, the probability of the prediction result being abnormal obtained by A~G in the anomaly detection model is 95.2%, which meets the preset threshold, then A~G is a set of strongly correlated features of the abnormal prediction result; the remaining features in the feature set are weakly correlated features, constituting a set of weakly correlated features.

[0067] Different weights can be assigned to strongly correlated features and weakly correlated features. For example, initially, each feature has a weight of 50%. After determining the set of strongly correlated features and the set of weakly correlated features, the weight of each strongly correlated feature can be increased, for example, to 70%.

[0068] Step S203: predicting the samples to be detected by using the trained anomaly detection model.

[0069] Step S204: In response to the sample to be detected being predicted by the anomaly detection model as a predicted result abnormal sample, the outlier degree of the predicted result abnormal sample and each of the training result normal samples is compared with the local anomaly limit value to determine whether the predicted result abnormal sample is a global or local anomaly type.

[0070] For example, the local anomaly limit value is γ max A similar correlation synthesis calculation method can also be used for abnormal prediction sample a to calculate a second comprehensive result of the correlation between the abnormal prediction sample and each normal training sample. Optionally, the correlation can be calculated in the same manner as the correlation used in the calculation of the local anomaly threshold value, such as the Mahalanobis distance.

[0071] For example, the calculation formula for the second comprehensive result is shown below:

[0072]

[0073] Then, the second comprehensive result is compared with the local abnormality limit value; if the second comprehensive result is higher than the local abnormality limit value, the abnormal sample of the predicted result is determined to be a global abnormality type; otherwise, the abnormal sample of the predicted result is determined to be a local abnormality type. For example, if γ a >γ max , a is the global anomaly, γ a ≤γ max , a is a local anomaly.

[0074] Step S205: Calculate the abnormality degree of each feature in the abnormal sample of the prediction result according to the abnormality type.

[0075] In some first embodiments, based on the strong and weak correlation features and their corresponding abnormality weights of the abnormal prediction results obtained from the training sample set, the abnormality type of the abnormal sample in the predicted result and the abnormality weight of each feature included in the abnormal sample can be combined to calculate the abnormality of each feature in the abnormal sample, so as to make the obtained abnormality more accurate. Of course, in other embodiments, the abnormality weights corresponding to the strong and weak correlation features can also be used without reference, and the calculation can be performed based on, for example, weights preset by experience.

[0076] In some embodiments, the abnormality type of the abnormal sample of the prediction result and the abnormality weight of each feature contained in the abnormal sample of the prediction result are combined to calculate the abnormality of each feature in the abnormal sample, including: for the t-th feature in the abnormal sample a of the prediction result having a characteristic value v at , calculate the abnormality ρ of the t-th feature according to the following formula anomaly_t , ρ anomaly_t The calculation method of is shown below:

[0077]

[0078] Among them, α u Represents the abnormality weight of the strongly correlated feature, α n Represents the abnormality weight of the weakly correlated feature, V St represents the feature value set of the tth feature of the normal sample of the training result on the normal sample set of the training result; C top_i_sim represents the i-th one of the q normal training sample clusters closest to the abnormal sample of the predicted result, represents the set of feature values of the tth feature on the ith normal training sample cluster closest to the abnormal sample of the predicted result; mean represents the mean, and Dev represents the deviation. For example, the values of q and r can be selected according to the actual situation, such as 3, 4, 5 or other. For example, α u Use 0.9, α n Use 0.1 etc.

[0079] According to the above formula, it can be understood that for the features of the abnormal samples of the prediction results of the global abnormal type, the correlation can be calculated with the features of the normal samples of each training result in the normal sample set S. This is because the abnormal samples of the prediction results of the global abnormal type are different enough from the whole S, and can accurately explain the abnormality of the features, so V can be used. St to perform calculations.

[0080] However, for the features in the abnormal samples of the prediction results of the local abnormal type, if the feature correlation comparison of the whole S is considered, the abnormal samples of the prediction results of the local abnormal type are not enough to show the difference from the whole S, and will be disturbed by the normal samples of the training results, making it difficult to accurately explain the abnormality of the features. Therefore, the feature value set of the feature on the q normal training sample clusters closest to the prediction result sample is used. To calculate the abnormality, instead of using V St .

[0081] Step S206: Determine the features in the prediction result prediction sample whose abnormality is higher than a preset feature abnormality threshold as abnormal features.

[0082] For example, you can set a preset feature anomaly threshold σ, ρ anomaly_t >σ, it means that the t-th feature of abnormal prediction sample a is an abnormal feature; otherwise, the t-th feature of abnormal prediction sample a is a normal feature. For example, σ can be 1.2, etc.

[0083] Alternatively, it is possible that, in extreme cases, if the abnormality of a feature that does not exist in the abnormal sample of the prediction result is higher than a preset feature abnormality threshold, a preset number of features with the largest abnormality can be used as abnormal features, for example, the top three features with the largest abnormality in the abnormal prediction sample are selected as abnormal features.

[0084] Step S207: Generate an abnormality explanation based on the determined abnormality features.

[0085] In some embodiments, anomaly explanations can be generated by mapping the anomaly features. Alternatively, the anomaly features can be sorted from highest to lowest by degree of abnormality, and then mapped to generate anomaly explanations. For example, the anomaly explanations can include information about the impact of an anomaly feature or combination of features on an anomaly detection model's prediction of an abnormal result. This explanation can provide analytical guidance for why the anomaly detection model predicted an abnormal result.

[0086] For intuitive expression Figure 2 The process of the method, such as Figure 3 As shown, a flow chart showing the abnormality interpretation method in an application example of the present application is shown.

[0087] like Figure 3 The process shown in the figure below:

[0088] On the one hand, the training sample set is input into the anomaly detection model for training, and the training result abnormal sample set and the training result normal sample set are obtained according to whether the prediction result is abnormal or not. The set of strongly correlated features and the set of weakly correlated features of the abnormal prediction results are searched based on the training result abnormal sample set and the training result normal sample set, and the abnormality weight of each feature is obtained accordingly.

[0089] Moreover, the local abnormality limit value is calculated based on the normal sample set of the training results for global or local type judgment.

[0090] On the other hand, the trained anomaly detection model is used to predict the sample to be detected. If it is abnormal, the sample to be detected is an abnormal sample of the prediction result; through the global or local abnormality type judgment, that is, comparing whether the outlier degree of the abnormal sample of the prediction result is higher than the local abnormality limit value, it is judged whether the abnormal sample of the prediction result is a local abnormal type sample or a global abnormal type sample.

[0091] Furthermore, combined with the abnormal type of the abnormal sample of the prediction result and the abnormality weight of each feature in the abnormal sample of the prediction result, the abnormality of each feature in the abnormal sample of the prediction result is calculated; under the condition that the abnormality is higher than the preset feature abnormality threshold, the abnormal features are determined from each feature in the abnormal sample of the prediction result, and the abnormal explanation language is formed according to the abnormal feature mapping.

[0092] like Figure 4 FIG2 is a block diagram showing an abnormality interpretation device in an embodiment of the present application. The implementation of the abnormality interpretation device can refer to the previous abnormality interpretation method embodiment, so the same technical content will not be repeated in this example.

[0093] The abnormality interpretation device 400 includes:

[0094] The sample classification module 401 is used to divide each training sample set into a training result abnormal sample set and a training result normal sample set based on whether each training sample in the training sample set for training the anomaly detection model is abnormal or not in the prediction result of the anomaly detection model.

[0095] The local anomaly limit calculation module 402 is configured to calculate a local anomaly limit value based on the outlier degree of each normal training result sample relative to other normal training result samples in the normal training result sample set.

[0096] Anomaly prediction module 403, used to predict the samples to be detected by using the trained anomaly detection model;

[0097] an anomaly type determination module 404, configured to, in response to a sample to be detected being predicted as a predicted result abnormal sample by the anomaly detection model, compare the outlier degree of the predicted result abnormal sample with each of the training result normal samples with the local anomaly threshold value to determine whether the predicted result abnormal sample is a global or local anomaly type;

[0098] Anomaly degree calculation module 405 is used to calculate the abnormality degree of each feature in the abnormal sample of the prediction result according to the abnormality type; wherein the global abnormality type and the local abnormality type are respectively used to determine different correlation sample ranges of the abnormal sample of the prediction result in the normal sample set of the training result, and each normal sample of the training result within the correlation sample range is used as the relevant sample for calculating the abnormality degree of the feature;

[0099] Abnormal feature determination module 406, used to determine the features in the prediction result prediction sample whose abnormality is higher than a preset feature abnormality threshold as abnormal features;

[0100] The anomaly explanation module 407 is configured to generate an anomaly explanation based on the determined anomaly features.

[0101] In some embodiments, the anomaly interpretation device 400 further includes:

[0102] The abnormality weight calculation module is used to search for the feature set most relevant to the abnormal prediction result based on the abnormal sample set of the training results and the normal sample set of the training results, and divide the feature set into a set of strongly correlated features and a set of weakly correlated features, and respectively assign different abnormality weights to the strongly correlated features and the weakly correlated features.

[0103] In some embodiments, the abnormality weight calculation module includes:

[0104] A correlation calculation module is used to calculate the mutual information coefficient between the feature vector and the label vector of each feature in the feature set corresponding to the training sample set;

[0105] The feature search module is used to perform a heuristic search in the feature set based on the maximum mutual information condition to obtain a feature combination as a set of strongly correlated features, and the remaining features constitute the set of weakly correlated features; wherein the feature combination satisfies that the probability that the prediction result of the anomaly detection model is anomaly is higher than a preset threshold.

[0106] In some embodiments, the local anomaly limit calculation module includes:

[0107] A clustering module, configured to cluster each normal training result sample in the normal training result sample set to form each normal training sample cluster;

[0108] A distance calculation module, configured to calculate a first comprehensive result of the correlation between each normal sample of the training results and other normal samples of the training results in each sample cluster;

[0109] A value obtaining module is used to obtain the maximum value among the first comprehensive results as the local abnormality limit value.

[0110] In some embodiments, the abnormality type determination module 404 includes:

[0111] A correlation synthesis module, configured to calculate a second comprehensive result of the correlation between the abnormal samples of the prediction results and the normal samples of the training results;

[0112] A comparison module is used to compare the second comprehensive result with the local abnormality limit value; if the second comprehensive result is higher than the local abnormality limit value, the prediction result abnormal sample is determined to be a global abnormality type; otherwise, the prediction result abnormal sample is determined to be a local abnormality type.

[0113] In some embodiments, the global anomaly type and the local anomaly type are respectively used to determine different correlation sample ranges of the prediction result abnormal samples in the training result normal sample set, and each training result normal sample within the correlation sample range is used as a relevant sample for calculating the abnormality of the feature.

[0114] In some embodiments, the abnormality calculation module 405 is configured to calculate the abnormality of each feature in the abnormal sample by combining the abnormality type of the abnormal sample of the prediction result and the abnormality weight of each feature included in the abnormal sample of the prediction result.

[0115] In some embodiments, the abnormality calculation module 405 combines the abnormality type of the predicted result abnormal sample and the abnormality weight of each feature contained in the predicted result abnormal sample to calculate the abnormality of each feature in the abnormal sample, including: for the t-th feature in the predicted result abnormal sample a with a feature value v at, calculate the abnormality ρ of the t-th feature according to the following formula anomaly_t :

[0116]

[0117] Among them, α u Represents the abnormality weight of the strongly correlated feature, α n Represents the abnormality weight of the weakly correlated feature, V St represents the feature value set of the tth feature of the normal sample of the training result on the normal sample set of the training result; C top_i_sim represents the i-th one of the q normal training sample clusters closest to the abnormal sample of the predicted result, It represents the set of feature values of the t-th feature on the i-th normal training sample cluster that is closest to the abnormal sample of the predicted result; mean represents the mean, and Dev represents the deviation.

[0118] In some embodiments, the anomaly interpretation device further comprises:

[0119] The abnormal feature selection module is used to select a preset number of features with the largest abnormality as abnormal features if the abnormality of the features that do not exist in the abnormal samples of the prediction results is higher than the preset feature abnormality threshold.

[0120] It should be noted that in Figure 4 The various functional modules in the embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a program instruction product. The program instruction product includes one or more program instructions. When the program instruction instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0121] and, Figure 4 The devices disclosed in the embodiments can be implemented through other module division methods. The device embodiments shown above are merely illustrative. For example, the module division is merely a logical function division. In actual implementation, other division methods may be used, such as multiple modules or modules that can be combined or dynamically transferred to another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the devices or modules shown or discussed can be through some interface. The indirect coupling or communication connection between the devices or modules can be electrical or other forms.

[0122] in addition, Figure 4Each functional module and submodule in the embodiments may be dynamically integrated into a single processing component, each module may exist physically independently, or two or more modules may be dynamically integrated into a single component. The aforementioned dynamic components may be implemented in hardware or as software functional modules. If the aforementioned dynamic components are implemented as software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0123] It should be noted that the processes or methods represented by the flowcharts of the above embodiments of the present application can be understood as modules, segments, or portions of code that include one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application also includes other implementations in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order depending on the functions involved.

[0124] For example, Figure 2 The order of the steps in the embodiments may be changed in specific scenarios and is not limited to the above.

[0125] In an embodiment of the present application, a computer-readable storage medium may be provided, which stores program instructions, and when the program instructions are executed, the aforementioned method embodiment (for example, Figure 2 ) process steps performed.

[0126] That is, the method steps in the above embodiments are implemented as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or are implemented as computer code originally stored in a remote recording medium or a non-transitory machine-readable medium downloaded via a network and to be stored in a local recording medium, so that the method represented herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor or programmable or dedicated hardware (such as an ASIC or FPGA).

[0127] In summary, the embodiments of the present application provide an anomaly interpretation method, an anomaly interpretation device, a computer device, and a storage medium, which predict the sample to be detected by the trained anomaly detection model; in response to the sample to be detected being predicted as a predicted result abnormal sample by the anomaly detection model, the outlier degree of the predicted result abnormal sample and each of the training result normal samples is compared with the local anomaly limit value to determine whether the predicted result abnormal sample is a global or local anomaly type; the abnormality degree of each feature in the predicted result abnormal sample is calculated according to the anomaly type; wherein the global anomaly type and the local anomaly type are respectively used to determine the different correlation sample ranges of the predicted result abnormal sample in the training result normal sample set, and each training result normal sample within the correlation sample range is used as a related sample for calculating the abnormality degree of the feature; the feature in the predicted result prediction sample whose abnormality degree is higher than the preset feature abnormality threshold is determined as an abnormal feature; and an anomaly interpretation is generated based on the determined abnormal feature. The present application makes the interpretation of abnormal samples more accurate by adding a method for sample discrimination of local anomaly types and quantification of local anomaly features.

[0128] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical concepts disclosed in this application shall be covered by the claims of this application.

Claims

1. A method for interpreting anomalies, characterized in that: include: Based on a training sample set for training anomaly detection model, dividing the training sample set into a training result abnormal sample set and a training result normal sample set according to a prediction result of the anomaly detection model on each training sample, wherein the training result abnormal sample set includes training samples determined to be abnormal by the anomaly detection model, and the training samples in the training sample set are obtained based on text or images; Calculating a local abnormality limit value based on the outlier degree of each normal training result sample relative to other normal training result samples in the normal training result sample set; Predicting an input sample to be detected using the trained anomaly detection model to obtain a prediction result, wherein the sample to be detected is obtained based on the text or image to be detected corresponding to the training sample; In response to the sample to be detected being predicted as a predicted result abnormal sample by the abnormality detection model, the outlier degree of the predicted result abnormal sample and each of the training result normal samples is compared with the local abnormality limit value to determine whether the predicted result abnormal sample is a global or local abnormality type; Calculating the abnormality of each feature in the abnormal sample of the prediction result according to the abnormality type and the abnormality weights of the features included in the abnormal sample of the prediction result, wherein the abnormality weights include abnormality weights corresponding to strong and weak correlation features determined based on the training sample set, or preset weights; Determine a feature in the prediction result prediction sample whose abnormality is higher than a preset feature abnormality threshold as an abnormal feature; Based on the determined abnormal features, an abnormality explanation is generated for analyzing the abnormal sample of the prediction result.

2. The abnormality interpretation method according to claim 1, characterized in that: Also includes: Based on the abnormal sample set of the training results and the normal sample set of the training results, a feature set most relevant to the abnormal prediction result is searched, and the feature set is divided into a set of strongly correlated features and a set of weakly correlated features, and different abnormality weights are respectively assigned to the strongly correlated features and the weakly correlated features.

3. The abnormality interpretation method according to claim 2, characterized in that: The step of searching for a feature set most relevant to the abnormal prediction result based on the abnormal sample set of the training result and the normal sample set of the training result, and dividing the feature set into a set of strongly relevant features and a set of weakly relevant features, includes: Calculate the mutual information coefficient between the feature vector and label vector of each feature in the feature set corresponding to the training sample set; A heuristic search is performed on the feature set based on the maximum mutual information condition to obtain a feature combination as a set of strongly correlated features, and the remaining features constitute the set of weakly correlated features; wherein the feature combination satisfies that the probability that the prediction result of the anomaly detection model is abnormal is higher than a preset threshold.

4. The abnormality interpretation method according to claim 1, characterized in that: The calculating a local abnormality limit value based on the outlier degree of each normal training result sample relative to other normal training result samples in the normal training result sample set includes: Clustering each normal training result sample in the normal training result sample set to form each normal training sample cluster; Calculating a first comprehensive result of the correlation between each normal sample of the training results and other normal samples of the training results in each sample cluster; The maximum value among the first comprehensive results is obtained as the local abnormality limit value.

5. The abnormality interpretation method according to claim 4, characterized in that: The comparing the outlier degree of the abnormal sample of the prediction result with the normal samples of each of the training results with the local abnormality limit value to determine whether the abnormal sample of the prediction result is a global or local abnormality type includes: Calculating a second comprehensive result of the correlation between the abnormal samples of the prediction results and the normal samples of the training results; comparing the second comprehensive result with the local abnormality limit value; If the second comprehensive result is higher than the local abnormality limit value, the abnormal sample of the prediction result is determined to be a global abnormality type; otherwise, the abnormal sample of the prediction result is determined to be a local abnormality type.

6. The abnormality interpretation method according to claim 1, characterized in that: The global anomaly type and the local anomaly type are respectively used to determine different correlation sample ranges of the prediction result abnormal samples in the training result normal sample set, and each training result normal sample within the correlation sample range is used as a relevant sample for calculating the abnormality of the feature.

7. The anomaly interpretation method according to claim 1, characterized in that: The step of calculating the abnormality degree of each feature in the abnormal sample according to the abnormality type and the abnormality degree weight of the feature included in the abnormal sample of the prediction result includes: For the prediction result abnormal sample a, the tth feature has the feature value v at , calculate the abnormality ρ of the t-th feature according to the following formula anomaly_t : Among them, α u Indicates the abnormality weight of the strongly correlated feature, α n Represents the abnormality weight of the weakly correlated feature, V St represents the feature value set of the tth feature of the normal sample of the training result on the normal sample set of the training result; C top_i_sim represents the i-th one of the q normal training sample clusters closest to the abnormal sample of the predicted result, It represents the set of feature values of the t-th feature on the i-th normal training sample cluster that is closest to the abnormal sample of the predicted result; mean represents the mean, and Dev represents the deviation.

8. The anomaly interpretation method according to claim 1, characterized in that: Also includes: If the abnormality of the feature that does not exist in the abnormal sample of the prediction result is higher than the preset feature abnormality threshold, the preset number of features with the largest abnormality will be taken as abnormal features.

9. An abnormality interpretation device, characterized in that: include: a sample classification module for dividing, based on a training sample set for training an anomaly detection model and according to a prediction result of the anomaly detection model on each training sample, the training sample set into a training result abnormal sample set and a training result normal sample set, wherein the training result abnormal sample set includes training samples determined to be abnormal by the anomaly detection model; A local anomaly limit calculation module, configured to calculate a local anomaly limit value based on the outlier degree of each normal training result sample relative to other normal training result samples in the normal training result sample set; Anomaly prediction module, used to predict the input sample to be detected by using the trained anomaly detection model to obtain a prediction result; an anomaly type determination module, configured to, in response to a sample to be detected being predicted by the anomaly detection model as a predicted result abnormal sample, compare the outlier degree of the predicted result abnormal sample with each of the training result normal samples with the local anomaly limit value to determine whether the predicted result abnormal sample is a global or local anomaly type; An abnormality calculation module, configured to calculate the abnormality of each feature in the abnormal sample of the predicted result based on the abnormality type and the abnormality weights of the features contained in the abnormal sample of the predicted result, wherein the abnormality weights include abnormality weights corresponding to strong and weak correlation features determined based on the training sample set, or preset weights; wherein the global abnormality type and the local abnormality type are respectively used to determine different correlation sample ranges of the abnormal sample of the predicted result in the normal sample set of the training result, and each normal sample of the training result within the correlation sample range is used as a relevant sample for calculating the abnormality of the feature; An abnormal feature determination module, used to determine a feature in the prediction result prediction sample whose abnormality is higher than a preset feature abnormality threshold as an abnormal feature; The abnormality explanation module is used to generate an abnormality explanation for analyzing the abnormal sample of the prediction result based on the determined abnormal features.

10. A computer device, characterized in that: include: communicator, memory and processor; The communicator is used for communicating with the outside; The memory is used to store program instructions; The processor is configured to run the program instructions to perform the abnormality interpretation method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that Program instructions are stored, and the program instructions are executed to perform the abnormality interpretation method according to any one of claims 1 to 8.

Images