Feature selection method and apparatus, and electronic device and medium

Abstract

The present disclosure provides a feature selection method, comprising: obtaining behavior data of a plurality of users; converting the behavior data of each user in the plurality of users to obtain a plurality of feature vectors, the plurality of feature vectors having the same data structure and each dimension corresponding to a behavior feature; based on the plurality of feature vectors, performing non-Gaussianity measurement and robustness measurement of non-Gaussianity on each behavior feature to obtain corresponding measurement results; and based on the measurement results, selecting the behavior feature corresponding to at least one dimension for detecting abnormal user behavior. The present disclosure also discloses a feature selection device, an electronic device and a computer readable storage medium.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and more particularly, to a feature selection method and device, an electronic device and a medium. BACKGROUND

[0002] Anomaly detection has been widely used in data mining, financial anti-fraud and many other fields. In the field of anomaly detection, the selection of which features to perform anomaly detection has always been one of the core factors that seriously affect the performance of algorithms. Good feature selection techniques can detect the anomaly patterns of target anomalies using only limited features, thereby significantly improving the efficiency of anomaly detection and effectively improving the effect of anomaly detection.

[0003] However, in the process of implementing the present disclosure, the inventors have found that the conventional method often improves the efficiency of anomaly detection and the effect of anomaly detection by introducing excessive information, but such a method cannot guarantee the preservation of effective information while causing feature information redundancy. SUMMARY

[0004] Therefore, the embodiments of the present disclosure provide a feature selection method and device capable of selecting features sensitive to anomalies and rich in information for anomaly detection.

[0005] One aspect of the embodiments of the present disclosure provides a feature selection method, comprising: obtaining behavior data of a plurality of users; converting the behavior data of each user in the plurality of users to obtain a plurality of feature vectors, the plurality of feature vectors having the same data structure and each dimension corresponding to a behavior feature; based on the plurality of feature vectors, performing non-Gaussianity measurement and robustness measurement of non-Gaussianity on each behavior feature to obtain corresponding measurement results; and based on the measurement results, selecting a behavior feature corresponding to at least one dimension for detecting abnormal user behavior.

[0006] According to the embodiments of the present disclosure, the non-Gaussianity measurement and the robustness measurement of non-Gaussianity on each behavior feature based on the plurality of feature vectors to obtain the corresponding measurement results comprises, for each behavior feature: obtaining a first probability distribution, the first probability distribution being a probability distribution of all feature values of the plurality of feature vectors in a single dimension; determining a first non-Gaussianity measurement coefficient based on the first probability distribution; obtaining a second probability distribution, the second probability distribution being a probability distribution of feature values excluding low-frequency feature values from the all feature values; determining a second non-Gaussianity measurement coefficient based on the second probability distribution; and calculating a difference value between the first non-Gaussianity measurement coefficient and the second non-Gaussianity measurement coefficient.

[0007] According to an embodiment of the present disclosure, the selecting, based on the metric result, the behavior feature corresponding to at least one dimension for detecting abnormal user behavior comprises: in response to the difference being greater than a preset value, taking the current behavior feature as a candidate feature for detecting abnormal user behavior.

[0008] According to an embodiment of the present disclosure, after the taking the current behavior feature as the candidate feature for detecting abnormal user behavior, the method further comprises: in response to the first probability distribution being a long-tail distribution, taking the current behavior feature as the candidate feature.

[0009] According to an embodiment of the present disclosure, the method further comprises: after obtaining all the candidate features, selecting a candidate feature satisfying a first preset condition; and for each candidate feature satisfying the first preset condition, in response to a first probability distribution corresponding to the candidate feature being a long-tail distribution, retaining the candidate feature.

[0010] According to an embodiment of the present disclosure, the method further comprises: selecting a candidate feature satisfying a second preset condition from the retained candidate features.

[0011] According to an embodiment of the present disclosure, before the obtaining the second probability distribution, the method further comprises: in response to the first probability distribution being a long-tail distribution, performing the operation of obtaining the second probability distribution.

[0012] Another aspect of the embodiments of the present disclosure provides a feature selection apparatus, comprising: an obtaining module configured to obtain behavior data of a plurality of users; a converting module configured to convert the behavior data of each user in the plurality of users to obtain a plurality of feature vectors, the plurality of feature vectors having the same data structure and each dimension corresponding to a behavior feature; a metric module configured to perform non-Gaussianity metric and robustness of non-Gaussianity metric on each behavior feature based on the plurality of feature vectors to obtain a corresponding metric result; and a selecting module configured to select, based on the metric result, the behavior feature corresponding to at least one dimension for detecting abnormal user behavior.

[0013] Another aspect of the embodiments of the present disclosure provides an electronic device, comprising: one or more processors; and a memory configured to store one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of the embodiments of the present disclosure.

[0014] Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method of the embodiments of the present disclosure.

[0015] Another aspect of the embodiments of the present disclosure provides a computer program, which comprises computer executable instructions for implementing the method of the embodiments of the present disclosure when executed.

[0016] According to the embodiments of the present disclosure, because the technical means of measuring the non-Gaussianity of the behavior features of the user in the anomaly detection scenario and the robustness of the non-Gaussianity are adopted, the technical problem that the reliability of the selected features is not high due to only measuring the non-Gaussianity in the related art is at least partially overcome, and thus the technical effect of improving the reliability of the selected features is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 An exemplary system architecture to which the feature selection method and device of the present disclosure can be applied is schematically shown;

[0019] Figure 2 A flowchart of the feature selection method according to the embodiments of the present disclosure is schematically shown;

[0020] Figure 3 A flowchart of measuring the non-Gaussianity of each behavior feature and the robustness of the non-Gaussianity according to the embodiments of the present disclosure is schematically shown;

[0021] Figure 4 A block diagram of the feature selection device according to the embodiments of the present disclosure is schematically shown; and

[0022] Figure 5 A block diagram of an electronic device suitable for implementing the feature selection method and device according to the embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It is to be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent, however, that one or more embodiments can be practiced without these specific details. In other instances, well-known structures and functions have not been described in detail in order to avoid obscuring the concepts of the present disclosure.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the term "includes" and tautological equivalents thereof, means that the named feature, step, operation, and / or component is included, but not to the exclusion of the presence or addition of one or more other features, steps, operations, and / or components.

[0025] All terms used herein including technical and scientific terms have the meanings commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein are not intended to have any ideologically or overly formalistic meanings, but should be interpreted in a manner consistent with the context of the present specification.

[0026] In the case where expressions such as "at least one of A, B, and C, etc." are used, it should be generally construed that the meaning is understood by one of ordinary skill in the art as it is commonly used (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.). In the case where expressions such as "at least one of A, B, or C, etc." are used, it should be generally construed that the meaning is understood by one of ordinary skill in the art as it is commonly used (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, a system having B alone, a system having C alone, a system having both A and B, a system having both A and C, a system having both B and C, and / or a system having A, B, and C, etc.).

[0027] Embodiments of the present disclosure provide a feature selection method capable of selecting features sensitive to anomalies and having high information content for anomaly detection, and a feature selection device capable of applying the method. The method includes obtaining behavior data of a plurality of users. The behavior data of each user in the plurality of users is converted to obtain a plurality of feature vectors, and the plurality of feature vectors have the same data structure and each dimension corresponds to a behavior feature. Based on the plurality of feature vectors, a non-Gaussianity measurement and a robustness measurement of non-Gaussianity are performed on each behavior feature to obtain corresponding measurement results. Based on the measurement results, a behavior feature corresponding to at least one dimension is selected for detecting an abnormal user behavior.

[0028] Figure 1 An exemplary system architecture to which the feature selection method and device of the present disclosure can be applied is schematically shown. It should be noted that, Figure 1 The shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0029] As Figure 1As shown, the system architecture 100 according to this embodiment can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, and the like.

[0030] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, and the like. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, and the like (only as examples).

[0031] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0032] The server 105 can be a server providing various services, such as a background management server providing support for websites browsed by users using the terminal devices 101, 102, 103 (only as an example). The background management server can analyze and process received user requests and the like, and feed back the processing results (such as web pages, information, or data, and the like obtained or generated according to user requests) to the terminal devices.

[0033] It should be noted that the feature selection method provided by the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the feature selection apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 105. The feature selection method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the feature selection apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the feature selection method provided by the embodiments of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or by other terminal devices different from the terminal devices 101, 102, or 103. Accordingly, the feature selection apparatus provided by the embodiments of the present disclosure can also be arranged in the terminal devices 101, 102, or 103, or in other terminal devices different from the terminal devices 101, 102, or 103.

[0034] It should be understood that, Figure 1The number of terminal devices, networks and servers in the system is merely illustrative. Any number of terminal devices, networks and servers can be provided according to implementation needs.

[0035] Figure 2 A flowchart of a feature selection method according to an embodiment of the present disclosure is illustratively shown.

[0036] As shown in Figure 2 The method may, for example, include operations S201-S204.

[0037] In operation S201, behavior data of a plurality of users is acquired.

[0038] It should be understood that the assumption of anomaly detection is that the activities of intruders are abnormal to the activities of normal subjects. According to this concept, for example, an "activity profile" of normal activities of a subject can be established, and the current activity of the subject is compared with the normal activity recorded in the "activity profile". When the current activity of the subject violates the statistical rule described in the "activity profile", it is considered that the current activity of the subject can be an "intrusion" behavior. The difficulty of anomaly detection lies in how to establish the "activity profile" and how to design a statistical algorithm so as not to regard normal operation as "intrusion" or ignore the real "intrusion" behavior. The feature selection method provided by the embodiments of the present disclosure can achieve the establishment of the "activity profile". And the feature selection method provided by the embodiments of the present disclosure can also achieve not regarding normal operation as "intrusion" and not ignoring the real "intrusion" behavior.

[0039] Since the behavior data of a user can represent the activity characteristics of the user, and the activity characteristics of a plurality of users can determine which user behavior is a real "intrusion" behavior. Therefore, the embodiments of the present disclosure acquire the behavior data generated by a plurality of users within a certain time range as a sample set of candidate features. Specifically, the behavior data of a user may, for example, be read from a relational data table.

[0040] In operation S202, the behavior data of each user in the plurality of users is converted to obtain a plurality of feature vectors, and the plurality of feature vectors have the same data structure and each dimension corresponds to a behavior feature.

[0041] It should be understood that the activities of any one user can exhibit a plurality of features in a period of time. Therefore, in order to facilitate feature analysis, the behavior data of each user can be expressed as a feature vector, so that each dimension of the feature vector corresponds to a feature. And in order to maintain the consistency of the feature vectors, the behavior data of all users in the sample set can be converted into a plurality of multi-dimensional feature vectors of the same format. Further, after the behavior data of each user is expressed as a corresponding feature vector, the behavior data of all users in a sample set can also be expressed in the form of a feature matrix.

[0042] Taking the application scenario of a social network as an example, specifically, the behavior data generated by operations such as access and adding a relationship between users in the social network can be converted into a feature vector required by an algorithm by various technical means such as business experience or graph embedding, so as to complete the data basis preparation for feature selection.

[0043] It should be noted that the graph embedding technology using low-order approximation in this process often has good results in retaining more local information. For example, the user can be taken as a node type, and the access between two user nodes can be taken as an edge type to construct the graph data of the access between users. Then, the graph embedding technology such as LINE that can effectively retain local information is used to embed the graph data of the access between users into a 128-dimensional space (that is, the behavior data of each user in the social network can be expressed by a 128-dimensional feature vector), which completes the data expression.

[0044] In operation S203, based on the plurality of feature vectors, the non-Gaussianity of each behavior feature and the robustness of the non-Gaussianity are measured to obtain corresponding measurement results.

[0045] It should be understood that in the field of anomaly detection, there are two definitions of anomaly: 1. Abnormal data and normal data are generated by different patterns and are different in some properties; 2. Abnormal data accounts for a very small proportion of the overall data, that is, the overall data is not contaminated.

[0046] Continuing to take the application scenario of a social network as an example, in order to make a profit maliciously, abnormal users and normal users exhibit different behavior characteristics. A large number of abnormal users will concentrate on accessing users or topics with low popularity, and will cheat normal users into clicking and accessing by increasing the popularity and confidence of target users and target topics in the social network in a short time to achieve a profit. However, in terms of the entire social network, such behavior-specific users are relatively rare.

[0047] Based on the above definition, it can be determined that the features that can better reflect the properties of abnormal users should conform to certain specific data distribution patterns, in which the values of most data are similar, but the values of local data are significantly different from the values of most data. Therefore, the non-Gaussianity of the behavior characteristics of the user can be measured to determine whether the behavior characteristics of the user conform to a certain specific data distribution pattern.

[0048] In order to measure whether the behavior characteristics of the user conform to such a specific data distribution pattern, the embodiments of the present disclosure use a series of measurement coefficients with measurement capability, including but not limited to kurtosis coefficient, skewness coefficient, etc.

[0049] The kurtosis is also referred to as a peakness, and is a characteristic number used to represent the peak height of a probability distribution curve at the average value. The kurtosis reflects the sharpness or flatness of the top end of the probability distribution curve. The probability distribution with a larger kurtosis has more data in a similar value range, and a small amount of data has a large difference.

[0050] In addition, the skewness is also referred to as a bias coefficient, and is a statistical parameter used to represent the asymmetry of a random distribution. Since the kurtosis can only reflect the average and dispersion of the probability distribution curve, but cannot reflect the symmetry (i.e., the skewness), another parameter, i.e., the bias coefficient, is introduced. The larger the absolute value of the skewness is, the more serious the skewness is.

[0051] Taking the social network as an example, after the metric coefficient used to evaluate the advantages and disadvantages of the features is determined, the features can be selected based on the plurality of feature vectors obtained in the foregoing. For example, for a 128-dimensional feature vector, each dimension can be regarded as a feature, and the feature values in each dimension are traversed dimension by dimension, and then the metric coefficient of each feature is calculated.

[0052] The calculation formula of the kurtosis is as shown in formula (1),

[0053]

[0054] where n represents the number of users in the sample set, x i represents the feature value corresponding to the i th user, represents the average value of the feature values of the n users.

[0055] For any feature dimension, if it is found through calculation that all the feature values in this feature dimension as a whole present a non-Gaussian distribution (i.e., a specific probability distribution), it is preliminarily considered that the feature corresponding to the dimension feature can be used as an effective feature for detecting anomalies, and therefore the dimension feature can be listed as a candidate feature.

[0056] In the embodiment of the present disclosure, the feature selection method based on the non-Gaussian measure does not require any prior knowledge, and can effectively solve the zero sample problem in the cold start scenario, and can greatly reduce the cold start cost in the anomaly detection scenario. However, this method only considers the probability distribution of a single feature, and considers that the feature with a strong long-tail distribution can better reflect the information of the abnormal value in the global data, but does not consider whether the non-Gaussianity of the data in the single feature is caused by the abnormal value, and therefore a large number of feature dimensions that cannot represent the anomaly are still selected, resulting in unnecessary calculation cost.

[0057] Based on this, the feature selection method provided by the embodiments of the present disclosure not only measures the non-Gaussianity of the features to evaluate the effectiveness of the features for anomaly detection, but also further measures the robustness of the non-Gaussianity of the features to evaluate the sensitivity of the features for anomaly detection, so as to select the features that are more likely to reflect the abnormal pattern from the probability distribution of the features, and further select the features with lower robustness (the lower the robustness, the higher the sensitivity of the features for anomaly detection) as the final features for anomaly detection.

[0058] Figure 3 The flowchart of measuring the non-Gaussianity and the robustness of the non-Gaussianity of each behavior feature is schematically shown according to the embodiments of the present disclosure.

[0059] As shown in Figure 3 As an optional embodiment, operation S203 measures the non-Gaussianity and the robustness of the non-Gaussianity of each behavior feature based on the plurality of feature vectors, and obtains the corresponding measurement results. For example, operations S301-S305 can be performed for each behavior feature.

[0060] In operation S301, a first probability distribution is obtained, which is the probability distribution of all feature values of the plurality of feature vectors in a single dimension.

[0061] Specifically, for each feature dimension, all feature values of the plurality of feature vectors in the dimension can be traversed, so as to calculate the probability distribution of the feature values of each feature dimension, and obtain a plurality of first probability distributions.

[0062] In operation S302, a first non-Gaussianity measurement coefficient is determined based on the first probability distribution.

[0063] For each first probability distribution, the calculation formula (1) of the kurtosis coefficient is as above, which will not be repeated here.

[0064] In operation S303, a second probability distribution is obtained, which is the probability distribution of the feature values excluding the low-frequency feature values from all feature values.

[0065] It should be understood that for a single-dimension feature, if the first probability distribution of the feature values in a feature dimension is a non-Gaussian distribution, it is initially considered that the feature corresponding to the feature dimension is effective for anomaly detection and can be used in an anomaly detection scenario. In order to further determine whether the non-Gaussianity here is caused by an outlier, it can be judged whether the second probability distribution of the feature values in the feature dimension after removing the low-frequency feature values has become a Gaussian distribution. If the second probability distribution has become a Gaussian distribution, it is considered that the non-Gaussianity of the corresponding first probability distribution is caused by an outlier, and the robustness of the non-Gaussianity is low, so the sensitivity of the feature corresponding to the feature dimension to anomaly detection is high, and finally it can be used in an anomaly detection scenario. If the second probability distribution is still a non-Gaussian distribution, it is considered that the non-Gaussianity of the corresponding first probability distribution is not caused by an outlier, and the robustness of the non-Gaussianity is high, so the sensitivity of the feature corresponding to the feature dimension to anomaly detection is low, and it is actually not suitable for being used in an anomaly detection scenario, and therefore can be deleted from the candidate features.

[0066] It can be seen that through the embodiments of the present disclosure, more reasonable features can be selected for an anomaly detection scenario.

[0067] In addition, in the embodiments of the present disclosure, for continuous variables, low-frequency feature values can be removed by an equidistant binning algorithm. Specifically, feature values in two to four bins with low proportions can be removed. For discrete variables, 20% to 40% of feature values with low frequencies can be directly removed.

[0068] It should be noted that in the embodiments of the present disclosure, when the non-Gaussianity and the robustness of the non-Gaussianity of the features are measured, the first probability distribution and the second probability distribution of each feature can be calculated first, and then it is determined which features can be used in an anomaly detection scenario, or for each feature, the first probability distribution of the feature can be calculated first, and then the second probability distribution of the feature is further calculated after it is determined that the first probability distribution is a non-Gaussian distribution and a long-tail distribution, and then it is determined based on the second probability distribution that the feature can be used in an anomaly detection scenario.

[0069] For the latter method, as an optional embodiment, before the operation S303 of acquiring the second probability distribution, for example, it can also include: determining whether the first probability distribution is a long-tail distribution, wherein if it is determined that the first probability distribution is a long-tail distribution, the operation of acquiring the second probability distribution is performed.

[0070] It should be noted that if a feature's probability distribution follows a long-tailed distribution, it is considered to have a high level of anomaly information. Conversely, if a feature's probability distribution does not follow a long-tailed distribution, it is considered to have a low level of anomaly information. Therefore, when selecting features, features with higher anomaly information content can be chosen for anomaly detection to improve its effectiveness.

[0071] Specifically, the kurtosis coefficient can be calculated based on the first probability distribution, and the kurtosis coefficient can be used to determine whether the first probability distribution is a long-tailed distribution. The larger the kurtosis coefficient, the longer the long tail of the first probability distribution. Therefore, in this embodiment, a first probability distribution with a kurtosis coefficient greater than a certain threshold can be defined as a long-tailed distribution.

[0072] In operation S304, the second non-Gaussianity metric coefficient is determined based on the second probability distribution.

[0073] For each second probability distribution, taking the kurtosis coefficient as an example, its calculation formula (1) is as above, and will not be repeated here.

[0074] In operation S305, the difference between the first non-Gaussianity metric coefficient and the second non-Gaussianity metric coefficient is calculated.

[0075] Taking kurtosis coefficients as an example, assuming the kurtosis coefficient calculated based on the first probability distribution is Cv1 and the kurtosis coefficient calculated based on the second probability distribution is Cv2, then the difference between the two kurtosis coefficients is (Cv1-Cv2). The larger the difference (Cv1-Cv2), the greater the effectiveness of the corresponding feature in anomaly detection.

[0076] Since anomalous information is not the only factor causing a high level of non-Gaussianity, special data scenarios often lead to anomalous non-Gaussianity of features. Therefore, to improve the reliability of feature selection, embodiments of this disclosure propose a feature selection framework that simultaneously considers the non-Gaussianity of candidate features and the robustness of non-Gaussianity to anomalies.

[0077] In operation S204, based on the measurement results, behavioral features corresponding to at least one dimension are selected for detecting abnormal user behavior.

[0078] Specifically, based on the above measurement results, features that satisfy long-tail distribution and have low robustness due to non-Gaussianity can be selected as features for detecting abnormal user behavior.

[0079] Since the larger the difference (Cv1-Cv2), the greater the effectiveness of the corresponding feature in anomaly detection. Therefore, as an optional embodiment, operation S204, based on the measurement results, selects behavioral features corresponding to at least one dimension for detecting abnormal user behavior, which may include the following operations.

[0080] If the difference is greater than a preset value, the current behavioral feature is used as a candidate feature for detecting abnormal user behavior.

[0081] In this embodiment of the disclosure, the robustness assessment of non-Gaussianity is started by calculating the sensitivity of each feature to abnormal information to evaluate the potential contribution of each feature to anomaly detection, and then the features that cause non-Gaussianity due to abnormal information are selected as feature candidates.

[0082] Continuing with the example of social networks, we first analyze the 128-dimensional features. After removing outliers from a single-dimensional feature, if the non-Gaussianity of that feature remains high, it is considered that the feature is too robust to anomalies, indicating that it contains insufficient outlier information and should not be selected. Conversely, if the non-Gaussianity of the feature disappears, meaning it has changed from a non-Gaussian distribution to a Gaussian distribution, it is considered that the feature is less robust to anomalies, indicating that it contains a lot of outlier information and can be selected.

[0083] Furthermore, as an optional embodiment, after using the current behavioral features as candidate features for detecting abnormal user behavior, the method may also include, for example, the following operations.

[0084] Since the first probability distribution is a long-tailed distribution, the current behavioral features are continued to be used as candidate features.

[0085] Specifically, in this embodiment of the disclosure, after determining that the first probability distribution of a feature is a non-Gaussian distribution and the second probability distribution of the feature is Gaussian, it is further verified whether the first probability distribution of the feature is a long-tailed distribution. If the first probability distribution of the feature is a long-tailed distribution, the feature is ultimately determined as a feature for anomaly detection. Otherwise, if the first probability distribution of the feature is not a long-tailed distribution, the feature is abandoned for anomaly detection.

[0086] Through the embodiments disclosed herein, it can be ensured that the selected features are sensitive to anomalies and that the selected features contain sufficient anomaly information, thereby avoiding the risk that the selected features may fail to effectively express the anomaly characteristics, and thus ensuring the stable reliability of the selected features in the anomaly detection task.

[0087] Alternatively, as an optional embodiment, the method may also include, for example, the following operations.

[0088] After obtaining all candidate features, select the candidate features that meet the first preset condition.

[0089] For each candidate feature that satisfies the first preset condition, in response to the first probability distribution corresponding to the candidate feature being a long-tailed distribution, the candidate feature is retained.

[0090] Specifically, in this embodiment, a first probability distribution and a second probability distribution for each feature can be determined first, and then candidate features can be determined based on these first and second probability distributions. The candidate features here need to meet the following conditions: their first probability distributions are all non-Gaussian distributions, and their second probability distributions are all Gaussian distributions. Based on this, feature candidates that meet a first preset condition (e.g., features ranked in the top N (e.g., N=50) of robustness from low to high) are first selected from these candidate features, and then long-tail distribution judgment is performed.

[0091] To further reduce feature complexity while ensuring detection capabilities, this embodiment selects candidate features based on the non-Gaussianity and robustness of non-Gaussianity. These candidate features are then ranked based on kurtosis coefficients. The top N candidate features are selected from these features according to their kurtosis coefficients from largest to smallest for long-tail distribution judgment, thus completing the unsupervised feature selection process. This ensures the effectiveness of subsequent anomaly detection while effectively reducing computational costs.

[0092] Furthermore, as an optional embodiment, the method may also include, for example, selecting candidate features that satisfy a second preset condition from the retained candidate features.

[0093] Specifically, the number of features selected can be determined based on the specific scenario of anomaly detection. Experiments show that setting the selection ratio between 10% and 20% generally yields good results, especially when the number of features selected is small. This approach often ensures the effectiveness of subsequent anomaly detection while effectively reducing computational costs.

[0094] The embodiments disclosed herein can effectively reduce the feature dimensions involved in subsequent calculations. In scenarios with large amounts of data, this can significantly reduce computational overhead, thereby improving the applicability of anomaly detection algorithms in big data environments.

[0095] For example, the feature selection technique proposed in this disclosure can approximate the detection effect of using the full set of features in anomaly detection tasks such as identifying fake traffic, malicious groups, and upstream crimes in anti-money laundering on social networks, using only 5% to 10% of the original feature set. Due to the removal of a large number of redundant features, the computational cost of subsequent anomaly detection work can be significantly reduced, and the stability of subsequent algorithm performance can be improved. Furthermore, in specific tasks, the feature selection strategy provided in this disclosure considers the sensitivity of features to anomalies, thus favoring feature dimensions with anomalous business implications, which has a significant advantage in interpretability compared to existing feature selection techniques.

[0096] Figure 4A block diagram of a feature selection apparatus according to an embodiment of the present disclosure is shown schematically.

[0097] like Figure 4 As shown, the feature selection device 400 includes an acquisition module 410, a conversion module 420, a measurement module 430, and a selection module 440.

[0098] Specifically, module 410 is used to acquire behavioral data from multiple users.

[0099] The transformation module 420 is used to transform the behavioral data of each user among multiple users to obtain multiple feature vectors. The multiple feature vectors have the same data structure and each dimension corresponds to a behavioral feature.

[0100] The measurement module 430 is used to perform non-Gaussianity measurement and non-Gaussian robustness measurement on each behavioral feature based on multiple feature vectors, and obtain the corresponding measurement results.

[0101] Selection module 440 is used to select behavioral features corresponding to at least one dimension for detecting abnormal user behavior based on the measurement results.

[0102] It should be noted that the implementation methods of the apparatus portion of this disclosure are the same as or similar to those of the method portion, and will not be repeated here.

[0103] Any one or more of the modules according to embodiments of this disclosure, or at least a portion thereof, may be implemented in one module. Any one or more of the modules according to embodiments of this disclosure may be implemented by dividing them into multiple modules. Any one or more of the modules according to embodiments of this disclosure may be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules according to embodiments of this disclosure may be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0104] For example, any plurality of the acquisition module 410, conversion module 420, measurement module 430, and selection module 440 may be combined into one module / unit / subunit, or any one of these modules / units / subunits may be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits may be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the acquisition module 410, conversion module 420, measurement module 430, and selection module 440 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 410, conversion module 420, measurement module 430 and selection module 440 may be implemented at least partially as a computer program module that can perform corresponding functions when the computer program module is run.

[0105] It should be noted that the device implementation in the embodiments of this disclosure corresponds to or is similar to the method implementation in the embodiments of this disclosure. For a detailed description of the device implementation, please refer to the description of the method implementation, which will not be repeated here.

[0106] Figure 5 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0107] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage portion 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0108] RAM 503 stores various programs and data required for the operation of system 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0109] According to embodiments of this disclosure, system 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. System 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0110] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0111] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0112] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0113] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0115] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0116] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A feature selection method, comprising: Obtain behavioral data from multiple users; The behavioral data of each of the multiple users is transformed to obtain multiple feature vectors. The multiple feature vectors have the same data structure and each dimension corresponds to a behavioral feature. Based on the multiple feature vectors, a non-Gaussianity measure and a robustness measure of non-Gaussianity are performed on each behavioral feature to obtain corresponding measurement results. The non-Gaussianity measure is used to evaluate whether the behavioral data conforms to a specific data distribution pattern. The coefficients of the non-Gaussianity measure include at least one of the following: kurtosis coefficient and skewness coefficient. The robustness measure is used to determine whether the non-Gaussianity of the data in a single-dimensional feature is caused by outliers. Based on the measurement results, behavioral features corresponding to at least one dimension are selected for detecting abnormal user behavior; based on the multiple feature vectors, non-Gaussianity and robustness measures are performed on each behavioral feature to obtain corresponding measurement results, including for each behavioral feature: Obtain a first probability distribution, which is the probability distribution of all feature values ​​of the plurality of feature vectors in a single dimension; The first non-Gaussianity metric coefficient is determined based on the first probability distribution; Obtain a second probability distribution, which is the probability distribution of all feature values ​​excluding low-frequency feature values. The second non-Gaussianity metric coefficient is determined based on the second probability distribution; and Calculate the difference between the first non-Gaussianity metric coefficient and the second non-Gaussianity metric coefficient.

2. The method according to claim 1, wherein, The step of selecting behavioral features corresponding to at least one dimension for detecting abnormal user behavior based on the measurement results includes: In response to the difference being greater than a preset value, the current behavioral feature is used as a candidate feature for detecting abnormal user behavior.

3. The method according to claim 2, wherein, After considering the current behavioral features as candidate features for detecting abnormal user behavior, the method further includes: In response to the first probability distribution being a long-tailed distribution, the current behavioral feature is used as the candidate feature.

4. The method according to claim 2, wherein, The method further includes: After obtaining all candidate features, select the candidate features that meet the first preset condition; For each candidate feature that satisfies the first preset condition, in response to the first probability distribution corresponding to the candidate feature being a long-tailed distribution, the candidate feature is retained.

5. The method according to claim 4, wherein, The method further includes: Select candidate features that meet the second preset condition from the remaining candidate features.

6. The method according to claim 1, wherein, Before obtaining the second probability distribution, the following steps are also included: In response to the first probability distribution being a long-tailed distribution, the operation of obtaining the second probability distribution is performed.

7. A feature selection device, comprising: The acquisition module is used to acquire behavioral data from multiple users. The conversion module is used to convert the behavioral data of each of the multiple users to obtain multiple feature vectors. The multiple feature vectors have the same data structure and each dimension corresponds to a behavioral feature. The measurement module is used to perform non-Gaussianity measurement and non-Gaussian robustness measurement on each behavioral feature based on the multiple feature vectors, obtaining corresponding measurement results. The non-Gaussianity measurement is used to evaluate whether the behavioral data conforms to a specific data distribution pattern, and the measurement coefficient of the non-Gaussianity measurement includes at least one of the following: kurtosis coefficient and skewness coefficient. The robustness measurement is used to determine whether the non-Gaussianity of the data in a single-dimensional feature is caused by outliers. The selection module is used to select behavioral features corresponding to at least one dimension for detecting abnormal user behavior based on the measurement results. The measurement module is also used for: for each behavioral feature: Obtain a first probability distribution, which is the probability distribution of all feature values ​​of the plurality of feature vectors in a single dimension; The first non-Gaussianity metric coefficient is determined based on the first probability distribution; Obtain a second probability distribution, which is the probability distribution of all feature values ​​excluding low-frequency feature values. The second non-Gaussianity metric coefficient is determined based on the second probability distribution; and Calculate the difference between the first non-Gaussianity metric coefficient and the second non-Gaussianity metric coefficient.

8. An electronic device, comprising: One or more processors; as well as Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 6.

Images