An Android malware detection method based on cost-sensitive learning

Abstract

The present application belongs to the field of Android malware detection in information security, and particularly relates to an Android malware detection method based on cost-sensitive learning. The method comprises the following steps: obtaining an original Android software sample data set; decompiling all samples in the data set to obtain source code files; extracting permissions and four components from the decompiled source code files as original features; calculating sample weights by using a sample-sensitive weight calculation method, and obtaining a sample-sensitive weight sequence; applying the sample-sensitive weight sequence to the feature selection stage, and selecting a sensitive feature subset by using a cost-sensitive feature selection method; applying the sample-sensitive weight sequence to the model training stage, and detecting Android malware from the sensitive feature subset by using a cost-sensitive learning-based classification method. The present application can improve the detection efficiency and accuracy of the Android malware classification model.

Description

Technical Field

[0001] This invention belongs to the field of Android malware detection in information security, and specifically relates to an Android malware detection method based on cost-sensitive learning, addressing the problem of imbalanced datasets in machine learning. Background Technology

[0002] Since the advent of the mobile internet era, the number of smartphone users has surged. Among them, the Android system, due to its open-source nature and portability, occupies a major share of the smartphone market. While mobile applications bring convenience to people's lives, they also expose users to more security threats. Due to the open-source nature and popularity of the Android system, as well as the current imperfections in application market management mechanisms, Android malware is constantly increasing, seriously threatening users' privacy, property security, and even personal safety. According to a G-DATA mobile security report, more than 2.5 million new malware programs were added to Android devices in 2021, highlighting the extremely serious mobile security problem.

[0003] Due to the surge in malware numbers and the inefficiency of manual detection, recent research has incorporated machine learning algorithms for Android malware detection. In classification problems, machine learning algorithms typically train predictive models using labeled sample datasets, automatically making predictions based on past observations. However, the composition of the training dataset significantly impacts the model's performance, including dataset quality, temporal bias, and imbalance. Imbalanced training datasets are a prevalent problem and a major concern in data analysis. Imbalanced training datasets can lead to classification models biased towards the majority class, making minority class samples highly susceptible to misclassification. This imbalance problem also exists in Android malware detection research. Statistics show that malware typically constitutes only 8%-12% of the entire dataset in real-world scenarios. Furthermore, misclassifying malware as benign and allowing its widespread dissemination can have incalculable consequences, while the impact is relatively smaller. Existing machine learning-based Android malware detection methods often employ imbalanced datasets, but the performance of detection models significantly degrades when the dataset is imbalanced. Therefore, reducing the impact of class imbalance and improving the detection accuracy of malicious samples is a key research challenge in Android malware detection.

[0004] Currently, numerous methods have been proposed in related research to address the class imbalance problem in datasets, mainly including data-level processing and algorithm-level processing. Data-level processing adjusts imbalanced datasets to balanced datasets by changing the number of samples in the majority or minority class. Among these, undersampling and oversampling techniques are the most commonly used. Algorithm-level processing improves existing classification algorithms to enhance the model's classification performance for minority class samples. The most commonly used method is the cost-sensitive method, which sets different misclassification costs for each class during the learning process. By increasing the misclassification cost for the minority class, the accuracy of the classifier in classifying the minority class is further improved. However, traditional cost-sensitive methods only embed cost sensitivity into the classification algorithm, ignoring the impact of dataset class imbalance on the feature selection process. This leads to the loss of a large number of important features during the feature selection stage, severely affecting the accuracy of subsequent classification models. Summary of the Invention

[0005] To address the issues of high false positive rates and insufficient generalization ability in Android malware detection models caused by imbalanced datasets in existing detection technologies, and to reduce the impact of dataset imbalance on Android malware detection models while improving their efficiency and accuracy, this invention proposes a cost-sensitive learning-based Android malware detection method. This method calculates sample sensitivity weights based on sample category and importance, and applies them to both feature selection and classification algorithms. After selecting a subset of sensitive features in the feature selection stage, a cost-sensitive classification algorithm is used for model training, thereby improving the detection efficiency of the classification model while increasing the detection accuracy for minority class samples.

[0006] An Android malware detection method based on cost-sensitive learning includes the following steps:

[0007] S1: Obtain the original Android software sample dataset, including malicious Android samples and benign Android samples; decompile all samples in the dataset to obtain source code files;

[0008] S2: Extract permissions and the four major components from the decompiled source code file as original features, and construct an original feature vector from the original features to represent the sample;

[0009] S3: Calculate the sample weights for all samples using the sample-sensitive weight calculation method, and obtain the sample-sensitive weight sequence;

[0010] S4: Apply the sample-sensitive weight sequence to the feature selection stage, and use a cost-sensitive feature selection method to select a subset of sensitive features;

[0011] S5: Apply the sample sensitive weight sequence to the model training stage, and use a classification method based on cost-sensitive learning to detect Android malware on the sensitive feature subset.

[0012] The beneficial effects of this invention are:

[0013] This invention proposes a cost-sensitive learning-based method for detecting Android malware. First, the method classifies samples based on their importance using the KNN algorithm, and calculates sample sensitivity weights by combining sample category and importance. This allows different samples to be assigned different importance values, thus avoiding interference from noisy data and improving model performance. Second, redundant and irrelevant features are filtered based on feature support, discriminative power, and correlation. Then, a cost-sensitive logistic regression algorithm is used to calculate feature importance based on the sample sensitivity weights, and features are selected according to their importance. This effectively reduces the impact of class imbalance on feature selection, improving the quality of the selected feature subset and model efficiency. Finally, a cost-sensitive classification model is trained based on the sample sensitivity weights to further reduce the impact of class imbalance on classification model performance. This method effectively reduces the performance error caused by class imbalance in the classification model, improving detection efficiency and accuracy. Attached Figure Description

[0014] Figure 1 This is a flowchart of an Android malware detection method based on cost-sensitive learning according to an embodiment of the present invention;

[0015] Figure 2 This is a diagram showing the overall structure of the detection model according to an embodiment of the present invention;

[0016] Figure 3 This is a flowchart of the feature selection algorithm in an embodiment of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] An embodiment of the present invention provides an Android malware detection method based on cost-sensitive learning, such as... Figure 1 As shown, the method includes the following steps:

[0019] S1: Obtain the original Android software sample dataset, including malicious Android samples and benign Android samples; decompile all samples in the dataset to obtain source code files;

[0020] S2: Extract permissions and the four major components from the decompiled source code file as original features, and construct an original feature vector from the original features to represent the sample;

[0021] S3: Calculate the sample weights for all samples using the sample-sensitive weight calculation method, and obtain the sample-sensitive weight sequence;

[0022] S4: Apply the sample-sensitive weight sequence to the feature selection stage, and use a cost-sensitive feature selection method to select a subset of sensitive features;

[0023] S5: Apply the sample sensitive weight sequence to the model training stage, and use a classification method based on cost-sensitive learning to detect Android malware on the sensitive feature subset.

[0024] In a preferred embodiment of the present invention, the method further includes step S6, wherein step S6 specifically includes:

[0025] S6: Divide the original dataset into a training set and a test set. Use the training set to train the model, and the test set to test the model's performance. Specifically, the training dataset uses the subset of sensitive features selected by the cost-sensitive feature selection algorithm in step S4 as input to train the classification model, and the test dataset uses the subset of sensitive features selected by the cost-sensitive feature selection algorithm in step S4 as input to test the classification model.

[0026] In this embodiment of the invention, an original Android software sample dataset is obtained, which contains malicious and benign samples. These samples need to be decompiled to obtain source code files such as AndroidManifest.xml, and software features can then be extracted from the source code files.

[0027] In this embodiment of the invention, step S2 requires decompiling all samples using the decompilation tool androguard. It mainly utilizes the androlyze.py module in androguard to extract features of subsequent permission features and features of the four major components, such as application permission requests, service broadcast components, bytecode features, graph features, etc.

[0028] In this embodiment, the androlyze.py module of AndroGuard is mainly used to extract features from permission features and the features of the four main components, and to construct the original feature vector. The feature vector construction process is as follows:

[0029] Step S21: Traverse all features of all samples in the original dataset and construct the original feature vector matrix. Each column of the matrix represents a feature, and each row represents a sample. The matrix values ​​consist of "0" and "1", where "0" indicates that the application does not contain this feature, and "1" indicates that the application contains this feature. All values ​​of the original feature vector matrix are initialized to "0".

[0030] Step S22: Traverse each sample in the original dataset. If the sample contains a feature from the original feature vector matrix, change the value of that feature in the matrix from "0" to "1".

[0031] Step S23: After traversing all samples in the original dataset, the original feature vector matrix is ​​obtained.

[0032] In this embodiment of the invention, step S3 requires calculating the sample weights for all samples using a sample-sensitive weight calculation method. Traditional cost-sensitive methods often directly embed the class proportion as the class weight into the algorithm, giving the same weight to samples of the same class, without considering the different importance of different samples within the same class. Therefore, this method combines class proportion and sample importance to comprehensively calculate the sample-sensitive weights. The sample-sensitive weight calculation process is as follows:

[0033] Step S31: Calculate the category weights based on the category proportions of the Android software samples, and use them as the initial weights for the samples. The sample weights are calculated as follows:

[0034]

[0035] Where, N a N represents the total number of samples. y This represents the total number of samples in category y, which includes both malicious Android samples and benign Android samples. The category weights W0 and W1 for benign and malicious samples are then calculated.

[0036] Step S32: Quantify the distance between samples based on the K-nearest neighbor algorithm and classify the samples into three types: noisy samples, safe samples, and dangerous samples.

[0037] In some embodiments, the specific method for classifying sample types can be as follows:

[0038] If the majority class sample accounts for more than 1 / 2 of the k nearest neighbor samples, the sample is considered a dangerous sample. If the majority class sample accounts for less than or equal to 1 / 2 of the k nearest neighbor samples, the sample is considered a safe sample. If all k nearest neighbor samples are majority class samples, the sample is considered a noisy sample.

[0039] Step S33: Calculate the initial weights of the samples based on the class imbalance ratio of the sample dataset, and adjust the sample weights according to the sample type to obtain the sample sensitivity weight sequence. The sample weight allocation scheme is as follows:

[0040] S331: First, assign initial weights to samples based on whether they are malicious or benign samples. If sample X... i If the sample is benign, the initial weight is W0; otherwise, it is W1. S332: If sample X... i If the sample is noisy, then the initial sample weights are reset to 0; if sample X... i If sample X is a dangerous sample, then increase the initial sample weight; if sample X i If the sample is a safe sample, then the initial sample weights are maintained;

[0041] The formulas used for updating sample weights include:

[0042]

[0043] Among them, W c ' represents the updated initial weights of the sample, a represents the enhancement factor, and W c This represents the initial weight of the sample, which is the category weight calculated based on the category ratio of the Android software samples.

[0044] In a preferred embodiment of the present invention, if sample X i If the sample is noisy, the weights are reset to 0; if sample X... i For a dangerous sample, if the enhancement factor is set to a = 2, then the sample weight is increased to twice the initial weight; if sample X i For safe samples, the initial weights are maintained.

[0045] By employing the methods described above in this embodiment of the invention, different importance levels can be assigned to different samples, thereby avoiding interference from noisy data and improving model performance.

[0046] S333: Finally, obtain the dataset sample weight sequence w s Each sample is assigned a corresponding weight for feature selection and model training.

[0047] In this embodiment of the invention, step S4 requires using a cost-sensitive feature selection algorithm to select a feature subset. Traditional cost-sensitive methods only embed cost-sensitive weights into the classification algorithm, ignoring the impact of dataset imbalance on the feature selection process. Therefore, this method embeds cost-sensitive weights into the feature selection algorithm to reduce the impact of dataset imbalance on the classification model. The flowchart of the cost-sensitive feature selection algorithm is as follows. Figure 2 As shown. The cost-sensitive feature selection algorithm process is as follows:

[0048] Step S41: Calculate feature support and discriminancy, and use the sum of feature support and discriminancy as the feature weight. Based on this, set a weight threshold to filter features. Since Android applications involve a large number of features and feature dimensions, and may contain many redundant and irrelevant features, traditional cost-sensitive feature selection methods are easily affected by these irrelevant and redundant features, impacting feature selection performance. Therefore, this scheme first quickly filters irrelevant and redundant features based on feature support, discriminancy, and relevance before using cost-sensitive logistic regression for feature selection, avoiding interference with subsequent algorithms. Feature f i Support is expressed as Discrimination is expressed as The initial weights of features are represented using support and discriminant, and the calculation methods for support and discriminant are as follows:

[0049]

[0050]

[0051] Wherein, N(f) i B) and N(f i M) represent features f respectively i The number of times a feature appears in the benign and malicious sample sets, where i = 1, 2, ..., n, and n represents the total number of features. N(B) and N(M) represent the number of benign and malicious samples in the dataset, respectively.

[0052] Step S42: Calculate the correlation between features. Since two features with high correlation can provide similar information, a correlation threshold is set. For feature pairs with correlation higher than the threshold, one feature is randomly selected and removed from the feature set. The selection method is as follows:

[0053] For all feature combinations {f} in the feature set i ,f j The feature f is calculated using the Pearson correlation coefficient. i and f j The degree of correlation between them, c ij By setting the correlation threshold c threshold Select features, if c ij ≥c threshold Then feature f j Remove from the feature set. Feature x i and feature y i The formula for calculating the correlation is as follows:

[0054]

[0055] Where, x iand y i These represent two characteristic variables, and These represent the mean values ​​of the corresponding feature variables, and n is the total number of features.

[0056] In a preferred embodiment of the present invention, a feature correlation evaluation value can be used to remove features from the feature set. The feature correlation evaluation value can be defined as follows:

[0057]

[0058] Where n represents the number of features in the feature subset f. c represents the average correlation between each feature and the category in the feature subset f. ii c represents the average correlation between features in the feature subset f. ij For feature x i and feature y i The Pearson correlation coefficient.

[0059] This embodiment uses the number of features n in the feature subset, combined with the Pearson correlation coefficients between feature categories and between features, to calculate the feature evaluation function value, fully mining the implicit information between features and features, and between features and categories in the feature subset. This method can remove feature variables that are not relevant to category prediction and can eliminate the nonlinear effects of traditional Pearson correlation measurement.

[0060] For the feature removal process, the embodiments of the present invention provide the following process:

[0061] Step S421: Calculate the feature set (f1, f2, ..., f n Each feature f in ) d Feature correlation evaluation value J(f) d );

[0062] Step S422: If J(f) a )=max{J(f d If f )}, then feature f a Add to the output feature set F out and remove it from the input feature set F in Delete;

[0063] Step S423: Input feature set F in The features not selected are respectively compared with the output feature set F out The features in the data are matched to obtain the feature correlation evaluation value of the combined feature set after matching.

[0064] Step S424: Add the feature corresponding to the maximum combined correlation evaluation value to the output feature set F.out And remove this feature from the input feature set F in Delete;

[0065] Step S425: Determine the output feature set F out If the number of features reaches the preset threshold L, proceed to step S426; otherwise, proceed to step S424.

[0066] Step S426: Use the updated output feature set as the final optimal feature subset.

[0067] Step S43: Embed the sample-sensitive weight sequence calculated in step three as the misclassification cost of the samples into the logistic regression loss function to obtain a cost-sensitive logistic regression algorithm, and use this cost-sensitive logistic regression algorithm to calculate the importance of each feature. Finally, set an importance threshold to select features. The logistic regression function is as follows:

[0068]

[0069] Where, x i For the input data of the model, θ i The parameters are to be determined. The logistic regression function value P(Y=1|X) is the result of input feature set X=[x1,x2,…,x…]. n Given the condition that the sample class Y is 1, the conditional probability is given. The purpose of the logistic regression algorithm is to determine the conditional probability of each variable x. i The best fit parameter value θ i This is typically achieved using gradient descent iteratively. During the iteration process, a loss function is used to adjust the selected parameter θ for each iteration. i An evaluation is performed. The cost-sensitive logistic regression loss function is as follows:

[0070]

[0071] Where w s This represents the sample-sensitive weight sequence, where m is the total number of samples, and y is the weighted average weight. i h is the true label for sample i. θ (x i Let θ be the predicted label for sample i. j For each feature, there are parameters to be determined, and C is a hyperparameter that controls the degree of regularization.

[0072] In this embodiment of the invention, the sample-sensitive weight sequence calculated in step S3 is embedded in the classification algorithm in step S5 to increase the emphasis on minority class samples during model training, thereby obtaining a cost-sensitive classification algorithm. The model training process uses logistic regression, random forest, and adaptive augmentation algorithms, respectively.

[0073] In this embodiment of the invention, step S6 can employ a ten-fold cross-validation method for model training and testing. Specifically, the feature vector set is randomly divided into ten disjoint data groups. Each time, nine groups are selected as the training set, and the remaining group is used as the test set. Finally, the average of the ten results is taken as the final test data for model performance.

[0074] Figure 3 This is a diagram of an Android malware detection framework based on cost-sensitive learning, as described in an embodiment of the present invention. Figure 3 As shown, the detection framework mainly includes a weight calculation module, a feature selection module, and a model training module. After obtaining the Android application, the permission features and component features are extracted and input into the weight module. After processing, a sample weight sequence is obtained. The sample weight sequence is then input into the feature selection module and the model classification module. The feature selection module is used to filter out a feature subset, and the model training module is used to classify the filtered feature subset to obtain the corresponding classification result.

[0075] The above-described embodiments further illustrate the purpose, technical solution, and advantages of the present invention. It should be understood that the above-described embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made to the present invention within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for Android malware detection based on cost-sensitive learning, characterized in that, The method comprises the following steps: S1: obtaining an original Android software sample data set, including Android malicious samples and Android benign samples; and performing decompilation on all samples in the data set to obtain source code files; S2: extracting permissions and four components from the decompiled source code files as original features, and constructing an original feature vector to represent the samples; S3: calculating sample weights by using a sample sensitive weight calculation method, and obtaining a sample sensitive weight sequence; S4: applying the sample sensitive weight sequence to the feature selection stage, and selecting a sensitive feature subset by using a cost-sensitive feature selection method, which specifically comprises: calculating feature support and discriminability, and using the sum of the feature support and the discriminability as a feature weight, filtering the features by using a weight threshold; calculating the correlation degree between the features, and removing one feature from the feature set at random for the feature pair with a correlation degree higher than a threshold; embedding the calculated sample sensitive weight sequence into a loss function of a logistic regression as a misclassification cost, and calculating the importance of each feature by using the cost-sensitive logistic regression algorithm; S5: applying the sample sensitive weight sequence to the model training stage, and detecting Android malicious software by using a classification method based on cost-sensitive learning for the sensitive feature subset. 2.The Android malware detection method based on cost-sensitive learning of claim 1, wherein, The step S3 comprises calculating a class weight according to a class proportion of the Android software samples and taking the class weight as a sample initial weight; quantifying distances between the Android software samples based on a K-nearest neighbor algorithm, and classifying the samples into three types of noise samples, safe samples and dangerous samples; and adjusting the sample initial weight according to a sample type of the Android software samples to obtain a sample sensitive weight sequence. 3.The Android malware detection method based on cost-sensitive learning of claim 2, characterized in that, The sample initial weight is calculated in the following manner: ， wherein, represents the total number of samples, represents the total number of samples of category y, category y including Android malicious samples and Android benign samples.

4. The Android malware detection method based on cost-sensitive learning according to claim 2, characterized in that, The sample weight adjustment process comprises: (1) calculating an initial sample weight of a corresponding malicious class or benign sample according to a malicious sample or benign sample class of the sample; (2) if the sample is a noise sample, then the initial sample weight is set to 0; if the sample is a dangerous sample, then the initial sample weight is increased; and if the sample is a safe sample, then the initial sample weight is maintained. (3) Obtain the sample weight sequence of the dataset and give each sample a corresponding weight.

5. The Android malware detection method based on cost-sensitive learning according to claim 4, characterized in that, The formula used for updating the sample weight comprises: ， wherein, represents the updated sample initial weight, represents the improvement factor, represents the sample initial weight, i.e., the category weight calculated according to the category proportion of the Android software sample.

6. The Android malware detection method based on cost-sensitive learning according to claim 1, characterized in that, The calculation manner of the feature support and the feature discriminability comprises: ， ， wherein, with respectively represent the features the number of occurrences in the benign sample set and the malicious sample set, where i = 1, 2, …, n, n represents the total number of features; N(B) and N(M) represent the number of benign samples and malicious samples in the data set, respectively.

7. The Android malware detection method based on cost-sensitive learning according to claim 1, characterized in that, The logistic regression function is as follows: ， wherein, is the input data for the model, is the parameter to be solved; the value of the logistic regression function is the conditional probability that the sample class Y is 1 given the input feature set .

8. The Android malware detection method based on cost-sensitive learning according to claim 1, characterized in that, The cost-sensitive logistic regression loss function is as follows: ， wherein, denotes a sequence of sample sensitive weights, m is the total number of samples, is the true label of sample i, is the predicted label of sample i; is the parameter to be solved for each feature, C is a hyperparameter that controls the degree of regularization, and n represents the total number of features.

Images