Anti-aging efficient malicious app detection method for constructing api correlation confidence

Abstract

The application relates to an anti-aging high-efficiency malicious APP detection method for constructing API correlation confidence, and belongs to the technical field of cyberspace security. First, the method is used for coping with the deviation caused by the continuous upgrading of an Android system on malicious software analysis by performing layer abstraction on API package names in APK files; second, the method is used for extracting high-level behavior semantics of software by calculating the correlation confidence between APIs; and finally, the method is used for completing the detection of malicious software by constructing classifiers for APKs in different release periods, selecting representative classifiers to learn the behavior patterns between API combinations, and completing the detection of malicious software. Current malicious software detection methods have weak software intention representation and anti-aging capability, and are difficult to detect continuously evolving and variant malicious software. The application establishes a high-level behavior intention representation mode, proposes an anti-aging strategy for a malicious software classifier, and can effectively detect continuously evolving malicious software, and has high detection efficiency and anti-aging property.

Description

Technical Field

[0001] This invention belongs to the field of cyberspace security technology and relates to an anti-aging and efficient malicious APP detection method for constructing API association confidence. Background Technology

[0002] Based on the different methods of data acquisition, Android malware detection methods can be broadly divided into dynamic detection methods and static detection methods.

[0003] 1. Dynamic Android malware detection methods involve running the software to obtain its behavioral data, thereby determining whether it is malware. First, this type of method requires running the software on an Android physical device or in a sandbox to acquire behavioral data, such as user-level API calls, kernel-level API calls, and network traffic data. Then, machine learning algorithms are used to analyze the data and obtain the detection results. This type of method has advantages in combating attack techniques such as code obfuscation and dynamic loading. However, it suffers from inherent drawbacks such as low code coverage and high resource consumption, resulting in low detection accuracy and making it difficult to meet the industrial needs of efficient and rapid detection scenarios.

[0004] 2. Static analysis methods refer to malware detection by directly analyzing the static data contained in APK files. These methods directly analyze the source code of the installation package file, AndroidManifest.xml, and other files to uncover the characteristics of malicious code and build a detector. Compared to dynamic analysis methods, static analysis methods have higher code coverage and can provide multi-layered analysis data, ensuring accurate detection results. Therefore, this type of method has become a focus of interdisciplinary research in machine learning and malicious app analysis, and the research work of this invention also focuses on static analysis methods. Based on the feature construction method of the analysis process, existing static analysis methods can be divided into detection methods based on statistical features and analysis methods based on behavioral semantics. The characteristics and existing problems of these two methods will be analyzed in turn below.

[0005] (1) Detection methods based on statistical features: The method that can effectively distinguish between the features of malicious software and normal software by screening features through statistical methods is called the statistical feature detection method.

[0006] (2) Behavioral semantics-based detection methods: In response to the ever-evolving malicious apps, researchers have proposed a large number of malware detection methods based on semantic analysis. These methods aim to learn the behavioral intent of the software and realize software representation at a high-level behavioral semantic level.

[0007] Furthermore, production models experience aging. This is because the statistical properties of the target variable the model attempts to predict change unpredictably over time, including abrupt, gradual, and iterative changes. As time progresses, the model's predictive accuracy decreases, and most Android malware detection models age rapidly.

[0008] In summary, based on the analysis of the existing work, the existing static malware detection methods have the following problems: (1) Statistical feature-based methods generally use the structured data already present in the installation package as the analysis material. These methods rely only on surface-level statistical features and cannot establish high-level behavioral intent representation patterns, making them difficult to cope with continuously evolving and mutating malware. (2) Most existing behavioral semantic-based methods rely on data such as program function call graphs or control flow graphs. The graph construction and analysis process is highly complex, resulting in low detection efficiency. In addition, there is room for improvement in the comprehensiveness of semantic construction in these methods. (3) Existing static malware detection methods suffer from model aging, resulting in a gradual decline in the accuracy of malware detection models after they are put into production. Summary of the Invention

[0009] The purpose of this invention is to address the aforementioned problems in malicious app detection research by proposing an anti-aging and efficient malicious app detection method that constructs API association confidence.

[0010] The design principle of this invention is as follows: First, the method addresses the biases caused by the continuous upgrades of the Android system to malware analysis by abstracting the API package names in the APK file layer by layer; second, it extracts the high-level behavioral semantics of the software by calculating the association confidence between APIs; finally, it builds classifiers for APKs released at different times, selects representative classifiers to learn the behavioral patterns between API combinations, and completes the detection of malware.

[0011] The technical solution of the present invention is achieved through the following steps:

[0012] Step 1, Preprocessing;

[0013] Step 1.1: Unzip the APK file to obtain the DEX file;

[0014] Step 1.2: Decompile the DEX files to obtain the Smali code for each APK;

[0015] Step 2, API abstraction;

[0016] Step 2.1: Extract all developer-defined methods from the Smali code of the APK and generate a method dictionary for each APK;

[0017] Step 2.2: In each method, retrieve the API call instructions, including invoke-virtual, invoke-direct, invoke-static, and invoke-super, and obtain all API information called by the instructions;

[0018] Step 2.3: Use API package names arranged in a hierarchical manner to represent specific APIs, and remove high-level attributes with fine-grained meanings from the package names;

[0019] Step 3, Feature extraction: In the abstracted API set, calculate the association confidence between two abstract APIs for each abstract API to generate the APK confidence matrix;

[0020] Step 4, classifier prediction;

[0021] Step 4.1: Obtain the APKs released within the fixed sliding time window and input their corresponding confidence matrix;

[0022] Step 4.2: Build a new classifier model for each sliding time window;

[0023] Step 4.3, check if the new classifier model is aging: if so, cluster all historical models, extract representative models from different clusters, and proceed to step 4.5; if not, proceed to step 4.4.

[0024] Step 4.4, Model similarity measurement, determines whether the new model is similar to the historical model: if so, delete the old model and save the new model to the historical model list; if not, only save the new model to the historical list.

[0025] Step 4.5: Representative models in the historical list output prediction results using a weighted majority voting method.

[0026] Beneficial effects

[0027] Compared to the MaMaDroid method (Onwuzurike L, Mariconti E, Andriotis P, et al. MaMaDroid: Detecting android malware by building markov chains of behavioral models (extended version)[J]. ACM Transactions on Privacy and Security (TOPS), 2019, 22(2):14.), this invention can effectively detect continuously evolving malware and has a certain degree of anti-aging properties. In addition, the process of constructing the function call graph requires parsing the global call relationships of all APIs in the APK file. The method proposed in this invention avoids this time-consuming operation, thus possessing higher detection efficiency and detection stability. Attached Figure Description

[0028] Figure 1 The flowchart of the anti-aging and efficient malicious APP detection method for constructing API association confidence is shown in this invention.

[0029] Figure 2 This is a schematic diagram illustrating the confidence level associated with the abstract API of this invention.

[0030] Figure 3 This is a schematic diagram of the confidence matrix of the present invention.

[0031] Figure 4 This is a flowchart of the classifier prediction process of the present invention. Detailed Implementation

[0032] To better illustrate the purpose and advantages of the present invention, the implementation methods of the present invention will be further described in detail below with reference to examples.

[0033] The datasets are sourced from three databases: Drebin, AMD, and Androzoo, as detailed in Table 1. The Drebin database contains 5560 malicious samples from 179 families, all dating between 2010 and 2012. Additionally, 5945 normal samples with the same age distribution as those in Drebin were obtained from the Androzoo database, and the dataset #Drebin was constructed using both malicious and normal samples. The AMD database contains 20843 malicious samples dating between 2010 and 2016, belonging to 71 malicious families. Furthermore, 20519 normal samples with the same age distribution were obtained from the Androzoo database, and the dataset #AMD was constructed using both malicious and normal samples. During actual APK processing, due to package decompression or file parsing errors, some APK samples could not be correctly converted into feature vectors. Therefore, in the #Drebin dataset (containing 11505 samples), 11331 samples were successfully preprocessed (5448 malicious samples and 5883 normal samples). In the dataset #AMD (containing 41,362 samples), 40,533 samples were successfully preprocessed (20,583 malicious samples and 19,950 normal samples).

[0034] Table 1. Overview of Experimental Dataset Information for Method Characteristic Analysis

[0035]

[0036] This experiment was conducted on a computer with the following specifications: Intel(R) Core(TM) i7-6700, CPU 3.40GHz, 8GB RAM, Windows 7 64-bit operating system, and Python 3.6, scikit-learn 0.22, and Androguard 3.3.5 programming tools.

[0037] Malware detection is a binary classification problem; therefore, the experiment uses the evaluation metrics commonly used in classification problems, as shown in Table 2. In the table, FP represents the number of applications that the classifier incorrectly classifies as malicious; FN represents the number of applications that the classifier incorrectly classifies as normal; TP represents the number of applications correctly classified as malicious; and TN represents the number of applications correctly classified as normal.

[0038] Table 2 Evaluation metrics used in malware detection experiments

[0039]

[0040] The specific procedure for this experiment is as follows:

[0041] Step 1, Preprocessing;

[0042] Step 1.1: Unzip the APK file to obtain the DEX file;

[0043] Step 1.2: Decompile the DEX files to obtain the Smali code for each APK;

[0044] Step 2, API abstraction;

[0045] Step 2.1: Extract all developer-defined methods from the Smali code of the APK and generate a method dictionary for each APK;

[0046] Step 2.2: In each method, retrieve the API call instructions, including invoke-virtual, invoke-direct, invoke-static, and invoke-super, and obtain all API information called by the instructions;

[0047] Step 2.3: Use API package names arranged in a hierarchical manner to represent specific APIs, and remove high-level attributes with fine-grained meanings from the package names;

[0048] Taking "Android.telephone.gsm" as an example, level one "Android" indicates that this API belongs to the Android system functions; level two "telephone" indicates that this API has functions related to telephone operations; level three "gsm" indicates that this function provides system services using specific GSM telephone functions. Therefore, the API can be further abstracted by removing higher-level attributes with finer-grained meanings. In the actual API abstraction process, the first two levels of attributes are retained, abstracting the 443 packages in the Android system API into 73. Adding custom APIs and obfuscated APIs, a total of 75 abstract APIs are included.

[0049] Step 3, Feature extraction: In the abstracted API set, calculate the association confidence between two abstract APIs for each abstract API to generate the APK confidence matrix;

[0050] The association strength between the two abstract APIs is calculated using confidence: Confidence(X→Y)=δ(X∪Y) / δ(X), where δ(X)=|{t i |X∈t i ,t i Let ∈T}|, where X and Y are two abstract APIs, δ(X) represents the number of times X occurs, and δ(X∪Y) represents the number of times X and Y occur simultaneously. In malware analysis, each abstract API is called an item, and the set of items can be represented as: I=[I1,I2,I3,I4,...,I... d], d = 75, if an APK sample contains k methods, then the APK sample can be represented as T = [t1, t2, t3, t4, ..., t k The association is established by focusing on the rules between two items, such as {java.io} -> {java.net}, {android.net} -> {org.xml}. The strength of the association between rules is calculated using confidence. All 75 abstract API calls are generated. Each association rule. The abstract API calculates the association confidence level relationship; see [link / reference]. Figure 2 To obtain the confidence matrix as a vector representation of the Android application, see... Figure 3 .

[0051] Step 4, classifier prediction. The classifier prediction process is detailed below. Figure 4 .

[0052] Step 4.1: Obtain the APKs released within the sliding time window and input their corresponding confidence matrix. The sliding time window can be set to 10 days.

[0053] Step 4.2: Build a new classifier model for each sliding time window;

[0054] Step 4.3, check if the new classifier model is aging: if so, cluster all historical models, extract representative models from different clusters, and proceed to step 4.5; if not, proceed to step 4.4.

[0055] The method for detecting model aging involves determining whether the model's classification accuracy is less than 90%. If it is, the model is considered aged; if it is not, the model is considered normal. The model clustering method involves all historical models predicting APKs released within a time window, with a prediction result of 1 indicating correctness and 0 indicating incorrectness. If the number of models in the historical model list is M, and the number of APKs within the time window is b, then the number of prediction results is M × b. The prediction results are clustered using the expectation-maximization algorithm; prediction results within the same cluster belong to the same model group.

[0056] Step 4.4, Model similarity measurement, determines whether the new model is similar to the historical model: if so, delete the old model and save the new model to the historical model list; if not, only save the new model to the historical list.

[0057] The similarity calculation method uses the Q-statistic between classifiers, and its calculation formula is: Q i,j =(N 1，1 N 0，0 -N 0，1 N 1 ，0 ) / (N 1，1N 0，0 +N 0，1 N 1，0 ), where N a,b This represents the number of samples where the classification results of two classifiers are combined on the same sample, according to classifier c. i Classified as a, according to classifier c j The classification is 'b', with 1 for correct classification and 0 for incorrect classification. The Q value varies between -1 and 1. A positive Q value indicates that the model classifies the same sample correctly or incorrectly simultaneously; a negative Q value indicates that the same sample is classified incorrectly. A threshold θ is set to determine the similarity of the models. If Q ≥ θ... ， If the classifier model is similar to a historical model, the new model is added to the historical model list, and the similar historical models are deleted. If Q < θ ， The new model will only be saved in the historical model list, and θ can be set to 0.5.

[0058] Step 4.5: Representative models in the historical list output prediction results using a weighted majority voting method.

[0059] Test Results: The experiment demonstrated the effectiveness of the anti-aging and efficient malicious app detection method based on API association confidence on the Drebin and AMD public datasets. The performance results of the anti-aging and efficient malicious app detection method based on API association confidence under different classification algorithms on the two datasets are shown in Tables 3 and 4.

[0060] Table 3# Experimental results of different algorithms on the AMD dataset

[0061]

[0062] Table 4# Experimental results of different algorithms on the Drebin dataset

[0063]

[0064] To address the challenge of detecting continuously evolving malware, a detailed analysis and comparison of detection accuracy and efficiency with the MaMaDroid method further demonstrates the superior analytical efficiency of this invention while maintaining detection accuracy. Table 5 shows the results of the accuracy comparison experiment, displaying 36 sets of F-score test results for the two methods during malware classification. Table 6 shows the time efficiency test results; groups marked with "*" were excluded from the experiment due to abnormal samples with analysis times exceeding 20 minutes.

[0065] Table 5. Comparative Experiment Results of Malware Classification F-Score Test

[0066]

[0067] Note: In the table, "old" represents a normal app sample from 2014, and "new" represents a normal app sample from 2017. The numbers from 2013 to 2017 represent malicious app samples from that year.

[0068] Table 6. Comparative Experimental Detection Efficiency Test Results

[0069]

[0070] Note: In the table, A refers to the experimental results of the MaMaDroid method, and B refers to the experimental results of this invention.

[0071] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting malicious apps that is resistant to aging and highly efficient, based on API association confidence, characterized in that... The method includes the following steps: Step 1, Preprocessing; Step 1.1: Unzip the APK file to obtain the DEX file; Step 1.2: Decompile the DEX files to obtain the Smali code for each APK; Step 2, API abstraction; Step 2.1: Extract all developer-defined methods from the Smali code of the APK and generate a method dictionary for each APK; Step 2.2: In each method, retrieve the API call instructions, including invoke-virtual, invoke-direct, invoke-static, and invoke-super, and obtain all API information called by the instructions; Step 2.3: Use API package names arranged in a hierarchical manner to represent specific APIs, and remove high-level attributes with fine-grained meanings from the package names; Step 3, Feature extraction: In the abstracted API set, calculate the association confidence between two abstract APIs for each abstract API to generate the APK confidence matrix; Step 4, classifier prediction; Step 4.1: Obtain the APKs released within the fixed sliding time window and input their corresponding confidence matrix; Step 4.2: Build a new classifier model for each sliding time window; Step 4.3, check if the new classifier model is aging: if so, cluster all historical models, extract representative models from different clusters, and proceed to step 4.5; if not, proceed to step 4.

4. Step 4.4, Model similarity measurement, determines whether the new model is similar to the historical model: if so, delete the old model and save the new model to the historical model list; if not, only save the new model to the historical list. Step 4.5: Representative models in the historical list output prediction results using a weighted majority voting method.

2. The anti-aging and efficient malicious APP detection method for constructing API association confidence as described in claim 1, characterized in that: In step 2.3, the API abstraction process requires removing high-level attributes with fine-grained meanings from the package name, retaining only the first two levels of attributes.

3. The method for detecting anti-aging and highly efficient malicious apps by constructing API association confidence as described in claim 1, characterized in that: In step 4.3, if the classifier model is aging, all historical models need to be clustered and representative models of different clusters need to be extracted. Among them, all historical models predict multiple APK confidence matrices within the sliding time window, and the cluster category of the predicted value is used as the category of its corresponding model.

4. The anti-aging and efficient malicious APP detection method for constructing API association confidence as described in claim 1, characterized in that: In step 4.4, the addition and deletion of models in the historical model list are managed through model similarity measurement. When the model list is full, the similarity of each pair of models in the list is measured, the two most similar models are selected, and one of the models is deleted.

Images