Training method and detection method of malicious software detection model and related equipment

Abstract

The invention provides a malicious software detection model training method, a malicious software detection method and related equipment. The method comprises the following steps: constructing an uncertainty evaluation model; and training the uncertainty evaluation model through the obtained first data set to obtain a prediction probability set corresponding to each piece of training data in the first data set. And for the training data corresponding to each malicious software, based on the prediction probability set of the training data corresponding to the malicious software and a preset uncertainty index type, calculating to obtain a plurality of uncertainty indexes. According to the uncertainty indexes corresponding to each piece of malicious software, adopting a clustering algorithm to cluster the training data corresponding to all the malicious software, and dividing the training data into a first sample combination and a second sample combination; and based on the first sample combination, the second sample combination and the training data corresponding to the benign software, training a pre-constructed malicious software detection model. The malicious software detection performance of the model can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of software engineering testing technology, and in particular to a malware detection model training method, a detection method, and related equipment. Background Art

[0002] To achieve the goal of malware detection, a common practice is to model malware detection as a binary classification task, collecting an equal number of benign and malicious software as a training set. However, due to the wide variety of malicious families, malware datasets suffer from a serious long-tail distribution problem, where the number of malicious samples in the leading malicious families far exceeds the number of samples in the trailing families. This difference in sample distribution between malicious families allows the model to adequately fit malicious families with a large number of samples, resulting in good detection performance. Conversely, for malicious families with a small number of samples, the model fits poorly and performs poorly. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a malware detection model training method, detection method and related equipment to solve the impact of the long-tail distribution problem of malware datasets on the performance of the detection model.

[0004] Based on the above objectives, the first aspect of the present application provides a method for training a malware detection model, comprising:

[0005] Construct uncertainty assessment models;

[0006] Training the uncertainty assessment model using the acquired first data set to obtain a set of prediction probabilities corresponding to each training data in the first data set; wherein the first data set includes training data corresponding to malware and training data corresponding to benign software;

[0007] For each piece of malware corresponding to the training data, multiple uncertainty indicators are calculated based on the predicted probability set of the malware corresponding to the training data and the preset uncertainty indicator type;

[0008] Based on multiple uncertainty indicators corresponding to each malware, a clustering algorithm is used to cluster the training data corresponding to all malware into a first sample combination and a second sample combination;

[0009] The pre-built malware detection model is trained based on the first sample combination, the second sample combination, and the training data corresponding to the benign software.

[0010] Based on the same inventive concept, the second aspect of the present application further provides a method for detecting malware, comprising: detecting malware using a malware detection model trained using the malware detection model training method described in the first aspect.

[0011] Based on the same inventive concept, a third aspect of the present application further provides a training device for a malware detection model, comprising:

[0012] A building module, configured to build an uncertainty assessment model;

[0013] a prediction module configured to train the uncertainty assessment model using the acquired first data set to obtain a set of prediction probabilities corresponding to each training data in the first data set, wherein the first data set includes training data corresponding to malware and training data corresponding to benign software;

[0014] a calculation module configured to calculate, for each piece of training data corresponding to the malware, a plurality of uncertainty indicators based on a set of predicted probabilities of the training data corresponding to the malware and a preset uncertainty indicator type;

[0015] a clustering module configured to cluster the training data corresponding to all malware using a clustering algorithm according to multiple uncertainty indicators corresponding to each malware, and divide the clustering into a first sample combination and a second sample combination;

[0016] The training module is configured to train a pre-built malware detection model based on the first sample combination, the second sample combination, and training data corresponding to benign software.

[0017] Based on the same inventive concept, the fourth aspect of this application also provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable by the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0018] Based on the same inventive concept, the fifth aspect of the present application further provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0019] As can be seen from the above description, the malware detection model training method, detection method, and related devices provided in this application include: constructing an uncertainty assessment model; training the uncertainty assessment model using an acquired first dataset to obtain a set of predicted probabilities corresponding to each training data item in the first dataset; wherein the first dataset includes training data corresponding to malware and training data corresponding to benign software. The predicted probability set includes multiple prediction results of the uncertainty assessment model for each software item, providing a data foundation for subsequent uncertainty indicator analysis. For each piece of training data corresponding to malware, multiple uncertainty indicators are calculated based on the predicted probability set of the malware training data and preset uncertainty indicator types. The uncertainty of each piece of malware data can be quantified using the uncertainty indicators. Based on the multiple uncertainty indicators corresponding to each piece of malware, a clustering algorithm is used to cluster all the training data corresponding to the malware, dividing them into a first sample group and a second sample group; and a pre-constructed malware detection model is trained based on the first sample group, the second sample group, and the training data corresponding to the benign software. Using the divided malware training data and benign software training data with different degrees of uncertainty, a training dataset with a uniform distribution of malware families can be constructed, allowing for phased training of the malware detection model to improve its detection performance. This application uses the uncertainty index obtained by uncertainty evaluation model to robustly train the malware detection model based on deep neural network, thereby reducing the impact of the long-tail distribution problem in the malware dataset on the performance of the malware model and improving the performance of the model in detecting malware. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A flowchart of a method for training a malware detection model according to an embodiment of the present application is provided;

[0022] Figure 2 This is a flow chart of a method for training a malware detection model according to an embodiment of the present application;

[0023] Figure 3 A schematic diagram of the structure of a training device for a malware detection model according to an embodiment of the present application;

[0024] Figure 4This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] Related technologies addressing long-tail distribution issues typically involve explicit data processing. The goal is to manually balance the data through various techniques, including undersampling, oversampling, and reweighting, thereby mitigating the impact of the long-tail distribution issue.

[0028] Undersampling reduces the number of samples in the leading categories by implementing a sample discarding strategy; oversampling increases the number of samples in the trailing categories through data augmentation and sample reuse. These two techniques can achieve a balance in the number of samples across different categories. Reweighting assigns lower weights to the leading categories and higher weights to the trailing categories, allowing the model to focus more on samples in the trailing categories and less on samples in the leading categories, resulting in a more balanced fit for samples across different categories. However, due to the complexity of Android software, the sample representations constructed by oversampling lack practical semantics. The sample discarding operation of undersampling reduces the diversity of the dataset. Reweighting is ineffective for Android software samples because Android software is non-independently distributed in the sample space.

[0029] The approach to addressing long-tail distributions is to adjust model components to make them compatible with long-tail datasets. Specifically, the impact of long-tail datasets on the model can be mitigated by designing specialized loss functions, adjusting the model structure, and employing ensemble models. However, adjusting model components requires designing appropriate components and hyperparameters for each dataset to be effective, which increases the cost of the approach.

[0030] In view of this, this application proposes a training method for a malware detection model. Based on uncertainty assessment, the training data set of malware is divided, and then combined with the training data set of benign software to build a data set that is evenly distributed among different malicious families, thereby reducing the impact of the long-tail distribution problem on the Android malware detection model and improving the performance of the Android malware detection model based on deep neural networks.

[0031] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0032] This application proposes a training method for malware detection model, refer to Figure 1 , including the following steps:

[0033] Step 101: Construct an uncertainty assessment model.

[0034] Specifically, uncertainty assessment models make slight modifications to a base model and then observe whether the model's predictions for the same sample change significantly. If the difference between the model's predictions for the sample after the slight modification and the original predictions for the sample is large, then the model's uncertainty for the sample is high, meaning that the predictions are unreliable.

[0035] The uncertainty assessment method used in this application is the variational Bayesian inference method, and the uncertainty assessment model constructed is a Bayesian neural network. It is implemented based on Bayesian theory. The weight parameters of the Bayesian neural network are random variables that obey a Gaussian distribution, and each weight has its own Gaussian distribution. For each prediction, all parameters in the uncertainty assessment model are sampled from the distribution, which means that the parameters of the uncertainty assessment model will be different for each prediction.

[0036] Step 102: Train the uncertainty assessment model using the acquired first data set to obtain a set of prediction probabilities corresponding to each data in the first data set; wherein the first data set includes data corresponding to malware and data corresponding to benign software.

[0037] Specifically, the first dataset can be obtained from an open-source database or by crawling the web, with no particular limitation on the specific acquisition method. The first dataset includes data corresponding to both malware and benign software. The malware data is unbalanced, i.e., it has a long-tail distribution.

[0038] During prediction, the uncertainty assessment model samples from each Gaussian distribution to obtain weighted values. These weighted values ​​are then used in the forward propagation to calculate the prediction result. Because the weights are randomly sampled, the prediction result is also a random variable, represented as a probability distribution. For each software application, the corresponding prediction result represents the probability that the software is malware or benign software.

[0039] The uncertainty assessment model is trained using the acquired first data set. Specifically, all data in the first data set is divided into training data and test data. The uncertainty assessment model is trained using the training data, and tested using the test data. After testing, a set of prediction probabilities corresponding to each data point in the first data set is obtained. The set of prediction probabilities includes multiple prediction probabilities. For each data point in the first data set, a prediction is performed using the uncertainty assessment model to obtain a prediction probability. The number of times the uncertainty assessment model is tested is the same as the number of prediction probabilities in the set of prediction probabilities.

[0040] Step 103: For each piece of malware corresponding data, multiple uncertainty indicators are calculated based on the predicted probability set of the malware corresponding data and the preset uncertainty indicator type.

[0041] From all predicted probability sets, we select the set of predicted probabilities corresponding to malware. Each set of predicted probabilities corresponding to malware reflects the uncertainty of the uncertainty assessment model in assessing malware. However, uncertainty cannot be quantified based solely on the set of predicted probabilities. Therefore, we need to use uncertainty indicators to quantify uncertainty, which is beneficial for the subsequent classification of malware datasets.

[0042] There are many types of indicators for measuring uncertainty. This application divides each uncertainty indicator type into two groups based on the algorithms of various uncertainty indicator types. The first group is defined as intra-prediction uncertainty. This group of uncertainty indicator types characterizes the degree of dispersion between prediction probabilities within a prediction probability set. The greater the degree of dispersion, the greater the degree of uncertainty in the data prediction by the uncertainty assessment model. The second group is defined as label dispersion. This group of uncertainty indicator types characterizes the degree of difference between the prediction probability set and the uniform distribution of the true label. The greater the degree of difference, the worse the uncertainty assessment model fits the data.

[0043] According to the algorithm of each uncertainty indicator type in the above two groups of uncertainty indicator types, the uncertainty indicator corresponding to each uncertainty indicator type is calculated as the basis for subsequent division of the data set.

[0044] Step 104: Based on the multiple uncertainty indicators corresponding to each malware, a clustering algorithm is used to cluster the data corresponding to all malware, and divide the data into a first sample combination and a second sample combination.

[0045] Specifically, the clustering algorithm performs clustering based on various uncertainty indicators, dividing the data corresponding to all malware into a first sample combination and a second sample combination. The first and second sample combinations correspond to different degrees of uncertainty. A higher degree of uncertainty indicates a greater difficulty for the uncertainty model to predict the software in the sample data of that combination. The probability that the malware corresponding to this sample data belongs to the tail family is high, and the uncertainty assessment model has poor detection performance for this sample data. A lower degree of uncertainty indicates a lower difficulty for the uncertainty model to predict the software in the sample data of that combination. The probability that the malware corresponding to this sample data belongs to the head family is high, and the uncertainty assessment model has good detection performance for this sample data.

[0046] Step 105: Train a pre-built malware detection model based on the first sample combination, the second sample combination, and data corresponding to benign software.

[0047] Specifically, by dividing the malware dataset into a first sample combination and a second sample combination based on an uncertainty metric, this effectively separates malware that is easily predicted by the model from malware that is difficult to predict. Easily predicted malware typically belongs to the top malware families and has a larger number of samples. Difficult-to-predict malware typically belongs to the bottom malware families and has a smaller number of samples. Subsequently, the first and second sample combinations can be combined with the datasets corresponding to benign software to reconstruct a dataset that is evenly distributed across different malware families. Using this reconstructed dataset to train the malware detection model can mitigate the impact of long-tail distribution problem datasets on the malware detection model.

[0048] Based on steps 101 to 105 above, the malware detection model training method provided in this embodiment includes: constructing an uncertainty assessment model; training the uncertainty assessment model using an acquired first dataset to obtain a set of prediction probabilities corresponding to each training data item in the first dataset; wherein the first dataset includes training data corresponding to malware and training data corresponding to benign software. The set of prediction probabilities includes multiple prediction results of the uncertainty assessment model for each software item, providing a data foundation for subsequent uncertainty indicator analysis. For each piece of training data corresponding to malware, multiple uncertainty indicators are calculated based on the set of prediction probabilities for the malware-related training data and preset uncertainty indicator types. The uncertainty of each piece of malware data can be quantified using the uncertainty indicators. Based on the multiple uncertainty indicators corresponding to each piece of malware, a clustering algorithm is used to cluster all the training data corresponding to the malware, dividing them into a first sample group and a second sample group; and a pre-constructed malware detection model is trained based on the first sample group, the second sample group, and the training data corresponding to the benign software. Using the divided malware training data and benign software training data with varying degrees of uncertainty, a training dataset with a uniform distribution of malware families can be constructed, allowing for phased training of the malware detection model to improve its detection performance. This application uses the uncertainty index obtained by uncertainty evaluation model to robustly train the malware detection model based on deep neural network, thereby reducing the impact of the long-tail distribution problem in the malware dataset on the performance of the malware model and improving the performance of the model in detecting malware.

[0049] In addition, due to the inherent complexity of Android malware, many techniques used in computer vision and natural language processing to address long-tail distribution problems are difficult to transfer to the field of Android malware detection. These methods require modifications to the model structure, parameters, or loss function, limiting their versatility. Furthermore, it is necessary to ensure that the Android samples generated during the data augmentation process retain semantic meaning. This application does not require any modifications to the underlying model or to the samples, so the training method for the malware detection model in this application is transferable.

[0050] In some embodiments, the uncertainty assessment model is trained using the acquired first data set to obtain a set of predicted probabilities corresponding to each data in the first data set, including:

[0051] The first data set is divided into multiple sub-data sets, each sub-data set is used as a target test set, and the uncertainty assessment model is trained for multiple rounds to obtain a set of prediction probabilities corresponding to each data in the first data set.

[0052] Specifically, the first dataset is divided into multiple sub-datasets. For example, the first dataset is divided into five sub-datasets, each containing the same amount of data. Each sub-dataset is used as the target test set in turn, and the other sub-datasets are used as the target training set. The uncertainty assessment model is then trained multiple times to obtain a set of predicted probabilities corresponding to each data point in the first dataset.

[0053] Furthermore, the uncertainty assessment model is trained for multiple rounds using each sub-dataset as a target test set to obtain a set of prediction probabilities corresponding to each training data in the first data set, including:

[0054] For each of the multiple training rounds, perform the following operations:

[0055] One of the sub-datasets that is not used as the target test set in the multiple sub-datasets is used as the target test set, and all other sub-datasets other than the target test set are used as target training sets;

[0056] The uncertainty assessment model is trained based on the target training set, and the trained uncertainty model is tested multiple times based on the target test set to obtain the prediction probability set corresponding to each training data in the target test set;

[0057] In response to determining that there is a sub-dataset among the multiple sub-datasets that is not used as the target test set, entering the next round of training;

[0058] In response to determining that there is no sub-dataset among the multiple sub-datasets that is not used as the target test set, exiting the multiple rounds of training.

[0059] During specific implementation, after dividing the first data set into multiple sub-data sets, a seat target test set is randomly selected from the multiple sub-data sets, and the other sub-data sets are used as target training sets. The uncertainty assessment model is trained with the target training set, and after the training, the uncertainty assessment model is tested with the target test set. For each data in the target test set, the data is input into the uncertainty assessment model, and the uncertainty assessment model can output the predicted probability corresponding to the data. At this time, it can be set to perform multiple tests on each data in the target test set through the uncertainty assessment model. For each data, after multiple tests, a predicted probability set can be obtained. For example, the number of multiple tests is set to 10 times, and the number of predicted probabilities in the predicted probability set is 10.

[0060] Afterwards, a determination is made among the multiple sub-datasets to determine whether any sub-datasets are not used as target test sets. If so, one of the sub-datasets that are not used as target test sets is randomly selected as the target test set, and the remaining sub-datasets are used as target training sets. The uncertainty assessment model is then trained for the next round until each sub-dataset has been used as a target test set, at which point multiple rounds of training are exited. This ensures that each data point in the first data set can be used as data in a target test set, and a corresponding set of predicted probabilities can be obtained. Each set of predicted probabilities can be used to evaluate the corresponding malware uncertainty indicator in subsequent data processing, providing a data foundation for the partitioning of malware data sets.

[0061] In some embodiments, the training data corresponding to each malware is used to calculate multiple uncertainty indicators based on a set of predicted probabilities of the training data corresponding to the malware and a preset uncertainty indicator type, including:

[0062] For each uncertainty indicator type, the corresponding uncertainty indicator is determined based on the prediction probability set and the algorithm of the uncertainty indicator type.

[0063] Specifically, the predicted probability set determined by the above embodiment can be expressed as P = {p1, p2, ..., p N Where N represents the total number of prediction probabilities in the prediction probability set. The degree of difference between the prediction probabilities in the prediction probability set is considered uncertainty. The greater the difference, the higher the uncertainty, and the smaller the difference, the lower the uncertainty.

[0064] This application divides uncertainty metrics into two groups: the first group is intra-prediction uncertainty, and the second group is label dispersion. The first group of metrics characterizes the degree of dispersion between predicted probabilities; greater dispersion indicates greater uncertainty about the model's predictions. Uncertainty metrics within the first group include prediction entropy, KL divergence, standard deviation, maximum minus next maximum, next minimum minus minimum, and mean minus median.

[0065] The second group of indicators characterizes the uniform distribution Y = (y,y,…,y) of the model prediction probability set P and the true label y. N The second group includes uncertainty indicators such as Euclidean distance, Chebyshev distance, KL divergence, negative log-likelihood, and Manhattan distance.

[0066] The calculation method for each uncertainty indicator type is as follows:

[0067] ① Prediction entropy ET: It characterizes the degree of confusion between each prediction probability in the prediction probability set.

[0068]

[0069] Where ∈ = 10 -10 It is a small constant used to avoid log(0) and prevent numerical calculation errors.

[0070] ②First KL divergence: characterizes the degree of difference between two probability distributions.

[0071] First calculate the mean of the predicted probability set The mean is then expanded into a distribution vector Q with the same length as the predicted probability set, and then the degree of difference between the distribution vector Q and the predicted probability set P is calculated.

[0072]

[0073]

[0074] ③ Standard deviation STD: represents the degree of dispersion between each prediction probability in the prediction probability set.

[0075]

[0076] ④ Maximum value minus second largest value: the maximum value of the predicted probability in the predicted probability set minus the second largest value of the predicted probability

[0077]

[0078] ⑤ The second smallest value minus the minimum value: the second smallest value of the predicted probability in the predicted probability set Subtract the minimum value min of the predicted probability.

[0079]

[0080] ⑥ Mean minus median: the mean of the predicted probability in the predicted probability set minus the median of the predicted probability.

[0081] Δ (median,mean) =mean(P)-median(P)

[0082] ⑦Euclidean distance: measures the straight-line distance between two points, indicating the difference between the predicted probability and the true label distribution.

[0083]

[0084] ⑧Chebyshev distance: Calculates the maximum difference between two points in all coordinate axis directions, which is used to measure the farthest deviation between the predicted probability and the true label in any dimension.

[0085]

[0086] ⑨ Second KL divergence: measures the difference between one probability distribution and another probability distribution, emphasizing the relative entropy increase between the predicted probability distribution and the true label distribution.

[0087]

[0088] ⑩ Negative log likelihood: By calculating the negative log likelihood value of the true label under the predicted probability distribution, the degree of match between the confidence of the model prediction and the true label is measured.

[0089]

[0090] Manhattan distance: Calculates the sum of the absolute values ​​of the differences between two points in each coordinate axis direction, which represents the total deviation between the predicted probability and the true label.

[0091]

[0092] Based on the calculation methods of the above 11 types of uncertainty indicators, 11 uncertainty indicators corresponding to each malicious software are calculated, providing a data basis for the subsequent division of the malicious software dataset.

[0093] In some embodiments, clustering the training data corresponding to all malware using a clustering algorithm based on multiple uncertainty indicators corresponding to each malware to divide the training data into a first sample combination and a second sample combination includes:

[0094] Clustering the training data corresponding to all malware using a K-means clustering algorithm based on multiple uncertainty indicators corresponding to all malware to obtain a first sample combination and a second sample combination;

[0095] The uncertainty of the first sample combination is higher than the uncertainty of the second sample combination.

[0096] Specifically, in each round of training of the uncertainty assessment model, 11 uncertainty indicators are calculated and quantified for the data of each malware. In order to further distinguish the fitting difficulty of the malware data, this embodiment uses a clustering method to divide the malware data set. Exemplarily, the clustering method can be the K-means clustering algorithm (K-Means algorithm). The K-means algorithm is an unsupervised clustering algorithm based on distance. It divides the data set into K clusters through iterative optimization. The core steps include randomly initializing the centroid, assigning data points to the nearest centroid, and updating the centroid position until convergence. The malware data set is divided into two clusters by the K-means algorithm to obtain the corresponding first sample combination and second sample combination. Among them, the uncertainty of the first sample combination is higher than the uncertainty of the second sample combination.

[0097] The first sample combination has high uncertainty and is identified as a "difficult-to-fit malicious sample," while the second sample combination has low uncertainty and is identified as an "easy-to-fit malicious sample." This method automatically classifies sample difficulty based on the inherent structure of uncertainty within malware data, without requiring manual threshold setting. This provides foundational support for subsequent adversarial robustness improvements and sample selection strategies.

[0098] In some embodiments, the training of a pre-built malware detection model based on the first sample combination, the second sample combination, and the training data corresponding to the benign software includes:

[0099] Combining the first sample combination and the data corresponding to the benign software into a first training set;

[0100] Combining the second sample combination and the data corresponding to the benign software into a second training set;

[0101] The malware detection model is trained using the second training set;

[0102] The first training set is used to fine-tune the trained malware detection model.

[0103] Specifically, when constructing a uniformly distributed training set, the first sample combination is combined with an equal amount of benign software data to form the first training set. The amount of malware data in the first training set is equal to the amount of benign software data. The second sample combination is combined with an equal amount of benign software data to form the second training set. The amount of malware data in the second training set is equal to the amount of benign software data.

[0104] The malware detection model is trained on the second training set, prioritizing training on the easier-to-fit training set. Subsequently, the malware detection model is fine-tuned on the harder-to-fit training set using the first training set. This curriculum learning approach improves model training stability and accelerates model convergence. For imbalanced datasets, learning the majority class first (which is easier to fit) and then fine-tuning the minority class can improve overall model performance.

[0105] It should be noted that the embodiments of the present application can be further described in the following manner:

[0106] Figure 2 FIG1 shows a flow chart of a training method for a malware detection model according to an embodiment of the present application. Figure 2 As shown, the training method is divided into three stages: Stage I: uncertainty assessment, Stage II: malware clustering, and Stage III: uncertainty-based curriculum learning. In Stage I, the first dataset collected includes an unbalanced malware dataset and a benign software dataset. The uncertainty assessment model is trained and tested using this first dataset to obtain a set of predicted probabilities. In Stage II, an uncertainty index is calculated based on the predicted probabilities. Based on this uncertainty index, the malware dataset is clustered using a clustering model, dividing it into easy-to-fit and difficult-to-fit malware datasets. In Stage III, the easy-to-fit malware dataset is combined with an equal number of benign software datasets to obtain the first training set, and the malware detection model is trained based on this first training set. The difficult-to-fit malware dataset is combined with an equal number of benign software datasets to obtain the second training set, and the malware detection model is fine-tuned based on this second training set.

[0107] Based on the same inventive concept, this application also proposes a malware detection method, including:

[0108] The malware detection model trained by the malware detection model training method of the aforementioned embodiment is used to detect malware.

[0109] Specifically, the software to be detected is input into a trained malware detection model, which then outputs the probability that the software to be detected is malware. This probability is then used to determine whether the software to be detected is malware.

[0110] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0111] It should be noted that the above describes some embodiments of the present application. In some cases, the actions or steps described in the above embodiments can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a training device for a malware detection model.

[0113] refer to Figure 3 , the training device of the malware detection model comprises:

[0114] A construction module 201 is configured to construct an uncertainty assessment model;

[0115] Prediction module 202 is configured to train the uncertainty assessment model using the acquired first data set to obtain a set of prediction probabilities corresponding to each training data in the first data set, wherein the first data set includes training data corresponding to malware and training data corresponding to benign software;

[0116] The calculation module 203 is configured to calculate a plurality of uncertainty indicators for each piece of training data corresponding to the malware based on a set of predicted probabilities of the training data corresponding to the malware and a preset uncertainty indicator type;

[0117] The clustering module 204 is configured to cluster the training data corresponding to all malware using a clustering algorithm according to the multiple uncertainty indicators corresponding to each malware, and divide the clustering into a first sample combination and a second sample combination;

[0118] The training module 205 is configured to train a pre-built malware detection model based on the first sample combination, the second sample combination, and the training data corresponding to the benign software.

[0119] In some embodiments, the prediction module 202 is further configured to divide the first data set into multiple sub-data sets, use each sub-data set as a target test set, perform multiple rounds of training on the uncertainty assessment model, and obtain a set of prediction probabilities corresponding to each data in the first data set.

[0120] In some embodiments, the prediction module 202 is further configured to perform the following operations for each round of training in the multiple rounds of training:

[0121] One of the sub-datasets that is not used as the target test set in the multiple sub-datasets is used as the target test set, and all other sub-datasets other than the target test set are used as target training sets;

[0122] The uncertainty assessment model is trained based on the target training set, and the trained uncertainty model is tested multiple times based on the target test set to obtain the prediction probability set corresponding to each training data in the target test set;

[0123] In response to determining that there is a sub-dataset among the multiple sub-datasets that is not used as the target test set, entering the next round of training;

[0124] In response to determining that there is no sub-dataset among the multiple sub-datasets that is not used as the target test set, exiting the multiple rounds of training.

[0125] In some embodiments, the calculation module 203 is further configured to determine a corresponding uncertainty indicator for each uncertainty indicator type based on the prediction probability set and the algorithm of the uncertainty indicator type.

[0126] In some embodiments, the clustering module 204 is further configured to cluster the training data corresponding to all malware using a K-means clustering algorithm based on multiple uncertainty indicators corresponding to all malware to obtain a first sample combination and a second sample combination; wherein the uncertainty of the first sample combination is higher than the uncertainty of the second sample combination.

[0127] In some embodiments, the training module 205 is further configured to combine the first sample combination and the data corresponding to the benign software into a first training set; combine the second sample combination and the data corresponding to the benign software into a second training set; use the second training set to train the malware detection model; and use the first training set to fine-tune the trained malware detection model.

[0128] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a malware detection device.

[0129] The malware detection device includes:

[0130] The detection module detects malware using a malware detection model trained using a malware detection model training method.

[0131] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0132] The apparatus of the above embodiment is used to implement the training method of the corresponding malware detection model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0133] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the training method of the malware detection model described in any of the above embodiments is implemented.

[0134] Figure 4 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0135] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0136] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0137] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0138] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0139] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0140] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0141] The electronic device of the above embodiment is used to implement the training method of the corresponding malware detection model in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0142] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the malware detection model training method described in any of the above embodiments.

[0143] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0144] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the training method of the malware detection model as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0145] Based on the same concept, corresponding to any of the above-mentioned embodiments, the present application also provides a computer program product, including computer program instructions. When the computer program instructions are run on a computer, the computer executes the method described in any of the above embodiments, which has the beneficial effects of the corresponding method embodiments and will not be repeated here.

[0146] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application is limited to these examples. In line with the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0147] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0148] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0149] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

Claims

1. A method for training a malware detection model, characterized in that: include: Construct uncertainty assessment models; Training the uncertainty assessment model using the acquired first data set to obtain a set of predicted probabilities corresponding to each data in the first data set; wherein the first data set includes data corresponding to malware and data corresponding to benign software; For each piece of malware corresponding data, multiple uncertainty indicators are calculated based on a set of predicted probabilities of the malware corresponding data and a preset uncertainty indicator type; Based on multiple uncertainty indicators corresponding to each malware, a clustering algorithm is used to cluster the data corresponding to all malware into a first sample combination and a second sample combination; The pre-built malware detection model is trained based on the first sample combination, the second sample combination, and data corresponding to the benign software.

2. The method according to claim 1, characterized in that The uncertainty assessment model is trained by using the acquired first data set to obtain a set of prediction probabilities corresponding to each data in the first data set, including: The first data set is divided into multiple sub-data sets, each sub-data set is used as a target test set, and the uncertainty assessment model is trained for multiple rounds to obtain a set of prediction probabilities corresponding to each data in the first data set.

3. The method according to claim 2, characterized in that The uncertainty assessment model is trained for multiple rounds using each sub-dataset as a target test set to obtain a set of prediction probabilities corresponding to each training data in the first data set, including: For each of the multiple training rounds, perform the following operations: One of the sub-datasets that is not used as the target test set in the multiple sub-datasets is used as the target test set, and all other sub-datasets other than the target test set are used as target training sets; The uncertainty assessment model is trained based on the target training set, and the trained uncertainty model is tested multiple times based on the target test set to obtain the prediction probability set corresponding to each training data in the target test set; In response to determining that there is a sub-dataset among the multiple sub-datasets that is not used as the target test set, entering the next round of training; In response to determining that there is no sub-dataset among the multiple sub-datasets that is not used as the target test set, exiting the multiple rounds of training.

4. The method according to claim 1, wherein The training data corresponding to each malware is used to calculate multiple uncertainty indicators based on the predicted probability set of the training data corresponding to the malware and the preset uncertainty indicator type, including: For each uncertainty indicator type, the corresponding uncertainty indicator is determined based on the prediction probability set and the algorithm of the uncertainty indicator type.

5. The method according to claim 1, wherein The method of clustering the training data corresponding to all malware using a clustering algorithm according to the multiple uncertainty indicators corresponding to each malware into a first sample combination and a second sample combination includes: Clustering the training data corresponding to all malware using a K-means clustering algorithm based on multiple uncertainty indicators corresponding to all malware to obtain a first sample combination and a second sample combination; The uncertainty of the first sample combination is higher than the uncertainty of the second sample combination.

6. The method according to claim 1, wherein The training of the pre-built malware detection model based on the first sample combination, the second sample combination, and the training data corresponding to the benign software includes: Combining the first sample combination and the data corresponding to the benign software into a first training set; Combining the second sample combination and the data corresponding to the benign software into a second training set; The malware detection model is trained using the second training set; The first training set is used to fine-tune the trained malware detection model.

7. A method for detecting malware, characterized in that: include: Malware is detected by using a malware detection model trained using the malware detection model training method according to any one of claims 1 to 6.

8. A training device for a malware detection model, characterized in that: include: A building module, configured to build an uncertainty assessment model; a prediction module configured to train the uncertainty assessment model using the acquired first data set to obtain a set of prediction probabilities corresponding to each training data in the first data set, wherein the first data set includes training data corresponding to malware and training data corresponding to benign software; a calculation module configured to calculate, for each piece of training data corresponding to the malware, a plurality of uncertainty indicators based on a set of predicted probabilities of the training data corresponding to the malware and a preset uncertainty indicator type; a clustering module configured to cluster the training data corresponding to all malware using a clustering algorithm according to multiple uncertainty indicators corresponding to each malware, and divide the clustering into a first sample combination and a second sample combination; The training module is configured to train a pre-built malware detection model based on the first sample combination, the second sample combination, and training data corresponding to benign software.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images