Dataset construction, malware identification, identification model construction method and device

Abstract

The present invention provides a method and device for constructing a data set, identifying malware, and constructing an identification model. Based on new samples and samples in a seed set, if a new sample is determined to be an abnormal sample using an anomaly detection algorithm, the new sample is added to the seed set until the number of samples in the seed set meets a preset condition. The present invention determines abnormal samples based on an anomaly detection algorithm, so that the abnormal samples and the samples in the seed set have a higher degree of abnormality, and removes new samples with a lower degree of abnormality than the seed set, thereby reducing redundant data in the seed set, reducing the scale of the seed set, and thus saving data storage space. At the same time, when a seed set with a smaller scale is applied to malware identification, the training time of the malware identification model can also be shortened, and each sample in the seed set after removing redundant data has a strong representativeness, avoiding the problem of noise caused by excessive redundant data in the data set in the traditional method affecting the accuracy of the identification model.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for constructing a data set, identifying malware, and constructing an identification model. Background Art

[0002] With the continuous development of computer programming technology, software based on various computer languages ​​has made it easier for people to complete various tasks and work on computers. However, malware with malicious content has also emerged, maliciously attacking normal data files or stealing the fruits of others' labor. Therefore, it is very important to identify whether the software under test is malware.

[0003] Existing intelligent malware identification technologies usually use machine learning methods to identify malware, and the application of machine learning algorithms is inseparable from training data sets. The training data sets in existing intelligent malware identification technologies include a large number of malicious and non-malware software, but the internal homogeneity of the training data sets is relatively serious (there are only very small differences between two similar software), that is, the redundancy of the data sets is very high, and redundant data not only wastes data storage space, prolongs the training time of the malware identification model, but may even reduce the accuracy of the malware identification model. Summary of the invention

[0004] The present invention provides a method and device for constructing a data set, identifying malware, and constructing an identification model, so as to solve the defect of high redundancy of data sets in the prior art.

[0005] The present invention provides a method for constructing a data set for malware identification, comprising:

[0006] New sample acquisition step: acquiring a new sample; wherein the sample type of the new sample is consistent with the sample type in the pre-constructed seed set, wherein the seed set includes malicious samples and / or non-malicious samples;

[0007] Abnormal sample determination step: based on the new sample and the samples in the seed set, using an abnormality detection algorithm to determine whether the new sample is an abnormal sample, and if so, adding the new sample to the seed set;

[0008] Update step: cyclically execute the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets a preset condition.

[0009] According to the data set construction method provided by the present invention, the abnormal sample determination step specifically includes:

[0010] Based on the new sample and the samples in the seed set, an isolation forest algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set.

[0011] According to a data set construction method provided by the present invention, based on the new sample and the samples in the seed set, an isolation forest algorithm is used to determine whether the new sample is an abnormal sample, and if so, the new sample is added to the seed set, including:

[0012] Use the isolation forest algorithm or the extended isolation forest algorithm to construct N isolated trees based on the seed set;

[0013] Based on the N isolated trees, the new sample is scored as an outlier, and if the score is higher than a preset threshold, the new sample is added to the seed set.

[0014] According to a data set construction method provided by the present invention, outlier scoring is performed on the new sample based on the N isolated trees, comprising:

[0015] Based on the N isolated trees, determining a depth value of the new sample on each isolated tree;

[0016] According to each depth value, an outlier scoring function is used to perform an outlier scoring on the new sample.

[0017] The present invention also provides a malware identification method, comprising:

[0018] Obtain the software to be identified;

[0019] Inputting the software into a malware identification model to obtain an identification result of the software;

[0020] Among them, the malware identification model is obtained after machine learning training of a first seed set containing malicious samples constructed by any of the dataset construction methods described above, and / or a second seed set containing non-malicious samples constructed by any of the dataset construction methods described above.

[0021] The present invention also provides a method for constructing a malware identification model, comprising:

[0022] Using any of the above-mentioned dataset construction methods to construct a first seed set containing malicious samples; and / or,

[0023] Using any of the above dataset construction methods to construct a second seed set containing non-malicious samples;

[0024] Based on the first seed set and / or the second seed set, a machine learning model is trained using a machine learning method to obtain a malware identification model.

[0025] The present invention also provides a data set construction device for malware identification, comprising:

[0026] A new sample acquisition unit, used to acquire a new sample; wherein the sample type of the new sample is consistent with the sample type in the pre-constructed seed set, wherein the seed set includes malicious samples and / or non-malicious samples;

[0027] an abnormal sample determination unit, configured to determine whether the new sample is an abnormal sample by using an abnormality detection algorithm based on the new sample and the samples in the seed set, and if so, add the new sample to the seed set;

[0028] An updating unit is used to cyclically execute the steps in the new sample acquisition unit and the steps in the abnormal sample determination unit until the number of samples in the seed set meets a preset condition.

[0029] The present invention also provides a malware identification device, comprising:

[0030] An acquisition unit, used for acquiring software to be identified;

[0031] an identification unit, configured to input the software into a malware identification model to obtain an identification result of the software;

[0032] Among them, the malware identification model is obtained after machine learning training of a first seed set containing malicious samples constructed by any of the dataset construction methods described above, and / or a second seed set containing non-malicious samples constructed by any of the dataset construction methods described above.

[0033] The present invention also provides a malware identification model construction device, comprising:

[0034] A first construction unit is configured to construct a first seed set containing malicious samples by using any of the above-mentioned data set construction methods; and / or,

[0035] A second construction unit, configured to construct a second seed set containing non-malicious samples by using any of the above-mentioned data set construction methods;

[0036] A training unit is used to train a machine learning model based on the first seed set and / or the second seed set using a machine learning method to obtain a malware identification model.

[0037] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the data set construction methods described above are implemented; and / or, when the processor executes the computer program, the steps of the malware identification method described above are implemented; and / or, when the processor executes the computer program, the steps of the malware identification model construction method described above are implemented.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of any one of the data set construction methods described above are implemented; and / or when the processor executes the computer program, the steps of the malware identification method described above are implemented; and / or when the processor executes the computer program, the steps of the malware identification model construction method described above are implemented.

[0039] The present invention also provides a computer program product having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the steps of any one of the data set construction methods described above; and / or, when the processor executes the computer program, implements the steps of the malware identification method described above; and / or, when the processor executes the computer program, implements the steps of the malware identification model construction method described above.

[0040] The data set construction, malware identification, and identification model construction methods and devices provided by the present invention are based on new samples and samples in the seed set. If a new sample is determined to be an abnormal sample by an anomaly detection algorithm, the new sample is added to the seed set until the number of samples in the seed set meets a preset condition. It can be seen that the present invention determines abnormal samples based on the anomaly detection algorithm, so that the abnormal samples and the samples in the seed set have a higher degree of abnormality, and eliminates new samples with a lower degree of abnormality than the seed set, thereby reducing redundant data in the seed set, reducing the scale of the seed set, and saving data storage space. At the same time, when a seed set with a smaller scale is applied to malware identification, the training time of the malware identification model can also be shortened, and each sample in the seed set after eliminating redundant data has a strong representativeness, avoiding the problem of noise caused by excessive redundant data in the data set in the traditional method affecting the accuracy of the recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 It is a flowchart of the method for constructing a data set provided by the present invention;

[0043] Figure 2 It is a schematic diagram of the seed set construction process provided by the present invention;

[0044] Figure 3 It is a schematic diagram of the seed set update process provided by the present invention;

[0045] Figure 4 It is a flowchart of the malware identification method provided by the present invention;

[0046] Figure 5 It is a flowchart of the method for building a malware identification model provided by the present invention;

[0047] Figure 6 It is a structural schematic diagram of a data set construction device provided by the present invention;

[0048] Figure 7 is a schematic diagram of the structure of the malware identification device provided by the present invention;

[0049] Figure 8 It is a schematic diagram of the structure of the malware identification model building device provided by the present invention;

[0050] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Malware refers to viruses, worms, and Trojan horse programs that perform malicious tasks on computer systems. Malware can steal user information, affect the user's experience of using electronic devices, and even seriously threaten the user's property safety. Therefore, it is necessary to identify malware. In existing intelligent malware identification technologies, machine learning is usually used to identify malware, and the application of machine learning algorithms is inseparable from training data sets. In existing intelligent malware identification technologies, data sets are constructed based on a large number of malicious and non-malware software. However, due to the strong similarity between software, the internal homogeneity of the simply collected data sets is relatively serious, such as several adjacent small versions of the same software, or different versions of viruses belonging to the same family, or the same software infected by similar viruses, etc., resulting in a high degree of redundancy in the data set. Redundant data not only wastes data storage space, but also prolongs the training time of the malware identification model. At the same time, compared with the laboratory environment, the real scenario of malware identification is often more complex. Since malware is updated quickly, it is necessary to obtain new samples to add to the training set to update the recognition model, resulting in a large scale of the model training set. Correspondingly, the redundant data in the training set will also increase and bring noise to the training of the recognition model, affecting the accuracy of the recognition model.

[0053] To this end, the present invention provides a method for constructing a data set for malware identification. Figure 1 is a flow chart of the method for constructing a data set provided by the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0054] Step 110, new sample acquisition step: acquire new samples; wherein the sample type of the new samples is consistent with the sample type in the pre-constructed seed set, wherein the seed set contains malicious samples and / or non-malicious samples.

[0055] In this step, in order to train the malware recognition model, it is necessary to obtain new samples as the training set data of the malware recognition model, and then the malware recognition model trains one or more malicious / non-malicious binary classification models based on these training set data, and then uses the model to identify the malware. Among them, the sample type of the new sample is consistent with the sample type in the pre-built seed set, and the seed set contains malicious samples, and / or, non-malicious samples, that is, the new sample contains malicious samples, and / or, non-malicious samples. Among them, malicious samples include software that attempts to damage computers, collect sensitive information or illegally access other computers, and non-malicious samples include software that works normally on computers. The above-mentioned malicious samples or non-malicious samples can be downloaded from the sample site, but considering that if the malicious samples or non-malicious samples downloaded from the sample site are directly added to the seed set, there may be some identical malicious samples or non-malicious samples in the seed set, which increases the data size of the seed set. Therefore, after downloading and obtaining malicious samples or non-malicious samples, this embodiment will screen and deduplicate the samples to eliminate samples that are exactly the same in the seed set. For example, a hash digest algorithm (MD5, SHA1, etc.) can be used to deduplicate the samples and eliminate samples with the same digest value as the samples in the seed set, thereby ensuring that the sample data in the seed set are different.

[0056] In addition, in order to find the most effective features from the original features of the samples (invariance of samples of the same type, discriminability of different samples, and robustness to noise), this embodiment extracts features from the deduplicated samples, and converts the original features of the samples into a set of features with obvious physical or statistical significance or cores, thereby reducing data redundancy and accurately obtaining features that can reflect the essence of the samples, promoting the learning and generalization of the subsequent malware recognition model on the samples, and improving the prediction accuracy of the recognition model.

[0057] like Figure 2 As shown, this embodiment obtains malicious samples or non-malicious samples by downloading from sample sites, then deduplicates the downloaded samples, removes the downloaded samples that are identical to the data in the seed set, and then extracts features from the deduplicated downloaded samples, and represents the downloaded samples in a quantized form (usually a high-dimensional real vector) until the seed set reaches a preset size. It should be noted that the seed set obtained in this embodiment is relatively small in size, for example, the data size of the seed set can be as large as 10,000, which is equivalent to 1% of the data size of the seed set obtained in the existing intelligent malware identification technology, which greatly reduces the size of the seed set and saves storage space.

[0058] In addition, although the obtained seed set contains malicious samples or non-malicious samples, in actual malware identification application scenarios, malware and non-malicious software are updated quickly. In order to ensure the prediction accuracy of the identification model, it is necessary to adapt to the update speed of the software and frequently obtain new samples from the sample site for updating and training the identification model. In this embodiment, new samples can be obtained from the sample site regularly, and the sample type of the obtained new samples is consistent with the sample type in the seed set, including malicious samples or non-malicious samples.

[0059] Step 120, abnormal sample determination step: Based on the new sample and the samples in the seed set, an abnormality detection algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set.

[0060] In this step, it should be noted that the traditional method of obtaining a dataset for malware identification is achieved through the steps of sample downloading - sample deduplication - feature extraction - dataset construction. That is, although the traditional method can remove completely identical samples through sample deduplication (such as hash algorithm), the different software obtained after deduplication using the hash algorithm may be homogenized (the difference between the software is very small), such as several adjacent small versions of the same software, or different versions of viruses belonging to the same family, or the same software infected by similar viruses, etc., which leads to a large number of similar samples in the final dataset, that is, redundant samples. Redundant samples not only cannot provide effective information for recognition model learning, but also cause the dataset to expand excessively, thereby seriously reducing the efficiency of recognition model training. In addition, because the popularity of different software varies greatly, the redundancy of different samples in the final dataset also varies greatly, which ultimately leads to the introduction of a large bias in the recognition model training, affecting the prediction effect of the recognition model.

[0061] Therefore, after obtaining a new sample, this embodiment determines whether the new sample is an abnormal sample based on the anomaly detection algorithm. If so, it indicates that there is a large difference between the new sample and the sample in the seed set, so the new sample is added to the seed set, so that the sample in the seed set has a strong representativeness. If it is determined that the new sample is not an abnormal sample, it indicates that the difference between the new sample and the sample in the seed set is small, that is, the new sample is a redundant sample, so the new sample is removed to avoid the redundancy of the seed set sample. Among them, anomaly detection is a detection process to find out its behavior is very different from the expected object. This embodiment uses the anomaly detection algorithm to regard the abnormal data considered by the algorithm as high-quality non-redundant data, and regards the non-abnormal data as redundant data, that is, to remove non-abnormal data with high redundancy, that is, low abnormality, so that a high-quality data set can be constructed at a low cost. At the same time, based on the data set determined by the anomaly detection algorithm, the samples in the final data set are all highly representative, which not only greatly reduces the noise around the sample point, but also reduces the data scale without reducing the data coverage capability, effectively improving the training convergence speed, prediction accuracy and generalization performance of the recognition model.

[0062] The present application innovatively solves the problem that the existing sample set cannot be generated efficiently and with high quality in a reverse thinking way. In the sample set generation process of the present application, the abnormal data is detected by using an anomaly detection algorithm, and then the abnormal data is used as high-quality non-redundant data. It can be understood that since the non-abnormal data has a high similarity with the existing samples in the sample set, the non-abnormal data can be used as redundant data, and the abnormal data can be used as non-redundant data. It can be understood that, in general, the anomaly detection algorithm is used to remove abnormal data, and the present application creatively reverses this process, using the anomaly detection algorithm to add valid sample data, thereby reducing the redundant data in the seed set, reducing the size of the seed set, and thus saving data storage space. At the same time, when a seed set with a smaller scale is applied to malware identification, the training time of the malware identification model can also be shortened, and each sample in the seed set after removing the redundant data has a strong representativeness, avoiding the problem that the noise caused by the data set containing too much redundant data in the traditional method affects the accuracy of the recognition model.

[0063] In addition, in order to improve the quality of the data set, traditional methods usually require manual secondary confirmation of the data annotation (which may even involve multiple rounds of review and confirmation), which is extremely time-consuming and labor-intensive. It can be seen that this embodiment can significantly reduce the manual review cost of high-quality data sets while reducing the size of the data set.

[0064] Step 130, update step: looping the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets a preset condition.

[0065] In this step, it should be noted that, since there is redundancy within the pre-constructed seed set, after the new sample is added to the seed set by the method of step 110 and step 120, the pre-constructed seed set only accounts for a very low proportion in the seed set obtained in step 130, thereby achieving the dilution of the redundancy in the seed set and making it negligible. It can be seen that, by cyclically executing steps 110 and 120, this embodiment can not only obtain new data set samples for training the recognition model, but also dilute the redundancy of the seed set, reduce the size of the data set, and save the storage overhead of the data set. Among them, the preset condition may refer to the number of samples in the seed set reaching a preset size, or the number of updates reaching a preset number, which is not specifically limited in this embodiment.

[0066] The data set construction method provided by the present invention is based on new samples and samples in the seed set. If the new sample is determined to be an abnormal sample by an anomaly detection algorithm, the new sample is added to the seed set until the number of samples in the seed set meets the preset conditions. It can be seen that the present invention determines abnormal samples based on the anomaly detection algorithm, so that the abnormal samples and the samples in the seed set have a higher degree of abnormality, and eliminates new samples with a lower degree of abnormality than the seed set, thereby reducing redundant data in the seed set, reducing the scale of the seed set, and saving data storage space. At the same time, when a seed set with a smaller scale is applied to malware identification, the training time of the malware identification model can also be shortened, and each sample in the seed set after eliminating redundant data has a strong representativeness, avoiding the problem of noise caused by too much redundant data in the data set in the traditional method affecting the accuracy of the recognition model.

[0067] Based on the above embodiment, the abnormal sample determination step specifically includes:

[0068] Based on the new sample and the samples in the seed set, the isolation forest algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set.

[0069] In this embodiment, it should be noted that Isolation Forest is a fast outlier detection method based on Ensemble, which has linear time complexity and high accuracy. Unlike other anomaly detection algorithms that use quantitative indicators such as distance and density to characterize the degree of alienation between samples, the Isolation Forest algorithm detects outliers by isolating sample points. Specifically, the algorithm uses a binary search tree structure called an isolation tree to isolate samples. Due to the small number of outliers and their alienation from most samples, outliers will be isolated earlier, that is, outliers will be closer to the root node of the isolation tree, while normal values ​​will be farther away from the root node. In addition, compared with traditional algorithms such as LOF and K-means, the Isolation Forest algorithm has better robustness to high-dimensional data.

[0070] In this embodiment, by adopting the isolation forest algorithm, an isolated tree is established in the seed set, and the new sample is compared and analyzed with each node in the isolated tree to determine the depth of the new sample on the isolated tree. The smaller the depth, the closer it is to the root node of the isolated tree, indicating that there is a large difference between the new sample and the sample in the seed set. It should be noted that the isolation forest algorithm can calculate the abnormal score value based on the depth value of the new sample in the isolated tree. The larger the abnormal score value, the closer the new sample is to the root node of the isolated tree. It can be understood that by setting a preset value, if the abnormal score value is greater than the preset value, the new sample is determined to be an abnormal sample and added to the seed set.

[0071] In addition, since the data set used for the recognition model adopts binary sample data that can be recognized by computers, if the clustering method is used to cluster the entire data set and then take the cluster core as the low redundancy set, although it is possible to determine whether the new sample is different from the seed set sample, the algorithm complexity of the clustering method is higher than the isolation forest algorithm, especially for binary sample data vectors represented by ultra-high dimensional vectors (usually reaching several k dimensions or even dozens of k dimensions), its running overhead seriously reduces its own availability.

[0072] The data set construction method provided by the present invention uses an isolation forest algorithm to determine whether a new sample is an abnormal sample based on the new sample and the samples in the seed set, which not only reduces the operating overhead cost but also has good robustness.

[0073] Based on the above embodiment, based on the new sample and the samples in the seed set, an isolation forest algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set, including:

[0074] Use the isolation forest algorithm or the extended isolation forest algorithm to construct N isolated trees based on the seed set;

[0075] Based on N isolated trees, the new samples are scored as outliers. If the score is higher than the preset threshold, the new samples are added to the seed set.

[0076] In this embodiment, it should be noted that since the feature vectors of malicious samples or non-malicious samples may be mixed with various forms such as discrete and continuous, and the dimensions are usually large, and the large amount of data leads to time-consuming normalization of the vectors, the isolation forest algorithm can be used to avoid these problems; if the feature vectors of malicious samples or non-malicious samples are continuous normalization results, the extended isolation forest algorithm can be used to further reduce the bias factors in the constructed data set. It can be seen that this embodiment can select the corresponding anomaly detection algorithm according to the feature vector situation of malicious samples or non-malicious samples to improve the calculation efficiency.

[0077] In this embodiment, the isolation forest algorithm or the extended isolation forest algorithm constructs N isolation trees on the seed set, where each tree targets only a randomly sampled subset of the seed set, and then randomly selects one from all the features to split the data set, and repeats this process until the current data set is a single data set or the depth reaches a threshold, where N can be set by a domain expert, such as an initial value of 100, which can be subsequently increased according to a specified strategy, such as each time N increases by 5 compared to the previous time.

[0078] In this embodiment, after N isolated trees are constructed in the seed set, outlier scores are performed on new samples to filter out samples with scores lower than a preset threshold (the preset threshold can be set by an expert, such as 0.5, and samples below the preset threshold are considered to be redundant / low-quality samples), and samples with outlier scores higher than the preset threshold are added to the seed set until the size of the seed set reaches α times the size before this iteration (the α value can be set by a domain expert, such as α=2.0), thereby ensuring that the new samples are different from the existing seed set without affecting the prediction accuracy of the recognition model, thereby diluting the redundancy of the seed set obtained in step 110.

[0079] like Figure 3 As shown, after obtaining the seed set in step 110 according to sample downloading - sample deduplication - feature extraction, an isolation tree is constructed in the seed set, and the quality of the new sample is scored based on the isolation tree to determine whether the new sample is a high-quality non-redundant sample. If so, the new sample is added to the seed set sample and the seed set sample is updated.

[0080] Based on the above embodiment, outlier scoring is performed on new samples based on N isolated trees, including:

[0081] Based on N isolated trees, determine the depth value of the new sample on each isolated tree;

[0082] According to each depth value, an outlier scoring function is used to score the new samples as outliers.

[0083] In this embodiment, N isolated trees are constructed on the seed set, each of which is only for a randomly sampled subset of the seed set, and then one of the features is randomly selected from all features to split the seed set, and the process is repeated until the current seed set is a single data set or the depth reaches a threshold. The new sample corresponds to a depth value on each tree. The smaller the depth value, the greater the difference between the new sample and the seed set sample. The isolation forest algorithm calculates the mean depth of the new sample on all isolated trees and substitutes it into the outlier scoring function to obtain the outlier scoring result:

[0084]

[0085] Among them, s(x,n) represents the outlier scoring result, x represents the new sample, n represents the number of nodes used to construct the isolated tree, E(h(x)) represents the mean depth of x on all isolated trees, and H represents the total depth value of the isolated tree.

[0086] It should be noted that the isolation forest algorithm is one of many anomaly detection algorithms, and the application of almost all anomaly detection algorithms is still anomaly detection (such as sudden changes in traffic, a certain data in a lot of data is very different from other data, etc.). The anomaly detection algorithm in this embodiment is not limited to the isolation forest algorithm, and the corresponding anomaly detection algorithm can be selected according to the actual situation. The main highlight of this application is that it is not to eliminate abnormal data, but to regard the abnormal data considered by the algorithm as high-quality non-redundant data, and to regard non-abnormal data as redundant data (for non-abnormal data with high redundancy, that is, low abnormality, it should be eliminated instead). The application of this reverse thinking has led to the application being able to construct a very high-quality data set at a very low cost.

[0087] In addition, in order to improve the quality of the data set as much as possible, manual confirmation of the data labeling is often required (or even multiple rounds of review and confirmation). However, for binary samples of software, manual confirmation is extremely time-consuming and labor-intensive. Therefore, this application can also indirectly significantly reduce the manual review cost of high-quality data sets.

[0088] At present, there is no effective work on reducing redundancy for binary sample data. From the perspective of machine learning, we can certainly use clustering methods to cluster the entire data set, and then take the cluster core as a low-redundancy set. However, the algorithm complexity of the clustering method is higher than that of the isolation forest algorithm, especially for binary samples, which are usually represented as ultra-high-dimensional (usually reaching several k dimensions, or even dozens of k dimensions) vectors. Its running overhead seriously reduces its availability.

[0089] Based on the above embodiments, Figure 4 As shown, the present invention provides a malware identification method, the method comprising the following steps:

[0090] Step 410: Obtain software to be identified;

[0091] Step 420: input the software into a malware identification model to obtain an identification result of the software;

[0092] Among them, the malware identification model is obtained after machine learning training of a first seed set containing malicious samples constructed by the data set construction method of any of the above embodiments, and / or a second seed set containing non-malicious samples constructed by the data set construction method of any of the above embodiments.

[0093] In this embodiment, it should be noted that, since the first seed set containing malicious samples is constructed by the data set construction method of any of the above embodiments, and / or the second seed set containing non-malicious samples is constructed by the data set construction method of any of the above embodiments, redundant samples in the first seed set and the second seed set are eliminated, the scale of the first seed set and the second seed set is reduced, so that the samples in the final data set are highly representative, which not only greatly reduces the noise around the sample points, but also reduces the data scale without reducing the data coverage capability, effectively improving the training convergence speed, prediction accuracy and generalization performance of the recognition model. It can be seen that this embodiment can accurately obtain the recognition result by inputting the software to be identified into the malware recognition model, and the recognition model can be updated and trained accordingly according to the update of the software sample, so that the recognition result of the software to be identified can be updated in real time.

[0094] Based on the above embodiments, Figure 5 As shown, the present invention provides a method for building a malware identification model, the method comprising the following steps:

[0095] Step 510: construct a first seed set containing malicious samples using the data set construction method of any of the above embodiments; and / or,

[0096] Step 520: Use the data set construction method of any of the above embodiments to construct a second seed set containing non-malicious samples;

[0097] Step 530: Based on the first seed set and / or the second seed set, the machine learning model is trained using a machine learning method to obtain a malware identification model.

[0098] In this embodiment, it should be noted that the first seed set containing malicious samples constructed by the data set construction method of any of the above embodiments, and / or the second seed set containing non-malicious samples constructed by the data set construction method of any of the above embodiments, eliminates redundant samples in the first seed set and the second seed set, reduces the size of the first seed set and the second seed set, so that the samples in the final data set are highly representative, which not only greatly reduces the noise around the sample points, but also reduces the data size without reducing the data coverage capability, effectively improves the training convergence speed, prediction accuracy and generalization performance of the recognition model, so that the acquired recognition model can accurately determine whether the software to be identified is malware.

[0099] The data set construction device provided by the present invention is described below. The data set construction device described below and the data set construction method described above can be referenced to each other.

[0100] Based on the above embodiments, Figure 6 As shown, the present invention provides a data set construction device for malware identification, the device comprising:

[0101] A new sample acquisition unit 610 is used to acquire a new sample; wherein the sample type of the new sample is consistent with the sample type in the pre-constructed seed set, wherein the seed set includes malicious samples and / or non-malicious samples;

[0102] An abnormal sample determination unit 620 is used to determine whether the new sample is an abnormal sample based on the new sample and the samples in the seed set by using an abnormal detection algorithm, and if so, add the new sample to the seed set;

[0103] The updating unit 630 is used to cyclically execute the steps in the new sample acquisition unit and the steps in the abnormal sample determination unit until the number of samples in the seed set meets a preset condition.

[0104] Based on the above embodiment, the abnormal sample determination unit 620 is specifically configured to:

[0105] Based on the new sample and the samples in the seed set, the isolation forest algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set.

[0106] Based on the above embodiment, the abnormal sample determination unit 620 is used to determine whether the new sample is an abnormal sample based on the new sample and the samples in the seed set by using the isolation forest algorithm, and if so, add the new sample to the seed set, including:

[0107] Use the isolation forest algorithm or the extended isolation forest algorithm to construct N isolated trees based on the seed set;

[0108] Based on N isolated trees, the new samples are scored as outliers. If the score is higher than the preset threshold, the new samples are added to the seed set.

[0109] Based on the above embodiment, the preset condition is that the first ratio reaches a preset value, and the first ratio is the ratio of the number of first samples in the seed set after the new sample is added to the seed set this time to the number of second samples in the seed set before the new sample is added this time.

[0110] Based on the above embodiment, outlier scoring is performed on new samples based on N isolated trees, including:

[0111] Based on N isolated trees, determine the depth value of the new sample on each isolated tree;

[0112] According to each depth value, an outlier scoring function is used to score the new samples as outliers.

[0113] Based on the above embodiments, Figure 7 As shown, the present invention provides a malware identification device, the device comprising:

[0114] An acquisition unit 710, used to acquire software to be identified;

[0115] The identification unit 720 is used to input the software into the malware identification model to obtain the identification result of the software;

[0116] Among them, the malware identification model is obtained after machine learning training using a first seed set containing malicious samples constructed using the data set construction method described in any of the above embodiments, and / or a second seed set containing non-malicious samples constructed using the data set construction method described in any of the above embodiments.

[0117] Based on the above embodiments, Figure 8 As shown, the present invention provides a malware identification model construction device, the device comprising:

[0118] A first construction unit 810 is configured to construct a first seed set containing malicious samples by using the data set construction method described in any of the above embodiments; and / or,

[0119] A second construction unit 820, configured to construct a second seed set including non-malicious samples by using the data set construction method described in any of the above embodiments;

[0120] The training unit 830 is used to train the machine learning model in a machine learning manner based on the first seed set and / or the second seed set to obtain a malware identification model.

[0121] Fig. 9 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the data set construction method, which includes: a new sample acquisition step: acquiring a new sample; wherein the sample type of the new sample is consistent with the sample type in the pre-constructed seed set, wherein the seed set contains malicious samples and / or non-malicious samples; an abnormal sample determination step: based on the new sample and the samples in the seed set, using an abnormal detection algorithm to determine whether the new sample is an abnormal sample, and if so, adding the new sample to the seed set; an update step: looping the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets the condition.

[0122] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the data set construction method provided by the above-mentioned methods, and the method includes: a new sample acquisition step: acquiring a new sample; wherein the sample type of the new sample is consistent with the sample type in a pre-constructed seed set, wherein the seed set contains malicious samples and / or non-malicious samples; an abnormal sample determination step: based on the new sample and the samples in the seed set, using an anomaly detection algorithm to determine whether the new sample is an abnormal sample, and if so, adding the new sample to the seed set; an update step: cyclically executing the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets the conditions.

[0124] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned data set construction methods, the methods comprising: a new sample acquisition step: acquiring a new sample; wherein the sample type of the new sample is consistent with the sample type in a pre-constructed seed set, wherein the seed set contains malicious samples, and / or non-malicious samples; an abnormal sample determination step: based on the new sample and the samples in the seed set, using an anomaly detection algorithm to determine whether the new sample is an abnormal sample, and if so, adding the new sample to the seed set; an update step: cyclically executing the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets a condition.

[0125] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0126] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a data set for malware identification, characterized in that: include: New sample acquisition step: acquiring new samples; wherein the sample type of the new samples is consistent with the sample type in the pre-constructed seed set, wherein the seed set contains malicious samples and / or non-malicious samples, and the samples in the seed set are screened and deduplicated so that the sample data in the seed set are different; Abnormal sample determination step: Based on the new sample and the samples in the seed set, an abnormal detection algorithm is used to determine whether the new sample is an abnormal sample. If so, the abnormal sample is used as non-redundant data and the new sample is added to the seed set; if it is determined that the new sample is not an abnormal sample, the new sample is removed as a redundant sample to avoid redundancy in the seed set samples; multiple adjacent small versions of the same software, or different versions of viruses belonging to the same family, or the same software infected by similar viruses, will result in multiple similar samples in the data set, and the similar samples are the redundant samples; Update step: cyclically execute the new sample acquisition step and the abnormal sample determination step until the number of samples in the seed set meets a preset condition; the preset condition refers to that the number of samples in the seed set reaches a preset scale, or the number of updates reaches a preset number.

2. The method for constructing a data set according to claim 1, characterized in that: The abnormal sample determination step specifically includes: Based on the new sample and the samples in the seed set, an isolation forest algorithm is used to determine whether the new sample is an abnormal sample. If so, the new sample is added to the seed set.

3. The method for constructing a data set according to claim 2, characterized in that: Based on the new sample and the samples in the seed set, an isolation forest algorithm is used to determine whether the new sample is an abnormal sample, and if so, the new sample is added to the seed set, including: Use the isolation forest algorithm or the extended isolation forest algorithm to construct N isolated trees based on the seed set; Based on the N isolated trees, the new sample is scored as an outlier, and if the score is higher than a preset threshold, the new sample is added to the seed set.

4. The method for constructing a data set according to claim 3, characterized in that: Based on the N isolated trees, outlier scoring is performed on the new sample, including: Based on the N isolated trees, determining a depth value of the new sample on each isolated tree; According to each depth value, an outlier scoring function is used to perform an outlier scoring on the new sample.

5. A method for identifying malware, characterized in that: include: Obtain the software to be identified; Inputting the software into a malware identification model to obtain an identification result of the software; The malware identification model is obtained after machine learning training using a first seed set containing malicious samples constructed using the dataset construction method described in any one of claims 1 to 4, and / or a second seed set containing non-malicious samples constructed using the dataset construction method described in any one of claims 1 to 4.

6. A method for constructing a malware identification model, characterized in that: include: Constructing a first seed set containing malicious samples using the data set construction method according to any one of claims 1 to 4; and / or, Constructing a second seed set containing non-malicious samples using the data set construction method according to any one of claims 1 to 4; Based on the first seed set and / or the second seed set, the machine learning model is trained using a machine learning method to obtain a malware identification model.

7. A data set construction device for malware identification, characterized in that: include: A new sample acquisition unit, used to acquire a new sample; wherein the sample type of the new sample is consistent with the sample type in the pre-constructed seed set, wherein the seed set includes malicious samples and / or non-malicious samples, and the samples in the seed set are screened and deduplicated so that the sample data in the seed set are different; The abnormal sample determination unit is used to determine whether the new sample is an abnormal sample based on the new sample and the samples in the seed set by using an abnormal detection algorithm. If so, the abnormal sample is used as non-redundant data and the new sample is added to the seed set; if it is determined that the new sample is not an abnormal sample, the new sample is removed as a redundant sample to avoid redundancy in the seed set samples; multiple adjacent small versions of the same software, or different versions of viruses belonging to the same family, or the same software infected by similar viruses, will result in multiple similar samples in the data set, and the similar samples are the redundant samples; An updating unit is used to cyclically execute the steps in the new sample acquisition unit and the steps in the abnormal sample determination unit until the number of samples in the seed set meets a preset condition; the preset condition refers to that the number of samples in the seed set reaches a preset scale, or the number of updates reaches a preset number.

8. A malware identification device, characterized in that: include: An acquisition unit, used for acquiring software to be identified; an identification unit, configured to input the software into a malware identification model to obtain an identification result of the software; The malware identification model is obtained after machine learning training using a first seed set containing malicious samples constructed using the dataset construction method described in any one of claims 1 to 4, and / or a second seed set containing non-malicious samples constructed using the dataset construction method described in any one of claims 1 to 4.

9. A malware identification model construction device, characterized in that: include: A first construction unit, configured to construct a first seed set containing malicious samples by using the data set construction method according to any one of claims 1 to 4; and / or, A second construction unit, configured to construct a second seed set containing non-malicious samples by using the data set construction method according to any one of claims 1 to 4; A training unit is used to train a machine learning model based on the first seed set and / or the second seed set using a machine learning method to obtain a malware identification model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the data set construction method as described in any one of claims 1 to 4 are implemented; and / or, when the processor executes the program, the steps of the malware identification method as described in claim 5 are implemented; and / or, when the processor executes the program, the steps of the malware identification model construction method as described in claim 6 are implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the steps of the data set construction method as described in any one of claims 1 to 4 are implemented; and / or, when the processor executes the program, the steps of the malware identification method as described in claim 5 are implemented; and / or, when the processor executes the program, the steps of the malware identification model construction method as described in claim 6 are implemented.

12. A computer program product having executable instructions stored thereon, characterized in that: When the instruction is executed by the processor, the processor implements the steps of the data set construction method as described in any one of claims 1 to 4; and / or, when the processor executes the program, the processor implements the steps of the malware identification method as described in claim 5; and / or, when the processor executes the program, the processor implements the steps of the malware identification model construction method as described in claim 6.

Images