A Method, System and Medium for Inferring the Truth of a Malware Family

Abstract

This application is a method, system and medium for inferring the truth of malware families. Aiming at the problem of inaccurate labeling in existing methods, a method for inferring the truth of malware families is provided, including the following steps: distributing the samples to be tested to multiple antivirus engines for scanning, obtaining and collecting malware labels through the reports of the antivirus engines; realizing the tokenization of malware labels by decomposing the malware labels into token sequences; regarding the type label and the platform label as locators, and determining the locators by searching the obtained token sequences; searching for and outputting family labels according to the position distribution of the locators; dividing the labeling situations according to the extracted family labels to obtain strongly labeled samples, weakly labeled samples and unlabeled samples; using the n-vote voting method to output the truth inference results of the strongly labeled samples. The position-first search algorithm has a higher accuracy rate for extracting family labels; only the strongly labeled samples are used for truth inference to ensure the accuracy rate of malware family recognition.

Description

Technical Field

[0001] The present invention relates to the field of malware, and more particularly to a method, system and medium for inferring the truth of a malware family. Background Art

[0002] Online malware analysis platforms, such as VirusTotal, aggregate well-known antivirus engines worldwide and provide rich APIs to facilitate uploading samples and downloading the scanning results of antivirus engines. When an antivirus engine detects a sample as malware, it reports a malware label to elaborate on the detection result. The malware label consists of tags such as family, type, and platform. The above property is widely used by the security community for labeling and constructing malware benchmark data, which is called crowdsourcing malware annotation.

[0003] Crowdsourcing malware annotation includes two main tasks: category tag extraction and truth inference. Among them, category tag extraction realizes the association between the tags in the malware label and the corresponding tag categories, and truth inference realizes inferring the unknown truth of the sample from the extracted tags.

[0004] The first challenge in applying the above paradigm for malware family annotation is how to accurately extract the family tags from the chaotic malware labels. CARO proposed a virus naming convention in 1991, attempting to bring consistency to the naming of malware labels. Although most antivirus engines today use similar components (type, platform, and family) recommended by CARO to construct malware labels, each antivirus engine has its own unique tag vocabulary and naming pattern. The difficulties in extracting category tags are as follows: (1) The naming patterns of antivirus engines cannot be enumerated by regular expressions. For example, although the malware labels elf:mirai-fug and win32:adware-gen are the same in structure, adware is a type tag while mirai is a family tag; (2) The tag vocabularies of antivirus engines are always changing, and most engines do not disclose their custom tags, especially family tags, which makes it very difficult to establish an effective machine learning-based tag classification model. Some existing methods have tried to solve the problem of extracting family tags, such as Euphony and AVClass, but the accuracy of these methods is still low.

[0005] The second challenge in malware family annotation using the above annotation paradigm is how to accurately infer the true family of an unknown sample from inconsistent family labels. Antivirus engines usually have different opinions on which family a sample belongs to. Existing works, such as Euphony and AVClass, usually assume that the quality of antivirus engines is equally reliable and heavily rely on the n-vote voting scheme to infer the true family of a sample. This scheme is prone to bias in the case where the number of votes for family labels is close. Summary of the Invention

[0006] Aiming at the problems existing in the prior art, the present invention provides a method, system and medium for inferring the true family of malware, and innovatively proposes: a location-first search method and a family truth annotation method based on label situation division. The former effectively solves the problem of extracting family labels from chaotic malware; the latter collects only samples with no disputes about the family truth through label situation division to improve the quality of family annotation.

[0007] A method for inferring the true family of malware includes the following steps:

[0008] S1, Distribute the sample to be tested to multiple antivirus engines for scanning, and obtain and collect malware labels through the reports of the antivirus engines;

[0009] S2, Use the location-first search method to extract family labels from the obtained malware labels, including:

[0010] Tokenize the malware label by decomposing it into a label sequence; regard the type label and platform label as locators, and determine the locators by searching the obtained label sequence; search for and output family labels according to the position distribution of the locators;

[0011] S3, Perform label situation division according to the extracted family labels to obtain strongly labeled samples, weakly labeled samples and unlabeled samples; use the n-vote voting method to output the true family inference result of the strongly labeled samples to obtain the true family inference result of the sample.

[0012] Further, in step S2, the determined locators form an initial locator label vocabulary.

[0013] Further, the specific process of searching for and outputting family labels according to the position distribution of the locators is: search for locator labels in the label sequence according to the locator label vocabulary. If the position distribution of the locators is continuous, search at both ends of the locator positions; if the position distribution of the locators is discontinuous, search in the middle of the locator positions, and return the label with the smallest position number in the search area as the family label.

[0014] Further, the specific process of step S3 is as follows: Extract the sample x i whose family label is where the number of occurrences of family label k is N k , I(.) is the sign function; Arrange all in descending order, denoted as Perform label situation division according to formula (1):

[0015]

[0016] In the formula: J is the total number of antivirus engines; j represents any antivirus engine; represents the k-th family label of the i-th sample; m is the first hyperparameter; K is the total number of family labels.

[0017] Further, the specific process of step S4 is as follows: Infer that a sample belongs to family k if and only if at least n antivirus engines consider that the sample belongs to family k and k has the most votes.

[0018] Further, the inferred family label is:

[0019]

[0020] wherein, is an invalid inference, indicating that family labeling cannot be performed on this sample.

[0021] A malware family truth inference system includes:

[0022] A crowdsourcing module that distributes samples to be tested to multiple antivirus engines for scanning, and obtains and collects malware labels through the reports of the antivirus engines;

[0023] A family label extraction module that extracts family labels from the obtained malware labels using the position-first search method, including: tokenizing the malware labels by decomposing them into label sequences; regarding the type label and the platform label as locators, and determining the locators by searching the obtained label sequences; searching for and outputting family labels according to the position distribution of the locators;

[0024] A family truth inference module that performs label situation division based on the extracted family labels to obtain strongly-tagged samples, weakly-tagged samples, and null-tagged samples; adopts an n-vote voting scheme to output the truth inference results of the strongly-tagged samples, and obtains the sample family truth inference results.

[0025] A malware family truth inference system includes a memory and one or more processors. An executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the above-mentioned malware family truth inference method.

[0026] A computer-readable storage medium stores a program, which when executed by a processor, implements the above-mentioned malware family truth inference method.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The malware family truth inference method, system and medium provided by the present invention analyze a large amount of malware label data, obtain the relationship between family markers and locator markers (type markers and platform markers), and the proposed location-priority search algorithm has a higher accuracy rate for extracting family markers. At the same time, the present application divides the marker situations and only performs truth inference on strongly marked samples, further ensuring the accuracy rate of malware family identification. Description of the Drawings

[0029] Figure 1 It is a flowchart of Embodiment 1.

[0030] Figure 2 It is a schematic diagram of family marker extraction in Embodiment 1.

[0031] Figure 3 It is a schematic diagram of family truth inference in Embodiment 1.

[0032] Figure 4 It is an architecture diagram of Embodiment 2. Detailed Embodiments

[0033] The following introduces a preferred embodiment of the present invention with reference to the drawings to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0034] In the drawings, components with the same structure are denoted by the same numerical labels, and components with similar structures or functions are denoted by similar numerical labels. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. To make the illustration clearer, the thickness of some components in the drawings is appropriately exaggerated.

[0035] Embodiment 1

[0036] This embodiment is a malware family truth inference method, as Figures 1 to 3 shown, including the following steps:

[0037] The uploaded samples enter the database module for query and archiving; the data samples calculate the hash value and query in the database; if not present, crowdsourcing, family marker extraction, and family truth inference are performed to update the database; if present, the labeling result is directly returned.

[0038] The samples to be tested are distributed to multiple antivirus engines for scanning, and malicious software labels are obtained and collected through the reports of the antivirus engines.

[0039] The family markers are extracted from the obtained malicious software labels using the position - first search method, including:

[0040] The malicious software label tokenization is achieved by decomposing the malicious software label into a token sequence (for example, the malicious software label of trojan.ransom.gandcrab is decomposed into the token sequence (trojan,ransom,gandcrab)); the type token and platform token are regarded as locators, and the locators are determined by searching the obtained token sequence, and the determined locators form an initial locator - token vocabulary; the family marker is searched and output according to the position distribution of the locators: search for the locator tokens in the token sequence according to the locator - token vocabulary, if the position distribution of the locators is continuous, search at both ends of the locator positions; if the position distribution of the locators is discontinuous, search in the middle of the locator positions, and return the token with the smallest position number in the search area as the family marker.

[0041] Tokenize the malicious software label l to obtain the token sequence t=(t 1 ,t 2 ,…,t m ); determine the positions of the locator tokens in the token sequence according to the locator - token vocabulary v (loc) ; search and output the family marker according to the position distribution of the locator tokens. Specifically: if the position distribution of the locators is continuous, search at both ends of the locator positions; if the position distribution of the locators is discontinuous, search in the middle of the locator positions. Then return the token with the smallest position number in the search area as the family marker. The position - first search method is described in Table 1.

[0042] Table 1

[0043]

[0044]

[0045] Perform label - situation division based on the extracted family markers to obtain strongly - labeled samples, weakly - labeled samples, and unlabeled samples: extract the family marker of sample x i as where the number of occurrences of the family marker k is N k , I(.) is the sign function; all are sorted in descending order and denoted as According to formula (1), the marking situation is divided as follows:

[0046]

[0047] In the formula: J is the total number of anti-virus engines; j represents any anti-virus engine; represents the k-th family mark of the i-th sample; m is the first hyperparameter; K is the total number of family marks.

[0048] The n-vote voting scheme is adopted to output the truth inference result of the strong mark sample, and the sample family truth inference result is obtained: It is inferred that a sample belongs to family k if and only if at least n anti-virus engines think that the sample belongs to family k and the number of votes for k is the largest. The inferred family mark is:

[0049]

[0050] Among them, is an invalid inference, indicating that the family annotation cannot be performed on this sample. In this embodiment, m = 2 and n = 2.

[0051] Embodiment 2

[0052] As Figure 4 shown, it is an example architecture diagram of a malware family truth inference system of a preferred embodiment of the present invention. The entire system adopts a browser / server architecture, and the core functions are implemented on the server side.

[0053] The malware family truth inference system of this embodiment includes:

[0054] A database module, configured to store the crowdsourcing results, family mark extraction results, and family truth inference results of samples;

[0055] A crowdsourcing module, which distributes the samples to be tested to multiple anti-virus engines for scanning, and obtains and collects malware tags through the reports of the anti-virus engines;

[0056] A family mark extraction module, which extracts family marks from the obtained malware tags by using the position-first search method, including: tokenizing the malware tags by decomposing them into a tag sequence; regarding the type tag and the platform tag as locators, and determining the locators by searching the obtained tag sequence; searching the position distribution of the locators to output family marks;

[0057] The family truth inference module divides the marked situations according to the extracted family marks to obtain strongly marked samples, weakly marked samples, and unmarked samples; and adopts an n-vote voting scheme to output the truth inference results of the strongly marked samples, thereby obtaining the family truth inference results of the samples.

[0058] The browser side of the malware family truth inference system based on position-first search and marked situation division is configured to upload data samples from users and present the marked results. Users only need to use any electronic device with browser functions to access the Internet to upload malware and view the family truth inference results output by the annotation system.

[0059] Embodiment 3

[0060] A malware family truth inference system, characterized in that it includes a memory and one or more processors. Executable code is stored in the memory, and when the one or more processors execute the executable code, it is used to implement the malware family truth inference method described in Embodiment 1.

[0061] Embodiment 4

[0062] A computer-readable storage medium, characterized in that a program is stored thereon, and when the program is executed by a processor, it implements the malware family truth inference method described in Embodiment 1.

[0063] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art shall fall within the protection scope determined by the claims.

Claims

1. A malware family truth inference method, characterized in that: The steps include: S1, distribute the samples to be tested to multiple antivirus engines for scanning, and obtain and collect malware labels through antivirus engine reports; S2, extracts family tags from the obtained malware labels using a position-first search method, including: The malware label is tokenized by decomposing it into a tag sequence; the type tag and platform tag are regarded as locators, and the locator is determined by searching the tag sequence; the family tag is searched and output according to the position distribution of the locator; the determined locator forms an initial locator tag vocabulary; the family tag is searched and output according to the position distribution of the locator. The specific process is: according to the locator tag vocabulary, the locator tag is searched in the tag sequence, if the position distribution of the locator is continuous, the search is performed at both ends of the locator position; if the position distribution of the locator is discontinuous, the search is performed in the middle of the locator position; then the tag with the smallest position number in the search area is returned as the family tag; S3, according to the extracted family markers, the marking situation is divided to obtain strong marking samples, weak marking samples and unmarked samples; the n-vote voting method is used to output the truth inference results of the strong marking samples, and the truth inference results of the sample family are obtained; the specific process of marking situation division is: extract sample x i The family marker is The number of occurrences of family marker k is N k , I(.) is a symbolic function; Arrange in descending order, denoted as According to formula (1), the marking situation is divided: Where: J is the total number of antivirus engines; j represents any antivirus engine; represents the kth family marker of the i-th sample; m is the first hyperparameter; K is the total number of family markers.

2. According to the malware family truth inference method described in claim 1, the specific process of using the n-vote voting method to output the truth inference result of the strongly labeled sample is: inferring that a sample belongs to family k if and only if greater than or equal to n antivirus engines believe that the sample belongs to family k and k has the largest number of votes.

3. The malware family truth inference method according to claim 2, characterized in that: The inferred family markers are: in, This is an invalid inference, indicating that family annotation cannot be performed for this sample.

4. A malware family truth inference system, characterized in that: include: The crowdsourcing module distributes the samples to be tested to multiple antivirus engines for scanning, and obtains and collects malware labels through antivirus engine reports; The family tag extraction module extracts the family tag from the obtained malware tag by using the position-first search method, including: tokenizing the malware tag by decomposing the malware tag into a tag sequence; treating the type tag and the platform tag as locators, and determining the locator by searching the obtained tag sequence; searching and outputting the family tag according to the position distribution of the locator; forming an initial locator tag vocabulary with the determined locator; searching and outputting the family tag according to the position distribution of the locator. The specific process is: searching for the locator tag in the tag sequence according to the locator tag vocabulary, if the position distribution of the locator is continuous, searching at both ends of the locator position; if the position distribution of the locator is discontinuous, searching in the middle of the locator position; and then returning the tag with the smallest position number in the search area as the family tag; The family truth inference module divides the labeling situation according to the extracted family labels to obtain strong labeling samples, weak labeling samples and unlabeled samples; the n-vote voting method is used to output the truth inference results of the strong labeling samples to obtain the sample family truth inference results; the specific process of labeling situation division is: extract sample x i The family marker is The number of occurrences of family marker k is N k , I(.) is a symbolic function; Arrange in descending order, denoted as According to formula (1), the marking situation is divided: Where: J is the total number of antivirus engines; j represents any antivirus engine; represents the kth family marker of the i-th sample; m is the first hyperparameter; K is the total number of family markers.

5. A malware family truth inference system, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the malware family truth inference method described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, the malware family truth inference method described in any one of claims 1-3 is implemented.

Images