Machine learning based malware family signature extraction and visualization method

Abstract

The present application relates to computer technology, machine learning and information security field, especially to a malware family main feature extraction and visualization method based on machine learning, including feature selection, family classification, feature contribution extraction, normalization processing and visualization steps; through static or dynamic analysis, the behavior data of malware samples are collected, features are selected and converted into matrix form, classification is carried out based on machine learning algorithm, feature contribution is extracted and normalized, and according to the feature contribution matrix, different features are visualized through color coding method; macroscopic visualization displays the similarities and differences between families, and microscopic visualization displays specific feature content; the present application effectively extracts and displays the main features and differences of malware families, provides decision basis for classification algorithm, and has the characteristics of high efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the fields of computer technology, machine learning, and information security, and in particular to a method for extracting and visualizing key features of malware families based on machine learning. Background Technology

[0002] In the field of machine learning-based malware classification, despite the existence of various high-precision and high-efficiency classification algorithms, several key issues remain unresolved due to the "black box" effect of machine learning algorithms:

[0003] Do machine learning classification algorithms truly learn the representative characteristics of malware families?

[0004] Machine learning classification algorithms learn which feature differences to correctly classify malware families.

[0005] Although researchers have proposed various interpretable machine learning frameworks to explain the decision-making basis and process of machine learning algorithms, in the field of information security, especially in the classification of malware families, existing methods cannot confirm whether high-contribution features are the main features of malware, nor can they specifically show the similarities and differences in features between different families.

[0006] Therefore, there is an urgent need for a method that can effectively extract the main features of malware, specifically display the content of these features, and visualize the similarities and differences in features between different families. Summary of the Invention

[0007] To address the numerous problems existing in the prior art, this invention provides a method for extracting and visualizing the main features of malware families based on machine learning. This invention extracts and visualizes the main features of malware families through feature selection, machine learning classification, feature contribution extraction, normalization processing, and color encoding. It utilizes the RGBA color overlap principle to display the similarities and differences in features between different families, ultimately achieving accurate extraction and intuitive display of malware family features, providing clear classification criteria and detailed feature difference analysis.

[0008] A machine learning-based method for extracting and visualizing key features of malware families includes the following steps:

[0009] The collected malware samples are analyzed, and the features that can be used for family classification are selected and converted into a matrix form that can be used for machine learning algorithms.

[0010] Classification of malware families based on machine learning algorithms;

[0011] Analyze the characteristics of machine learning classification algorithms, and extract the classification contribution of each feature to each family based on the classification results, generating a feature contribution matrix;

[0012] The extracted feature contribution values ​​are normalized to obtain the normalized feature contribution matrix.

[0013] Based on the feature contribution matrix, different features are visualized using a color coding method;

[0014] The visualization results showcase the similarities and differences in characteristics among different families, highlighting the main features and differences of each family.

[0015] Preferably, behavioral data of malware samples are collected through static or dynamic analysis, and features are selected, including API call sequences, network communication data, and byte order.

[0016] Preferably, the selection of features includes: selecting the first 512 bytes of content starting from the Entry Point of the malware sample as a feature.

[0017] Preferably, a machine learning algorithm is selected, and the model is trained based on cross-validation and the optimal parameters are found.

[0018] Preferably, the machine learning algorithm is the Linear SVM algorithm, and the Linear SVM decision function calculation expression is:

[0019]

[0020] in, For the input feature vector, The weight vector assigned to all features. This is the bias parameter.

[0021] Preferably, it also includes: extracting the weight vector. elements in As the feature contribution, and generate the feature contribution matrix. ,in, Indicates the first The weights of each feature.

[0022] Preferably, when the total number of families When using the One-VS-Rest strategy, Classification tasks are divided into The binary classification subtask yields the classification contribution matrix of each feature to all families. ,in, Indicates the number of features. Indicates the number of families. Indicates the first The feature is related to the first The contribution of each family.

[0023] Preferably, a scaling function is used to normalize the feature contribution values ​​to obtain a normalized feature contribution matrix. The scaling function is calculated as follows:

[0024]

[0025] in, This is the scaling factor. This is the bias value.

[0026] Preferably, based on the feature contribution matrix, different features are visualized using a color coding method, including:

[0027] The feature contribution is represented using the RGBA color model, where the Alpha channel value represents the feature contribution. The calculation expression is as follows:

[0028]

[0029] in, Features transparency, For features The number of families that made contributions.

[0030] Preferably, visualizations include:

[0031] Use Matplotlib's matshow function to visualize the feature contribution matrix macroscopically, showing the similarities and differences in features among different families;

[0032] The specific feature content in the micro-visual feature contribution matrix shows the main features of each family and their differences.

[0033] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows:

[0034] This invention achieves precise extraction and visualization of the main features of malware families through feature selection, classification, and contribution analysis techniques.

[0035] This invention achieves a clear display of the differences in features among different malware families by using feature contribution normalization processing and coloring techniques.

[0036] This invention achieves a comprehensive comparison and detailed analysis of the main characteristics of malware families through macroscopic and microscopic visualization techniques. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the method of the present invention. Detailed Implementation

[0038] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0039] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0040] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0041] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0042] The accompanying drawings illustrate several block diagrams and / or flowcharts. It should be understood that some blocks, or combinations thereof, in the block diagrams and / or flowcharts can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that, when executed by the processor, these instructions can create means for implementing the functions / operations described in these block diagrams and / or flowcharts. The technology of this disclosure can be implemented in hardware and / or software (including firmware, microcode, etc.). Alternatively, the technology of this disclosure can take the form of a computer program product stored on a computer-readable storage medium, which is available for use by or in conjunction with an instruction execution system.

[0043] like Figure 1 As shown, a machine learning-based method for extracting and visualizing key features of malware families includes the following steps:

[0044] The collected malware samples are analyzed, and the features that can be used for family classification are selected and converted into a matrix form that can be used for machine learning algorithms.

[0045] This invention collects malware samples from publicly available resources, such as VirusShare and VirusTotal;

[0046] Preferably, behavioral data of malware samples are collected through static or dynamic analysis, and features are selected, including API call sequences, network communication data, and byte order.

[0047] Behavioral data of malware samples are collected through static or dynamic analysis, and features such as API call sequences, network communication data, and byte order are selected from them.

[0048] Preferably, the selection of features includes: selecting the first 512 bytes of content starting from the Entry Point of the malware sample as a feature.

[0049] The extracted malware features and corresponding family tags are converted into a matrix using programming languages ​​such as Python and C. Of these, the number of families is The number of features is ;

[0050] Classification of malware families based on machine learning algorithms;

[0051] Preferably, a machine learning algorithm is selected, and the model is trained based on cross-validation and the optimal parameters are found.

[0052] Choose machine learning algorithms, such as SVM, random forest, KNN, KMeans, etc., and train the model and find the optimal parameters based on cross-validation.

[0053] Preferably, the machine learning algorithm is the Linear SVM algorithm, and the Linear SVM decision function calculation expression is:

[0054]

[0055] in, For the input feature vector, The weight vector assigned to all features. This is the bias parameter.

[0056] In this embodiment of the invention, Linear SVM (LSVM) is selected as the classification algorithm; family classification is performed with optimal parameters, and the family classification results are obtained.

[0057] Analyze the characteristics of machine learning classification algorithms, and extract the classification contribution of each feature to each family based on the classification results, generating a feature contribution matrix;

[0058] Preferably, it also includes: extracting the weight vector. elements in As the feature contribution, and generate the feature contribution matrix. ,in, Indicates the first The weights of each feature.

[0059] Preferably, when the total number of families When using the One-VS-Rest strategy, Classification tasks are divided into The binary classification subtask yields the classification contribution matrix of each feature to all families. ,in, Indicates the number of features. Indicates the number of families. Indicates the first The feature is related to the first The contribution of each family.

[0060] When the total number of families In binary classification, each row of the feature contribution matrix contains the first row. The contribution of each feature to the classification of the two families;

[0061] When the total number of families In multi-class classification tasks, a One-VS-Rest-based strategy can be used to segment the class classification task into... A binary sub-task,

[0062] The extracted feature contribution values ​​are normalized to obtain the normalized feature contribution matrix.

[0063] Preferably, a scaling function is used to normalize the feature contribution values ​​to obtain a normalized feature contribution matrix. The scaling function is calculated as follows:

[0064]

[0065] in, This is the scaling factor. This is the bias value.

[0066] Use programming languages ​​and visualization tools (such as Matplotlib, Seaborn, etc.) to visualize the distribution of feature contributions and determine the normalization function. Fixed points in The value;

[0067] Use normalization processing function Feature contribution matrix Normalization is performed, which involves normalizing the feature contributions from different value ranges to the same range, resulting in a matrix. ;

[0068] Based on the feature contribution matrix, different features are visualized using a color coding method;

[0069] Preferably, based on the feature contribution matrix, different features are visualized using a color coding method.

[0070] This includes: using the RGBA color model to represent feature contribution, where the Alpha channel value represents the feature contribution. The calculation expression is as follows:

[0071]

[0072] in, Features transparency, For features The number of families that made contributions.

[0073] In an embodiment of the present invention, the color encoding method includes:

[0074] Different families are assigned different colors, such as red, green, and blue for three families, while other families can be assigned gray. These colors are used for easier calculation. The values ​​are processed, including:

[0075] red:

[0076] green:

[0077] blue:

[0078] grey:

[0079] The Alpha channel value represents the transparency of the color. In this invention, it is used to indicate the magnitude of the feature contribution. The higher the contribution, the less transparent the color, and vice versa.

[0080] When a feature contributes to the classification of all families, meaning it is a feature shared by all families, it is considered an invalid feature. Therefore, this method is limited in its calculation of contribution. It needs to be less than the total number of families. ;

[0081] The visualization results showcase the similarities and differences in characteristics among different families, highlighting the main features and differences of each family.

[0082] Preferably, visualizations include:

[0083] Use the matshow function in the Matplotlib library for macroscopic visualization to show the similarities and differences between features of different families in the feature contribution matrix;

[0084] Microscopic visualization is performed to display the specific features in the feature contribution matrix, showing the main features of each family and their differences.

[0085] Macro-level visualization: It can show the similarities and differences in characteristics between different families, that is, it shows the similarities and differences in characteristics between different families as a whole, but does not show specific characteristics.

[0086] Visualize matrices using the matshow function of Matplotlib, a Python visualization tool. The Y-axis represents the family name, and the X-axis represents the first 512 bytes of the malware sample starting from the Enyry Point.

[0087] Microscopic visualization: It can reflect the specific content of different family characteristics, that is, present the specific content of family characteristics in the visualization results, so as to extract the main characteristic content of each family, and analyze the specific characteristic differences between different families or different subspecies within the same family.

[0088] Based on the macroscopic visualization results, the contents of the 512-byte sequence are displayed.

[0089] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0090] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0091] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0092] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0093] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0094] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0095] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0096] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0097] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for extracting and visualizing key features of malware families based on machine learning, characterized in that, Includes the following steps: The collected malware samples are analyzed, and the features that can be used for family classification are selected and converted into a matrix form that can be used for machine learning algorithms. Classification of malware families based on machine learning algorithms; Analyze the characteristics of machine learning classification algorithms, and extract the classification contribution of each feature to each family based on the classification results, generating a feature contribution matrix; The extracted feature contribution values ​​are normalized to obtain the normalized feature contribution matrix. Extracting weight vectors elements in As the feature contribution, and generate the feature contribution matrix. ,in, Indicates the first The weight of each feature; when the total number of families When using the One-VS-Rest strategy, Classification tasks are divided into The binary classification subtask yields the classification contribution matrix of each feature to all families. ,in, Indicates the number of features. Indicates the number of families. Indicates the first The feature is related to the first The contribution of each family; The feature contribution values ​​are normalized using a scaling function to obtain the normalized feature contribution matrix. The scaling function is calculated as follows: in, This is the scaling factor. This is the bias value; Based on the normalized feature contribution matrix, different features are visualized using a color coding method; The visualization results showcase the similarities and differences in characteristics among different families, highlighting the main features and differences of each family.

2. The method according to claim 1, characterized in that, Behavioral data of malware samples are collected through static or dynamic analysis, and features are selected, including API call sequences, network communication data, and byte order.

3. The method according to claim 2, characterized in that, Feature selection includes: selecting the first 512 bytes of content starting from the Entry Point of the malware sample as a feature.

4. The method according to claim 1, characterized in that, Choose a machine learning algorithm, train the model based on cross-validation, and find the optimal parameters.

5. The method according to claim 4, characterized in that, The machine learning algorithm is the Linear SVM algorithm, and the Linear SVM decision function calculation expression is: in, For the input feature vector, The weight vector assigned to all features. For bias parameters, This refers to the feature dimension, i.e., the number of features.

6. The method according to claim 1, characterized in that, Based on the feature contribution matrix, different features are visualized using a color coding method, including: The feature contribution is represented using the RGBA color model, where the Alpha channel value represents the feature contribution. The calculation expression is as follows: in, Features transparency, For features The number of families that made contributions.

7. The method according to claim 1, characterized in that visualization include: Use the matshow function in the Matplotlib library for macroscopic visualization to show the similarities and differences between features of different families in the feature contribution matrix; Microscopic visualization is performed to display the specific features in the feature contribution matrix, showing the main features of each family and their differences.

Images