Identification method, system and equipment for open set shelled software and medium
By extracting function call graph structure information and performing lightweight statistical calculations using disassemblers, and combining this with a multi-model single-class identification strategy, the difficulty of detecting unknown types of packed software in existing technologies has been solved, achieving efficient and accurate identification of packed software, especially in resource-constrained environments.
Patent Information
- Application Number
- CN202511282284.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-09
- Publication Date
- 2026-01-02
AI Technical Summary
Existing methods for detecting packed software have significant limitations when dealing with unknown types of packed software under open set conditions. They are difficult to adapt to new types of attacks and have low computational efficiency in resource-constrained environments.
The function call graph structure information is extracted using a disassembler, and combined with lightweight statistical computation and a multi-model single-class recognition strategy, feature vectors are generated and a single-class support vector machine algorithm is used for classification and recognition.
It improves the ability to detect unknown types of packed software, enhances the accuracy and adaptability of identification, reduces the consumption of computing resources, and is suitable for resource-constrained environments.
Smart Images
Figure CN121256784A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of identification technology for open set packed software, and particularly relates to an identification method, system, device and medium for open set packed software. BACKGROUND
[0002] Packed technology is widely used in modern software protection, especially in the detection and defense of malicious software. Existing packed software detection methods mainly rely on entropy features, deep networks or other heuristic methods to identify known types of packed software. Such methods show high efficiency and accuracy in the detection of known sample sets, and are widely used in antivirus tools and malicious software analysis platforms.
[0003] However, current methods mostly assume that all samples to be tested belong to a known fixed type set, i.e. the so-called closed set condition. In actual scenarios, as attackers continuously iterate and improve the packing form of malicious software, new types of packing technology emerge in an endless stream. Existing methods show obvious limitations when dealing with unknown types of packed software under open set conditions. This deficiency greatly limits the adaptability of existing detection technology in the face of new attacks, and also allows malicious software to evade detection and lurk. SUMMARY
[0004] To solve the above technical problems, the present application provides the following technical solutions:
[0005] In a first aspect, the present application provides an identification method for open set packed software, comprising extracting function call graph structure information from a binary executable file to be identified through a disassembler tool and its custom plug-in, and generating a structured graph representation file;
[0006] Based on the graph representation file, graph statistical features and file section structure features are extracted to form a spliced feature vector;
[0007] Based on a multi-model single-class identification strategy constructed for open set identification, the binary executable file corresponding to the current feature vector is classified and identified.
[0008] As a preferred scheme of the identification method for open set packed software of the present application, based on the graph representation file, the graph statistical features and the file section structure features are extracted to form a spliced feature vector, comprising,
[0009] Based on the graph representation file, a set of pre-defined graph statistical features are calculated;
[0010] A set of section structure features are extracted from the header information of the binary executable file to be identified;
[0011] The graph statistical features and the section structure features are spliced to form a feature vector for identification.
[0012] As a preferred scheme of the identification method for open-set packed software, the feature vector is classified and identified based on a multi-model single-class identification strategy constructed for open-set identification, and the identification method comprises the following steps of:
[0013] The feature vector is input into a target identification model set for determination, wherein the target identification model set comprises a plurality of single-class identification models corresponding to a plurality of known packing types one by one.
[0014] The type of the binary executable file to be identified corresponding to the current feature vector is determined according to the determination result.
[0015] As a preferred scheme of the identification method for open-set packed software, the type of the binary executable file to be identified corresponding to the current feature vector is determined according to the determination result, and the determination method comprises the following steps of:
[0016] If the current feature vector is determined by all single-class identification models as not belonging to the known packing type corresponding thereto, the type of the binary executable file to be identified corresponding to the current feature vector is determined as an open-set unknown type packed software.
[0017] If the current feature vector is determined by any single-class identification model as belonging to the known packing type corresponding thereto, the type of the binary executable file to be identified corresponding to the current feature vector is determined as a known type.
[0018] As a preferred scheme of the identification method for open-set packed software, the graph representation file comprises node information recording function node numbers and names, and directed edge information recording the correspondence between caller and callee node numbers.
[0019] As a preferred scheme of the identification method for open-set packed software, the single-class identification model is obtained by training sample feature vectors corresponding to known packing types, and is used to define the distribution range of the type in the feature space.
[0020] The single-class identification model is obtained by using a single-class support vector machine algorithm.
[0021] As a preferred scheme of the identification method for open-set packed software, the graph statistical features and the file section structure features are extracted based on the graph representation file, and lightweight statistical calculation is used without using a neural network model.
[0022] In a second aspect, the present application provides an identification system for open set shell software, comprising: a generation module configured to extract function call graph structure information from a binary executable file to be identified by using a disassembling tool and a self-defined plug-in thereof, and generate a structured graph representation file;
[0023] a splicing module configured to extract graph statistical features and file section structure features based on the graph representation file, and form a spliced feature vector;
[0024] an identification module configured to perform classification identification on the binary executable file corresponding to the current feature vector based on a multi-model one-class identification strategy constructed for open set identification.
[0025] In a third aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0026] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the above method.
[0027] Compared with the prior art, the present application has the following beneficial effects: by combining the function call graph structure information extracted by the disassembling tool and the feature extracted by the lightweight statistical calculation, and using the multi-model one-class identification strategy, the accuracy and adaptability of shell software identification are effectively improved, especially the detection capability of unknown type shell software under the open set condition is significantly enhanced, and the calculation efficiency is high and the resource consumption is low, which is suitable for resource limited environment. Further, the one-class identification model trained by the one-class support vector machine algorithm can accurately define the distribution range of the known shell type in the feature space, thereby realizing efficient identification of unknown types. BRIEF DESCRIPTION OF DRAWINGS
[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.
[0029] Figure 1 It is an overall flowchart of the open set shell software identification method.
[0030] Figure 2 It is an outlier identification (LOF) example diagram.
[0031] Figure 3 It is an one-class support vector machine (One Class SVM) example diagram.
[0032] Figure 4 This is a schematic diagram of the feature space in the closed set recognition problem.
[0033] Figure 5 A schematic diagram of the threshold setting scheme.
[0034] Figure 6 This is a schematic diagram of a single model scheme.
[0035] Figure 7 This is a schematic diagram of multiple model schemes. Detailed Implementation
[0036] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.
[0037] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for identifying shelled software in open-source systems, including:
[0038] Using disassemblers and their custom plugins, function call graph structure information is extracted from the binary executable file to be identified, and a structured graph representation file is generated.
[0039] Based on the graph representation file, graph statistical features and file segment structure features are extracted to form a concatenated feature vector;
[0040] Based on a multi-model single-class recognition strategy built for open set recognition, the binary executable file corresponding to the current feature vector is classified and recognized.
[0041] It should be noted that binary executable files are often protected by packing techniques, which complicates direct analysis of these files. Existing identification methods are mostly aimed at packed software under closed-set conditions, that is, assuming that all samples belong to a known type. However, with the continuous iteration of malware technology and the emergence of new packing methods, the challenge of identifying packed software of unknown types under open-set conditions urgently needs to be overcome. At the same time, traditional deep learning methods have high computational resource requirements and are difficult to run efficiently in resource-constrained environments, limiting their practicality.
[0042] Therefore, to address the aforementioned problems, this invention, through steps S100-S300, utilizes disassemblers and plugins to extract function call graph information and generate structured graph files, laying the foundation for subsequent analysis; by extracting graph statistics and file segment structure features to form feature vectors, rapid feature generation is achieved, reducing dependence on computing resources; and a multi-model single-class recognition strategy is adopted to effectively address the detection of unknown types in open sets.
[0043] Example 2, refer to Figures 1 to 7 This is one embodiment of the present invention. Based on the above embodiment, a method for identifying shelled software for open sets is provided.
[0044] In this embodiment of the application, step S100 uses a disassembler and its custom plugins to extract function call graph structure information from the binary executable file to be identified, generating a structured graph representation file, including the following steps A1-A2:
[0045] A1: The diagram shows that the file includes node information that records the function node number and name;
[0046] A2: The graph representation file also includes directed edge information that records the correspondence between the node numbers of the caller and the callee.
[0047] It should be noted that packed software is a binary file, and therefore cannot be directly viewed using a text reader. On Windows platforms, it is commonly presented as a PE file, so it must first be converted into a file type that we can understand and easily extract features from. While disassemblers can generate disassembly instructions composed of mnemonics from binary files based on their instruction set, allowing for identification and analysis, disassemblers themselves do not directly support extracting function call graphs. Therefore, a plugin script is needed to accomplish this. The plugin script extracts node and directed edge information from the function call graph in the binary file, thus automating the conversion from binary files to structured network data, providing the foundation and prerequisite for all subsequent analysis.
[0048] In an optional implementation, step S100, extracting the function call graph structure information from the binary executable file to be identified using a disassembler and its custom plugins, can be achieved using IDA Pro disassembler and its custom plugins. Specifically, ensure IDA Pro is installed, load the binary executable file to be identified (e.g., a PE file), and IDA Pro will automatically perform a preliminary analysis of the loaded binary file, including identifying functions and data segments. After the analysis is complete, a custom plugin script written in Python is used to further extract the function call graph structure information. By running the script, the script will extract the function call graph information and save the results.
[0049] In another optional implementation, step S100, which involves extracting the function call graph structure information from the binary executable file to be identified using a disassembler and its custom plugins, can also be achieved using the BinaryNinja disassembler and its custom plugins. Specifically, ensure that the BinaryNinja disassembler is installed, load the binary executable file to be identified (such as a PE file), and BinaryNinja will automatically perform a preliminary analysis of the loaded binary file, including identifying functions and data segments. After the analysis is complete, a custom plugin script written in Python is used to extract the function call graph structure information. By running the script, the script will extract the function call graph information and save the results.
[0050] In this embodiment of the application, step S200, based on the graph representation file, extracts graph statistical features and file segment structural features to form a concatenated feature vector, including the following steps B1-B3:
[0051] It should be noted that since the detection task is a preliminary step in the analysis task, and since malware analysis devices are generally not equipped with complex computing devices such as GPUs, and the analysis is often performed only in a virtual environment to ensure security and prevent long-term persistent and highly destructive malware, such as mining software or ransomware, from interfering with the normal use of the system, the extraction and use of features must be fast. Therefore, this step will not use deep networks or graph neural networks to extract the graph representation language files of packed software, but will use the statistical features of the graph as the output.
[0052] It should be emphasized that this step uses lightweight statistical computation and does not use neural network models.
[0053] B1: Calculate a set of predefined graph statistical features based on the graph representation file;
[0054] Understandably, graph statistical features include the number of vertices, the number of directed edges, the maximum in-degree and minimum out-degree of vertices, the relationship between the number of vertices and directed edges, the absolute value of the difference between the number of vertices and directed edges, the maximum degree, the minimum degree, the number of connected graphs, whether the first connected graph has a cycle, the number of root vertices, the number of leaf vertices, the number of isolated vertices, the number of vertices partitioned by an edge, the number of vertices merged by an edge, the number of vertices passed by an edge, the in-degree of the program entry vertex, and the out-degree of the program entry vertex, etc. (as shown in Table 1 below), which can reflect the characteristics of the software's structure and behavior.
[0055] Table 1. Statistical Characteristics and Overview
[0056]
[0057] It should be noted that since the graph representation file is a text file, the information of nodes and edges is stored line by line, which facilitates quick reading and statistical analysis of features.
[0058] It should be noted that the networkx library can be used to assist in the extraction of statistical features from the graph.
[0059] B2: Extract a set of segment structure features from the header information of the binary executable file to be identified;
[0060] Understandably, segment structure features include total number of segments, maximum segment size (unit: bytes), minimum segment size, maximum number of segments, minimum number of segments, and software size, which can reflect the composition and storage characteristics of a file.
[0061] It should be noted that segment structure features can be extracted using the pefile library in Python.
[0062] B3: Combine the graph statistical features and segment structure features to form a feature vector for identification.
[0063] In an optional implementation, step B3 concatenates the graph statistical features and segment structure features to form a feature vector for recognition. This can be achieved by using NumPy for feature concatenation, which involves converting the extracted graph statistical features and segment structure features into NumPy arrays respectively, and then using the numpy.concatenate() function to concatenate the two arrays into a one-dimensional array to form a complete feature vector. The concatenated feature vector can then be saved as a file (e.g., .npy format) or directly used as input to the subsequent classification and recognition model.
[0064] In another optional implementation, the concatenation of graph statistical features and segment structure features in step B3 to form a feature vector for identification can also be performed using Pandas. That is, the extracted graph statistical features and segment structure features are created as Pandas DataFrame objects respectively, and the pandas.concat() function is used to concatenate the two DataFrames column by column to form a DataFrame containing all features. The concatenated DataFrame is then converted into a NumPy array to form a complete feature vector for subsequent processing.
[0065] In this embodiment of the application, step S300 classifies and identifies the binary executable file corresponding to the current feature vector based on a multi-model single-class identification strategy constructed for open set identification, including the following steps C1-C2:
[0066] It should be noted that existing multi-class classification models assume that there are only samples of a fixed type in the feature space and train a confidence model to classify the data into specific categories. However, although such models can use confidence thresholds to identify open set shelled software, their performance is poor, and model reconstruction requires retraining the entire model.
[0067] C1: Input the feature vector into the target recognition model set for judgment, wherein the target recognition model set contains multiple single-class recognition models that correspond one-to-one with multiple known shell types;
[0068] Specifically, the single-class recognition model in step C1 includes the following steps C11-C12:
[0069] C11: The single-class recognition model is trained using the feature vectors of samples corresponding to known shell types, and is used to define the distribution range of that type in the feature space;
[0070] C12: The single-class recognition model is trained using the single-class support vector machine algorithm.
[0071] It should be noted that outlier identification, as an anomaly detection method, shares a similar objective with the single-class support vector machine algorithm in open set identification tasks: identifying outliers or unknown types in the feature space. On the one hand, such as... Figure 2 The image shows a density example of LOF (Local Optical Array), where red indicates the highest density and lighter colors indicate lower density. Outlier identification doesn't require a separate model; instead, it uses all training set samples as comparison objects. The process for determining if a test point is an outlier is as follows: 1. Calculate the nearest sample points to the test point as its neighbors. The number of neighbors can be customized. 2. Calculate the density of the neighbors, which is the average distance from each sample point to its neighbors. Similarly, calculate the density of the test point. 3. If the density of the test point is higher than the average density of its neighbors, it is considered a sample of the current class; otherwise, it is considered an outlier. On the other hand, if... Figure 3 As shown (an example of a single-class support vector machine), the single-class support vector machine specifically consists of the following processes: 1. Map the feature vectors of the training samples to a high-dimensional space, and calculate the initial hypersphere, which is the center (arithmetic mean of each dimension) and radius (farthest distance from the center in each sample) of all sample points in the high-dimensional space. 2. Calculate the perpendicular distance from all points to the hypersphere, and take the samples with shorter distances as the support vector set. 3. Using the support vectors, shrink the hypersphere inward until the hyperspheres near all support vectors have been advanced. 4. Optimize the position of the hypersphere. That is, compared with outlier identification (LOF), the single-class support vector machine distinguishes known and unknown types in the feature space through the hypersphere, which has higher recognition efficiency and accuracy.
[0072] Furthermore, to verify the advantages and applicability of the One-Class Support Vector Machine (SVM) algorithm, 21 sets (AU) of experiments were conducted to analyze the recall rates of the outlier detection (LOF) and the One-Class SVM algorithm for the visible types in the training dataset, as well as the recall rates for the open set (unseen types). The mean values of SR (ASR) and UR (AUR) were also calculated, and the mean values of ASR and AUR were used as the final metrics. The results are shown in Tables 2 and 3 below.
[0073] Table 2 Results using the LOF algorithm
[0074]
[0075]
[0076] Table 3 shows the results using the One-Class SVM algorithm.
[0077]
[0078] As shown above, the LOF algorithm performs very stably on known types (SR is mostly above 98%), but generally performs poorly on unknown types (UR is usually only around 10%-30%). In contrast, the One Class SVM algorithm, especially the UR value, mostly exceeds 70% or even approaches 80%. At the same time, the AR and AUR of One Class SVM are significantly higher than those of LOF. Therefore, it can be proven that it is more effective and accurate in identifying shelled software under open set conditions.
[0079] It should be further clarified that previous problems in detecting packed software all fall under the category of identifying closed sets of packed software. This problem presupposes that the feature vector space exists as a partition according to certain rules, and packed software within the same partitioned region belongs to a specific label, such as... Figure 4 The diagram illustrates a region partitioning method, where the numbers represent type identifiers. Each region shares a boundary with its neighbors without any gaps. This is because in typical supervised learning problems, the label with the highest confidence level is used as the predicted label. This results in each sample in the feature vector space partitioned by the model uniquely existing in one region or on the boundaries of multiple regions (e.g., two labels with equal confidence levels). Below, we will introduce three solutions for addressing the problem of identifying open sets in shelled software:
[0080] Option 1 - Threshold Setting: To detect unknown types of packed software, in Figure 4Building upon this foundation, 2-SPIFF proposes for the first time a threshold-based unknown label recognition scheme, namely... Figure 4 A class gap was added. When the highest confidence level predicted by the model is below a threshold, it falls into the class gap and is predicted as an unknown type; otherwise, it is judged as an unknown type (unknown type software). For example... Figure 5 As shown, light-colored areas represent unknown types of packed software, while dark-colored areas correspond to various types of packed software. The 2-SPIFF scheme provides a new approach to solving the problem of identifying open sets of packed software.
[0081] Option 2 - Single Model: The advantage of the threshold setting scheme is that it can maintain high accuracy in recognizing known types while providing the ability to identify unknown types of packed software. It is not a scheme designed independently for unknown types of packed software, but rather an extension of the scheme for identifying closed sets of packed software. The single model scheme treats all packed software in the training set as one type of packed software, using a single-class recognition model to identify unknown packed software, such as... Figure 6 As shown, the single-model approach no longer relies on the multi-classification space assumption; it only classifies the feature space into known-type space and unknown-type space based on known-type shelled software.
[0082] Solution 3 - Multi-class Model (An ensemble of multiple single-class models): Because different types differ in distribution and density, a single-class model may misclassify some marginal samples of known types as unknown types. The multi-class model extends the ant colony model by using a single-class model for each independent classification. This approach retains more known types within the model's coverage area while, compared to a single-class model, eliminating unknown types between categories. For example... Figure 7 As shown, the multi-class model scheme uses the method of increasing the number of models to more accurately distinguish between known and unknown types. Different colors indicate that the training data sources of the models are different. Therefore, the method framework of this patent adopts a multi-class recognition model scheme.
[0083] C2: Determine the type of the binary executable file to be identified corresponding to the current feature vector based on the judgment result.
[0084] Specifically, in step C2, the type of the binary executable file to be identified corresponding to the current feature vector is determined based on the judgment result, including the following steps C21-C22:
[0085] C21: If the current feature vector is determined by all single-class recognition models to not belong to its corresponding known packer type, then the type of the binary executable file to be identified corresponding to the current feature vector is determined to be open set unknown type packed software.
[0086] C22: If the current feature vector is determined by any single-class recognition model to belong to its corresponding known packer type, then the type of the binary executable file to be identified corresponding to the current feature vector is determined to be a known type.
[0087] It should be noted that, compared to using existing multi-class classification models to identify open-set shelled software, the multi-model single-class identification strategy used in this step is more in line with the principles of open-set shelled software identification. That is, a separate single-class identification model is trained for each class, using a single-class support vector machine algorithm and the feature set of training samples for each class to construct the identification model for each class. Therefore, for the multi-model training process, the input is the training feature set with class labels, and the output is the set of identification models, with the length of the output equal to the number of types. For the open-set shelled software identification process, the input is the feature vector of the software to be identified, and the output is whether the software belongs to the type in the current shelled software training set or to the open-set shelled software category.
[0088] In summary, this invention effectively improves the accuracy and adaptability of packed software identification by combining function call graph structure information extracted using disassemblers and feature extraction using lightweight statistical calculations, and employing a multi-model single-class identification strategy. In particular, it significantly enhances the detection capability of unknown types of packed software under open set conditions, while also exhibiting high computational efficiency and low resource consumption, making it suitable for resource-constrained environments. Furthermore, the single-class identification model trained using a single-class support vector machine algorithm can accurately define the distribution range of known packed software types in the feature space, thereby achieving efficient identification of unknown types.
[0089] Example 3 illustrates a schematic scheme for identifying open-source bundled software. It should be noted that the technical solution of this system for identifying open-source bundled software is based on the same concept as the technical solution of the aforementioned method for identifying open-source bundled software. Details not described in detail in the technical solution of the system for identifying open-source bundled software in this embodiment can be found in the description of the technical solution of the aforementioned method for identifying open-source bundled software.
[0090] This embodiment also provides a system for identifying shelled software in open systems, including:
[0091] The generation module is used to extract function call graph structure information from the binary executable file to be identified using a disassembler and its custom plugins, and generate a structured graph representation file.
[0092] The splicing module is used to extract graph statistical features and file segment structure features based on the graph representation file, and form a spliced feature vector.
[0093] The identification module is used to classify and identify the binary executable file corresponding to the current feature vector based on a multi-model single-class identification strategy built for open set identification.
[0094] This embodiment also provides an electronic device applicable to the identification of open-set packed software, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the identification method for open-set packed software as proposed in the above embodiment.
[0095] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, it implements the identification method for open-set shelled software as proposed in the above embodiments.
[0096] The storage medium proposed in this embodiment and the method for identifying shelled software with a highly open set proposed in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.
[0097] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.
[0098] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. An identification method for open set shell software, characterized in that: The method comprises the steps of: extracting function call graph structure information from a binary executable file to be identified through a disassembler tool and a custom plug-in thereof, and generating a structured graph representation file; based on the graph representation file, extracting graph statistical features and file section structure features to form a spliced feature vector; based on a multi-model one-class identification strategy constructed for open set identification, performing classification identification on the binary executable file corresponding to the current feature vector.
2. The method of claim 1, wherein the open set of shells is identified by: The method comprises the steps of: based on the graph representation file, calculating a set of predefined graph statistical features; extracting a set of section structure features from the header information of the binary executable file to be identified; splicing the graph statistical features and the section structure features to form a feature vector for identification.
3. The method for identifying shelled software for open-source systems as described in claim 2, characterized in that: The method comprises the steps of: inputting the feature vector into a target identification model set for determination, wherein the target identification model set comprises a plurality of one-class identification models corresponding to a plurality of known packing types one by one; determining the type of the binary executable file to be identified corresponding to the current feature vector according to the determination result.
4. The method of claim 3, wherein the open set of shells is identified by: The method comprises the steps of: if the current feature vector is determined by all the one-class identification models as not belonging to the known packing type corresponding thereto, determining that the type of the binary executable file to be identified corresponding to the current feature vector is an open set unknown type of packed software; if the current feature vector is determined by any one of the one-class identification models as belonging to the known packing type corresponding thereto, determining that the type of the binary executable file to be identified corresponding to the current feature vector is a known type.
5. The method for identifying shelled software for open-source systems as described in claim 4, characterized in that: The graph representation file comprises node information recording function node numbers and names, and directed edge information recording the correspondence between caller and callee node numbers.
6. The method for identifying shelled software for open-source systems as described in claim 5, characterized in that: The one-class identification model is trained using a sample feature vector corresponding to a known packing type, and is used to define the distribution range of the type in the feature space; The one-class identification model is trained using a one-class support vector machine algorithm.
7. An identification method for open set shell software according to any one of claims 2-6, characterized in that: The extraction of graph statistical features and file section structure features based on the graph representation file adopts lightweight statistical calculation and does not use a neural network model.
8. A recognition system for open set shell software, applying the method according to any one of claims 1 to 7, characterized in that, The method comprises the steps of: a generation module for extracting function call graph structure information from a binary executable file to be identified through a disassembler tool and a custom plug-in thereof, and generating a structured graph representation file; a splicing module for extracting graph statistical features and file section structure features based on the graph representation file, and forming a spliced feature vector; an identification module for performing classification identification on the binary executable file corresponding to the current feature vector based on a multi-model one-class identification strategy constructed for open set identification. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.
10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 7.