A method and apparatus for constructing a binary feature database

Abstract

This application discloses a method for constructing a binary feature database. The method involves obtaining a binary file, extracting binary features from the binary file, and constructing a full feature database using these features. Then, the full feature database is filtered to remove reusable features. By removing reusable features from the full feature database, this application avoids the reuse of binary features between components during binary detection, thereby preventing false alarms in binary detection matching.

Description

Technical Field

[0001] This application belongs to the technical research field of database construction, and specifically relates to a binary feature database construction method and a binary feature database construction device. Background Technology

[0002] With the rapid development of the Internet of Things (IoT) and the Industrial Internet, research on cyberspace security has received increasing attention from both industry and academia. Due to the unavailability of a large amount of non-open-source source code, binary code similarity detection has become a key technology for vulnerability discovery and software security analysis.

[0003] In binary-to-binary matching, even though the actual instruction sequences of two binary programs differ, semantic similarity can still be used for matching and detection. Binary-to-binary homology detection primarily compares the similarity of binary files, which are the main representations of applications. Disassembling or decompiling the binary program yields its intermediate representation code, and the textual features of this intermediate code are directly used to measure the similarity of the binary programs. However, most works perform semantic analysis on the intermediate code to extract useful features for similarity detection. Currently, many studies propose using multiple features from binary files for detection, such as constant strings, control flow graphs, function call graphs, structured signatures, and basic block attributes. Detection techniques mainly include inverted index detection, longest common subsequence calculation, and path matching algorithms.

[0004] Because the features in binary programs are not as rich as those in source code, and because a large number of features appear in a large number of components, simple reuse, partial reuse, and derived reuse of binary features occur between components, which in turn causes false alarms in the matching results, making binary detection more difficult.

[0005] To address the aforementioned issues, this application proposes a binary feature database construction method and a binary feature database construction apparatus. Summary of the Invention

[0006] To address the shortcomings of the prior art, this application provides a binary feature database construction method that filters the constructed full feature database and removes reusable features, thereby avoiding the reuse of binary features among components during binary detection.

[0007] The technical effect to be achieved in this application is accomplished through the following solution:

[0008] Firstly, this application provides a method for constructing a binary feature database, applied to binary software component analysis and detection, the method comprising:

[0009] Obtain the binary file;

[0010] Obtain binary features from the binary file, and construct a full feature database using the binary features;

[0011] The full feature database is filtered to remove reusable features;

[0012] The filtering process involves traversing the binary features in the full feature database, comparing every two binary features, determining the similarity between each pair of binary features, and determining the reusable features based on the similarity.

[0013] Optionally, obtaining the binary file includes:

[0014] Acquire open-source component data and use the open-source component data to build an open-source component information database;

[0015] The binary file is obtained from the open-source component information repository.

[0016] Optionally, obtaining the binary file from the open-source component information repository includes:

[0017] Crawl binary packages from the aforementioned open-source component information repository;

[0018] The binary package is decompressed, and the binary file is identified and obtained from the decompressed file package.

[0019] Optionally, the open-source component data includes basic information about the open-source components and their dependencies.

[0020] Optionally, the step of acquiring open-source component data and using the open-source component data to construct an open-source component information database includes:

[0021] The basic information of the open-source components and their dependencies are obtained through web scraping.

[0022] An open-source component information repository is constructed using the basic information of the open-source components and their dependencies.

[0023] Optionally, filtering the full feature database to remove reusable features includes:

[0024] The full feature database is filtered by the open-source component dependency relationships in the open-source component information repository to remove reusable features.

[0025] Optionally, obtaining binary features from the binary file and constructing a full feature database using the binary features includes:

[0026] The binary file is decompiled, and file characteristics are obtained from the decompiled file;

[0027] Extract the binary features that can be used for binary detection from the file features;

[0028] The full feature database is constructed using the binary features.

[0029] Secondly, this application provides a binary feature database construction apparatus, the apparatus comprising:

[0030] The acquisition module is used to acquire binary files;

[0031] A construction module is used to obtain binary features from the binary file and construct a full feature database using the binary features;

[0032] The filtering module is used to filter the full feature database and remove reusable features.

[0033] Optionally, the acquisition module is used to:

[0034] Acquire open-source component data and use the open-source component data to build an open-source component information database;

[0035] The binary file is obtained from the open-source component information repository.

[0036] Optionally, the acquisition module is used to:

[0037] Crawl binary packages from the aforementioned open-source component information repository;

[0038] The binary package is decompressed, and the binary file is identified and obtained from the decompressed file package.

[0039] Optionally, the acquisition module is used to:

[0040] The basic information of the open-source components and their dependencies are obtained through web scraping.

[0041] An open-source component information repository is constructed using the basic information of the open-source components and their dependencies.

[0042] Optionally, the filtering module is used for:

[0043] The full feature database is filtered by the open-source component dependency relationships in the open-source component information repository to remove reusable features.

[0044] Optionally, the building module is used for:

[0045] The binary file is decompiled, and file characteristics are obtained from the decompiled file;

[0046] Extract the binary features that can be used for binary detection from the file features;

[0047] The full feature database is constructed using the binary features.

[0048] Thirdly, this application provides a readable medium including execution instructions, which, when executed by a processor of an electronic device, cause the electronic device to perform any of the methods described in the first aspect.

[0049] Fourthly, this application provides an electronic device, the electronic device including a processor and a memory storing execution instructions, wherein when the processor executes the execution instructions stored in the memory, the processor performs the method as described in any of the first aspects.

[0050] This application has the following advantages:

[0051] This application discloses a method for constructing a binary feature database. The method involves obtaining a binary file, extracting binary features from the binary file, and constructing a full feature database using these binary features. Then, the full feature database is filtered to remove reusable features. By removing reusable features from the full feature database, this application avoids the reuse of binary features between components during binary detection, thereby preventing false alarms in binary detection matching. Attached Figure Description

[0052] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 The flowchart of the binary feature database construction method described in one embodiment of this application Figure 1 ;

[0054] Figure 2 The flowchart of the binary feature database construction method described in one embodiment of this application Figure 2 ;

[0055] Figure 3 This is a schematic diagram of the binary feature database construction device described in one embodiment of this application;

[0056] Figure 4 This is a schematic diagram of the structure of the electronic device described in one embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In related technologies, the features in binary programs are less abundant than those in source code, and because a large number of features appear in numerous components, simple reuse, partial reuse, and derived reuse of binary features occur between components, leading to false positives in matching results and making binary detection more difficult. To address these issues, this application proposes a method for constructing a binary feature database. This method involves obtaining a binary file; then, extracting binary features from the binary file and constructing a full feature database using these features; finally, filtering the full feature database to remove reused features. By removing reused features from the full feature database, this application avoids the reuse of binary features between components during binary detection, thereby preventing false positives in binary detection matching.

[0059] The non-limiting embodiments of this application are described in detail below with reference to the accompanying drawings.

[0060] As attached Figure 1 This paper illustrates a binary feature database construction method according to an embodiment of the present application. The method is applied to binary software component analysis and detection, and includes:

[0061] S101: Get the binary file.

[0062] Constructing the binary feature database requires a sufficient number of binary files, and these binary files should be diverse to facilitate the subsequent acquisition of various binary features and to improve the feature data of the binary feature database.

[0063] In one embodiment, obtaining the binary file can involve acquiring open-source component data and using that data to construct an open-source component information repository. Then, the binary file is obtained from the open-source component information repository. Open source code allows access to a wide variety of source code systems, thus providing access to diverse open-source component data. Constructing the open-source component information repository using this data facilitates the retrieval of open-source component data, thereby obtaining the binary file from it. Obtaining a variety of binary files from the open-source component information repository satisfies the diverse needs of binary files, enabling the construction of a comprehensive binary feature database.

[0064] In one embodiment, obtaining the binary file from the open-source component information repository can be achieved by crawling a binary package from the repository; then, the binary package is decompressed, and the binary file is identified and obtained from the decompressed file package. The open-source component information repository contains various types of open-source component data, and the binary file is obtained from this data. Because the open-source component information repository stores a large amount of open-source component data, crawling is necessary to quickly obtain the binary package. The binary files in the open-source component data are all in compressed packages; that is, the compressed package is the binary package. The compressed package may contain other files, therefore, when obtaining the binary file, the binary package needs to be decompressed before the binary file can be identified.

[0065] In one embodiment, the open-source component data includes basic information about the open-source components and their dependencies. The basic information is code such as binary packages or binary files. The open-source component dependencies refer to the correspondence between application software built using code from binary files and the function libraries it depends on. Application software needs to call numerous functions from system function libraries to achieve its functionality.

[0066] In one embodiment, the acquisition of open-source component data and the construction of an open-source component information database using this data can be achieved by web scraping to obtain basic information about the open-source components and their dependencies. Then, the open-source component information database is constructed using this basic information and dependencies. With the prevalence of the internet, many open-source components are available online. To quickly acquire these components, web scraping can be used to obtain their basic information and dependencies. The open-source component information database includes the basic information and dependencies. Application software built by users using the binary code from the basic information of open-source components requires calls to functions in the system database to achieve its functionality; therefore, the application software depends on these functions. The open-source component dependencies can prompt users to install the corresponding function library packages when installing the application software, enabling it to run. The binary code needs to demonstrate the functions it needs to call to achieve its functionality. Binary code implementing the same functionality calls the same functions, but the binary code itself is not necessarily identical; it can be similar code.

[0067] S102: Obtain binary features from the binary file and construct a full feature database using the binary features.

[0068] All binary features obtained from binary files are used to construct a full feature database, which facilitates the use of these binary features to measure the similarity of binary programs when detecting binary software components.

[0069] In one embodiment, the step of obtaining binary features from the binary file and constructing a full feature database using these binary features can involve decompiling the binary file to obtain file features from the decompiled file; then, extracting binary features usable for binary program detection from these file features; and finally, constructing the full feature database using these binary features. The file features include constant strings, control flow graphs, function call graphs, structured signatures, basic block attributes, partial class names, and function names. Extracting binary features usable for binary program detection from these file features facilitates the detection of binary program similarity. Although much information present in the source code is not present in the binary file, some configuration information still exists, such as constant strings, partial class names, and function names. Using this existing configuration information as binary features to detect binary program similarity can improve the accuracy of the detection.

[0070] S103: Filter the full feature database and remove reusable features;

[0071] The filtering process involves traversing the binary features in the full feature database, comparing every two binary features to determine their similarity, and then determining the reusable feature based on the similarity. Two binary features with a similarity exceeding a preset value are identified as one of the reusable features. The similarity calculation can be performed using relevant algorithms. There are many algorithms available for similarity calculation, and these can be adjusted according to actual circumstances. There are numerous existing technologies for similarity calculation, and these are not the focus of this application; therefore, they will not be elaborated upon further.

[0072] Including all binary features in the full feature database might result in many similar binary programs having their features stored in the database, even though these similar programs have different binary features. However, these different binary features can still detect similar binary programs. Therefore, detecting similar binary programs only requires one set of binary features, eliminating the need to use more binary features to search for similar programs. Repeatedly using the binary characteristics of similar programs for detection increases detection time and the number of steps, thus reducing efficiency. Removing duplicate binary features, leaving only one set to detect similar binary programs, improves the efficiency of binary detection.

[0073] In one embodiment, filtering the full feature database to remove reusable features can be achieved by filtering the full feature database through the open-source component dependency relationships in the open-source component information repository. Open-source component dependencies reveal the functions called by the binary program, and from these functions, the functionality of the application software built from the binary file can be determined. If different binary files have the same functionality, then these binary files are considered similar. These similar binary files can be detected using a set of binary features. Other binary features used to detect these similar binary files can be removed to avoid duplicate detection, thereby preventing false positives in binary detection matching and improving detection efficiency.

[0074] Specifically, starting with a specific open-source component, its corresponding component dependency network is located using dependency information from the open-source component database. The dependency level of each component in this network is calculated and numerically identified. Starting with the component with the highest dependency level, the open-source components in the network are traversed sequentially from highest to lowest dependency level. During this traversal, the similarity between the binary features of the dependent component and the dependent component is calculated using relevant algorithms. Two binary features with high similarity are marked, and the binary features in the dependent component are marked as reused features and removed from the dependent component's features. This process is repeated until all open-source components have been processed.

[0075] Specifically, as shown in the attached document Figure 2 As shown, in one embodiment of this application, the binary feature database construction method may include:

[0076] Crawl binary packages from the aforementioned open-source component information repository;

[0077] Decompress the binary package to obtain the decompressed file package;

[0078] Identify and retrieve binary files from the decompressed file package;

[0079] The binary file is decompiled, and file characteristics are obtained from the decompiled file;

[0080] Extract the binary features that can be used for binary detection from the file features;

[0081] Construct a full feature database using the aforementioned binary features;

[0082] The mapping relationship between binary features, binary files, and the binary features is stored in the in-use feature library of the full feature database; wherein, all file features, including the removed binary features, are stored in the original feature library of the full feature database, and can be directly called when repeated binary features are needed later.

[0083] As attached Figure 3 The diagram shows a schematic representation of a binary feature database construction apparatus according to an embodiment of this application. The apparatus includes:

[0084] The acquisition module is used to acquire binary files;

[0085] A construction module is used to obtain binary features from the binary file and construct a full feature database using the binary features;

[0086] The filtering module is used to filter the full feature database and remove reusable features.

[0087] Optionally, the acquisition module is used to:

[0088] Acquire open-source component data and use the open-source component data to build an open-source component information database;

[0089] The binary file is obtained from the open-source component information repository.

[0090] Optionally, the acquisition module is used to:

[0091] Crawl binary packages from the aforementioned open-source component information repository;

[0092] The binary package is decompressed, and the binary file is identified and obtained from the decompressed file package.

[0093] Optionally, the acquisition module is used to:

[0094] The basic information of the open-source components and their dependencies are obtained through web scraping.

[0095] An open-source component information repository is constructed using the basic information of the open-source components and their dependencies.

[0096] Optionally, the filtering module is used for:

[0097] The full feature database is filtered by the open-source component dependency relationships in the open-source component information repository to remove reusable features.

[0098] Optionally, the building module is used for:

[0099] The binary file is decompiled, and file characteristics are obtained from the decompiled file;

[0100] Extract the binary features that can be used for binary detection from the file features;

[0101] The full feature database is constructed using the binary features.

[0102] Since the binary feature database construction device adopts all the technical solutions of all implementation methods of the binary feature database construction method, it has at least all the beneficial effects brought about by the technical solutions of the above and below implementation methods, which will not be repeated here.

[0103] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include RAM, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk storage device. Of course, the electronic device may also include other hardware required for other services.

[0104] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0105] Memory is used to store instructions for execution. Specifically, instructions for execution are computer programs that can be executed. Memory can include main memory and non-volatile memory, and it provides the processor with execution instructions and data.

[0106] In one possible implementation, the processor reads the corresponding execution instructions from non-volatile memory into memory and then executes them. Alternatively, it may obtain the corresponding execution instructions from other devices to form a binary feature database construction method at the logical level. The processor executes the execution instructions stored in memory to implement the binary feature database construction method provided in any embodiment of this application through the executed instructions.

[0107] The above is as stated in this application. Figure 1The binary feature database construction method provided in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0108] The steps of the method disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0109] This application also proposes a readable medium that stores execution instructions. When the stored execution instructions are executed by the processor of an electronic device, the electronic device can execute the binary feature database construction method provided in any embodiment of this application, and is specifically used to execute the above-mentioned binary feature database construction method.

[0110] The electronic devices described in the foregoing embodiments may be computers.

[0111] Those skilled in the art will understand that the embodiments of this application can be provided as methods or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or a combination of software and hardware.

[0112] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0113] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for constructing a binary feature database, applied to binary software component analysis and detection, characterized in that, The method includes: Obtain open-source component data and construct an open-source component information database using the open-source component data; the open-source component data includes basic information about open-source components and their dependencies; Obtain the binary file from the aforementioned open-source component information repository; Obtain binary features from the binary file, and construct a full feature database using the binary features; The full feature database is filtered by the open-source component dependency relationships in the open-source component information repository to eliminate reusable features; The filtering process involves traversing the binary features in the full feature database, comparing every two binary features, determining the similarity between each pair of binary features, and determining the reusable features based on the similarity.

2. The binary feature database construction method as described in claim 1, characterized in that, Obtaining the binary file from the open-source component information repository includes: Crawl binary packages from the aforementioned open-source component information repository; The binary package is decompressed, and the binary file is identified and obtained from the decompressed file package.

3. The binary feature database construction method as described in claim 1, characterized in that, The step of acquiring open-source component data and using the open-source component data to construct an open-source component information database includes: The basic information of the open-source components and their dependencies are obtained through web scraping. The open-source component information repository is constructed using the basic information of the open-source components and their dependencies.

4. The binary feature database construction method as described in claim 1, characterized in that, The step of obtaining binary features from the binary file and constructing a full feature database using the binary features includes: The binary file is decompiled, and file characteristics are obtained from the decompiled file; Extract the binary features that can be used for binary detection from the file features; The full feature database is constructed using the binary features.

5. A binary feature database construction device, characterized in that, The device includes: An acquisition module is used to acquire open-source component data and construct an open-source component information repository using the open-source component data; the open-source component data includes basic information about the open-source components and their dependencies; and binary files are acquired from the open-source component information repository. A construction module is used to obtain binary features from the binary file and construct a full feature database using the binary features; The filtering module is used to filter the full feature database based on the open-source component dependencies in the open-source component information library, and remove reusable features. The filtering process involves traversing the binary features in the full feature database, comparing every two binary features, determining the similarity between each pair of binary features, and determining the reusable features based on the similarity.

6. A readable medium, characterized in that, The readable medium includes execution instructions that, when executed by the processor of the electronic device, cause the electronic device to perform the method as described in any one of claims 1-4.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory storing execution instructions. When the processor executes the execution instructions stored in the memory, the processor performs the method as described in any one of claims 1-4.

Images