Binary function identification method and system combining embedded similarity and function call graph

Abstract

The application discloses a binary function identification method and system combining embedded similarity and function call graph, and the method comprises the following steps: compiling source code to generate a binary file; analyzing the binary file to export internal features of each function and a call relationship between functions; performing internal feature vectorization processing on the functions to obtain a feature vector of each function; regarding all functions with the same symbol as an equivalence class, and combining the call relationship between the functions to construct an extended function call graph representing a call relationship between symbol names; constructing a binary function database based on the feature vector of the function and the extended function call graph; and using the binary function database to complete function query of an unknown binary program to obtain a binary function identification result in the unknown binary program. The application can automatically analyze a binary file lacking debugging information and function symbols, and quickly identify functions in the binary file.

Description

Technical Field

[0001] This invention relates to a binary function recognition method and system that combines embedded similarity with function call graphs, belonging to the field of computer application technology. Background Technology

[0002] Analyzing binary programs, such as malware and embedded firmware, without debugging information and function symbols is often a time-consuming and challenging process, requiring significant manpower for manual reverse engineering. Existing function-level solutions, such as IDA FLIRT and fcatalog, typically rely on raw bytes as features for function identification. While these methods are fast and simple, they are highly sensitive to even minor changes. Different compiler versions and compilation options can easily render these byte-feature-dependent methods ineffective. Another approach is third-party component identification techniques, which summarize syntactic rule information such as strings and constants to identify known components of unknown programs at the library or module level. This method is coarse-grained and cannot be applied to template functions.

[0003] In recent years, many schemes have emerged that utilize machine learning techniques to calculate function similarity. These methods embed binary functions into a vector space using neural network models and use distance metrics to evaluate similarity. Through extensive training, neural network models can be made capable of handling binary function similarity problems under different compilation conditions. However, these methods typically only perform well on closed-set recognition tasks; in open-set recognition tasks, they are not good at rejecting false positives, resulting in a large number of false positives and poor usability.

[0004] To address this problem, we propose a binary function recognition system that combines embedding similarity with function call graphs. This system embeds binary functions as vectors using a trained neural network model, stores known binary functions and their call relationships in a function database, and performs retrieval based on vector distance and call relationships, thereby achieving rapid identification of functions in unknown binary programs. This system exhibits high efficiency and accuracy, as well as good scalability. Summary of the Invention

[0005] The purpose of this invention is to propose a binary function identification method and system that combines embedded similarity and function call graphs. This method offers high efficiency and accuracy, as well as good scalability. It is suitable for the rapid identification of functions in unknown binary programs.

[0006] To achieve rapid identification of functions applicable to unknown binary programs, this invention employs the following technical solution:

[0007] A binary function recognition method combining embedding similarity and function call graphs, the method comprising:

[0008] Collect projects with source code, compile the source code, and generate binary files;

[0009] The binary file is analyzed to derive the internal characteristics of each function and the call relationships between functions; for binary files with symbol information, the symbol names of the functions are derived, and the symbol names of the functions are correlated with the internal characteristics and function call graphs of the functions.

[0010] The internal features of the function are vectorized to obtain the feature vector of each function;

[0011] Treat all functions with the same symbol as equivalence classes, and combine the calling relationships between these functions to construct an extended function call graph that represents the calling relationships between symbol names;

[0012] A binary function database is constructed based on the feature vectors of the functions and the extended function call graph.

[0013] The binary function database is used to query the functions of an unknown binary program, and the binary function identification results in the unknown binary program are obtained.

[0014] Furthermore, the internal features of the function include: the sequence of assembly instructions within the basic block and the function's control flow graph.

[0015] Furthermore, the binary file is analyzed to derive the internal characteristics of the functions, including:

[0016] The regularized binary file is analyzed using a binary analysis tool to derive the unregularized assembly instruction sequence and function control flow graph within the basic blocks; wherein, the binary analysis tool includes: Ghidra;

[0017] Regularize the unregularized assembly instruction sequence to obtain the assembly instruction sequence within the basic block.

[0018] Further, the step of regularizing the unregularized assembly instruction sequence to obtain the assembly instruction sequence within the basic block includes:

[0019] If the value of the immediate value is not between 0x100 and -0x100, then "IMM" is used instead of the immediate value;

[0020] When operands involve taking addresses, use <ptr>+base+imm、 <ptr>+base+index and <ptr>Encoding is performed using the format +base+index+imm; where... <ptr>It is a special token used to identify memory access. Base is the base address register, index represents the index register and multiplier, and imm represents the immediate value of the offset.

[0021] Furthermore, the internal feature vectorization of the functions is performed to obtain the feature vector of each function, including:

[0022] The assembly instruction sequence within each basic block of the function is processed using a natural language processing model to obtain a vector embedding representation of the basic block.

[0023] Using a graph neural network model, basic blocks are used as nodes and control flow graphs are used as edges. Several rounds of information propagation are performed to update the vector representation of the nodes.

[0024] The updated node vector representations are synthesized using a GNN network to obtain the feature vectors of the internal features of the function.

[0025] Furthermore, the binary function database is used to query the functions of the unknown binary program to obtain the binary function identification results of the unknown binary program, including:

[0026] Obtain the first similarity threshold a, the second similarity threshold b, and the third similarity threshold d; where a <b;

[0027] For an unknown binary program, calculate the feature vector of each function to be detected and the function call graph between the functions to be detected;

[0028] The similarity s of the function pairs is obtained by calculating the cosine distance between the feature vector of the function to be detected and the feature vector of each function in the binary function database. 0 The matching status of the function pair is determined by combining the first similarity threshold a and the second similarity threshold b; wherein the matching status includes: impossible matching function pair, potential matching function pair, or high probability matching function pair.

[0029] When a function pair is determined to be a potential matching function pair, its corresponding neighbors are obtained based on both the function call graph and the extended function call graph. At least one round of neighbor verification is then performed based on these neighbors to obtain the similarity s of the function pair. t Where t is the number of adjacent verification rounds;

[0030] If a function pair is determined to be a high-probability matching function pair, or if the similarity s of a function pair is... t If the similarity is greater than the third similarity threshold d, then the corresponding function in the binary function database of the function pair is determined to be the binary function recognition result of the function to be detected.

[0031] Further, determining the matching situation of the function pair by combining the first similarity threshold a and the second similarity threshold b comprises:

[0032] when s 0 <a, determining the function pair as an impossible matching function pair;

[0033] when a<s 0 <b, determining the function pair as a potential matching function pair;

[0034] when b<s 0 , determining the function pair as a high-probability matching function pair.

[0035] Further, performing at least one round of adjacency verification according to neighbors to obtain the similarity s of potential matching function pairs t , comprising:

[0036] calculating a similarity reward value for the t-th round of adjacency verification according to the number of high-probability matching function pairs in the neighbors;

[0037] adding the similarity reward value of the t-th round of adjacency verification to the similarity s t-1 to obtain an updated similarity s t ; wherein the similarity s in the first round of adjacency verification t-1 is the initial similarity s 0 ;

[0038] if at least one new similarity s t greater than the second similarity threshold b appears in the t-th round of adjacency verification, determining the corresponding function pair as a high-probability matching function pair, and after setting t = t+1, returning to the step of calculating the similarity reward value for the t-th round of adjacency verification according to the number of high-probability matching function pairs in the neighbors;

[0039] if no similarity s t greater than the second similarity threshold b appears in the t-th round of adjacency verification, outputting the similarity s of each potential matching function pair t .

[0040] Further, the similarity reward value p of the t-th round of adjacency verification is p=ln(1+cnt)*c; wherein cnt represents the number of high-probability matching function pairs in the neighbors, and c is an adjustable parameter.

[0041] A binary function recognition system combining embedding similarity and function call graph, the system comprises:

[0042] The training module collects projects with source code, compiles the source code to generate binary files, analyzes the binary files to derive the internal features of each function and the call relationships between functions, specifically, for binary files with symbol information, derives the symbol names of functions, and correlates these symbol names with the internal features and function call graphs; the internal features of each function are vectorized to obtain a feature vector for each function; all functions with the same symbol are treated as equivalence classes, and combined with the call relationships between functions, an extended function call graph representing the call relationships between symbol names is constructed; based on the function feature vectors and the extended function call graph, a binary function database is constructed.

[0043] The testing module is used to query functions of an unknown binary program using the binary function database, and obtain the binary function identification results in the unknown binary program.

[0044] Compared to existing technologies, this invention can automatically analyze binary files lacking debugging information and function symbols, and quickly identify functions within them, determining whether a function is similar to a function in a function database, thereby improving reverse engineering efficiency. Compared to other binary function identification systems, this system comprehensively considers both the binary information of the function and the function call graph information, resulting in higher efficiency, scalability, and better accuracy. Attached Figure Description

[0045] Figure 1 The overall flowchart of this invention.

[0046] Figure 2 Example diagram of feature embedding within a function.

[0047] Figure 3 Function database query flowchart.

[0048] Figure 4 Example diagram of adjacent verification phases. Detailed Implementation

[0049] The technical solution of the present invention will be further described below with reference to the embodiments and accompanying drawings.

[0050] Figure 1 The diagram shows the workflow of the binary function recognition system.

[0051] First, analysts need to build a function database containing a large amount of function embedding information and call relationships, which includes the following steps:

[0052] Step 1: Collect projects with source code and compile the source code to generate binary files.

[0053] This invention requires first collecting projects with source code, and then compiling the source code using various compilation options (including different optimization levels, function inlining thresholds, etc.) while preserving debugging and symbol information, to generate binary files with symbol information.

[0054] Step 2: Use binary analysis tools to analyze the binary file and write scripts to export the internal characteristics of functions and the calling relationships between functions in the file.

[0055] The internal features of this invention include the assembly instruction sequence within a basic block and the control flow graph of functions. The call relationship between functions refers to the function call graph. Specifically, this invention uses Ghidra or other binary analysis software to analyze binary files and runs scripts provided by this framework to extract the control flow graph of each function (segmenting basic blocks and determining the jump relationships within basic blocks of functions), the assembly instruction sequence within the basic blocks, and the call relationships between functions. Furthermore, for binary files with symbolic information, the symbolic names of functions are also exported and correlated with the internal features of the functions and the function call graph.

[0056] In step 2, the script also performs regularization on the assembly instructions. That is, when collecting binary function feature information, to reduce the dictionary size of the neural network model, the assembly code needs to be regularized. For immediate values, if their value is not between 0x100 and -0x100, "IMM" is used as a substitute; for operands where the address is taken, "IMM" is used. <ptr>+base+imm”," <ptr>+base+index”," <ptr>Encode using the format "+base+index+imm", where " <ptr>Yes, it's a special token used to identify memory access. Among them, <ptr>It is a special token used to identify memory access. Base is the base address register, index represents the index register and multiplier, and imm represents the immediate value of the offset.

[0057] Step 3: Using the collected function control flow graph and assembly instruction information, train the script to train the neural network, and use the trained neural network to embed the collected binary functions into the vector space, and store them together with the function call relationships into the function database.

[0058] Figure 2 This demonstrates the process of embedding binary function features into vectors. First, a Natural Language Processing (NLP) model is used to process each basic block of the function, and the final internal state of the model serves as the vector embedding representation of the basic block. Then, a Graph Neural Network (GNN) model is used, with the basic blocks as nodes and the Control Flow Graph (CFG) as edges, to perform five rounds of information propagation, updating the vector representation of the nodes. Finally, a GNN readout function is used to synthesize the information of all nodes in the entire graph to obtain the vector representation of the entire function.

[0059] In one embodiment, when constructing the function database, if binary functions with different function bodies but the same symbolic name are encountered, a maximum of five copies of the same symbolic name function are retained. An appropriate number of variants of the same symbolic function can improve the system's recall rate.

[0060] In one embodiment, when constructing the function database, the function call graph in the function database uses symbol names as nodes to record all occurrences of call relationships. If different files have different inlining choices during compilation, a pair of functions may exist in one file but not in another. In this case, the call relationship of this pair of functions should be recorded. Therefore, this invention treats all functions with the same symbol as equivalence classes to construct the call graph and stores this graph ("extended function call graph") in the database.

[0061] In one embodiment, the neural network model used in this invention integrates HBMP, GGNN, and Set2Set models, ultimately generating 128-bit embeddings. During neural network training, the cosine function is used to calculate the distance between two functions, and triplet loss is used for training with a margin set to 0.5. This training method ensures that similar binary functions are as close as possible in the vector space, while dissimilar functions are as far apart as possible. For simplicity, similar binary functions refer to functions compiled from the same source code, although their compilers and compilation options may differ.

[0062] Thereafter, the constructed function database can be used to query functions of unknown binary programs. For an unknown binary program, function information is collected by the same method described above, embedded into vectors using the same neural network model, and finally queried in the function database by a script.

[0063] Figure 3 shows the function database query process. After obtaining functions from the unstripped binary file, querying in the function database can be divided into three steps:

[0064] 1) Initialization phase.

[0065] First, the similarity between all binary functions to be queried and all binary functions in the function database needs to be calculated. This similarity is evaluated using cosine distance: the shorter the distance, the higher the similarity.

[0066] Then, two thresholds a < b are determined. Analysts can manually modify the values of a and b. Higher values will lead to higher accuracy but reduce the recall rate. The default values of the system are 0.6 and 0.8 respectively. When the similarity of a pair of matches is greater than a, they are a pair of "potential matches". When the similarity is greater than b, they are a pair of "high-probability matches".

[0067] 2) Adjacent verification phase.

[0068] Figure 4 shows an example diagram of the adjacent verification phase. Circles in the diagram represent functions, and arrows represent call relationships between functions. Among them, two circles circled by black solid lines are a pair of "potential matches", which have 2 and 3 callers and 2 and 1 callees respectively. A pair of callers and a pair of callees, marked by black solid and dashed lines, are "high-probability matches". Therefore, this pair of "potential matches" is further verified by two pairs of "high-probability matches" and obtains additional similarity scores in sequence.

[0069] Specifically, the adjacency verification phase can be performed in multiple rounds. In each round, for a "potential match" that is not yet a "high-probability match," a similarity reward value is calculated based on the number of "high-probability matches" among its neighbors (n-order callers and callees). The calculation formula is ln(1+cnt)*c, where cnt is the number of "high-probability matches" among the caller and callee, and c is an adjustable parameter with a default value of 0.3. For functions in the binary file to be matched, their neighbors are obtained from the function call graph of that binary file; for binary functions in the database, their neighbors are obtained from the "extended function call graph" in the database. If the sum of the basic similarity value (calculated using vector space distance) and the similarity reward value is higher than b, then the match is also considered a "high-probability match" and will affect the calculation of the similarity reward value of other "potential matches" in the next round. The adjacency verification phase continues until no new "high-probability matches" appear in a round.

[0070] In one embodiment, for a pair of potential mappings, in addition to counting the direct caller and direct callee, the higher-order caller of caller and callee can also be considered. That is, the invention can set a parameter, such as x. When x = 1, "direct caller and direct callee" are considered; when x = n, "callers and callees of orders 1 to n" are considered. This is also to address the impact of function inlining on the change in the call graph.

[0071] 3) Final derivation stage.

[0072] In the final derivation phase, a new threshold d needs to be determined to accept or reject each "potential match". The default value of d in the system is 0.8, which means that only "high-probability matches" are accepted. Similar to a and b, it can be manually modified to balance precision and recall.

[0073] Although specific embodiments and accompanying drawings of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. The invention should not be limited to the content disclosed in the preferred embodiments and accompanying drawings; the scope of protection of the invention is defined by the claims.< / ptr> < / ptr> < / ptr> < / ptr> < / ptr> < / ptr> < / ptr> < / ptr> < / ptr>

Claims

1. A binary function recognition method combining embedding similarity and function call graphs, characterized in that, The method includes: Collect projects with source code, compile the source code, and generate binary files; The binary file is analyzed to derive the internal characteristics of each function and the call relationships between functions; for binary files with symbol information, the symbol names of the functions are derived, and the symbol names of the functions are correlated with the internal characteristics and function call graphs of the functions. The internal features of the functions are vectorized to obtain the feature vectors of each function; Treat all functions with the same symbol as equivalence classes, and combine the calling relationships between these functions to construct an extended function call graph that represents the calling relationships between symbol names; A binary function database is constructed based on the feature vectors of the functions and the extended function call graph. The binary function database is used to query the functions of an unknown binary program to obtain the binary function identification results in the unknown binary program; Specifically, the binary function database is used to query functions of an unknown binary program to obtain the binary function identification results of the unknown binary program, including: Obtain the first similarity threshold Second similarity threshold and the third similarity threshold ;in, ; For an unknown binary program, calculate the feature vector of each function to be detected and the function call graph between the functions to be detected; The similarity between function pairs is obtained by calculating the cosine distance between the feature vector of the function to be detected and the feature vector of each function in the binary function database. And combined with the first similarity threshold Second similarity threshold The matching status of the function pair is determined; wherein the matching status includes: impossible matching function pair, potential matching function pair, or high probability matching function pair; When a function pair is identified as a potential matching function pair, its corresponding neighbors are obtained based on both the function call graph and the extended function call graph. At least one round of neighbor verification is then performed based on these neighbors to determine the similarity of the function pair. ;in, The number of adjacent verification rounds; If a function pair is determined to be a high-probability matching function pair, or the similarity of a function pair is... Greater than the third similarity threshold If the function in the binary function database is identified as the binary function recognition result of the function to be detected, then the corresponding function in the binary function database of the function pair is determined.

2. The method as described in claim 1, characterized in that, The internal features of the function include: the sequence of assembly instructions within the basic block and the function's control flow graph.

3. The method as described in claim 2, characterized in that, The binary file is analyzed to derive the internal characteristics of the functions, including: The regularized binary file is analyzed using a binary analysis tool to derive the unregularized assembly instruction sequence and function control flow graph within the basic blocks; wherein, the binary analysis tool includes: Ghidra; Regularize the unregularized assembly instruction sequence to obtain the assembly instruction sequence within the basic block.

4. The method as described in claim 3, characterized in that, The step of regularizing the unregularized assembly instruction sequence to obtain the assembly instruction sequence within the basic block includes: If the value of the immediate value is not between 0x100 and -0x100, then "IMM" is used to replace the immediate value; When operands involve taking addresses, use <ptr>+ base + imm、 <ptr>+ base + index and <ptr>Encoding is performed using the method of + base + index + imm; where... <ptr> It is a special token used to identify memory access. Base is the base address register, index represents the index register and multiplier, and imm represents the immediate value of the offset.< / ptr> < / ptr> < / ptr> < / ptr> 5. The method as described in claim 2, characterized in that, The internal features of each function are vectorized to obtain the feature vector of each function, including: The assembly instruction sequence within each basic block of the function is processed using a natural language processing model to obtain a vector embedding representation of the basic block. Using a graph neural network model, basic blocks are used as nodes and control flow graphs are used as edges. Several rounds of information propagation are performed to update the vector representation of the nodes. The updated node vector representations are synthesized using a GNN network to obtain the feature vectors of the internal features of the function.

6. The method as described in claim 1, characterized in that, Combined with the first similarity threshold Second similarity threshold Determine the matching status of the function pair, including: exist In the case of this, the function pair is determined to be an impossible matching function pair; exist In the case of this, the function pair is determined to be a potential matching function pair; exist In the case of [condition], the function pair is determined to be a high-probability matching function pair.

7. The method as described in claim 1, characterized in that, The process involves performing at least one round of neighbor verification based on neighbors to obtain the similarity of potential matching function pairs. ,include: Calculate the number of high-probability matching function pairs among the neighbors. The similarity reward value for each round of adjacent verification; The first Similarity reward value and similarity in round-adjacent validation Add them together to get the similarity score. Among them, the similarity in the first round of adjacent verification For similarity ; If the first At least one new similarity appears in the adjacent verification round. Greater than the second similarity threshold If so, then the function pair is considered a high-probability matching function pair, and let Then, return to the point where the number of highly probable matching function pairs among the neighbors is calculated, and calculate the first... The similarity reward value for each round of adjacent verification; If the first No similarity was found in the adjacent verification round. Greater than the second similarity threshold Then output the similarity of each potential matching function pair. .

8. The method as described in claim 7, characterized in that, The first Similarity reward value for adjacent validation rounds ;in, This indicates the number of highly likely matching function pairs among the neighbors. This is an adjustable parameter.

9. A binary function recognition system combining embedding similarity and function call graphs, characterized in that, The system includes: The training module collects projects with source code, compiles the source code to generate binary files, analyzes the binary files to derive the internal features of each function and the call relationships between functions, specifically, for binary files with symbol information, derives the symbol names of functions, and correlates these symbol names with the internal features and function call graphs; the internal features of functions are vectorized to obtain the feature vectors of each function; all functions with the same symbol are treated as equivalence classes, and combined with the call relationships between functions, an extended function call graph representing the call relationships between symbol names is constructed; based on the function feature vectors and the extended function call graph, a binary function database is constructed. The testing module is used to query functions of an unknown binary program using the binary function database, and obtain the binary function identification results in the unknown binary program; Specifically, the binary function database is used to query functions of an unknown binary program to obtain the binary function identification results of the unknown binary program, including: Obtain the first similarity threshold Second similarity threshold and the third similarity threshold ;in, ; For an unknown binary program, calculate the feature vector of each function to be detected and the function call graph between the functions to be detected; The similarity between function pairs is obtained by calculating the cosine distance between the feature vector of the function to be detected and the feature vector of each function in the binary function database. And combined with the first similarity threshold Second similarity threshold The matching status of the function pair is determined; wherein the matching status includes: impossible matching function pair, potential matching function pair, or high probability matching function pair; When a function pair is identified as a potential matching function pair, its corresponding neighbors are obtained based on both the function call graph and the extended function call graph. At least one round of neighbor verification is then performed based on these neighbors to determine the similarity of the function pair. ;in, The number of adjacent verification rounds; If a function pair is determined to be a high-probability matching function pair, or the similarity of a function pair is... Greater than the third similarity threshold If the function in the binary function database is identified as the binary function recognition result of the function to be detected, then the corresponding function in the binary function database of the function pair is determined.

Images