A binary code similarity detection method to eliminate false positives

Abstract

This invention discloses a binary code similarity detection method that eliminates false positives, belonging to the field of static software vulnerability detection and analysis. The method involves: extracting control flow graph information from the binary code using reverse engineering tools to construct structural feature vectors and attribute feature vectors for all nodes; generating a first similarity matrix based on these vectors; approximating the first similarity matrix with a low-rank matrix to obtain the vector representation of each node; quickly aligning the graph nodes; converting the distance metric between the node vector representations into a second similarity score; and finally, converting the second similarity score into the final similarity score between the two binary codes. This invention eliminates false positives caused by "embedding collisions" in binary code similarity detection.

Description

Technical Field

[0001] This invention relates to the field of static software vulnerability detection and analysis, and in particular to a binary code similarity detection method for eliminating false positives. Background Art

[0002] Binary code similarity detection has been used as a core foundation for various security and software engineering applications, including malware clustering, code clone detection, and vulnerability auditing. Currently, mainstream binary code similarity detection methods primarily rely on neural networks (NNs) to extract structural information, such as control flow and data flow graphs. However, neural network-based techniques, which capture lexical, control structure, or data flow level information of binary code for representation learning, are often too coarse-grained, significantly losing semantic information and resulting in a large number of false positives among the retrieved top-k candidates. Furthermore, it exhibits low detection capability and stability under various challenging settings, such as across optimization levels and obfuscation.

[0003] Currently, there are many challenges in binary code similarity detection:

[0004] (1) The "embedding conflict" problem: Since the READOUT function in most neural network-based binary code similarity detection methods is a summation function, some logically dissimilar binary code segments may have similar summations of their different node information embeddings. This makes it impossible for mainstream binary code similarity detection methods to be applied to the elimination of such false alarms.

[0005] (2) The compiler, compilation options and optimizer all have a significant impact on the final generated binary code, resulting in different generated binary code in different environments even if the source code is the same, thus affecting similarity detection.

[0006] (3) When extracting the semantics of binary code, neural network-based models lose semantic information to a large extent.

[0007] Graph alignment can be used to eliminate false alarms caused by "embedding collisions." However, there is currently a lack of methods for equivalence testing using graph alignment in the context of binary code. Therefore, it is important and necessary to develop relevant analysis methods and systems. Summary of the Invention

[0008] To address the problems in existing methods, this invention provides a low-cost and comprehensive graph alignment method to eliminate false positives in binary code similarity detection.

[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0010] A binary code similarity detection method to eliminate false positives includes the following steps:

[0011] S1: Perform reverse analysis on the binary code to be detected to obtain the control flow graph of the binary code;

[0012] S2: Based on the structure of the control flow graph, construct structural feature vectors for all nodes in the control flow graph;

[0013] S3: Extract semantic information from the control flow graph and construct attribute feature vectors for all nodes in the control flow graph based on the semantic information;

[0014] S4: Construct a similarity matrix between control flow graphs based on structural feature vectors and attribute feature vectors;

[0015] S5: Calculate the similarity score between binary codes based on the similarity matrix, and perform binary code similarity detection based on the similarity score.

[0016] Furthermore, the control flow graph includes information about the basic blocks in the binary code and their execution order.

[0017] Further, step S2 specifically includes:

[0018] S2.1: Construct an undirected graph based on the structure of the control flow graph, wherein the weight of each edge in the undirected graph is 1;

[0019] S2.2: For each node in the undirected graph, take all other nodes in the same undirected graph and assign a discount factor to the selected nodes: Where k is the distance between the node and the current node;

[0020] S2.3: For each node in the undirected graph, calculate the number of neighboring nodes at different distances, and construct the node's structural feature vector using the discount factor as weights.

[0021] d s [k] = len(neighbors) × δ k

[0022] Where, d s [k] is the k-th dimension of the structural feature vector, and len(neighbors) is the number of nodes that are k distances away from the current node.

[0023] Further, step S4 specifically includes:

[0024] S4.1: Select p nodes as marker nodes from the control flow graphs of the two binary codes to be detected. Calculate the similarity between each node and the first node of the p marker nodes based on the structural feature vector and attribute feature vector, and form an n×p first similarity matrix C, where n is the sum of the number of nodes in the control flow graphs of the two binary codes to be detected.

[0025] S4.2: Based on the first similarity matrix C, obtain the vector representation of each node through low-rank approximation and SVD matrix decomposition;

[0026] S4.3: Calculate the cosine similarity value between the vector representations of the nodes of each control flow graph, and use it as the second similarity between the nodes to form an n×n second similarity matrix.

[0027] Further, step S5, which involves calculating the similarity score between binary codes based on the similarity matrix, specifically involves: for each row of the second similarity matrix, calculating the maximum value among the elements in that row as the row maximum value, and averaging all the row maximum values; for each column of the second similarity matrix, calculating the maximum value among the elements in that column as the column maximum value, and averaging all the column maximum values; and then averaging the average of the row maximum values ​​and the average of the column maximum values ​​again, with the final average being the similarity score between the binary codes.

[0028] Compared with existing technologies, the present invention has the following advantages:

[0029] 1. This invention innovatively introduces graph alignment technology into binary code similarity detection, thus overcoming the false alarm problem caused by "embedding collision" in existing binary code similarity detection technologies, thereby achieving higher search accuracy.

[0030] 2. This invention employs The method performs low-rank approximation and SVD matrix decomposition, representing the node information of the binary code control flow graph as a low-dimensional vector, thereby obtaining the similarity score between binary codes more efficiently. This overcomes the problem of excessively high time cost of similarity detection in existing technologies, thus achieving higher detection efficiency. Attached Figure Description

[0031] Figure 1 A schematic diagram of the overall structure of a low-cost and comprehensive graph alignment method for eliminating false alarms in binary code similarity detection;

[0032] Figure 2 A schematic diagram for obtaining structural and attribute feature vectors from a control flow graph;

[0033] Figure 3 To construct a similarity matrix and utilize A schematic diagram illustrating the method of performing low-rank matrix approximation and decomposition to obtain node vector representations;

[0034] Figure 4 To achieve rapid node alignment, a diagram is shown that calculates the similarity score between two graphs and makes a judgment based on a set threshold. Detailed Implementation

[0035] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0036] like Figure 1 As shown, the invention is mainly divided into three parts: structure and attribute embedding vector extraction, construction of similarity matrix and node alignment.

[0037] The workflow for extracting structural features and attribute feature vectors is as follows: Figure 2 As shown, the following steps are included:

[0038] S1: Perform reverse analysis on the binary code to be tested to obtain the control flow graph of the binary code.

[0039] S1.1: Collecting Control Flow Graph (CFG) Information Using Reverse Engineering Tools: For binary code (functions or program blocks) requiring similarity detection, use reverse engineering tools (such as IDA Pro, Ghidra, or Radare2) to disassemble the target binary file. Distinguish each binary code segment into multiple basic blocks based on jump instructions and obtain the jump relationships between these basic blocks. Use these basic blocks as nodes to construct the control flow graph (CFG) for each binary code segment, representing the execution path of instructions in the program. Preprocess the CFG to generate unique identifiers for the basic blocks, facilitating subsequent graph alignment and similarity calculation.

[0040] S1.2: Use NetworkX to construct an undirected graph to represent control flow graph information. The connections between nodes in the undirected graph are the same as those in the control flow graph, but the direction of the edges is removed, and the weight of all edges is assigned to 1.

[0041] S2: Based on the structure of the control flow graph, construct structural feature vectors for all nodes in the control flow graph, such as... Figure 2 As shown.

[0042] S2.1: Construct the k-hop neighborhood of each node recursively. The k-hop neighborhood consists of all nodes that are k steps away from the target node. Specifically, for each node in the undirected graph, first find its direct neighbors as its 1-hop neighborhood. Direct neighbors are nodes directly adjacent to the target node. A dynamic data structure is used to continuously track the processed neighbor nodes, recursively calculating the hop neighborhood of the current node until the farthest node is reached. In each hop, a discount factor is assigned to each neighbor node, calculated based on distance. Different discount factors δ1, δ2, δ3, ..., δ are assigned to neighbor nodes at different distances from the current node. k ,…,in This represents the discount factor for neighboring nodes that are k steps away from the current node.

[0043] S2.2: Constructing the Degree Distribution Sequence: In the k-hop neighborhood of each node, calculate the degree distribution of all neighboring nodes, and calculate the neighbor degrees by weighting them according to the discount factor, thereby constructing the structural feature vector d of the node. s :

[0044] d s [k] = len(neighbors) × δ k

[0045] Where, d s [j] represents the k-th dimension of the structural feature vector. The number of dimensions of the structural feature vector is not less than the longest path length of the undirected graph. len(neighbors) is the number of neighboring nodes in the k-hop neighborhood of this node.

[0046] The structural feature vectors of each node are superimposed to generate a degree distribution matrix, where each row represents a node and contains the neighborhood information of that node and its corresponding degree distribution features.

[0047] S3: Construct attribute feature vectors for all nodes in the control flow graph based on the semantic information of the control flow graph.

[0048] In this embodiment, an NLP model is used to extract semantic information from binary code. The code content of functions or basic blocks in the control flow graph, as well as the connections between them, are used as the input text for the model.

[0049] S3.1: Pre-training Phase: The BERT model is used to pre-train the semantic information of the binary code. The BERT model is an NLP model based on the Transformer architecture. In the Masked Language Model (MLM) task, the tokens are masked to simulate semantic filling in real-world contexts, and all adjacent nodes are identified for pre-training in the Adjacent Node Prediction (ANP) task. Two graph-level tasks, Intra-Block Graph (BIG) and Graph Classification (GC), are added to further explore semantic information and optimize block representation learning. Intra-Block Graph (BIG): By determining whether sampled blocks belong to the same graph, the intra-graph association between different blocks is learned. Graph Classification (GC): Based on the different platforms / optimization levels to which blocks belong, the graph is classified to learn platform / optimization level affiliation.

[0050] S3.2: Attribute Feature Extraction: Information about each control flow block is obtained using a control flow graph (CFG) parsing tool. This information is then processed into block embedding vectors using a trained NLP model, and these vectors are used as attribute feature vectors for each graph node.

[0051] S4: Construct a similarity matrix between control flow graphs based on structural feature vectors and attribute feature vectors, including the following steps:

[0052] S4.1: Construct the first similarity matrix from all nodes in the two graphs to the “signature” node.

[0053] (1) The Euclidean distance of the structural feature vectors and the Euclidean distance of the attribute feature vectors are weighted and summed to obtain the total distance. The total distance is then converted into a first similarity score between 0 and 1 using an exponential function. The specific formula is as follows:

[0054] sim(u,v)=exp[-γ s ·dist(d u d v )-γ a ·dist(f u f v )]

[0055] Where, γ s γ a For weights, d u d v Let f be the structural feature vector of node u and node v. u f v Let u and v be the attribute feature vectors of nodes u and v, and dist(·) represents the calculation of the Euclidean distance.

[0056] (2) Select a small number of marker nodes. These marker nodes should be selected proportionally from both graphs. The similarity of each node is compared only with a small number of p marker nodes, where p is much smaller than the total number of nodes n in the two control flow graphs. Calculate the first similarity between the p marker nodes and all n nodes to form an n×p first similarity matrix C.

[0057] S4.2: As Figure 3 As shown, a decomposition representation of a low-rank approximate matrix is ​​constructed by using the similarity of each node to these marker nodes to construct the node representation. Specifically, this is done by: The method performs a low-rank approximation, decomposing the high-dimensional kernel function to obtain an approximate low-rank part, i.e. in Let W be the pseudo-inverse of the p×p similarity matrix W, which is composed of the first-node similarity between each pair of p marker nodes; then let W be the pseudo-inverse of W. Perform SVD decomposition, the specific formula is as follows: Where U and V are orthogonal matrices, and Σ is a diagonal matrix, then the formula is used... We obtain an n×p matrix Y, where each row of matrix Y is a vector representation of each node;

[0058] S4.3: For example Figure 4 The diagram shows how to quickly align graph nodes.

[0059] The cosine similarity value of the vector representations between each pair of nodes is calculated as the second similarity. A similarity matrix is ​​constructed based on the second similarity between each pair of nodes. The element in the i-th row and j-th column of the matrix is ​​the second similarity score between the i-th node in one control flow graph and the j-th node in another control flow graph, thus forming the second similarity matrix.

[0060] S5: Extract the information from the second similarity matrix and calculate the final similarity score between the two images.

[0061] For each row of the matrix, the maximum value of its elements is calculated. This aims to capture the strongest alignment signal in each row, reflecting the maximum matching degree along the row direction. Similarly, for each column of the matrix, the maximum value of its elements is also calculated. This operation reflects the maximum matching degree along the column direction. The average of the sets of maximum values ​​for the rows and the average of the sets of maximum values ​​for the columns are calculated. These two averages are then taken to obtain the final similarity score between the two images.

[0062] S6: Determine whether the two binary code segments are similar based on the final score.

[0063] This step can be based on a preset score threshold or ranking threshold. If based on the preset score threshold, codes with a final similarity score greater than the threshold are considered similar, while those below the threshold are considered dissimilar. For binary code retrieval scenarios, a preset ranking threshold can be used to determine similarity. The query code snippet is compared with all code snippets in the database using the method described above to calculate the final similarity score. All participating code snippets are then sorted from highest to lowest based on their final similarity score; the top 10 snippets are considered similar to the query code snippet, and the rest are considered dissimilar.

[0064] In one specific embodiment of the present invention, it may further include: using a portion of the graph nodes for faster graph alignment, for example, selecting to align the first 70% of the nodes to achieve an improvement in system speed.

[0065] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A binary code similarity detection method to eliminate false positives, characterized in that, The following steps are involved: S1: Perform reverse analysis on the binary code to be detected to obtain the control flow graph of the binary code; S2: Based on the structure of the control flow graph, construct structural feature vectors for all nodes in the control flow graph; S3: Extract semantic information from the control flow graph and construct attribute feature vectors for all nodes in the control flow graph based on the semantic information; S4: Construct a similarity matrix between control flow graphs based on structural feature vectors and attribute feature vectors; S4.1: Select from the control flow graphs of the two binary codes to be tested Each node is used as a marker node, and the relationship between each node and its attribute feature vector is calculated based on the structural feature vector and attribute feature vector. The first node similarity of each marker node constitutes... The first similarity matrix C, where The sum of the number of nodes in the control flow graphs of the two binary codes to be detected; S4.2: Based on the first similarity matrix C, obtain the vector representation of each node through low-rank approximation and SVD matrix decomposition; S4.3: Calculate the cosine similarity value between the vector representations of the nodes in each control flow graph, and use it as the second similarity between the nodes to form a second similarity. The second similarity matrix; S5: Calculate the similarity score between binary codes based on the similarity matrix, and perform binary code similarity detection based on the similarity score.

2. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, The control flow graph includes the basic blocks in the binary code and their execution order information.

3. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, Step S2 specifically involves: S2.1: Construct an undirected graph based on the structure of the control flow graph, wherein the weight of each edge in the undirected graph is 1; S2.2: For each node in the undirected graph, take all other nodes in the same undirected graph and assign a discount factor to the selected nodes: , where k is the distance between the node and the current node; S2.3: For each node in the undirected graph, calculate the number of neighboring nodes at different distances, and construct the node's structural feature vector using the discount factor as weights: ； in, It is the k-th dimension of the structural eigenvector. It is the number of nodes that are k distances from the current node.

4. The binary code similarity detection method for eliminating false alarms according to claim 2, characterized in that, Step S3 specifically involves: inputting the control flow graph into the NLP model, extracting multiple block embedding vectors through the NLP model, and using the block embedding vectors as the attribute feature vectors of the corresponding nodes.

5. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, Step S4.2 specifically involves: based on The first-node similarity between each pair of marker nodes is used to construct... Given a similarity matrix W, find its pseudo-inverse. Then the pseudo-inverse matrix Obtained through SVD decomposition in It is an orthogonal matrix. It is a diagonal matrix, and then the orthogonal matrix and diagonal matrix obtained from the decomposition are used to obtain... Vector representation of matrix The vector representation matrix Y has each row as a vector representation of each node.

6. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, The method for calculating the similarity of the first node is as follows: ； in, , As weight, , Let u and v be the structural feature vectors. , Let be the attribute feature vectors of nodes u and v. This indicates the calculation of Euclidean distance.

7. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, Step S5, which involves calculating the similarity score between binary codes based on the similarity matrix, specifically involves: For each row of the second similarity matrix, the maximum value among the elements in that row is calculated as the row maximum value, and the average of all row maximum values ​​is taken. For each column of the second similarity matrix, the maximum value among the elements in that column is calculated as the column maximum value, and the average of all column maximum values ​​is taken. The average of the average of the row maximum values ​​and the average of the column maximum values ​​are then taken, and the final average is the similarity score between the binary codes.

8. The binary code similarity detection method for eliminating false alarms according to claim 1, characterized in that, The binary code similarity detection based on similarity scores is specifically as follows: Binary code similarity detection is performed based on a preset score threshold or a ranking threshold.

Images