Binary function similarity detection method based on agent strategy optimization
The binary function similarity detection method optimized by intelligent agent policy, which combines control flow graphs and abstract control flow graphs with graph neural networks and reinforcement learning algorithms, solves the problems of high false alarm rate and poor adaptability in existing technologies, and achieves efficient and accurate binary function similarity detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-04-03
AI Technical Summary
Existing binary code similarity detection methods suffer from high false positive and false negative rates when faced with complex code transformations, and deep learning models lack adaptability and interpretability, making it difficult to meet the needs of software supply chain security.
An agent-based strategy optimization method is adopted. By disassembling the binary file to extract the control flow graph and abstract control flow graph, and combining graph neural network and reinforcement learning algorithm, the embedding vector of the function is generated. Then, the similarity judgment strategy is optimized through reinforcement learning to achieve adaptive function similarity detection.
It improves the robustness and accuracy of detection, reduces the amount of computation and computing resources consumed, is suitable for large-scale third-party library detection, and reduces the probability of false positives and false negatives.
Smart Images
Figure CN121786498A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer security, and in particular to a binary function similarity detection method based on agent policy optimization. Background Technology
[0002] With the widespread reuse of open-source code and the proliferation of third-party libraries in modern software development, software supply chain security has become a core concern in the field of information security. While the introduction of third-party library code improves development efficiency, it also brings potential security vulnerabilities, malicious code injection, and intellectual property infringement risks. Therefore, accurately detecting whether binary programs contain unauthorized or risky third-party library code is of great significance for software composition analysis, vulnerability discovery, malware detection, and copyright protection.
[0003] Traditional binary code similarity detection methods are mainly divided into two categories: string matching-based methods and static function feature-based methods. The former compares variable information such as symmetric predicate strings and constants, but its accuracy drops significantly when faced with code obfuscation, compiler optimization, or the removal of debugging information. The latter typically extracts static structural features such as function control flow graphs, data flow graphs, or instruction sequences, and uses graph matching or sequence alignment algorithms to calculate similarity. While these methods are robust to simple code transformations, they rely on predefined feature engineering and fixed similarity measurement rules, making them difficult to adapt to complex semantic equivalence transformations caused by different compilers, optimization levels, instruction set architectures, and advanced obfuscation techniques, resulting in high false positive and false negative rates.
[0004] In recent years, deep learning techniques, especially graph neural networks, have been introduced to learn more robust function representations. However, existing deep learning-based solutions typically treat it as a static pattern matching problem, relying on supervised training on large-scale labeled data, and the models lack dynamic decision-making and adaptive adjustment capabilities. When faced with code variants or novel obfuscation techniques outside the training data distribution, their generalization performance is limited, and the interpretability of the model's decision-making process is poor, making it difficult to meet the credibility requirements of security analysis results. Summary of the Invention
[0005] The purpose of this invention is to provide a binary function similarity detection method based on agent policy optimization, aiming to provide a binary function similarity detection method that can adaptively learn and optimize matching strategies and has stronger robustness to complex code transformations.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides a binary function similarity detection method based on agent policy optimization, used to detect the similarity between a target binary file and functions in a candidate third-party library. The method includes: S1: Disassembling the target binary file and candidate third-party library files to extract the control flow graph and abstract control flow graph corresponding to each function; wherein, the control flow graph is a graph constructed with basic function blocks as nodes and control flow relationships or function call relationships between basic blocks as edges; the abstract control flow graph is a graph formed by abstracting and simplifying the node features and edge relationships of the control flow graph; S2: Inputting the extracted control flow graph and abstract control flow graph into an agent, where the agent is an integrated graph. Neural networks and reinforcement learning algorithms are used to develop a model for function similarity determination. A graph neural network is used to learn features from the input control flow graph and abstract control flow graph, generating embedding vectors for each function. Reinforcement learning algorithms are then used to optimize the agent's similarity determination strategy. Step S3: The control flow graph and abstract control flow graph of the function in the target binary file are input into the trained agent, and similarity is calculated between them and the embedding vectors of functions in candidate third-party libraries. A preset threshold is used to determine if they are similar function pairs. Step S4: The determination results from step S3 are compared with the true labels. Reward or penalty mechanisms are used to adjust the agent's policy parameters, enabling the agent to self-optimize.
[0007] The disassembly process in step S1 includes: parsing instructions of the target binary file and candidate third-party library files, identifying the start and end boundaries of functions, extracting basic block information of each function and the relationship between basic blocks, so as to construct a control flow graph and an abstract control flow graph.
[0008] In step S2, the difference between the predicted similarity and the true label of the agent's output from the integrated graph neural network and reinforcement learning algorithm is calculated using a loss function. Combined with the reward and penalty mechanism of reinforcement learning, the feature extraction parameters of the graph neural network and the policy parameters of reinforcement learning are simultaneously optimized. The loss function is: in, The similarity predicted by the agent. This is a real label.
[0009] In step S2, the reinforcement learning algorithm is the Q-learning algorithm.
[0010] In step S3, similarity calculation is achieved through a similarity index that reflects the degree of spatial association between two embedded vectors. The similarity index is based on the dimensionality and spatial distribution characteristics of the embedded vectors using the formula... Calculated; in, For functions in the target binary file to be detected, Functions in candidate third-party libraries; This is the embedding vector corresponding to the function in the target binary file to be detected. This is the embedding vector corresponding to the function in the candidate third-party library.
[0011] .
[0012] In step S3, the preset threshold is determined by statistically analyzing the similarity distribution of historical similar function pairs and dissimilar function pairs during the training process. The size of the preset threshold varies depending on the requirements of different detection scenarios.
[0013] In step S4, the reward or punishment mechanism includes: when the judgment result of the agent integrating the graph neural network and the reinforcement learning algorithm is consistent with the real label, a positive incentive signal is fed back to the agent; when the judgment result is inconsistent with the real label, a negative constraint signal is fed back to the agent, and the parameters of the reinforcement learning algorithm are adjusted based on the incentive signal or the constraint signal.
[0014] In control flow graphs and abstract control flow graphs, each node carries instruction-related information such as the instruction operation type and operand characteristics of the corresponding basic block, and each edge carries control flow transition probability or call priority information.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This application provides a binary function similarity detection method based on agent policy optimization. It extracts the function's control flow graph and abstract control flow graph through disassembly. Nodes in the graphs carry key information such as instruction operation type and operand characteristics, while edges carry control flow transition probabilities or call priority information. Combined with the powerful graph structure learning capabilities of graph neural networks, it can uncover the underlying structural and behavioral features of functions, avoiding the shortcomings of traditional methods that rely on easily invalidated features such as static function names and constants, making feature representations more closely aligned with the function's essence. Using cosine similarity as the similarity index, it accurately calculates the degree of association between functions through the dimensional and spatial distribution features of the embedded vectors. Compared to traditional graph matching algorithms, it can more finely characterize the differences in function similarity, reducing judgment bias caused by coarse feature matching. Since the embedded vectors generated by the graph neural network have compact feature representation capabilities, similarity calculation only requires cosine similarity operations based on the embedded vectors. Compared to the complex graph structure comparison of traditional graph matching algorithms, the computational load is significantly reduced, greatly improving the matching efficiency between the target binary file and candidate third-party library functions. Therefore, it is suitable for large-scale third-party library detection scenarios.
[0016] 2. The method provided in this application quantifies the deviation between predicted similarity and the true label through a loss function during the agent training phase. Simultaneously, it combines the reward and penalty mechanism of reinforcement learning to optimize the feature extraction parameters of the graph neural network and the policy parameters of reinforcement learning. This achieves collaborative calibration of feature extraction and decision-making, fundamentally improving the model's prediction accuracy and effectively reducing the probability of misclassification and missed classification. The Q-learning reinforcement learning algorithm is used to optimize the agent's decision-making strategy. Combined with the bias quantification effect of the loss function, the model can quickly locate defects in feature extraction and decision-making logic during training, simultaneously optimizing both parameters. This avoids the slow convergence problem caused by single optimization, reduces the number of iterations required for model training and the dependence on labeled data, and lowers computational resource consumption. Attached Figure Description
[0017] Figure 1 This is a flowchart of a binary function similarity detection method based on agent policy optimization provided in an embodiment of this application. Detailed Implementation
[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0019] For example, such as Figure 1 As shown in the figure, this application provides a binary function similarity detection method based on agent policy optimization, including: S1: Disassemble the target binary file and candidate third-party library files, and extract the control flow graph (CFG) and abstract control flow graph (ACFG) for each function. The control flow graph is a graph with basic function blocks as nodes and control flow relationships or function call relationships between basic blocks as edges. The abstract control flow graph is a graph formed by abstracting and simplifying the node features and edge relationships of the control flow graph.
[0020] In control flow graphs and abstract control flow graphs, each node carries instruction-related information such as the instruction operation type and operand characteristics of the corresponding basic block, and each edge carries control flow transition probability or call priority information.
[0021] The disassembly process in step S1 includes: parsing the target binary file and candidate third-party library files, such as candidate TPL binary files, identifying the start and end boundaries of functions, extracting the basic block information of each function and the relationship between basic blocks, in order to construct a control flow graph and an abstract control flow graph.
[0022] For example, each function is represented as a graph. ,in A set of nodes represents the basic blocks in a function. Let be the set of edges representing control flow and call relationships. Therefore, the graph representation of each function is as follows: The objective function and the TPL function are obtained respectively. and .
[0023] S2: Input the extracted control flow graph and abstract control flow graph into the agent. The agent is a model that integrates graph neural network (GNN) and reinforcement learning algorithm for function similarity determination. The GNN performs feature learning on the input control flow graph and abstract control flow graph to generate the embedding vector corresponding to each function. The reinforcement learning algorithm is used to optimize the similarity determination strategy of the agent.
[0024] The agent uses graph neural networks to learn representations of the function graph, acquires embeddings through deep learning, and optimizes the decision policy using reinforcement learning. For example, the target function graph... And candidate TPL function graph Embedding is performed using a graph neural network to obtain the embedding vector for each function. The embedding vector. During agent training, through the objective function graph. And candidate TPL function graph Calculate similarity and adjust the strategy using a reward mechanism. Objective function graph. And candidate TPL function graph The corresponding embedding vector is obtained through the agent. Then, the similarity is calculated using the following formula: As one possible implementation, in step S2, the agent's policy is optimized using the reinforcement learning algorithm Q-learning. The difference between the predicted similarity of the agent's output (integrated with the graph neural network and the reinforcement learning algorithm) and the true label is calculated using a loss function. Combined with the reward and penalty mechanism of reinforcement learning, the feature extraction parameters of the graph neural network and the policy parameters of the reinforcement learning are simultaneously optimized. The loss function is: in, The similarity predicted by the agent. The true label refers to the benchmark identifier that clearly marks whether the functions in the target binary file and the functions in the candidate third-party library constitute similar function pairs, based on known function origin information, code ownership relationships, or manual verification results. It is the core reference standard for measuring the accuracy of the agent's judgment results and triggering model optimization, possessing objectivity and uniqueness. Through the above process, the agent is trained, enabling it to have accurate and efficient discrimination capabilities.
[0025] S3: Input the control flow graph and abstract control flow graph of the function in the binary file of the target to be detected into the trained agent, calculate the similarity with the embedding vector of the function in the candidate third-party library, and determine whether they are similar function pairs according to the preset threshold.
[0026] It should be understood that the target binary file to be detected refers to the specific binary file that needs to be used by the trained agent for actual similarity determination after entering the inference stage. It is a specific subset of the target binary file in the inference stage. The target binary file covers both the agent training stage and the inference stage, and is a general term for the detection object throughout the entire process. In the training stage, i.e., in step S2, it serves as the source of training data, providing the control flow graph and abstract control flow graph of the function for the agent's feature learning and policy training. In the inference stage, i.e., the target binary file to be detected, it serves as the final detection object.
[0027] In step S3, similarity calculation is achieved through a similarity index that reflects the degree of spatial association between two embedded vectors. The similarity index is based on the dimensionality and spatial distribution characteristics of the embedded vectors using the formula... Calculated; in, For functions in the target binary file to be detected, Functions in candidate third-party libraries; This is the embedding vector corresponding to the function in the target binary file to be detected. This is the embedding vector corresponding to the function in the candidate third-party library.
[0028] .
[0029] In step S3, a preset threshold is set. The similarity distribution of historical similar and dissimilar function pairs during training is statistically determined, with the preset threshold varying depending on the requirements of different detection scenarios. When the similarity exceeds the preset threshold, the two functions are considered similar, and a similar function pair is output.
[0030] S4: Compare the judgment result of step S3 with the real label, and adjust the agent's policy parameters through reward or punishment mechanisms to achieve self-optimization of the agent.
[0031] In step S4, the agent learns from feedback in similarity function judgments to further optimize the model. During training, the agent continuously adjusts its strategy, using reward and penalty mechanisms to improve the accuracy of similarity judgments. When the decision of the agent integrating the graph neural network and the reinforcement learning algorithm is consistent with the true label, a positive incentive signal is fed back to the agent; when the decision is inconsistent with the true label, a negative constraint signal is fed back to the agent, and the parameters of the reinforcement learning algorithm are adjusted based on the incentive signal or the constraint signal.
[0032] In reinforcement learning, the update rule adjusts the agent's strategy based on reward feedback. .in, It is in state Take action below value, It's the learning rate. It is a discount factor.
[0033] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.
[0034] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A binary function similarity detection method based on agent policy optimization, used to detect the similarity between a target binary file and functions in candidate third-party libraries, characterized in that, include: S1: Disassemble the target binary file and candidate third-party library files to extract the control flow graph and abstract control flow graph corresponding to each function. The control flow graph is a graph with basic function blocks as nodes and control flow relationships or function call relationships between basic blocks as edges. The abstract control flow graph is a graph formed by abstracting and simplifying the node features and edge relationships of the control flow graph. S2: Input the extracted control flow graph and abstract control flow graph into an agent, which is a model integrating graph neural networks and reinforcement learning algorithms for function similarity determination. The graph neural network performs feature learning on the input control flow graph and abstract control flow graph to generate embedding vectors for each function. Reinforcement learning algorithms are then used to optimize the agent's similarity determination strategy. S3: Input the control flow graph and abstract control flow graph of the function in the target binary file to be detected into the trained agent. Similarity calculation is performed between this agent and the embedding vectors of functions in the candidate third-party libraries. A preset threshold is used to determine whether they are similar function pairs. S4: Compare the determination results of step S3 with the true labels. Adjust the agent's strategy parameters through reward or punishment mechanisms to achieve self-optimization of the agent.
2. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that, The disassembly process in step S1 includes: parsing the target binary file and candidate third-party library files, identifying the start and end boundaries of functions, extracting the basic block information of each function and the relationship between the basic blocks, so as to construct the control flow graph and the abstract control flow graph.
3. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that: In step S2, the difference between the predicted similarity and the true label output of the agent integrating the graph neural network and reinforcement learning algorithm is calculated using a loss function. Combined with the reward and penalty mechanism of reinforcement learning, the feature extraction parameters of the graph neural network and the policy parameters of reinforcement learning are simultaneously optimized. The loss function is: in, The similarity predicted by the agent. This is a real label.
4. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that: In step S2, the reinforcement learning algorithm is the Q-learning algorithm.
5. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that: In step S3, the similarity calculation is achieved through a similarity index that reflects the degree of spatial association between two embedded vectors. This similarity index is based on the dimensionality and spatial distribution characteristics of the embedded vectors, expressed by a formula. Calculated; in, For functions in the target binary file to be detected, Functions in candidate third-party libraries; This is the embedding vector corresponding to the function in the target binary file to be detected. This is the embedding vector corresponding to the function in the candidate third-party library.
6. The binary function similarity detection method based on agent policy optimization according to claim 5, characterized in that, 。 7. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that: In step S3, the preset threshold is determined by statistically analyzing the similarity distribution of historical similar function pairs and dissimilar function pairs during the training process. The size of the preset threshold varies depending on the requirements of different detection scenarios.
8. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that, In step S4, the reward or punishment mechanism includes: when the judgment result of the agent integrating the graph neural network and the reinforcement learning algorithm is consistent with the real label, a positive incentive signal is fed back to the agent; when the judgment result is inconsistent with the real label, a negative constraint signal is fed back to the agent, and the parameters of the reinforcement learning algorithm are adjusted based on the incentive signal or the constraint signal.
9. The binary function similarity detection method based on agent policy optimization according to claim 1, characterized in that, In the control flow graph and the abstract control flow graph, each node carries instruction-related information such as the instruction operation type and operand characteristics of the corresponding basic block, and each edge carries control flow transition probability or call priority information.