Method, system and storage medium for detecting opaque predicates in malware

By analyzing the predicate basic blocks and constraints in the control flow graph, and inferring branch executability, the problems of path explosion and detection efficiency in the prior art are solved, and efficient opaque predicate detection is achieved.

CN117407876BActive Publication Date: 2024-11-19CHANGSHU INSTITUTE OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311689087.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-11
Publication Date
2024-11-19
Estimated Expiration
2043-12-11

AI Technical Summary

Technical Problem

Existing opaque predicate detection methods are prone to cause path explosion problems during path exploration, resulting in inefficient detection, and rely on user interaction or relies on backtracking boundary value selection, and accuracy depends on multiple factors.

Method used

By obtaining the predicate basic blocks and their constraints for jumping branches in the control flow diagram of the program, using symbolic execution technology to analyze these constraints to infer the executability of the branch and judge redundancy, thereby avoiding path exploration and improving detection efficiency.

Benefits of technology

It effectively avoids the problem of path explosion, reduces the complexity and overhead of path exploration, improves the efficiency of opaque predicate detection, and can accurately judge whether there are redundant branches and opaque predicates in the program.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117407876B_ABST
    Figure CN117407876B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system and storage medium for detecting opaque predicates in malware, including: obtaining a predicate basic block in a control flow graph of a program; obtaining the constraint conditions of each jump branch in the predicate basic block; analyzing the constraint conditions to infer the executableness of the branch, if there is a solution that satisfies the condition, then the branch is judged to be executable, if there is no solution that satisfies the condition, then the branch is judged to be not executable and is a redundant branch; if there is a redundant branch, then the program is judged to have an opaque predicate, otherwise, then the program is judged to have no opaque predicate. The method does not perform path exploration to verify the accessibility of the path, but uses the constraint conditions that must be satisfied by the branch jump to represent the execution selection logic of the predicate basic block, infers the executableness of the branch by analyzing the constraint conditions, judges the redundancy of the predicate basic block, and completes the opacity decision of the predicate basic block, which can effectively avoid the path explosion problem and improve the efficiency of opaque predicate detection.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of software detection, and the present invention relates to a method, system and storage medium for detecting opaque predicates in malware. Background Art

[0002] Opaque predicate obfuscation is mainly used to protect the key code of the program to resist reverse analysis and prevent code theft. However, more and more malware are now using opaque predicate obfuscation technology to hinder security analysis in order to evade security detection or hide their malicious code. For malware that uses opaque predicate obfuscation technology, it becomes more difficult to analyze it because it is protected by opaque predicate obfuscation. Therefore, how to effectively detect opaque predicates in malware and restore the real execution logic of the program has become an important research field.

[0003] In programming, conditional statements are used to select execution of certain code blocks or jump to different code blocks based on specific conditions, and each conditional branch has the possibility of being executed. However, opaque predicates construct branch structures by using tautologies or contradictions as conditional statements, thereby introducing redundant branches to complicate the control flow graph. In actual execution, these redundant branches are never executed, and the instructions in the basic blocks they connect are invalid instructions, just to increase the difficulty of understanding the program and confuse the control flow.

[0004] Opaque predicate detection is a challenging task. Researchers are committed to developing effective techniques and algorithms to reveal the true meaning of opaque predicates and restore the execution logic of the program. Currently, opaque predicate detection methods based on symbolic execution mainly include forward symbolic execution and backward symbolic execution.

[0005] LOOP (Logic Oriented Opaque Predicate) is a logic-oriented opaque predicate detection and DROP (Drop Removes Opaque Predicate) is a representative method for opaque predicate detection based on forward symbolic execution. LOOP is a logic-oriented opaque predicate detection tool proposed by Jiang Ming et al. It constructs first-order logic formulas on the execution path through dynamic taint analysis and symbolic execution, and solves these formulas through constraint solvers to detect opaque predicates. However, when the taint information fails to be correctly propagated to all relevant execution paths or variables, it may cause logical errors or incompleteness of the execution path, thereby triggering false positives. DROP is an opaque predicate detection and removal tool based on the IDA Pro (Interactive Disassembler Professional) interactive decompilation environment proposed by Thomas Rinsma. It can automatically detect and remove false control flows constructed by opaque predicates visually. DROP draws on the idea of ​​LOOP in its core method and provides interactive input for analysts to help analyze the internal logic of the program. By providing specific types of constraints, DROP can detect stronger types of control flow obfuscation. However, this method relies on user interaction, is not well suited for large-scale program analysis, and some of the flaws in the LOOP method still exist in DROP.

[0006] Another opaque predicate detection method based on backward symbolic execution was proposed by Sébastien Bardin and others, called Backward-bounded Dynamic Symbolic Execution (BB-DSE). BB-DSE uses computable bounded reasoning to collect all the constraints of the path within the specified number of steps of backward backtracking, and completes the detection of opaque predicates by analyzing these constraints. However, the accuracy of BB-DSE depends on the selection of the backtracking boundary value. The accuracy of predicate detection can only be guaranteed when enough constraints are collected during the backtracking process.

[0007] Whether it is an opaque predicate detection method based on forward symbolic execution or backward symbolic execution, path exploration is a key step in these detection methods. However, when the number of predicate basic blocks increases, the execution path will grow exponentially, which will bring about the problem of path explosion. Summary of the invention

[0008] The purpose of the present invention is to provide a method, system and storage medium for detecting opaque predicates in malware. The method does not perform path exploration to verify the reachability of the path, but uses the constraints that must be satisfied by branch jumps to represent the execution selection logic of the predicate basic block. The executableness of the branch is inferred by analyzing these constraints, and the redundancy of the predicate basic block is judged accordingly to complete the opacity decision of the predicate basic block, which can effectively avoid the path explosion problem and improve the efficiency of opaque predicate detection.

[0009] The technical solution to achieve the purpose of the present invention is:

[0010] A method for detecting opaque predicates in malware comprises the following steps:

[0011] S01: Get the predicate basic block in the control flow graph of the program;

[0012] S02: Obtain the constraints of each jump branch in the predicate basic block;

[0013] S03: Analyze the executableness of the constraint condition inference branch. If there is a solution that satisfies the condition, the branch is judged to be executable. If there is no solution that satisfies the condition, the branch is judged to be not executable and is judged to be a redundant branch.

[0014] S04: If there is a redundant branch, it is determined that there is an opaque predicate in the program; otherwise, it is determined that there is no opaque predicate in the program.

[0015] In the preferred technical solution, the method for determining the predicate basic block in step S01 includes:

[0016] S11: Obtain basic blocks using control flow graph;

[0017] S12: filtering the system method basic blocks in the control flow graph to obtain a basic block set containing only user codes;

[0018] S13: judging a basic block in the user code basic block set, if a basic block has multiple successor basic blocks in the control flow graph and the basic block contains a conditional jump instruction, and the out-degree on the control flow graph is greater than or equal to 2, then it is judged to be a predicate basic block.

[0019] In a preferred technical solution, the method of obtaining the constraint conditions of each jump branch in the predicate basic block in step S02 includes:

[0020] Obtain the successor basic block of the predicate basic block. The algorithm traverses each successor basic block and obtains the corresponding constraint conditions from the constraint conditions to express the branch jump logic.

[0021] In the preferred technical solution, before step S03, the following steps are further included:

[0022] The constraints are simplified and converted into the SMT-LIB (Satisfiability ModuloTheories Library) standard input format. The simplification operation includes one or more of algebraic simplification, logical simplification, equivalent transformation, and symbolic simplification.

[0023] In a preferred technical solution, the method for analyzing the executability of the constraint condition inference branch in step S03 includes:

[0024] Use the constraint solver to solve the constraints and obtain their satisfiability results to query whether there is a solution that satisfies the constraints.

[0025] The present invention also discloses a system for detecting opaque predicates in malware, comprising:

[0026] A predicate basic block acquisition module acquires the predicate basic blocks in the control flow graph of the program;

[0027] The constraint condition acquisition module obtains the constraint conditions of each jump branch in the predicate basic block;

[0028] The branch redundancy analysis module analyzes the executableness of the branch inferred by the constraint conditions. If there is a solution that satisfies the conditions, the branch is judged to be executable. If there is no solution that satisfies the conditions, the branch is judged to be not executable and is judged to be a redundant branch.

[0029] The opaque predicate detection module determines whether an opaque predicate exists in the program if there are redundant branches, otherwise, it determines whether an opaque predicate does not exist in the program.

[0030] In a preferred technical solution, the method for obtaining the constraint conditions of each jump branch in the predicate basic block in the constraint condition acquisition module includes:

[0031] Obtain the successor basic block of the predicate basic block. The algorithm traverses each successor basic block and obtains the corresponding constraint conditions from the constraint conditions to express the branch jump logic.

[0032] In the preferred technical solution, the branch redundancy analysis module also includes:

[0033] The simplification processing module performs a simplification processing operation on the constraint conditions and converts them into the SMT-LIB standard input format. The simplification processing operation includes one or more of algebraic simplification, logical simplification, equivalent transformation, and symbolic simplification.

[0034] In a preferred technical solution, the method for analyzing the executableness of the constraint condition inference branch in the branch redundancy analysis module includes:

[0035] Use the constraint solver to solve the constraints and obtain their satisfiability results to query whether there is a solution that satisfies the constraints.

[0036] The present invention further discloses a computer storage medium on which a computer program is stored. When the computer program is executed, the above-mentioned method for detecting opaque predicates in malware is implemented.

[0037] Compared with the prior art, the present invention has the following significant advantages:

[0038] The optimization algorithm uses the predicate basic block as the analysis unit and screens the predicate basic block based on the control flow graph. The symbolic value of the variable is tracked by symbolic execution technology to obtain the constraints of each jump branch in the predicate basic block. Unlike the method based on symbolic execution, the algorithm of the invention does not perform path exploration to verify the accessibility of the path, but uses the constraints that must be satisfied by the branch jump to represent the execution selection logic of the predicate basic block, and infers the executableness of the branch by analyzing these constraints. The redundancy of the predicate basic block is judged accordingly to complete the opacity decision of the predicate basic block. The path explosion problem can be effectively avoided, and the detection of opaque predicates can be completed. In addition, the algorithm of the present invention can also reduce the complexity and overhead of path exploration and improve the efficiency of opaque predicate detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method for detecting opaque predicates in malware in this embodiment;

[0040] Figure 2 4 is a functional block diagram of the opaque predicate detection system in malware of this embodiment. DETAILED DESCRIPTION

[0041] The principle of the present invention is: take the predicate basic block as the analysis unit, and screen the predicate basic block based on the control flow graph. Track the symbolic value of the variable through symbolic execution technology to obtain the constraints of each jump branch in the predicate basic block. Different from the method based on symbolic execution, the algorithm of the present invention does not perform path exploration to verify the reachability of the path, but uses the constraints that must be satisfied by the branch jump to represent the execution selection logic of the predicate basic block, and infers the executableness of the branch by analyzing these constraints. Based on this, the redundancy of the predicate basic block is judged to complete the opacity decision of the predicate basic block. Through the algorithm of the present invention, the path explosion problem can be effectively avoided, and the detection of opaque predicates can be completed. In addition, the algorithm of the present invention can also reduce the complexity and overhead of path exploration and improve the efficiency of opaque predicate detection.

[0042] Embodiment 1:

[0043] like Figure 1As shown, a method for detecting opaque predicates in malware includes the following steps:

[0044] S01: Get the predicate basic block in the control flow graph of the program;

[0045] S02: Obtain the constraints of each jump branch in the predicate basic block;

[0046] S03: Analyze the executableness of the constraint condition inference branch. If there is a solution that satisfies the condition, the branch is judged to be executable. If there is no solution that satisfies the condition, the branch is judged to be not executable and is judged to be a redundant branch.

[0047] S04: If there is a redundant branch, it is determined that there is an opaque predicate in the program; otherwise, it is determined that there is no opaque predicate in the program.

[0048] In a preferred embodiment, the method for determining the predicate basic block in step S01 includes:

[0049] S11: Obtain basic blocks using control flow graph;

[0050] S12: filtering the system method basic blocks in the control flow graph to obtain a basic block set containing only user codes;

[0051] S13: judging a basic block in the user code basic block set, if a basic block has multiple successor basic blocks in the control flow graph and the basic block contains a conditional jump instruction, and the out-degree on the control flow graph is greater than or equal to 2, then it is judged to be a predicate basic block.

[0052] Filter the system method basic blocks in the control flow graph basic blocks; determine whether the control flow graph basic blocks have been visited, if the basic blocks have not been visited, mark the basic blocks as visited; determine whether there are function calls in the control flow graph basic blocks, if there are function calls in the basic blocks, jump to the basic blocks connected after the call returns; according to the characteristics that the basic blocks have multiple successors and the out-degree on the control flow graph is greater than or equal to 2, and the conditions that the basic blocks contain conditional jump instructions. Not only can the predicate basic blocks be extracted quickly and accurately, but also the influence of system methods and function call instructions on the predicate basic block screening can be eliminated.

[0053] In a preferred embodiment, the method of obtaining the constraint conditions of each jump branch in the predicate basic block in step S02 includes:

[0054] Obtain the successor basic block of the predicate basic block. The algorithm traverses each successor basic block and obtains the corresponding constraint conditions from the constraint conditions to express the branch jump logic.

[0055] Specifically, the symbolic value of the variable is tracked through symbolic execution technology to obtain the constraint conditions of each jump branch in the predicate basic block.

[0056] In a preferred embodiment, before step S03, the following steps are further included:

[0057] The constraints are simplified and converted into the standard input format of the Satisfiability ModuloTheories Library (SMT-LIB). The simplification operation includes one or more of algebraic simplification, logical simplification, equivalent transformation, and symbolic simplification.

[0058] In a preferred embodiment, the method of analyzing the executableness of the constraint condition inference branch in step S03 includes:

[0059] Use the constraint solver to solve the constraints and obtain their satisfiability results to query whether there is a solution that satisfies the constraints.

[0060] The user uses a specific language to describe the constraints of the object (variable), and the constraint solver will try to solve the value of each variable that can satisfy all the constraints. Existing constraint solvers include SMT (Satisfiability Modulo Theories) solver, Z3 solver, and so on.

[0061] In another embodiment, a computer storage medium stores a computer program, which implements the above-mentioned method for detecting opaque predicates in malware when executed.

[0062] In another embodiment, Figure 2 A system for detecting opaque predicates in malware is shown, comprising:

[0063] A predicate basic block acquisition module 10 acquires a predicate basic block in a control flow graph of a program;

[0064] A constraint condition acquisition module 20 is used to acquire the constraint conditions of each jump branch in the predicate basic block;

[0065] The branch redundancy analysis module 40 analyzes the executableness of the branch inferred by the constraint condition. If there is a solution that satisfies the condition, the branch is judged to be executable. If there is no solution that satisfies the condition, the branch is judged to be not executable and is judged to be a redundant branch.

[0066] The opaque predicate detection module 50 determines whether an opaque predicate exists in the program if a redundant branch exists, and vice versa, determines whether an opaque predicate does not exist in the program.

[0067] It also includes a simplification processing module 30, which performs simplification processing operations on the constraint conditions and converts them into the SMT-LIB standard input format. The simplification processing operations include one or more of algebraic simplification, logical simplification, equivalent transformation, and symbolic simplification.

[0068] Specifically, the workflow of the opaque predicate detection system in malware is described below by taking a preferred embodiment as an example:

[0069] The specific idea of ​​the present invention is: to analyze in units of predicate basic blocks, and to use constraints to represent the execution of branch selection. Then, the constraint solver is used to verify the satisfiability of the corresponding constraints to query whether there is a solution that satisfies the constraints. If there is a solution that satisfies the conditions, it means that the branch is executable in actual execution. Correspondingly, if there is no solution that satisfies the conditions, it means that the branch is not executable in actual execution, that is, it is a redundant branch. In order to further improve the solving efficiency, these constraints are subjected to constraint simplification processing. The simplified constraints will be converted into the standard input form of STM-LIB. This method of branch redundancy analysis based on predicate basic blocks can effectively discover redundant branches in predicate basic blocks, thereby realizing the detection of opacity of predicate basic blocks. The specific algorithm construction is as follows:

[0070] Input: Control flow graph G = (V, E) (G: Graph, graph; V: Vertex, vertex; E: Edge, edge)

[0071] Output: Opaque predicate

[0072] Step 1: Get all predicate basic blocks in the control flow graph G and store them in the list predicateBlocks (predicate blocks).

[0073] Step 2: Get all predicate basic block constraints in the control flow graph G and store them in the list constraints.

[0074] Step 3: Get the successor basic blocks of the predicate basic block and store them in the list successors (successor basic blocks).

[0075] Step 4: Create a redundant branch list redundantBranch.

[0076] Step 5: Create a constraint solver satisfiability verification list.

[0077] Step 6: Traverse each successor basic block successor in the successor basic block list successors and execute steps 7 to 15.

[0078] Step 7: Get the constraint corresponding to the predicateBlock jump successor.

[0079] Step 8: Perform algebraic simplification, logical simplification, equivalent transformation and other simplification operations on the constraint condition to obtain the simplified constraint condition simplifyConstraint (simplified constraint condition).

[0080] Step 9: Convert the simplified constraint condition simplifyConstraint to the SMT-LIB standard input format and store it in smtInput (satisfiability modulo theory input).

[0081] Step 10: Use the constraint solver to solve smtInput and verify its satisfiability.

[0082] Step 11: Store the satisfiability verification result isSat (is satisfied) of smtInput in the list satisfactory.

[0083] Step 12: Determine the smtInput satisfiability verification result isSat. If the smtInput satisfiability verification isSat is true, execute step 13; otherwise, execute steps 14 and 15.

[0084] Step 13: Mark successor as a valid branch.

[0085] Step 14: Mark successor as a redundant branch.

[0086] Step 15: Add (predicateBlock, successor) to redundantBranch

[0087] Step 16: Determine whether the successor traversal is complete. If the traversal is complete, execute step 17. If the traversal is not complete, execute step 6.

[0088] Step 17: Determine whether the redundant branch list redundantBranch is empty. If it is not empty, execute step 18, otherwise execute step 19.

[0089] Step 18: Output True indicates that an opaque predicate and corresponding redundantBranch information are detected in the program, and the corresponding opaque predicate basic block is displayed in redundantBranch.

[0090] Step 19: The output False indicates that it is detected that there is no opaque predicate in the program.

[0091] The algorithm takes the program control flow graph as input. First, it obtains the predicate basic block in the control flow graph, then obtains the constraints in the control flow graph, and then obtains the list of successor basic blocks of the predicate basic block, and creates a redundant branch list. Next, the algorithm traverses each successor basic block and obtains the corresponding constraints from the constraints to express the branch jump logic. Then, the constraints are simplified by algebraic simplification, logical simplification, equivalent transformation, symbolic simplification, etc., and converted into the SMT-LIB standard input format. Then, the algorithm uses the constraint solver to solve the constraints and obtain their satisfiability results. If the constraints are satisfiable, the branch is marked as executable; if the constraints are not satisfiable, the branch is marked as not executable and added to the redundant branch list. Finally, whether the predicate basic block is opaque is determined based on whether the redundant branch list is empty. If there are redundant branches, the predicate basic block is opaque, and True and the redundant branch list are output. The algorithm uses the constraints of the predicate basic block to jump to the successor basic block for satisfiability analysis to determine the branch redundancy. And by using the feature of whether there are redundant branches, it is possible to determine whether the predicate basic block is opaque, thereby realizing the detection of opaque predicates in the program.

[0092] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principles of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. A method for detecting opaque predicates in malware, characterized in that: The following steps are involved: S01: Get the predicate basic block in the control flow graph of the program; The judgment method of the predicate basic block includes: S11: Obtain basic blocks using control flow graph; S12: filtering the system method basic blocks in the control flow graph to obtain a basic block set containing only user codes; S13: judging a basic block in the user code basic block set, if a basic block has multiple successor basic blocks in the control flow graph and the basic block contains a conditional jump instruction, and the out-degree on the control flow graph is greater than or equal to 2, then it is judged to be a predicate basic block; S02: Obtaining the constraint conditions of each jump branch in the predicate basic block; the method for obtaining the constraint conditions of each jump branch in the predicate basic block includes: obtaining the successor basic block of the predicate basic block, traversing each successor basic block by algorithm, and obtaining the corresponding constraint conditions from the constraint conditions to express the branch jump logic; Performing a simplification operation on the constraint conditions and converting them into the SMT-LIB standard input format, wherein the simplification operation includes one or more of algebraic simplification, logical simplification, equivalent transformation, and symbolic simplification; S03: Analyze the executableness of the constraint inference branch. If there is a solution that satisfies the condition, the branch is judged to be executable. If there is no solution that satisfies the condition, the branch is judged to be not executable and is judged to be a redundant branch. The method for analyzing the executableness of the constraint inference branch includes: Use the constraint solver to solve the constraints and obtain their satisfiability results to check whether there is a solution that satisfies the constraints; S04: If there is a redundant branch, it is determined that there is an opaque predicate in the program; otherwise, it is determined that there is no opaque predicate in the program.

2. A system for detecting opaque predicates in malware, characterized in that: include: A predicate basic block acquisition module acquires the predicate basic blocks in the control flow graph of the program; The judgment method of the predicate basic block includes: S11: Obtain basic blocks using control flow graph; S12: filtering the system method basic blocks in the control flow graph to obtain a basic block set containing only user codes; S13: judging a basic block in the user code basic block set, if a basic block has multiple successor basic blocks in the control flow graph and the basic block contains a conditional jump instruction, and the out-degree on the control flow graph is greater than or equal to 2, then it is judged to be a predicate basic block; A constraint condition acquisition module is used to acquire the constraint conditions of each jump branch in the predicate basic block. The method for acquiring the constraint conditions of each jump branch in the predicate basic block includes: acquiring the successor basic block of the predicate basic block, traversing each successor basic block by algorithm, and acquiring the corresponding constraint condition expression branch jump logic from the constraint conditions; The branch redundancy analysis module analyzes the executableness of the branch inferred by the constraint condition. If there is a solution that satisfies the condition, the branch is judged to be executable. If there is no solution that satisfies the condition, the branch is judged to be not executable and is judged to be a redundant branch. The method for analyzing the executableness of the branch inferred by the constraint condition includes: Use the constraint solver to solve the constraints and obtain their satisfiability results to check whether there is a solution that satisfies the constraints; The opaque predicate detection module determines whether an opaque predicate exists in the program if there are redundant branches, otherwise, it determines whether an opaque predicate does not exist in the program.

3. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for detecting opaque predicates in malware according to claim 1 is implemented.