A Deep Learning-Based Optimized Code Decompilation Method and System

Abstract

This invention relates to a method and system for decompiling optimized code based on deep learning. The method includes the following steps: obtaining a low-level intermediate language (LIR) and a high-level intermediate language (HIR) using a low-level programming language (LPL) and a high-level programming language (HPL) as training datasets; training a deep learning model using the training datasets to learn the mapping rules between LIRs and HIRs; using the trained deep learning model to translate the LIR of the LPL to be decompiled into the HIR; performing data flow recovery and control structure recovery on the HIR obtained by the deep learning model to generate HPL code; and using a similarity matching algorithm to find source code similar to the generated HPL code, transferring the semantic information from the source code to the generated HPL code. This invention can automatically convert LPL to HPL and has high accuracy for both optimized and non-optimized binary decompilation.

Description

Technical Field

[0001] This invention belongs to the fields of software technology and information security technology, and relates to software reverse engineering technology, specifically to an optimized code decompilation method and system based on deep learning. Background Technology

[0002] Compilation typically translates computer programs written in high-level languages ​​(such as C / C++) into low-level languages ​​(such as assembly language or machine code), i.e., those that run on the target CPU (such as x86, ARM). Due to the differences between the two PLs (programming languages), unnecessary information (e.g., variable / function names) is usually removed. Furthermore, optimization techniques aim to minimize or maximize certain properties of the executable program during this process, such as reducing execution time, memory usage, and storage size. Note that the optimized target program should produce the same result as the unoptimized original program. Optimization typically has several levels (e.g., from O0 to O3 in the GCC compiler). Some optimization strategies perform machine-dependent optimizations, further refining the optimization by leveraging the characteristics of the target machine language.

[0003] Decompilation is a technique that converts a compiled executable program or LPL (Low-Level Programming Language) into a functionally equivalent and easily readable HPL (High-Level Programming Language). The decompiled code is often used to understand hard-to-read LPLs, discover program vulnerabilities, analyze malicious behavior, or perform clone detection. It can also help port programs without source code to different architectures (e.g., different hardware or operating systems). Existing decompilers mostly use rule-based pattern matching to elevate LPLs to an intermediate representation, which is crucial for decompilers to recover HPLs. However, the known instruction sets are highly diverse, often requiring analysts to expend significant effort (e.g., years of development) to summarize the rules. Worse still, frequent changes in decompilation rules due to variations in compilation / optimization strategies mean analysts must constantly modify the rules they previously generated. This process is both time-consuming and error-prone, as rules are likely to conflict (especially many older rules).

[0004] Machine translation is a technique in natural language processing that translates one language into another, such as English to French. An encoder encodes the input language into a specific vector, and a decoder then decodes this vector to obtain the output language. Models can learn mapping rules between input and output sentences and words through large amounts of data. Machine translation models can automatically learn mapping rules between LPL and HPL, achieving a decompilation-like effect. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for decompiling optimized code based on a deep learning model. This invention can automatically convert LPL to HPL and exhibits high accuracy in decompiling both optimized and non-optimized binary code. Compared to state-of-the-art decompilers such as Hex-Rays in IDA Pro, this invention provides more accurate decompilation of optimized code with better readability.

[0006] The technical solution adopted in this invention is as follows:

[0007] A deep learning-based optimized code decompilation method includes the following steps:

[0008] Use the low-level programming language LPL and the high-level programming language HPL to obtain the low-level intermediate language LIR and the high-level intermediate language HIR as training datasets;

[0009] A deep learning model is trained using the training dataset to learn the mapping rules between LIR and HIR. The trained deep learning model is then used to translate the LIR of the LPL to be decompiled into HIR.

[0010] Data flow recovery and control structure recovery are performed on the HIR obtained from the deep learning model to generate HPL code;

[0011] A similarity matching algorithm is used to find source code that is similar to the generated HPL code, and the semantic information in the source code is transferred to the generated HPL code.

[0012] Furthermore, the step of obtaining a low-level intermediate language (LIR) and a high-level intermediate language (HIR) using the low-level programming language LPL and the high-level programming language HPL as a training dataset includes:

[0013] Obtain the function-level LIR and HIR from LPL and HPL;

[0014] Data dependency analysis was performed on function-level LIRs and HIRs to obtain the data dependency graph DDG for the LIRs. l Data dependency graph DDG with HIR h ;

[0015] DDG l and DDG h First, the basic blocks of the CFG are matched, and then the sub-regions of the data dependency graph are matched within the basic blocks to obtain more fine-grained LIR and HIR code fragment pairs.

[0016] A program that randomly generates high-level language statements is used to generate various types of statements in a certain proportion. These statements are then compiled at various compilation optimization levels O0-O3 to generate LIR and HIR. A large number of LIR and HIR code snippets are deduplicated to form a training dataset.

[0017] Furthermore, the deep learning model is a GGS-NN model, which uses the data dependency graph of LIR as the input of the model and the sequence of HIR as the output of the model.

[0018] Furthermore, the data stream recovery includes: for each pair of LIR and HIR, matching operands according to the type of instruction, the order of instructions, and the number of source operands and destination operands of the instruction, and handling special cases according to rules.

[0019] Furthermore, the operand matching process includes: first, classifying instructions according to the number of operands; and when encountering a normal three-operand type instruction, assigning the operands in the LIR to the HIR in sequence.

[0020] Furthermore, the rule-based processing of special cases includes:

[0021] For multi-operand type instructions, according to the instruction syntax rules, intermediate variables are used to construct a template to fill the operands in the LIR into the HIR;

[0022] For instructions of function call type, the same template is built to populate the parameter list and return value in the HIR;

[0023] For instructions involving optimization, a template is built using optimization rules, and the operands in the HIR are calculated and filled based on the operands in the LIR.

[0024] Furthermore, the step of using a similarity matching algorithm to find source code similar to the generated HPL code, and transferring the semantic information from the source code to the generated HPL code, includes:

[0025] Use a similarity matching algorithm to find functions P in an open-source project that are similar to the generated HPL code. src The function P to be recovered des Encode it as a vector;

[0026] For function P des sum function P src Construct abstract syntax trees separately, and then extract their contextual features for P. des The variables in P src Find variables with similar characteristics in P and transfer their semantic information to P. desAmong the corresponding variables, P, which ultimately recovers the semantic information, is obtained. des .

[0027] A deep learning-based optimized code decompilation system employing the above method includes:

[0028] The dataset building module is used to obtain the low-level intermediate language (LIR) and high-level intermediate language (HIR) using the low-level programming language (LPL) and the high-level programming language (HPL) as training datasets.

[0029] The deep model translation module is used to train a deep learning model using the training dataset, learn the mapping rules between LIR and HIR, and use the trained deep learning model to translate the LIR of the LPL to be decompiled into HIR.

[0030] The HPL code generation module is used to perform data flow recovery and control structure recovery on the HIR obtained from the deep learning model and generate HPL code.

[0031] The semantic information recovery module is used to find source code similar to the generated HPL code using a similarity matching algorithm, and to transfer the semantic information in the source code to the generated HPL code.

[0032] Compared with the prior art, the positive effects of the present invention are as follows:

[0033] This invention also has high accuracy for removing symbol information and optimizing binary data.

[0034] This invention can add semantic information to the recovered source code, increasing the readability of the code. Attached Figure Description

[0035] Figure 1 This is the overall flowchart of the present invention. Wherein, CFG represents the Control Flow Graph; DDG represents the Data-Dependence Graph; SEQ represents the Sequence; LIR represents Low-level Intermediate Representation; and HIR represents High-level Intermediate Representation.

[0036] Figure 2 This diagram illustrates the division of translation units. TU stands for Translation Unit.

[0037] Figure 3 This is a diagram of the model structure. Here, mul represents the multiplication instruction.

[0038] Figure 4 This is a schematic diagram for semantic information recovery. Detailed Implementation

[0039] The present invention will be further described below with reference to specific embodiments and accompanying drawings.

[0040] 1. The overall process of the deep learning-based optimized code decompilation method of the present invention is as follows: Figure 1 As shown, it includes the following key points;

[0041] 1.1. Dataset Construction: Obtain low-level intermediate language (LIR) and high-level intermediate language (HIR) code pairs from LPL and HPL, and perform data dependency analysis to obtain graph-structured LIR and sequence-structured HIR.

[0042] 1.2. Deep Model Translation: Select a suitable deep learning model for code translation, train the deep learning model using the constructed dataset, and learn the mapping rules between LIR and HIR. Using the trained deep learning model, translate the LIR of the LPL to be decompiled into HIR.

[0043] 1.3. Generate HPL code: Based on the HIR, restore the data flow and control structure, and generate HPL code in a C-like language.

[0044] 1.4. Semantic Information Recovery: A similarity matching algorithm is used to find source code that is similar to the generated HPL code, and the semantic information in the source code is transferred to the generated HPL code.

[0045] 2. As described in point 1.1, dataset construction involves the following steps:

[0046] 2.1. Disassembly and CFG Construction: Given a binary file, it is necessary to first divide the file into functions and disassemble it. This embodiment uses the open-source tool Angr to obtain the functions in the binary file and disassemble it to obtain the LPL of each function. Simultaneously, this embodiment utilizes Angr's static analysis techniques to obtain the CFG of each function.

[0047] 2.2. Acquisition of Intermediate Language: Since LPL and HPL are not suitable for direct model learning, LPL was improved. First, compiler-implemented information not reflected in the code was added. Then, redundant instructions were removed based on data dependencies, resulting in the final LIR. The CFG of LPL can be mapped to the CFG of LIR. For HPL, an optimized high-level intermediate language (HIR) was obtained by compiling with a compiler. Data dependency analysis was performed on the HIR code to remove redundant instructions, resulting in the final HIR.

[0048] 2.3. Division of Translation Units: The function-level LIR and HIR are split into smaller corresponding units to facilitate better model learning. The method for splitting the function-level LIR and HIR into smaller corresponding units is to perform data dependency analysis on the LIR and HIR to obtain the LIR data dependency graph (DDG). l Data dependency graph DDG with HIR h ; Regarding DDG l and DDG h First, the basic blocks of the CFG are matched, and then the sub-regions of the data dependency graph are matched within the basic blocks, which can yield more fine-grained LIR and HIR code fragment pairs.

[0049] Figure 2 This is a diagram illustrating the division of translation units. Wherein:

[0050] (a) The graph is a data dependency graph of LIR DDG l The first stage of partitioning. This stage partitions the graph based on the out-degree of the nodes. Nodes with an out-degree greater than or equal to 2 serve as boundaries, dividing the graph into several non-overlapping regions. Each node represents an instruction, and the content of each node in the graph consists of the corresponding AArch64 instruction and its position number in the basic block. For example, bl0 represents the first bl instruction in this code segment. Edges represent the data transfer relationship between two instructions. For example, x0_5 indicates that the variable x0_5 calculated by bl0 is passed to the add3 instruction. TU represents the unit obtained from this stage of partitioning, pointing to the dashed box in the graph. The dashed arrows represent the data transfer relationship between units established based on the implementation data dependencies.

[0051] Figure (b) shows the result of merging the cells obtained in (a). The merging method is that if a cell only has a dashed outward edge, then it is merged with the cell it points to. The 5 cells in (a) are finally merged into the 3 cells TU1, TU2 and TU3 in (b).

[0052] (c) shows the cell partitioning of the data dependency graph for the corresponding HIR code. The partitioning method is the same as in (a) and (b), ultimately resulting in three HIR cells: TU1, TU2, and TU3. Nodes and edges represent instruction and data dependencies, respectively. The corresponding HIR code is shown within the solid-line box.

[0053] 2.4. Dataset Construction: Using a program that randomly generates high-level language statements of various types according to a certain ratio, and compiling them at various compilation optimization levels O0-O3 to generate LIR and HIR. A large number of small code snippet pairs are obtained using the methods in 2.1 to 2.3, and after deduplication, a training dataset is formed.

[0054] 3. As mentioned in point 1.2, the deep model translation part selects the GGS-NN (Gated Graph Sequence Neural Network) model as the model for translating LIRs. The data dependency graph of the LIR is used as the input of the model. The sequence of HIRs is used as the output of the model.

[0055] Figure 3 Here is the model structure diagram. First, the LIR's DDG is input, then the nodes are initialized, and then GG-NN (Gated Graph Neural Network) is used to encode the LIR's DDG. Then, the decoder part uses an LSTM network with Luong attention mechanism to decode, outputting the already predicted partial sequence, and outputting the next word to be predicted.

[0056] 4. As shown in point 1.3, generating the HPL code involves the following steps:

[0057] 4.1. Data Stream Recovery: After obtaining the instruction pair, this invention designs a data stream-based method to recover HIR operands. The data stream-based method refers to: for each LIR and HIR pair, matching operands based on the instruction type, instruction order, and the number of source and destination operands, and handling some special cases based on rules.

[0058] The operand matching process involves first classifying instructions according to the number of operands, and then, when encountering a normal three-operand instruction, assigning the operands in the LIR to the HIR in sequence.

[0059] Specifically, the handling of certain special cases using rules refers to:

[0060] 1) Multi-operand type instructions, such as the madd instruction, typically correspond to multiple instructions in the HIR. Based on the syntax rules of these instructions, intermediate variables are used to construct a template to fill the operands in the LIR into the HIR.

[0061] 2) For instructions of the same type as function calls, the template is also built to fill the parameter list and return value in the LIR into the HIR.

[0062] 3) Instructions involving optimization, such as integer division optimization. A template is built using optimization rules, and the operands in the HIR are calculated and filled based on the operands in the LIR.

[0063] 4.2. Control structure restoration: Restoration is performed using existing control structure restoration techniques.

[0064] 4.3. HPL Generation: The HIR is restored to the HPL through statement rule mapping.

[0065] 5. As shown in point 1.4, the semantic information recovery part includes the following steps:

[0066] 5.1. Use a similarity matching algorithm to find similar functions P in open source projects. src The function P to be recovered des The function P is encoded as a vector and compared with function vectors in the candidate dataset. The candidate dataset consists of application scenarios based on binary files, and similar projects downloaded from the open-source community. The function P that is closest to the target function is selected from the candidate dataset. src .

[0067] 5.2. For function P des sum function P src They construct their abstract syntax trees respectively, and then extract their contextual features, including the position of the statement in the function, the type of the statement, and the operation of the statement. For P des The variables in P src Find variables with similar characteristics in P and transfer their semantic information to P. des In the corresponding variables.

[0068] Figure 4 This is a schematic diagram for semantic information recovery. Wherein, S des Represents a binary file, P des P represents the recovered source code. src This represents the source code of functions found in the candidate dataset that are similar to the function to be matched, including the following steps:

[0069] 1) Place S des Decompiling yields the source code P des In the candidate dataset, a similarity matching algorithm was used to match P. des Similar functions P src .

[0070] 2) Standardize the decompiled code, including syntax checking.

[0071] 3) Perform simple optimizations on the restored source code, removing redundant assignment statements, for example... Figure 4 The bottom left integer v1 = a.

[0072] 4) Extract P respectively des and P src Contextual features of variables in the text.

[0073] 5) Traverse P des Each variable in P srcFind variables in the context that have similar characteristics and have not been used before, and assign them to P. des The corresponding variable in the original variable is used as the new variable name, and finally, the semantic information of P is recovered. des .

[0074] Compared with existing deep learning-based decompilation schemes, the technical solution of this invention can better handle optimized code. The unit partitioning scheme proposed in this invention improves performance by 4.1%-71.23% compared with directly using models to translate assembly code and high-level languages ​​or syntax trees.

[0075] In one embodiment of the present invention, the model training process includes:

[0076] 1) Use regular expressions to randomly generate source code files for training.

[0077] 2) Use Clang+LLVM to compile the source code into binary files and obtain the intermediate generated HIR.

[0078] 3) Use Angr to decompile the binary file to obtain the LIR and CFG of each function.

[0079] 4) Using the unit partitioning method proposed in this scheme, HIR and LIR are divided into smaller matching units.

[0080] 5) Use the model structure mentioned in this scheme for training.

[0081] In one embodiment of the present invention, the decompilation process includes:

[0082] 1) Use Angr to decompile the binary file to obtain the LIR and CFG of each function.

[0083] 2) Using the unit partitioning method proposed in this scheme, the LIR is divided into smaller matching units.

[0084] 3) Use the trained model to translate LIR into HIR.

[0085] 4) Use the variable filling method proposed in this scheme to fill the variables in the LIR into the HIR.

[0086] 5) Reorder the small units to form basic blocks, and reassemble the basic blocks into HIR CFGs according to the LIR CFG.

[0087] 6) Use pattern matching to restore the control structure to the IF / WHILE / FOR / SWTICH / GOTO statements in the high-level language.

[0088] 7) Use the defined template to identify the parameters and return value of the function.

[0089] 8) Translate the translated HIR into a high-level language based on the template.

[0090] 9) Combine high-level language statements, parameters, return values, and control structures to form a complete high-level language function, and then decompile it.

[0091] 10) Use the method proposed in this scheme to recover semantic information and perform simple optimizations on the code.

[0092] Based on the same inventive concept, another embodiment of the present invention provides a deep learning-based optimized code decompilation system employing the method of the present invention, comprising:

[0093] The dataset building module is used to obtain the low-level intermediate language (LIR) and high-level intermediate language (HIR) using the low-level programming language (LPL) and the high-level programming language (HPL) as training datasets.

[0094] The deep model translation module is used to train a deep learning model using the training dataset, learn the mapping rules between LIR and HIR, and use the trained deep learning model to translate the LIR of the LPL to be decompiled into HIR.

[0095] The HPL code generation module is used to perform data flow recovery and control structure recovery on the HIR obtained from the deep learning model and generate HPL code.

[0096] The semantic information recovery module is used to find source code similar to the generated HPL code using a similarity matching algorithm, and to transfer the semantic information in the source code to the generated HPL code.

[0097] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0098] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), which stores a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0099] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. A deep learning-based optimized code decompilation method, characterized in that, Includes the following steps: Use the low-level programming language LPL and the high-level programming language HPL to obtain the low-level intermediate language LIR and the high-level intermediate language HIR as training datasets; A deep learning model is trained using the training dataset to learn the mapping rules between LIR and HIR. The trained deep learning model is then used to translate the LIR of the LPL to be decompiled into HIR. Data flow recovery and control structure recovery are performed on the HIR obtained from the deep learning model to generate HPL code; A similarity matching algorithm is used to find source code that is similar to the generated HPL code, and the semantic information in the source code is transferred to the generated HPL code. The step of using a similarity matching algorithm to find source code similar to the generated HPL code, and transferring the semantic information from the source code to the generated HPL code, includes: Use a similarity matching algorithm to find functions P in an open-source project that are similar to the generated HPL code. src This includes: the function P to be recovered des The function is encoded into a vector, and this vector is compared with the function vectors in the candidate dataset. The function P that is closest to the candidate dataset is selected. src ; For function P des sum function P src Construct abstract syntax trees separately, and then extract their contextual features for P. des The variables in P src Find variables with similar characteristics in P, and transfer the semantic information of the variables with similar characteristics to P. des Among the corresponding variables, P, which ultimately recovers the semantic information, is obtained. des .

2. The method according to claim 1, characterized in that, The method of obtaining a low-level intermediate language (LIR) and a high-level intermediate language (HIR) using the low-level programming language LPL and the high-level programming language HPL as training datasets includes: Obtain the function-level LIR and HIR from LPL and HPL; Data dependency analysis was performed on function-level LIRs and HIRs to obtain the data dependency graph DDG for the LIRs. l Data dependency graph DDG with HIR h ; DDG l and DDG h First, the basic blocks of the CFG are matched, and then the sub-regions of the data dependency graph are matched within the basic blocks to obtain more fine-grained LIR and HIR code fragment pairs; where CFG represents the control flow graph. A program that randomly generates high-level language statements is used to generate various types of statements in a certain proportion. These statements are then compiled at various compilation optimization levels O0-O3 to generate LIR and HIR. A large number of LIR and HIR code snippets are deduplicated to form a training dataset.

3. The method according to claim 1, characterized in that, The deep learning model is the GGS-NN model, which uses the LIR data dependency graph as the input of the model and the HIR sequence as the output of the model.

4. The method according to claim 1, characterized in that, The data stream recovery includes: for each pair of LIR and HIR, matching operands according to the instruction type, instruction order, and the number of source and destination operands of the instruction, and performing rule-based processing for special cases.

5. The method according to claim 4, characterized in that, The operand matching process includes: first, classifying instructions according to the number of operands; and when encountering a normal three-operand type instruction, assigning operands from the LIR to the HIR in sequence.

6. The method according to claim 5, characterized in that, The rule-based processing of special cases includes: For multi-operand type instructions, according to the instruction syntax rules, intermediate variables are used to construct a template to fill the operands in the LIR into the HIR; For instructions of function call type, the same template is built to populate the parameter list and return value in the HIR; For instructions involving optimization, a template is built using optimization rules, and the operands in the HIR are calculated and filled based on the operands in the LIR.

7. A deep learning-based optimized code decompilation system employing the method described in any one of claims 1 to 6, characterized in that, include: The dataset building module is used to obtain the low-level intermediate language (LIR) and high-level intermediate language (HIR) using the low-level programming language (LPL) and the high-level programming language (HPL) as training datasets. The deep model translation module is used to train a deep learning model using the training dataset, learn the mapping rules between LIR and HIR, and use the trained deep learning model to translate the LIR of the LPL to be decompiled into HIR. The HPL code generation module is used to perform data flow recovery and control structure recovery on the HIR obtained from the deep learning model and generate HPL code. The semantic information recovery module is used to find source code similar to the generated HPL code using a similarity matching algorithm, and to transfer the semantic information in the source code to the generated HPL code. The step of using a similarity matching algorithm to find source code similar to the generated HPL code, and transferring the semantic information from the source code to the generated HPL code, includes: Use a similarity matching algorithm to find functions P in an open-source project that are similar to the generated HPL code. src This includes: the function P to be recovered des The function is encoded into a vector, and this vector is compared with the function vectors in the candidate dataset. The function P that is closest to the candidate dataset is selected. src ; For function P des sum function P src Construct abstract syntax trees separately, and then extract their contextual features for P. des The variables in P src Find variables with similar characteristics in P, and transfer the semantic information of the variables with similar characteristics to P. des Among the corresponding variables, P, which ultimately recovers the semantic information, is obtained. des .

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 6.

Images