A project-level CPG file generation method, system, device and storage medium

Abstract

The application belongs to the field of CPG file generation, and particularly relates to a project-level CPG file generation method, system, device and storage medium, which comprises the following steps: generating an abstract syntax tree according to a filtered file set, traversing nodes in the abstract syntax tree to generate a scope, and obtaining node relationships; establishing a dependency edge according to the abstract syntax tree and the node relationships; processing a branch structure, a loop structure and a jump structure according to the abstract syntax tree, and obtaining a control flow edge; traversing the abstract syntax tree to establish the dependency edge, otherwise, storing a call block position and call information into an import function table to establish a call edge, and obtaining a single CPG file according to the call edge, the dependency edge, the control flow edge and a basic block; and generating a project-level CPG file according to a file index generated from the abstract syntax tree and the single CPG file. The application can provide structured data support for static analysis of a large-scale Lua project, and can greatly improve the efficiency of the static analysis.

Description

Technical Field

[0001] This invention belongs to the field of CPG file generation, specifically relating to a method, system, device, and storage medium for generating project-level CPG files. Background Technology

[0002] Lua project-level CPG modeling refers to the unified modeling of the AST (Abstract Syntax Tree), control flow graph, and program dependency graph of an entire project consisting of multiple Lua source files. This results in a property graph representation that allows for cross-file queries and associations. Based on this, program understanding, impact analysis, refactoring, and security testing are then performed. The core idea of ​​CPG is to integrate the abstract syntax tree, control flow graph, and program dependency graph into a single property graph. This allows for the expression of complex rules through graph traversal, thereby more efficiently discovering structural problems and vulnerabilities in the program. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a method, system, device and storage medium for generating project-level CPG files, thereby improving the efficiency of static analysis of large-scale Lua projects.

[0004] A method for generating project-level CPG files includes:

[0005] Obtain the Lua script file set, parse and filter the Lua script file set to obtain the filtered file set, which includes multiple Lua script files;

[0006] Based on the selected file set, an abstract syntax tree is generated. The nodes that generate scopes in the abstract syntax tree are traversed. A snapshot of the current local variable table is generated. When entering a scope, a CPG subgraph is generated. When exiting a scope, the upper-level snapshot is restored based on the current local variable table snapshot to obtain the node relationships.

[0007] Based on the abstract syntax tree and node relationships, the left and right values ​​of assignment statement nodes are aligned to establish dependency edges;

[0008] Based on the abstract syntax tree, branch structure processing, loop structure processing, and jump structure processing are performed to obtain the control flow edges;

[0009] Traverse the abstract syntax tree. When a call node is detected, if the called object is a local function or a member function in the local table, copy the CPG subgraph defined by the local function or member function, and establish a function dependency edge between the call block and the function entry block. Otherwise, store the call block position and call information in the import function table, and establish a function call edge. The called object exists in the call node, the call block is generated through the called object, the function entry block is generated through the local function or a member function in the local table, and the function dependency edge represents the jump relationship during program execution.

[0010] Based on the call edges, dependency edges, control flow edges, and the basic blocks generated by filtering the file set, we obtain the CPG file corresponding to each Lua script file;

[0011] Generate a file index based on the abstract syntax tree;

[0012] Generate a project-level CPG file based on the file index and the corresponding CPG file for each Lua script file.

[0013] Optionally, an abstract syntax tree is generated based on the filtered document set, including:

[0014] The filtered file set is parsed to obtain a file directory, which includes a root directory and subdirectories;

[0015] Traverse the root directory and subdirectories, retaining valid directories containing Lua script files;

[0016] The valid directories are numbered according to their hierarchical size to obtain a numbered directory;

[0017] Based on the numbered directories, the target directories are retrieved sequentially to create file tree nodes, and Lua path information is stored in the file tree nodes. If the target directory has no parent directory, the target directory is mounted to the root directory node. If the target directory has a parent directory, the target directory is mounted under the parent directory node, until all numbered directories are accessed. All file tree nodes form a file tree, and the file tree nodes include parent directory nodes and root directory nodes.

[0018] Optionally, the step of aligning the left and right values ​​of assignment statement nodes and establishing dependency edges based on the abstract syntax tree and node relationships includes:

[0019] Based on the abstract syntax tree and node relationships, obtain the assignment node;

[0020] Extract the assignment statement from the assignment node;

[0021] Align the left-hand side of the assignment statement with the expression on the right-hand side of the equals sign to obtain an alignment statement;

[0022] Determine if the left-hand side of an alignment statement is a local variable;

[0023] If the lvalue is a local variable, it is renamed according to the current scope;

[0024] If the lvalue is not a local variable, a CPG basic block is created and the expression on the right side is extracted from the CPG basic block;

[0025] Rename the local variables in the expression on the right, and determine whether the expression on the right is a function call;

[0026] If the expression on the right is a function call, then a dependency edge is established.

[0027] Optionally, the step of processing branching structures, loop structures, and jump structures based on the abstract syntax tree to obtain control flow edges includes:

[0028] Branching structure processing based on the abstract syntax tree includes:

[0029] Create branching control flow blocks and converging basic control flow blocks based on the abstract syntax tree;

[0030] Will The pointer points to the forked block and is initialized. For a linear list, enter the branching block. If a `test` pointer exists, access the subtree of the abstract syntax tree pointing to the branch's condition. Then, create the branch's entry block and save the subtree information of the abstract syntax tree pointed to by the `test` pointer to the basic control flow block. The pointer points to the entry block;

[0031] Access the abstract syntax tree subtree of the branch pointed to by the body pointer, and save the exit control flow block of the generated CPG subgraph to... linear list and The pointer points to the forked control flow block;

[0032] If the orelse pointer is not null, then restore the context and access the next branch pointed to by the orelse pointer;

[0033] If the orelse pointer is null, restore the context and exit the current branch. All basic control flow blocks in the linear list are edged with the converging basic control flow block, and... The pointer points to the convergence basic control flow block;

[0034] Processing loop structures based on the abstract syntax tree includes:

[0035] Create the starting basic control flow block and the ending basic control flow block based on the abstract syntax tree;

[0036] Will The pointer points to the starting basic control flow block, traverses the judgment conditions pointed to by the test pointer, and saves the abstract syntax tree information of the test pointer to the entry block;

[0037] Access the abstract syntax tree subtree pointed to by the body pointer and generate the corresponding CPG subgraph. Add control flow edges to the exit control flow block and the loop exit block of the CPG subgraph.

[0038] Jump structure processing based on the abstract syntax tree includes:

[0039] Traverse the abstract syntax tree. When the goto structure is encountered, create a goto control flow block and check if the target basic control flow block corresponding to the label exists in the hash table. If it exists, establish a control flow dependency edge between the target basic control flow block and the basic control flow block corresponding to the label. If it does not exist, store the goto control flow block in the linear list corresponding to the label in the hash table.

[0040] When traversing the label structure, create a label control flow block, and establish control flow dependency edges between all goto control flow blocks in the linear list corresponding to the label in the hash table and the label control flow block. Finally, store the label control flow block in the hash table and establish a jump structure.

[0041] Based on the abstract syntax tree, branching structure processing, loop structure processing, and jump structure processing are performed to obtain control flow edges.

[0042] Optionally, generating a project-level CPG file based on the file index and the corresponding CPG file for each Lua script file includes:

[0043] An adjacency matrix is ​​built based on the file index, and a dependency dominance tree is built based on the adjacency matrix.

[0044] Based on the dependency dominance tree, the global topological relationships are obtained;

[0045] Based on the global topology, a project-level CPG file is generated.

[0046] Optionally, obtaining the global topological relationship based on the dependency dominance tree includes:

[0047] Based on the file index, the dependent directories are obtained;

[0048] Based on the dependency directory, determine whether the current dependency exists in the current directory;

[0049] If the current dependency does not exist in the current directory, then check if the current dependency exists in the recursive subdirectory;

[0050] If the current dependency does not exist in the recursive subdirectory, then determine whether the global variable has been searched;

[0051] If global variables have been searched, determine whether it is possible to revert to the parent directory;

[0052] If it is not possible to return to the parent directory, the dependency is a C package or is missing, and the topological relationship of the root directory is obtained;

[0053] Get the topological relationship of CPG subdirectories;

[0054] Based on the topological relationships of the root directory and the CPG subdirectories, the global topological relationships are obtained.

[0055] Optionally, obtaining the topological relationship of CPG subdirectories includes:

[0056] If the current dependency exists in the current directory, or in a recursive subdirectory, or if the global variable has not been searched and the dependency exists in the global variable directory, then save the dependency, establish edges in the adjacency matrix, and obtain the topological relationship of the CPG subdirectories.

[0057] A project-level CPG file generation system, comprising:

[0058] The acquisition module is used to acquire a set of Lua script files, parse and filter the set of Lua script files, create an abstract syntax tree, and filter the set of files to include multiple Lua script files.

[0059] The traversal module is used to generate an abstract syntax tree based on the filtered file set, traverse the nodes in the abstract syntax tree that generate the scope, generate a snapshot of the current local variable table, generate a CPG subgraph when entering the scope, and restore the upper-level snapshot based on the current local variable table snapshot when exiting the scope to obtain the node relationship;

[0060] The dependency edge building module is used to align the left and right values ​​of assignment statement nodes and build dependency edges based on the abstract syntax tree and node relationships.

[0061] The control flow edge creation module is used to process branching structures, loop structures, and jump structures based on the abstract syntax tree to obtain control flow edges.

[0062] The edge creation module is used to traverse the abstract syntax tree. When a call node is detected, if the called object is a local function or a member function in the local modified table, the CPG subgraph of the local function or member function definition is copied, and a dependency edge is established between the call block and the function entry block. Otherwise, the call block position and call information are stored in the import function table, and the function call edge is established. The called object exists in the call node, the call block is generated through the called object, and the function entry block is generated through the local function or the member function in the local modified table.

[0063] The single-file generation module is used to generate the CPG file corresponding to each Lua script file based on the call edge, dependency edge, control flow edge, and basic blocks generated by the filtered file set.

[0064] The indexing module is used to generate a file index based on the abstract syntax tree;

[0065] The project-level file generation module is used to generate project-level CPG files based on the file index and the corresponding CPG file for each Lua script file.

[0066] A terminal device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, it employs a project-level CPG file generation method.

[0067] A computer-readable storage medium storing a computer program, wherein when the computer program is loaded and executed by a processor, a project-level CPG file generation method is employed.

[0068] The beneficial effects of this invention are: by unpacking firmware and reconstructing the file tree, a lightweight Lua script index is established, providing basic support for static analysis. For single-file code attribute modeling, a scope-sensitive assignment alias recognition mechanism is designed; multi-scope variable lifecycle tracking is achieved through environment snapshot management; and control flow modeling combining branching, looping, and jump structures fully covers Lua syntax features.

[0069] In project-level CPG construction, a dependency dominance tree generation scheme based on the Tarjan-Kahn algorithm is proposed: the Tarjan algorithm is used to deconstruct strongly connected components, and the Kahn algorithm is combined with topological sorting to determine the bottom-up module construction order. Through a multi-process pool parallel architecture, precise mapping of cross-file function calls and subgraph merging are achieved, enabling efficient integration of project-level CPGs. Experimental results show that this method significantly improves the static analysis efficiency of large-scale Lua projects while maintaining modeling accuracy. Attached Figure Description

[0070] Figure 1 This is a schematic diagram illustrating the construction of an abstract syntax tree for a project-level CPG file generation method according to the present invention.

[0071] Figure 2 This is a schematic diagram of the algorithm flow for generating CPG subgraphs from assigned AST nodes according to the present invention;

[0072] Figure 3 This is a schematic diagram of the abstract syntax tree node structure of the branch structure of this invention;

[0073] Figure 4 This is a schematic diagram of the CPG modeling of the cyclic structure of the present invention, wherein, Figure 4 (a) is the CPG subgraph generated by the while loop. Figure 4 (b) is the CPG subgraph generated by the repeat-until loop. Figure 4 (c) is the CPG subgraph generated by the numerical for loop. Figure 4(d) is the CPG subgraph generated by the generic for loop;

[0074] Figure 5 This is a schematic diagram of the CPG generation algorithm for the goto-label jump structure of this invention;

[0075] Figure 6 This is a flowchart illustrating the algorithm for constructing dependencies between Lua project modules according to the present invention. Detailed Implementation

[0076] A method for generating project-level CPG files includes:

[0077] S1. Obtain the Lua script file set, parse and filter the Lua script file set, create an abstract syntax tree, and filter the file set to include multiple Lua script files.

[0078] S2. Based on the filtered file set, generate an abstract syntax tree, traverse the nodes in the abstract syntax tree that generate scopes, generate a snapshot of the current local variable table, generate a CPG subgraph when entering a scope, and restore the upper-level snapshot based on the current local variable table snapshot when exiting a scope to obtain the node relationships.

[0079] Specifically, based on Lua's multi-level nested scope feature, an environment snapshot mechanism is established to achieve precise variable lifecycle management. When traversing the AST (Abstract Syntax Tree) to nodes that can generate scopes (function declarations, table and metatable declarations, control structures, do-end structures), a snapshot of the current local variable table is first saved. Then, the AST is traversed within that scope, and a CPG subgraph is generated. Finally, when exiting the scope, the snapshot of the parent environment is restored. This mechanism effectively distinguishes between global and local variables and solves the problem of identifying variables with the same name in different scopes by renaming local variables.

[0080] Based on the selected file set, an abstract syntax tree is generated, including:

[0081] Specifically, the set of files to be filtered includes multiple Lua script files, which are text files that store source code written in the Lua language, with the common extension .lua.

[0082] The filtered file set is parsed to obtain a file directory, which includes a root directory and subdirectories;

[0083] Traverse the root directory and subdirectories, retaining valid directories containing Lua script files;

[0084] The valid directories are numbered according to their hierarchical size to obtain a numbered directory;

[0085] like Figure 1As shown, target directories are retrieved sequentially according to the numbered directories to create file tree nodes, and Lua path information is stored in the file tree nodes. If the target directory has no parent directory, the target directory is mounted to the root directory node. If the target directory has a parent directory, the target directory is mounted under the parent directory node, until all numbered directories are accessed. All file tree nodes form a file tree, and the file tree nodes include parent directory nodes and root directory nodes.

[0086] Specifically, a breadth-first search is first used to traverse the root directory of the unpacked project, filtering out directories containing Lua files in the recursive path, and numbering them in ascending order of path hierarchy to ensure that parent directories are accessed first. Then, directories are retrieved sequentially and nodes are created to store the Lua file path information in the current directory, mounting it as a subdirectory to the nearest parent directory node. Once all nodes are mounted, a final traversal of the file tree is performed, mounting the path information of all Lua files in its recursive subdirectories to each node in the file tree.

[0087] The files were parsed to extract the source code scripts to be analyzed, and a lightweight file tree was constructed based on this. This preprocessing process eliminated unnecessary file interference, laying the data structure foundation for the subsequent extensive lookups of Lua module dependency relationships.

[0088] S3. Based on the abstract syntax tree and node relationships, align the left and right values ​​of the assignment statement nodes and establish dependency edges;

[0089] The process involves aligning the left and right values ​​of assignment statement nodes based on the abstract syntax tree and node relationships, and establishing dependency edges, such as... Figure 2 As shown, it includes:

[0090] Based on the abstract syntax tree and node relationships, obtain the assignment node;

[0091] Extract the assignment statement from the assignment node;

[0092] Align the left-hand side of the assignment statement with the expression on the right-hand side of the equals sign to obtain an alignment statement;

[0093] Determine if the left-hand side of an alignment statement is a local variable;

[0094] If the lvalue is a local variable, it is renamed according to the current scope;

[0095] If the lvalue is not a local variable, a CPG basic block is created and the expression on the right side is extracted from the CPG basic block;

[0096] Rename the local variables in the expression on the right, and determine whether the expression on the right is a function call;

[0097] If the expression on the right is a function call, then a dependency edge is established.

[0098] Specifically, in the assignment statement processing, a multi-stage pollution propagation rule is established to address the unique syntactic features of Lua. When parsing an Assign type AST node, alignment operations are performed between the lvalue and the expression on the right side of the equals sign, and redundant lvalues ​​or expressions on the right side of the equals sign are truncated. Then, pollution sources are identified in the expression on the right side of the equals sign, extracting the involved variables, function calls, and closure structures, and renaming local variables according to the local variable table of the current scope. Next, the scope of the lvalue is determined; if it is a local variable, it is renamed using a scope prefix and added to the local function table; if it is a global variable, the CPG global variable table is updated. For cases where the expression on the right side of the equals sign contains a function call, since Lua's syntax allows multiple return values, dependency edges are established between all lvalues ​​that may be affected by the function and that function. Furthermore, dynamic alignment is performed during the subsequent search of static taint paths to compensate for accuracy, ensuring analysis efficiency while keeping the false alarm rate within an acceptable range.

[0099] For the definition and invocation of functions and methods, the system temporarily stores the parameter list in the basic control flow block, and then performs parameter alignment during the static taint analysis phase.

[0100] For complex structure handling, this solution employs a conservative pollution propagation strategy to improve the feasibility of analysis within limited time and space complexity, particularly for multi-level nested tables and metatable operations in Lua. When processing table operations, any member assignment is assumed to update the pollution flag for the entire table. While this method sacrifices some precision, it effectively avoids exponential complexity growth.

[0101] S4. Based on the abstract syntax tree, perform branching structure processing, loop structure processing, and jump structure processing to obtain the control flow edge;

[0102] Will The pointer points to the forked block and is initialized. For a linear list, enter the branching block. If a `test` pointer exists, access the subtree of the abstract syntax tree pointing to the branch's condition. Then, create the branch's entry block and save the subtree information of the abstract syntax tree pointed to by the `test` pointer to the basic control flow block. The pointer points to the entry block;

[0103] Access the abstract syntax tree subtree of the branch pointed to by the body pointer, and save the exit control flow block of the generated CPG subgraph to... linear list and The pointer points to the forked control flow block;

[0104] If the orelse pointer is not null, then restore the context and access the next branch pointed to by the orelse pointer;

[0105] If the orelse pointer is null, restore the context and exit the current branch. All basic control flow blocks in the linear list are edged with the converging basic control flow block, and... The pointer points to the convergence basic control flow block;

[0106] Specifically, control structure modeling is key to CPG's ability to achieve a unified representation of syntax, control, and data at multiple levels, and it permeates the entire process from code understanding and path verification to vulnerability detection. This work starts with the syntactic features of Lua and relies on a single-file generated AST to accurately model three types of control flow structures.

[0107] Processing loop structures based on the abstract syntax tree includes:

[0108] Create the starting basic control flow block and the ending basic control flow block based on the abstract syntax tree;

[0109] Will The pointer points to the starting basic control flow block, traverses the judgment conditions pointed to by the test pointer, and saves the abstract syntax tree information of the test pointer to the entry block;

[0110] Access the abstract syntax tree subtree pointed to by the body pointer and generate the corresponding CPG subgraph. Add control flow edges to the exit control flow block and the loop exit block of the CPG subgraph.

[0111] The Lua language uses the `if`, `if else`, and `else` keywords to implement branching structures. The branching structure abstract syntax tree constructed by `luaparser` includes, for example,... Figure 3 The key structures shown are as follows: the AST nodes generated by the three branch control structure keywords all contain three key structures: the control condition `test` (except for the `Else` branch control node), the pointer to the internal structure of the branch `body`, and the pointer to `orelse`. Figure 3 In this code, `test` represents the test condition, `body` represents the main code block, `orelse` is a pointer to the next possible branch to be executed (the next `else if` or `else` node), and `None` is a special flag. The `if`, `else if`, and `else` nodes are the `if`, `else if`, and `else` conditional nodes, respectively. These are keywords for conditional branching in a program.

[0112] A depth-first search is used to traverse the AST nodes of the branch and generate a CPG subgraph. First, branching and converging basic control flow blocks are created, and the pointer to the current basic control flow block is set. Point to the forked block and initialize. Linear list. Save the context and enter the branch. If the `test` pointer exists, access the AST subtree pointing to the branch's decision condition. Then, create the entry block for that branch and save the AST subtree information pointed to by `test` to this basic control flow block. Points to the entry block. Then accesses the AST subtree within the branch pointed to by the body pointer, saving the exit control flow block of the generated CPG subgraph. The basic control flow block it points to linear list and The pointer points to the forked control flow block. Finally, if the `orelse` pointer is not null, the context is restored and the next branch pointed to by the `orelse` pointer is accessed; if the `orelse` pointer is null, the context is restored and the current branch is exited, and the linear list is rewritten. All basic control flow blocks in the pooling basic control flow block establish edges with each other, and... The pointer points to the convergence basic control flow block, and the algorithm for generating the CPG subgraph from the AST of this branch ends.

[0113] In Lua's basic syntax, the keywords `while`, `for`, `repeat`, and `until` are used to implement loop control structures. Although these three keywords implement three different loop control structures, the underlying idea of ​​traversing their ASTs to generate the corresponding CPG subgraphs is essentially the same.

[0114] The AST node corresponding to the "while" keyword contains a test pointer and a body pointer, pointing to the AST subtrees generated by the condition and the loop body, respectively. When the depth-first traversal reaches the AST node of the while loop, the basic control flow blocks for the start and end of the loop are first created. The pointer points to the loop start block and saves the context. Upon entering the loop, it first iterates through the condition pointed to by the `test` pointer, saves the AST information of `test` to the entry block, and marks the implicit propagation token. Then it visits the AST subtree pointed to by the `body` pointer, generates the corresponding CPG subgraph, and sets the exit control flow block of the subgraph (…). Add control flow edges between the underlying control flow block (the one it points to) and the loop exit block. Finally, restore the context and... Point to the loop exit block and exit the loop.

[0115] The AST structure generated by the repeat-until loop is completely identical to that generated by the while loop. The only difference is that the repeat-until loop executes first and then checks the condition. Therefore, when constructing the CPG subgraph from the repeat-until loop AST, it is necessary to first traverse the AST subtree pointed to by the body pointer, and then traverse the AST subtree pointed to by the test pointer.

[0116] The loop control structure implemented by the "for" keyword is relatively complex. Lua has two types of structures: numeric for (Fornum) and generic for (Forin). They both use an iterator type to generate the loop condition. Therefore, unlike while loops and repeat-until loops which calculate the condition after each iteration, the loop constructed by the "for" keyword calculates the necessary condition before entering the loop.

[0117] Therefore, when generating a CPG subgraph from a numerical for (Fornum) AST structure, it is necessary to first traverse the starting assignment (target and start), ending point (stop), and step size of the AST node to generate the CPG subgraph in sequence and add implicit propagation markers, and then traverse the loop body to create the corresponding CPG subgraph. Similarly, for a generic for (Forin) AST structure, it is also necessary to first traverse the iterator (targets and iter) to generate the CPG subgraph and add implicit propagation markers before performing subsequent operations.

[0118] like Figure 4 The figure shows the CPG models constructed by the proposed scheme for four loop structures. Figure 4 In this context, Entry represents the entry point, indicating the starting position of the program flow when it enters this code block or loop.

[0119] Exit is the exit point, indicating the position where the program flow leaves this code block or the end of a loop.

[0120] The body represents the main code block.

[0121] While: While... This is a keyword in loop statements used to create a loop that repeats as long as a condition is met.

[0122] Test(): Test(), specifically the loop condition of the while loop, that is, the expression that is checked before each iteration. If the result is "true", the loop body is executed.

[0123] Repeat: This is a label indicating that a block of code that can be executed repeatedly is starting from here.

[0124] Until Test(): Until the [condition] is true. This is a join point, indicating that the flow will exit the loop along this path "when the condition of Test() is true".

[0125] Start is the starting value, Stop is the termination condition, and step is the step size.

[0126] The loop function `For Target = Start(), Stop(), Step()` takes values ​​from `Start()` to `Stop()` for the loop variable `Target`, with a step size of `Step()`.

[0127] Iter() is an iterable object.

[0128] The `For Targets in Iter()` statement is the loop header declaration. It declares a variable `Targets`. The loop automatically retrieves the first, second, ..., ... element from `Iter()` sequentially until the last element, and assigns each element to the variable `Targets`.

[0129] Create the starting basic control flow block and the ending basic control flow block based on the abstract syntax tree;

[0130] Will The pointer points to the starting basic control flow block, traverses the judgment conditions pointed to by the test pointer, and saves the abstract syntax tree information of the test pointer to the entry block;

[0131] Access the abstract syntax tree subtree pointed to by the body pointer and generate the corresponding CPG subgraph. Add control flow edges to the exit control flow block and the loop exit block of the CPG subgraph to establish a loop structure.

[0132] Traverse the abstract syntax tree. When the goto structure is encountered, create a goto control flow block and check if the target basic control flow block corresponding to the label exists in the hash table. If it exists, establish a control flow dependency edge between the target basic control flow block and the basic control flow block corresponding to the label. If it does not exist, store the goto control flow block in the linear list corresponding to the label in the hash table.

[0133] When traversing the label structure, create a label control flow block, and establish control flow dependency edges between all goto control flow blocks in the linear list corresponding to the label in the hash table and the label control flow block. Finally, store the label control flow block in the hash table and establish a jump structure.

[0134] Specifically, jump structure modeling.

[0135] The Lua language uses the "break" keyword to exit a single-level loop; the "return" keyword to return from a function or method; and the paired "goto" and "label" keywords to perform unconditional jumps.

[0136] Modeling of conditional jump control flow is achieved using a stack data structure. Before constructing a single-file CPG, separate working stacks are first established for loops and functions.

[0137] When creating a CPG subgraph of a loop structure, the loop termination block is first stored on the loop working stack before traversing the loop body AST subtree. During traversal of the loop body AST subtree, if a `break` statement is encountered, a `break` block is created and a control flow dependency edge is established with the block at the top of the loop working stack. Finally, upon exiting the loop traversal, the loop working stack is popped. Similarly, the same method can be used to model the "return" control structure using a function working stack.

[0138] Since the jumps implemented by the keywords "goto" and "label" are unconditional (forward or backward jumps), the order in which the same pair of "goto" and "label" appears in the AST is not fixed, and multiple "goto" can correspond to one jump label. Therefore, a hash table is needed to model this.

[0139] When traversing to a `goto` structure, a `goto` control flow block is created, and it is checked whether a control flow block corresponding to the `label` exists in the hash table. If it exists, a control flow dependency edge is established between this control flow block and the control flow block corresponding to the label; otherwise, the `goto` block is stored in the linear list corresponding to the `label` in the hash table. When traversing to a `label` structure, a `label` control flow block is created, and control flow dependency edges are established between it and all `goto` blocks in the linear list corresponding to this label in the hash table. Finally, the `label` control flow block is stored in the hash table. Figure 5 The flowchart shown is the corresponding to this algorithm.

[0140] Based on the abstract syntax tree, branch structure modeling, loop structure modeling, and jump structure modeling are performed to obtain control flow edges.

[0141] S5. Traverse the abstract syntax tree. When a call node is detected, if the called object is a local function or a member function in the local table, copy the CPG subgraph defined by the local function or member function, and establish a dependency edge between the call block and the function entry block. Otherwise, store the call block position and call information in the import function table, and establish a function call edge. The called object exists in the call node, the call block is generated through the called object, the function entry block is generated through the local function or a member function in the local table, and the function dependency edge represents the jump relationship during program execution.

[0142] Specifically, accurate identification of function call dependencies is a core element in constructing program dependency edges, and its modeling accuracy directly determines the completeness of the CPG. Considering the function design characteristics of large Lua projects such as WebUI backend source code, global functions, module member functions, and methods generally exhibit cross-file call characteristics, while local functions and anonymous functions are mostly used for single-call scenarios within a file.

[0143] Local functions are constrained by the "local" keyword, and their lifetime is strictly limited to the current scope, not visible to parent and sibling scopes. To reduce the computational complexity of subsequent static taint analysis, the impact of lifetime on function calls is fully considered during the generation of a single-file CPG.

[0144] During the AST traversal to generate the CPG, when a call node is detected: if the called object is a local function or a member function in the table modified by "local", then the CPG subgraph of that function is copied, and a dependency edge is established between the call block and the function entry block; otherwise, the call block location and call information are stored in the import function table, and the function call edge is dynamically established during static taint analysis. For member functions in a table, when the main program flow exports the table using the "return" keyword, the file becomes a Lua module, and its public members have cross-file calling capabilities. The CPG subgraph of such member functions needs to be stored in the function export table. The function names of global functions, module member functions, and methods, along with their corresponding subgraph indices, constitute the export function table, serving as a single-file code attribute graph. Components of.

[0145] S6. Based on the call edges, dependency edges, control flow edges, and the basic blocks generated by filtering the file set, obtain the CPG file corresponding to each Lua script file;

[0146] Specifically, code attribute diagram Using quadruples for formal representation:

[0147]

[0148] in, This represents the basic block that makes up the CPG, used to represent a Lua expression or statement. It represents the edges that constitute the CPG, and serves as The connecting function between them. , , These represent the control flow edge, data flow edge, and call edge of the code property graph, respectively.

[0149] This section will explain the modeling process of generating a CPG from a single Lua file, including single-file code attribute diagrams. The definition is as follows:

[0150]

[0151] in, The subgraph structure generated corresponding to the main program flow. This corresponds to the collection of subgraphs generated by each function in the Exported Functions table. Global Variables Table Construct cross-file data flow edges and import function tables. and dependency module table It accurately depicts the call relationship edges across files.

[0152] S7. Generate a file index based on the abstract syntax tree;

[0153] S8. Generate a project-level CPG file based on the file index and the CPG file corresponding to each Lua script file.

[0154] The step of generating a project-level CPG file based on the file index and the corresponding CPG file for each Lua script file includes:

[0155] An adjacency matrix is ​​built based on the file index, and a dependency dominance tree is built based on the adjacency matrix and the CPG file corresponding to each Lua script file.

[0156] The global topological relationships are obtained by dominating the dependency tree.

[0157] Based on the global topology, a project-level CPG file is generated.

[0158] The process of obtaining the global topological relationship based on the dependency dominance tree includes:

[0159] Based on the file index, the dependent directories are obtained;

[0160] Based on the dependency directory, determine whether the current dependency exists in the current directory;

[0161] If the current dependency does not exist in the current directory, then check if the current dependency exists in the recursive subdirectory;

[0162] If the current dependency does not exist in the recursive subdirectory, then determine whether the global variable has been searched;

[0163] If global variables have been searched, determine whether it is possible to revert to the parent directory;

[0164] If it is not possible to return to the parent directory, the dependency is a C package or is missing, and the topological relationship of the root directory is obtained;

[0165] Get the topological relationship of CPG subdirectories;

[0166] Based on the topological relationships of the root directory and the CPG subdirectories, the global topological relationships are obtained.

[0167] Obtaining the topological relationship of CPG subdirectories includes:

[0168] If the current dependency exists in the current directory, or in a recursive subdirectory, or if the global variable has not been searched and the dependency exists in the global variable directory, then save the dependency system, establish edges in the adjacency matrix, and obtain the topological relationship of the CPG subdirectories.

[0169] Specifically, project-level code attribute diagram The combined code property subgraph generated from Lua files in the project is defined as follows, where i corresponds to the index in the Lua script build file tree:

[0170]

[0171] for Each subgraph in the graph relies on function call edges formed by cross-file function calls. Value flow edges formed by global variable references across files Connected together.

[0172] The construction first requires building a Lua dependency dominance tree using the Tarjan-Kahn algorithm and calculating the topological order of dependencies, and then using a process pool to build it in parallel.

[0173] Construction of Lua Dependency Dominance Tree Based on Tarjan-Kahn Algorithm

[0174] The cross-file function call feature of Lua projects requires precise control over the CPG subgraph of a single file. The generation order is determined by this scheme. This scheme establishes an adjacency matrix using a pre-built file tree index, accurately representing the dependencies of Lua modules introduced by require / dofile with directed edges. However, the working paths of large Lua projects are extremely complex, making it difficult to precisely locate the working path of each script in an information-less reverse security analysis scenario. Therefore, in the reverse analysis of the Lua module loading mechanism, a method such as... Figure 6 The progressive dependency retrieval strategy shown is as follows: first, locate the same directory as the calling script; if not found, traverse the recursive subdirectories, then load the global variable path, and finally complete the dependency loop determination by backtracking through the parent directory, thus forming a complete program dependency topology network.

[0175] In the preprocessing stage, the Tarjan algorithm is used to find the contractions in the dependency graph, decomposing the cyclic dependency structure into multiple sets of disconnected supernodes. This algorithm is based on depth-first search and traverses... A working stack is required. Maintain the sequential number of the node that is first visited in the depth-first search. And the earliest node in the stack that can be backtracked to from the current node and its subtree. The value is denoted as Using global variables outside of functions Manage traversal order, use Record the abbreviated point number and maintain the status flag. Used to determine if a node exists in the work stack. .

[0176] The core of this process is the use of the Tarjan algorithm to identify the dependency adjacency matrix. Strongly connected components .

[0177] Then, the Kahn algorithm is used to implement hierarchical topological sorting, and the in-degree value is dynamically maintained. A queue of nodes with a value of zero By peeling away the dependencies between supernodes layer by layer, a dominator tree structure with strict bottom-up characteristics is generated. The leaf nodes of this dominator tree represent basic modules without external dependencies, upon which the topological sequence between nodes can be derived. This provides sequential constraints for subsequent parallel construction.

[0178] A project-level CPG construction method based on process pools.

[0179] Based on the hierarchical characteristics of the dependency dominance tree, the generation order of each CPG subgraph is obtained from the traversal order. The work queue is initialized according to the number of CPU cores. The main control thread submits single-file CPG build tasks to the process pool in batches according to the topological order of the dependency dominance tree. A priority queue is used to ensure that the underlying modules complete subgraph generation first.

[0180] After all single-file CPG subgraphs have been generated, the adjacency matrix is ​​used as the basis for the calculation. Record cross-file dependencies. Then merge the copies of exported function tables in all CPG subgraphs to generate a global symbol table, and finally complete the integration of the project-level CPG.

[0181] A project-level CPG file generation system, comprising:

[0182] The acquisition module is used to acquire a set of Lua script files, parse and filter the set of Lua script files to obtain a filtered set of files, which includes multiple Lua script files.

[0183] The traversal module is used to generate an abstract syntax tree based on the filtered file set, traverse the nodes in the abstract syntax tree that generate the scope, generate a snapshot of the current local variable table, generate a CPG subgraph when entering the scope, and restore the upper-level snapshot based on the current local variable table snapshot when exiting the scope to obtain the node relationship;

[0184] The dependency edge building module is used to align the left and right values ​​of assignment statement nodes and build dependency edges based on the abstract syntax tree and node relationships.

[0185] The control flow edge creation module is used to process branching structures, loop structures, and jump structures based on the abstract syntax tree to obtain control flow edges.

[0186] The edge creation module is used to traverse the abstract syntax tree. When a call node is detected, if the called object is a local function or a member function in the local modified table, the CPG subgraph of the local function or member function definition is copied, and a dependency edge is established between the call block and the function entry block. Otherwise, the call block position and call information are stored in the import function table, and the function call edge is established. The called object exists in the call node, the call block is generated through the called object, and the function entry block is generated through the local function or the member function in the local modified table.

[0187] The single-file generation module is used to generate the CPG file corresponding to each Lua script file based on the call edge, dependency edge, control flow edge, and basic blocks generated by the filtered file set.

[0188] The indexing module is used to generate a file index based on the abstract syntax tree;

[0189] The project-level file generation module is used to generate project-level CPG files based on the file index and the corresponding CPG file for each Lua script file.

[0190] This application also discloses a terminal device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor loads and executes the computer program, a project-level CPG file generation method is used.

[0191] The terminal device can be a computer device such as a desktop computer, a laptop computer, or a cloud server. The terminal device includes, but is not limited to, a processor and a memory. For example, the terminal device may also include input / output devices, network access devices, and buses.

[0192] The processor can be a central processing unit (CPU). Of course, depending on the actual use, it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc., and this application does not limit it.

[0193] The memory can be an internal storage unit of the terminal device, such as a hard disk or RAM of the terminal device, or an external storage device of the terminal device, such as a plug-in hard disk, smart memory card (SMC), secure digital card (SD), or flash memory card (FC) equipped on the terminal device. Furthermore, the memory can be a combination of internal storage units and external storage devices of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output. This application does not limit this.

[0194] In this terminal device, a project-level CPG file generation method from the above embodiments is stored in the terminal device's memory and loaded and executed on the terminal device's processor for convenient use.

[0195] This application also discloses a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, it employs a project-level CPG file generation method described in the above embodiments.

[0196] The computer program can be stored in a computer-readable medium. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or certain middleware. The computer-readable medium includes any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the computer-readable medium includes, but is not limited to, the above-mentioned components.

[0197] In this computer-readable storage medium, a project-level CPG file generation method from the above embodiments is stored in the computer-readable storage medium and loaded and executed on the processor to facilitate the storage and application of the above method.

[0198] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of protection of this application is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in detail for the sake of brevity.

[0199] One or more embodiments in this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of this application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments in this application should be included within the protection scope of this application.

Claims

1. A project-level CPG file generation method, characterized by, The method comprises the following steps: obtaining a Lua script file set, parsing and screening the Lua script file set to obtain a screened file set, and the screened file set comprising a plurality of Lua script files; generating an abstract syntax tree according to the screened file set, traversing nodes of the abstract syntax tree to generate scopes, generating a current local variable table snapshot, generating a CPG subgraph when entering a scope, and restoring an upper snapshot according to the current local variable table snapshot when exiting the scope to obtain node relationships; aligning a left value and a right value of an assignment statement node according to the abstract syntax tree and the node relationships, and establishing a dependency edge; performing branch structure processing, loop structure processing and jump structure processing according to the abstract syntax tree to obtain control flow edges; traversing the abstract syntax tree, and when a call node is detected, if a called object is a local function or a member function in a table modified by local, copying a CPG subgraph defined by the local function or the member function, and establishing a function dependency edge between a call block and a function entry block, otherwise, storing a call block position and call information in an imported function table, and establishing a function call edge, wherein the called object exists in the call node, the call block is generated through the called object, the function entry block is generated through the local function or the member function in the table modified by local, and the function dependency edge represents a jump relationship during program execution; generating a CPG file corresponding to each Lua script file according to a basic block generated by the call edge, the dependency edge, the control flow edge and the screened file set; generating a file index according to the abstract syntax tree; generating a project-level CPG file according to the file index and the CPG file corresponding to each Lua script file; the method of generating the project-level CPG file according to the file index and the CPG file corresponding to each Lua script file comprises: establishing an adjacency matrix according to the file index, and establishing a dependency relationship dominance tree according to the adjacency matrix and the CPG file corresponding to each Lua script file; obtaining a global topological relationship according to the dependency relationship dominance tree; generating the project-level CPG file according to the global topological relationship; the method of obtaining the global topological relationship according to the dependency relationship dominance tree comprises: obtaining a dependency directory according to the file index; judging whether a current dependency exists in a current directory according to the dependency directory; if the current dependency does not exist in the current directory, judging whether the current dependency exists in a recursive subdirectory; if the current dependency does not exist in the recursive subdirectory, judging whether a global variable has been searched; if the global variable has been searched, judging whether a parent directory can be backed up; if the parent directory cannot be backed up, the dependency is a C language package or missing, and a topological relationship of a root directory is obtained; obtaining a topological relationship of a CPG subdirectory; obtaining the global topological relationship according to the topological relationship of the root directory and the topological relationship of the CPG subdirectory.

2. The project-level CPG file generation method of claim 1, wherein, the method of generating the abstract syntax tree according to the screened file set comprises: parsing the screened file set to obtain a file directory, wherein the file directory comprises a root directory and a subdirectory; traversing the root directory and the subdirectory to retain valid directories having Lua script files; numbering the valid directories according to a hierarchical size to obtain numbered directories. According to the numbered directory, a target directory is taken out in sequence to create a file tree node, and path information of Lua is stored in the file tree node, if the target directory has no parent directory, the target directory is mounted to a root directory node, if the target directory has a parent directory, the target directory is mounted to a parent directory node, until all numbered directories are accessed, all file tree nodes form a file tree, and the file tree node includes a parent directory node and a root directory node.

3. The project-level CPG file generation method of claim 1, wherein, The left value and the right value of the assignment statement node are aligned according to the abstract syntax tree and the node relationship, and a dependency edge is established, including: An assignment node is obtained according to the abstract syntax tree and the node relationship; An assignment statement is extracted from the assignment node; The left value in the assignment statement and the expression on the right side of the equal sign are aligned to obtain an aligned statement; It is judged whether the left value of the aligned statement is a local variable; If the left value is a local variable, the current scope is renamed; If the left value is not a local variable, a CPG basic block is created, and the expression on the right side is extracted from the CPG basic block; The local variable in the expression on the right side is renamed, and it is judged whether the expression on the right side is a function call; If the expression on the right side is a function call, a dependency edge is established.

4. The project-level CPG file generation method of claim 1, wherein, The branch structure processing, the loop structure processing and the jump structure processing are performed according to the abstract syntax tree, and control flow edges are obtained, including: The branch structure processing according to the abstract syntax tree includes: A bifurcation control flow block and a convergence basic control flow block are created according to the abstract syntax tree; The pointer points to the branching block, initializes linear list, enters the branching block, if the test pointer exists, accesses the subtree of the abstract syntax tree pointing to the branching judgment condition, then creates the entry block of the branch and saves the information of the subtree of the abstract syntax tree pointed to by the test pointer to the basic control flow block, and pointer points to the entry block; The access body pointer points to the abstract syntax tree sub-tree of the internal structure of the branch, and the exit control flow block of the generated CPG sub-graph is saved to the linear table and the pointer points to the bifurcation control flow block; If the orelse pointer is not empty, the context is restored, and the next branch pointed to by the orelse pointer is accessed; If the orelse pointer is null, restore the context, exit the current branch, and all basic control flow blocks in the linear list are connected with the sink basic control flow block by edges, and the pointer points to the sink basic control flow block; The loop structure processing according to the abstract syntax tree includes: A start basic control flow block and a termination basic control flow block are created according to the abstract syntax tree; The method comprises the following steps: The pointer points to the starting basic control flow block, traverses the judgment condition pointed by the test pointer, and saves the abstract syntax tree information of the test pointer to the entry block. The abstract syntax tree subtree pointed to by the body pointer is accessed, and a corresponding CPG subgraph is generated, and an exit control flow block of the CPG subgraph and a loop exit block are added to a control flow edge; The jump structure processing according to the abstract syntax tree includes: The abstract syntax tree is traversed, when a goto structure is traversed, a goto control flow block is created, and it is checked whether a target basic control flow block corresponding to the label tag exists in a hash table, if the target basic control flow block exists, the target basic control flow block and the basic control flow block corresponding to the label tag are established to a control flow dependency edge, if the target basic control flow block does not exist, the goto control flow block is stored in a linear table corresponding to the label tag in the hash table; When a label structure is traversed, a label control flow block is created, and all goto control flow blocks in the linear table corresponding to the label tag in the hash table and the label control flow block are established to a control flow dependency edge, and finally the label control flow block is stored in the hash table; The branch structure processing, the loop structure processing and the jump structure processing are performed according to the abstract syntax tree, and control flow edges are obtained.

5. The method of claim 1 wherein, The topological relationship of the CPG subdirectory includes: If the current dependency exists in the current directory or the current dependency exists in the recursive subdirectory or the global variable has not been searched and the dependency exists in the global variable directory, the dependency is saved, an edge is established in an adjacency matrix, and the topological relationship of the CPG subdirectory is obtained.

6. A project-level CPG file generation system characterized by, It includes: The acquisition module is configured to acquire a Lua script file set, parse and screen the Lua script file set, and obtain a screened file set including a plurality of Lua script files; The traversal module is configured to generate an abstract syntax tree according to the screened file set, traverse nodes of a scope generated in the abstract syntax tree, generate a current local variable table snapshot, generate a CPG subgraph when entering the scope, restore an upper snapshot according to the current local variable table snapshot when exiting the scope, and obtain node relationships; The dependency edge establishing module is configured to align a left value and a right value of an assignment statement node according to the abstract syntax tree and the node relationships, and establish a dependency edge; The control flow edge establishing module is configured to perform branch structure processing, loop structure processing, and jump structure processing according to the abstract syntax tree, and obtain a control flow edge; The call edge establishing module is configured to traverse the abstract syntax tree, and when a call node is detected, if a called object is a local function or a member function in a table modified by local, copy a CPG subgraph defined by the local function or the member function, and establish a dependency edge between a call block and a function entry block, otherwise, store a call block position and call information in an imported function table, and establish a call edge of the function, the called object being in the call node, the call block being generated through the called object, and the function entry block being generated through the local function or the member function in the table modified by local; The single file generating module is configured to generate a basic block according to the call edge, the dependency edge, the control flow edge, and the screened file set, and obtain a CPG file corresponding to each Lua script file; The index module is configured to generate a file index according to the abstract syntax tree; The project-level file generating module is configured to generate a project-level CPG file according to the file index and the CPG file corresponding to each Lua script file; The project-level CPG file is generated according to the file index and the CPG file corresponding to each Lua script file, including: an adjacency matrix is established according to the file index, and a dependency relationship dominance tree is established according to the adjacency matrix and the CPG file corresponding to each Lua script file; a global topological relationship is obtained according to the dependency relationship dominance tree; the project-level CPG file is generated according to the global topological relationship; the global topological relationship is obtained according to the dependency relationship dominance tree, including: a dependency directory is obtained according to the file index; whether a current dependency exists in a current directory is judged according to the dependency directory; if the current dependency does not exist in the current directory, whether the current dependency exists in a recursive subdirectory is judged; if the current dependency does not exist in the recursive subdirectory, whether a global variable has been searched is judged; if the global variable has been searched, whether a parent directory can be backed up is judged; if the parent directory cannot be backed up, the dependency is a C language package or missing, and a topological relationship of a root directory is obtained; a topological relationship of a CPG subdirectory is obtained; the global topological relationship is obtained according to the topological relationship of the root directory and the topological relationship of the CPG subdirectory.

7. A terminal device comprising a memory and a processor, characterized in that The memory stores a computer program capable of running on the processor, and the processor loads and executes the computer program to adopt the method in any one of claims 1 to 5.

8. A computer-readable storage medium having stored therein a computer program, characterized in that, The computer program, which is loaded and executed by a processor, employs the method as claimed in any one of claims 1 to 5.

Images