Method, device and storage medium for collaborative detection of cloned code by cloud-edge-end
Through symbol execution technology and special solution judgment, the problem of insufficient detection accuracy of cloned code in the existing technology is solved, and function-level binary code cloning detection is realized, which improves detection accuracy and interpretability.
Patent Information
- Application Number
- CN202411146900.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-21
AI Technical Summary
When detecting cloned codes with similar semantics but different syntaxes, the prior art lacks generalization ability, overfitting, and incomplete feature extraction, resulting in a lack of accuracy and interpretability of the detection results.
Symbol execution is used as feature extraction scheme, and suspected cloned code is filtered through function name judgment, function parameter judgment, import and export symbol judgment, symbol execution is performed on it, path constraint features are obtained, and special solutions are used to determine the equivalence of constraint expressions to achieve Function-level binary code cloning detection.
Improves the accuracy of cloned code detection, provides stronger interpretability, can recognize semantic similar codes, and in some cases achieve higher accuracy.
Smart Images

Figure CN118656125B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of software engineering technology, and in particular to a method, device and storage medium for collaborative detection of cloned code between a cloud and an edge. Background Art
[0002] Open source code plays a key role in promoting innovation and technological progress. Although the reuse of open source code makes software development more convenient, there is also a potential risk of introducing vulnerabilities. In order to effectively meet this challenge, code clone detection has become one of the important methods to discover reused code in actual production environments and academic research. Code clone detection is an important technical means in the field of traditional software engineering and software security, and plays a key role in code vulnerability assessment, vulnerability mining, software supply chain security, and code maintenance. When assessing code vulnerabilities, cloned code may cause low-quality code and potential security risks. By detecting cloned code, potential security risks can be identified, thereby improving the overall security of the software. At the same time, cloned code may also have similar code vulnerabilities. By detecting cloned code, new vulnerabilities can be discovered and repaired in a timely manner. In terms of software supply chain security, third-party libraries or open source projects may have outdated problems, and downstream software developers may not update related codes in a timely manner. By detecting cloned code, potential supply chain security risks can be identified to ensure the security and credibility of the software. In addition, although cloned code can reduce the difficulty of writing code, a large amount of copying may also lead to code duplication and redundancy, increasing the difficulty of maintenance. Through code clone detection, the maintainability of code can be improved, maintenance costs can be reduced, and the continuous development and improvement of software can be promoted.
[0003] The technical principle of code clone detection is to determine whether the code pair is a similar code pair, that is, a clone code pair, by comparing the code features of the code pair. Clone code refers to two pieces of code with similar syntax or semantic structure, so there may be similar security vulnerabilities and quality problems. Through code clone detection, similar code vulnerabilities can be quickly discovered and repaired in time, thereby significantly reducing the possibility of the occurrence of known code vulnerabilities. According to whether the source code of the code base to be detected is available, code clone detection technology is divided into source code clone detection technology and binary code clone detection technology. The main technical process of code clone detection technology is to extract code features, detect code feature similarity, and determine the code similarity threshold. The two most important technologies are code feature extraction and code similarity judgment. In code feature extraction, the simplest code feature is the text information of the code, among which the binary code is usually converted into assembly code using disassembly technology to extract code features. Common code features include control flow graphs, function call graphs, symbolic constraints, code blocks, syntax trees, etc. These features have been proven to be effective by a large number of products in academia and industry. In terms of code similarity judgment models, the most common models in academia are various machine learning models. As shown in the figure, neural networks, decision tree models, random forests, etc. These models have achieved remarkable results in detecting syntactically similar code pairs, but they often fail to achieve the expected results when faced with cloned codes with similar semantics but different syntax. The specific manifestations are as follows: 1. The single model used in the existing technology may have insufficient generalization ability or overfitting problems. This means that the training set affects the final machine learning effect, and a large-scale high-quality data set is required. 2. The features used in the existing technology may not fully capture the characteristics and patterns of the fault. This may lead to a lack of comprehensiveness and accuracy in the results. 3. The existing technology does not have strong interpretability. Due to the use of machine learning, it is impossible to explain whether its features are truly effective.
[0004] Therefore, how to improve the detection accuracy of cloned code has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The present invention aims to solve at least one of the technical problems existing in the prior art or related technology, and discloses a method, device and storage medium for collaborative detection of cloned code between the cloud and the edge. Symbolic execution is introduced as a feature extraction scheme to realize binary code clone detection at the function level, which helps to discover potential vulnerabilities and errors in the program and provide an important analysis and verification tool for the security and reliability of the program.
[0006] The first aspect of the present invention discloses a method for collaboratively detecting cloned codes between cloud and edge, comprising: parsing two executable files to be compared, screening out suspected cloned code pairs through function name judgment, function parameter judgment, and import and export symbol judgment; performing symbolic execution on the suspected cloned code pairs, obtaining path constraint features, and following the following strategy during symbolic execution: when the symbolic execution runs on a certain node for a time exceeding a first threshold, pausing the symbolic execution simulation of the current branch, and discarding the branch; when the symbolic execution runs on a certain branch, if it is detected that the occupied memory space resources exceed a second threshold, pausing the symbolic execution simulation of the current branch, and discarding the branch; when the symbolic execution encounters a loop structure or a recursive call structure, only running the internal code of the loop and recursive call once, and then jumping out of the loop and recursion; according to the path constraint features obtained by symbolic execution, each function path corresponds to a set of function constraints, and a special solution is used to determine whether the constraint expressions of the suspected cloned code pairs are equivalent. If the number of equivalent paths of the suspected cloned code pairs reaches a preset value, the suspected cloned code pairs are determined to be cloned codes.
[0007] In this technical solution, symbolic execution is introduced as a feature extraction solution to achieve binary code clone detection at the function level. The development of symbolic execution provides a new direction for code clone detection. Symbolic execution technology executes programs by replacing specific input values with symbolic values, thereby achieving a comprehensive exploration of various possible execution paths of the program. By collecting and solving path constraints, the symbolic executor can discover potential vulnerabilities and errors in the program, providing an important analysis and verification tool for the security and reliability of the program.
[0008] According to the method for cloud-edge-end collaborative detection of cloned code disclosed in the present invention, preferably, the step of using a special solution to determine whether the constraint expressions of a suspected cloned code pair are equivalent specifically includes: extracting integers from two sets of constraint expressions to be compared as initial seeds, and using the integers and integers near the integers as inputs of the constraint expressions to determine whether the constraint expressions are satisfiable; if for the two sets of constraint expressions, the satisfiability of more than 80% of the integer special solutions is similar, then it is determined to be an equivalent constraint, that is, the paths corresponding to the two sets of constraint expressions are equivalent paths.
[0009] According to the method for cloud-edge-end collaborative detection of cloned code disclosed in the present invention, preferably, function name judgment specifically includes: parsing two executable files to be compared, obtaining corresponding symbol tables, filtering out functions, and obtaining function signatures, wherein the function signature includes function name, function parameters, and function return value; obtaining the disassembly code and program control flow graph of the code through disassembly calculation, and only performing subsequent judgment on functions with more than 5 nodes in the control flow graph; judging whether the names of the functions are the same: if two functions have the same function name, they are added to the list of functions to be compared.
[0010] According to the method for cloud-edge-end collaborative detection of cloned code disclosed in the present invention, preferably, function parameter judgment specifically includes: detecting whether the number of parameters is the same: adding functions with the same number of parameters to the list of functions to be compared, and if the number of parameters of two functions is different, directly discarding this function pair.
[0011] According to the method for cloud-edge collaborative detection of cloned code disclosed in the present invention, preferably, the import and export symbol judgment specifically includes: locating the corresponding function according to the import symbols and export symbols in the symbol table of the executable file, and identifying the code pairs above the similarity threshold as suspected cloned code pairs by comparing the Jaccord similarity of the function names.
[0012] According to the method for cloud-edge collaborative detection of cloned code disclosed in the present invention, preferably, symbolic execution specifically includes: obtaining a symbol table and a function signature in the symbol table; obtaining the number and types of parameters of the function to be analyzed from the function signature; constructing formal parameters with the same storage space according to the number of function parameters and the space used, and using the formal parameters as the initialization conditions for symbolic execution at the function level.
[0013] According to the method for collaborative cloud-edge-end detection of cloned code disclosed in the present invention, preferably, angr, a Python-based binary analysis framework, is used to parse the executable file.
[0014] According to the method for cloud-edge-end collaborative detection of cloned code disclosed in the present invention, preferably, the first threshold is 5 minutes; and the second threshold is 8GB.
[0015] The second aspect of the present invention discloses a device for collaboratively detecting cloned code between cloud and edge, comprising: a memory for storing program instructions; a processor for calling the program instructions stored in the memory to implement a method for collaboratively detecting cloned code between cloud and edge as described in any of the above technical solutions.
[0016] The third aspect of the present invention discloses a computer-readable storage medium, which stores a program code, and the program code is used to implement a method for cloud-edge-end collaborative detection of cloned code as described in any of the above technical solutions.
[0017] The beneficial effects of the present invention include at least: the present invention does not use machine learning to extract features and perform feature comparison, and has strong interpretability. The present invention obtains new code features and path constraints of binary code through symbolic execution technology, thereby realizing code semantic clone detection and improving the accuracy of code clone detection. Different from the machine learning solution, the present invention uses symbolic execution to formally execute the entire function, which can identify the deep semantics of the function and achieve higher accuracy in certain semantically similar situations. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 A flow chart of a method for cloud-edge-end collaborative detection of cloned code according to an embodiment of the present invention is shown.
[0019] Figure 2 A schematic block diagram of an apparatus for collaboratively detecting cloned codes by the cloud and the edge according to an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0020] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0022] According to one embodiment of the present invention, a method for collaboratively detecting cloned codes between cloud and edge is disclosed, comprising: parsing two executable files to be compared, screening out suspected cloned code pairs through function name judgment, function parameter judgment, and import and export symbol judgment; performing symbolic execution on the suspected cloned code pairs, obtaining path constraint features, and following the following strategy during symbolic execution: when the symbolic execution runs on a certain node for a time exceeding a first threshold, pausing the symbolic execution simulation of the current branch, and discarding the branch; when the symbolic execution runs on a certain branch, if it is detected that the occupied memory space resources exceed a second threshold, pausing the symbolic execution simulation of the current branch, and discarding the branch; when the symbolic execution encounters a loop structure or a recursive call structure, only running the internal code of the loop and recursive call once, and then jumping out of the loop and recursion; according to the path constraint features obtained by symbolic execution, each function path corresponds to a set of function constraints, and a special solution is used to determine whether the constraint expressions of the suspected cloned code pairs are equivalent. If the number of equivalent paths of the suspected cloned code pairs reaches a preset value, the suspected cloned code pairs are determined to be cloned codes.
[0023] According to the above embodiment, preferably, the step of using a special solution to determine whether the constraint expressions of the suspected clone code pair are equivalent specifically includes: extracting integers from the two sets of constraint expressions to be compared as initial seeds, and using the integers and integers near the integers as inputs of the constraint expressions to determine whether the constraint expressions are satisfiable; if for the two sets of constraint expressions, the satisfiability of more than 80% of the integer special solutions is similar, then it is determined to be an equivalent constraint, that is, the paths corresponding to the two sets of constraint expressions are equivalent paths.
[0024] According to the above embodiment, preferably, the step of screening out suspected clone code pairs through function name judgment, function parameter judgment and import and export symbol judgment specifically includes: parsing two executable files to be compared, obtaining corresponding symbol tables, screening out functions, and obtaining function signatures, wherein the function signature includes function names, function parameters and function return values; obtaining the disassembly code and program control flow graph of the code through disassembly calculation, and only performing subsequent judgment on functions with more than 5 nodes in the control flow graph; judging whether the names of the functions are the same: if the two functions have the same function name, they are added to the list of functions to be compared; detecting whether the number of parameters is the same: functions with the same number of parameters are added to the list of functions to be compared, and if the number of parameters of the two functions is different, the function pair is directly discarded; locating the corresponding function according to the import symbols and export symbols in the symbol table of the executable file, and identifying the code pairs above the similarity threshold as suspected clone code pairs by comparing the Jaccord similarity of the function names.
[0025] According to the above embodiment, preferably, symbolic execution specifically includes: obtaining a symbol table and a function signature in the symbol table; obtaining the number and types of parameters of the function to be analyzed from the function signature; constructing formal parameters with the same storage space according to the number of function parameters and the space used, and using the formal parameters as initialization conditions for symbolic execution at the function level.
[0026] According to the above embodiment, preferably, the executable file is parsed using angr, a binary analysis framework based on Python.
[0027] According to the above embodiment, preferably, the first threshold is 5 minutes; and the second threshold is 8 GB.
[0028] According to the above embodiment, preferably, when obtaining the function signature and symbol table, they can be obtained through various general disassembly tools, such as ida, Ghidra, angr, etc.
[0029] According to the above embodiment, preferably, when performing symbolic execution, it can be implemented through symbolic execution tools such as s2e.
[0030] like Figure 1As shown, according to another embodiment of the present invention, a specific implementation method of a cloud-edge collaborative detection of cloned code is also disclosed, including: performing symbol table parsing on two binary executable files to be compared to obtain a function list, and then constructing a preliminary screener to filter out a portion of code pairs that are obviously not cloned code before performing code comparison; at the same time, since the semantics of some short code snippets are too weak, the practical significance of detecting cloned code of these codes is relatively small, so these code snippets are not detected. After obtaining the function to be compared (suspected cloned code pair), its path constraint features are obtained through symbolic execution, and then the code cloning detection results are determined by constraint equivalence comparison. The specific scheme is as follows:
[0031] How to implement the initial screener:
[0032] First, for the two binary executable files to be compared, first parse their symbol tables, filter out the functions in them, and get their function signatures. This part is implemented through angr, which can help developers get the symbol table and help parse the symbols, determine which symbols are function names, and get the function signature through angr, which mainly includes the function name, function parameters and function return value.
[0033] Next, the disassembly code and program control flow graph of the code are obtained through disassembly calculation. At this time, functions with control flow graphs less than or equal to 5 nodes are screened, because most of these functions are very simple and do not have rich semantics. At this time, the remaining functions all have control flow graphs greater than 5 nodes.
[0034] Then, a simple code pair screening is performed to screen out the code pairs that may be cloned codes. First, based on the parsed function signatures, it is first determined whether the names of the two functions are the same. If the two functions have the same function name, then the two functions are very likely to be functions with similar semantics and are directly added to the list of functions to be compared. Secondly, it is detected whether the number of parameters is the same. Only functions with the same number of parameters may be function overloads or similar functions. Otherwise, the two functions have different input forms and are generally not similar functions. Therefore, if the number of parameters of the two functions is different, this function pair is directly discarded.
[0035] Next, use angr to obtain the imported and exported symbols in the symbol table of the file, which are usually the functions used by the binary file or the interfaces exposed to the outside world. By comparing the Jaccord similarity of these function names, code pairs above a certain threshold are selected as candidate code pairs (such as 0.8).
[0036] After the above function name judgment, function parameter judgment, and import and export symbol judgment, we can basically screen out the function pairs that may be code clones. For these function pairs, we will perform symbolic execution to obtain their path constraint characteristics to determine whether they are cloned code pairs. Since symbolic execution consumes a lot of resources, by using the preliminary screener, a large number of function pairs do not need to be compared, which significantly saves computing resources.
[0037] In order to solve the path explosion problem of symbolic execution, the present invention also proposes a path pruning algorithm:
[0038] The path explosion problem of symbolic execution is caused by the control flow structure and data dependencies in the program. For example, the loop structure will cause part of the code in the program to be executed repeatedly. If the number of loops is uncertain or the loop body contains branch statements, the symbolic executor will need to explore all possible paths for each loop iteration, resulting in an exponential number of paths, which exceeds the computing resources of the computer. The present invention studies how to correctly perform path pruning to obtain the most representative set of function paths for code cloning feature extraction. Specifically, the present invention performs path pruning through three factors: time factor, space factor and structural factor.
[0039] Time factor: When symbolic execution runs for more than a certain time (such as 5 minutes) on a certain node, it may be due to a program error crash or the symbolic execution constraint solver cannot solve quickly. At this time, the symbolic execution simulation of this branch is paused and the branch is abandoned.
[0040] Space factors: When symbolic execution runs on a branch, it is detected that the memory space resources used by the script exceed a certain threshold, such as 8 GB. At this time, path explosion is very likely to occur, so the symbolic execution simulation of this branch is suspended and the branch is abandoned.
[0041] Structural factors: When symbolic execution encounters a loop structure or a recursive call structure, the code inside the loop and recursive call will only run once, and then the loop and recursion will be jumped out. This processing method not only extracts the code features inside the loop and recursive call, but also does not cause the path explosion problem caused by recursion and loops.
[0042] According to the above embodiment, through the multi-factor path branch reduction algorithm, the symbolic execution at the function level is finally realized, and the symbolic execution operation can be performed on the code of the function scale without interruption and crash. The specific time and space factors are realized by checking the operating system resources and the current timestamp through the python code. The structural factors are also recorded in the python code for the executed nodes. The loop jump and call are used to delete the relevant constraints in the current constraints so that the loop is jumped out.
[0043] The specific process of path constraint extraction:
[0044] The present invention intends to implement function-level code clone detection, so it is necessary to extract function-level code features. When extracting path constraints, it is necessary to perform symbolic execution on the entire function. Specifically, the present invention will use angr as a binary symbolic execution tool. First, angr is used to obtain the symbol table and the function signature in the symbol table. The number and type of parameters of the function to be analyzed are obtained from the function signature. According to the number of function parameters and the space used, formal parameters with the same storage space are constructed, and these parameters are used as initialization conditions for symbolic execution at the function level. After symbolic execution, each function path will obtain a set of function constraints. Most of these constraints are expressions related to function parameters and have solution domains. Only some specific function parameter values can satisfy these expressions.
[0045] The specific process of constraint equivalence comparison:
[0046] The function pairs that have undergone the initial screening will be judged by comparing the path constraint features harvested by the symbolic execution of the function to determine whether the two functions are really clone code pairs. Since the constraint equivalence proof will consume huge computing resources when solving nonlinear problems, the present invention uses a special solution to determine whether two constraint expressions are equivalent. Specifically, the integers in the two sets of expressions to be compared are first extracted as initial seeds, and these integers and integers near these integers are used as inputs of the constraint expression to determine whether the constraint expression is satisfiable. If the satisfiability of most integer special solutions for two sets of constraint expressions is similar, such as above the threshold of 80%, then the two sets of path constraints are equivalent constraints, and the two paths are equivalent paths, that is, the two functions are likely to be similar functions. If most of the paths of the two functions are equivalent paths, then the two functions are similar functions, that is, clone function pairs. When performing integer extraction, it is implemented by the built-in method of the angr package of Python. When determining whether the special solution can satisfy the constraint expression, it is determined by the z3 constraint solver built into angr.
[0047] like Figure 2 As shown, according to another embodiment of the present invention, a device 200 for cloud-edge-end collaborative detection of cloned code is also disclosed, including: a memory 201, for storing program instructions; a processor 202, for calling the program instructions stored in the memory to implement the method for cloud-edge-end collaborative detection of cloned code as in the above embodiment.
[0048] According to another embodiment of the present invention, a computer-readable storage medium is disclosed, which stores program code, and the program code is used to implement the method for cloud-edge-end collaborative detection of cloned code as in the above embodiment.
[0049] According to the above-mentioned embodiments of the present invention, the present invention proposes a path pruning algorithm based on spatial factors, temporal factors, and structural factors, so as to realize symbolic execution operations at the function level; a symbolic execution scheme at the constructor level according to the function signature; an approximate comparison of the constraint expression is obtained using the integer value in the path constraint as the initial seed, so as to determine whether the functions are similar; by using the path constraint features generated by symbolic execution, code cloning is detected from a semantic level, thereby improving the accuracy of cloned code detection and improving the interpretability of the code.
[0050] All or part of the steps in the various methods of the above embodiments can be completed by controlling the relevant hardware through a program, and the program can be stored in a readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other readable medium that can be used to carry or store data.
[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A method for collaborative detection of cloned code between cloud, edge and end, characterized in that: include: Parse the two executable files to be compared, and filter out suspected clone code pairs by judging the function name, function parameter, and import and export symbol. The suspected cloned code is symbolically executed to obtain path constraint features. The following strategies are followed during symbolic execution: When the symbolic execution time on a certain node exceeds the first threshold, the symbolic execution simulation of this branch is suspended and this branch is abandoned; When symbolic execution is running on a certain branch and it is detected that the occupied memory space resources exceed the second threshold, the symbolic execution simulation of this branch is suspended and this branch is abandoned; When symbolic execution encounters a loop structure or a recursive call structure, it only runs the internal code of the loop and recursive call once, and then jumps out of the loop and recursion; According to the path constraint features obtained by symbolic execution, each function path corresponds to a set of function constraints, and a special solution is used to determine whether the constraint expressions of the suspected clone code pair are equivalent. If the number of equivalent paths of the suspected clone code pair reaches a preset value, the suspected clone code pair is determined to be a clone code; The step of using a special solution to determine whether the constraint expressions of the suspected clone code pair are equivalent specifically includes: Extract the integers in the two sets of constraint expressions to be compared as initial seeds, and use the integers and integers near the integers as inputs of the constraint expressions to determine whether the constraint expressions are satisfied; If for two sets of constraint expressions, the satisfiability of more than 80% of the integer solutions is similar, then they are determined to be equivalent constraints, that is, the paths corresponding to the two sets of constraint expressions are equivalent paths; The function name determination specifically includes: Parse two executable files to be compared, obtain corresponding symbol tables, filter out functions, and obtain function signatures, wherein the function signatures include function names, function parameters, and function return values; Obtain the disassembled code and program control flow graph of the code through disassembly calculation, and only perform subsequent judgment on the functions with more than 5 nodes in the control flow graph; Determine whether the function names are the same: If two functions have the same function name, add them to the list of functions to be compared; The function parameter judgment specifically includes: Check whether the number of parameters is the same: functions with the same number of parameters are added to the list of functions to be compared. If the number of parameters of two functions is different, the function pair is directly discarded; The import and export symbol judgment specifically includes: According to the import symbols and export symbols in the symbol table of the executable file, the corresponding functions are located. By comparing the Jaccord similarity of the function names, the code pairs with a similarity higher than the threshold are identified as suspected clone code pairs. The symbolic execution specifically includes: Get the symbol table and the function signature in the symbol table; Get the number and type of parameters of the function to be analyzed from the function signature; According to the number of function parameters and the space used, formal parameters with the same storage space are constructed, and the formal parameters are used as initialization conditions for symbolic execution at the function level.
2. The method for cloud-edge-end collaborative detection of cloned code according to claim 1, characterized in that: The Python-based binary analysis framework angr is used to parse the executable file.
3. The method for cloud-edge-end collaborative detection of cloned code according to claim 1, characterized in that: The first threshold is 5 minutes; the second threshold is 8 GB.
4. A device for collaborative detection of cloned code between cloud and edge, characterized in that: include: A memory for storing program instructions; A processor, used to call the program instructions stored in the memory to implement the method for cloud-edge collaborative detection of cloned code as described in any one of claims 1 to 3.
5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code is used to implement the method for cloud-edge collaborative detection of cloned code as described in any one of claims 1 to 3.
