A method for locating software non-repeatable compilation faults and automatically generating patches

Abstract

The present invention belongs to the field of software compilation, and is specifically a method for locating software non-repeatable compilation faults and automatically generating patches, including: constructing a compilation environment, obtaining a source code package to generate artifacts; judging whether the artifacts generated twice are consistent; calling a compilation tracking program to track the compilation process of the source code package; locating non-repeatable compilation faults; obtaining historical repair patches and automatically generating repair patches for software source code packages; repairing software source code packages; and repairing software soundness detection. The method of the present invention can locate software non-repeatable compilation faults and automatically generate repair patches, avoiding the tedious work of developers manually locating and repairing in previous repair work, and greatly saving time.

Description

Technical Field

[0001] The invention belongs to the field of software compilation, and relates to a technology for locating software non-repeatable compilation faults and automatically generating fault repair patches, in particular to a software non-repeatable compilation fault locating and patch automatic generation method. Background Art

[0002] With the rapid development of network technology and computer technology, people have more and more ways to obtain open source software, such as various application stores, resource websites, etc. Ensuring the security and credibility of the obtained software has become an increasingly serious problem. Therefore, the open source community led by Debian verifies the credibility of the software package by ensuring that the distributed software package can be recompiled. Reproducible compilation means that any party can generate a bit-for-bit identical copy of a specific artifact by compiling the same source code under the same compilation environment using the same compilation instructions.

[0003] Although some existing papers and invention patents can be used to repair software faults, their fault location accuracy and repair methods still have many deficiencies in software non-repeatable compilation faults. For example, the automatic repair method, system and medium for software code quality problems based on code change history (patent application number: CN201911338355.0) first obtains the code of the target to be repaired; then finds the matching repair method from the repair pattern library based on the code change history; and finally generates a repair patch for the target code. However, this method does not target the repair of software non-repeatable compilation faults. In 2018, Ren et al. proposed the RepLoc framework in the paper Non-repeatable compilation location. This framework extracts information from the build log, narrows the information search scope through heuristic component filtering, and provides a list of possible files. However, this framework is highly dependent on the build log, so it can only analyze high-level build commands, and the integrity of the information in the build log will also affect the efficiency of the framework. In 2019, Ren et al. further proposed a non-repeatable compilation location method RrpTrace based on system call logs and applied for a related patent (patent application number: CN201910549431.6). This patent first compiles the dependency graph by differential analysis of the system call log information; then enhances the dependency graph by analyzing the parent-child process relationship during the compilation process; finally, by traversing the dependency graph and sorting it, the final problematic compilation command is obtained for subsequent repair work. This patent can effectively verify the correspondence between the software source code and the binary package. Once an inconsistency occurs, it can locate the compilation command that may cause the software to be non-repeatable. However, this method has high requirements for storage space during the tracing process, and can only reach the file level, and the positioning granularity needs to be improved.

[0004] In summary, the current software repeatable compilation fault location and repair methods have the following two problems: first, the positioning granularity is not fine enough, and the existing methods can only achieve file-level positioning, and the required storage space is large; second, the current software package repeatable compilation fault repair method is mainly to modify the system call or command line parameters during the execution process, and the accumulated repair experience has not been utilized, and there is a lack of automatic generation of source code package repair patches. Summary of the invention

[0005] The purpose of the present invention is to provide a method for locating software non-repeatable compilation faults and automatically generating patches. The present invention is conducive to improving the granularity of locating software non-repeatable compilation faults, reducing storage space, and automatically generating fault repair patches.

[0006] The technical solution of the present invention:

[0007] A method for locating a software non-repeatable compilation fault and automatically generating a patch, the specific steps are as follows:

[0008] Step 1: Build two compilation environments 1 and 2 with differences.

[0009] Specifically, the compilation environment consists of a fixed part and a perturbation part. The fixed part remains unchanged in the two compilation environments, including the compiler type, related dependencies required for compilation, etc. The perturbation part refers to the difference between the two compilation environments, including the language environment, time, compilation directory, etc.

[0010] Step 2: Compile the source code package in compilation environment No. 1 and No. 2 respectively to obtain artifact No. 1 and artifact No. 2.

[0011] Specifically, the source code package is compiled according to the compilation rules in compilation environment No. 1 and compilation environment No. 2. The artifact generated by the compilation in compilation environment No. 1 is generated artifact No. 1, and the artifact generated by the compilation in compilation environment No. 2 is generated artifact No. 2.

[0012] Step 3: Determine whether the artifact generated in No. 1 is consistent with the artifact generated in No. 2.

[0013] Specifically, use the tool diffoscope to recursively compare the generated artifact No. 1 and the generated artifact No. 2 at the bit level to determine whether the two are completely consistent. If the comparison results are completely consistent, the software can be compiled repeatedly, the consistency result is reported, and the process terminates. If the comparison results are inconsistent, the output file of the comparison tool is used as a difference log, and the file name used as the basis for fault location is extracted from the difference log; a difference file list file is generated based on the extracted file name list, and the file is used as the input of step 6.

[0014] The diffoscope tool is developed by the Reproducible builds community and is free software licensed under the GNU General Public License version 3. It is available on multiple platforms. The tool can compare binary files in a more understandable form and can generate comparison files in multiple formats.

[0015] Step 4: Call the suspected system call process propagation graph generation program to generate the suspected system call process propagation graph.

[0016] Specifically, if a system call can generate content that causes the software package to be non-reproducible, the system call is called a suspected system call, and the content that causes the software package to be non-reproducible is called suspected content. During the source code package compilation process, the suspected system call process propagation graph generation program is called to monitor all system calls during the source code package compilation process with ptrace (process trace) system call as the core, generate a suspected system call process propagation graph, and use the graph as the input of step 6.

[0017] The overall process is as follows: record all suspected system calls and the process information that issued the suspected system call; track read and write operations, record the propagation path of suspected content between processes; generate a suspected system call process propagation graph. The nodes in the graph are custom process information classes, including the process number that issued the suspected system call, the suspected system call issued, the suspected content, the written file containing the suspected content, and the read file containing the suspected content. The edges in the graph are the propagation directions of the suspected content between processes.

[0018] Ptrace is a system call for process tracing in Unix and some Unix-like operating systems. It provides the ability for a parent process to observe and control the execution of its child processes, and allows the parent process to inspect and replace the values ​​of the child process kernel image (including registers). On Windows platforms, system call monitoring can be implemented using the Debugging API.

[0019] Step 5: Call the compilation tool information extraction program to generate a log file of the correspondence between the process and the file line number.

[0020] Specifically, when the source code package is compiled, the parameter value of a specific function is obtained through breakpoints according to the debugging symbol table of the compilation tool used, and the information of the running process and the compilation command executed by the process are obtained. Based on the obtained process information and compilation command information, a log file of the correspondence between the process and the file line number can be generated, and the graph is used as the input of step 6.

[0021] Step 6: Locate the non-repeatable compilation fault and obtain the compilation command that causes the software non-repeatable compilation fault.

[0022] Specifically: obtain the difference file list file generated in step 3, the suspected system call process propagation graph generated in step 4, and the process and compiled script file line number correspondence log file generated in step 5. Search in the suspected system call process propagation graph, match the write file name of the node in the graph containing the suspected content, and obtain the process p1 that writes the difference file. Find the process p2 that issued the suspected system call according to the search path of the system call process propagation graph; search in the process and compiled script file line number correspondence log to locate the code line and compilation command that executes process p2 in the compiled script.

[0023] Step 7: Get the historical fix patch and get the source and target commands.

[0024] Specifically, according to the compilation command located in step 6, obtain the historical repair patch that has been released and has a similar fault to the non-repeatable compilation fault of the software. Use the script to extract the repair content from the patch, that is, the compilation command before the repair and the compilation command after the repair. The compilation command before the repair is called the source command, and the compilation command after the repair is called the target command.

[0025] Step 8: Convert the source command and target command into AST to obtain the repair node.

[0026] According to the syntax analysis of the compilation principle, both the source command and the target command are converted into AST (Abstract SyntaxTree). In the AST, CommandNode is the root node, WordNode and RedirectNode are the intermediate nodes, and the parsed command content is the leaf node. Compare ASTT1 converted from the source command and ASTT2 converted from the target command. The modified ASTT2 leaf nodes are the repaired nodes, and the content of the leaf nodes is the repaired command content.

[0027] AST (Abstract Syntax Tree) refers to an abstract syntax tree, which is an abstract representation of the grammatical structure of the source code.

[0028] Step 9: Repair the AST corresponding to the command obtained by locating, and compile the command after the repair.

[0029] Specifically: according to the syntax analysis of the compilation principle, the command located in the compilation script in step 6 is converted into the corresponding ASTT3. The repair node transformation of AST T2 obtained in step 8 is applied to AST T3 to obtain the ASTT4 after the command located and repaired, and the ASTT4 after the command located and repaired is converted back into the compilation command, which is the compilation command after repair.

[0030] Step 10: Generate a repair patch and repair the software source package.

[0031] Specifically, according to the compilation principle, the compilation command after repair in step 8 is used to generate a repair patch for the software non-repeatable compilation failure. According to the generated repair patch, the compilation script is repaired.

[0032] Step 11: Fix package sanity check.

[0033] Specifically, the source code package after repair is compiled. If the source code package can be compiled normally, and the generated artifacts are not missing compared with the artifacts generated by compiling the source code package before repair; and the generated artifacts can function normally, it means that the software package is sound, and the software package can be tested for repeatable compilation in step 1. If it passes the test, the automatically generated repair patch is returned. If the repaired software package fails the soundness test, return to step 7 to regenerate the patch.

[0034] Beneficial effects of the present invention: The method of the present invention can locate software non-repeatable compilation faults and automatically generate repair patches, avoiding the tedious work of developers manually locating and repairing in previous repair work, thereby greatly saving time. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention is a schematic diagram of the workflow of a method for locating software non-repeatable compilation faults and automatically generating patches. DETAILED DESCRIPTION

[0036] The method of the present invention is described in detail below in conjunction with the accompanying drawings, technical solutions and examples.

[0037] In this embodiment, the method of the present invention is deployed on a Linux server, and the specific configuration of the server is shown in Table 1. This embodiment is described by taking the software package when as an example, and requires the source code package of when and related dependencies such as the dpkg package manager, the make compilation tool, the diffoscope comparison tool, and the reprotest compilation tool. The method of the present invention is composed of programs such as a source code package twice compilation comparison program, a compilation tracking program, and a repair patch generation program.

[0038] Table 1: Linux server configuration information table

[0039] Processor model Memory operating system Intel Core i9-10900 32GB Ubuntu 18.04.01

[0040] like Figure 1As shown, a method for locating software non-repeatable compilation faults and automatically generating patches of the present invention is performed according to the following process: constructing a compilation environment, obtaining a source code package generation artifact; determining whether the two generated artifacts are consistent; calling a compilation tracking program to track the compilation process of the source code package; locating non-repeatable compilation faults; obtaining historical repair patches and automatically generating repair patches for the software source code package; repairing the software source code package; and repairing software soundness detection.

[0041] The steps in this embodiment are described using ptrace on the Linux platform as an example, and the specific steps can be adjusted according to the platform. The specific steps are as follows:

[0042] Step 1: Build two compilation environments 1 and 2 with differences.

[0043] Specifically: build two differentiated compilation environments No. 1 and No. 2.

[0044] In this embodiment, the compilation directory in the compilation environment No. 1 is b1, the time zone is UTC-8 (PST—Pacific Standard Time), and the time starts at 0:00. The compilation directory in the compilation environment No. 2 is b2, UTC+8 (CT—China Standard Time), and the time starts at 12:00.

[0045] Step 2: Compile the source code package in compilation environment No. 1 and No. 2 respectively to obtain artifact No. 1 and artifact No. 2.

[0046] Specifically, in compilation environment No. 1 and compilation environment No. 2, the source code package is compiled according to the compilation rules to obtain generated artifact No. 1 and generated artifact No. 2.

[0047] In this embodiment, the build command used is as follows: 'dpkg-buildpackage-us-uc', which completes the compilation of the when software package twice. The generated artifact No. 1 is saved in the directory b1, and the generated artifact No. 2 is saved in the directory b2.

[0048] dpkg (Debian Packager) is the package management system of "Debian", which facilitates the installation, update and removal of software. All "Linux" distributions derived from "Debian" use "dpkg", such as "Ubuntu", "Knoppix", etc.

[0049] Step 3: Determine whether the artifact generated in No. 1 is consistent with the artifact generated in No. 2.

[0050] Specifically: Use the tool diffoscope to perform a bit-level recursive comparison of artifact No. 1 and artifact No. 2 to determine whether the two are completely consistent. In this embodiment, the command diffoscope-jsondiff.json b1 b2 is used to complete the bit-level recursive comparison of all artifacts generated in b1 and b2. If the comparison results of b1 and b2 are inconsistent, the output file of the comparison tool is diff.json. Use a script to extract the file name / usr / share / man / man1 / when.1.gz as the basis for fault location from the diff.json file, and generate a difference file list file difffiles.json.

[0051] Step 4: Call the suspected system call process propagation graph generation program to generate the suspected system call process propagation graph.

[0052] Specifically, during the source code package compilation process, the suspected system call process propagation graph generation program is called to monitor all system calls during the source code package compilation process with the ptrace (process trace) system call as the core, and generate a suspected system call process propagation graph. In this embodiment, the suspected system call process propagation graph generation program is called to track the execution of the 'dpkg-buildpackage-us-uc' command to complete the compilation process tracking of the when software package. A suspected system call process propagation graph G1 is generated.

[0053] Step 5: Call the compilation tool information extraction program to generate a log file of the correspondence between the process and the file line number.

[0054] Specifically, when the source code package is compiled, the parameter value of a specific function is obtained through breakpoints, the running process information and the compilation command information executed by the process are obtained, and a log file of the correspondence between the process and the file line number is generated.

[0055] In this embodiment, the make compilation tool is used, make-dbgsym with debugging information is installed, the start_waiting_job and job_next_command function execution of make is interrupted, the function parameters are obtained, and the corresponding relationship log file correspond.json between the process and the file line number is generated.

[0056] The make compilation tool is responsible for generating the final executable file and other non-source code files from the project's source code.

[0057] Step 6: Locate the non-repeatable compilation fault and obtain the compilation command that causes the software non-repeatable compilation fault.

[0058] Specifically: In this embodiment, the difference file list file difffiles.json generated in step 3 is obtained, the suspected system call process propagation graph G1 is generated in step 5, and the process and compiled script file line number correspondence log file correspond.json is generated in step 5. The difference file list file difffiles.json is used as input, and a search is performed in the suspected system call process propagation graph G1, and the write files containing suspected content in the nodes in the graph are matched by regular matching, so as to obtain the process 84 that writes the difference file. According to the search path of the system call process propagation graph, the process 63 that issued the suspected system call is found. The process number 63 is used as input, and the process and compiled script file line number correspondence log correspond.json is searched to locate line 49 in the makefile file of the when source code package. The build command is gzip-9<when.1> when.1.gz.

[0059] Step 7: Get the historical fix patch and get the source and target commands.

[0060] Specifically: the command located in this embodiment is gzip-9<when.1> when.1.gz. Through the script, we searched the patch released by Debain and found that the patch of rockdodger software was available. We downloaded the patch of rockdodger and extracted the patch contents. The patch contents of rockdodger are as follows:

[0061] -gzip-9debian / tmp / usr / share / doc / rockdodger / changelog.Debian

[0062] +gzip-9n debian / tmp / usr / share / doc / rockdodger / changelog.Debian

[0063] The former is the source command and the latter is the target command.

[0064] Step 8: Convert the source command and target command into AST to obtain the repair node.

[0065] In this embodiment: the source command and the target command are converted into AST T1 and T2, T1 and T2 both correspond to three WorldNode nodes, T1 is gzip, -9, debian / tmp / usr / share / doc / rockdodger / changelog.Debian, T2 is gzip, -9n, debian / tmp / usr / share / doc / rockdodger / changelog.Debian. Comparing the two AST trees, it can be seen that the second WorldNode node of T2 is a repair node.

[0066] Step 9: Repair the AST corresponding to the command obtained by locating, and compile the command after the repair.

[0067] Specifically: The compilation command obtained by locating the when package non-repeatable compilation fault is gzip-9<when.1> when.1.gz, convert the command into ASTT3, T3 contains two WordNode nodes and two RedirectNode nodes, namely gzip, -9,<when.1,> when.1.gz.. Apply the repair content transformation in the rockdodger repair patch to the command-converted ASTT4 obtained by locating the when package's non-repeatable compilation fault. The nodes of T4 are two WordNode nodes and two RedirectNode nodes, which are gzip, -9n,<when.1,> When.1.gz. Convert the ASTT4 command after positioning and repair to the compilation command to get gzip-9n<when.1> when.1.gz.

[0068] Step 10: Generate a repair patch and repair the software source package.

[0069] In this example, the repair patch when.sh is generated by modifying the command line 49 of the makefile file in the when software package to gzip-9n<when.1> when.1.gz.

[0070] Step 11: Fix package sanity check.

[0071] Specifically: In this embodiment, the repaired when source code package is compiled, the source code package is successfully compiled, and the compiled artifacts are saved in the file directory b3, wherein the files / usr / share / man / man1 / when.1.gz, / usr / bin / when, / usr / share / doc / when / NEWS.Debian.gz, / usr / share / doc / when / copyright, / usr / share / doc / when / changelog.Debian.gz contained therein are the same as the artifacts compiled and generated by the unrepaired when source code package. There is no functional difference in the compiled artifacts, and step 1 is entered to perform repeatable compilation detection on the repaired source code.

[0072] The repaired when software package is tested for repeatable compilation. The artifacts generated by the two compilations are consistent, indicating that the repair patch is effective and the repair work of the when software package is completed.

[0073] The method of the present invention effectively reduces the consumption of storage space during the tracking process; improves the granularity of fault location, and realizes file line-level location; automatically generates a repair script based on historical repair information to complete the source code package repair, greatly reducing the investment of time and manpower and effectively saving costs.

Claims

1. A method for locating software non-repeatable compilation faults and automatically generating patches, characterized in that: Here are the steps: Step 1: Build two differentiated compilation environments 1 and 2; The compilation environment consists of a fixed part and a perturbation part. The fixed part remains unchanged in the two compilation environments, including the compiler type and related dependencies required for compilation. The perturbation part refers to the difference between the two compilation environments, including the language environment, time, and compilation directory. Step 2: Compile the source code package in compilation environment No. 1 and No. 2 respectively to obtain artifact No. 1 and artifact No. 2; Under compilation environment No. 1 and compilation environment No. 2, the source code package is compiled according to the compilation rules; the artifact generated by the compilation under compilation environment No. 1 is generated artifact No. 1, and the artifact generated by the compilation under compilation environment No. 2 is generated artifact No. 2; Step 3: Determine whether the artifact generated by No. 1 is consistent with the artifact generated by No. 2; Use the tool diffoscope to recursively compare the artifacts generated in No. 1 and No. 2 at the bit level to determine whether the two are completely consistent; If the comparison results of the two are completely consistent, the software can be compiled again, the consistency results are reported, and the process is terminated; if the comparison results of the two are inconsistent, the output file of the comparison tool is used as a difference log, and the file name used as the basis for fault location is extracted from the difference log; a difference file list file is generated based on the extracted file name list, and the file is used as the input of step 6; Step 4: Call the suspected system call process propagation graph generation program to generate the suspected system call process propagation graph; If a system call can generate content that causes the software package to be unrecompilable, the system call is called a suspected system call, and the content that causes the software package to be unrecompilable is called suspected content. During the compilation of the source code package, call the suspected system call process propagation graph generation program, monitor all system calls in the compilation process of the source code package with ptrace system call as the core, generate a suspected system call process propagation graph, and use the graph as the input of step 6; the overall process is as follows: record all suspected system calls and the process information that issues the suspected system call; track read and write operations, and record the propagation path of suspected content between processes; generate a suspected system call process propagation graph; the nodes in the graph are custom process information classes, including the process number that issues the suspected system call, the suspected system call issued, the suspected content, the write file containing the suspected content, and the read file containing the suspected content. The edges in the graph are the propagation directions of the suspected content between processes; Step 5: Call the compilation tool information extraction program to generate a log file of the correspondence between the process and the file line number; When the source code package is compiled, according to the debugging symbol table of the compilation tool used, the parameter value of the specific function during runtime is obtained through the breakpoint, and the running process information and the compilation command information executed by the process are obtained; according to the obtained process information and compilation command information, a log file of the correspondence between the process and the file line number is generated, and the graph is used as the input of step 6; Step 6: Locate the non-repeatable compilation fault and obtain the compilation command that causes the software non-repeatable compilation fault; Get the difference file list file generated in step 3, the suspected system call process propagation graph generated in step 4, and the process and compiled script file line number correspondence log file generated in step 5; search in the suspected system call process propagation graph, match the write file name of the node in the graph containing the suspected content, and obtain the process p1 that writes the difference file; find the process p2 that issues the suspected system call according to the system call process propagation graph search path; search in the process and compiled script file line number correspondence log to locate the code line and compilation command that executes process p2 in the compiled script; Step 7: Get the historical repair patch, source command and target command; According to the compilation command located in step 6, obtain a historical repair patch that has a similar fault to the non-repeatable compilation fault of the software and has been released; use a script to extract the repair content from the patch, where the repair content is the compilation command before the repair and the compilation command after the repair; The compilation command before repair is called source command, and the compilation command after repair is called target command; Step 8: Convert the source command and target command into AST to obtain the repair node; According to the syntax analysis of the compilation principle, both the source command and the target command are converted into AST. In the AST, CommandNode is the root node, WordNode and RedirectNode are the intermediate nodes, and the parsed command content is the leaf node. Compare ASTT1 converted from the source command and ASTT2 converted from the target command. The modified ASTT2 leaf nodes are the repaired nodes, and the content of the leaf nodes is the repaired command content. Step 9: Repair the AST corresponding to the command obtained by locating, and compile the command after repair; According to the syntax analysis of the compilation principle, the commands located in the compilation script in step 6 are converted into corresponding ASTT3; Apply the repair node transformation of AST T2 obtained in step 8 to AST T3 to obtain ASTT4 after the command obtained by positioning is repaired, and convert the command obtained by positioning and repairing ASTT4 into a compilation command again, which is the compilation command after repair; Step 10: Generate a repair patch and repair the software source code package; According to the compilation principle, the compilation command repaired in step 8 is used to generate a repair patch for the software non-repeatable compilation failure; and the compilation script is repaired according to the generated repair patch; Step 11: Repair the package sanity check; Compile the repaired source package. If the source package can be compiled normally and the generated artifacts are not missing compared with the artifacts compiled by the source package before the repair; and the generated artifacts can function normally, it means that the software package is sound. Then the software package can be tested for repeatable compilation in step 1. If it passes the test, the automatically generated repair patch is returned. If the repaired software package fails the soundness test, return to step 7 to regenerate the patch.

Images