Automatically implement incremental code walkthrough system and incremental code walkthrough method based on DevOps platform

Abstract

The present invention relates to an incremental code walkthrough system and an incremental code walkthrough method that are automatically implemented based on a DevOps platform, and belongs to the technical field of code walkthrough. The incremental code walkthrough system includes a DevOps platform, a web server, a file server, a SonarQube code quality analysis module, and a Jenkins server; the Jenkins server integrates the DevOps platform, the web server, the file server, and the SonarQube code quality analysis module, and then implements incremental code walkthrough through a Jenkins pipeline and a script file. The present invention can achieve code walkthrough coverage of any range while effectively simplifying the code walkthrough workflow through the interaction between walkthrough personnel and front-end pages, and fundamentally improve the quality of code walkthrough by combining automated static scanning.

Description

Technical Field

[0001] The present invention relates to an incremental code walking system and method, and in particular to an incremental code walking system and method for automatically implementing incremental code walking based on a DevOps platform, which belongs to the technical field of code walking technology. Background Art

[0002] Code review is a necessary task for every company engaged in software development and maintenance. Currently, there are two ways to implement code review in the industry:

[0003] The first is to submit the code manually, but the code volume is large, and the code branch permissions of developers are not open to code checkers, which means that developers need to manually move the code to the specified path for code check. Faced with a large number of files and complex paths, the result of manual extraction is that the spot check files are not comprehensive, the code check scope is not complete, the code quality feedback is not true, the effect is not ideal, the spot check results are not recorded, the quality problems are difficult to track later, and the process is not closed-loop. If all the codes are checked, it will be very time-consuming, and the same code will be checked repeatedly, resulting in a waste of resources.

[0004] The second method is to use the Jenkins plug-in to pull the full code. Install the SVN plug-in in Jenkins and configure it accordingly, including setting the address of the SVN server, authentication information, etc. Create a new task in Jenkins and select the build type as "Freestyle project" or other suitable task types. In the configuration page of the task, select the source code management as SVN and fill in the corresponding SVN warehouse address, credentials and other information. In the build step of the task, you can configure the execution of specific commands or scripts to pull the full code. Usually, you can use the SVN command line tool or the steps provided by the SVN plug-in of Jenkins to pull the code. Configure the build trigger as needed, such as timed triggering, polling the SVN warehouse, etc., to determine when to pull the code. Save the task configuration and manually trigger the task to run or set it to automatically trigger. Jenkins will pull the full code from the SVN warehouse according to the configured steps, but this will take up more storage space, especially when frequently building or pulling a large amount of historical code. Pulling the full code may cause insufficient server storage space. As the company develops, the amount of code increases, the pulling time is getting longer and longer, and it is not flexible enough.

[0005] The Chinese invention patent with the authorization announcement number CN111309441B discloses a microservice deployment method based on Jenkins to implement DevOps. It uses the gitlab+jenkins pipeline to complete automatic construction and packaging through advance standardization and unified configuration, and generates a helm chart that can manage images and configuration files, so as to be deployed to various environments, greatly reducing the cost of human communication and the possibility of misoperation, and improving efficiency. The patent discloses a deployment method based on Jenkins and DevOps, and does not disclose a code walkthrough method. Therefore, the solution is not suitable for the optimization of the code walkthrough method and cannot solve the problems in this application. Summary of the invention

[0006] Technical purpose: In order to overcome the shortcomings of the prior art, the present invention provides an incremental code walking system and an incremental code walking method based on a DevOps platform, which can automatically extract incremental code and perform code walking to generate a walking report through the interaction between the walking personnel and the front-end page. At the same time, the incremental code and directory structure are compressed and stored on the file server for download by the walking personnel, which simplifies the work while ensuring the comprehensiveness of the code walking and improving the overall code quality.

[0007] Technical solution: To achieve the above-mentioned purpose, the present invention discloses a system for automatically implementing incremental code walkthrough based on a DevOps platform, including a DevOps platform, a web server, a file server, a SonarQube code quality analysis module and a Jenkins server; the Jenkins server integrates the DevOps platform, the web server, the file server, and the SonarQube code quality analysis module, and then implements incremental code walkthrough through the Jenkins pipeline and script files.

[0008] The incremental code walkthrough method for automatically implementing the incremental code walkthrough system based on the DevOps platform includes the following steps:

[0009] Step S1, the DevOps platform obtains code extraction parameters according to user input, generates corresponding task instructions, and securely forwards these instructions to the backend service through the web server;

[0010] Step S2: The Jenkins server receives the backend service parameters and configures the corresponding Jenkins pipeline task to execute. The Jenkins pipeline task triggers the incremental code extraction script according to the predefined script and parameters.

[0011] Step S3: The incremental code extraction script runs in the Jenkins pipeline;

[0012] Step S4: The Jenkins pipeline sends the incremental code extracted in step S3 to the SonarQube code quality analysis module through a secure data transmission protocol. After the Jenkins pipeline executes the shell script to compress the incremental code and directory structure, the file storage module securely stores the compressed files in the file server for downloading by the inspection personnel.

[0013] Step S5: After receiving the incremental code, the SonarQube code quality analysis module uses its built-in code review rules and algorithms to perform in-depth static analysis on the incremental code and generate a walkthrough report;

[0014] Step S6, the DevOps platform will feed back the walkthrough report to the user through the DevOps user interface module. If there are problems in the walkthrough report, the developer can optimize and repair the code based on this information, and resubmit the code to the SVN version control system after optimization and repair. After submission, re-execute step S1 to perform a new round of incremental code scanning, and repeat the above operations until the results are correct.

[0015] Furthermore, in step S3, the incremental code extraction script logic is:

[0016] (1) Based on the extraction parameters passed in by the user, use the svn diff command to obtain the XML file of the SVN changes;

[0017] (2) Use Python script to extract the list of changed files and filter out the SVN file paths that only have modification operations;

[0018] (3) Traverse this file list, and for each file:

[0019] a) Use the svn info command to determine whether the file exists in the latest version;

[0020] b) Obtain the user ID of the user who last modified the file from the specified time to the present, and obtain the user name by combining it with the user mapping table;

[0021] c) Use the svn export command to download the file;

[0022] d) Determine whether it is a link file, if so, delete it;

[0023] e) Use grep, sed, and awk commands to calculate the total number of lines and the number of valid lines in a file;

[0024] f) Use svn diff to obtain all changes of the file from the specified time to the present, and calculate the number of valid modified lines;

[0025] g) Filter out files with a low number of modified lines or a low modification rate based on user selection;

[0026] h) Construct the directory structure according to the relative path of the file on the SVN server;

[0027] i) Integrate the modified file information into json and return it to the user.

[0028] Furthermore, in step S1, the extraction parameters include but are not limited to version number, file path, extraction range, file type, time, configuration library address and account number.

[0029] Furthermore, in step S4, the storage process includes data encryption, access control and version management.

[0030] Furthermore, in step S5, the static analysis includes but is not limited to code complexity assessment, potential error detection and code style checking.

[0031] Furthermore, the valid number of lines is the number of lines after removing blank lines and blank characters, and the valid modified number of lines is the sum of the number of newly added or modified lines after removing blank lines and blank characters.

[0032] Beneficial effect: The present invention integrates the DevOps platform, web server, file server, and SonarQube code quality analysis module through the Jenkins server, helping the inspectors to automate the incremental code inspection work, reducing the workload of the inspectors, alleviating the storage space pressure of the server due to the full code extraction, and improving work efficiency; the method can achieve code inspection coverage of any range while effectively simplifying the code inspection workflow through the interaction between the inspectors and the front-end page, and combined with automated static scanning, fundamentally improve the quality of code inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0034] Figure 2 The figure is a schematic diagram of the overall process of incremental code walkthrough in the present invention. DETAILED DESCRIPTION

[0035] The following is combined with Figure 1 To Attachment Figure 2 The principles and features of the present invention are described, and the examples given are only used to explain the present invention, rather than to limit the scope of the present invention.

[0036] An incremental code walkthrough system is automatically implemented based on a DevOps platform, including a DevOps platform, a web server, a file server, a SonarQube code quality analysis module, and a Jenkins server; the Jenkins server integrates the DevOps platform, a web server, a file server, and the SonarQube code quality analysis module, and then implements incremental code walkthrough through Jenkins pipelines and script files.

[0037] The incremental code walkthrough method for automatically implementing the incremental code walkthrough system based on the DevOps platform includes the following steps:

[0038] Step S1, the DevOps platform obtains code extraction parameters according to user input, generates corresponding task instructions by splicing these parameters on the front end, and securely forwards these instructions to the backend service through the web server HTTP POST request;

[0039] Step S2: The Jenkins server receives the backend service parameters and configures the corresponding Jenkins pipeline task to execute. The Jenkins pipeline task triggers the incremental code extraction script according to the predefined script and parameters, specifically:

[0040] After receiving the parameters from the backend service through the Jenkins REST API interface, the Jenkins server replaces the execution parameters into the predefined Jenkins pipeline XML-type task template, and then sends the replaced XML-type task template information to the Jenkins server through the Create Task Jenkins REST API interface provided by the Jenkins service; After receiving the XML-type task template information, the Jenkins server obtains the XML-type task template information through the Jenkins REST API interface, and then the server-side Python script uses the xml.etree.ElementTree library to parse the XML configuration file, extract the parameters and integrate them into the predefined Jenkins task configuration, which includes all the information and parameters required to create a Jenkins pipeline, such as basic information such as pipeline name, source code information such as SVN address and credentials, build parameters, build environment and other parameters; then call the Jenkins REST API interface to create the corresponding pipeline task, and at the same time, the Jenkins server defines the code extraction parameters entered by the user in the Jenkinsfile using the parameters block and sends them to the incremental code extraction script through an HTTP POST request. After the pipeline task configuration is completed, the backend service executes the created Jenkins pipeline task by sending an HTTP POST request through the Execute Task Jenkins REST API interface provided by the Jenkins service and triggers the incremental code extraction script through the Jenkinsfile;

[0041] Step S3: The incremental code extraction script runs in the Jenkins pipeline;

[0042] The incremental code extraction script logic is:

[0043] (1) Based on the time period and SVN repository address entered by the user, use the svn diff command to obtain the XML file of the SVN changes;

[0044] (2) Use Python script to extract the list of changed files and filter out the SVN file paths that only have modification operations;

[0045] (3) Traverse this file list, and for each file:

[0046] a) Use the svn info command to determine whether the file exists in the latest version, because the latest version will be used when extracting the file content and calculating the number of lines later;

[0047] b) Obtain the ID of the user who last modified the file from the specified time to the present, and obtain the user name by combining it with the user mapping table;

[0048] c) Use the svn export command to download the file;

[0049] d) Determine whether it is a link file, if so, delete it;

[0050] e) Calculate the total number of lines and the number of valid lines in the file by using grep, sed, or awk commands, where the valid number of lines is the number of lines after removing blank lines and blank characters;

[0051] f) Use svn diff to obtain all changes to the file from the specified time to the present, and calculate the number of valid modified lines, which is the sum of the number of new or modified lines after removing blank lines and blank characters;

[0052] g) Filter out files with a low number of modified lines or a low modification rate based on user selection;

[0053] h) Construct the directory structure according to the relative path of the file on the SVN server;

[0054] i) Integrate the modified file information into json and return it to the user;

[0055] Step S4, the Jenkins pipeline sends the incremental code extracted in step S3 to the SonarQube code quality analysis module through the HTTP protocol through a secure data transmission protocol. The Jenkins pipeline executes a shell script containing the tar and mcconfig host add commands to compress the incremental code and directory structure extracted in step S3. The file storage module stores the compressed files in the file server through the HTTP protocol for download by the inspection personnel. The specific download method is: by writing the URL address of the file server into the DevOps platform code in advance, by clicking the file server link provided by the DevOps platform, jump to the file server for download; the storage process includes data encryption, access control and version management to ensure the security, traceability and long-term preservation of the code;

[0056] Step S5, after receiving the incremental code extracted from the Jenkins pipeline in step S4 through the HTTP protocol, the SonarQube code quality analysis module uses its built-in code review rules and algorithms to perform in-depth static analysis on the incremental code and generate a walkthrough report, wherein the rules cover various programming languages ​​and code quality issues, such as code complexity, potential bugs, code style, duplicate code, etc., and the algorithms are used to calculate code metrics, such as cyclomatic complexity, code coverage, etc.; the static analysis includes but is not limited to code complexity assessment, potential error detection, and code style inspection to identify and report potential quality issues in the code;

[0057] The static analysis is implemented by SonarScanner of SonarQube code quality analysis module, which analyzes code structure and code quality without executing the code. SonarScanner uses a series of predefined or customized rules to check the code and generate reports. These rules cover code standards, potential defects, complexity, duplicate code and other aspects;

[0058] Step S6, the DevOps platform provides a link to the address of the project corresponding to the SonarQube code quality analysis module, and feeds back the inspection report to the user through the DevOps user interface module. If there are problems in the inspection report, the developer can optimize and repair the code based on this information. For optimization, the developer uses IDE, such as IntelliJ IDEA's refactoring function, to optimize the code based on these problem reports. The code can be optimized by simplifying the code logic, reducing conditional judgments, merging similar code blocks, reducing nesting depth, etc. to reduce code complexity; for repair, the developer will follow the problem repair suggestions provided by the SonarQube code quality analysis module, and use code editing tools such as VScode to manually modify the code. After the developer completes the code optimization and repair, the code will be resubmitted to the SVN version control system. After the submission is completed, step S1 will be re-executed to perform a new round of incremental code scanning, and the above operations will be repeated until the result is correct.

[0059] The advantages of this incremental code review method are as follows:

[0060] (1) Process automation: One-click execution of code walkthrough through the DevOps platform;

[0061] (2) Efficiency: Helps code reviewers quickly obtain the files needed for code review;

[0062] (3) Accuracy: Based on SVN submission history and rule-based methods, there will be no extra submissions or omissions;

[0063] (4) Flexibility: By changing the method rules, the code files can be flexibly extracted for inspection, which is convenient and fast;

[0064] (5) Simplicity: Only incremental code is pulled to reduce server storage space usage;

[0065] (6) Universality: SVN is currently widely applicable and is suitable for all teams that use SVN for code review.

[0066] The present invention uses a DevOps platform and an incremental code extraction method to flexibly configure code file extraction parameters, accurately extract files that need to be code checked according to the parameters, conduct a code check, generate a code check report, and achieve one-click automatic completion of the code check work, reduce the workload of related links, and improve the overall code quality. The method can achieve code check coverage of any range while effectively simplifying the code check workflow through the interaction between the checker and the front-end page, and combine with automated static scanning to fundamentally improve the quality of code check.

[0067] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. Automatically implement an incremental code walkthrough method based on a DevOps platform, characterized in that: The following steps are involved: Step S1, the DevOps platform obtains code extraction parameters according to user input, generates corresponding task instructions, and securely forwards these instructions to the backend service through the web server; Step S2: The Jenkins server receives the backend service parameters and configures the corresponding Jenkins pipeline task to execute. The Jenkins pipeline task triggers the incremental code extraction script according to the predefined script and parameters. Step S3: The incremental code extraction script runs in the Jenkins pipeline; Step S4: The Jenkins pipeline sends the incremental code extracted in step S3 to the SonarQube code quality analysis module through a secure data transmission protocol. After the Jenkins pipeline executes the shell script to compress the incremental code and directory structure, the file storage module securely stores the compressed files in the file server for downloading by the inspection personnel. Step S5: After receiving the incremental code, the SonarQube code quality analysis module uses its built-in code review rules and algorithms to perform in-depth static analysis on the incremental code and generate a walkthrough report; Step S6, the DevOps platform will feed back the walkthrough report to the user through the DevOps user interface module. If there are problems in the walkthrough report, the developer can optimize and repair the code based on this information, and resubmit the code to the SVN version control system after optimization and repair. After submission, re-execute step S1 to perform a new round of incremental code scanning, and repeat the above operations until the results are correct.

2. The incremental code walkthrough method according to claim 1, characterized in that: In step S3, the incremental code extraction script logic is: (1) Based on the extraction parameters passed in by the user, use the svn diff command to obtain the XML file of the SVN changes; (2) Use Python script to extract the list of changed files and filter out the SVN file paths that only have modification operations; (3) Traverse this file list, and for each file: a) Use the svn info command to determine whether the file exists in the latest version; b) Obtain the user ID of the user who last modified the file from the specified time to the present, and obtain the user name by combining it with the user mapping table; c) Use the svn export command to download the file; d) Determine whether it is a link file, if so, delete it; e) Use grep, sed, and awk commands to calculate the total number of lines and the number of valid lines in a file; f) Use svn diff to obtain all changes of the file from the specified time to the present, and calculate the number of valid modified lines; g) Filter out files with a low number of modified lines or a low modification rate based on user selection; h) Construct the directory structure according to the relative path of the file on the SVN server; i) Integrate the modified file information into json and return it to the user.

3. The incremental code walkthrough method according to claim 1, characterized in that: In step S1, the extraction parameters include but are not limited to version number, file path, extraction range, file type, time, configuration library address and account number.

4. The incremental code walkthrough method according to claim 1, characterized in that: In step S4, the storage process includes data encryption, access control and version management.

5. The incremental code walkthrough method according to claim 1, characterized in that: In step S5, the static analysis includes but is not limited to code complexity assessment, potential error detection and code style checking.

6. The incremental code walkthrough method according to claim 2, characterized in that: The valid number of lines is the number of lines after removing blank lines and blank characters, and the valid number of modified lines is the sum of the number of newly added or modified lines after removing blank lines and blank characters.

7. A system for automatically implementing incremental code walkthrough based on a DevOps platform based on the method of claim 1, characterized in that: It includes DevOps platform, web server, file server, SonarQube code quality analysis module and Jenkins server; the Jenkins server integrates DevOps platform, web server, file server, SonarQube code quality analysis module, and then implements incremental code walkthrough through Jenkins pipeline and script files.

Images