Code quality detection system and detection method based on Gitlab and GitlabRunner

Abstract

The present invention provides a code quality detection system and detection method based on Gitlab and GitlabRunner, the system comprises a code server and a code detection server; the working process is: 1. pushing new code to a remote code server and notifying the code detection server; 2. after receiving the notification, the code detection server retrieves the update on the code repository locally; 3. comparing the difference between the code pushed by the user and the original code, as the incremental code; 4. detecting the defects in the incremental code; 5. notifying the developer of the defects in the incremental code; the present invention automatically detects the code submitted by the user, and only extracts the defects in the incremental code and feeds them back to the user, which can avoid the code defects introduced by the newly submitted code from damaging the original branch, and the incremental inspection mechanism can also save the time of the code detection server to perform code quality detection and allow the developer to repair the defects introduced by the newly submitted code in a targeted manner.

Description

Technical Field

[0001] The present invention relates to the technical field of continuous code integration, and in particular to a code quality detection system and a detection method based on Gitlab and GitlabRunner. Background Art

[0002] Continuous integration is a software development engineering practice that allows teams to continuously receive feedback and make improvements, without having to wait until the later stages of development to find and fix defects. Continuous integration advocates that team members must frequently integrate their work, and even multiple integrations may occur every day. Each integration is verified through automated builds, including automatic compilation, release, and testing, so that errors can be discovered as soon as possible, allowing the team to develop working software faster.

[0003] Code gate is an important means of successful continuous integration. Code gate refers to scanning the code before it is merged into the main branch to find code defects as early as possible. In this way, the quality of the main branch code can be avoided from being damaged by the merging of new code, ensuring that the main branch code is always robust and of high quality.

[0004] The traditional code access control in the prior art can only perform full scanning, which has the following problems:

[0005] (1) First, full scanning takes a long time. Every time a developer submits data, he or she has to wait for a long time before knowing the access control results. This has a great impact on the efficiency of uploading data.

[0006] (2) Full scanning consumes a lot of resources on the access control detection server. In the context of frequent submissions by developers, the access control detection server will be under a heavy burden;

[0007] (3) Full scanning cannot distinguish between historical code quality defects and defects introduced by the newly submitted code, and cannot allow developers to improve the newly submitted code in a targeted manner.

[0008] If the market can develop a code quality detection system and detection method based on Gitlab and GitlabRunner, use the callback mechanism of GitlabRunner to trigger access control checks, use the git diff (comparison) tool to accurately define incremental codes, narrow the scope of code scanning, and quickly isolate defects introduced by new codes, it will have great market prospects. Summary of the invention

[0009] In order to solve the defects in the prior art, the present invention provides a code quality detection system based on Gitlab and GitlabRunner, which uses the callback mechanism of Gitlab Runner to trigger access control inspection and uses the git diff (comparison) tool to accurately define incremental code, which can narrow the scope of code scanning and quickly isolate defects introduced by new code.

[0010] To achieve the above-mentioned purpose, the present invention provides the following technical scheme: a code quality detection system based on Gitlab and GitlabRunner, which includes a code server and a code detection server; the code server includes a Gitlab instance, a code repository, and a configuration repository; the code detection server includes GitlabRunner and an execution module; the code server is used to establish a Gitlab instance and manage the code repository and the configuration repository; the code detection server is used to receive a code detection request sent by the code server, perform detection, and output the detection result.

[0011] The preferred technical solution is that the Gitlab instance is used to respond to the user's pull, clone and push code requests, and is responsible for the user's registration, logout and permission management; it is responsible for the registration, scheduling and full life cycle management of GitlabRunner.

[0012] The preferred technical solution is that the code repository is used to store code, track and manage code changes, and record the process of each code change.

[0013] In a preferred technical solution, the configuration repository is used to store files that control the execution steps of GitlabRunner.

[0014] The preferred technical solution is that the GitlabRunner is responsible for receiving the code detection request sent by the code server, then calling the execution module to perform code detection, and sending the code detection result to the code server.

[0015] In a preferred technical solution, the execution module includes a code download module, a code comparison module, and a code detection module; the code download module is used to pull new code submitted by users; the code comparison module is used to compare the new code submitted by users pulled by the download module with the original code to obtain the incremental code introduced by the modification; the code detection module is used to detect the incremental code provided by the code comparison module.

[0016] Another object of the present invention is to provide a method for detecting code quality based on Gitlab and GitlabRunner, which comprises the following steps:

[0017] Step 1: The developer configures Gitlab and uses git rebase master (the rebase command is based on the master branch, where the original code is located, master represents the master branch, i.e. the original code, the same below) to rebase the personal branch (the personal branch where the newly submitted code is located, i.e. the newly submitted code), push the personal branch to the code repository of the remote code server, and use GitlabRunner to notify the code detection server for detection;

[0018] Step 2: After receiving the notification from step 1, the code detection server uses the git fetch command to retrieve the updated content on the code repository to the local computer.

[0019] Step 3: Use git diff to get the difference between the personal branch and the main branch as the incremental part of the code;

[0020] Step 4: Detect defects in the incremental part;

[0021] Step 5: Notify the developer of the defect.

[0022] In a preferred technical solution, the content of the notification in step 1 includes the name of the personal branch and the address of the code repository of the remote code server.

[0023] The preferred technical solution is that the process of step three is as follows:

[0024] Use git diff (compare) with the --name-status (display only the name and status of modified files) parameter to get a list of files that are different between the pushed personal branch and the main branch. Then take out each file in the list in turn, use the git diff (compare) command to generate the specific difference between the file and the corresponding file on the remote main branch, and save the difference in the difference file.

[0025] The preferred technical solution is that the process of step 4 is as follows:

[0026] Use cppcheck, cpplint and other detection tools to perform code quality detection on the difference file. The detection results usually include the specific line number and column number where the defect is located. Then read the corresponding difference file of the file obtained in step 3. If the defect falls into the area where the incremental code is located identified by the difference file, it is determined that the defect is introduced by the incremental code.

[0027] Compared with the prior art, the code quality detection system based on Gitlab and GitlabRunner and the updating method thereof of the present invention have the following beneficial effects:

[0028] 1. The present invention utilizes the callback mechanism of GitlabRunner to automatically detect the user's submitted code, thereby preventing the code defects introduced by the personal development branch from damaging the main branch.

[0029] 2. The code detection tool of the present invention uses the git diff (comparison) tool to accurately define the incremental code and only detects the incremental part of the code, thereby reducing the workload of the code detection server and the waiting time of the user.

[0030] 3. By adopting the technical solution of the present invention, the system will only feedback to the developer the defects introduced in the incremental code submitted this time, avoiding the waste of development efficiency due to interference from historical code defects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the interaction of the code quality detection system based on Gitlab and GitlabRunner in Example 1 of the present invention.

[0032] Figure 2 This is a schematic diagram of the interaction process of the code quality detection system based on Gitlab and GitlabRunner in Example 1 of the present invention.

[0033] Figure 3 The figure is a schematic diagram of the workflow of the code detection server in Embodiment 1 of the present invention to obtain the incremental code.

[0034] Figure 4 The figure is a schematic diagram of the defect flow of the incremental code detected by the code detection server in Embodiment 1 of the present invention.

[0035] Figure 5 This is a schematic diagram of the interaction of the code quality detection system based on Gitlab and GitlabRunner in Example 2 of the present invention.

[0036] Figure 6 Schematic diagram of the working process of the detection system in Example 2 of the present invention.

[0037] Figure 7 The figure is a schematic diagram showing the principle of detecting incremental code defects by the detection system in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0038] Reference Figure 1 to Figure 2 The code quality detection system based on Gitlab and GitlabRunner and the updating method thereof are further described in the present invention.

[0039] Example 1

[0040] like Figures 1 to 4As shown in the figure, this system mainly consists of a code server and a code detection server. The code server is mainly responsible for establishing a Gitlab instance and managing the code repository and configuration repository. The code detection server mainly receives code detection requests from the code server, performs relevant detection, and finally returns the detection results to the code server.

[0041] like Figure 1 As shown in the figure, the composition of the code server and the code detection server is introduced. The code server consists of a Gitlab instance, a code repository, and a configuration repository. The code detection server mainly consists of GitlabRunner and an execution module. The code server and the code detection server are connected together through a network module. The following introduces the main functions of each module of the code server and the code detection server:

[0042] Key features of code-server:

[0043] Gitlab instance: used to respond to user pull, clone and push code requests. Responsible for user registration, logout and permission management. Responsible for GitlabRunner registration, scheduling and full life cycle management.

[0044] Code repository: Stores project code and is responsible for tracking and managing code changes, ensuring that each change is clearly recorded.

[0045] Configuration repository: stores the gitlab_ci.yml file specified by the project. The execution steps of GitlabRunner are specified in this file.

[0046] The main functions of the code detection server are:

[0047] GitlabRunner instance: responsible for receiving code detection requests from the code server, then calling the execution module for code detection, and sending the code detection results to the code server.

[0048] The execution module is responsible for checking the code execution according to the specified logic. The execution module is divided into the following sub-modules:

[0049] Code download module: responsible for pulling the latest code submitted by users.

[0050] Code comparison module: responsible for comparing the new code submitted by the user with the original code and obtaining the incrementally modified parts.

[0051] Code detection module: responsible for detecting the incremental code provided by the code comparison module.

[0052] The working process of the code quality detection system based on Gitlab and GitlabRunner of the present invention is as follows:

[0053] Step 1. Edit the.gitlab_ci.yml file in the following way and save it in the configuration repository;

[0054] <Code detection project name>:

[0055] Tags:

[0056] <Identifier of GitlabRunner>

[0057] Script:

[0058] <Detection command><Branch>

[0059] The content represented by the code detection project name identifier can be selected arbitrarily. The Tags field specifies the identifier of the GitlabRunner used to perform the detection, and the Script field specifies the command and corresponding parameters that the GitlabRunner needs to execute. The detection command is generally the name of an executable binary file, and the Branch represents the name of the branch where the code pushed by the user this time is located, that is, the personal branch.

[0060] Step 2. After Gitlab receives the user's push branch request, it will notify the GitlabRunner specified in the Tags field in Step 1 to execute the content in the Script field.

[0061] Step 3. GitlabRunner finds the corresponding execution module entry according to the detection command to be executed this time.

[0062] Step 4. The code download module uses the git fetch command to retrieve the updates on the code repository to the local.

[0063] Step 5. The code comparison module obtains the list of files with differences (denoted as L1) between the personal branch (hereinafter denoted as developer, the same below) and the master branch (that is, the branch where the original code is located, hereinafter denoted as master, the same below), and records each difference in the difference file.

[0064] Step 5.1. Use git diff developer master --name-status (to obtain the files with differences and their statuses between the personal branch and the master branch), and record the result of the command execution as L2.

[0065] Step 5.2. Use git log master..developer --pretty=oneline (to obtain the commit records that exist in the personal branch but do not exist in the master branch), and record the result of the command execution as C.

[0066] Step 5.3, take out each commit record of C in turn, record it as COMMIT, and execute gitshow COMMIT --name-only --pretty=oneline in turn (get the files modified in the commit). Merge all the modified files obtained in 5.3 together and remove duplicates to form a difference file list (recorded as L3).

[0067] Step 5.4: Take out each file in L3. If the file is a deleted file, exclude it from L3.

[0068] Step 5.5: Take out each file in L3. If the file does not belong to the code file, exclude it from L3.

[0069] Step 5.6: Take each file in L3 and exclude it from L3 if it is not in L2.

[0070] Step 5.7. Re-record the remaining L3 as L1.

[0071] Step 5.8. For each file in L1 (hereinafter referred to as file), execute the Git diff master:filedeveloper:file (get the difference between the file on the master branch and the personal branch) command and save the result in the difference file.

[0072] Step 6: The code detection module uses a code detection tool to obtain defects in the incremental part.

[0073] Step 6.1: For each file in L1, call public scanning tools such as cppcheck and clang-tidy to obtain a list of defect results for the file.

[0074] Step 6.2: Compare each defect with the incremental code position marked with @@ in the difference file one by one. If a defect falls into the incremental code position, it can be regarded as an incremental defect.

[0075] Step 6.3: Summarize incremental defects and form a defect report.

[0076] Step 7: Send the scan results back to the code server through GitlabRunner to end the code access control scan.

[0077] The present invention utilizes the above mechanism to automatically detect the user's submitted code, and only extracts defects in the incremental code to feedback to the user, which can not only prevent code defects introduced by personal development branches from damaging the main branch, but also the incremental inspection mechanism can save the time of the code detection server to perform intelligent detection, allowing developers to modify the defects introduced by the newly submitted code in a targeted manner.

[0078] Example 2

[0079] like Figure 5 As shown in the figure, the code quality detection system based on Gitlab and GitlabRunner includes the following modules:

[0080] The Gitlab server (also called the code server) is used to establish a Gitlab instance and manage the code repository and configuration repository.

[0081] The code detection server is used to receive code detection requests sent by the code server, perform detection, output detection results and feedback to developers.

[0082] When using this system, first configure Gitlab Runner to notify the access control inspection server (also known as the code detection server) when the user pushes a personal branch (hereinafter referred to as the developer branch, that is, the newly developed code content) to the remote end. The notification content needs to include the name of the user's personal branch and the address of the code repository on the remote code server.

[0083] After receiving the above notification, the program on the access control inspection server uses the git fetch command to retrieve the updates on the code repository back to the local computer.

[0084] Use git diff (comparison) with the --name-status (display only the name and status of modified files) parameter to get a list of files that are different between your personal branch (the branch where the newly submitted code is located, denoted as the developer branch, the same below) and the main branch (the branch where the original code is located, denoted as the master branch, the same below), and then use git diff (comparison) for each file in the list to generate the specific difference between the file and the corresponding file on the remote master branch, and save the difference in a difference file.

[0085] Code quality detection is performed on the files in the difference file list. The detection results usually include the specific line number and column number where the defect is located. Then the difference file is read. If the defect falls into the area where the new code exists as identified by the difference file, it is determined that the defect is introduced by incremental code.

[0086] Notify the developer of the defect.

[0087] After using git diff (comparison) with the --name-status parameter (displaying only the name and status of modified files) to get a list of files that differ between the developer branch and the master branch, you should filter them: determine whether the file is a code file or an ordinary non-code configuration file. Only code files will be scanned in the next step.

[0088] After using git diff (comparison) with the --name-status parameter (displaying only the name and status of modified files) to get a list of files that differ between the developer branch and the master branch, you should filter them to see whether the file is newly added, modified, or deleted. Deleted files should be skipped.

[0089] like Figure 6 and Figure 7 As shown, the specific working process of this embodiment 2 is as follows:

[0090] Step 1: Configure GitlabRunner to notify the access control server when the user pushes a personal branch (hereinafter referred to as the developer branch) to the remote end.

[0091] Step 1.1: Configure Gitlab Runner on the Gitlab server. This device triggers the notification script in step 1.2 when the user pushes a personal branch to the code repository.

[0092] Step 1.2: Edit the notification script on the Gitlab server. The script performs the following tasks: Notify the access control detection server of the user's personal branch name and Git repository address via the network.

[0093] Step 2: After receiving the notification in step 1.2, the program on the code detection server (also called: access control detection server) uses the git fetch command to retrieve the updates on the code repository locally.

[0094] Step 3: Use git diff (get) to get the difference between the branch pushed by the user and the master branch as the incremental part of the code.

[0095] Step 3.1. Use git diff developer master --name-status (get the files and their status that differ between the personal branch and the main branch) to get a list of files on the developer branch that differ from the master branch.

[0096] Step 3.2: Read the difference list. If the file is a deleted file, remove it from the final list of files to be detected.

[0097] Step 3.3: Read the difference list. If the file does not belong to a code file, remove it from the final list of files to be detected.

[0098] Step 3.4: Execute the Gitdiff master:file developer:file command (get the difference between the file in the master branch and the personal branch) for each file in the list of files to be checked (hereinafter referred to as file), obtain the corresponding difference file, and save it temporarily

[0099] Step 4: Use code detection tools to obtain defects in the incremental part.

[0100] Step 4.1: For the files in the list of files to be checked, call public scanning tools such as cppcheck and pclint to obtain a defect result list of the files.

[0101] Step 4.2: Compare each defect with the incremental code position marked with @@ in the difference file one by one. If a defect falls into the incremental code position, it can be regarded as an incremental defect.

[0102] Step 4.3: Summarize incremental defects and form a defect report.

[0103] Step 5: Send the defect report to the developer via email to end the code access control scan.

[0104] The present invention uses the callback mechanism of Gitlab Runner to trigger access control checks, and uses the git diff (comparison) tool to accurately define incremental codes, narrow the scope of code scanning, and then quickly isolate defects introduced by new codes, which can help developers quickly locate defects caused by newly developed codes.

[0105] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. It should be pointed out that any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A code quality detection system based on Gitlab and GitlabRunner, characterized by: It includes a code server and a code detection server; the code server includes a Gitlab instance, a code repository, and a configuration repository; the code detection server includes GitlabRunner and an execution module; The code server is used to establish a Gitlab instance and manage the code repository and configuration repository; the code detection server is used to receive a code detection request sent by the code server, perform detection, and output the detection result.

2. The code quality detection system based on Gitlab and GitlabRunner according to claim 1, characterized in that: The Gitlab instance is used to respond to user requests to pull, clone, and push code, and is responsible for user registration, logout, and permission management; it is responsible for the registration, scheduling, and full life cycle management of GitlabRunner.

3. The code quality detection system based on Gitlab and GitlabRunner according to claim 2, characterized in that: The code repository is used to store code, track and manage code changes, and record the process of each code change.

4. The code quality detection system based on Gitlab and GitlabRunner according to claim 3, characterized in that: The configuration repository is used to store files that control the execution steps of GitlabRunner.

5. The code quality detection system based on Gitlab and GitlabRunner according to claim 4, characterized in that: The GitlabRunner is responsible for receiving the code detection request sent by the code server, then calling the execution module to perform code detection, and sending the code detection result to the code server.

6. The code quality detection system based on Gitlab and GitlabRunner according to claim 5, characterized in that: The execution module includes a code download module, a code comparison module, and a code detection module; the code download module is used to pull the new code submitted by the user; the code comparison module is used to compare the new code submitted by the user pulled by the download module with the original code to obtain the incremental code introduced by the modification; The code detection module is used to detect the incremental code provided by the code comparison module.

7. The detection method of the code quality detection system based on Gitlab and GitlabRunner according to any one of claims 1 to 6, characterized in that: It includes the following steps: Step 1: The developer configures Gitlab, uses the rebase command on the personal branch where the newly submitted code is located, pushes the personal branch to the code repository of the remote code server, and uses Gitlab Runner to notify the code detection server to perform detection; the benchmark for using the rebase command is the original branch where the original code is located; Step 2: After receiving the notification in step 1, the code detection server uses the get command to retrieve the updated content on the code repository to the local computer; Step 3: Use the comparison command to get the difference between the personal branch and the original branch as the incremental part of the code; Step 4: Detect defects in the incremental part; Step 5: Notify the developer of the defect.

8. The detection method according to claim 7, characterized in that: The content of the notification in step 1 includes the name of the personal branch and the address of the code repository on the code server.

9. The detection method according to claim 8, characterized in that: The process of step three is as follows: Use the comparison command to get a list of files that are different between the pushed personal branch and the original branch, then take out each file in the list in turn, use the comparison command to generate the specific difference between the file and the corresponding file on the original branch, and save the difference in the difference file.

10. The detection method according to claim 9, characterized in that: The process of step 4 is as follows: Use cppcheck and cpplint detection tools to perform code quality detection on the difference file. The detection results usually include the specific line number and column number where the defect is located. Then read the difference file corresponding to the file obtained in step 3. If the defect falls into the area where the incremental code is located identified by the difference file, it is determined that the defect is introduced by the incremental code.

Images