Static code scanning method, device and computer equipment

Abstract

The present application discloses a static code scanning method, device and computer equipment, which obtains the first scan result of the code repository and saves it as a first rule file; determines whether the scan rule has changed according to the rule_info field; if not, calculates the md5 value of the second rule file, and determines whether the scanned file exists according to the md5 value and repo_id, if it exists, deletes the scan of the scanned file, if not, retains the scan of the scanned file, and obtains the second scan result; if there is a change, each scanned file needs to be rescanned, and the second scan result is obtained; the first scan result is merged with the second scan result to obtain the complete scan result of the second scan. The static code scanning method, device and computer equipment provided by the present application not only greatly reduces resource consumption, but also improves scanning efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of static code scanning, and in particular to a static code scanning method, device and computer equipment. Background Art

[0002] Static code scanning is an automated tool for analyzing the quality and security of source code. It works by checking the text content of the source code, based on AST syntax analysis or regular matching techniques, to find potential program errors, security vulnerabilities, and bad programming practices without actually running the code. Static code scanners detect problems in the code based on conventional rules and patterns, helping developers discover and fix potential defects at an early stage.

[0003] The two most common problems in static code scanning are: slow scanning and resource consumption. The reasons for these two problems are:

[0004] (1) Large code size: Large projects have a large amount of code, which requires more time to scan. The larger the code, the slower the scanning speed.

[0005] (2) Complex code: Complex code structure, nested conditions, and loops will increase scanning time and consume more resources.

[0006] (3) Large number of scanning rules: Detecting potential problems in the code requires running multiple rules and algorithms to discover potential security risks, errors, and bad practices. The more rules there are, the longer the scan takes.

[0007] (4) Poor hardware environment: Poor hardware performance during the scanning process will result in a slower speed, and the need for more complex code analysis will result in more resource consumption.

[0008] (5) Repeated code scanning: The same sub-code is submitted for scanning multiple times, resulting in a large waste of scanning resources.

[0009] At present, in order to deal with the problems of slow scanning speed and high resource consumption during static code scanning, the following methods are usually used:

[0010] (1) Parallel processing: Use multithreading or distributed computing technologies to decompose the code scanning task into multiple subtasks for parallel processing to improve scanning efficiency. However, this will significantly increase server resource costs.

[0011] (2) Incremental scanning: Scanning is based on the difference files submitted by the code. However, this scanning method has limited scanning scenarios.

[0012] In summary, streamlining scanning rules and reducing code complexity are objective factors that cannot truly solve the problem. Summary of the invention

[0013] To this end, the present application provides a static code scanning method, apparatus and computer device to solve the problems of slow scanning speed and high resource consumption of the static code scanning method in the prior art.

[0014] In order to achieve the above objectives, this application provides the following technical solutions:

[0015] In a first aspect, a static code scanning method includes:

[0016] Step 1: Obtain the first scan result of the code repository and save it as a first rule file; the data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field, and an op_time field;

[0017] Step 2: Obtain the scanned files and scanning rules of the code repository and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_properties_version field;

[0018] Step 3: Parse the rule_info field in the first rule file, and determine whether the scanning rule has changed according to the rule_id field, rule_version field, and ule_properties_version field in the second rule file;

[0019] Step 4: If the scanning rule has not changed, the md5 value of the file_md5 field in the second rule file is calculated, and whether the scanned file exists is determined according to the md5 values ​​and repo_id in the first rule file and the second rule file. If the scanned file exists, the scan of the scanned file is deleted; if the scanned file does not exist, the scan of the scanned file is retained to obtain the second scanning result;

[0020] Step 5: If the scanning rules are changed, each scanned file needs to be rescanned and the second scanning result is obtained;

[0021] Step 6: Combine the first scan result with the second scan result to obtain a complete scan result of the second scan.

[0022] Preferably, in step 1, the rule_info field includes the rule id, the rule problem line, the rule version number and the rule threshold version number.

[0023] Preferably, in step 4, the md5 value calculation formula of the file_md5 field is: md5 (relative directory of the file + file content).

[0024] Preferably, the method further includes: determining whether the scanned file is a newly-added scanned file, and if so, adding new data; and if it is a reused scan result, updating the data.

[0025] Preferably, the method further includes: performing data cleaning regularly according to the op_time field in the first rule file and the second rule file.

[0026] In a second aspect, a static code scanning device includes:

[0027] The first scanning result acquisition module is used to obtain the first scanning result of the code repository and save it as a first rule file; the data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field and an op_time field;

[0028] A scanned file acquisition module, used to acquire the scanned files and scanning rules of the code repository, and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_roperties_version field;

[0029] A scanning rule judgment module, used for parsing the rule_info field in the first rule file, and judging whether the scanning rule has changed according to the rule_id field, the rule_version field and the ule_properties_version field in the second rule file;

[0030] A scanning file judgment module, used for calculating the md5 value of the file_md5 field in the second rule file if the scanning rule has not changed, and judging whether the scanned file exists according to the md5 values ​​and repo_id in the first rule file and the second rule file; if the scanned file exists, deleting the scan of the scanned file; if the scanned file does not exist, retaining the scan of the scanned file to obtain a second scanning result;

[0031] The scanning module is used to rescan each scanned file if the scanning rule changes, and obtain the second scanning result;

[0032] The scanning result merging module is used to merge the first scanning result with the second scanning result to obtain the complete scanning result of the second scanning.

[0033] According to a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a static code scanning method when executing the computer program.

[0034] In a fourth aspect, a computer-readable storage medium stores a computer program, which implements the steps of a static code scanning method when executed by a processor.

[0035] In a fifth aspect, a computer program product includes a computer program or instructions, which implement the steps of a static code scanning method when executed by a processor.

[0036] Compared with the prior art, this application has at least the following beneficial effects:

[0037] The present application provides a static code scanning method, apparatus and computer equipment, including obtaining the first scanning result of a code repository and saving it as a first rule file; obtaining the scanned file and scanning rules of the code repository and saving them as a second rule file; parsing the rule_info field in the first rule file, and judging whether the scanning rule has changed according to the rule_id field, rule_version field and ule_properties_version field in the second rule file; if the scanning rule has not changed, calculating the md5 value of the file_md5 field in the second rule file, and judging whether the scanned file exists according to the md5 value and repo_id in the first rule file and the second rule file; if the scanned file exists, deleting the scan of the scanned file; if the scanned file does not exist, retaining the scan of the scanned file to obtain a second scanning result; if the scanning rule has changed, each scanned file needs to be rescanned to obtain a second scanning result; merging the first scanning result with the second scanning result to obtain a complete scanning result of the second scanning. The present application provides a static code scanning method, device and computer equipment which reuses the results of repeated scanning of files, selects newly added files for scanning, and finally merges the scanning results, which not only greatly reduces resource consumption but also improves scanning efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing the present application; for example, those skilled in the art are capable of easily making conventional adjustments or further optimizations to the addition / reduction / attribution division, specific shapes, positional relationships, connection methods, dimensional ratios, etc. of certain units (components) based on the technical concepts and exemplary drawings disclosed in the present application.

[0039] Figure 1 A flow chart of a static code scanning method provided in Example 1 of the present application;

[0040] Figure 2 A schematic diagram of the structure of a static code scanning method provided in Example 1 of the present application. DETAILED DESCRIPTION

[0041] The present application is further described below in detail through specific embodiments in conjunction with the accompanying drawings.

[0042] In the description of this application: unless otherwise specified, "plurality" means two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).

[0043] The terms such as "upper", "lower", "left", "right", "middle", etc. cited in this application are usually used to indicate the general relative position relationship for the purpose of intuitive understanding by referring to the drawings, and are not absolute limitations on the position relationship in the actual product.

[0044] Under the dual conditions of improving scanning speed and reducing resource consumption, whether using parallel processing or incremental scanning, a paradox seems to have formed between the two, and they cannot be satisfied at the same time. According to a large number of user surveys, when users scan the same warehouse each time, they find that 80% to 90% of the files are scanned repeatedly. Therefore, if the results of repeated scans can be reused, and then the newly added files are screened for scanning, and finally the scan results are merged, it will greatly reduce resource consumption and improve scanning efficiency. This can solve the paradox between the two and better meet user expectations.

[0045] Embodiment 1

[0046] See also Figure 1 and Figure 2 , this embodiment provides a static code scanning method, including:

[0047] S1: Obtain the first scan result of the code repository and save it as the first rule file; the data structure of the first rule file includes the file_md5 field, the file_name field, the rule_info field, the repo_id field, and the op_time field;

[0048] Specifically, although the first scan takes a long time and consumes more resources, in order to facilitate subsequent reuse, this embodiment saves the first scan result of the code repository as the base data and obtains the first rule file.

[0049] The data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field, and an op_time field. The data structure is shown in Table 1:

[0050] Table 1

[0051]

[0052] The rule_info field consists of four fields: rule id, problem lines of file rules (problem lines are separated by multiple commas), rule version number, and rule threshold version number. The rule_info field is used to minimize the size of the data structure. The purpose of the rule version number and rule threshold version number fields is to calculate whether the scanning rules have changed. The repo_id field is used by users to distinguish repositories. The op_time field is used to record the operation time of this file, which is mainly used for data cleaning. The calculation formula of file_md5 = md5 (relative directory of the file + file content), which constitutes a globally unique file under the repository.

[0053] S2: Obtain the scanned files and scanning rules of the code repository and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_properties_version field;

[0054] S3: Parse the rule_info field in the first rule file, and determine whether the scanning rule has changed according to the rule_id field, the rule_version field, and the ule_properties_version field in the second rule file;

[0055] S4: If the scanning rule has not changed, the md5 value of the file_md5 field in the second rule file is calculated, and the existence of the scanned file is determined according to the md5 values ​​and repo_id in the first rule file and the second rule file. If the scanned file exists, the scan of the scanned file is deleted. If the scanned file does not exist, the scan of the scanned file is retained to obtain the second scanning result.

[0056] S5: If the scanning rules are changed, each scanned file needs to be rescanned and the second scanning result is obtained;

[0057] Specifically, due to the change of scanning rules, the scanning result of each scanned file may change. Therefore, in this case, each file needs to be rescanned.

[0058] In the method provided in this embodiment, whether the scanning rules have changed is one of the core points of the method. First, the rule information of this scan needs to be obtained, including the rule id, rule version number and rule threshold version number, and then the existing reusable data is queried according to the md5 value and repo_id of the scanned file, and then the rule information in each reusable data is compared. When the rules are inconsistent, it is considered that the rules have changed, and then the next scan is performed directly. If the rules have not changed during the data comparison, the results of the reusable data are recorded, and then the reused files are deleted from the scanned folder, thereby reducing the amount of file scanning.

[0059] S6: Merge the first scan result and the second scan result to obtain a complete scan result of the second scan.

[0060] Specifically, after the reused scanning result (second scanning result) is processed, the remaining files are scanned first to obtain the execution result file, and then the overall result is merged with the reused result (second scanning result) to generate the complete scanning result of this scan.

[0061] It should be noted that the first scan and the second scan mentioned in this embodiment do not represent the actual number of scans, but are only for distinction.

[0062] The static code scanning method provided by this embodiment also includes: determining whether the scanned file is a newly scanned file, if it is a newly scanned file, adding data; if it is a reused scan result, updating the data. Specifically: after the scan result processing is completed, the file_bug data needs to be reorganized, and data is added and inserted according to the existing data and the result of this scan. If it is a newly scanned file, adding data; if it is a reused scan result, updating the data.

[0063] A static code scanning method provided by this embodiment also includes: regularly cleaning data according to the op_time field in the first rule file. Specifically: in order to avoid excessive expansion of data, it is necessary to regularly clean up the data. This embodiment cleans up older data according to the update time of the op_time field, for example: cleaning up 100 days of data. Data cleaning may cause an increase in the number of scanned files, so a corresponding balance needs to be made in the trade-offs of time and space.

[0064] In order to verify the effect of a static code scanning method provided by this embodiment, the inventors tested 50w, 100w, 200w and 500w lines of code on the same server, with the same code volume, rules and scanning tools. The results showed that the scanning speed can be increased by 6 times, and the resource consumption is relatively reduced by 3 times, which basically meets the requirements of most users. The method provided by this embodiment is not only simple to implement, but also can save resource consumption, greatly improve the scanning speed, and meet the application requirements.

[0065] The static code scanning method provided in this embodiment has the following characteristics:

[0066] (1) The scanned file results can be reused through logical processing, thereby reducing repeated scanning operations. Specifically: In the entire code scanning process, most of the reasons for scanning time and resource consumption are trapped in the scanning calculation of each file, each line of code, and each rule by the scanning tool. This embodiment avoids the consumption points in the scanning process as much as possible, and uses clever data retention and a small amount of calculation to improve the scanning speed and reduce resource consumption while obtaining complete results.

[0067] (2) Generate a specific data structure through algorithmic logic processing, and based on the data structure, it can effectively identify whether to re-scan the code.

[0068] (3) This embodiment implements data cleaning operations through algorithms to avoid data expansion.

[0069] In summary, the static code scanning method provided in this embodiment reuses the results of repeated scanning of files, selects newly added files for scanning, and finally merges the scanning results, which not only greatly reduces resource consumption but also improves scanning efficiency.

[0070] Embodiment 2

[0071] This embodiment provides a static code scanning device, including:

[0072] The first scanning result acquisition module is used to obtain the first scanning result of the code repository and save it as a first rule file; the data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field and an op_time field;

[0073] A scanned file acquisition module, used to acquire the scanned files and scanning rules of the code repository, and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_roperties_version field;

[0074] A scanning rule judgment module, used for parsing the rule_info field in the first rule file, and judging whether the scanning rule has changed according to the rule_id field, the rule_version field and the ule_properties_version field in the second rule file;

[0075] A scanning file judgment module, used for calculating the md5 value of the file_md5 field in the second rule file if the scanning rule has not changed, and judging whether the scanned file exists according to the md5 values ​​and repo_id in the first rule file and the second rule file; if the scanned file exists, deleting the scan of the scanned file; if the scanned file does not exist, retaining the scan of the scanned file to obtain a second scanning result;

[0076] The scanning module is used to rescan each scanned file if the scanning rule changes, and obtain the second scanning result;

[0077] The scanning result merging module is used to merge the first scanning result with the second scanning result to obtain the complete scanning result of the second scanning.

[0078] For the specific implementation content of each module in a static code scanning device, please refer to the definition of a static code scanning method above, which will not be repeated here.

[0079] Embodiment 3

[0080] This embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements steps of a static code scanning method when executing the computer program.

[0081] Embodiment 4

[0082] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a static code scanning method are implemented.

[0083] Embodiment 5

[0084] This embodiment provides a computer program product, including a computer program or instructions, which implements the steps of a static code scanning method when executed by a processor.

[0085] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). To make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A static code scanning method, characterized in that: include: Step 1: Get the first scan result of the code repository and save it as the first rule file; The data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field, and an op_time field; Step 2: Obtain the scanned files and scanning rules of the code repository and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_properties_version field; Step 3: Parse the rule_info field in the first rule file, and determine whether the scanning rule has changed according to the rule_id field, rule_version field, and ule_properties_version field in the second rule file; Step 4: If the scanning rule has not changed, the md5 value of the file_md5 field in the second rule file is calculated, and whether the scanned file exists is determined according to the md5 values ​​and repo_id in the first rule file and the second rule file. If the scanned file exists, the scan of the scanned file is deleted; if the scanned file does not exist, the scan of the scanned file is retained to obtain the second scanning result; Step 5: If the scanning rules are changed, each scanned file needs to be rescanned and the second scanning result is obtained; Step 6: Combine the first scan result with the second scan result to obtain a complete scan result of the second scan.

2. The static code scanning method according to claim 1, characterized in that: In step 1, the rule_info field includes the rule id, the rule problem line, the rule version number, and the rule threshold version number.

3. The static code scanning method according to claim 1, characterized in that: In step 4, the md5 value calculation formula of the file_md5 field is: md5 (relative directory of the file + file content).

4. The static code scanning method according to claim 1, characterized in that: Also includes: Determine whether the scanned file is a newly-added scanned file. If it is a newly-added scanned file, then add data; if it is a reused scan result, then update the data.

5. The static code scanning method according to claim 1, characterized in that: Also includes: Data cleaning is performed regularly according to the op_time field in the first rule file.

6. A static code scanning device, characterized in that: include: The first scan result acquisition module is used to obtain the first scan result of the code repository and save it as the first rule file; The data structure of the first rule file includes a file_md5 field, a file_name field, a rule_info field, a repo_id field, and an op_time field; A scanned file acquisition module, used to acquire the scanned files and scanning rules of the code repository, and save them as a second rule file; the data structure of the second rule file includes a file_md5 field, a file_name field, a repo_id field, a rule_id field, a rule_version field, and a rule_roperties_version field; A scanning rule judgment module, used for parsing the rule_info field in the first rule file, and judging whether the scanning rule has changed according to the rule_id field, the rule_version field and the ule_properties_version field in the second rule file; A scanning file judgment module, used for calculating the md5 value of the file_md5 field in the second rule file if the scanning rule has not changed, and judging whether the scanned file exists according to the md5 values ​​and repo_id in the first rule file and the second rule file; if the scanned file exists, deleting the scan of the scanned file; if the scanned file does not exist, retaining the scan of the scanned file to obtain a second scanning result; The scanning module is used to rescan each scanned file if the scanning rule changes, and obtain the second scanning result; The scanning result merging module is used to merge the first scanning result with the second scanning result to obtain the complete scanning result of the second scanning.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images