Test method of code detection tool, code detection method and computing equipment

Abstract

The invention provides a test method of a code detection tool, a code detection method and computing equipment, relates to the technical field of servers, and is used for providing a test result which can be used for accurately judging the detection capability of the code detection tool on code defects. The method comprises the steps of obtaining a preset use case set; the preset use case set comprises use cases corresponding to various code defect types, and the number of the use cases corresponding to one defect type is in positive correlation with the number of vulnerabilities belonging to the defect type; testing the code detection tool by utilizing a preset case set to obtain a test result; wherein the code detection tool has the function of detecting code defects; the test result is used for indicating the code defect detection capability of the code detection tool.

Description

Technical Field

[0001] This application relates to the technical field of servers, and particularly to a test method for a code detection tool, a code detection method, and a computing device. Background Art

[0002] Defect detection of code is a crucial part in the software development process. Defect detection can identify and fix errors and potential problems in the code, thereby significantly improving the quality of the software. By ensuring the correctness and robustness of the code, the reliability and stability of the software can be enhanced, and the failures and crashes during runtime can be reduced.

[0003] To reduce the workload of manual comparison and improve the evaluation accuracy, it can be considered to use a code detection tool to detect code defects. The code detection tool is used to detect defects in the code, and its detection ability is crucial because it determines whether the tool can correctly identify the problems existing in the code. Effective evaluation of the code detection tool can provide a reasonable reference for selecting a suitable code detection tool. Summary of the Invention

[0004] Embodiments of this application provide a test method for a code detection tool, a code detection method, and a computing device, which are used to provide a way to test the code detection tool, so that users can determine the detection ability of the code detection tool for code defects according to the test results.

[0005] To achieve the above object, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, embodiments of this application provide a test method for a code detection tool. The method includes: obtaining a preset use case set; the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type. Using the preset use case set to test the code detection tool to obtain a test result; wherein, the function of the code detection tool includes detecting code defects; the test result is used to indicate the detection ability of the code detection tool for code defects.

[0007] In the test method for the code detection tool provided by the embodiments of this application, the distribution of use cases in the obtained preset use case set is similar to the actual vulnerability distribution. Therefore, they are more likely to cover the real vulnerabilities existing in the code detection tool. In this way, when using the preset use case set to test the code detection tool, this test method can more accurately reveal the weak links in the code detection tool, and the obtained test result can more truly reflect the performance of the code detection tool when facing actual vulnerabilities. Thus, the obtained test result can be used to accurately judge the detection ability of the code detection tool for code defects.

[0008] In a possible implementation, the test results include at least one of a first result, a second result, and a third result; the first result is used to characterize the number of types of code defects that the code detection tool can detect; the second result is used to characterize the accuracy of the code detection tool in detecting code defects; the third result is used to characterize the duration taken by the code detection tool to detect a preset test case set.

[0009] In this implementation, when the test results include at least one of the first result, the second result, and the third result, by synthesizing multiple results, the detection ability, accuracy, and efficiency of the code detection tool can be comprehensively evaluated, so as to more accurately understand its performance in code defect detection.

[0010] In a possible implementation, the test cases include code samples and first labels; the first labels are used to indicate whether the code samples have defects and the types of defects in the case of defects; the first result includes the ratio of the number of types of code defects that the code detection tool can detect to the total number of types of code defects in the preset test case set; and / or, the second result is obtained based on the number of first test cases and the number of second test cases in the preset test case set; the code samples in the first test cases have defects of a certain type; the code samples of the second test cases are the code samples in the preset test case set that are detected by the code detection tool as having defects; and / or, the third result is the ratio of the duration taken by the code detection tool to detect the preset test case set to a reference duration.

[0011] In this implementation, a more specific determination method for calculating the first result, the second result, or the third result is provided, and more accurate first, second, or third results can be obtained.

[0012] In a possible implementation, the test cases include code samples and first labels; the first labels are used to indicate whether the code samples have defects and the types of defects in the case of defects; the code detection tool is tested using the preset test case set to obtain test results, including: for each test case in the preset test case set, the code detection tool is used to detect the code sample in the test case to obtain a second label for the test case; the second label is used to indicate whether the code sample has defects; based on the first label and the second label of each test case, the test results are determined.

[0013] In this implementation, the first label is a true indication of whether the code sample has defects and the specific type of defect when there are defects. It represents the actual state of the code sample and is a benchmark for evaluating the detection accuracy of the code detection tool. The second label is the result obtained after the code detection tool detects the code sample, indicating the defect situation that the code detection tool believes exists in the code sample. It reflects the performance of the code detection tool in the automated detection process. By comparing the first label and the second label, the performance of the code detection tool in code defect detection can be effectively evaluated, and more accurate test results can be obtained.

[0014] In a possible implementation, based on the first label and the second label of each use case, the test result is determined, including: using the use case where the first label indicates that the code sample has a defect as the first use case; using the use case where the second label indicates that the code sample has a defect as the second use case, and using the defect type indicated by the first label of the second use case as the defect type that the code detection tool can detect; determining the first result according to the number of defect types that the code detection tool can detect, and determining the second result according to the number of the first use cases and the number of the second use cases; the test result includes the first result and the second result, the first result is used to characterize the number of code defect types that the code detection tool can detect; the second result is used to characterize the accuracy of the code detection tool in detecting code defects.

[0015] In a possible implementation, the use case types in the preset use case set include the first type, the second type, and the third type; the code sample in the use case of the first type is the code in the preset software; the code sample in the use case of the second type is the virtual code generated based on the preset generation model; the code sample in the use case of the third type is the simulation code obtained by modifying the code in the preset software.

[0016] In this implementation, the preset use case set contains use cases of the first type, the second type, and the third type, that is, real use cases, virtual use cases, and simulation use cases, which can significantly improve the test efficiency and quality, enhance the controllability and repeatability of the test, support various test strategies and methods, and help to more accurately evaluate the defect detection effect of the code detection tool.

[0017] In a possible implementation, the defect type is the base weakness type in the Common Weakness Enumeration (CWE) type of general vulnerabilities.

[0018] In this implementation, setting the defect type in the preset use case set to the base weakness type can more accurately evaluate the ability of the code detection tool to detect code defects.

[0019] Second aspect, an embodiment of the present application provides a code detection method, which includes: obtaining a tool set according to at least one target defect type to be detected; the tool set includes at least one code detection tool, and each code detection tool can detect code defects of at least one defect type; the defect types that each code detection tool can detect are tested by a preset use case set, the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type; the defect types that at least one code detection tool can detect cover the target defect type; using each code detection tool in the tool set to perform defect detection on the target code.

[0020] It can be seen that in the solution provided by the embodiment of the present application, through multiple code detection tools in the tool set that can perform defect detection on the target defect type, defect detection is performed on the target code. The multiple tools review the code from different perspectives, can complement each other, reduce the possibility of missed reports, and thus can improve the accuracy of defect detection.

[0021] Third aspect, a computing device is provided, including: a processor and a memory, and the processor is connected to the memory. The memory is used to store computer execution instructions, and the processor executes the computer execution instructions stored in the memory, thereby implementing any method provided in the first aspect or the second aspect.

[0022] Fourth aspect, a chip is provided, the chip includes: a processor and an interface circuit; the interface circuit is used to receive code instructions and transmit them to the processor; the processor is used to run the code instructions to execute any method provided in the first aspect above.

[0023] Fifth aspect, a computer-readable storage medium is provided, storing computer execution instructions, and when the computer execution instructions run on a computer, the computer is enabled to execute any method provided in the first aspect or the second aspect.

[0024] Sixth aspect, a computer program product is provided, including computer execution instructions, and when the computer execution instructions run on a computer, the computer is enabled to execute any method provided in the first aspect or the second aspect.

[0025] Among them, the technical effects brought by any implementation manner in the third aspect to the sixth aspect can refer to the technical effects brought by different implementation manners in the first aspect or the second aspect, which will not be elaborated here. Description of the Drawings

[0026] Figure 1 It is a system framework diagram of a computing device provided by an embodiment of the present application;

[0027] Figure 2 Method flow of a test method for a code detection tool provided by an embodiment of the present application Figure 1 ;

[0028] Figure 3 Method flow of a test method for a code detection tool provided by an embodiment of the present application Figure 2 ;

[0029] Figure 4 Coverage comparison chart of a code detection tool provided by an embodiment of the present application;

[0030] Figure 5 Coverage comparison chart of another code detection tool provided by an embodiment of the present application;

[0031] Figure 6 Method flow chart of a code detection method provided by an embodiment of the present application;

[0032] Figure 7 Schematic diagram of a test device provided by an embodiment of the present application. Detailed implementation manners

[0033] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0034] Among them, in the description of the present application, unless otherwise specified, " / " means that the objects associated before and after are in an "or" relationship. For example, A / B may represent A or B; "and / or" in the present application is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. These three situations, where A and B may be singular or plural.

[0035] Moreover, in the description of the present application, unless otherwise specified, "a plurality of" means two or more than two. "At least one (item)" or its similar expression below refers to any combination of these items, including any combination of single item (item) or plural items (items). For example, at least one (item) of a, b, or c may represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, c may be single or multiple.

[0036] In addition, for the convenience of clearly describing the technical solutions of the embodiments of the present application, in the embodiments of the present application, terms such as "first" and "second" are used to distinguish identical or similar items with basically the same functions and effects. Those skilled in the art can understand that terms such as "first" and "second" do not limit the quantity and execution order, and "first", "second", etc. do not necessarily mean different. At the same time, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific way for easy understanding.

[0037] The following introduces the terms related to the embodiments of the present application.

[0038] Common Weakness Enumeration (CWE): An open and extensible common language used to describe the types of defects existing in code. CWE provides a standardized way to describe and classify defects in code, which may lead to the introduction of vulnerabilities. CWE enables researchers in different fields to use the same definitions when communicating about security issues caused by code defects, reducing ambiguity.

[0039] CWE-ID is used to indicate the type of defect in the code. CWE-ID usually consists of "CWE-" followed by a string of numbers, such as CWE-122, CWE-787, etc. These numbers represent specific types of defects, and each type has its unique description and possible impacts. CWE forms a multi-level defect type classification system through numbered types (such as class defects, basic defects, and variant defects, etc.).

[0040] CWE-1000 is a CWE research view, also known as the researcher view (navigate CWE). In CWE-1000, nearly 1000 CWE IDs are included, and these CWE IDs are classified according to multi-level division in the CWE-1000 view. CWE-1000 does not care about the defect detection method, the location of the defect in the code, or when the defect is introduced in the software development life cycle. This view is mainly based on organizing and classifying the defects existing in the code, covering almost all defect types in CWE.

[0041] Static code security scanning: Static code security scanning is a technical means to detect potential vulnerabilities by carefully analyzing the code without executing the code. Its purpose is to identify and prevent common security threats such as structured query language (SQL) injection and cross-site scripting (XSS). Through static code security scanning, the security of the code can be significantly improved, ensuring that the software built based on the code is more robust and reliable.

[0042] First, an exemplary introduction to the application scenario of the embodiments of the present application is given.

[0043] Static code security scanning is an important means for defect detection of code. It can discover potential security problems in the early stage of development, improve the quality and security of the code. At the same time, it can also reduce the repair cost, because the earlier the problem is discovered, the easier it is to repair.

[0044] Code detection tools are used to detect defects in the code. Their detection capabilities are crucial because they determine whether the tool can correctly identify the problems existing in the code. Effective evaluation of code detection tools can provide a reference for selecting the appropriate tools. There are many types of static code security scanning tools (or static code security scanning applications). Since different tools use different algorithms, technologies, capabilities for identifying specific vulnerabilities, and identification speeds. For example, some tools may be better at identifying memory leak problems, while others may be more focused on discovering logical errors or security vulnerabilities. Therefore, when selecting a code detection tool, it is necessary to comprehensively consider and evaluate according to the specific needs of the development team and the characteristics of the project.

[0045] The embodiments of the present application provide a test method for a code detection tool. The method includes: First, it is necessary to obtain a preset test case set corresponding to multiple code defect types, and the number of test cases corresponding to one defect type in the preset test case set is positively correlated with the number of vulnerabilities belonging to the defect type; then, use the preset test case set to test the code detection tool to obtain a test result; where the function of the code detection tool includes detecting code defects; the test result is used to indicate the detection ability of the code detection tool for code defects.

[0046] It can be understood that the number of vulnerabilities of a defect type is the frequency of occurrence of the defect type in actual applications. That is to say, the number of test cases corresponding to each defect type in the preset test case set is determined based on the actual occurrence frequency of the defect type. The higher the actual occurrence frequency of a certain defect type, the more common it is, and the more test cases corresponding to this defect type.

[0047] It can be seen that the use case distribution in the preset use case set in the embodiments of the present application is similar to the actual vulnerability distribution. Therefore, they are more likely to cover the real vulnerabilities existing in the code detection tool. In this way, when using the preset use case set to test the code detection tool, this testing method can more accurately reveal the weak links in the code detection tool, and the obtained test results can more truly reflect the performance of the code detection tool when facing actual vulnerabilities. Thus, the obtained test results can be used to accurately judge the detection ability of the code detection tool for code defects.

[0048] Furthermore, when a user selects a code detection tool (such as a code detection tool), based on the test results corresponding to multiple code detection tools, the user can determine a code detection tool that meets their own needs, without the need to manually compare the scan results to select a code detection tool. This can not only improve the efficiency of the user in selecting a code detection tool, but also, since the test results can be used to accurately judge the detection ability of the code detection tool for code defects, provide a reasonable basis for the user to select a code detection tool.

[0049] Next, an exemplary introduction to the system architecture of the embodiments of the present application will be given.

[0050] The embodiments of the present application provide a computing device, which may specifically be a network device or a terminal device. The network device may include a server, etc. Among them, the server may be a physical server, or may be two or more physical servers sharing different responsibilities and cooperating with each other to implement the various functions of the server.

[0051] Exemplarily, the server may be a blade server, a high-density server, a rack server, or a tower server, etc. The terminal device may include a personal digital assistant (PDA), an ultra-mobile personal computer (UMPC), a notebook computer, a netbook, a desktop computer, an all-in-one computer, etc.

[0052] It should be noted that the embodiments of the present application do not limit the device form of the computing device. Hereinafter, taking the server as an example, the system architecture of the computing device provided by the embodiments of the present application will be described.

[0053] Such as Figure 1As shown in the figure, it is the system framework diagram of the computing device provided by the embodiment of the present application. The hardware part of the computing device includes a processor, a basic input output system (BIOS) chip, an out-of-band controller, and a memory. The software part mainly includes BIOS, an out-of-band management module, and an operating system (OS).

[0054] The processor may include a central processing unit (CPU), and the CPU includes one or more CPU cores. All operations of the CPU for processing data are executed by the CPU cores. The more CPU cores included in the CPU, the faster the data processing speed. In the embodiment of the present application, the processor in the computing device executes the test method or the code detection method of the code detection tool.

[0055] The BIOS chip is a chip set on the motherboard for initializing and detecting various hardware during the startup process of the computing device. The BIOS chip includes a flash memory area.

[0056] The out-of-band management module is located inside the out-of-band controller, and the operating system is located inside the processor.

[0057] The out-of-band management module can be a management unit for non-business modules. For example, the out-of-band management module can remotely maintain and manage the computing device through a dedicated data channel. This out-of-band management module is completely independent of the operating system of the computing device and can communicate with the BIOS and the operating system through the out-of-band management interface of the computing device.

[0058] Exemplarily, the out-of-band management module may include a management unit for the operating state of the computing device, a management system in the management chip, a baseboard management controller (BMC) of the computing device motherboard, a system management mode (SMM), etc. It should be noted that the embodiment of the present application does not limit the specific form of the out-of-band management module, and the above is only an exemplary description.

[0059] The OS is a computer program that manages and controls the hardware and software resources of the computing device. Any other software must run under the support of the operating system. After the computing device is powered on, the BIOS first starts to perform a series of operations such as self-check and initialization, and then boots the OS to enable the user to use the computing device normally.

[0060] BIOS is a set of programs that are fixed on the BIOS chip on the motherboard of a computing device. The main function of BIOS is to provide the lowest-level and most direct hardware settings and controls for computing devices.

[0061] Memory, also called internal storage or main memory, is installed in memory slots on the motherboard of a computing device.

[0062] It should be noted that the system architecture and application scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0063] For ease of understanding, the testing method of the code detection tool provided in the embodiment of the present application is exemplarily introduced below with reference to the accompanying drawings.

[0064] See also Figure 2 The testing method of the code detection tool provided in the embodiment of the present application can be performed by Figure 1 The computing device in the code detection tool is executed, exemplarily, the testing method of the code detection tool includes the following steps S101-S102:

[0065] S101. Obtain a preset use case set.

[0066] Among them, the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to a defect type is positively correlated with the number of vulnerabilities belonging to the defect type.

[0067] It should be understood that a preset use case set refers to a set of use cases that are pre-designed and prepared in order to cover and detect specific types of code defects during software testing. These use cases are carefully designed and selected to effectively discover and handle potential problems in the software.

[0068] Code defects refer to various errors, loopholes or deficiencies that may be introduced during the code development process. Code defect types are obtained by classifying existing code defects according to certain standards. For example, various errors, loopholes or deficiencies that may be introduced during the code development process can be divided into different defect types such as syntax errors, logic errors, boundary errors and security issues. Among them, syntax errors refer to the code violating the grammatical rules of the programming language. Logical errors refer to problems with the logic of the code at runtime, resulting in failure to execute as expected. Boundary errors refer to problems when the code handles boundary conditions. Security issues refer to security risks in the code (such as SQL injection, command injection, buffer overflow, etc.), which are easy to be attacked or exploited.

[0069] In a possible implementation, the types of defects existing in the code can be described by CWE. Among them, the specific defect types are identified by CWE-ID. Different CWE-IDs represent different defect types, and each type has its unique description and possible impacts. CWE forms a multi-level defect type classification system through numbered types (such as class defects, basic defects, and variant defects, etc.).

[0070] In the embodiments of the present application, the preset test case set will include test cases corresponding to each defect type, that is, each test case is designed for a specific defect type and there are problems corresponding to the defect type. To facilitate determining the defect type of each test case during the testing process, a label can be set for each test case.

[0071] As a feasible implementation, a test case includes a code sample and a first label; the first label is used to indicate whether the code sample has a defect and, in the case of a defect, the defect type of the defect.

[0072] It can be understood that the code sample is the key part of the test case, which contains the code segments to be tested. The code detection tool will perform tests on these code samples to verify whether the code samples can be executed smoothly or whether there are defects. The first label can be an identifier or annotation attached to the code sample, used to indicate the defect status of the code sample. If the code sample has a defect, the first label can further specify the specific type of the defect existing in the code sample.

[0073] Exemplarily, in the case where the defect existing in the code sample is "parameters are read from the database without judgment, which may cause integer overflow", the corresponding label can be "Defect: Yes; Defect Type: Security Issue (Integer Overflow)".

[0074] In the embodiments of the present application, the number of test cases corresponding to a defect type in the preset test case set has a positive correlation with the number of vulnerabilities belonging to the defect type. It can be understood that the number of vulnerabilities of a defect type refers to the number of times or frequencies that a specific defect type is found in actual applications, which can reflect to a certain extent the frequency of occurrence of this defect type in actual applications.

[0075] The number of test cases corresponding to a defect type has a positive correlation with the number of vulnerabilities, that is, the number of test cases corresponding to each defect type in the preset test case set is determined based on the number of vulnerabilities of the defect type. The more the number of vulnerabilities of a certain defect type, the higher the frequency of occurrence of this defect type in actual applications may be. Therefore, this defect type needs to be focused on, and then the number of test cases corresponding to this defect type in the preset test case set is more.

[0076] As a feasible implementation method, the number of vulnerabilities of each defect type can be determined based on the vulnerability information recorded in the National Vulnerability Database (NVD), that is, the number of vulnerabilities corresponding to each CWE-ID can be determined.

[0077] Specifically, as an implementation method, the computing device first obtains the metadata information of all vulnerabilities in the NVD. Among them, the metadata information is used to indicate the CWE-ID corresponding to the vulnerability, that is, the code defect type of the defect existing in the vulnerability. Then, based on the metadata information, that is, the CWE-ID corresponding to each vulnerability, the number of occurrences of each CWE-ID is counted, and the proportion of the vulnerabilities corresponding to each CWE-ID in all vulnerabilities in the NVD vulnerability database is obtained. The computing device can determine the number of use cases corresponding to the CWE-ID in the preset use case set based on the proportion corresponding to each CWE-ID.

[0078] It can be understood that determining the number of vulnerabilities of each defect type based on the NVD vulnerability database has a clear and professional data source; the vulnerability information in the database has been strictly screened and verified, with high credibility. By counting the number of vulnerabilities corresponding to each CWE-ID in the NVD, it is possible to quickly identify which types of vulnerabilities are more common or serious in the system, so as to design more use cases for these critical vulnerabilities.

[0079] In some embodiments, in order to more comprehensively cover the functions and scenarios of software code, multiple types of use cases can be designed, and a comprehensive and systematic test plan can be formed by multiple types of use cases together.

[0080] As a feasible implementation method, the use case types in the preset use case set include the first type, the second type, and the third type. Among them, the code sample in the use case of the first type is the code in the preset software; the code sample in the use case of the second type is the virtual code generated based on the preset generation model; the code sample in the use case of the third type is the simulation code obtained by modifying the code in the preset software.

[0081] It can be understood that the code sample in the use case of the first type is the code in the preset software, that is, the code sample in the use case of the first type is the code in the real use case, which usually contains rich business logic and complex interaction processes. Using these codes as test cases can more truly reflect the actual running situation of the preset software. Testing the code detection tool with these real use cases can more effectively evaluate the ability of the code detection tool to identify and handle potential problems in actual code, thereby improving the authenticity and accuracy of detection.

[0082] As a feasible implementation method, code samples in the first type of use cases can be determined based on a product defect library and / or open-source software defects. The embodiments of the present application do not limit the determination method of code samples in the first type of use cases.

[0083] The code samples in the second type of use cases are virtual codes generated based on a preset generation model. It should be understood that the preset generation model has powerful pattern recognition capabilities and can identify potential patterns and rules from complex data. In code generation, the model can identify patterns such as the syntax structure of code, function definitions, and variable naming, and generate new code based on these patterns. Exemplarily, the preset generation model can be a neural network model, a diffusion model, etc.

[0084] That is to say, the virtual codes in the second type of use cases (virtual use cases) are not real codes written in actual development, but simulated codes automatically generated according to the preset generation model. They can also be called virtual use cases designed based on a simulation environment or data. Virtual use cases can simulate complex business scenarios or system behaviors when it is impossible to directly access actual software or data, so as to evaluate the performance and stability of the software. On the one hand, virtual codes can be generated in large quantities based on the preset generation model, so as to cover more test scenarios. Using virtual use cases to evaluate code detection tools helps to enhance the diversity of test scenarios, enabling a computing device to evaluate code detection tools under multiple test scenarios. Through virtual use cases, complex scenarios that are difficult to implement can be simulated, thereby evaluating the performance and stability of code detection tools in these scenarios. On the other hand, virtual use cases can be adjusted and modified at any time according to needs to adapt to different test requirements and environments, thereby improving the flexibility of testing code detection tools.

[0085] As a feasible implementation method, code samples in the second type of use cases can be determined based on OWASP Benchmark, Juliet Test Suite, or Openrasp. The embodiments of the present application do not limit the determination method of code samples in the first type of use cases.

[0086] The code samples in the third type of use cases are simulation codes obtained by modifying the codes in a preset software. For example, the codes in the preset software are modified so that the modified codes contain preset defects (such as access defects, etc.). The simulation codes are obtained by modifying the actual codes in the preset software, so they are highly similar to the real software operating environment. This similarity enables the simulation codes to more accurately simulate potential problems in the real software, thereby conducting a more effective evaluation of code detection tools. In addition, the simulation codes can be easily generated and modified, thereby reducing the complexity of evaluating code detection tools and improving the flexibility of testing code detection tools.

[0087] As can be seen, the preset use case set in this embodiment may include real use cases, virtual use cases, and simulation use cases. Real code samples contain rich business logic and complex interaction processes, and these characteristics enable them to more realistically reflect the actual running situation of the preset software. By using these codes as test cases, it can be ensured that the code detection tool faces codes highly similar to the real software environment during the evaluation process, thereby improving the accuracy and practicality of evaluating the code detection tool. Virtual use cases utilize virtual environments or data to simulate real scenarios, can be quickly generated, improve test efficiency, and reduce test costs. Simulation use cases are obtained by modifying the codes in the preset software. Since the simulation codes are highly similar to the codes in the software, using the simulation codes for evaluation can more accurately reflect the performance of the code detection tool in actual applications, improving the accuracy of evaluating the code detection tool. At the same time, the simulation codes can be conveniently generated and modified, which helps to discover the deficiencies of the code detection tool when dealing with complex codes and potential problems, thereby promoting the optimization and improvement of the tool. Thus, it can be seen that the preset use case set containing real use cases, virtual use cases, and simulation use cases can significantly improve the efficiency and quality of testing the code detection tool, enhance the controllability and repeatability of testing, support multiple test scenarios, and help to more accurately evaluate the defect detection effect of the code detection tool.

[0088] In some embodiments, the Common Weakness Enumeration (CWE) types include multiple defect types, such as pillar weakness, class weakness, base weakness, variant weakness, etc.

[0089] Among them, the pillar weakness is the highest-level defect type in CWE, which describes a category of errors. The characteristics of the pillar weakness are very abstract in describing the defect and do not point to the specific error location or the type of resource affected. For example, Incorrect Calculation (CWE-682) is an example of a pillar weakness. CWE-682 describes an error but does not indicate where this error occurs or which resource it affects.

[0090] The class weakness describes a category of defects with common characteristics and does not point to the specific error location or the type of resource affected. This enables the class weakness to be used for grouping and classifying related things, thereby helping researchers and developers better understand and solve security problems. For example, Uncontrolled Resource Consumption (CWE-400) is an example of a class weakness, which describes the problem (uncontrolled) that may occur when describing the behavior (consumption) related to any type of resource.

[0091] Variant defects describe defects in a very specific and detailed manner, including multiple dimensions such as the behavior, attributes, technologies, languages, and resources involved in the defects. This enables variant defects to precisely describe and locate specific problems in the code. For example, "using sizeof() on a pointer type" (CWE-467) is an example of a variant defect, which describes a problem that may occur when applying a function to a pointer resource in a language that supports pointer types.

[0092] Basic defects lie between variant defects and class defects, describing defects in an abstract way but with enough details to infer specific methods for detection and prevention. Basic defects are usually described by 2 to 3 dimensions among behavior, attributes, technologies, languages, and resources. This makes basic defects abstract enough to cover a wide range of defect types and contain enough details to support specific detection work. For example, "using externally-controlled format strings" (CWE-134) is an example of a basic defect, which describes a problem (improper operation) that may occur when taking an action (using) on a specific resource (format string) with a given attribute (externally-controlled).

[0093] As a feasible implementation method, the defect type in the embodiments of this application is the basic defect type in the Common Weakness Enumeration (CWE) type.

[0094] It can be understood that the description method of basic defects is both abstract and contains enough details, which enables researchers and developers to more accurately identify potential problems in software or hardware. By focusing on aspects such as the behavior, attributes, and resources involved in basic defects, defects in the code can be more effectively located and analyzed. Therefore, setting the defect type in the preset use case set to the basic weakness type can more accurately evaluate the performance of the code detection tool for detecting code defects.

[0095] As a feasible implementation method, after obtaining the CWE-1000 (research concept) view, through secondary filtering (retaining basic defect nodes and removing nodes that cannot map to vulnerabilities), the CWE types that the code detection tool may detect are obtained, and then the preset use case set is designed based on the CWE IDs of the CWE types.

[0096] Retaining basic defect nodes means focusing on the basic defect types defined in CWE, which are the core of security vulnerabilities. By removing nodes that cannot map to vulnerabilities, redundant information that has no direct association with specific vulnerabilities can be excluded, thus more accurately locating security vulnerabilities. The defect type in the resulting preset use case set focuses on basic defects with clear vulnerability mappings, which reduces the time for screening effective vulnerability mappings in a large amount of information and improves the evaluation efficiency.

[0097] S102. Test the code detection tool using a preset test case set to obtain a test result.

[0098] Among them, the functions of the code detection tool include detecting code defects; the test result is used to indicate the detection ability of the code detection tool for code defects.

[0099] It should be understood that the embodiments of the present application do not limit the code detection tool, and the code detection tool can be any static code security scanning tool. Exemplarily, the code detection tool can be Coverity, Fortify, Checkmarx, CodeQL or Deepsource.

[0100] After obtaining the preset test case set, input the code samples in the test cases in the preset test case set into the code detection tool one by one. The code detection tool can analyze the code samples, perform defect detection on them, and identify the corresponding defect types. Observe and record the test results of the code detection tool for each test case. It can be understood that the test results of the code detection tool can include the detected defect types, location detection time, and any relevant diagnostic information, etc., and the embodiments of the present application do not limit this.

[0101] After the code detection tool performs code defect detection on each test case in the preset test case set, it can conduct in-depth analysis based on the test results of the code detection tool for the test cases to determine the specific performance of the code detection tool's detection ability for code defects and obtain the test result.

[0102] Exemplarily, as a feasible implementation method, the detection ability of the code detection tool for code defects can be determined by calculating indicators such as CWE coverage rate (the ratio of the number of code defect types that the code detection tool can detect to the total number of code defect types in the preset test case set), recall rate (the ratio of the number of code samples with defects detected by the code detection tool to the number of code samples with defects in the preset test case set), etc.

[0103] It can be seen from the above S101 and S102 that the distribution of the test cases in the preset test case set in the embodiments of the present application is similar to the actual vulnerability distribution. Therefore, they are more likely to cover the real vulnerabilities existing in the code detection tool. In this way, when using the preset test case set to test the code detection tool, this test method can more accurately reveal the weak links in the code detection tool, and the obtained test results can more truly reflect the performance of the code detection tool when facing actual vulnerabilities. Thus, the obtained test results can be used to accurately judge the detection ability of the code detection tool for code defects.

[0104] It should be noted that since the classification rules of different code detection tools for defect types are not the same as those of CWE, that is, the ID of the defect type detected by each code detection tool (hereinafter referred to as the rule ID) is not the same as the CWE-ID. Therefore, in order to evaluate the detection capabilities of multiple code detection tools using the same standard, it is necessary to determine the mapping relationship between the rule ID and the CWE-ID. After obtaining the detection results of the code samples by the code detection tool, it is necessary to map the detected defect types from the corresponding rule ID to the CWE-ID.

[0105] Based on this, as a feasible implementation method, calibration is performed using the code defect type corresponding to the use case (such as the basic defect type in the CWE type) as the baseline. Specifically, for a use case corresponding to a certain code defect type CWE-ID1, if the code detection tool detects that there is a defect in the code sample in this use case based on the rule ID1, then the code defect type CWE-ID1 is the defect type that the code detection tool can detect. At this time, there is a mapping relationship between CWE-ID1 and the rule ID1 of the code detection tool. Further, the final mapping relationship between the rule ID and the CWE-ID can also be obtained by manually examining the mapping relationship between the rule ID and the CWE ID of each detection tool.

[0106] Exemplarily, please refer to Table 1. Table 1 shows a mapping relationship between the rule ID and the CWE-ID provided by an embodiment of the present application. Taking the code detection tool as Tool A as an example, Tool A detects that there is a defect in the use case corresponding to CWE-78 based on the rules A8 and A12. Then, the rules A8 and A12 of Tool A and CWE-78 form a mapping relationship.

[0107] Table 1

[0108] CWE-ID Tool A Rule ID Tool B Rule ID Tool C Rule ID CWE-78 A8, A12 B34 C90 CWE-1241 A458 B71 C292 CWE-XXX A-XXX B-XXX C-XXX

[0109] Another exemplarily, please refer to Table 2. Table 2 shows another mapping relationship between the rule ID and the CWE-ID provided by an embodiment of the present application. In Table 2, taking CWE-22 as an example, the defect described by CWE-22 refers to "improper restriction of the path name of a restricted directory". For Tool A, the rule ID corresponding to CWE-22 is "S2091". For Tool B, the rule ID corresponding to CWE-22 is "Detect possible partial path traversal vulnerabilities in Java code, detect possible partial path traversal vulnerabilities caused by input received from a remote source, and detect the situation of calling the openStream() method on input that may contain malicious URLs". For Tool C, the rule ID corresponding to CWE-22 is "JAVA-S1001".

[0110] Table 2

[0111]

[0112]

[0113] In this way, after each code detection tool detects the preset use case set, each defect type detected by the code detection tool can be mapped from the rule ID to the CWE-ID, so that the evaluation criteria of multiple code detection tools in the obtained test results are consistent.

[0114] In some embodiments, in order to more accurately judge the detection ability of the code detection tool for code defects, the ability of the code detection tool can be comprehensively judged by determining the detection accuracy rate, the types of detected defects, and the detection time of the code detection tool.

[0115] As a feasible implementation manner, the test result includes at least one of a first result, a second result, and a third result.

[0116] Among them, the first result is used to characterize the number of types of code defects that the code detection tool can detect. The second result is used to characterize the accuracy of the code detection tool in detecting code defects. The third result is used to characterize the duration taken by the code detection tool to detect the preset use case set.

[0117] The first result is used to characterize the number of types of code defects that the code detection tool can detect. It can be understood that the detection capabilities of different static code scanning tools are different, and some code detection tools are insufficient in detecting certain types of defects, resulting in the inability to accurately detect all defect types. Therefore, the detection ability of the code detection tool for code defects can be determined based on the number of types of code defects that the code detection tool can detect (the first result). Here, the types of code defects that the code detection tool can detect refer to that for a use case of a certain type of code defect, if the code detection tool detects that the code sample in the use case has a defect, it is considered that the code detection tool can detect that type of code defect.

[0118] As a specific implementation manner, the first result includes the ratio of the number of types of code defects that the code detection tool can detect to the total number of types of code defects in the preset use case set.

[0119] Since the preset use case set includes use cases corresponding to multiple types of code defects, if the number of types of code defects that the code detection tool can detect is larger, it indicates that the ability and coverage of the code detection tool in code defect detection are higher, that is, the detection ability of the code detection tool is stronger. Therefore, the detection ability of the code detection tool for code defects can be determined based on the ratio of the number of types of code defects that the code detection tool can detect to the total number of types of code defects in the preset use case set.

[0120] The second result is used to characterize the accuracy of the code detection tool in detecting code defects. It can be understood that accuracy here refers to the ability of the code detection tool to correctly identify and report the defects existing in the code. It reflects the effectiveness of the code detection tool in handling complex code structures, identifying potential risks, and providing reliable detection results.

[0121] As a specific implementation, the second result is obtained based on the number of the first use cases and the number of the second use cases in the preset use case set; the code samples in the first use cases have defects of the defect type; the code samples of the second use cases are the code samples detected by the code detection tool as having defects in the preset use case set.

[0122] That is to say, the second result can be determined based on the number of use cases with defects in the preset use case set and the number of use cases with defects detected by the code detection tool.

[0123] Specifically, as a feasible implementation, the second result may include one or more accuracy evaluation metrics. Exemplarily, the accuracy evaluation metrics may include: Accuracy, Precision, or Recall.

[0124] Accuracy is the ratio of the number of use cases in which the code detection tool correctly detects defects to the total number of use cases; Precision is the ratio of the number of actually defective use cases among the use cases detected by the code detection tool as having defects; Recall is the ratio of the number of use cases with actual defects correctly detected by the code detection tool.

[0125] The determination method of the accuracy evaluation metrics includes: after the code detection tool finishes detecting the use cases in the preset use case set, the following accuracy metrics can be counted:

[0126] True Positives (TP): The number of use cases with defects correctly detected by the code detection tool.

[0127] False Positives (FP): The number of use cases that the code detection tool wrongly considers as having defects but actually do not have defects.

[0128] True Negatives (TN): The number of use cases that the code detection tool correctly considers as not having defects.

[0129] False Negatives (FN): The number of use cases with actual defects that the code detection tool fails to detect.

[0130] Based on the above metrics, the accuracy evaluation metrics can be calculated respectively through the following formulas: The formula for accuracy is: Accuracy = (TP + TN) / (TP + TN + FP + FN); the formula for precision is: Precision = TP / (TP + FP); the formula for recall is: Recall = TP / (TP + FN).

[0131] It should be understood that in practical applications, since it is impossible to know which of all the undetected use cases are defect-free, that is, the above TN is usually not directly calculated. Therefore, as a feasible implementation method, in practical applications, the second result may include precision and recall. According to metrics such as precision and recall, the accuracy of the code detection tool in detecting code defects can be comprehensively evaluated. Among them, precision reflects the accuracy of the code detection tool in detecting defects; recall reflects the comprehensiveness of the code detection tool in detecting defects.

[0132] In some embodiments, if all types of defects are regarded as a whole to calculate the recall rate, the evaluation result may be biased due to the differences in the number, distribution, or characteristics of different types of defects. Therefore, as another feasible implementation method, the recall rate of each type of defect can be calculated separately, and then the total recall rate can be determined by weighted summation. This method can more specifically evaluate the recognition ability of the code detection tool for different types of defects, and thus more accurately reflect the performance of the code detection tool on different types of defects.

[0133] The third result is used to characterize the duration of the code detection tool for detecting the preset use case set. It can be understood that the third result refers to the total time spent by the code detection tool from the start of executing the use cases in the preset use case set to the completion of all use case detections. This time includes various links such as use case loading, execution, result collection, and analysis. To accurately calculate the third result, it is usually necessary to record the start time and end time when the code detection tool executes the test preset use case set, and calculate the total duration through these two time points. In addition, the total duration can also be broken down into the execution times of each test case to more deeply understand the performance of the code detection tool in detecting different test cases.

[0134] As a feasible implementation method, the third result is the ratio of the duration of the code detection tool for detecting the preset use case set to the reference duration.

[0135] Among them, the reference duration is preset, and the embodiments of the present application do not limit this. Exemplarily, as a feasible implementation method, the reference duration can be the sum of the detection times of multiple code detection tools.

[0136] It can be understood that the third result determined by the ratio of the time taken by the code detection tool to detect the preset test case set to the reference time can intuitively reflect the relative efficiency of the code detection tool when executing the preset test case set. By comparing with the reference time, it can be judged whether the code detection tool has achieved the expected detection speed or efficiency. When comparing multiple code detection tools or different versions of the same code detection tool, the third result can be used as an important performance indicator. By comparing the third results of different code detection tools, their relative efficiency can be evaluated, so as to select a code detection tool or version with better performance.

[0137] It can be seen that in the solution provided in this embodiment, the test result includes at least one of the first result, the second result, and the third result. Since the first result is used to characterize the number of types of code defects that the code detection tool can detect, through the first result, the ability of the code detection tool in code defect detection can be intuitively understood, including the number of defect types that can be detected. This helps to evaluate the detection scope and comprehensiveness of the code detection tool. The second result is used to characterize the accuracy of the code detection tool in detecting code defects. Through the second result, the accuracy of the code detection tool in detecting code defects can be evaluated, that is, whether the detected defects actually exist, and whether there are false positives or false negatives. This helps to ensure the reliability and effectiveness of the test results. The third result is used to characterize the time taken by the code detection tool to detect the preset test case set. Through the third result, the efficiency of the code detection tool when executing the preset test case set can be measured, that is, the time required for detection, which helps to evaluate the performance of the code detection tool. Therefore, when the test result includes at least one of the first result, the second result, and the third result, by comprehensively considering multiple results, the detection ability, accuracy, and efficiency of the code detection tool can be comprehensively evaluated, so as to more accurately understand its performance in code defect detection.

[0138] In some embodiments, in software testing, a test case is an important guiding document for guiding test execution personnel on how to conduct specific tests. The test case contains a code sample and a first label. The code sample is the specific object of the test. By executing this code, it can be verified whether its function works as expected. The first label is used to indicate whether the code sample has defects, which helps the tester quickly determine whether the code sample has passed the test; in the case of defects, the first label can further indicate the type of defect, which helps the tester more accurately locate the problem and take corresponding repair measures.

[0139] Based on this, the test result of the code detection tool can be determined through the first label in the test case and the detection result of the code detection tool.

[0140] Specifically, as a feasible implementation method, please refer toFigure 3 , S102 can be specifically implemented as the following steps:

[0141] S1021. For each use case in the preset use case set, use a code detection tool to detect the code sample in the use case to obtain a second label for the use case.

[0142] Among them, the second label is used to indicate whether the code sample has defects, that is, whether the code sample detected by the code detection tool has defects.

[0143] Before using the code detection tool to detect the code sample in the use case, it is necessary to ensure that the code detection tool is in a testable state, prepare test cases and code samples, as well as necessary test environments and tools. Input the code sample of each use case in the preset use case set into the code detection tool and run the corresponding detection logic. This may include various methods such as static code analysis, dynamic testing, and performance testing. Analyze the execution result of the code sample according to the output or log of the code detection tool and generate a second label.

[0144] It can be understood that the second label is used to indicate whether the code sample has passed the detection in the code detection tool, that is, whether defects have been detected. If the code sample fails the detection, the second label may further provide detailed information about the defects, such as the type, location, and severity of the defects. The embodiments of the present application do not limit this.

[0145] S1022. Based on the first label and the second label of each use case, determine the test result.

[0146] The first label is used to indicate whether the code sample has defects and the type of defects in the case of existing defects; that is, the first label is used to represent the actual defect situation of the code sample. The second is obtained by detecting the code sample in the use case by the code detection tool and is used to represent the predicted defect situation detected by the code detection tool. Therefore, the test result can be determined based on the first label and the second label of each use case.

[0147] If the two are consistent, it means that the code detection tool performs well when detecting this code sample and can accurately identify the actually existing defects (if any). If the two are inconsistent, it means that there is an error in the code detection tool during the detection process, and it may have missed reporting the actually existing defects or misreported the non-existing defects.

[0148] As a feasible implementation method, S1022 can be specifically implemented as the following steps:

[0149] S11. Use the use cases whose first label indicates that the code sample has defects as the first use cases.

[0150] S12. Use the test cases with the second label indicating that the code sample is defective as the second test cases, and use the defect types indicated by the first label of the second test cases as the defect types that the code detection tool can detect.

[0151] S13. Determine the first result based on the number of defect types that the code detection tool can detect, and determine the second result based on the number of the first test cases and the number of the second test cases.

[0152] Among them, the test result includes the first result and the second result. The first result is used to represent the number of code defect types that the code detection tool can detect; the second result is used to represent the accuracy of the code detection tool in detecting code defects.

[0153] As a feasible implementation method, the first result can be determined based on the ratio of the number of the second test cases to the number of the first test cases.

[0154] It should be understood that the specific determination process and analysis of the first result and the second result can refer to the above embodiments, and the present application will not elaborate here.

[0155] It can be seen from the above S1021 - S1022 that the first label is a true indication of whether the code sample is defective and the specific type of defect when there is a defect. It represents the actual state of the code sample and is a benchmark for evaluating the detection accuracy of the code detection tool. The second label is the result obtained after the code detection tool detects the code sample, indicating the defect situation that the code detection tool believes exists in the code sample. It reflects the performance of the code detection tool in the automated detection process. By comparing the first label and the second label, the performance of the code detection tool in code defect detection can be effectively evaluated, and a more accurate test result can be obtained.

[0156] In some embodiments, after obtaining the test result of the code detection tool, the final evaluation result can be calculated through the multi - dimensional evaluation indicators in the test result and the scoring formula, so as to facilitate the user to intuitively determine the performance of the code detection tool.

[0157] As a feasible implementation method, the method provided in the embodiments of the present application further includes: performing a weighted sum of the first result, the second result, and the third result to obtain the final test result of the code detection tool. Among them, the weights of the first result, the second result, and the third result can be set according to the actual situation.

[0158] Exemplarily, as a feasible implementation method, please refer to Table 3, which is used to represent the determination method of the final test result of the code detection tool.

[0159] Table 3

[0160]

[0161] It can be seen that serial number A in Table 3 is the first result in the above embodiments; serial number B in Table 3 is the second result in the above embodiments; serial number D in Table 3 is the third result in the above embodiments.

[0162] After the code detection tool finishes detecting all the use cases in the preset use case set, the test result of the code detection tool can be determined through Table 2 above. Further, according to the preset weights, the score of the code detection tool can be determined by weighted summation.

[0163] Exemplarily, when the weight is as shown in Table 3, the score S of the code detection tool can be determined by the following method: S = A * 50% + B * 40% + (1 - D * 10%).

[0164] It can be understood that the score can convert the test result into a specific value, making the comparison and evaluation of the test result more intuitive and objective. Through the score, it is easier to identify the performance of the code detection tool in detection performance, thus providing a more accurate and intuitive basis for staff to select the code scanning tool.

[0165] In some other embodiments, the score is usually the result of a comprehensive evaluation of multiple aspects of the code detection tool, such as overall performance, accuracy, ease of use, update frequency, support degree, etc. A high score means that the code detection tool performs well in multiple key areas and can provide users with efficient and reliable security vulnerability scanning services. However, a code detection tool may perform well in some aspects (such as high accuracy), but may be lacking in other aspects (such as low coverage). Therefore, both the score and the coverage rate of the code detection tool can be concerned simultaneously.

[0166] As a feasible implementation method, after obtaining the test result of the code detection tool, a corresponding coverage curve can be drawn based on the CWE-ID that the code detection tool can detect in the test result.

[0167] Specifically, in actual applications, since different code detection tools have different detection capabilities for different code languages, multiple code languages can be detected separately to determine the corresponding coverage curves.

[0168] Exemplarily, in combination with Figure 4As shown in the figure, when performing code detection on C language, the CWE types that code detection tool A can detect are only 2 (CWE-396, CWE-688), and the CWE types that code detection tool B can detect are 5 (CWE-242, CWE-367, CWE-676, CWE-685, CWE-688). The CWE coverage rate of code detection tool A is 40.17%, which is better than that of code detection tool B (18.75%).

[0169] Another exemplary one, in combination with Figure 5 As shown in the figure, when performing code detection on Java language, the CWE types that code detection tool A can detect are 6 (CWE-510, CWE-338, CWE-459, CWE-336, CWE-667, CWE-760,), and the CWE types that code detection tool B can detect are 4 (CWE-338, CWE-378, CWE-336, CWE-614). The CWE coverage rate of code detection tool A is 41.67%, which is better than that of code detection tool B (14.17%).

[0170] Combined with the above analysis, it can be seen that after obtaining the test results of the code detection tool, determining the score of the code detection tool based on the test results can intuitively select the tool that best supports security vulnerability scanning, and at the same time, it can display the coverage curve graph of each CWE ID by each code detection tool. This enables the staff to combine multiple code detection tools to obtain the highest comprehensive coverage rate of CWE IDs, thereby achieving the highest security vulnerability detection rate.

[0171] In some embodiments, as a feasible implementation method, in order to facilitate distinguishing whether a code sample has defects, that is, whether the use case is a positive example or a negative example, the positive examples in the preset use case set can be prefixed with "good" and suffixed with a number, which is convenient for the code detection tool to judge false positives through regular expressions; the negative examples are prefixed with "bad" and suffixed with a number, which is convenient for the code detection tool to judge missed reports through regular expressions.

[0172] Exemplarily, for a code sample with code defects, it can be prefixed with "bad". If there are multiple code samples with code defects in the same file, a number is used as the suffix, such as bad1, bad2.

[0173] For a code sample without code defects, it can be prefixed with "good". If there are normal code samples in the same file, a number is used as the suffix, such as good1, good2.

[0174] The embodiment of the present application also provides a code detection method, which can be applied to the above computing device, and the embodiment of the present application does not make any restrictions on this.

[0175] Please refer to Figure 6 , the code detection method provided by the embodiment of the present application includes:

[0176] S501. According to at least one target defect type to be detected, obtain a tool set.

[0177] Among them, the tool set includes at least one code detection tool, and each code detection tool can detect code defects of at least one defect type; the defect types that each code detection tool can detect are tested by using a preset use case set. The preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type; the defect types that at least one code detection tool can detect cover the target defect type.

[0178] That is to say, first determine the code defects of at least one defect type that each code detection tool can detect based on the preset use case set, and then use the code detection tools that can detect the target defect type as the tools in the tool set.

[0179] It should be understood that the method of testing each code detection tool based on the preset use case set can refer to the above embodiment, and the present application will not elaborate here.

[0180] S502. Use each code detection tool in the tool set to perform defect detection on the target code.

[0181] It should be understood that the code detection tools in the tool set can perform defect detection on the target defect type, that is, they can all detect the defects of the target defect type. Therefore, defect detection is performed on the target code based on using each code detection tool in the tool set.

[0182] When performing defect detection on the target code, each code detection tool in the tool set will run independently, analyze the code to find the defect types it specializes in. Multiple code detection tools review the code from different perspectives, which can complement each other and reduce the possibility of missed reports. For example, one code detection tool may be good at detecting memory leaks, while another is more proficient in discovering security vulnerabilities. Through cross-verification of multiple code detection tools, the number of false positives can be significantly reduced. If a defect is detected by multiple code detection tools at the same time, then it is very likely to be real.

[0183] It can be seen that the solution provided by the embodiment of the present application performs defect detection on the target code through multiple code detection tools in the tool set that can perform defect detection on the target defect type. Multiple code detection tools review the code from different perspectives, which can complement each other and reduce the possibility of missed reports, thereby improving the accuracy of defect detection.

[0184] The above mainly introduces the solution provided by the embodiments of the present application from the perspective of methods. To implement the above functions, the test device, code detection device, or electronic device includes the corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the manner of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0185] The embodiments of the present application can, according to the above method, exemplarily divide the functional modules of the test device, code detection device, or electronic device. For example, the test device can include each functional module corresponding to each functional division, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. It should be noted that the division of modules in the embodiments of the present application is illustrative, only a logical functional division, and there can be other division methods in actual implementation.

[0186] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a test device provided by the present application. As Figure 7 shown, the test device 60 includes: an acquisition module 61 and a test module 62.

[0187] Among them, the acquisition module 61 is used to acquire a preset use case set; the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type.

[0188] The test module 62 is used to test the code detection tool using the preset use case set to obtain a test result; among them, the function of the code detection tool includes detecting code defects; the test result is used to indicate the detection ability of the code detection tool for code defects.

[0189] In some embodiments, the test result includes at least one of a first result, a second result, and a third result; the first result is used to characterize the number of code defect types that the code detection tool can detect; the second result is used to characterize the accuracy of the code detection tool in detecting code defects; the third result is used to characterize the duration of the code detection tool in detecting the preset use case set.

[0190] In some embodiments, a use case includes a code sample and a first label; the first label is used to indicate whether there is a defect in the code sample and the type of the defect in the case of a defect; the first result includes the ratio of the number of code defect types that a code detection tool can detect to the total number of code defect types in a preset use case set; and / or, the second result is obtained based on the number of first use cases and the number of second use cases in the preset use case set; there is a defect of a defect type in the code sample of the first use case; the code sample of the second use case is a code sample in the preset use case set that is detected by the code detection tool to have a defect; and / or, the third result is the ratio of the time taken by the code detection tool to detect the preset use case set to a reference time.

[0191] In some embodiments, a use case includes a code sample and a first label; the first label is used to indicate whether there is a defect in the code sample and the type of the defect in the case of a defect; the test module 62 is specifically configured to, for each use case in the preset use case set, use the code detection tool to detect the code sample in the use case to obtain a second label of the use case; the second label is used to indicate whether there is a defect in the code sample; and determine a test result based on the first label and the second label of each use case.

[0192] In some embodiments, the test module 62 is specifically configured to use the use case whose first label indicates that the code sample has a defect as the first use case; use the use case whose second label indicates that the code sample has a defect as the second use case, and use the defect type indicated by the first label of the second use case as the defect type that the code detection tool can detect; determine the first result according to the number of defect types that the code detection tool can detect, and determine the second result according to the number of first use cases and the number of second use cases; the test result includes the first result and the second result, the first result is used to characterize the number of code defect types that the code detection tool can detect; the second result is used to characterize the accuracy of the code detection tool in detecting code defects.

[0193] In some embodiments, the use case types in the preset use case set include a first type, a second type, and a third type; the code sample in the use case of the first type is the code in the preset software; the code sample in the use case of the second type is virtual code generated based on a preset generation model; the code sample in the use case of the third type is simulation code obtained by modifying the code in the preset software.

[0194] In some embodiments, the defect type is the base weakness type in the Common Weakness Enumeration (CWE) type of general vulnerabilities.

[0195] An embodiment of the present application further provides a code detection device, including an acquisition module and a detection module.

[0196] Among them, the obtaining module is configured to obtain a set of tools according to at least one target defect type to be detected; the set of tools includes at least one code detection tool, and each code detection tool can detect code defects of at least one defect type; the defect types that each code detection tool can detect are tested by using a preset use case set, the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type; the defect types that at least one code detection tool can detect cover the target defect types.

[0197] The detection module is configured to use each code detection tool in the set of tools to perform defect detection on the target code.

[0198] An embodiment of the present application also provides a computing device, which includes a processor and a memory. The processor is connected to the memory, and the memory stores computer-executable instructions. When the processor executes the computer-executable instructions, the data processing method in the above embodiment is implemented. The specific form of the computing device in the embodiment of the present application is not limited in any way. For example, the computing device may specifically be a terminal device or a network device. Among them, the terminal device may be referred to as: terminal, user equipment (UE), terminal device, access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent or user device, etc. The terminal device may specifically be a mobile phone, an augmented reality (AR) device, a virtual reality (VR) device, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The network device may specifically be a server, etc. Among them, the server may be a physical or logical server, or may be two or more physical or logical servers sharing different responsibilities and cooperating with each other to implement the various functions of the server.

[0199] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program runs on a computer, the computer is enabled to execute the method performed by any one of the above computing devices.

[0200] For the explanations and beneficial effects descriptions of the relevant content in any one of the above computer-readable storage media, reference may be made to the corresponding embodiments above, and details are not repeated here.

[0201] The embodiments of the present application also provide a chip. The chip integrates a control circuit for implementing the functions of the above-mentioned computing device and one or more ports. Optionally, the functions supported by the chip can be referred to the above, and will not be elaborated here. Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiments can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a random access memory, etc. The above-mentioned processing unit or processor can be a central processing unit, a general-purpose processor, an application specific integrated circuit (ASIC), a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.

[0202] The embodiments of the present application also provide a computer program product containing instructions. When the instructions run on a computer, the computer is made to execute any one of the methods in the above embodiments. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that contains one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as an SSD), etc.

[0203] It should be noted that the devices for storing computer instructions or computer programs provided in the embodiments of the present application, such as but not limited to, the above-mentioned memory, computer-readable storage medium, and communication chip, etc., are all non-transitory.

[0204] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website, a computer, a server, or a data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated media. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0205] Although the present application has been described in connection with various embodiments, however, in the process of implementing the claimed present application, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit can implement several functions recited in the claims. Certain measures are recited in mutually different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0206] Although the present application has been described in connection with specific features and their embodiments, it is obvious that various modifications and combinations can be made without departing from the spirit and scope of the present application. Accordingly, the present specification and the drawings are only exemplary descriptions of the present application defined by the appended claims, and are considered to have covered any and all modifications, variations, combinations, or equivalents within the scope of the present application. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A testing method for a code detection tool, characterized in that, The method includes: Obtaining a preset use case set; the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one of the defect types is positively correlated with the number of vulnerabilities belonging to the defect type; Testing a code detection tool using the preset use case set to obtain a test result; wherein, the function of the code detection tool includes detecting code defects; the test result is used to indicate the code defect detection ability of the code detection tool.

2. The test method of the code detection tool according to claim 1, characterized in that, The test result includes at least one of a first result, a second result, and a third result; the first result is used to characterize the number of code defect types that the code detection tool can detect; The second result is used to characterize the accuracy of the code detection tool in detecting code defects; The third result is used to characterize the duration taken by the code detection tool to detect the preset use case set.

3. The testing method of the code detection tool according to claim 2, characterized in that, The use case includes a code sample and a first label; the first label is used to indicate whether the code sample has a defect and, in the case of a defect, the defect type of the defect; The first result includes the ratio of the number of code defect types that the code detection tool can detect to the total number of code defect types in the preset use case set; and / or, The second result is obtained based on the number of first use cases and the number of second use cases in the preset use case set; the code sample in the first use case has a defect of the defect type; the code sample of the second use case is the code sample in the preset use case set that is detected by the code detection tool as having a defect; and / or, The third result is the ratio of the duration taken by the code detection tool to detect the preset use case set to a reference duration.

4. The testing method of the code detection tool according to any one of claims 1-3, characterized in that The use case includes a code sample and a first label; the first label is used to indicate whether the code sample has a defect and, in the case of a defect, the defect type of the defect; The testing the code detection tool using the preset use case set to obtain a test result includes: For each use case in the preset use case set, using the code detection tool to detect the code sample in the use case to obtain a second label for the use case; the second label is used to indicate whether the code sample has a defect; Determining the test result based on the first label and the second label of each use case.

5. The testing method of the code detection tool according to claim 4, characterized in that, The determining the test result based on the first label and the second label of each use case includes: Regarding the use cases where the first label indicates that the code sample has a defect as first use cases; Regarding the use cases where the second label indicates that the code sample has a defect as second use cases, and using the defect type indicated by the first label of the second use cases as the defect types that the code detection tool can detect; Determine a first result according to the number of defect types that the code detection tool can detect, and determine a second result according to the number of the first use cases and the number of the second use cases; the test result includes the first result and the second result, and the first result is used to characterize the number of code defect types that the code detection tool can detect; the second result is used to characterize the accuracy of the code detection tool in detecting code defects.

6. The testing method of the code detection tool according to any one of claims 1-5, characterized in that, The use case types in the preset use case set include a first type, a second type, and a third type; the code samples in the use cases of the first type are the codes in the preset software; the code samples in the use cases of the second type are virtual codes generated based on a preset generation model; the code samples in the use cases of the third type are simulation codes obtained by modifying the codes in the preset software.

7. The testing method of the code detection tool according to any one of claims 1-6, characterized in that, The defect type is the base weakness type in the Common Weakness Enumeration (CWE) type of general vulnerabilities.

8. A code detection method, characterized in that, The method includes: According to at least one target defect type to be detected, obtain a tool set; the tool set includes at least one code detection tool, and each code detection tool can detect code defects of at least one defect type; the defect types that each code detection tool can detect are tested by using a preset use case set, the preset use case set includes use cases corresponding to multiple code defect types, and the number of use cases corresponding to one defect type is positively correlated with the number of vulnerabilities belonging to the defect type; the defect types that the at least one code detection tool can detect cover the target defect type. Use each code detection tool in the tool set to perform defect detection on the target code.

9. A computing device, characterized in that, Include: A processor and a memory; The processor is connected to the memory, and the memory is used to store computer execution instructions. The processor executes the computer execution instructions stored in the memory so that the computing device implements the method according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, Store computer instructions, and when the computer instructions run on a computing device, the computing device is caused to execute the method according to any one of claims 1-8.

Images