Fuzzy test case selection methods, devices, electronic equipment and storage media

Abstract

This application discloses a method, apparatus, electronic device, and storage medium for fuzzy test case selection. The method includes: acquiring fuzzy test cases; determining evaluation factors for the fuzzy test cases, the evaluation factors including at least one of the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case; and, in response to the evaluation factors meeting preset requirements, identifying the fuzzy test cases as target test cases. Through the above method, this application can perform targeted optimization and selection of fuzzy test cases, improving testing efficiency and product quality.

Description

Technical Field

[0001] This application relates to the field of testing technology, and in particular to a fuzzy test case selection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, in the field of testing technology, fuzzing is commonly used to generate fuzzy test cases to expand the test cases. However, the random and blind generation of a large number of fuzzy test cases increases the manpower and burden of testing for duplicate and invalid cases, resulting in low testing efficiency. In addition, due to the randomness of fuzzy test case generation, even a large number of test cases are unlikely to cover all the code, affecting the detection of anomalies and vulnerabilities, leading to poor product quality. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide a fuzzy test case filtering method, device, electronic device, and storage medium, which can perform targeted optimization filtering of fuzzy test cases, thereby improving testing efficiency and product quality.

[0004] To address the aforementioned technical problems, the first aspect of this application provides a method for filtering fuzzy test cases, comprising: acquiring fuzzy test cases; determining evaluation factors for the fuzzy test cases, wherein the evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzzy test cases and the priority of the fuzzy test cases; and determining the fuzzy test cases as target test cases in response to the evaluation factors meeting preset requirements.

[0005] To address the aforementioned technical problems, a second aspect of this application provides a fuzzy test case screening device, comprising: an acquisition module for acquiring fuzzy test cases; a determination module for determining evaluation factors of the fuzzy test cases, wherein the evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case; and a response module for determining the fuzzy test case as a target test case in response to the evaluation factors meeting preset requirements.

[0006] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising a memory and a processor coupled to each other, wherein the memory is used to store program data and the processor is used to execute the program data to implement the aforementioned method.

[0007] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program data, which, when executed by a processor, is used to implement the aforementioned method.

[0008] The beneficial effects of this application are as follows: Unlike existing technologies, this application obtains fuzzy test cases, then determines the evaluation factors of the fuzzy test cases. The evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case. Finally, in response to the evaluation factors meeting preset requirements, the fuzzy test cases are determined as target test cases. Specifically, the code coverage of the functional module corresponding to the fuzzy test cases can qualitatively filter the fuzzy test cases, maximizing the coverage and effectiveness of the test cases. The priority of the fuzzy test cases can quantitatively filter the fuzzy test cases, minimizing the number of test cases. This allows for targeted optimization and filtering of fuzzy test cases from both qualitative and / or quantitative perspectives, improving testing efficiency and product quality. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in this application, the accompanying drawings required in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Among them:

[0010] Figure 1 This is a flowchart illustrating an embodiment of the fuzzy test case selection method of this application;

[0011] Figure 2 This is another flowchart illustrating an embodiment of the fuzzy test case selection method of this application;

[0012] Figure 3 This application Figure 1 A flowchart illustrating an implementation method for step S12;

[0013] Figure 4 This is a screenshot of the code coverage test results.

[0014] Figure 5 This application Figure 1 A flowchart illustrating another embodiment of step S12;

[0015] Figure 6 This is a flowchart illustrating another embodiment of the fuzzy test case selection method of this application;

[0016] Figure 7 This is another flowchart illustrating another embodiment of the fuzzy test case selection method of this application;

[0017] Figure 8 This is a flowchart illustrating an embodiment of the training method for the code coverage evaluation model of this application;

[0018] Figure 9This is a flowchart illustrating an embodiment of the training method for the use case priority evaluation model of this application;

[0019] Figure 10 This is a schematic block diagram of an embodiment of the fuzzy test case filtering device of this application;

[0020] Figure 11 This is a schematic block diagram of the structure of an embodiment of the electronic device of this application;

[0021] Figure 12 This is a schematic block diagram of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0022] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] The terms "first" and "second" in this application are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0024] Some products (such as intelligent language interaction engines) have been developed for several years, accumulating thousands of test cases. However, during deployment, they still frequently encounter various scenarios not covered by current real-world test cases. Currently, to address this issue, the industry-standard fuzz testing method is generally used to generate test cases. This involves randomly mutating existing test case seeds to construct hundreds of times more test cases than before, aiming to probabilistically address some scenario coverage and anomaly detection issues by increasing the quantity of test cases.

[0025] However, the current non-directional, conventional fuzzing approach to test case generation still has many shortcomings. For example, randomly generating hundreds of thousands of test cases may have the same testing effect on a product as generating a subset (tens of thousands of test cases). It doesn't significantly improve code coverage or bug detection capabilities, but instead increases the manpower and burden of testing duplicate and invalid test cases. The inventors of this application have discovered that fuzzing is not simply about generating a large number of test cases, but rather focuses on how to select the best test cases. Fuzzing is not "fuzzy" at all; it requires a clear understanding of the testing objectives, the test objects, and the selection of test cases to generate optimal test cases more efficiently.

[0026] To this end, this application proposes a fuzzy test case screening method to screen fuzzy test cases in terms of both quality and quantity, thereby generating optimal test cases more efficiently and with higher quality, and thus improving testing efficiency and product quality.

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0028] Please see Figures 1 to 2 , Figure 1 This is a flowchart illustrating an embodiment of the fuzzy test case selection method of this application. Figure 2 This is another flowchart illustrating an embodiment of the fuzzy test case selection method of this application.

[0029] like Figure 1 As shown, the method may include the following steps:

[0030] Step S11: Obtain fuzz test cases.

[0031] Fuzz test cases are randomly generated test cases using fuzz testing techniques. By inputting fuzz test cases into the program and monitoring for anomalies, potential vulnerabilities or security flaws can be discovered. The fuzz testing process mainly includes the following basic stages: identifying the test target, identifying input, generating fuzz test data, executing the fuzz test data, monitoring for anomalies, and determining whether the discovered vulnerabilities can be exploited.

[0032] Defining clear testing objectives is crucial for deciding on the appropriate fuzzing tools or methods. First, identify the testing objectives based on the specific product being tested, then determine the test inputs. Fuzzing technology has a wide range of applications and can test a diverse range of objects, such as environment variables and parameters, web applications, file formats, network protocols, web browsers, and IoT systems. In some implementations, fuzzing test cases can be used to test intelligent language interaction engines.

[0033] Generally, fuzz test data can be generated by mutating test case seeds or by generating fuzz test cases directly from scratch. Test case seeds are existing test cases used to generate fuzz test cases. By randomly mutating test case seeds, hundreds of times more fuzz test cases can be constructed than before.

[0034] like Figure 2 As shown, in some implementations, after identifying the test target and the input, the fuzz test seed (corresponding to) can be initialized according to the test input protocol. Figure 2 Initialize the fuzz test seed module in the middle, and then perform mutation processing on the initialized fuzz test seed (corresponding to...). Figure 2 The seed mutation module is used to generate fuzzy test cases. Specifically, the initialization of the fuzzy test seed module and the seed mutation module are used to generate fuzzy test cases.

[0035] Specifically, the initialization method depends on the test input protocol. For example, based on the protocol format of the object under test, message data packets can be generated through protocol parsing and analysis, serving as test seeds. The mutation method can be selected as needed, and mutation methods may include, but are not limited to, replacement, insertion, and deletion.

[0036] Furthermore, before mutating fuzzy test seeds, they can be screened. This can be done by filtering fuzzy test seeds based on the reference code coverage of functional modules and the priority of the test seeds. Specifically, existing real test cases can be used to test the object under test, thereby obtaining the reference code coverage of each functional module. Then, real test cases corresponding to functional modules with lower reference code coverage can be selected as fuzzy test seeds. This allows for targeted mutation, and the mutated fuzzy test cases can then be used for testing, further improving the code coverage of that functional module. Alternatively, test seeds can be assigned priorities based on their importance to the test, and then higher-priority test seeds can be selected for mutation. The priority of the mutated fuzzy test cases will be closer to the priority of the original test seed. In other implementations, real test cases corresponding to functional modules with high user attention can also be used as fuzzy test seeds. For example, the user can determine the user's level of user attention for a functional module.

[0037] The data execution phase of fuzzing involves automatically sending data packets, opening files, or executing the program under test. This phase typically runs in parallel with test data generation. Monitoring anomalies involves monitoring for exceptions that occur during the testing process. Fuzzing requires setting up various forms of monitoring based on the program under test and the chosen type of fuzzing. If an error is found during fuzzing, it needs to be determined whether this error is an exploitable security vulnerability. This determination can be performed by the fuzzer or by a security testing expert.

[0038] In this embodiment, the steps of generating fuzzy test cases and obtaining fuzzy test seeds can be performed in parallel or sequentially, depending on actual needs. Specifically, fuzzy test cases can be generated simultaneously, and then the generated fuzzy test cases can be obtained and filtered; alternatively, all fuzzy test cases can be generated first, and then the generated fuzzy test cases can be obtained and filtered.

[0039] Step S12: Determine the evaluation factors for fuzz test cases. The evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzz test case and the priority of the fuzz test case.

[0040] In this embodiment, the evaluation factors of fuzzy test cases are used to filter fuzzy test cases.

[0041] The objects to be tested include at least one functional module, each used to implement a corresponding function. Code coverage may include, but is not limited to, at least one of line coverage and function coverage.

[0042] In some implementations, each evaluation factor is determined using a corresponding evaluation model. The evaluation model is a pre-trained model. Specifically, the code coverage of the functional module corresponding to the fuzzy test case can be determined using a code coverage evaluation model. The priority of the fuzzy test case can be determined using a test case priority evaluation model.

[0043] like Figure 2 As shown, after fuzzy test cases are generated, the corresponding evaluation factors can be determined by the fuzzy test case input code coverage evaluation model and / or test case priority evaluation model, respectively.

[0044] Step S13: In response to the evaluation factor meeting the preset requirements, the fuzzy test case is determined as the target test case.

[0045] like Figure 2As shown, after processing the fuzzy test cases using the code coverage evaluation model and / or the test case priority evaluation model, at least one evaluation factor can be obtained. Then, in response to the evaluation factor meeting the preset requirements, the fuzzy test case is determined as the target test case.

[0046] The preset requirements may include at least one of the following: the code coverage of the functional module corresponding to the fuzz test case meets the first preset requirement and the priority of the fuzz test case meets the second preset requirement.

[0047] In some implementations, the first preset requirement may be that the code coverage of the functional module corresponding to the fuzz test case is less than a preset code coverage threshold.

[0048] In some implementations, the second preset requirement can be that the priority of the fuzzy test case is a preset priority. The higher the importance or attention of the fuzzy test case, the higher its corresponding priority, and each priority has a corresponding expansion quantity threshold. In other implementations, the second preset requirement can also be that the expansion quantity corresponding to the priority of the fuzzy test case is less than the expansion quantity threshold. In still other implementations, the second preset requirement can also be that the priority of the fuzzy test case is a preset priority, and the expansion quantity corresponding to the priority of the fuzzy test case is less than the expansion quantity threshold. The priority levels can be selected according to actual needs; for example, priorities can be divided into high, medium, and low levels, with corresponding priority labels P1, P2, and P3, respectively. The preset priority can be set and modified according to actual conditions; for example, the preset priority is high (P1). When the priority of a fuzzy test case is P1, it indicates that the fuzzy test case has a high degree of importance or attention, thus retaining the fuzzy test case. Each priority has a corresponding expansion quantity threshold, which can be set and modified according to actual conditions. Typically, the number of test cases with the three priorities P1, P2, and P3 follows a Gaussian distribution, with a ratio of approximately 2:5:3. In one example, the expansion thresholds corresponding to priorities P1, P2, and P3 could be 2000:5000:3000, respectively. Of course, the ratio of different priorities is not limited to 2:5:3; other possible ratios are not specified here. If the expansion quantity corresponding to a priority has not yet reached the expansion threshold, it means that more fuzz test cases can be added for that priority.

[0049] The above scheme obtains fuzzy test cases, then determines the evaluation factors for these cases. These evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case. Finally, in response to the evaluation factors meeting preset requirements, the fuzzy test case is identified as the target test case. Specifically, the code coverage of the functional module corresponding to the fuzzy test case allows for qualitative screening of the fuzzy test cases, maximizing the coverage and effectiveness of the test cases. The priority of the fuzzy test cases allows for quantitative screening, minimizing the number of test cases. This enables targeted optimization and screening of fuzzy test cases from both qualitative and / or quantitative perspectives, improving testing efficiency and product quality.

[0050] Please see Figures 3 to 4 , Figure 3 This application Figure 1 A flowchart illustrating an implementation method for step S12. Figure 4 This is a screenshot of the code coverage test results.

[0051] In this embodiment, the evaluation factor includes the code coverage of the functional module corresponding to the fuzz test case. Correspondingly, step S12 may include steps S121 to S122.

[0052] Step S121: Determine the functional module corresponding to the fuzz test case.

[0053] For example, a trained code coverage evaluation model can be used to determine the functional modules corresponding to fuzzy test cases. Specifically, fuzzy test cases can be used as input to the code coverage evaluation model, and the output can be the functional modules corresponding to the fuzzy test cases.

[0054] Step S122: Obtain the code coverage of the functional modules corresponding to the fuzz test cases.

[0055] In some implementations, before step S122, code coverage testing can be performed on each functional module based on the first set of real test cases to obtain reference code coverage for each functional module. Then, based on the reference code coverage of the functional module corresponding to the fuzzy test case, the code coverage of the functional module corresponding to the fuzzy test case is obtained. Specifically, the functional modules of the object to be tested can be identified first, and then a round of code coverage testing can be performed on each functional module based on the first set of real test cases. This can specifically be a baseline test of C++ code coverage to obtain the code coverage of each functional module.

[0056] like Figure 4The image shows the baseline test results for C++ code coverage of a product. The code coverage test results include the types of functional modules (Directory), including multiple types such as A, B, and C, as well as the corresponding line coverage and function coverage. Line coverage is presented in three formats from left to right: icons, percentages, and fractions. Function coverage is presented in two formats from left to right: percentages and fractions. Different presentation formats allow viewers to obtain more information, and the specific presentation format can be set or modified according to actual needs; no limitations are specified here.

[0057] The first set of real test cases may include at least one real test case. Real test cases are pre-prepared or existing test cases used for code coverage testing. Since the number of test cases in the first set of real test cases is insufficient, fuzzing is used to generate fuzzing test cases to increase the number of test cases available. To generate fuzzing test cases in a targeted manner and improve their quality, the existing set of real test cases is used to obtain the reference code coverage for each functional module under the current test before filtering. Based on the reference code coverage, when filtering fuzzing test cases, more fuzzing test cases for functional modules with lower code coverage are retained, while fewer are retained for functional modules with higher code coverage. Furthermore, for functional modules with full code coverage, such as those with code coverage greater than or equal to 100%, their corresponding fuzzing test cases can be omitted. This improves the quality of generated fuzzing test cases and reduces the number of test cases.

[0058] In some implementations, after step S13 above—determining the fuzzy test case as the target test case in response to the evaluation factor meeting the preset requirements—it may further include updating the reference code coverage of the functional module corresponding to the fuzzy test case. It is understood that with each additional target test case, the reference code coverage of the corresponding functional module should also change, such as increasing. Therefore, to ensure the quality of fuzzy test case generation, the accuracy of the reference code coverage needs to be guaranteed, thus requiring updates to the reference code coverage.

[0059] Specifically, the number of target test cases for the functional module corresponding to the fuzzy test case can be obtained. Then, the ratio of the number of test cases corresponding to the current functional module to the number of target test cases is used as the reference code coverage of that functional module, thereby updating the reference code coverage. The number of test cases corresponding to the current functional module includes the number of real test cases and the number of fuzzy test cases. In one example, fuzzy test case a1 corresponds to functional module A, the target number of test cases for functional module A is 100, and the number of real test cases corresponding to functional module A is 20. Therefore, the current reference code coverage is 20 / 100 = 20%. After adding one fuzzy test case, the corresponding reference code coverage is (20+1) / 100 = 21%. After adding another fuzzy test case, the corresponding reference code coverage is (21+2) / 100 = 22%, and so on, to update the reference code coverage.

[0060] Please see Figures 5 to 6 , Figure 5 This application Figure 1 A flowchart illustrating another embodiment of step S12 is shown. Figure 6 This is a flowchart illustrating another embodiment of the fuzzy test case selection method of this application.

[0061] In this embodiment, when the evaluation factor includes the priority of fuzz test cases, the above step S13 includes steps S231 to S233.

[0062] Step S231: Add fuzz test cases to the test case preparation library.

[0063] like Figure 6 As shown, the test case preparation library is used to store fuzzy test cases. Ni represents the classification score corresponding to the i-th functional module output by the code coverage evaluation model, where i is a positive integer. Sj represents the classification score corresponding to the j-th priority output by the test case priority evaluation model, where j is a positive integer. The category with the highest classification score in Ni and Sj can be determined as the functional module and priority corresponding to the fuzzy test case, respectively.

[0064] In some implementations, a test case priority evaluation model is used to determine the priority of fuzz test cases and the corresponding expansion quantity threshold. The expansion quantity threshold can be set or modified according to actual conditions. The expansion quantity threshold can be an empirical value.

[0065] Step S232: Obtain the expansion quantity threshold corresponding to the priority of the fuzz test cases.

[0066] Step S233: In response to the fact that the number of expansions corresponding to the priority of the fuzzy test case is greater than the expansion number threshold, some or all of the fuzzy test cases with the same priority as the fuzzy test cases in the test case preparation library are determined as target test cases.

[0067] In some implementations, after identifying some or all of the fuzzy test cases with the same priority as the fuzzy test cases in the test case preparation library as target test cases, the expansion of fuzzy test cases of that priority can be stopped, and the target test cases can be formally included in the test case library for subsequent testing.

[0068] like Figure 6 As shown, in some implementations, the expansion quantity thresholds for priorities P1, P2, and P3 are TOPN1 = 200, TOPN2 = 500, and TOPN3 = 300, respectively. Taking priority P1 as an example, if the expansion quantity of priority P1 in the test case preparation library, i.e., the number of fuzzy test cases corresponding to priority P1, is greater than the expansion quantity threshold of 200, then some or all of the fuzzy test cases with the same priority as the fuzzy test cases in the test case preparation library can be determined as target test cases. For example, when the expansion quantity of priority P1 is greater than 200, only 200 fuzzy test cases can be determined as target test cases and included in the test case library. Of course, all fuzzy test cases can also be determined as target test cases and included in the test case library.

[0069] In other implementations, instead of adding fuzzy test cases to the test case preparation library, filtering can be performed directly. This involves directly comparing the expansion quantity corresponding to the priority of a fuzzy test case with an expansion quantity threshold. If the expansion quantity corresponding to the priority of a fuzzy test case is greater than the expansion quantity threshold, the fuzzy test case is identified as the target test case, and the expansion quantity corresponding to the priority of that fuzzy test case is updated. Updating the expansion quantity, for example, involves incrementing the expansion quantity corresponding to the priority of the fuzzy test case by one.

[0070] Please see Figure 7 , Figure 7 This is another flowchart illustrating another embodiment of the fuzzy test case selection method of this application. In this embodiment, the evaluation factors include the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case.

[0071] The method may include the following steps:

[0072] Step S21: Initialize the fuzz test seed according to the test input protocol.

[0073] Step S22: Mutate the initialized fuzz test seed to obtain fuzz test cases.

[0074] For a description of steps S21 to S22, please refer to step S11 above; it will not be repeated here.

[0075] Step S23: Determine the code coverage of the functional modules corresponding to the fuzz test cases.

[0076] Step S24: In response to the fact that the code coverage of the functional module corresponding to the fuzz test case meets the first preset requirement, determine the priority of the fuzz test case.

[0077] Step S25: In response to the priority of the fuzz test case meeting the second preset requirement, the fuzz test case is determined as the target test case.

[0078] For a description of steps S23 to S25, please refer to steps S12 to S13 above, which will not be repeated here. However, unlike the previous embodiments, in this embodiment, the evaluation factors include both the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case, and these two evaluation factors are used sequentially to filter the fuzzy test cases. Specifically, after obtaining the fuzzy test cases in steps S21 to S22, the code coverage evaluation model is first used to determine the code coverage of the functional module corresponding to the fuzzy test case. Then, the fuzzy test cases are first filtered based on the code coverage. After the first filtering is passed, the test case priority evaluation model is used to determine the priority of the fuzzy test cases. Then, the fuzzy test cases are second filtered based on the priority. After the second filtering is passed, the fuzzy test cases are determined as target test cases. In this embodiment, code coverage is used as a filtering condition before priority.

[0079] In other embodiments, priority can also be used as a filtering condition before code coverage. Correspondingly, the priority of fuzzy test cases can be determined first. In response to the priority of fuzzy test cases meeting a second preset requirement, the code coverage of the functional module corresponding to the fuzzy test cases can be determined. Then, in response to the code coverage of the functional module corresponding to the fuzzy test cases meeting a first preset requirement, the fuzzy test cases can be determined as target test cases.

[0080] Step S26: Use the feature values ​​learned by the evaluation model corresponding to each evaluation factor to adjust the parameters in the initialization process of the fuzzy test seed.

[0081] like Figure 2 or Figure 6As shown, in this embodiment, the feature values ​​(w1 and w2) learned by the code coverage evaluation model and / or test case priority evaluation model can also be applied to the seed initialization part of fuzzing through data feedback. This allows for more accurate and targeted generation of initial seed data for different business scenarios and characteristics, thereby improving the efficiency and quality of test case selection and generation in the code coverage evaluation model and test case priority evaluation model. Specifically, the feature values ​​in the code coverage evaluation model and / or test case priority evaluation model can be learned in real time and applied to the seed initialization part of fuzzing through data feedback. In particular, the parameters in the initialization fuzzing seed module can be adjusted using the feature values ​​to generate accurate and targeted test seeds.

[0082] The training process of the code coverage evaluation model and the use case priority evaluation model will be introduced below.

[0083] Please see Figure 8 , Figure 8 This is a flowchart illustrating an embodiment of the training method for the code coverage evaluation model of this application.

[0084] The code coverage evaluation model was trained using the following method:

[0085] Step S31: Obtain the second set of real test cases.

[0086] The second set of real test cases may include at least one real test case.

[0087] Step S32: Determine the functional module label corresponding to each real test case in the second set of real test cases.

[0088] Specifically, we can first identify the functional modules of the object to be tested, and then perform code coverage testing (e.g., baseline testing of C++ code coverage) on each functional module based on the second set of real test cases. This will give us the correspondence between each real test case in the second set of real test cases and the functional modules. Based on this correspondence, we can add the corresponding functional module tag to each real test case in the second set of real test cases.

[0089] Step S33: Based on the second set of real test cases and the corresponding functional module labels, train the pre-trained first classification model to obtain the code coverage evaluation model.

[0090] Optionally, the first classification model can be a BERT model. Here, the pre-trained first classification model is used directly for training. In subsequent training processes, only the previous parameters need to be fine-tuned, which can save training time and achieve good results.

[0091] Specifically, the second real test case can be used as input to the first classification model, and the corresponding functional module label can be used as output to train the pre-trained first classification model, thus obtaining the code coverage evaluation model. The trained code coverage evaluation model has the ability to automatically identify the functional module corresponding to the input test case.

[0092] Please see Figure 9 , Figure 9 This is a flowchart illustrating an embodiment of the training method for the use case priority evaluation model of this application.

[0093] The use case prioritization evaluation model was trained using the following method:

[0094] Step S41: Obtain the third set of real test cases.

[0095] The third set of real test cases may include at least one real test case. The first, second, and third sets of real test cases may be the same, partially the same (i.e., some real test cases may be the same), or completely different.

[0096] Step S42: Determine the priority label corresponding to each real test case in the third set of real test cases.

[0097] Specifically, each real test case in the third set of real test cases can be assigned a corresponding priority label. The determination of the priority labels can be jointly reviewed by R&D and testing departments to increase the feasibility of the training data.

[0098] Step S43: Based on the third real test case set and the corresponding priority labels, load the pre-trained second classification model for training to obtain the test case priority evaluation model.

[0099] Optionally, the second classification model can be the BERT model.

[0100] Specifically, the third real test case can be used as the input to the second classification model, and the corresponding priority label can be used as the output of the second classification model to train the pre-trained second classification model and obtain the test case priority evaluation model.

[0101] In some implementations, the loss function used during the training of the first classification model and / or the second classification model includes at least one of the cross-entropy loss function, the log-likelihood loss function, and the hinge loss function. By training the model and minimizing the loss function through backpropagation, an optimal classification model can be obtained, namely the code coverage evaluation model and the use case priority evaluation model.

[0102] The formula for the cross-entropy loss function is as follows:

[0103]

[0104] Where L is the loss, M is the total number of data points (e.g., the number of data points in the second or third real test case set), j is the number of data points to iterate through starting from 1, and z is the number of data points to traverse. j For the actual data labeling distribution, This is the distribution predicted by the model.

[0105] The formula for the log-likelihood loss function is as follows:

[0106]

[0107] in, Let y be the loss, and y be the true data label distribution. This is the distribution predicted by the model.

[0108] The formula for the hinge loss function is as follows:

[0109]

[0110] Where Li is the loss, s yi For the true data labeling distribution, s j The distribution predicted by the model is represented by Δ, which is an adjustment parameter. The value of Δ can be set according to the actual situation, such as different classification scenarios. For example, Δ can be set to 1.

[0111] The above solution optimizes and filters fuzzy test cases from both quantitative and qualitative perspectives. Specifically, regarding test case quality, it combines code coverage testing methods to summarize the characteristics of existing real test cases' code coverage test results across various functional modules of the test object, learning a code coverage evaluation model. This model can then assess the coverage of input fuzzy test cases at the code level for functional modules. Secondly, regarding test case quantity, it uses existing test cases and priority level labels (P1, P2, P3, ..., Pj) defined for each case. A classification model is trained based on these priority label features to determine the priority category of input fuzzy test cases. This allows for more targeted expansion of test cases of different priorities based on the priority evaluation results. Finally, based on the model evaluation results for both qualitative and quantitative modules, the fusion strategy is dynamically adjusted for different business scenarios to generate optimal test cases more efficiently and with higher quality, thereby improving the generalization and reliability of the tested engine.

[0112] Please see Figure 10 , Figure 10This is a schematic block diagram of an embodiment of the fuzzy test case filtering device of this application.

[0113] In this embodiment, the fuzzy test case filtering device 100 includes an acquisition module 110, a determination module 120, and a response module 130. The acquisition module 110 is used to acquire fuzzy test cases; the determination module 120 is used to determine the evaluation factors of the fuzzy test cases, and the evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzzy test case and the priority of the fuzzy test case; the response module 130 is used to determine the fuzzy test case as the target test case in response to the evaluation factors meeting the preset requirements.

[0114] In some implementations, when the evaluation factors include the code coverage of the functional module corresponding to the fuzzy test case, the code coverage of the functional module corresponding to the fuzzy test case is determined using a code coverage evaluation model; and / or, the determining module 120 is further used to determine the functional module corresponding to the fuzzy test case; and obtain the code coverage of the functional module corresponding to the fuzzy test case.

[0115] In some embodiments, the fuzzy test case filtering device further includes a testing module (not shown) for testing the code coverage of each functional module based on a first real test case set before obtaining the code coverage of the functional module corresponding to the fuzzy test case, and obtaining a reference code coverage for each functional module; the acquisition module 110 is used to obtain the code coverage of the functional module corresponding to the fuzzy test case based on the reference code coverage of the functional module corresponding to the fuzzy test case.

[0116] In some implementations, the fuzzy test case screening device further includes an update module (not shown), which is used to update the reference code coverage of the functional module corresponding to the fuzzy test case after determining the fuzzy test case as the target test case in response to the evaluation factor meeting the preset requirements.

[0117] In some implementations, the response module 130 is further configured to add fuzzy test cases to the test case preparation library; obtain an expansion quantity threshold corresponding to the priority of the fuzzy test cases; and, in response to the expansion quantity corresponding to the priority of the fuzzy test cases being greater than the expansion quantity threshold, determine some or all of the fuzzy test cases in the test case preparation library that have the same priority as the fuzzy test cases as the target test cases.

[0118] In some implementations, when the evaluation factor includes the code coverage of the functional module corresponding to the fuzzy test case, the preset requirement includes that the code coverage of the functional module corresponding to the fuzzy test case meets a first preset requirement, wherein the first preset requirement is that the code coverage of the functional module corresponding to the fuzzy test case is less than a preset code coverage threshold; and / or, the determining module 120 is further configured to determine the code coverage of the functional module corresponding to the fuzzy test case; and in response to the code coverage of the functional module corresponding to the fuzzy test case meeting the first preset requirement, determine the priority of the fuzzy test case; the response module 130 is further configured to determine the fuzzy test case as the target test case in response to the priority of the fuzzy test case meeting a second preset requirement.

[0119] In some implementations, the acquisition module 110 is further configured to initialize a fuzz test seed according to a test input protocol; and to mutate the initialized fuzz test seed to obtain fuzz test cases.

[0120] In some implementations, each evaluation factor is determined using a corresponding evaluation model. The fuzzy test case screening device also includes an adjustment module (not shown) for adjusting the parameters in the initialization process of the fuzzy test seed using the feature values ​​learned from the evaluation model corresponding to each evaluation factor.

[0121] In some implementations, the code coverage of the functional modules corresponding to the fuzzy test cases is determined using a code coverage evaluation model. The method further includes: obtaining a second set of real test cases; determining the functional module label corresponding to each real test case in the second set of real test cases; and training a pre-trained first classification model based on the second set of real test cases and the corresponding functional module labels to obtain a code coverage evaluation model.

[0122] In some implementations, the priority of fuzzy test cases is determined using a test case priority evaluation model. The method further includes: obtaining a third set of real test cases; determining the priority label corresponding to each real test case in the third set of real test cases; and training a pre-trained second classification model based on the third set of real test cases and the corresponding priority labels to obtain the test case priority evaluation model.

[0123] In some implementations, the loss function used during the training of the first classification model and / or the second classification model includes at least one of the cross-entropy loss function, the log-likelihood loss function, and the hinge loss function.

[0124] Please see Figure 11 , Figure 11 This is a schematic block diagram of the structure of an embodiment of the electronic device of this application.

[0125] The electronic device 200 includes a memory 210 and a processor 220 coupled to each other. The memory 210 is used to store program data, and the processor 220 is used to execute the program data to implement the steps in any of the above method embodiments.

[0126] Electronic device 200 may include, but is not limited to: personal computers (e.g., desktop computers, laptop computers, tablet computers, handheld computers, etc.), mobile phones, servers, wearable devices, as well as augmented reality (AR) and virtual reality (VR) devices, televisions, etc., without limitation.

[0127] Specifically, processor 220 controls itself and memory 210 to implement the steps in any of the above method embodiments. Processor 220 may also be referred to as a Central Processing Unit (CPU). Processor 220 may be an integrated circuit chip with signal processing capabilities. Processor 220 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 220 may be implemented by multiple integrated circuit chips.

[0128] Please see Figure 12 , Figure 12 This is a schematic block diagram of an embodiment of the computer-readable storage medium of this application.

[0129] The computer-readable storage medium 300 stores program data 310, which, when executed by a processor, is used to implement the steps in any of the above method embodiments.

[0130] The computer-readable storage medium 300 can be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or a medium capable of storing computer programs. It can also be a server storing the computer program, which can send the stored computer program to other devices for execution, or it can run the stored computer program itself.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0132] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0133] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0134] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A fuzzy test case selection method, characterized in that, include: Obtain fuzz test cases; Determine the evaluation factors for the fuzz test cases, wherein the evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzz test cases and the priority of the fuzz test cases; In response to the evaluation factor meeting the preset requirements, the fuzzy test case is determined as the target test case; When the evaluation factor includes the code coverage of the functional module corresponding to the fuzzy test case, the code coverage of the functional module corresponding to the fuzzy test case is determined using a code coverage evaluation model, including: using the trained code coverage evaluation model to determine the functional module corresponding to the fuzzy test case and obtain the code coverage of the functional module corresponding to the fuzzy test case; the code coverage evaluation model is trained by the following method: obtaining a second real test case set; determining the functional module label corresponding to each real test case in the second real test case set; and training a pre-trained first classification model based on the second real test case set and the corresponding functional module labels to obtain the code coverage evaluation model; The step of determining the fuzzy test case as the target test case in response to the evaluation factor meeting the preset requirements includes: adding the fuzzy test case to the test case preparation library; obtaining the expansion quantity threshold corresponding to the priority of the fuzzy test case; and determining some or all of the fuzzy test cases in the test case preparation library with the same priority as the fuzzy test case as the target test case in response to the expansion quantity corresponding to the priority of the fuzzy test case being greater than the expansion quantity threshold. The higher the importance or attention of the fuzzy test case, the higher its corresponding priority, and each priority has a corresponding expansion quantity threshold.

2. The method according to claim 1, characterized in that, Before obtaining the code coverage of the functional module corresponding to the fuzz test case, the method further includes: Based on the first set of real test cases, code coverage tests are performed on each functional module to obtain the reference code coverage for each functional module; The step of obtaining the code coverage of the functional module corresponding to the fuzz test case includes: Based on the reference code coverage of the functional module corresponding to the fuzzy test case, the code coverage of the functional module corresponding to the fuzzy test case is obtained.

3. The method according to claim 1, characterized in that, When the evaluation factor includes the code coverage of the functional module corresponding to the fuzzy test case, the preset requirement includes that the code coverage of the functional module corresponding to the fuzzy test case meets a first preset requirement, wherein the first preset requirement is that the code coverage of the functional module corresponding to the fuzzy test case is less than a preset code coverage threshold. And / or, determining the evaluation factor of the fuzz test case, and determining the fuzz test case as the target test case in response to the evaluation factor meeting a preset requirement, includes: Determine the code coverage of the functional modules corresponding to the fuzz test cases; In response to the fact that the code coverage of the functional module corresponding to the fuzz test case meets the first preset requirement, the priority of the fuzz test case is determined; In response to the priority of the fuzz test case meeting the second preset requirement, the fuzz test case is determined as the target test case.

4. The method according to claim 1, characterized in that, The process of obtaining fuzz test cases includes: Initialize the fuzz test seed according to the test input protocol; The initialized fuzz test seed is mutated to obtain the fuzz test cases; And / or, each of the evaluation factors is determined using a corresponding evaluation model, and the method further includes: Using the feature values ​​learned by the evaluation model corresponding to each evaluation factor, the parameters in the initial fuzzy test seed module are adjusted. The initial fuzzy test seed module is used to generate test seeds.

5. The method according to claim 1, characterized in that, The priority of the fuzz test cases is determined using a test case priority evaluation model, which is trained using the following method: Obtain the third set of real test cases; Determine the priority label corresponding to each real test case in the third set of real test cases; Based on the third set of real test cases and the corresponding priority labels, a pre-trained second classification model is loaded and trained to obtain the test case priority evaluation model.

6. A fuzzy test case filtering device, characterized in that, include: Get the module and get the fuzz test cases; A determination module is used to determine the evaluation factors of the fuzz test cases, wherein the evaluation factors include at least one of the code coverage of the functional module corresponding to the fuzz test cases and the priority of the fuzz test cases; A response module is used to determine the fuzz test case as the target test case in response to the evaluation factor meeting the preset requirements. When the evaluation factor includes the code coverage of the functional module corresponding to the fuzzy test case, the code coverage of the functional module corresponding to the fuzzy test case is determined using a code coverage evaluation model, including: using the trained code coverage evaluation model to determine the functional module corresponding to the fuzzy test case and obtain the code coverage of the functional module corresponding to the fuzzy test case; the code coverage evaluation model is trained by the following method: obtaining a second real test case set; determining the functional module label corresponding to each real test case in the second real test case set; and training a pre-trained first classification model based on the second real test case set and the corresponding functional module labels to obtain the code coverage evaluation model; The response module is used to add the fuzzy test cases to the test case preparation library; obtain the expansion quantity threshold corresponding to the priority of the fuzzy test cases; and, in response to the expansion quantity corresponding to the priority of the fuzzy test cases being greater than the expansion quantity threshold, determine some or all of the fuzzy test cases in the test case preparation library that have the same priority as the fuzzy test cases as the target test cases; wherein, the higher the importance or attention of the fuzzy test cases, the higher the corresponding priority, and each priority has a corresponding expansion quantity threshold.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor coupled to each other, the memory being used to store program data and the processor being used to execute the program data to implement the method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program data, which, when executed by a processor, is used to implement the method as described in any one of claims 1-5.

Images