Test strategy vulnerability mining method and device

Abstract

This invention provides a method and apparatus for discovering test strategy vulnerabilities. It utilizes the textual information of the defect to be analyzed and a test strategy knowledge graph to generate a preliminary list of potential test strategy vulnerabilities. Then, based on the description of the defect, its related code, and software project documentation, a deeper list of test strategy vulnerabilities is generated. Furthermore, by simulating the divergent thinking process of a test expert through a heuristic rule base, a list of potentially overlooked test scenarios related to or similar to the current defect, which may have been missed in the target test case set, is generated, providing forward-looking supplementary test suggestions. Based on the preliminary list of potential test strategy vulnerabilities, the deeper list of test strategy vulnerabilities, and the list of potentially overlooked test scenarios, a final list of test strategy vulnerabilities is generated. This achieves the goal of effectively conducting in-depth analysis of defects and uncovering hidden test strategy vulnerabilities behind them.

Description

Technical Field

[0001] This invention relates to the field of software testing technology, and in particular to a method and apparatus for discovering vulnerabilities in testing strategies. Background Technology

[0002] Currently, testers tend to limit themselves to verifying normal processes, resulting in insufficient coverage of unexpected paths such as abnormal scenarios and boundary conditions, creating blind spots in their thinking. Furthermore, the lack of explicit correlation between defects and test strategies makes it difficult to accurately pinpoint specific test strategy vulnerabilities from the test design phase once a defect not covered by test cases appears. Summary of the Invention

[0003] In view of this, the present invention provides a method and apparatus for mining test strategy vulnerabilities, which can effectively perform in-depth analysis of defects and uncover the test strategy vulnerabilities hidden behind the defects.

[0004] The first aspect of this invention provides a method for vulnerability discovery in testing strategies, comprising:

[0005] Receive defect information of a defect to be analyzed; wherein the defect information includes text information of the defect to be analyzed, related code of the defect to be analyzed, and software project documentation; the text information of the defect to be analyzed includes at least a description of the defect to be analyzed;

[0006] Based on the textual information of the defects to be analyzed and the test strategy knowledge graph, a preliminary list of potential test strategy vulnerabilities is generated.

[0007] Based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation, a deep testing strategy vulnerability list is generated.

[0008] Based on a heuristic rule base and a target test case set, a list of potential missed test scenarios is determined; wherein, the target test case set is a set of executed test cases related to the defect to be analyzed;

[0009] Based on the preliminary list of potential test strategy vulnerabilities, the deep test strategy vulnerability list, and the list of potential omitted test scenarios, a final list of test strategy vulnerabilities is generated.

[0010] Optionally, the method for constructing the test strategy knowledge graph includes:

[0011] Acquire multi-source data; wherein the multi-source data covers the entire lifecycle of software development and testing;

[0012] Entity nodes and semantic relationships between entity nodes are extracted from the multi-source data.

[0013] A test strategy knowledge graph is constructed based on the entity nodes and the semantic relationships between them.

[0014] Optionally, the textual information of the defect to be analyzed further includes: stack trace information of the defect to be analyzed; the generation of a preliminary list of potential test strategy vulnerabilities based on the textual information of the defect to be analyzed and the test strategy knowledge graph includes:

[0015] For each defect pattern node in the test strategy knowledge graph, a first matching degree is determined based on the text information of the defect to be analyzed and the keyword set of the defect pattern node.

[0016] Based on the description information of the defect to be analyzed and the description text of the defect pattern node, a second matching degree is determined;

[0017] Based on the description information of the defect pattern node and the stack trace information of the defect to be analyzed, a third matching degree is determined;

[0018] Based on the first matching degree, the second matching degree, and the third matching degree, the comprehensive correlation score between the defect to be analyzed and the defect pattern node is determined;

[0019] Test strategy nodes connected to defect pattern nodes whose overall correlation score is greater than the correlation score threshold are considered as preliminary potential test strategy vulnerabilities.

[0020] Optionally, the step of generating a deep testing strategy vulnerability list based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation includes:

[0021] Based on the guiding questions, the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation, an input sequence is generated;

[0022] The input sequence is fed into the causal reasoning model, and the answer to the guiding question is output.

[0023] Based on the answers to all the guiding questions, identify cognitive blind spots;

[0024] By mapping the aforementioned cognitive blind spots to a test strategy knowledge graph, deep test strategy vulnerabilities are obtained.

[0025] Optionally, the step of determining a list of potential missed test scenarios based on a heuristic rule base and a set of target test cases includes:

[0026] Identify the target scenarios within the target test case set;

[0027] For each target scenario, the target scenario is matched with the heuristic rules in the heuristic rule base to obtain the target heuristic rule;

[0028] Based on the aforementioned target heuristic rules, scene seeds are generated;

[0029] The scene seed and prompt words are input into the test scene description model, and the potential missing test scenes are output.

[0030] Optionally, the step of generating a final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list includes:

[0031] Based on the preliminary list of potential testing strategy vulnerabilities and the deep list of testing strategy vulnerabilities, the target testing strategy vulnerability is determined;

[0032] For each target test strategy vulnerability, the overall confidence level of the target test strategy vulnerability is determined based on the comprehensive correlation score of the initial potential test strategy vulnerability corresponding to the target test strategy vulnerability, the confidence level of the deep test strategy vulnerability corresponding to the target test strategy vulnerability, and the correlation strength between the target test strategy vulnerability and the potential omitted test scenario list.

[0033] Based on the overall confidence level of all the target test strategy vulnerabilities, a final list of test strategy vulnerabilities is generated.

[0034] Optionally, after generating the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list, the process further includes:

[0035] Based on the potential omission test scenario list and the final test strategy vulnerability list, candidate relationship triples are generated;

[0036] Based on the aforementioned cognitive blind spots, candidate entity nodes are generated;

[0037] The test strategy knowledge graph is updated based on the candidate relation triples and the candidate entity nodes to obtain the updated test strategy knowledge graph.

[0038] Optionally, after generating the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list, the process further includes:

[0039] The target testers are determined based on the testers' areas of responsibility and the areas where the defects to be analyzed occur.

[0040] The target tester's capability profile is updated using the final test strategy vulnerability list to obtain the updated capability profile.

[0041] Optionally, after updating the target tester's capability profile using the final test strategy vulnerability list to obtain the updated capability profile, the process further includes:

[0042] Based on the updated capability profile of the target tester, the test strategy knowledge graph, the guiding questions, the answers to the guiding questions, the list of potential missed test scenarios, and the final list of test strategy vulnerabilities, a personalized report for the target tester is generated.

[0043] A second aspect of the present invention provides a vulnerability discovery apparatus for testing strategies, comprising:

[0044] A receiving unit is configured to receive defect information of a defect to be analyzed; wherein the defect information includes text information of the defect to be analyzed, related code of the defect to be analyzed, and software project documentation; the text information of the defect to be analyzed includes at least a description of the defect to be analyzed.

[0045] The first list generation unit is used to generate a preliminary list of potential test strategy vulnerabilities based on the text information of the defects to be analyzed and the test strategy knowledge graph.

[0046] The second list generation unit is used to generate a deep testing strategy vulnerability list based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project document.

[0047] The third list generation unit is used to determine a list of potential missing test scenarios based on a heuristic rule base and a target test case set; wherein, the target test case set is a set of executed test cases related to the defect to be analyzed;

[0048] The fourth list generation unit is used to generate the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list.

[0049] Optionally, the construction unit of the test strategy knowledge graph includes:

[0050] An acquisition unit is used to acquire multi-source data; wherein the multi-source data covers the entire lifecycle of software development and testing;

[0051] An extraction unit is used to extract entity nodes and semantic relationships between entity nodes from the multi-source data.

[0052] The construction unit is used to construct a test strategy knowledge graph based on the entity nodes and the semantic relationships between them.

[0053] Optionally, the text information of the defect to be analyzed further includes: stack trace information of the defect to be analyzed; the first list generation unit includes:

[0054] The first matching degree determination unit is used to determine the first matching degree for each defect pattern node in the test strategy knowledge graph based on the text information of the defect to be analyzed and the keyword set of the defect pattern node.

[0055] The second matching degree determination unit is used to determine the second matching degree based on the description information of the defect to be analyzed and the description text of the defect pattern node;

[0056] The third matching degree determination unit is used to determine the third matching degree based on the description information of the defect pattern node and the stack trace information of the defect to be analyzed.

[0057] The comprehensive correlation score determination unit is used to determine the comprehensive correlation score between the defect to be analyzed and the defect pattern node based on the first matching degree, the second matching degree and the third matching degree;

[0058] The preliminary potential test strategy vulnerability determination unit is used to identify test strategy nodes connected to defect pattern nodes whose comprehensive correlation score is greater than the correlation score threshold as preliminary potential test strategy vulnerabilities.

[0059] Optionally, the second list generation unit includes:

[0060] An input sequence generation unit is used to generate an input sequence based on guiding questions, description information of the defect to be analyzed, relevant code of the defect to be analyzed, and software project documentation;

[0061] The answer generation unit is used to input the input sequence into the causal reasoning model and output the answer corresponding to the guiding question;

[0062] A cognitive blind spot generation unit is used to determine cognitive blind spots based on the answers to all the guiding questions.

[0063] The deep testing strategy vulnerability determination unit is used to map the cognitive blind spots to the testing strategy knowledge graph to obtain deep testing strategy vulnerabilities.

[0064] Optionally, the third list generation unit includes:

[0065] The identification unit is used to identify the target scenario in the target test case set.

[0066] The target heuristic rule determination unit is used to match each target scenario with the heuristic rules in the heuristic rule base to obtain the target heuristic rule;

[0067] A scene seed generation unit is used to generate scene seeds based on the target heuristic rules;

[0068] The potential omission test scenario determination unit is used to input the scenario seed and prompt words into the test scenario description model and output the potential omission test scenario.

[0069] Optionally, the fourth list generation unit includes:

[0070] The target testing strategy vulnerability determination unit is used to determine the target testing strategy vulnerability based on the preliminary potential testing strategy vulnerability list and the deep testing strategy vulnerability list.

[0071] The comprehensive confidence determination unit is used to determine the comprehensive confidence of each target test strategy vulnerability based on the comprehensive correlation score of the preliminary potential test strategy vulnerability corresponding to the target test strategy vulnerability, the confidence of the deep test strategy vulnerability corresponding to the target test strategy vulnerability, and the correlation strength between the target test strategy vulnerability and the potential omitted test scenario list.

[0072] The fourth list generation subunit is used to generate the final list of test strategy vulnerabilities based on the comprehensive confidence level of all the target test strategy vulnerabilities.

[0073] Optionally, the test strategy vulnerability discovery device further includes:

[0074] The candidate relation triple generation unit is used to generate candidate relation triples based on the potential omission test scenario list and the final test strategy vulnerability list.

[0075] A candidate entity node generation unit is used to generate candidate entity nodes based on the cognitive blind spot;

[0076] The knowledge graph is updated to which element, which is used to update the test strategy knowledge graph based on the candidate relation triples and the candidate entity nodes, to obtain the updated test strategy knowledge graph.

[0077] Optionally, the test strategy vulnerability discovery device further includes:

[0078] The target tester determination unit is used to determine the target tester based on the tester's responsibility domain and the area where the defect to be analyzed occurs;

[0079] The capability profile update unit is used to update the capability profile of the target tester using the final test strategy vulnerability list, so as to obtain the updated capability profile of the target tester.

[0080] Optionally, the test strategy vulnerability discovery device further includes:

[0081] The personalized report generation unit is used to generate a personalized report for the target tester based on the updated capability profile of the target tester, the test strategy knowledge graph, the guiding questions, the answers to the guiding questions, the list of potential missed test scenarios, and the final list of test strategy vulnerabilities.

[0082] A third aspect of the present invention provides an electronic device, comprising:

[0083] One or more processors;

[0084] A storage device on which one or more programs are stored;

[0085] When the one or more programs are executed by the one or more processors, the one or more processors implement the test strategy vulnerability discovery method as described in any one of the first aspects.

[0086] A fourth aspect of the present invention provides a computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the test strategy vulnerability mining method as described in any one of the first aspects.

[0087] As shown above, this invention provides a method and apparatus for discovering test strategy vulnerabilities. It utilizes the textual information of the defect to be analyzed and a test strategy knowledge graph to generate a preliminary list of potential test strategy vulnerabilities. Then, based on the description information of the defect to be analyzed, its related code, and software project documentation, a deeper list of test strategy vulnerabilities is generated. Furthermore, by simulating the divergent thinking process of a test expert through a heuristic rule base, a list of potentially overlooked test scenarios related to or similar to the current defect, which may be missing from the target test case set, is generated, providing forward-looking supplementary test suggestions. Based on the preliminary list of potential test strategy vulnerabilities, the deeper list of test strategy vulnerabilities, and the list of potentially overlooked test scenarios, a final list of test strategy vulnerabilities is generated. This achieves the goal of effectively conducting in-depth analysis of defects and uncovering the test strategy vulnerabilities hidden behind them. Attached Figure Description

[0088] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0089] Figure 1 A flowchart illustrating a method for discovering vulnerabilities in a testing strategy, as provided in an embodiment of the present invention;

[0090] Figure 2 An overall architecture diagram of a test strategy vulnerability mining method provided in another embodiment of the present invention;

[0091] Figure 3 This is a schematic diagram of a test strategy vulnerability discovery method provided in another embodiment of the present invention. Detailed Implementation

[0092] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0093] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0094] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0095] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0096] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0097] This invention provides a method for vulnerability discovery in testing strategies, such as... Figure 1 As shown, the specific steps include:

[0098] S101. Receive defect information of the defect to be analyzed.

[0099] The defect to be analyzed can be a defect that is not covered by test cases. The defect information includes the text information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation. The text information of the defect to be analyzed includes, but is not limited to, the description information of the defect to be analyzed and the stack trace information of the defect to be analyzed. There are no restrictions here.

[0100] It should be noted that software project documentation includes, but is not limited to, requirements documents and design documents; this is not a specific definition. The requirements document describes the content to be developed. Defects caused by developing this content, as well as the code used to fix these defects (i.e., the related code), can be recorded through a defect management system. The defects and related code recorded in the defect management system can be linked to the software project documentation through identifiers for easy later use. The design document describes the system's architecture, module division, and technical solutions.

[0101] In the actual application of this invention, the source of the requirements document can be provided by the product manager or feedback from users during use; there is no limitation here.

[0102] Stack trace information is a snapshot or record of the call stack at a specific point in time (such as when a program encounters an exception or error), used to show the program's execution path. It typically includes information such as method name, class name, file name, and line number, helping developers locate problems.

[0103] S102. Based on the textual information of the defects to be analyzed and the test strategy knowledge graph, generate a preliminary list of potential test strategy vulnerabilities.

[0104] In the practical application of this invention, a pre-built test strategy knowledge graph is used to quickly associate the defects to be analyzed with known common defect patterns and corresponding test strategy vulnerabilities, providing a preliminary, high-confidence analysis direction.

[0105] Optionally, in another embodiment of the present invention, one implementation of the method for constructing a test strategy knowledge graph specifically includes the following steps (steps A1 to A3):

[0106] Step A1: Obtain multi-source data.

[0107] The multi-source data covers the entire software development and testing lifecycle. This multi-source data includes, but is not limited to, historical defect report repositories, version control systems (code commit logs, code changes), test case management systems, requirements and design documents, test summaries and post-test analysis reports, and common defect classification standards (such as CWE, OWASP Top 10), which are not limited here.

[0108] In the practical application of this invention, after acquiring multi-source data, the multi-source data can also be preprocessed. Preprocessing includes, but is not limited to, text cleaning and standardization, code parsing and feature extraction, entity linking and alignment, which are not limited here.

[0109] Specifically, text cleaning and standardization can be implemented by, but is not limited to, denoising unstructured text, removing irrelevant characters, standardizing terminology, and segmenting sentences; no specific limitations are made here.

[0110] The implementation of code parsing and feature extraction can be, but is not limited to, using static code analysis tools (such as parsers based on abstract syntax trees, ASTs) to parse code changes and extract key information, such as modified functions / methods, involved API interfaces, affected data structures / types, etc., which are not limited here.

[0111] The implementation of entity linking and alignment can be, but is not limited to, using entity linking technology to disambiguate and normalize mentions in different data sources that point to the same real-world concept. For example, associating the “authentication module” in a document, the AuthService class in the code, and the “login function” in a defect with the same “user authentication service” node in the knowledge graph, etc., is not limited here.

[0112] Step A2: Extract entity nodes and semantic relationships between entity nodes from multi-source data.

[0113] In the practical application of this invention, the specific implementation method for extracting entity nodes from multi-source data includes the following steps (steps A21 to A22):

[0114] Step A21: Entity recognition. In the actual application of this invention, a hybrid method of rules and models can be used, which is not limited here.

[0115] Specifically, based on domain dictionaries and pattern definition rules, and using a pre-trained language model finely tuned on software engineering text for named entity recognition, core entity nodes of preset types are automatically extracted from the text.

[0116] Among them, the core entity nodes of the preset types can be test strategies, defect patterns, functional modules / technical components, test design cognitive patterns, etc., which are not limited here.

[0117] It should be noted that testing strategies can include, but are not limited to, "equivalence class partitioning", "boundary value analysis", "state transition testing", "security fuzzing", etc., and are not limited here.

[0118] Defect patterns can include, but are not limited to, "null pointer dereferencing", "resource leakage", "SQL injection", "race condition", etc., which are not limited here.

[0119] Functional modules / technical components may include, but are not limited to, "order payment service", "Redis cache cluster", "front-end routing component", etc., and are not limited here.

[0120] Test design cognitive patterns can include, but are not limited to, "optimistic path assumptions," "ignoring concurrent scenarios," and "vague security boundaries," etc., which are not limited here.

[0121] Among them, test design cognitive patterns can be extracted from in-depth texts such as defect root cause analysis.

[0122] In the practical application of this invention, the fine-tuned pre-trained language model can be a pre-trained language model fine-tuned through supervised learning, such as BERT, RoBERTa, or variants thereof, etc., which is not limited here. The training method can be, but is not limited to, supervised learning, which is not limited here. The training samples are a large amount of labeled software engineering domain data (such as defect reports and code comments labeled with entity types and relationships), which is not limited here.

[0123] In the practical application of this invention, the task of the fine-tuned pre-trained language model is to classify the fragments identified from multi-source data into core entity nodes of a preset category.

[0124] For example, inputting a text (such as a defect description) and outputting the entity type label (such as B-TestStrategy, I-TestStrategy, O, etc.) corresponding to each token (word or character) in the text, thereby identifying entity boundaries and types.

[0125] Step A22: Attribute extraction and assignment, specifically adding key attributes to entity nodes. For example, add attributes such as "triggering conditions," "scope of impact," and "remediation suggestions" to the "defect mode" node; add attributes such as "applicable scenarios," "implementation steps," and "best practices" to the "test strategy" node. There are no restrictions here.

[0126] In the practical application of this invention, information extraction technology can be used to obtain key attributes from multi-source data. For example, "triggering conditions" may appear directly in the defect description.

[0127] In the practical application of this invention, the implementation method for extracting semantic relationships between entity nodes from multi-source data can employ, but is not limited to, a relation classifier based on syntactic dependency analysis and a pre-trained language model to extract semantic relationships between entity nodes from text (here, text includes but is not limited to the "root cause analysis" section of defect reports, test summary reports, sections describing inter-module interactions in design documents, and sections explaining functions or constraints in code comments, etc., which are not limited here. In particular, texts rich in logic, such as defect root cause analysis and test summaries, are included).

[0128] The main relationship types include, but are not limited to: prevention, applied to, caused by, related to, etc., without limitation here.

[0129] It should be noted that: Prevention (detects / ): connects "test strategy" and "defect pattern", indicating that the strategy can be used to detect or prevent such defects; Applies to (applies_to / ): connects "test strategy" and "functional module / technical component", indicating that the strategy is often applicable to the testing of such modules; Caused by (caused_by / ): connects "defect pattern" and "test design cognitive pattern", indicating that such defects are often caused by a certain blind spot or cognitive limitation in test thinking; Related to (related_to / ): connects nodes of the same or different types, indicating that there is a conceptual, logical or statistical correlation between them.

[0130] In the practical application of this invention, the relation classifier based on syntactic dependency analysis and pre-trained language models can be a pre-trained language model that has been fine-tuned through supervised learning, such as BERT, RoBERTa or its variants, etc., and is not limited here.

[0131] The training method can be, but is not limited to, supervised learning; no specific method is specified here. The training samples consist of a large amount of labeled data from the software engineering domain (such as defect reports and code comments labeled with entity types and relationships); no specific method is specified here.

[0132] For example, given a sentence text and two identified entities within the sentence (e.g., a "test strategy" entity and a "defect pattern" entity), the output would be: the most likely relationship type between the two entities (e.g., detects, apply_to, etc.), or "no relationship".

[0133] Step A3: Construct a test strategy knowledge graph based on entity nodes and the semantic relationships between them.

[0134] In practical applications of this invention, native graph databases (such as Neo4j and NebulaGraph) can be used to store triples in the form of (head entity, relation, tail entity) and to store node attributes. Then, graph embedding techniques are used to map the nodes and relations in the graph into low-dimensional dense vectors. These vectors can capture the structural and semantic information in the graph, which can be used for subsequent efficient similarity calculation, link prediction, and association reasoning.

[0135] Graph embedding is a representation learning method that maps entity nodes (or edges) in a graph to a low-dimensional dense vector space. Commonly used algorithms include, but are not limited to, Node2Vec, TransE, and Graph Neural Networks (GNNs). These vectors can capture the semantic information of nodes and their relationships within the graph structure.

[0136] Optionally, in another embodiment of the present invention, one implementation of step S102 specifically includes the following steps (steps B1 to B5):

[0137] Step B1: For each defect pattern node in the test strategy knowledge graph, determine the first matching degree based on the text information of the defect to be analyzed and the keyword set of the defect pattern node.

[0138] In the practical application of this invention, algorithms such as TF-IDF and Jaccard similarity can be used, but are not limited to, to calculate the matching degree between the defective text and the keyword set of the "defective pattern" node to obtain the first matching degree. No limitation is made here.

[0139] Step B2: Determine the second matching degree based on the description information of the defect to be analyzed and the description text of the defect pattern node.

[0140] In the practical application of this invention, lightweight sentence embedding models (such as Sentence-BERT) can be used to convert the defect description and the "defect pattern" description text into vectors, and then their cosine similarity can be calculated as a second matching degree. No limitation is made here.

[0141] Step B3: Determine the third matching degree based on the description information of the defect pattern node and the stack trace information of the defect to be analyzed.

[0142] In the practical application of this invention, the description information of the defect pattern node and the stack trace information of the defect to be analyzed can be matched based on rules and regular expressions. For example, if a match is found, the result is 1; otherwise, it is 0. This is not limited here.

[0143] Step B4: Based on the first matching degree, the second matching degree, and the third matching degree, determine the comprehensive correlation score between the defect to be analyzed and the defect pattern node.

[0144] Specifically, the calculation method for the comprehensive association score between the defect to be analyzed and the defect pattern node can be to assign configurable weights to the scores of the above three dimensions (such as keyword weight 0.3, semantic weight 0.5, and stack weight 0.2), and then perform weighted summation to obtain the comprehensive association score between the defect to be analyzed and the defect pattern node. There are no restrictions here.

[0145] Step B5: The test strategy nodes connected to defect pattern nodes whose overall correlation score is greater than the correlation score threshold are considered as preliminary potential test strategy vulnerabilities.

[0146] The correlation score threshold is preset and modified by experts or authorized technical personnel, and is not limited here.

[0147] In the practical application of this invention, after obtaining the initial potential test strategy vulnerabilities, all the initial potential test strategy vulnerabilities will be merged and deduplicated to finally generate an initial potential test strategy vulnerability list.

[0148] S103. Based on the description information of the defects to be analyzed, the relevant code of the defects to be analyzed, and the software project documents, generate a list of vulnerabilities for in-depth testing strategies.

[0149] This invention goes beyond pattern matching in existing technologies by understanding and analyzing the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documents. It also delves into the deep technical reasons and logical roots of the defect, and reveals the wrong assumptions made in the test design, the ignored system states or combinations of conditions, that is, digging out the root cause of "why didn't we think of testing this".

[0150] Optionally, in another embodiment of the present invention, one implementation of step S103 specifically includes the following steps (steps C1 to C4):

[0151] Step C1: Generate the input sequence based on the guiding questions, the description of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation.

[0152] Specifically, the description of the defect to be analyzed, the relevant code of the defect, the requirement context from the software project documentation, and a pre-set guiding question are concatenated and separated by a special delimiter. For example:

[0153] [CLS] Description of the defect to be analyzed: {D.description} [SEP], Related code of the defect to be analyzed: {C} [SEP], Requirement context in the software project document: {R} [SEP], Preset guiding question: {Q_i} [SEP].

[0154] Step C2: Input the input sequence into the causal reasoning model and output the answer to the guiding question.

[0155] The architecture of the causal reasoning model can be based on the BERT (Bidirectional Encoder Representations from Transformers) architecture, with large-scale domain-specific fine-tuning to build a causal reasoning model focused on "root cause analysis of software defects and test vulnerability analysis".

[0156] In the practical application of this invention, the training data for the causal inference model is supervised learning using paired data consisting of a large amount of historical defect reports, code changes, test cases, and post-analysis reports. The learning objective is to establish a mapping from (defect description, related code, requirement context) to (root cause, missed test scenario).

[0157] It should be noted that the causal reasoning model adopts a multi-round question-and-answer reasoning approach, which solidifies the reasoning process into a set of pre-set, guiding questions [Q_1,Q_2,Q_3,...]. These guiding questions aim to deconstruct the thought process of test design.

[0158] In the practical application of this invention, the causal reasoning model encodes the entire context for each guiding question, and then generates the answer A_i to Q_i at a specific position in the input sequence through an additional decoding head (such as a linear layer + Softmax) and "mask language modeling". This process utilizes BERT's powerful bidirectional attention mechanism, which can simultaneously understand the complex relationships between defects, code, requirements, and problems.

[0159] Step C3: Identify cognitive blind spots based on the answers to all guiding questions.

[0160] It's important to note that the answers to the later questions in the chain of guiding questions most clearly reveal knowledge gaps. Therefore, one can identify these knowledge gaps by finding answers to the later guiding questions.

[0161] In the practical application of this invention, [Q_1:A_1,Q_2:A_2,Q_3:A_3] can be connected sequentially to automatically form a logical chain from "defect phenomenon" to "test design blind spot", that is, a causal chain. For example, A_1 points out "assuming the input is a positive integer", A_2 derives "not considering extremely large integers or negative numbers", and A_3 summarizes as "ignoring integer overflow and chain reaction".

[0162] Step C4: Map cognitive blind spots to the test strategy knowledge graph to obtain deep test strategy vulnerabilities.

[0163] In the actual application of the application, each deep testing strategy vulnerability can also be assigned a confidence level determined by the model output or rules, which is not limited here.

[0164] S104. Based on the heuristic rule base and the target test case set, determine the list of potential missed test scenarios.

[0165] The target test case set is a set of executed test cases related to the defect to be analyzed.

[0166] This invention is not limited to explaining existing defects. Furthermore, it uses heuristic rules to simulate the divergent thinking process of test experts, generating other potential test scenarios that may be missed in the current test set and are related to or similar to the current defects, thus providing forward-looking supplementary test suggestions.

[0167] Each heuristic rule in the heuristic rule base includes a heuristic type, triggering condition, and scene seed template.

[0168] Optionally, in another embodiment of the present invention, one implementation of step S104 specifically includes the following steps (steps D1 to D4):

[0169] Step D1: Identify the target scenarios in the target test case set.

[0170] Specifically, the target scenario can be quickly identified through scenario markers in the target test case set, but not limited to. The target scenario can be, but is not limited to, mainstream scenarios, success scenarios, etc., and is not restricted here.

[0171] Step D2: For each target scenario, match the target scenario with the heuristic rules in the heuristic rule base to obtain the target heuristic rule.

[0172] Specifically, for each target scenario, the target scenario is matched with heuristic rules in the heuristic rule base, and the heuristic rules that match successfully are used as the target heuristic rules.

[0173] Step D3: Generate scene seeds based on goal-oriented heuristic rules.

[0174] Assume the target heuristic rule includes the heuristic type of reverse thinking, the trigger condition is the existence of test cases verifying "operation A succeeds under condition B", and the scenario seed template considers the scenario of "operation A fails under abnormal conditions or when condition B is not met", or "the reverse operation of operation A".

[0175] When the triggering condition is met: there is a test case verifying that "operation A succeeds under condition B", such as "user successfully submits order", then the scenario seed template can be used to generate a scenario seed: "Consider the scenario of 'user fails to submit order'".

[0176] Step D4: Input the scene seed and prompt words into the test scene description model, and output the potential missing test scenes.

[0177] Since the generated scene seeds lack specific details and cannot be directly used for testing, this invention provides a test scene description model, which can be a lightweight text generation model (such as a medium-sized model with fine-tuned instructions) to transform scene seeds into specific, reasonable, and verifiable potential omission test scenes.

[0178] In the practical application of this invention, fine-tuning can be achieved, but is not limited to, collecting historical defect reports and having experts manually annotate standard answers for a pre-defined chain of questions (such as "What is the test hypothesis?"), forming training pairs of (question context, answer). Then, a pre-trained model (such as BERT) is selected and retrained on the prepared data with a low learning rate. The model learns a specific pattern for defect analysis by comparing its own output with the standard answer and adjusting its parameters accordingly. Finally, the model's performance is validated using data not used in training; once it meets the standards, it can be used to analyze new defects.

[0179] In the practical application of this invention, prompts can also be designed to guide the model through step-by-step reasoning, for example:

[0180] Given the original test scenario: 'User successfully submits an order'. Please apply the 'reverse thinking' heuristic. Step by step consider the various specific reasons why the user's order submission might fail:

[0181] 1. From the user / client perspective: network connection interruption, form data tampering, duplicate submissions...

[0182] 2. From a business logic perspective: insufficient inventory, product removal, user account freezing...

[0183] 3. From the server / system perspective: payment gateway timeout, database write failure, risk control interception...

[0184] Based on the above analysis, please generate three specific and testable descriptions of 'order submission failure' scenarios.

[0185] This structured prompting and guidance model simulates the reasoning path of human experts, producing logically coherent, comprehensive, and detailed descriptions of specific scenarios, while avoiding the generation of empty, repetitive, or irrelevant content.

[0186] S105. Based on the preliminary potential test strategy vulnerability list, the in-depth test strategy vulnerability list, and the potential omission test scenario list, generate the final test strategy vulnerability list.

[0187] Optionally, in another embodiment of the present invention, one implementation of step S105 specifically includes the following steps (steps E1 to E3):

[0188] Step E1: Based on the preliminary list of potential testing strategy vulnerabilities and the deep testing strategy vulnerability list, identify the target testing strategy vulnerability.

[0189] First, using the graph embedding vector of the "Test Strategy" node in the test strategy knowledge graph, we calculate the semantic similarity of each item in the preliminary potential test strategy vulnerability list V_preliminary and the deep test strategy vulnerability list V_semantic.

[0190] Specifically, for policies in V_preliminary (such as "boundary value analysis"), the graph embedding vectors pre-generated in the test policy knowledge graph are used directly. New descriptions in V_semantic (such as "insufficient numerical boundary validation") are converted into semantic vectors using a sentence encoding model (such as Sentence-BERT). Then, the cosine similarity between these two vectors is calculated. The method can be, but is not limited to, calculating cosine similarity. The formula is: Similarity = (Vec_New·Vec_A) / (||Vec_New||*||Vec_A||), where the similarity (between -1 and 1) quantifies their semantic closeness; the closer the value is to 1, the more similar the meanings.

[0191] In practical applications of this invention, the vector of a new text item can also be compared with all policy vectors in the test policy knowledge graph. If its similarity to an existing policy exceeds a preset threshold, it is determined to be the same vulnerability and merged into the same test policy vulnerability v_i. If it is below the threshold, it is considered a new vulnerability.

[0192] For each specific scenario in the potential omission test scenario list H_hypotheses, keyword extraction and semantic analysis are used to categorize or map it to one or more specific "test strategy" nodes in the test strategy knowledge graph, thus establishing a connection.

[0193] Step E2: For each target test strategy vulnerability, determine the overall confidence level of the target test strategy vulnerability based on the comprehensive correlation score of the initial potential test strategy vulnerability corresponding to the target test strategy vulnerability, the confidence level of the deep test strategy vulnerability corresponding to the target test strategy vulnerability, and the correlation strength between the target test strategy vulnerability and the list of potential omitted test scenarios.

[0194] In the practical application of this invention, the comprehensive confidence level S_i of the target testing strategy vulnerability can be calculated using the following formula:

[0195] S_i=α*Score_surface(v_i)+β*Confidence_semantic(v_i)+γ*Relevance_heuristic(v_i);

[0196] Wherein, Score_surface(v_i): the comprehensive correlation score of the preliminary potential test strategy vulnerability in V_preliminary corresponding to the target test strategy vulnerability v_i;

[0197] Confidence_semantic(v_i): The confidence level of the deep test strategy vulnerability in V_semantic corresponding to the target test strategy vulnerability v_i, determined by the model output or rule judgment.

[0198] Relevance_heuristic(v_i): Measures the strength of the association between v_i and the scenarios in H_hypotheses (e.g., how many generated scenario hypotheses can be covered by the test strategy represented by v_i; the higher the coverage, the stronger the association).

[0199] It should be noted that α, β, and γ are configurable weight parameters that satisfy α+β+γ=1, used to balance the credibility of results at different inference levels.

[0200] Step E3: Generate the final list of test strategy vulnerabilities based on the overall confidence level of all target test strategy vulnerabilities.

[0201] Specifically, the target test strategy vulnerabilities are sorted according to the overall confidence level of all target test strategy vulnerabilities to generate the final test strategy vulnerability list.

[0202] Optionally, in another embodiment of the present invention, the potential severity of the "defect mode" associated with the target test strategy vulnerability (obtained from the test strategy knowledge graph) and the frequency of occurrence of the target test strategy vulnerability in the project history can also be combined for weighted calculation to prioritize the final test strategy vulnerability list V_final, so as to more effectively guide the priorities of improvement.

[0203] Optionally, in another embodiment of the present invention, an implementation manner of the test strategy vulnerability mining method further includes the following steps (steps F1 to step F3):

[0204] Step F1: Generate candidate relation triples based on the list of potential missing test scenarios and the final test strategy vulnerability list.

[0205] In the actual application process of the present invention, rule templates can be predefined, for example: the rule of "if event X occurs, then create relationship Y" is set in advance.

[0206] Suppose the system analyzes an uncovered defect D_123 (defect ID) and concludes that this defect exposes a test strategy vulnerability V_456 (such as "omission of idempotency testing").

[0207] Rule template: If it is analyzed that defect D is caused by vulnerability V, then create the triple <D, exposes, V>. When triggered, the system will automatically generate a new triple:

[0208] Head entity: D_123;

[0209] Relationship: exposes (exposes);

[0210] Tail entity: V_456;

[0211] Through similar rules, the following can also be generated:

[0212] <V_456, suggests_strategy, T_789> (vulnerability V_456 suggests adopting test strategy T_789);

[0213] <P_zhangsan, has_blindspot, V_456> (tester P_zhangsan has a blind spot V_456).

[0214] Step F2: Generate candidate entity nodes based on cognitive blind spots.

[0215] In the actual application process of the present invention, new "test design cognitive mode" nodes (such as "ignoring the testing of eventual consistency between microservices", which is not limited here) can be identified and generated from the description of cognitive blind spots through text clustering and summarization techniques.

[0216] Step F3: Update the test strategy knowledge graph based on the candidate relation triples and candidate entity nodes to obtain the updated test strategy knowledge graph.

[0217] Specifically, candidate relation triples and candidate entity nodes, along with their source (original ID of the defect to be analyzed), confidence level, timestamp, and other information, are submitted as incremental data to the knowledge graph construction engine. The engine performs disambiguation and alignment operations and integrates the new knowledge into the existing test strategy knowledge graph. Simultaneously, a consistency check is performed. If the newly added knowledge logically conflicts with existing knowledge (e.g., both A detects B and A cannot detect B), a manual review process is triggered, or an automatic decision is made based on the confidence level.

[0218] In the practical application of this invention, for newly extracted or generated candidate entity nodes and candidate relation triples, the semantic similarity between their vector representations and existing entity nodes in the test strategy knowledge graph is calculated to determine whether to create a new node or merge it with the existing entity node with the closest semantics, and to enhance or adjust the weight of the related relations.

[0219] It should be noted that each relation (edge) in the knowledge graph used for testing strategies is assigned an initial confidence level. This confidence level is dynamically adjusted based on the frequency with which it is verified or used in a new round of analysis (as frequency increases, confidence increases accordingly). Simultaneously, a decay mechanism is introduced: for knowledge that has not been referenced or verified in any analysis process for a long time, its confidence level will slowly decrease over time to maintain the timeliness and accuracy of the knowledge base.

[0220] Optionally, in another embodiment of the present invention, one implementation of the test strategy vulnerability discovery method further combines the final test strategy vulnerability list with the tester's capability profile to generate highly personalized improvement reports and learning tasks with direct guidance significance, and tracks the improvement effects to form a closed loop driving continuous growth in individual capabilities, specifically including:

[0221] Based on the testers' responsibility domains and the areas where the defects to be analyzed occur, target testers are identified; the capability profiles of the target testers are updated using the final test strategy vulnerability list, resulting in the updated capability profiles of the target testers.

[0222] The domain of responsibility for tester P is the list of functional modules and technical components that tester P is currently responsible for or has recently participated in testing, and there are no restrictions here.

[0223] In the practical application of this invention, a dynamic capability profile will be constructed and maintained for each tester P, which can typically be represented as a multi-dimensional vector or a subgraph of a knowledge graph, and may include, but is not limited to:

[0224] Responsibility domain: A list of functional modules and technical components that P is currently responsible for or has recently participated in testing.

[0225] Historical Defect Associations: A list that records defects introduced by P (development role) or that should have been tested by P but were not detected. Each defect is associated with the type of vulnerability in the test strategy it exposed (from V_final).

[0226] Capability Model: A multi-dimensional vector, where each dimension corresponds to a "testing strategy" or "cognitive pattern" in the knowledge graph, and its value represents P's mastery level in that dimension (proficiency score, such as 0.0-1.0). The initial proficiency score can be evaluated by the quality of its historically designed test cases (such as coverage and defect detection rate).

[0227] Learning and Improvement Log: List of recommended learning materials, training sessions, and improvement tasks completed by P.

[0228] In practical applications of this invention, an initial score calculation model for a tester's competency profile can be provided to objectively calculate the initial proficiency score (0.0-1.0) for each "testing strategy" dimension in the tester's competency profile. This model is based on historical data, ensuring fair and traceable scoring.

[0229] Specifically, the "test case quality" is broken down into several quantifiable metrics, each with a weight, and then a weighted sum is obtained to arrive at a comprehensive score.

[0230] Among them, the quantitative metrics refer to the following metrics calculated for each tester P and each dimension S of the "test strategy" they are responsible for (such as "boundary value analysis"):

[0231] 1. Use Case Effectiveness, Definition: Measures whether the designed use cases can accurately find defects. It is evaluated through the "Defect Detection Rate". Calculation Formula: Defect Detection Rate = P (Number of defects found by executing self-designed use cases) / P (Total number of defects found by the module under your responsibility during the testing cycle). It should be noted that a higher rate indicates more effective use cases that accurately identify real problems.

[0232] 2. Test Case Coverage Completeness: Definition: Measures the degree to which test cases cover the requirement scenarios and code logic. It is evaluated through "Coverage Contribution". Calculation formula: Coverage Contribution = (Number of lines of code or requirement items covered by test cases designed by P) / (Total number of lines of code or requirement items in the module P is responsible for). It should be noted that a higher contribution indicates a more comprehensive test design.

[0233] 3. Use Case Design Efficiency, defined as: measuring whether use cases can accurately locate problems and reduce debugging costs. It is evaluated through "Defect Location Accuracy". The calculation formula is: Defect Location Accuracy = 1 / (P - Average Fix Time for Found Defects). The shorter the fix time, the more accurate the error information provided by the use case, and the higher the location accuracy score. It should be noted that high accuracy means the use case has a clear objective, helping development to fix problems quickly.

[0234] Integrating the above indicators, we form the initial score calculation formula:

[0235] Initial proficiency score (S) = α * defect detection rate + β * coverage contribution + γ * defect location accuracy;

[0236] Where α, β, and γ are configurable weighting coefficients that satisfy α + β + γ = 1. For example, if the ability to detect defects is given more importance, α = 0.5, β = 0.3, and γ = 0.2 can be set.

[0237] It should be noted that the above data was automatically obtained from the test management system, version control system, and defect tracking system.

[0238] In the actual application of this invention, the calculated original score will be normalized and mapped to the range of 0.0 to 1.0.

[0239] To illustrate the above, let's take an example: Suppose tester Xiao Wang is responsible for the "order payment" module.

[0240] In this round of testing, a total of 20 defects were found in this module, 15 of which were discovered by test cases designed by Xiao Wang himself. The defect discovery rate is 15 / 20 = 0.75.

[0241] The payment module has 1000 lines of code, and Xiao Wang's test cases cover 700 of them. His coverage contribution is 700 / 1000 = 0.7.

[0242] Xiao Wang discovered 15 defects, with an average repair time of 2 hours. The defect location accuracy is 1 / 2 = 0.5.

[0243] If the weights are set to α=0.5, β=0.3, and γ=0.2, then Xiao Wang's initial score in the relevant strategy dimension (such as "payment process test") is:

[0244] Initial score = 0.5*0.75 + 0.3*0.7 + 0.2*0.5 = 0.375 + 0.21 + 0.1 = 0.685.

[0245] In the practical application of this invention, the method for updating the tester's capability profile includes: for each vulnerability v_i in V_final, finding its corresponding dimension in the capability model of P, and adjusting (deducting) the score of that dimension according to the severity level and analysis confidence S_i of the vulnerability. The deduction formula is configurable, for example: new score = original score * (1 - attenuation factor * severity weight). Severe vulnerabilities will lead to a greater decrease in score. The severity weight is the score corresponding to the severity level, which can be preset and is not specified here.

[0246] In the practical application of this invention, the defect D_uncovered and its associated V_final can also be recorded in the "historical defect association" of P.

[0247] Optionally, in another embodiment of the present invention, one implementation of the test strategy vulnerability discovery method further includes:

[0248] Based on the updated capability profile of the target tester, the test strategy knowledge graph, guiding questions, the answers to the guiding questions, the list of potential missed test scenarios, and the final list of test strategy vulnerabilities, a personalized report for the target tester is generated.

[0249] In the practical application of this invention, a structured report template can be preset, including sections such as "Overview Summary", "In-depth Defect Case Analysis", "Trends in Your Testing Capabilities", "Personalized Improvement Suggestions", and "Recommended Learning and Practice Tasks", without any limitations here.

[0250] Among them, the blind spot trend can be identified based on the updated ability model of the target tester P, identifying dimensions with persistently low scores or significant recent declines. By clustering analysis of the data in its "historical defect association", the test strategy types or thinking patterns that the target tester P frequently overlooked can be summarized.

[0251] In-depth defect case analysis can use the defect to be analyzed as a core case. Using natural language generation technology, the causal chain (including guiding questions and their corresponding answers, as detailed in the above examples) and a list of potential missed test scenarios are organized into a fluent, narrative-style analysis article that clearly explains "how this defect occurred?", "why it wasn't detected in the initial test?", and "from which other relevant angles could testing have been conducted?", making the analysis process easy to understand and absorb.

[0252] Personalized improvement suggestions include strategy recommendations and learning material recommendations. Strategy recommendations refer to querying the knowledge graph for "testing strategy" nodes that are strongly correlated with the target tester P's weak dimensions (scores below the threshold), and extracting their attribute information such as "implementation steps," "checklists," and "best practices" as specific improvement methods. Learning material recommendations refer to associating and recommending learning resources such as internal wiki pages, external courses, technical articles, and classic cases linked to the strategy node in the testing strategy knowledge graph; there are no restrictions on these resources.

[0253] The recommended learning and practice tasks are generated automatically by combining the specific code modules or interfaces within the target tester P's current "domain of responsibility," their personal weaknesses (vulnerabilities in V_final), and system-generated extended scenarios (H_hypotheses). For example: "Please design three boundary test cases regarding 'network timeout and retry mechanisms' for the 'order cancellation interface' you are responsible for." This task can be directly created or linked to the project management system, becoming a to-do item for the target tester P.

[0254] In the practical application of this invention, the generated personalized reports can also be pushed to the target tester P regularly (e.g., every two weeks) or triggered (when a high-severity blind spot is found) via email, instant messaging tools, or integrated portals, without limitation here.

[0255] The system tracks the completion status of the "practical tasks" in the report by the target tester P, as well as the defect density of the module under their responsibility in the subsequent period after the task is completed, especially the recurrence of similar defects.

[0256] Positive feedback (such as task completion and defect density reduction) is used as a positive incentive to improve the score of the target tester P on the corresponding dimension of the capability model (increased reward), and the effectiveness of the recommended strategy is evaluated, forming a complete capability improvement closed loop of "analysis-feedback-improvement-verification".

[0257] like Figure 2 The diagram shown illustrates the overall architecture of a vulnerability discovery method for testing strategies, as provided in this embodiment of the invention. The method comprises four stages:

[0258] Phase 1 is knowledge base construction, which is based on multi-source data (multi-source heterogeneous data) for preprocessing to generate a knowledge graph (test strategy knowledge graph).

[0259] Phase two is a hierarchical causal tracing analysis, corresponding to steps S102 to S104 in this invention. It inputs uncovered defect D (the defect to be analyzed), code C (the relevant code of the defect to be analyzed), and document R (software project document), and performs three progressively layered reasoning algorithms. The aim is to deeply analyze each "defect not covered by test cases" and uncover the test design vulnerabilities hidden behind it.

[0260] Among them, surface-level correlation reasoning generates a preliminary list of potential test strategy vulnerabilities based on the textual information of the defects to be analyzed and the test strategy knowledge graph, generating a potential strategy vulnerability list V_pre (preliminary potential test strategy vulnerability list); deep semantic reasoning generates a layer test strategy vulnerability list V_sem (deep test strategy vulnerability list) based on the description information of the defects to be analyzed, the relevant code of the defects to be analyzed, and the software project documents; and innovative thinking reasoning determines the extended test scenario hypothesis H_hyp (potentially omitted test scenario list) based on the heuristic rule base and the target test case set.

[0261] Phase three involves result integration and knowledge accumulation. Through vulnerability integration and priority sorting, a final policy vulnerability list V_final (the final test policy vulnerability list) is generated. Then, new knowledge is extracted and the knowledge graph is incrementally updated based on V_final.

[0262] Phase four involves personalized report generation and feedback. Based on V_final and the personal ability profile of the target tester P, a personalized report is generated and pushed to the target tester P. Task tracking and separation are performed in real time, and the personal ability profile is updated based on the improvement effect.

[0263] The present invention will be illustrated by taking the analysis of the vulnerability of "duplicate payment of orders" in an e-commerce system as an example.

[0264] Phase 1: First, the system creates a continuously accumulating test knowledge base (test strategy knowledge graph), which records various common defects (such as "idempotency problems"), test methods (such as "boundary value analysis"), and blind spots in thinking (such as "optimistic path assumptions").

[0265] Phase Two: Hierarchical Causal Tracing Analysis;

[0266] 1. Trigger: Testers submit the "duplicate order payment" defect to the system for analysis (not covered by test cases).

[0267] 2. Rapid screening (surface reasoning): The system automatically matches the knowledge base and quickly suggests that "idempotency tests and network anomaly tests may have been missed".

[0268] 3. In-depth analysis (deep reasoning): The intelligent model analyzes the defects in depth and discovers the root cause: the test only considered the two clear results of "payment successful" and "payment failed", ignoring the intermediate state of "payment request has been sent but the result is unknown". This is precisely the blind spot in test design thinking.

[0269] 4. Expanding Associations (Innovative Reasoning): The system further utilizes rules such as "reverse thinking" to automatically deduce other relevant test scenarios, such as potential vulnerabilities like "payment successful but callback notification lost" and "user cancels order during payment".

[0270] Phase Three: Integration and Learning;

[0271] 1. Integration Results: The system merges and deduplicates the analysis results from each layer, sorts them by importance, and outputs the final conclusion: the "interface idempotency test" and "asynchronous timeout handling test" need to be completed with the highest priority.

[0272] 2. Knowledge Accumulation: The system automatically associates new insights gained from this analysis (such as the mindset of "ignoring the eventual consistency test of asynchronous operations") and newly discovered defects-testing strategies with the knowledge base, making the entire system smarter.

[0273] Phase Four: Personalized Report Generation and Feedback;

[0274] 1. Identify the person responsible: The system found that the person in charge of testing the defective module was "Xiao Wang".

[0275] 2. Report Generation: The system automatically generates a personalized report for Xiao Wang, including the following:

[0276] In-depth analysis: Explaining in simple terms "why this defect was not detected during testing".

[0277] Blind spots in capabilities: It was clearly pointed out that Xiao Wang has shortcomings in "fault tolerance testing" and "concurrency testing".

[0278] Action Guide: Recommend relevant learning materials and automatically create a practice task: "Design three test cases covering the 'concurrency of cancellation and payment callbacks' scenario for the 'order cancellation interface' that you are responsible for."

[0279] 3. Forming a closed loop: Xiao Wang learns and completes the practical task. Subsequent tracking shows that the number of similar defects in the module he is responsible for has decreased, and the system has correspondingly improved his ability score, completing the closed loop of "analysis-learning-improvement-verification".

[0280] As shown in the above scheme, this invention provides a method for discovering test strategy vulnerabilities. It utilizes the textual information of the defect to be analyzed and a test strategy knowledge graph to generate a preliminary list of potential test strategy vulnerabilities. Then, based on the description information of the defect to be analyzed, its related code, and software project documentation, a deeper list of test strategy vulnerabilities is generated. Furthermore, by simulating the divergent thinking process of a test expert through a heuristic rule base, a list of potentially overlooked test scenarios related to or similar to the current defect, which may be missing from the target test case set, is generated, enabling proactive test supplementation suggestions. Based on the preliminary list of potential test strategy vulnerabilities, the deeper list of test strategy vulnerabilities, and the list of potentially overlooked test scenarios, a final list of test strategy vulnerabilities is generated. This achieves the goal of effectively conducting in-depth analysis of defects and uncovering the test strategy vulnerabilities hidden behind them.

[0281] Another embodiment of the present invention provides a vulnerability discovery device for testing strategies, such as... Figure 3 As shown, it specifically includes:

[0282] The receiving unit 301 is used to receive defect information of the defect to be analyzed.

[0283] The defect information includes the text information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation; the text information of the defect to be analyzed includes at least the descriptive information of the defect to be analyzed.

[0284] The first list generation unit 302 is used to generate a preliminary list of potential test strategy vulnerabilities based on the text information of the defects to be analyzed and the test strategy knowledge graph.

[0285] The second list generation unit 303 is used to generate a deep testing strategy vulnerability list based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documents.

[0286] The third list generation unit 304 is used to determine a list of potential missing test scenarios based on a heuristic rule base and a set of target test cases.

[0287] The target test case set is a set of executed test cases related to the defect to be analyzed.

[0288] The fourth list generation unit 305 is used to generate the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list.

[0289] For details on the specific operation of the units disclosed in the above embodiments of the present invention, please refer to the corresponding method embodiments, such as... Figure 1 As shown, it will not be elaborated further here.

[0290] As can be seen from the above scheme, the present invention provides a test strategy vulnerability mining device. After the receiving unit 301 receives the defect information of the defect to be analyzed, the first list generation unit 302 uses the text information of the defect to be analyzed and the test strategy knowledge graph to generate a preliminary potential test strategy vulnerability list. Then, the second list generation unit 303 generates a deep test strategy vulnerability list based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documents. Further, the third list generation unit 304 simulates the divergent thinking process of test experts through a heuristic rule base to generate a list of potential omission test scenarios that may be missed in the target test case set and are related to or similar to the current defect, thereby realizing forward-looking test supplement suggestions. Finally, the fourth list generation unit 305 generates a final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omission test scenario list. Thus, the purpose of effectively conducting in-depth analysis of defects and mining the test strategy vulnerabilities hidden behind the defects is achieved.

[0291] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0292] Another embodiment of the present invention provides an electronic device, comprising:

[0293] One or more processors.

[0294] A storage device on which one or more programs are stored.

[0295] When the one or more programs are executed by the one or more processors, the one or more processors implement the test strategy vulnerability discovery method as described in the above embodiments.

[0296] Another embodiment of the present invention provides a computer storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the test strategy vulnerability mining method as described in the above embodiments.

[0297] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0298] It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0299] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0300] Another embodiment of the present invention provides a computer program product, which, when executed, is used to perform the above-described test strategy vulnerability discovery method.

[0301] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, it performs the functions defined in the methods of the embodiments of the present invention.

[0302] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in this invention is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely exemplary forms of implementing the invention.

[0303] While several specific implementation details are included in the foregoing discussion, these should not be construed as limiting the scope of the invention. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0304] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with technical features of the present invention (but not limited to) that have similar functions.

Claims

1. A method for vulnerability discovery in testing strategies, characterized in that, include: Receive defect information of a defect to be analyzed; wherein the defect information includes text information of the defect to be analyzed, related code of the defect to be analyzed, and software project documentation; the text information of the defect to be analyzed includes at least a description of the defect to be analyzed; Based on the textual information of the defects to be analyzed and the test strategy knowledge graph, a preliminary list of potential test strategy vulnerabilities is generated. Based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation, a deep testing strategy vulnerability list is generated. Based on a heuristic rule base and a target test case set, a list of potential missed test scenarios is determined; wherein, the target test case set is a set of executed test cases related to the defect to be analyzed; Based on the preliminary list of potential test strategy vulnerabilities, the deep test strategy vulnerability list, and the list of potential omitted test scenarios, a final list of test strategy vulnerabilities is generated.

2. The vulnerability discovery method for testing strategies according to claim 1, characterized in that, The method for constructing the test strategy knowledge graph includes: Acquire multi-source data; wherein the multi-source data covers the entire lifecycle of software development and testing; Entity nodes and semantic relationships between entity nodes are extracted from the multi-source data. A test strategy knowledge graph is constructed based on the entity nodes and the semantic relationships between them.

3. The vulnerability discovery method for testing strategies according to claim 1, characterized in that, The textual information of the defect to be analyzed also includes: stack trace information of the defect to be analyzed; the preliminary list of potential test strategy vulnerabilities is generated based on the textual information of the defect to be analyzed and the test strategy knowledge graph, including: For each defect pattern node in the test strategy knowledge graph, a first matching degree is determined based on the text information of the defect to be analyzed and the keyword set of the defect pattern node. Based on the description information of the defect to be analyzed and the description text of the defect pattern node, a second matching degree is determined; Based on the description information of the defect pattern node and the stack trace information of the defect to be analyzed, a third matching degree is determined; Based on the first matching degree, the second matching degree, and the third matching degree, the comprehensive correlation score between the defect to be analyzed and the defect pattern node is determined; Test strategy nodes connected to defect pattern nodes whose overall correlation score is greater than the correlation score threshold are considered as preliminary potential test strategy vulnerabilities.

4. The vulnerability discovery method for testing strategies according to claim 1, characterized in that, Based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation, a deep testing strategy vulnerability list is generated, including: Based on the guiding questions, the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project documentation, an input sequence is generated; The input sequence is fed into the causal reasoning model, and the answer to the guiding question is output. Based on the answers to all the guiding questions, identify cognitive blind spots; By mapping the aforementioned cognitive blind spots to a test strategy knowledge graph, deep test strategy vulnerabilities are obtained.

5. The vulnerability discovery method for testing strategies according to claim 1, characterized in that, The list of potential missed test scenarios, determined based on a heuristic rule base and a set of target test cases, includes: Identify the target scenarios within the target test case set; For each target scenario, the target scenario is matched with the heuristic rules in the heuristic rule base to obtain the target heuristic rule; Based on the aforementioned target heuristic rules, scene seeds are generated; The scene seed and prompt words are input into the test scene description model, and the potential missing test scenes are output.

6. The vulnerability discovery method for testing strategies according to claim 1, characterized in that, The final test strategy vulnerability list is generated based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list, including: Based on the preliminary list of potential testing strategy vulnerabilities and the deep list of testing strategy vulnerabilities, the target testing strategy vulnerability is determined; For each target test strategy vulnerability, the overall confidence level of the target test strategy vulnerability is determined based on the comprehensive correlation score of the initial potential test strategy vulnerability corresponding to the target test strategy vulnerability, the confidence level of the deep test strategy vulnerability corresponding to the target test strategy vulnerability, and the correlation strength between the target test strategy vulnerability and the potential omitted test scenario list. Based on the overall confidence level of all the target test strategy vulnerabilities, a final list of test strategy vulnerabilities is generated.

7. The vulnerability discovery method for testing strategies according to claim 4, characterized in that, After generating the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list, the process further includes: Based on the potential omission test scenario list and the final test strategy vulnerability list, candidate relationship triples are generated; Based on the aforementioned cognitive blind spots, candidate entity nodes are generated; The test strategy knowledge graph is updated based on the candidate relation triples and the candidate entity nodes to obtain the updated test strategy knowledge graph.

8. The vulnerability discovery method for testing strategies according to claim 4, characterized in that, After generating the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list, the process further includes: The target testers are determined based on the testers' areas of responsibility and the areas where the defects to be analyzed occur. The target tester's capability profile is updated using the final test strategy vulnerability list to obtain the updated capability profile.

9. The vulnerability discovery method for testing strategies according to claim 8, characterized in that, After updating the target tester's capability profile using the final test strategy vulnerability list to obtain the updated capability profile, the process further includes: Based on the updated capability profile of the target tester, the test strategy knowledge graph, the guiding questions, the answers to the guiding questions, the list of potential missed test scenarios, and the final list of test strategy vulnerabilities, a personalized report for the target tester is generated.

10. A vulnerability discovery device for testing strategies, characterized in that, include: A receiving unit is configured to receive defect information of a defect to be analyzed; wherein the defect information includes text information of the defect to be analyzed, related code of the defect to be analyzed, and software project documentation; the text information of the defect to be analyzed includes at least a description of the defect to be analyzed. The first list generation unit is used to generate a preliminary list of potential test strategy vulnerabilities based on the text information of the defects to be analyzed and the test strategy knowledge graph. The second list generation unit is used to generate a deep testing strategy vulnerability list based on the description information of the defect to be analyzed, the relevant code of the defect to be analyzed, and the software project document. The third list generation unit is used to determine a list of potential missing test scenarios based on a heuristic rule base and a target test case set; wherein, the target test case set is a set of executed test cases related to the defect to be analyzed; The fourth list generation unit is used to generate the final test strategy vulnerability list based on the preliminary potential test strategy vulnerability list, the deep test strategy vulnerability list, and the potential omitted test scenario list.

Images