A software evaluation outline generation method, device and program product for human-AI layered collaboration

Abstract

A software evaluation outline generation method and device for human-AI hierarchical collaboration, a computer readable storage medium and a program product belong to the technical field of software testing, to solve the problems of incomplete test coverage caused by AI context limitation, low efficiency and unstable quality of pure manual writing, lack of human high-level control in end-to-end automation, and high cost of model retraining for knowledge update in the existing software evaluation outline generation. The present application comprises obtaining software requirement documents and parsing them into structured data, extracting test items to build an outline framework; responding to manual editing optimization to form a complete framework; using a large language model to generate the content of each test item; inputting the generated content into the same model to restore the requirements and calculate the semantic similarity with the original requirements, to determine whether human intervention is needed; outputting the final outline after human confirmation. The present application realizes significant improvement in the efficiency and coverage of the evaluation outline writing through human-machine hierarchical collaboration, and ensures that the generated content is consistent with the original requirements.

Description

Technical Field

[0001] This invention belongs to the field of software testing technology, and in particular relates to a method, apparatus, computer-readable storage medium, and program product for generating software evaluation outlines through layered collaboration between humans and AI. Background Technology

[0002] The software testing outline is the core foundational document for software testing. It details the objects, methods, requirements, content, and pass / fail criteria for software testing, serving as a guiding document for the smooth conduct of the entire software testing activity. The quality of the testing outline directly affects the sufficiency and effectiveness of the software testing, and consequently, the accuracy and reliability of the quality evaluation conclusions for the tested software.

[0003] Traditional software testing outline writing relies heavily on the professional knowledge and practical experience of testers, involving manual analysis of software requirements documents, item-by-item test item design, manual writing of test methods, and application of criteria. This approach results in long writing cycles and high labor costs; the quality is highly dependent on the individual abilities of testers, making it difficult to guarantee consistent quality; and the traditional manual writing method is no longer efficient enough to meet the timeliness requirements of agile development and continuous delivery for testing activities. These problems make traditional software testing outline writing methods increasingly inadequate for meeting the current demands for efficiency, comprehensiveness, and high quality in software evaluation.

[0004] In recent years, the rapid development of large language model technology has provided a new technical path for the intelligent transformation of the software testing field. Researchers have explored applying large language models to tasks such as test case generation and test script writing, and intelligent software testing has become a key practical direction for ensuring software quality and accelerating software iteration cycles.

[0005] However, while existing technologies, through the introduction of large language models, have improved the automation level of software test outline and test case generation to some extent, and have increased efficiency compared to the traditional manual coding method, the following significant shortcomings still exist: First, there is a lack of deep understanding of the overall business context and logic of the software under test. Existing methods input the software requirements document, either as a whole or in segments, into a large language model, which then directly generates test content. However, large language models have inherent limitations in their perception of context. Limited by the context window length of the attention mechanism, the model struggles to fully process the entire content of a large software requirements document, let alone retain the context information it has already processed. This makes it difficult for the model to accurately grasp the implicit relationships between the overall business context, system architecture, and requirements of the software under test, resulting in the final test outline or test cases easily missing some test points and posing a risk of incomplete test coverage.

[0006] Second, the end-to-end automated process lacks human oversight of the high-level test logic. Existing methods completely delegate the generation process to the model, without setting up interactive nodes for human review and optimization of the high-level test logic. This results in expert experience not being effectively integrated into the test outline generation process, and the generated test outlines failing to reach a professional level in terms of the rationality of test strategies and the rationality of test item division.

[0007] Third, knowledge updates are costly and time-consuming. Generating knowledge by pre-training or fine-tuning large language models requires retraining the model each time the knowledge base is updated or adapted to new testing standards. This consumes significant computing resources, results in long update cycles, and prevents real-time updates and continuous accumulation of knowledge. Consequently, the system struggles to adapt to the diverse testing needs of different industries and projects, limiting its flexibility for widespread application. Summary of the Invention

[0008] Based on the above analysis, the present invention aims to provide a method, apparatus, computer-readable storage medium, and program product for generating software evaluation outlines through layered collaboration between humans and AI, in order to solve the problems in existing software evaluation outline generation, such as incomplete test coverage due to AI's contextual limitations, low efficiency and unstable quality of purely manual writing, lack of high-level human control over end-to-end automation, and high cost of retraining models for knowledge updates.

[0009] In a first aspect, embodiments of the present invention provide a method for generating a software evaluation outline through layered collaboration between humans and AI, the method comprising the following steps: Obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data; Several test items are obtained from the parsed structured data to obtain a software evaluation outline framework with several test items; In response to manual editing of the software evaluation outline framework, the software evaluation outline framework is optimized and adjusted to form a complete software evaluation outline framework; Based on the test items, the test content corresponding to each test item in the complete software evaluation outline framework is generated using a large language model; The test content is input into the same large language model to reverse-engineer the corresponding software requirements. The semantic similarity between the restored software requirements and the original software requirements is calculated. Based on the semantic similarity, it is determined whether manual intervention is required. If manual intervention is required, the marked test content is modified or confirmed in response to the manual intervention editing operation. In response to the final manual confirmation, the final software evaluation outline is output in a standard format.

[0010] Based on the above solution, the present invention also makes the following improvements: Furthermore, the step of obtaining several test items from the parsed structured data to obtain a software evaluation outline framework with several test items includes: From the parsed structured data, the test type to which each sentence in each chapter of the software requirements document belongs is identified; based on the test type to which each sentence belongs, software requirements belonging to the same test type in the same chapter are merged into one test item, thereby obtaining a software evaluation outline framework with several test items; the test types include functional testing, performance testing, and security testing.

[0011] Furthermore, identifying the test type to which each sentence in each chapter of the software requirements document belongs includes: A feature vocabulary library is pre-built for each test type, and the feature vocabulary library contains feature words and phrases corresponding to that test type; Using a sentence segmenter from a natural language processing library, the parsed continuous text of each chapter is segmented into independent sentences, and the chapter title and number information of each sentence in the document are recorded. For each sentence, the sentence is broken down into individual words, and the TF-IDF matching degree of each word with the words in the feature word library for each test type is calculated. The matching degrees of each word are then collected in order to form a feature vector. At the same time, the semantic vector of the sentence is obtained using the BERT model. The feature vector and the semantic vector are concatenated and then passed through a fully connected neural network layer for multi-label classification to determine the test type to which each sentence belongs.

[0012] Furthermore, the semantic similarity between the calculated and restored software requirements and the original software requirements includes: Original software requirements text and the required text output by the reverse restoration module A unified preprocessing process is performed, which includes case normalization, punctuation standardization, number and unit formatting, and special symbol protection. The preprocessed data is embedded using the Sentence-BERT semantic embedding model. and Encode them into semantic vectors respectively and ; Calculation using cosine similarity and Semantic similarity between them.

[0013] Furthermore, the step of determining whether manual intervention is needed based on the semantic similarity includes: If the semantic similarity is greater than or equal to the second preset threshold, the generated content is directly adopted. If the first preset threshold ≤ the semantic similarity < the second preset threshold, fill in the outline and mark it as a suggestion for manual review; If the semantic similarity is less than the first preset threshold, it is marked as requiring manual review and modification.

[0014] Furthermore, the step of generating test content corresponding to each test item in the complete software evaluation outline framework using a large language model based on the test items includes: Based on the professional field corresponding to the software requirements document, determine the corresponding test requirements knowledge base and prompt words; based on the test type of the test item, retrieve knowledge entries related to the test type from the test requirements knowledge base; for each test item, input the software requirements of the test item, the prompt words, and the knowledge entries into the large language model to generate the corresponding test method, test item description, and pass criteria; fill the test method, test item description, and pass criteria into the position corresponding to the test item in the software evaluation outline framework to form the test content corresponding to the test item in the software evaluation outline framework.

[0015] Furthermore, the optimization and adjustment of the software evaluation outline framework includes: Manually add, delete, modify, and query all test items; add test items that the system can automatically identify, delete unnecessary test items, modify the names or hierarchical relationships of test items, and adjust the logical order between test items.

[0016] In a second aspect, embodiments of the present invention provide a software evaluation outline generation device for hierarchical collaboration between humans and AI, wherein the device is used to implement at least one of the methods described in the first aspect of the present invention, the device comprising: The parsing and domain segmentation module is used to obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data; The framework construction module is used to obtain several test items from the parsed structured data to obtain a software evaluation outline framework with several test items. The framework optimization module is used to optimize and adjust the software evaluation outline framework in response to manual editing operations, so as to form a complete software evaluation outline framework. The content generation module is used to generate test content corresponding to each test item in the complete software evaluation outline framework based on the test items and using a large language model. The verification module is used to input the test content into the same large language model to reverse reconstruct the corresponding software requirements, calculate the semantic similarity between the reconstructed software requirements and the original software requirements, and determine whether manual intervention is required based on the semantic similarity. If manual intervention is required, the module responds to the editing operation after manual intervention by modifying or confirming the marked test content. The output module is used to respond to the final confirmation operation by outputting the final software evaluation outline in a standard format.

[0017] Thirdly, embodiments of the present invention provide a computer-readable storage medium for generating a software evaluation outline through layered collaboration between humans and AI. The computer-readable storage medium stores a computer program, which, when executed, implements at least one of the methods described in the first aspect of the present invention.

[0018] Fourthly, embodiments of the present invention provide a computer program product for generating a software evaluation outline through layered collaboration between humans and AI. The program product includes a stored computer program. When the computer program is run by a processor, the computer program code instructions cause the program product to perform at least one of the methods described in the first aspect of the present invention.

[0019] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. Improve test coverage and reduce test point omissions. This invention employs a two-level classification mechanism of "coarse classification (professional domain) + fine classification (test type)" to systematically traverse every sentence in the software requirements document and determine the test type based on the context of the chapter, effectively avoiding the omission of test points due to experience differences or oversights during manual writing. Simultaneously, by manually reviewing and optimizing test items, it fully leverages the advantages of testing experts' understanding of the overall business background and implicit requirements of the software, compensating for the limitations of large language models in context awareness. This effectively solves the dual problems of overall logical deficiencies caused by "AI-driven all-inclusive" approaches and the tendency of "purely manual" approaches to overlook boundary scenarios, significantly improving the coverage of the software evaluation outline.

[0020] 2. Ensure the generated content matches the original requirements. This invention introduces an adversarial verification process. It takes test content generated by a large language model as input and requires the same model to reverse-engineer the original software requirements. By calculating the semantic similarity between the reconstructed requirements and the original requirements, it automatically identifies deviations between the generated content and the original requirements. Items with low similarity are marked as requiring manual intervention, ensuring that the generated content faithfully reflects the original requirements. This effectively avoids potential deviations in requirements or comprehension during AI generation, further guaranteeing the accuracy and reliability of the evaluation outline.

[0021] 3. Improve writing efficiency and reduce labor costs. This invention automates most of the test point discovery and test method writing through AI, requiring testers to only review and optimize key points, significantly reducing the time spent on manual, line-by-line writing. Compared to the traditional fully manual writing model, it effectively shortens the test outline writing cycle and significantly reduces labor costs.

[0022] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0023] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0024] Figure 1 A flowchart illustrating a method for generating a software evaluation outline based on hierarchical collaboration between humans and AI, as provided in an embodiment of the present invention. Detailed Implementation

[0025] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0026] Example 1 This embodiment discloses a method for generating software evaluation outlines through layered collaboration between humans and AI. The method flow is as follows: Figure 1 As shown, it includes the following steps: Step S1: Obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data.

[0027] In the specific implementation process, the software requirements document (QR document) of the software under test is obtained. The QR document format includes, but is not limited to, .doc, .docx, and .pdf formats. For .doc and .docx documents, the Apache POI library is used for parsing; for .pdf documents, the PDFBox library is used. During parsing, text, titles, paragraphs, chapter structure, and text content within each chapter are extracted, establishing a one-to-one correspondence between chapter names and text content. Simultaneously, the table content in the document is parsed, including copying and pasting table content and extracting data row by row. Images and formulas are extracted from the document and placed in their corresponding positions in the subsequently generated outline. The parsed structured data is stored in JSON format, containing a chapter directory tree and structured text, tables, and image data, specifically including text content, table data, and image / formula references under each node in the chapter directory tree.

[0028] Furthermore, based on preset professional field classification rules, the professional field corresponding to the software requirement document is determined; wherein, the preset professional field classification rules include the pre-defined professional field to which the software requirement document belongs. The corresponding workflow branch and knowledge base are loaded according to the determined professional field, which includes, but is not limited to: aerospace measurement and control, financial settlement, industrial control, rail transportation, and communications.

[0029] Step S2: Obtain several test items from the parsed structured data to obtain a software evaluation outline framework with several test items.

[0030] Specifically, several test items are obtained from the parsed structured data to obtain a software evaluation outline framework with several test items, including: From the parsed structured data, identify the test type to which each sentence in each chapter of the software requirements document belongs; based on the test type to which each sentence belongs, merge software requirements belonging to the same test type in the same chapter into one test item, thereby obtaining a software evaluation outline framework with several test items; the test types include functional testing, performance testing, and security testing.

[0031] Specifically, identifying the test type to which each sentence in each chapter of the software requirements document belongs includes: Step S21: Build a feature vocabulary library in advance for each test type. The feature vocabulary library contains the feature words and phrases corresponding to that test type.

[0032] For example, the feature word library for functional testing includes one or more of the following: should be able to, must, support, implement, allow, if, when, verify, check; the feature word library for performance testing includes one or more of the following: response time, throughput, concurrency, transactions per second, CPU usage, memory usage, milliseconds, within seconds; the feature word library for security testing includes one or more of the following: password, encryption, decryption, authorization, permission, authentication, SQL injection, XSS, vulnerability, sensitive data.

[0033] Step S22: Using a sentence segmenter from a natural language processing library, the parsed continuous text of each chapter is segmented into independent sentences, and the chapter title and number information of each sentence in the document are recorded.

[0034] Furthermore, using sentence segmenters from natural language processing libraries such as NLTK or Spacy, the parsed continuous text of each chapter is segmented into independent sentences, and the chapter title and number information of each sentence in the document are recorded. For example, for the sentence "The system should complete the user login operation within 3 seconds" under the section "3.2.1 User Login Performance Requirements" in the software requirements document, the system records the section number of this sentence as "3.2.1" and the section title as "User Login Performance Requirements".

[0035] Step S23: For each sentence, break it down into individual words, calculate the TF-IDF matching degree between each word and the words in the feature word library for each test type, and collect the matching degrees of each word in order to form a feature vector; at the same time, use the BERT model to obtain the semantic vector of the sentence; after concatenating the feature vector and the semantic vector, perform multi-label classification through a fully connected neural network layer to determine the test type to which each sentence belongs.

[0036] Specifically, for each sentence, the sentence is broken down into individual words, the TF-IDF matching degree of each word is calculated with the words in the feature word library for each test type, and the matching degrees of each word are collected in order to form a feature vector.

[0037] All words in the feature word library of all test types are merged and deduplicated to form a total dictionary with a size of N. The position of each word is used as a fixed index of the feature vector. It is then determined whether the words extracted from the current sentence are included in the total dictionary: if they are included, the TF-IDF matching degree between the word and the corresponding word in the total dictionary is calculated and filled into the corresponding position in the vector; if they are not included, 0 is filled into the corresponding position. Finally, a numerical vector of length N composed of TF-IDF values ​​is obtained, which is the feature vector of the sentence.

[0038] For example, the sentence "The system should complete the user login operation within 3 seconds" is broken down into individual words, resulting in "system", "should", "within 3 seconds", "complete", "user", "login", and "operation". Each of these words is then checked to see if it is included in the general dictionary.

[0039] Furthermore, the semantic vector of the sentence is obtained simultaneously using the BERT model; the current sentence is input into the pre-trained BERT model, and the vector corresponding to the [CLS] label of the BERT model output layer is taken as the semantic vector of the sentence, with a dimension of 768.

[0040] Furthermore, the feature vector and the semantic vector are concatenated and then subjected to multi-label classification through a fully connected neural network layer to determine the test type to which each sentence belongs.

[0041] The two vectors are concatenated end to end to obtain a concatenated vector with dimension N+768. The concatenated vector is then input into a fully connected neural network classification layer. The Softmax function outputs the probability distribution of the sentence belonging to functional testing, performance testing, and security testing. The test type with the highest probability is taken as the fine classification result of the sentence.

[0042] For example, for the sentence "The system should complete the user login operation within 3 seconds.", the probability of the classification result being "performance test" is 0.85, the probability of "functional test" is 0.10, and the probability of "security test" is 0.05. Therefore, the sentence is classified as performance test.

[0043] Furthermore, based on the test type to which each sentence belongs, software requirements belonging to the same test type in the same chapter are merged into one test item, thus obtaining a software evaluation outline framework with several test items.

[0044] Test items should be named using the format "section name + test type". For example, if section 3.2.1 of the software requirements document is titled "User Login Performance Requirements" and categorized as performance testing, then the corresponding test item name would be "User Login Performance Requirements Performance Test".

[0045] For example, a software evaluation outline framework with several test items is as follows: 6. [Project Name] Test Item 6.1 Functional Testing 6.1.1 Functional Testing of [Requirement Module A] 6.1.2 Functional Testing of [Requirement Module B] 6.2 Performance Testing 6.2.1 [Scenario 1] Performance Testing 6.2.2 [Scenario 2] Performance Testing 6.3 Security Testing Step S3: In response to the manual editing operation of the software evaluation outline framework, optimize and adjust the software evaluation outline framework to form a complete software evaluation outline framework.

[0046] In practice, software testing experts optimize the software evaluation outline framework through interactive operations such as adding, deleting, modifying, and querying, based on their understanding of the business background and overall logic of the software under test. Specifically, this includes adding features such as automatic identification of missing test items, deleting unnecessary test items, modifying the names or hierarchical relationships of test items, and adjusting the logical order between test items.

[0047] It should be noted that software testing experts can also select or deselect specific software requirement document sections associated with a particular test item to ensure the accuracy of the tracking relationship between test items and software requirements, and mark the selected software requirement sections with a color.

[0048] Furthermore, after optimization, the system generates a complete software evaluation outline framework based on the final determined framework, including chapters such as introduction, test scope, test strategy, test resources and environment, test execution plan, test items and methods, entry / exit criteria, risks and contingency plans.

[0049] Step S4: Based on the test items, use the large language model to generate the test content corresponding to each test item in the complete software evaluation outline framework.

[0050] In the specific implementation process, based on the professional field corresponding to the software requirements document, the corresponding test requirements knowledge base and prompt words are determined; based on the test type of the test item, knowledge entries related to the test type are retrieved from the test requirements knowledge base; for each test item, the software requirements of the test item, the prompt words, and the knowledge entries are input into the large language model to generate the corresponding test method, test item description, and pass criteria; the test method, test item description, and pass criteria are filled into the position corresponding to the test item in the software evaluation outline framework to form the test content corresponding to the test item in the software evaluation outline framework.

[0051] Specifically, each knowledge base entry includes a software requirement description, a corresponding test method, a test item description, and pass criteria.

[0052] For example, in the "Aerospace Measurement and Control Field - Performance Testing" knowledge base, one knowledge record is as follows: Software requirement description: "The system should complete telemetry data parsing within 3 seconds." Test method: "Use a simulator to send 100 sets of telemetry data frames and record the parsing completion time"; Test item description: "Verify the real-time performance of telemetry data parsing"; The guiding principle is: "95% of data frames are parsed within 3 seconds."

[0053] Furthermore, based on the determined professional field, a corresponding domain-specific prompt word template is loaded, and based on the determined test type, type-specific constraints are added to the domain-specific prompt word template to form the final prompt word instruction.

[0054] Example of a general prompt word template for the aerospace telemetry and control field: "You are a senior software testing expert in the XXX field. Please generate the corresponding test methods, test item descriptions, and pass / fail criteria based on the following software requirements description. Requirements: XXX." Example of type-specific constraints (performance test): "Note: This is a performance testing requirement. Please focus on XXX." Final prompt word instruction example: "You are a senior software testing expert in the aerospace telemetry and control field. Based on the following software requirements description, please generate the corresponding test methods, test item descriptions, and pass criteria. Requirements: Test methods should be specific, operable, and conform to aerospace software testing standards; test item descriptions should be concise and clear; pass criteria should be quantifiable and verifiable."

[0055] Note: This is a performance testing requirement. Please focus on response time thresholds, throughput requirements, resource utilization limits, and concurrent user scenario design. Furthermore, the determined prompts, knowledge items, and software requirements for the test items are input into the large language model to generate corresponding test methods, test item descriptions, and pass criteria. These are then filled into the corresponding position of the test item in the software evaluation outline framework, forming the test content corresponding to the test item in the software evaluation outline framework.

[0056] For example, here is a sample of the test content: Requirement description: The system should complete the parsing of telemetry data within 3 seconds.

[0057] The generated test method is as follows: Use a simulator to send 100 sets of standard format telemetry data frames, record the entire process time from receipt to parsing, repeat the process 10 times and take the average value.

[0058] The generated test item description is: to verify the real-time performance of telemetry data parsing.

[0059] The generated pass criteria are: 95% of the data frames are parsed within 3 seconds, with no parsing errors.

[0060] Step S5: Input the test content into the same large language model to reverse reconstruct the corresponding software requirements, calculate the semantic similarity between the reconstructed software requirements and the original software requirements, and determine whether manual intervention is required based on the semantic similarity; if manual intervention is required, modify or confirm the marked test content in response to the manual intervention editing operation.

[0061] In practice, for each generated test item, the system uses the test content as input to construct reverse prompts. These reverse prompts are then input into the same large language model to obtain the reconstructed requirement text. .

[0062] For example, the reverse suggestion words are as follows: "You are a requirements analysis expert. Below is a section of test content from a software testing outline, including test methods, test item descriptions, and pass / fail criteria. Based on this content, please reconstruct the original software requirements. The reconstructed requirements should be complete and accurate, and should use the original requirements' expression as much as possible."

[0063] Test content: Test method: {Generated test method} Test item description: {Generated test item description} Pass criteria: {Generated pass criteria} Please output the restored required text directly, without adding any additional explanations. Furthermore, the original software requirements text and the required text output by the reverse restoration module Unified preprocessing is performed, including case normalization, punctuation standardization, number and unit formatting, and protection of special symbols.

[0064] The preprocessed data is embedded using the Sentence-BERT semantic embedding model. and Encode them into semantic vectors respectively and ; Calculation using cosine similarity and semantic similarity between The calculation formula is as follows:

[0065] In the formula, This represents the i-th component in the semantic vector of the original software requirements text. This represents the i-th component in the semantic vector of the demand text output by the reverse reconstruction module. Indicates the total number of components.

[0066] Furthermore, based on the semantic similarity, it is determined whether manual intervention is required; if manual intervention is required, the marked test content is modified or confirmed in response to the manual intervention editing operation.

[0067] The determination of whether manual intervention is needed based on the semantic similarity includes: If the semantic similarity is greater than or equal to the second preset threshold, the generated content is directly adopted. If the first preset threshold ≤ the semantic similarity < the second preset threshold, fill in the outline and mark it as a suggestion for manual review; If the semantic similarity is less than the first preset threshold, it is marked as requiring manual review and modification.

[0068] Preferably, the first preset threshold is set to 0.7 and the second preset threshold is set to 0.85.

[0069] Furthermore, after each human intervention, the system records the original requirement text, the generated content text, the restored requirement text, semantic similarity, and the human's final decision (whether to modify, and what content was modified). The system periodically (e.g., weekly) uses the feedback data recorded in the system to retrain or optimize the selection model for the first and second preset thresholds, or to update the entity extraction rules, to achieve continuous improvement.

[0070] Step S6: In response to the final confirmation operation by a human, output the final software evaluation outline in a standard format.

[0071] In the specific implementation process, the system responds to the final confirmation operation of the software testing experts and outputs the complete software evaluation outline in a preset standard format (such as a Word document or PDF document).

[0072] The output software evaluation outline includes: introduction, overall testing requirements, test preparation, test items (including test requirement description, test method, test item description, and pass criteria for each test item), test execution requirements, test evaluation criteria, appendix, and other complete chapters.

[0073] Example 2 This embodiment discloses a software evaluation outline generation device for hierarchical collaboration between humans and AI. The device is used to implement the method described in any one of the specific embodiments of this invention. The device includes: The parsing and domain segmentation module is used to obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data; The framework construction module is used to obtain several test items from the parsed structured data to obtain a software evaluation outline framework with several test items. The framework optimization module is used to optimize and adjust the software evaluation outline framework in response to manual editing operations, so as to form a complete software evaluation outline framework. The content generation module is used to generate test content corresponding to each test item in the complete software evaluation outline framework based on the test items and using a large language model. The verification module is used to input the test content into the same large language model to reverse reconstruct the corresponding software requirements, calculate the semantic similarity between the reconstructed software requirements and the original software requirements, and determine whether manual intervention is required based on the semantic similarity. If manual intervention is required, the module responds to the editing operation after manual intervention by modifying or confirming the marked test content. The output module is used to respond to the final confirmation operation by outputting the final software evaluation outline in a standard format.

[0074] Example 3 This embodiment discloses a computer-readable storage medium for generating a software evaluation outline for hierarchical collaboration between humans and AI. The computer-readable storage medium stores a computer program, and when the computer program is executed, it implements the method described in any one of the specific embodiments of the present invention.

[0075] Example 4 This embodiment discloses a program product for generating a software evaluation outline through hierarchical collaboration between humans and AI, including program code instructions. When the program product is executed by a computing device, the program code instructions cause the computing device to execute the method described in any one of the specific embodiments of the present invention, 1.

[0076] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating software evaluation outlines through layered collaboration between humans and AI, characterized in that, The method includes: Obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data; Several test items are obtained from the parsed structured data to obtain a software evaluation outline framework with several test items; In response to manual editing of the software evaluation outline framework, the software evaluation outline framework is optimized and adjusted to form a complete software evaluation outline framework; Based on the test items, the test content corresponding to each test item in the complete software evaluation outline framework is generated using a large language model; The test content is input into the same large language model to reverse-engineer the corresponding software requirements. The semantic similarity between the restored software requirements and the original software requirements is calculated. Based on the semantic similarity, it is determined whether manual intervention is required. If manual intervention is required, the marked test content is modified or confirmed in response to the manual intervention editing operation. In response to the final manual confirmation, the final software evaluation outline is output in a standard format.

2. As described in claim 1, characterized in that, The process of extracting several test items from the parsed structured data to obtain a software evaluation outline framework with several test items includes: From the parsed structured data, the test type to which each sentence in each chapter of the software requirements document belongs is identified; based on the test type to which each sentence belongs, software requirements belonging to the same test type in the same chapter are merged into one test item, thereby obtaining a software evaluation outline framework with several test items; the test types include functional testing, performance testing, and security testing.

3. As described in claim 2, characterized in that, The identification of the test type to which each sentence in each chapter of the software requirements document belongs includes: A feature vocabulary library is pre-built for each test type, and the feature vocabulary library contains feature words and phrases corresponding to that test type; Using a sentence segmenter from a natural language processing library, the parsed continuous text of each chapter is segmented into independent sentences, and the chapter title and number information of each sentence in the document are recorded. For each sentence, the sentence is broken down into individual words, and the TF-IDF matching degree of each word with the words in the feature word library for each test type is calculated. The matching degrees of each word are then collected in order to form a feature vector. At the same time, the semantic vector of the sentence is obtained using the BERT model. The feature vector and the semantic vector are concatenated and then passed through a fully connected neural network layer for multi-label classification to determine the test type to which each sentence belongs.

4. As described in claim 1, characterized in that, The semantic similarity between the calculated and restored software requirements and the original software requirements includes: Original software requirements text and the required text output by the reverse restoration module A unified preprocessing process is performed, which includes case normalization, punctuation standardization, number and unit formatting, and special symbol protection. The preprocessed data is embedded using the Sentence-BERT semantic embedding model. and Encode them into semantic vectors respectively and ; Calculation using cosine similarity and Semantic similarity between them.

5. As described in claim 4, characterized in that, The step of determining whether manual intervention is needed based on the semantic similarity includes: If the semantic similarity is greater than or equal to the second preset threshold, the generated content is directly adopted. If the first preset threshold ≤ the semantic similarity < the second preset threshold, fill in the outline and mark it as a suggestion for manual review; If the semantic similarity is less than the first preset threshold, it is marked as requiring manual review and modification.

6. As described in claim 1, characterized in that, The step of generating test content corresponding to each test item in the complete software evaluation outline framework using a large language model based on the test items includes: Based on the professional field corresponding to the software requirements document, determine the corresponding test requirements knowledge base and prompt words; based on the test type of the test item, retrieve knowledge entries related to the test type from the test requirements knowledge base; for each test item, input the software requirements of the test item, the prompt words, and the knowledge entries into the large language model to generate the corresponding test method, test item description, and pass criteria; fill the test method, test item description, and pass criteria into the position corresponding to the test item in the software evaluation outline framework to form the test content corresponding to the test item in the software evaluation outline framework.

7. As described in claim 1, characterized in that, The optimization and adjustment of the software evaluation outline framework includes: Manually add, delete, modify, and query all test items; add test items that the system can automatically identify, delete unnecessary test items, modify the names or hierarchical relationships of test items, and adjust the logical order between test items.

8. A software evaluation outline generation device for hierarchical collaboration between humans and AI, wherein the device implements the method according to any one of claims 1 to 7, characterized in that, The device includes: The parsing and domain segmentation module is used to obtain the software requirements document of the software under test and perform structured parsing to obtain the parsed structured data; The framework construction module is used to obtain several test items from the parsed structured data to obtain a software evaluation outline framework with several test items. The framework optimization module is used to optimize and adjust the software evaluation outline framework in response to manual editing operations, so as to form a complete software evaluation outline framework. The content generation module is used to generate test content corresponding to each test item in the complete software evaluation outline framework based on the test items and using a large language model. The verification module is used to input the test content into the same large language model to reverse reconstruct the corresponding software requirements, calculate the semantic similarity between the reconstructed software requirements and the original software requirements, and determine whether manual intervention is required based on the semantic similarity. If manual intervention is required, the module responds to the editing operation after manual intervention by modifying or confirming the marked test content. The output module is used to respond to the final confirmation operation by outputting the final software evaluation outline in a standard format.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed, implements the method of any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a stored computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images