Software testing method and device, storage medium and electronic device

Abstract

The invention discloses a software testing method and device, a storage medium and an electronic device, and relates to the technical field of software test.The software testing method comprises the steps that a requirement document and interface definition information of a target software function are obtained from a knowledge base; test information is generated according to the demand document and the interface definition information, the test information comprises a test case and an automatic test script, and the automatic test script is used for executing the test case; executing the automated test script in a test environment to test the target software function to obtain test data; and determining a test result of the target software function according to the test data and the test case. By adopting the technical scheme, the problem of how to improve the software testing efficiency is solved.

Description

Technical Field

[0001] This application relates to the field of software testing technology, and more specifically, to a software testing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Currently, in the field of software testing, traditional functional test case writing mainly relies on manual methods. Test engineers manually write test cases based on their experience and analysis of software requirements. This approach is not only inefficient, but also results in inconsistent test case quality among different engineers, making it difficult to guarantee scenario coverage and meet the needs of rapid software development iterations. Regarding API test script development, current technologies require testers to possess strong programming skills and relevant technical knowledge, manually writing code to implement API test scripts. This makes the testing work highly dependent on technical personnel, limiting the participation of non-technical personnel in the team and hindering the improvement of API automated test coverage. Therefore, there is a question regarding how to improve software testing efficiency in related technologies.

[0003] No effective solution has yet been proposed for improving software testing efficiency in related technologies. Summary of the Invention

[0004] This application provides a software testing method, apparatus, storage medium, and electronic device to at least address the problem of how to improve software testing efficiency in related technologies.

[0005] According to one embodiment of this application, a software testing method is provided, comprising: obtaining a requirement document and interface definition information of a target software function from a knowledge base; generating test information based on the requirement document and the interface definition information, wherein the test information includes test cases and automated test scripts, wherein the automated test scripts are used to execute the test cases; executing the automated test scripts in a test environment to test the target software function and obtain test data; and determining the test result of the target software function based on the test data and the test cases.

[0006] In an exemplary embodiment, before obtaining the requirement documents and interface definition information of the target software function from the knowledge base, the method further includes: performing structured parsing on data from different data sources to obtain structured data; determining the retrieval index of the structured data according to the software function corresponding to the structured data; performing semantic segmentation on the structured data to obtain multiple data blocks; performing metadata annotation on the multiple data blocks to obtain annotated data blocks; and constructing the knowledge base based on the annotated data blocks.

[0007] In one exemplary embodiment, generating test information based on the requirements document and the interface definition information includes: determining a target prompt word template corresponding to the type of the target software function from a prompt word template library; determining first supplementary content based on the interface definition information, determining second supplementary content based on the requirements document, and supplementing the target prompt word template based on the first supplementary content and the second supplementary content to obtain target prompt words, wherein the first supplementary content is used to determine the test interface, the second supplementary content is used to determine the test requirements corresponding to the test interface, and the target prompt words are used to instruct a large language model to generate test cases and automated test scripts that meet the test requirements for the test interface; and obtaining the test cases and automated test scripts generated by the large language model based on the target prompt words.

[0008] In one exemplary embodiment, generating test information based on the requirements document and the interface definition information includes: determining the requirements to be tested based on the requirements document; regenerating the test cases if it is determined that the test cases do not fully cover the requirements to be tested; and generating the automated test script based on the interface definition information and the test cases if it is determined that the test cases fully cover the requirements to be tested.

[0009] In one exemplary embodiment, determining the test result of the target software function based on the test data and the test cases includes: obtaining preset test data of the target software function from the test cases; determining the test result as passed if the preset test data is consistent with the test data; and determining the test result as failed if the preset test data is inconsistent with the test data.

[0010] In one exemplary embodiment, the method further includes: if it is determined that the target software includes an intelligent language interaction module, testing the intelligent language interaction module in the following manner: obtaining an evaluation dataset of input from a target object, wherein the evaluation dataset includes an input question and expected output content; testing the intelligent language interaction module according to the input question in the evaluation dataset to obtain the output content of the intelligent language interaction module for the input question; obtaining a language interaction test result generated by a large language model based on the evaluation dataset, evaluation rule prompts, and the output content, wherein the evaluation rule prompts are used to instruct the large language model to score the output content according to preset evaluation rules.

[0011] In an exemplary embodiment, obtaining the language interaction test results generated by the large language model based on the evaluation dataset, evaluation rule prompts, and the output content includes: determining the score calculated by the output content under the preset evaluation rules based on the comparison results between the expected output content in the evaluation dataset and the output content, thereby obtaining score values ​​in multiple dimensions, wherein the preset evaluation rules include multiple dimensions of scoring criteria; and determining that the test result of the intelligent language interaction module is passed when it is determined that the score value of each dimension is greater than a first preset score value and the sum of the score values ​​of the multiple dimensions is greater than a second preset score value.

[0012] According to another aspect of the embodiments of this application, a software testing apparatus is also provided, comprising: an acquisition module, configured to acquire requirement documents and interface definition information of a target software function from a knowledge base; a generation module, configured to generate test information based on the requirement documents and the interface definition information, wherein the test information includes test cases and automated test scripts, wherein the automated test scripts are used to execute the test cases; a testing module, configured to execute the automated test scripts in a test environment to test the target software function and obtain test data; and a determination module, configured to determine the test result of the target software function based on the test data and the test cases.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored in the computer-readable storage medium, and the computer program is configured to execute the above-described software testing method at runtime.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-described software testing method through the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0016] In this embodiment, the requirement documents and interface definitions of the target software function are first extracted from a knowledge base. Based on these requirements documents and interface definitions, test information, including test cases and automated test scripts, is generated, whereby the automated test scripts are used to execute the test cases. The automated test scripts are then run in a test environment to test the target software function and obtain test data. Finally, the test results of the target software function are determined based on the test data and test cases. By adopting the above scheme, the problem of how to improve software testing efficiency is solved, thereby achieving the effect of improving software testing efficiency. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the hardware environment for a software testing method according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a software testing method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram (a) of a software testing method according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram (II) of a software testing method according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram (iii) of a software testing method according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram (four) of a software testing method according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram (V) of a software testing method according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of an anomaly detection model according to an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] According to one aspect of the embodiments of this application, a software testing method is provided. This software testing method is widely used in software development and other application scenarios. Optionally, in this embodiment, the above-described software testing method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal device 102 and server 104. For example... Figure 1 As shown, server 104 is connected to terminal device 102 via a network and can be used to provide services (such as application services) to the terminal or clients installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data processing services for server 104.

[0030] The aforementioned network may include, but is not limited to, at least one of the following: wired network, wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity), Bluetooth. The terminal device 102 may not be limited to PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projector, smart TV, smart clothes rack, smart curtains, smart audio-visual equipment, smart socket, smart speaker, smart speaker box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart robot vacuum cleaner, smart window cleaning robot, smart mopping robot, smart air purifier, smart steam oven, smart microwave oven, smart water heater, smart air purifier, smart water dispenser, smart door lock, etc.

[0031] This embodiment provides a software testing method. Figure 2 This is a flowchart of a software testing method according to an embodiment of this application, which includes the following steps:

[0032] Step S202: Obtain the requirements document and interface definition information of the target software functions from the knowledge base;

[0033] Step S204: Generate test information based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, and the automated test scripts are used to execute the test cases;

[0034] Step S206: Execute the automated test script in the test environment to test the target software function and obtain test data;

[0035] Step S208: Determine the test results of the target software function based on the test data and the test cases.

[0036] The above steps first extract the requirement documents and interface definitions for the target software functionality from the knowledge base. Then, based on these requirements documents and interface definitions, test information, including test cases and automated test scripts, is generated, with the automated test scripts used to execute the test cases. Next, the automated test scripts are run in the test environment to test the target software functionality and obtain test data. Finally, the test results for the target software functionality are determined based on the test data and test cases. By adopting this approach, the problem of how to improve software testing efficiency is solved, thereby achieving the effect of improving software testing efficiency.

[0037] In an exemplary embodiment, before obtaining the requirement documents and interface definition information of the target software function from the knowledge base, the method further includes: performing structured parsing on data from different data sources to obtain structured data; determining the retrieval index of the structured data according to the software function corresponding to the structured data; performing semantic segmentation on the structured data to obtain multiple data blocks; performing metadata annotation on the multiple data blocks to obtain annotated data blocks; and constructing the knowledge base based on the annotated data blocks.

[0038] Optionally, in the above embodiments, the structured parsing of data from different data sources includes: accessing multiple types of data sources, such as requirements specifications (Word documents or Excel spreadsheets), technical design documents, API interface documents (such as Swagger / OpenAPI format), database table structures (SQL scripts), UI prototype diagrams (such as Pixso annotation information), historical test case libraries (Excel spreadsheets), meeting minutes (Markdown or plain text format), etc.

[0039] Structured parsing of data from different data sources involves using specific parsing tools to process input documents in various formats. For example, python-docx (a Python document processing library) is used to parse heading levels and tables in Word documents; PyPDF2 (a Python PDF processing library) is used to extract PDF text; html2text (an HTML-to-text tool) is used to process HTML-formatted interface documents; and OCR tools are used to extract text information from UI prototypes. Data cleaning and standardization are key steps in this stage, including removing duplicate paragraphs based on the SimHash algorithm and standardizing terminology according to the enterprise's business dictionary (e.g., unifying "user ID" as user_id).

[0040] Determining the retrieval index involves creating an index for the data according to a three-level classification system of "business domain - module - function point," such as "e-commerce - order module - payment interface," to improve subsequent retrieval efficiency. Semantic segmentation processing uses recursive character text segmentation tools such as LangChain (a framework for developing language model-driven applications) to divide long texts into blocks of 200-500 characters based on punctuation and line breaks, while merging logically related paragraphs by calculating semantic similarity (e.g., cosine similarity > 0.7). Metadata annotation adds tags to each data block, including source document, business module, element type (text, table, list), and priority (extracted from requirements: "high / medium / low"). Finally, these annotated data blocks are stored in a vector database, thus constructing a knowledge base that supports efficient semantic retrieval.

[0041] Optionally, in the above embodiments, the structured data undergoes semantic segmentation to obtain multiple data blocks, including basic segmentation and enhanced segmentation. Basic segmentation uses LangChai's RecursiveCharacterTextSplitter to initially segment the data based on punctuation and line breaks, with each block length controlled between 200-500 characters (to adapt to large model context windows). Enhanced segmentation preserves logical connections and separates blocks based on special elements. Logical connections preserve continuous requirement descriptions across paragraphs (such as "prerequisites" and "execution steps"). Semantic similarity is calculated; blocks with similarity greater than a preset value (e.g., cosine similarity > 0.7) are merged into one block. Semantic element-based independent segmentation treats tables, lists, and image OCR results as separate blocks, labeled with type tags (e.g., [TABLE] - payment parameters).

[0042] In one exemplary embodiment, generating test information based on the requirements document and the interface definition information includes: determining a target prompt word template corresponding to the type of the target software function from a prompt word template library; determining first supplementary content based on the interface definition information, determining second supplementary content based on the requirements document, and supplementing the target prompt word template based on the first supplementary content and the second supplementary content to obtain target prompt words, wherein the first supplementary content is used to determine the test interface, the second supplementary content is used to determine the test requirements corresponding to the test interface, and the target prompt words are used to instruct a large language model to generate test cases and automated test scripts that meet the test requirements for the test interface; and obtaining the test cases and automated test scripts generated by the large language model based on the target prompt words.

[0043] Optionally, in the above embodiments, the prompt word template library is a collection of pre-built scenario-based templates, such as "interface test template" and "UI test template". These templates define dynamically populated fields using variable placeholders in the form of {variable name}. The target prompt word template is automatically selected based on the type of the target software function (such as "interface test" or "UI process test"). The first supplementary content usually comes from interface definition information, such as the interface path, request method, and parameter list parsed from the Swagger document. The second supplementary content comes from the relevant functional descriptions, business rules, and constraints in the requirements document. The process of obtaining the target prompt word is to fill these specific contents into the placeholders of the template. For example, an interface test template might be: "Based on the following interface information (request method: {method}, parameters: {params}), this method is used for {business function description}, generate test cases containing normal requests, missing parameters, type errors, and boundary values." The system will replace {method} with "POST", {params} with the parsed parameter list, and {business function description} with the parsed business rules, thus forming a complete and clear instruction (i.e., target prompt) and sending it to the large language model.

[0044] In addition, this solution also includes a collaborative optimization mechanism, such as comparing the “coverage” and “effectiveness” of test cases generated by different templates through A / B testing (a comparative testing method), thereby continuously optimizing the template library.

[0045] In one exemplary embodiment, generating test information based on the requirements document and the interface definition information includes: determining the requirements to be tested based on the requirements document; regenerating the test cases if it is determined that the test cases do not fully cover the requirements to be tested; and generating the automated test script based on the interface definition information and the test cases if it is determined that the test cases fully cover the requirements to be tested.

[0046] Optionally, in the above embodiments, determining the requirements to be tested based on the requirements document means extracting all functional points that need to be verified from the structured requirements data blocks. Determining incomplete test case coverage is achieved through automated verification, for example, through keyword matching analysis (checking whether the generated test cases cover all requirements, with a matching rate >80%), or through consistency checks (ensuring that parameter names and operation steps in the test cases are consistent with the original document). If verification finds uncovered requirements or missing test cases, the system triggers a "regeneration" process. This process includes: using the missing requirement information as a new context to re-invoke the large language model; only after confirming that the test cases fully cover the requirements to be tested will the system enter the stage of generating automated test scripts. This stage will again utilize the large language model, along with a Prompt template customized for mainstream frameworks such as pytest (a Python testing framework), to convert test cases described in natural language into executable code scripts and automatically inject smart assertions (statements used to verify program execution results).

[0047] In one exemplary embodiment, determining the test result of the target software function based on the test data and the test cases includes: obtaining preset test data of the target software function from the test cases; determining the test result as passed if the preset test data is consistent with the test data; and determining the test result as failed if the preset test data is inconsistent with the test data.

[0048] Optionally, in the above embodiments, the preset test data refers to the expected results defined in the test cases. These expected results are the behavior the software should exhibit when working correctly, explicitly set during the test case generation phase. Test data, on the other hand, refers to the actual system behavior and data captured after executing the automated test script in the test environment. Consistency assessment is a core function of the automated testing framework. For interface testing, this might mean comparing the expected HTTP status codes, response body JSON structure, and key field values ​​to see if they completely match the actual response. For UI testing, it might involve checking whether expected page elements, text content, or page redirects occur. For example, if the preset test data for a test case is that the page displays a welcome message for user A after successful login, and the test data captured after the automated script executes is "Welcome user A," then the system considers it a pass; if the actual text is empty or other text, then it is considered a fail. All inconsistent cases are automatically recorded and detailed failure reports are generated.

[0049] In one exemplary embodiment, the method further includes: if it is determined that the target software includes an intelligent language interaction module, testing the intelligent language interaction module in the following manner: obtaining an evaluation dataset of input from a target object, wherein the evaluation dataset includes an input question and expected output content; testing the intelligent language interaction module according to the input question in the evaluation dataset to obtain the output content of the intelligent language interaction module for the input question; obtaining a language interaction test result generated by a large language model based on the evaluation dataset, evaluation rule prompts, and the output content, wherein the evaluation rule prompts are used to instruct the large language model to score the output content according to preset evaluation rules.

[0050] Optionally, in the above embodiments, this embodiment expands the testing scope from traditional software functions to the quality assessment of AI agents (i.e., intelligent language interaction modules), constituting a complete "full-process" testing scheme. The evaluation dataset is the foundation of the assessment, typically organized in tabular form, containing "input questions" and "expected output content" (i.e., reference answers). For example, the input is "In what year was the Shenzhen Special Economic Zone established?", and the expected output is "1980". Testing the intelligent language interaction module refers to automatically and in batches submitting the input questions from the evaluation dataset to the AI ​​agent under test (such as an intelligent voice assistant) and collecting its actual output content. The evaluation rule prompt is a carefully designed instruction used to guide another large language model, acting as a "referee," in scoring. This prompt explicitly informs the referee model of the specific preset evaluation rules, which are multi-dimensional, including accuracy (whether the information is correct), relevance (whether the answer is irrelevant), harmlessness (whether it contains offensive or unethical content), fluency (whether the language is fluent), etc. The referee model will comprehensively evaluate each output of the agent according to these rules and generate language interaction test results, which usually include quantitative scores and detailed scoring reasons.

[0051] In an exemplary embodiment, obtaining the language interaction test results generated by the large language model based on the evaluation dataset, evaluation rule prompts, and the output content includes: determining the score calculated by the output content under the preset evaluation rules based on the comparison results between the expected output content in the evaluation dataset and the output content, thereby obtaining score values ​​in multiple dimensions, wherein the preset evaluation rules include multiple dimensions of scoring criteria; and determining that the test result of the intelligent language interaction module is passed when it is determined that the score value of each dimension is greater than a first preset score value and the sum of the score values ​​of the multiple dimensions is greater than a second preset score value.

[0052] Optionally, in the above embodiments, this embodiment provides quantifiable and comprehensive passing standards for the evaluation of AI agents, ensuring the objectivity and operability of the evaluation results. The multi-dimensional scoring values ​​refer to the scores given by the judging model for each evaluation rule (such as accuracy, relevance, and harmlessness), for example, using a 1-5 point system. The first preset scoring value is the minimum passing score set for each single dimension; this is a "one-vote veto" mechanism to ensure that the agent does not have serious deficiencies in any key dimension. For example, if the first preset scoring value is set to 3 points, then even if an answer is very fluent and relevant, if its accuracy dimension score is only 2 points (i.e., it contains a major factual error), the overall evaluation of the answer will be lowered. The second preset scoring value is the minimum requirement set for the total score across all dimensions, ensuring that the agent's overall performance reaches a good level. For example, if the second preset scoring value is set to 80% of the total score (in a 5-point system with three dimensions and a maximum score of 15 points, the second preset scoring value is 12 points). Determining a test result as passing means that both of the above conditions must be met simultaneously: neither the score in any single dimension is too low, nor the overall performance is too good. This dual standard ensures a more comprehensive and rigorous guarantee of the quality of AI agents upon deployment. Ultimately, the system generates a visual evaluation report, clearly displaying scores for each dimension and the total score, providing data support for optimization decisions.

[0053] To better understand the process of the above software testing method, the implementation flow of the above software testing method will be described below in conjunction with optional embodiments, but this is not intended to limit the technical solution of the embodiments of this application.

[0054] In an optional embodiment, Figure 3 This illustrates a test case generation process based on a large AI model, specifically including the following steps:

[0055] 1. AI identifies requirements and generates use cases.

[0056] Objective: To address the issues of efficiency and standardization in test case writing.

[0057] Process: Utilize AI to automatically identify and understand software requirements, generating initial test cases. Through a large language model and the Prompt project, automatically convert requirement functionalities into executable test cases.

[0058] 2. Adopt / add / delete / modify based on the generated use cases.

[0059] Objective: To ensure the quality of test cases.

[0060] Process: The AI-generated test cases are inspected, supplemented, and adjusted. This includes evaluating the accuracy, completeness, and applicability of the generated test cases and modifying them as needed. Through collaborative optimization and dynamic adjustments during the Prompt management phase, and multiple rounds of clarification during the large model invocation phase, the generated test cases are ensured to meet actual testing requirements.

[0061] 3. Accumulate it into the use case library.

[0062] Objective: To document and consolidate use cases for future reuse.

[0063] Process: The reviewed and adjusted test cases are stored in a test case library as part of the RAG (Retrieval Enhancement Generation) corpus, which is used to feed back into the large model. The generated results are automatically synchronized to internal enterprise systems, including test management tools and knowledge bases, via a WebHook interface for subsequent model fine-tuning and optimization.

[0064] The entire process can be deployed privately within the enterprise, ensuring data security and meeting the customization needs of the model.

[0065] In an optional embodiment, Figure 4 This illustrates a test case generation process based on a large model, which includes the following steps:

[0066] 1. Preprocessing: Standardize and structure multi-source heterogeneous data, support rich text format parsing (Word / PDF / HTML, etc.), build a vectorized knowledge base, realize semantic retrieval capabilities, establish a three-level index system (business domain - module - function point), and ensure data quality through data cleaning rules (duplicate removal, terminology standardization, missing value handling).

[0067] 2. Document loading / segmentation: This function identifies and extracts multiple elements from documents, supporting four element types: characters (e.g., text paragraphs, function descriptions), tables (e.g., parameter lists, state matrices), lists (e.g., operation steps, requirement items), and images (e.g., flowcharts, UI prototypes). A semantic segmentation strategy, combined with logical association analysis, is used to add structured metadata annotations to each segmented block.

[0068] 3. Prompt Management: Establish a collaborative Prompt template library, support multi-role collaborative creation processes, version management, implement A / B testing mechanisms, compare template effects, dynamically optimize template content based on historical data, and provide variable placeholders to achieve flexible template adaptation.

[0069] 4. Large Model Invocation: In this stage, the constructed Prompt is used to invoke the large model to generate test cases.

[0070] 5. Parsing and Adaptation: Finally, the generated test cases need to be parsed and adapted into different formats, providing multi-format output capabilities. For example, Markdown format is easy for personnel to read and review, JSON format is used to support system integration, and MindMap format is used for visualization. Simultaneously, a quality verification system is established to ensure output accuracy, and completeness and consistency are checked through automated rules.

[0071] In an optional embodiment, Figure 5 This is a diagram illustrating the process of building a test case library, such as... Figure 5 As shown, firstly, in the testing platform, the interface definition serves as input, and then the corresponding test cases are generated by the single-interface test case assistant. For tests requiring script support, the single-interface script test case assistant generates test scripts based on these test cases. When handling more complex multi-interface scenarios, the multi-interface scenario test case assistant intervenes, generating test cases covering multiple interfaces. All generated test cases are stored in the test case library. Simultaneously, these test cases and scripts are passed as input to the large model. The large model receives prompts from the testing platform and, through its internal processing mechanism, outputs test cases or scripts. The output of the large model can be directly generated test cases or optimizations and adjustments to existing test cases. Finally, these outputs are fed back into the test case library for subsequent test execution and evaluation. The entire process automates the process from interface definition to test case generation, processing, and storage, improving the efficiency and accuracy of testing.

[0072] In an optional embodiment, Figure 6 This diagram illustrates the architecture and workflow of an intelligent agent evaluation platform, specifically divided into four parts: business flow, capabilities, data, and infrastructure. The business flow includes steps such as uploading test sets, reviewing and publishing test sets, selecting agents to be tested, selecting evaluation sets, selecting models and prompts, executing tasks, and viewing results. The capability layer involves evaluation set management (uploading, storage, review, and publishing), evaluation execution (human evaluation, referee models + rule prompts, scoring rules), task management, and evaluation results (result analysis and evaluation reports). The data layer uses MySQL and Redis databases to store and manage evaluation data. The infrastructure layer includes AI model services, prompt projects, and a testing platform, which together support the execution of the entire evaluation process. The entire system aims to evaluate the quality and performance of intelligent agents' responses in an automated and standardized manner, ensuring that key indicators such as the accuracy and fluency of their answers meet business requirements.

[0073] In one optional embodiment, examples of the evaluation set are shown in Table 1:

[0074] Table 1

[0075]

[0076] Session ID: Indicates the round of the session. The same session ID is considered as one conversation. If not filled in, each is an independent conversation by default.

[0077] Reference question: The question to be input to the agent.

[0078] Suggested answer: The content the agent expects to reply with.

[0079] In an optional embodiment, different prompt words are constructed based on business categories. For example, for an agent answering open-ended questions, the corresponding rules are as follows:

[0080] "Your task is to rate the quality of the AI ​​assistant's responses and provide the final answer or conclusion directly, without showing the thought process."

[0081] You clearly understand that when a user issues a command about a scenario (defined as ${scene_desc}), an AI assistant's response should meet the following criteria (listed in descending order of importance): [Criterion Start] ${metric} [Criterion End]. Based on the user command, we have collected the following response from an AI assistant. Please comprehensively evaluate this response and provide an assessment based on your knowledge of the assistant's response criteria in the current scenario. The following is the user command and assistant response data: [Data Start] [User Command]: ${input} [Reply]:${answer} [Reference Answer]: ${reference_output} [End of data] Think it over carefully for a while, and then give a score and the reason for the score.

[0082] The scoring uses a ${max_score} scale (1-5), and the meaning of each score level is as follows: [Start of level meaning] ${score_desc} [End of level meaning].

[0083] Require:

[0084] 1. The total score is [[an integer score between 1 and ${max_score}]].

[0085] 2. The reasons for the scoring are as follows:

[0086] Advantages of the current reply:

[0087] 1. (Please list the points you think are good in the current response, and give a score between 1 and ${max_score} for each point.)

[0088] Shortcomings of the current response:

[0089] 1. (Please list the points you think the current response is lacking, and give a score between 1 and ${max_score} for each point.)

[0090] The variables are as follows:

[0091] ${scene}: Answers open-ended questions.

[0092] ${scene_desc}: Open communication commands, usually used to ask a question in an open domain, and the response is also open-ended, such as casual conversation, asking for advice, or seeking recommendations.

[0093] ${metric}: 1. Accuracy: Ensure the accuracy of the information provided, adhere to common sense and facts, and avoid misleading users. 2. Relevance: Answers must address the user's question, avoiding irrelevant content and ensuring the information is relevant. 3. Cultural Sensitivity and Harmlessness: Understand and respect users' cultural backgrounds and differences, act ethically, avoid cultural biases and insensitive expressions, and avoid including any potentially offensive content. 4. Information Richness: Provide detailed information while ensuring accuracy, especially background information that users may not have explicitly requested but that is helpful in understanding the question. 5. Clarity: Answer questions using clear and easy-to-understand language, avoiding potentially misunderstanding technical jargon or complex structures. 6. User Engagement: Encourage further user interaction, demonstrate attention and consideration to user questions, and promote communication through asking questions or providing feedback. 7. Empathy: Consider the user's emotional state when answering, appropriately expressing empathy and understanding, especially when answering questions with emotional connotations. 8. Constructive Feedback: Maintain a positive and constructive attitude even when facing critical or negative questions, providing valuable responses and suggestions.

[0094] ${max_score}: 5 points.

[0095] ${score_desc}: 1 point, the response has major flaws, completely deviates from the standard, and should not occur in practice; 2 points, some parts of the response meet the standard and can be adopted, but as a whole, the response quality is unsatisfactory; 3 points, the response has both strengths and weaknesses, but the overall strengths outweigh the weaknesses within the required evaluation criteria; 4 points, the response quality is acceptable, generally meets the standard, but there are a few minor issues that can be improved; when a reference answer is given, this level represents the response quality presented by the reference answer; 5 points, the response is perfect, strictly meets the standard in all aspects; when a reference answer is given, this level represents a response quality superior to the reference answer.

[0096] In an optional embodiment, Figure 7 The diagram illustrates the evaluation process of an intelligent agent, which is divided into three main parts: evaluation management, intelligent agent evaluation process, and required data.

[0097] Evaluation Management: Start the evaluation process, then upload the evaluation dataset. The uploaded evaluation dataset needs to be reviewed. If the review is approved, the evaluation dataset will be used; if it is rejected, it needs to be re-uploaded.

[0098] Intelligent agent evaluation process: First, create a new evaluation task, then select the intelligent agent to be tested. After selecting the evaluation set and evaluation rules, decide on the evaluation method, which can be manual or automatic evaluation. In manual evaluation, directly execute the task and output the intelligent agent's response, then perform manual review, and finally output an evaluation report and end the evaluation. In automatic evaluation, select a referee model, execute the task and output the intelligent agent's response, score, and reasoning, similarly perform manual review, output an evaluation report, and end the evaluation. The output evaluation report includes the input evaluation rules, evaluation set, corresponding referee model, model score, and scoring reasoning. The evaluation report is used to evaluate the quality of the intelligent agent's response. Finally, it will calculate the average score and the number of data points in each score range, and can also export the results to evaluate the quality of the intelligent agent.

[0099] Required data includes a list of rules, which can be preset or newly created. When creating a new rule, you can create a model template or a custom prompt.

[0100] Through the above embodiments, this application successfully constructs an integrated intelligent platform covering test asset generation, automated execution, and AI agent evaluation. It transforms the creation mode of test cases and automated scripts from manual writing heavily reliant on the experience of senior experts to automated generation based on large models and precise Prompt projects. This significantly shortens test asset generation time and accelerates agile development iteration through automated testing. In terms of accessibility, the solution encapsulates classic testing methodologies and interface specifications, enabling even junior personnel and business experts to generate professional test assets through natural language interaction. It also minimizes the technical barrier to developing interface automated scripts, effectively addressing the core pain point of low automated test coverage in core businesses and achieving universal applicability of testing capabilities. Regarding evaluation, the solution innovatively introduces an automated evaluation system based on a referee model, providing a multi-dimensional and quantifiable unified evaluation standard for the performance of AI agents. This completely changes the predicament of traditional manual evaluation, which is characterized by strong subjectivity, low efficiency, and difficulty in reproducing results. It provides a reliable data-driven closed loop for quality assurance and iterative optimization of AI applications, thus offering an efficient and reliable integrated intelligent testing solution.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0102] Figure 8 This is a structural block diagram of a software testing apparatus according to an embodiment of this application; as shown below. Figure 8 As shown, it includes:

[0103] Module 82 is used to retrieve the requirements documents and interface definition information of the target software functions from the knowledge base;

[0104] The generation module 84 is used to generate test information based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, and the automated test scripts are used to execute the test cases;

[0105] Test module 86 is used to execute the automated test script in a test environment to test the function of the target software and obtain test data;

[0106] The determination module 88 is used to determine the test results of the target software function based on the test data and the test cases.

[0107] The aforementioned apparatus first extracts the requirement documents and interface definitions for the target software functionality from a knowledge base. Based on these requirements documents and interface definitions, test information, including test cases and automated test scripts, is generated, with the automated test scripts used to execute the test cases. The automated test scripts are then run in a test environment to test the target software functionality and obtain test data. Finally, the test results for the target software functionality are determined based on the test data and test cases. This approach solves the problem of improving software testing efficiency, thereby achieving the desired effect.

[0108] In an exemplary embodiment, the acquisition module 82 is used to perform structured parsing on data from different data sources to obtain structured data; determine the retrieval index of the structured data according to the software function corresponding to the structured data; perform semantic segmentation on the structured data to obtain multiple data blocks; perform metadata annotation on the multiple data blocks to obtain annotated data blocks; and construct the knowledge base based on the annotated data blocks.

[0109] In an exemplary embodiment, the generation module 84 is configured to: determine a target prompt word template corresponding to the type of the target software function from a prompt word template library; determine first supplementary content based on the interface definition information; determine second supplementary content based on the requirements document; supplement the target prompt word template based on the first supplementary content and the second supplementary content to obtain a target prompt word, wherein the first supplementary content is used to determine a test interface, the second supplementary content is used to determine the test requirements corresponding to the test interface, and the target prompt word is used to instruct a large language model to generate test cases and automated test scripts that meet the test requirements for the test interface; and obtain the test cases and automated test scripts generated by the large language model based on the target prompt word.

[0110] In one exemplary embodiment, the testing module 86 is configured to: determine the requirement to be tested based on the requirements document; regenerate the test cases if it is determined that the test cases do not fully cover the requirement to be tested; and generate the automated test script based on the interface definition information and the test cases if it is determined that the test cases fully cover the requirement to be tested.

[0111] In an exemplary embodiment, the determining module 88 is configured to: obtain preset test data of the target software function from the test cases; determine that the test result is passed if the preset test data is consistent with the test data; and determine that the test result is failed if the preset test data is inconsistent with the test data.

[0112] In an exemplary embodiment, the apparatus is further configured to, upon determining that the target software includes an intelligent language interaction module, test the intelligent language interaction module by: acquiring an evaluation dataset of input from a target object, wherein the evaluation dataset includes an input question and expected output content; testing the intelligent language interaction module based on the input question in the evaluation dataset to obtain the output content of the intelligent language interaction module in response to the input question; and acquiring a language interaction test result generated by a large language model based on the evaluation dataset, evaluation rule prompts, and the output content, wherein the evaluation rule prompts are used to instruct the large language model to score the output content according to preset evaluation rules.

[0113] In an exemplary embodiment, the device is further configured to determine the score calculated by the output content under the preset evaluation rules based on the comparison result between the expected output content in the evaluation dataset and the output content, thereby obtaining multiple dimension score values, wherein the preset evaluation rules include multiple dimension score standards; and if it is determined that the score value of each dimension is greater than a first preset score value, and the sum of the score values ​​of the multiple dimensions is greater than a second preset score value, then the test result of the intelligent language interaction module is determined to be passed.

[0114] Embodiments of this application also provide a storage medium including a stored program, wherein the program executes any of the methods described above when it is run.

[0115] Optionally, in this embodiment, the storage medium may be configured to store program code for performing the following steps:

[0116] S1, Obtain the requirements documents and interface definition information for the target software functions from the knowledge base;

[0117] S2, Generate test information based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, wherein the automated test scripts are used to execute the test cases;

[0118] S3, Execute the automated test script in the test environment to test the target software functions and obtain test data;

[0119] S4. Determine the test results of the target software function based on the test data and the test cases.

[0120] Embodiments of this application also provide an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0121] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0122] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0123] S1, Obtain the requirements documents and interface definition information for the target software functions from the knowledge base;

[0124] S2, Generate test information based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, wherein the automated test scripts are used to execute the test cases;

[0125] S3, Execute the automated test script in the test environment to test the target software functions and obtain test data;

[0126] S4. Determine the test results of the target software function based on the test data and the test cases.

[0127] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0128] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0129] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0130] The embodiments described herein also provide a computer program that includes computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps in any of the above method embodiments.

[0131] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0132] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0133] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A software testing method, characterized in that, include: Obtain the requirements documents and interface definitions for the target software functions from the knowledge base; Test information is generated based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, and the automated test scripts are used to execute the test cases; The automated test script is executed in the test environment to test the functionality of the target software and obtain test data. The test results of the target software function are determined based on the test data and the test cases.

2. The software testing method according to claim 1, characterized in that, Before retrieving the requirements documents and interface definitions for the target software functions from the knowledge base, the method further includes: Structured data is obtained by parsing data from different data sources. The retrieval index for the structured data is determined based on the software functions corresponding to the structured data; The structured data is semantically segmented to obtain multiple data blocks; Metadata is annotated on the multiple data blocks to obtain annotated data blocks, and the knowledge base is constructed based on the annotated data blocks.

3. The software testing method according to claim 1, characterized in that, Test information is generated based on the requirements document and the interface definition information, including: Determine the target prompt word template corresponding to the type of the target software function from the prompt word template library; The first supplementary content is determined based on the interface definition information, the second supplementary content is determined based on the requirements document, and the target prompt word template is supplemented based on the first supplementary content and the second supplementary content to obtain the target prompt word. The first supplementary content is used to determine the test interface, the second supplementary content is used to determine the test requirements corresponding to the test interface, and the target prompt word is used to instruct the large language model to generate test cases and automated test scripts that meet the test requirements for the test interface. Obtain the test cases and automated test scripts generated by the large language model based on the target prompt words.

4. The software testing method according to claim 1, characterized in that, Test information is generated based on the requirements document and the interface definition information, including: The requirements to be tested are determined based on the aforementioned requirements document; If it is determined that the test cases do not fully cover the requirements to be tested, the test cases will be regenerated. If it is determined that the test cases fully cover the requirements to be tested, the automated test script is generated based on the interface definition information and the test cases.

5. The software testing method according to claim 1, characterized in that, Determining the test results of the target software function based on the test data and the test cases includes: Obtain preset test data for the target software function from the test cases; If the preset test data is consistent with the test data, the test result is determined to be passed; If the preset test data is found to be inconsistent with the test data, the test result is determined to be unsuccessful.

6. The software testing method according to claim 1, characterized in that, The method further includes: If the target software is determined to include an intelligent language interaction module, the intelligent language interaction module is tested in the following manner: Obtain the evaluation dataset of the target object input, wherein the evaluation dataset includes the input question and the expected output content; The intelligent language interaction module is tested based on the input questions in the evaluation dataset to obtain the output content of the intelligent language interaction module in response to the input questions; Obtain language interaction test results generated by the large language model based on the evaluation dataset, evaluation rule prompts, and the output content, wherein the evaluation rule prompts are used to instruct the large language model to score the output content according to preset evaluation rules.

7. The software testing method according to claim 6, characterized in that, Obtain the language interaction test results generated by the large language model based on the evaluation dataset, evaluation rule prompts, and the output content, including: Based on the comparison between the expected output content in the evaluation dataset and the output content, the score calculated by the output content under the preset evaluation rules is determined, resulting in multiple-dimensional score values. The preset evaluation rules include multiple-dimensional scoring criteria. If the score for each dimension is greater than the first preset score, and the sum of the scores for the multiple dimensions is greater than the second preset score, then the test result of the intelligent language interaction module is determined to be passed.

8. A software testing apparatus, characterized in that, include: The acquisition module is used to retrieve the requirements documents and interface definition information of the target software functions from the knowledge base; A generation module is used to generate test information based on the requirements document and the interface definition information, wherein the test information includes test cases and automated test scripts, and the automated test scripts are used to execute the test cases; The testing module is used to execute the automated test scripts in a test environment to test the functionality of the target software and obtain test data. The determination module is used to determine the test results of the target software function based on the test data and the test cases.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when executed, performs the method described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 7 through the computer program.

Images