Non-functional test method and device of software system and electronic equipment

Abstract

The invention provides a non-functional testing method and device of a software system and electronic equipment. According to the scheme, the method comprises the following steps: obtaining: obtaining system information of to-be-tested software; a matching step: matching the system information of the to-be-tested software with a non-functional test rule in the RAG knowledge base to obtain non-functional test details; a generation step: generating a non-functional test case according to the non-functional test details; an execution step: executing the non-functional test case to perform a non-functional test on the to-be-tested software, and collecting a test coverage rate in the process of performing the non-functional test; a supplementing step: generating a supplementary test case based on the test coverage rate, and supplementing the supplementary test case into the non-functional test case; and a repeating step: repeating the execution step and the supplement step for at least one time until the test coverage rate is higher than a preset test coverage rate. According to the scheme, the problem that in the prior art, a software system non-function test scheme is still limited is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a non-functional testing method for a software system, a non-functional testing device for a software system, a computer-readable storage medium, and an electronic device. Background Art

[0002] In existing software system testing solutions, system information is obtained from non-functional test maps, data is integrated, and a set of non-functional test cases for the target system is matched from the entire set. This method primarily relies on information in the non-functional test map and a preset case set to match and generate test cases. This matching mechanism is unable to identify and generate test cases for specific code changes or configuration adjustments, especially for details not recorded in the map or case set, resulting in limited test coverage and poor test accuracy. Existing technologies also generate test cases based on requirement documents and test rules. In this method, test rules are manually set by testers based on PRD documents and preset standards, resulting in a low degree of automation. This not only increases the workload, but also the generation and application of rules relies heavily on the tester's subjective judgment and experience, which can easily lead to incomplete or inaccurate test case generation due to human factors. Furthermore, this method is highly dependent on PRD documents. If the PRD document description is incomplete or of low quality, the generated test cases will not fully reflect the system requirements and non-functional characteristics, thus affecting the effectiveness of the test.

[0003] That is, the solutions for non-functional testing of software systems in the existing technology still have limitations, including insufficient comprehensiveness of testing, poor accuracy, and low degree of automation. Summary of the Invention

[0004] The main purpose of this application is to provide a non-functional testing method for a software system, a non-functional testing device for a software system, a computer-readable storage medium and an electronic device, so as to at least solve the limitations of the existing solutions for non-functional testing of software systems, including insufficient comprehensiveness of testing, poor accuracy and low degree of automation.

[0005] To achieve the above object, according to one aspect of the present application, a non-functional testing method for a software system is provided, including: an acquisition step of acquiring system information of the software to be tested, where the system information includes system configuration information and code change information of the software to be tested; a matching step of matching the system information of the software to be tested with non-functional testing rules in a RAG knowledge base to obtain non-functional testing details, wherein the non-functional testing rules are stored in the RAG knowledge base in the form of vectors, and the non-functional testing details include at least one of installation testing, interface testing, and performance testing; a generation step of generating non-functional test cases according to the non-functional testing details; an execution step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing process, where the test coverage characterizes the coverage degree of the code of the software to be tested when the non-functional test cases are executed; a supplement step of generating supplementary test cases based on the test coverage and supplementing the supplementary test cases into the non-functional test cases; a repetition step of repeating the execution step and the supplement step at least once in sequence until the test coverage is higher than a preset test coverage.

[0006] Optionally, before the matching step of matching the system information of the software to be tested with non-functional testing rules in the RAG knowledge base to obtain non-functional testing details, the method further includes: generating a rule file according to preset non-functional testing rules, where the preset non-functional testing rules include features at the system configuration level and features at the code level; storing the rule file in an initial RAG knowledge base; using a text segmentation method to segment the rule file in the initial RAG knowledge base to obtain a set of data segments including non-functional testing rule information; using a vectorization technique to convert each data segment in the set of data segments into a data segment feature vector to obtain the RAG knowledge base composed of multiple data segment feature vectors.

[0007] Optionally, the matching step of matching the system information of the software to be tested with non-functional testing rules in the RAG knowledge base to obtain non-functional testing details includes: using a similarity algorithm to match the system information with the non-functional testing rules in the RAG knowledge base to obtain non-functional testing rules matching the system information; inputting the non-functional testing rules matching the system information into a large language model for parsing to obtain the non-functional testing details, and the large language model is pre-trained by using a historical system information data set on an initial large language model.

[0008] Optionally, a similarity algorithm is used to match the system information with the non-functional test rules in the RAG knowledge base to obtain the non-functional test rules that match the system information, including: analyzing the system information through a feature extraction algorithm to generate a feature vector of the system information; using a similarity algorithm to calculate the similarity between the feature vector and the non-functional test rules in the RAG knowledge base to obtain multiple similarity results; determining the non-functional test rule corresponding to the maximum similarity result among the multiple similarity results as the non-functional test rule that matches the system information.

[0009] Optionally, generating non-functional test cases according to the non-functional test details includes: inputting the non-functional test details and a preset test case format as prompt words into a large language model to generate the non-functional test cases, where the non-functional test cases include test steps, input data, and expected outputs, and the large language model is pre-trained using a historical system information data set on an initial large language model.

[0010] Optionally, performing steps, executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing, including: performing non-functional testing on the software to be tested according to the non-functional test cases; using a test coverage tool to collect the test coverage during the execution of the non-functional testing.

[0011] Optionally, a supplementary step, generating supplementary test cases based on the test coverage and supplementing the supplementary test cases to the non-functional test cases, includes: parsing the test coverage to extract the uncovered test points of the non-functional test cases; matching the uncovered test points with the non-functional test rules in the RAG knowledge base to obtain supplementary non-functional test details; inputting the supplementary non-functional test details into the large language model to generate the supplementary test cases, and supplementing the supplementary test cases to the non-functional test cases.

[0012] According to another aspect of the present application, a non-functional testing device for a software system is provided, including: an acquisition unit for performing an acquisition step: the acquisition step of acquiring the system information of the software to be tested, where the system information includes the system configuration information and code change information of the software to be tested; a matching unit for performing a matching step: the matching step of matching the system information of the software to be tested with the non-functional testing rules in the RAG knowledge base to obtain non-functional testing details, where the non-functional testing rules are stored in the RAG knowledge base in the form of vectors, and the non-functional testing details include at least one of installation testing, interface testing, and performance testing; a first generation unit for performing a generation step: the generation step of generating non-functional test cases according to the non-functional testing details; an execution unit for performing an execution step: the execution step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing process, where the test coverage characterizes the coverage degree of the code of the software to be tested when the non-functional test cases are executed; a supplement unit for performing a supplement step: the supplement step of generating supplementary test cases based on the test coverage and supplementing the supplementary test cases into the non-functional test cases; a repetition unit for performing a repetition step: the repetition step of repeating the execution step and the supplement step at least once in sequence until the test coverage is higher than a preset test coverage.

[0013] According to still another aspect of the present application, a computer-readable storage medium is provided, where the computer-readable storage medium includes a stored program, and when the program runs, it controls the device where the computer-readable storage medium is located to execute any one of the non-functional testing methods of the software system.

[0014] According to yet another aspect of the present application, an electronic device is provided, including: one or more processors, a memory, and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the one or more programs include those for executing any one of the non-functional testing methods of the software system.

[0015] Applying the technical solution of the present application, the acquisition step is to obtain the system information of the software to be tested, where the system information includes the system configuration information and code change information of the software to be tested; the matching step is to match the system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details, where the non-functional test rules are stored in the RAG knowledge base in the form of vectors, and the non-functional test details include at least one of installation test, interface test, and performance test; the generation step is to generate non-functional test cases according to the non-functional test details; the execution step is to execute the non-functional test cases to perform non-functional testing on the software to be tested, and collect the test coverage rate during the non-functional testing process, where the test coverage rate represents the coverage degree of the code of the software to be tested when the non-functional test cases are executed; the supplement step is to generate supplementary test cases based on the test coverage rate and supplement the supplementary test cases to the non-functional test cases; the repetition step is to repeat the execution step and the supplement step at least once in sequence until the test coverage rate is higher than the preset test coverage rate. This solution reduces the labor cost, ensures high-quality testing of the non-functional characteristics of the software, effectively reduces the risk of missed testing, and improves the accuracy of non-functional testing by obtaining the system information of the software to be tested, matching the non-functional test rules in the RAG knowledge base, generating and supplementing non-functional test cases until the test coverage rate is higher than the preset test coverage rate, thus solving the problem that the existing solutions for non-functional testing of software systems still have limitations, including insufficient comprehensiveness, poor accuracy, and low automation degree of testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings forming a part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0017] Figure 1 The hardware structure block diagram of a mobile terminal showing a method for performing non-functional testing of a software system according to an embodiment of the present application is shown;

[0018] Figure 2 The flowchart showing a method for non-functional testing of a software system according to an embodiment of the present application is shown;

[0019] Figure 3 The flowchart showing a specific method for non-functional testing of a software system according to an embodiment of the present application is shown;

[0020] Figure 4 The structure block diagram of a device for non-functional testing of a software system according to an embodiment of the present application is shown.

[0021] Among them, the above-mentioned drawings include the following reference numerals:

[0022] 102, Processor; 104, Memory; 106, Transmission Device; 108, Input / Output Device. Detailed Implementation Manner

[0023] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments may be combined with each other. The following will describe this application in detail with reference to the drawings and in combination with the embodiments.

[0024] In order to enable those skilled in the art to better understand the solution of this application, the following will clearly and completely describe the technical solutions in the embodiments of this application with reference to the drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances for the embodiments of this application described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] As introduced in the background art, the existing solutions for non-functional testing of software systems still have limitations, including incomplete testing, poor accuracy, and low automation. To solve the problem that the existing solutions for non-functional testing of software systems still have limitations, the embodiments of this application provide a non-functional testing method for software systems, a non-functional testing device for software systems, a computer-readable storage medium, and an electronic device.

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

[0028] The method embodiments provided in the embodiments of this application can be executed on a mobile terminal, a computer terminal, or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 is the hardware structure block diagram of a mobile terminal of a non-functional testing method for a software system according to an embodiment of the present invention. As Figure 1 shown, the mobile terminal may include one or more ( Figure 1Only one processor 102 (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a field-programmable gate array FPGA) and a memory 104 for storing data are shown. Among them, the above mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those of ordinary skill in the art can understand that Figure 1 The structure shown is only schematic and does not limit the structure of the above mobile terminal. For example, the mobile terminal may further include more or fewer components than Figure 1 shown therein, or have a different configuration from Figure 1 shown therein.

[0029] The memory 104 can be used to store computer programs. For example, software programs and modules of application software, such as the computer program corresponding to the display method of device information in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, implements the above method. The memory 104 may include a high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely disposed relative to the processor 102, and these remote memories may be connected to the mobile terminal through a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one instance, the transmission device 106 includes a network adapter (abbreviated as NIC), which can be connected to other network devices through a base station and thus can communicate with the Internet. In one instance, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] In this embodiment, a non-functional testing method for a software system running on a mobile terminal, a computer terminal, or a similar computing device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And, although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0031] Figure 2 is a flowchart of a non-functional testing method for a software system according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0032] Step S201, the acquisition step, acquires the system information of the software to be tested. The above system information includes the system configuration information and code change information of the software to be tested;

[0033] Specifically, the acquisition step mainly comprehensively collects information related to the software to be tested for subsequent testing. The system information of the software to be tested includes the system configuration information and code change information of the software to be tested. Among them, the system configuration information refers to the details of various settings of the software system in the running environment, including the type and version of the operating system, middleware configuration (such as Web server, database management system), network environment parameters (such as port mapping, firewall settings), hardware resource limitations (such as memory size, number of CPU cores), configuration information of external services relied on by the system, etc. These information are crucial for understanding the actual running environment, resource constraints, and non-functional characteristics of the software. The code change information refers to the code-level changes that have occurred in the current version of the software compared to the previous version, including newly added code modules, modified functions, deleted logical code, changes in variable types or names, etc. The collection of code change information helps to identify possible new function points or potential defects, which is very useful for accurately positioning the key areas of non-functional testing.

[0034] Obtain the system configuration information and code change information of the software to be tested through the system configuration platform or the specified configuration file, compare the differences between the current code to be tested and the baseline code, and record the change points, such as newly added modules, modified functions, etc. The system configuration information reveals the external conditions for the software to run, while the code change information focuses on the internal changes of the software. The combination of the two provides a solid foundation for subsequent non-functional test rule matching and non-functional test case generation, helps to ensure the comprehensiveness and effectiveness of the test, and avoids missing non-functional performance issues in important test scenarios or configurations.

[0035] The acquisition step is the basis of the non-functional test method for the entire software system, ensuring that all subsequent test activities (such as non-functional rule matching, non-functional test case generation, non-functional test execution, etc.) are carried out based on the latest and most comprehensive system information, thereby improving the reliability of the test results and the efficiency of the test process.

[0036] Step S202, the matching step, matches the above system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details. Among them, the above non-functional test rules are stored in the above RAG knowledge base in the form of vectors, and the above non-functional test details include at least one of installation test, interface test, and performance test;

[0037] Specifically, the RAG knowledge base, namely the Retrieve-Augmented Generation (RAG) knowledge base, is a knowledge base that combines retrieval and generation models and is used to store and retrieve a large amount of structured and unstructured information to assist in generation tasks. In this embodiment, the RAG knowledge base is used to store non-functional test rules, and these non-functional test rules are represented in vector form. Vector storage is a method of converting text information into a numerical form, which can directly utilize numerical calculations for matching and subsequent processing, improving efficiency and accuracy. The system information of the software to be tested obtained is matched with the non-functional test rules in the RAG knowledge base to obtain non-functional test details. Non-functional test details refer to specific non-functional test items determined according to the matching results of the system information of the software to be tested and the non-functional test rules in the RAG knowledge base for the characteristics of the software to be tested. Non-functional tests cover multiple aspects, such as installation testing (checking the installation and configuration process of the software), interface testing (verifying the compatibility and stability of software interfaces), performance testing (evaluating the performance of the software under specific loads), etc. Through matching, non-functional test points related to the specific configuration and code changes of the software to be tested are identified, and non-functional test details are generated.

[0038] Through the matching step, it is possible to identify which non-functional characteristics need to be focused on and tested, thereby generating highly targeted non-functional test details and ensuring the coverage and effectiveness of the tests. The matching step automates the rule matching process that originally required manual judgment, improving the efficiency and accuracy of test design.

[0039] Step S203, the generation step, generates non-functional test cases according to the above non-functional test details;

[0040] Specifically, after determining the non-functional test details through the above matching step, a large language model is used to convert these non-functional test details into executable non-functional test cases. Large language models, such as the Qwen series, chatGLM series, etc., are pre-trained and can be further fine-tuned to adapt to specific tasks. In this embodiment, they are used to generate non-functional test cases. Specifically, the large language model receives non-functional test details as input, considers the characteristics of the software to be tested, test objectives, and environmental configurations, and generates a series of instructions and data that meet the test requirements. These instructions and data will guide how to execute the test, including how to set up the test environment, how to construct test data, and how to determine whether the test results meet the expectations.

[0041] Through the generation step, non-functional testing is transformed from abstract rules and test points into specific operation guides and data inputs, greatly enhancing the automation level and efficiency of testing. The generation of non-functional test cases determines the coverage and depth of testing, and is also the key to the accuracy and reliability of test results.

[0042] Step S204, Execution step, execute the above non-functional test cases to perform non-functional testing on the software to be tested, and collect test coverage during the process of performing the above non-functional testing, where the above test coverage represents the coverage degree of the code of the software to be tested when the above non-functional test cases are executed;

[0043] Specifically, in the non-functional testing execution phase, specific test operations are carried out according to the previously generated non-functional test cases. These non-functional test cases include a series of test scenarios designed to cover various non-functional characteristics of the software, such as performance, security, compatibility, maintainability, etc. During the test execution, test coverage data is collected, which is a key indicator for evaluating the test effect. Test coverage refers to the percentage of the code of the software to be tested covered by the executed non-functional test cases, usually including various indicators such as line coverage, branch coverage, and condition coverage. A higher test coverage means that more software code and logical paths have been tested and verified, which helps to reduce potential undetected defects and errors and improve software quality.

[0044] Non-functional test cases are generated based on the specific non-functional requirements of the software to be tested and the non-functional test rules in the RAG knowledge base. This means that the design of non-functional test cases directly targets possible non-functional problems in the software to be tested, such as performance bottlenecks, security vulnerabilities, compatibility issues, etc. Therefore, these characteristics can be tested deeply and accurately. Therefore, the execution of non-functional test cases realizes the comprehensive verification of the non-functional characteristics of the software to be tested. During the non-functional testing execution process, the real-time collected test coverage data can reflect the coverage of the current non-functional test cases on the code of the software to be tested. By collecting test coverage, it provides a data basis for the generation of subsequent supplementary test cases. This is because test coverage intuitively shows which code, logical paths, and configuration items have been tested and which parts have not been covered. According to the current coverage situation, more targeted new test cases (i.e., the subsequent mentioned supplementary test cases) can be generated to fill the test gaps and further improve the integrity and effectiveness of testing.

[0045] Step S205, Supplementary step, generate supplementary test cases based on the above test coverage, and supplement the above supplementary test cases into the above non-functional test cases;

[0046] Specifically, by collecting test coverage, evaluate whether each code block, configuration item, and interface has been adequately covered by test cases. If the test coverage of a certain part does not meet the test coverage requirements, then this part will be regarded as the key area that needs to be supplemented with tests. Based on the identified key areas that need to be supplemented with tests, and based on the non-functional test rules in the RAG knowledge base and the specific attributes of the software to be tested, generate a series of new test cases (i.e., supplementary test cases) specifically for detecting the uncovered or low-coverage code and configurations. These supplementary test cases cover more hidden input situations or explore different execution paths to ensure that every part of the software to be tested can be tested. Add the supplementary test cases to the original non-functional test cases for the next round of non-functional testing.

[0047] Through the supplementary steps, the test cases can be continuously optimized, and the omissions in the tests can be specifically filled, so as to achieve a comprehensive verification of the non-functional characteristics of the software to be tested. This not only reduces the workload of manually designing and adjusting test cases, but also significantly improves the test efficiency and the software quality assurance level. This automated supplementary test case generation process can flexibly cope with the rapid iteration of software versions, ensuring that each test is as comprehensive and effective as possible.

[0048] Step S206, repeat the steps, and repeat and execute the above execution steps and the above supplementary steps at least once until the above test coverage is higher than the preset test coverage.

[0049] Specifically, based on the existing non-functional test cases, conduct non-functional testing on the software to be tested. During the test execution process, continuously monitor and record the test coverage to understand the coverage degree of the current non-functional test cases on the code of the software to be tested. After the test is completed, analyze the test coverage to determine whether the preset test coverage threshold (i.e., the preset test coverage) has been reached. If the test coverage is lower than the preset test coverage, it is considered that the test is not comprehensive enough and there are undetected problems. Then, based on the RAG knowledge base and the currently collected test coverage, generate new test cases (i.e., supplementary test cases). These supplementary test cases are designed to cover the software code and configuration points that have not been tested or have been insufficiently tested to improve the test coverage. Add the supplementary test cases to the non-functional test cases and prepare to enter the next round of non-functional test execution. This round of testing will focus on the supplementary test cases to ensure that the remaining part of the software to be tested can be effectively covered. Repeat and execute the execution steps and the supplementary steps at least once until the test coverage is higher than the preset test coverage. The preset test coverage can be set according to the specific requirements and the importance of the software, and can be 95%, 98% or even higher to ensure the high quality and stability of the software.

[0050] Through repeated steps, the non-functional testing process forms a closed-loop optimization mechanism that can continuously detect and improve the non-functional characteristics of the software under test. This iterative testing strategy not only helps to discover and fix potential defects in the software under test, but also improves the testing efficiency and accuracy, avoids testing omissions caused by imperfect one-time design, and ensures that the software under test can meet the expected functional and performance standards before release.

[0051] In this embodiment, first, the system information of the software under test is obtained, and then it is matched with the non-functional testing rules in the RAG knowledge base to generate a detailed non-functional testing specification, and then non-functional test cases are generated. During the test execution process, the test coverage rate is collected, and supplementary test cases are generated based on the test coverage rate. Through iterative execution and supplementation, until the test coverage rate is higher than the preset test coverage rate. This solution improves the automation degree of the non-functional testing of the software under test through non-functional testing rule matching and iterative testing strategy, realizes comprehensive, accurate and efficient non-functional testing of the software under test, significantly improves the quality and coverage of non-functional testing, and thus solves the problem that the existing solutions for non-functional testing of software systems still have limitations, including insufficient comprehensiveness, poor accuracy and low automation degree.

[0052] In the specific implementation process, before matching the above-mentioned system information of the software under test with the non-functional testing rules in the RAG knowledge base to obtain the non-functional testing specification, the above method further includes: generating a rule file according to the preset non-functional testing rules, where the preset non-functional testing rules include characteristics at the system configuration level and characteristics at the code level; storing the rule file in the initial RAG knowledge base; using a text segmentation method to segment the rule file in the initial RAG knowledge base to obtain a set of data segments including non-functional testing rule information; using a vectorization technique to convert each data segment in the set of data segments into a data segment feature vector, and obtaining the above-mentioned RAG knowledge base composed of multiple above-mentioned data segment feature vectors.

[0053] The above content elaborates on the operations before the above-mentioned matching steps, that is, the process of constructing the RAG knowledge base. First, a rule file is generated according to the preset non-functional test rules, which include characteristics at the system configuration level (such as service deployment method, concurrent processing ability, etc.) and characteristics at the code level (such as exception handling mechanism, resource management code, etc.). The rule file is based on the analysis of industry standards and historical test data to cover all non-functional test requirements that the software may encounter. The generated preset rule file is stored in an initial RAG knowledge base. The RAG knowledge base is a database for storing and managing non-functional test rules, providing the basic data for subsequent rule matching. The rule file in the initial RAG knowledge base needs to be further processed to improve the efficiency and accuracy of subsequent matching. Specifically, a text segmentation method is used to process the rule file. In this embodiment, the chunk segmentation method can be used to segment the rule file into a set of smaller data fragments, and each data fragment contains one or more non-functional test rule information. This segmentation is beneficial for subsequent vectorization processing and rule matching. Using vectorization technology, each data fragment in the data fragment set is converted into a data fragment feature vector, forming an optimized RAG knowledge base composed of multiple data fragment feature vectors. Vectorization technology converts text information into numerical vectors, which is not only convenient for computer storage and processing, but also enables the rule matching process to use machine learning algorithms for efficient and accurate retrieval, improving the efficiency and accuracy of non-functional test rule matching.

[0054] Through the above steps, a structured, efficient and easily retrievable RAG knowledge base is constructed, providing a solid data foundation for subsequent non-functional test rule matching and non-functional test case generation. This method not only reduces the consumption of computing resources in the rule matching process, but also improves the intelligence and automation level of the entire test process.

[0055] After constructing the RAG knowledge base, the above system information of the software to be tested is matched with the non-functional test rules in the RAG knowledge base to obtain non-functional test details, including: using a similarity algorithm to match the above system information with the above non-functional test rules in the above RAG knowledge base to obtain non-functional test rules that match the above system information; inputting the non-functional test rules that match the above system information into a large language model for parsing to obtain the above non-functional test details, and the above large language model is pre-trained by using a historical system information data set on an initial large language model.

[0056] The above content further refines the above matching steps, that is, how to use similarity algorithms and large language models to accurately match and parse non-functional test rules to obtain detailed non-functional test details. Specifically, first, the system information (including the system configuration information and code change information of the software to be tested) is analyzed and processed through a feature extraction algorithm to generate a feature vector that can represent the key characteristics of the system information. The feature extraction algorithm can be an algorithm in natural language processing technologies such as the bag-of-words model, term frequency-inverse document frequency (TF-IDF), etc., which can convert text information into a numerical form for subsequent mathematical operations and comparisons. The feature vector can capture important elements in the system information, such as code structure, configuration parameters, performance indicators, etc., and is the basis for subsequent similarity calculations.

[0057] Next, a similarity algorithm (such as cosine similarity, Euclidean distance, etc.) is used to calculate the similarity between the generated feature vector of the system information and the feature vector of the non-functional test rules stored in the RAG knowledge base. The similarity algorithm quantifies the similarity degree between the two vectors and generates a similarity value. By calculating the similarity between the feature vector of the system information and the non-functional test rules in the RAG knowledge base, multiple similarity results are obtained, and the non-functional test rule with the maximum similarity value is selected as the non-functional test rule that best matches the system information. This means that the non-functional test rule that best matches the system information is most suitable for the characteristics of the software to be tested and can guide the generation of the most relevant and effective non-functional test cases.

[0058] Through feature extraction, similarity calculation, and selection of the maximum similarity result, this embodiment can intelligently and accurately select non-functional test rules for specific system information from the RAG knowledge base, providing a solid foundation for subsequent non-functional test case generation and non-functional test execution. This matching strategy based on feature vectors and similarity algorithms not only greatly improves the efficiency and accuracy of matching but also reduces the influence of human factors, ensuring the comprehensiveness and in-depthness of testing, which is of great significance for improving software quality and test automation level.

[0059] After determining the non-functional test rules that match the system information, these non-functional test rules are input into a pre-trained large language model for in-depth analysis. A large language model is a deep learning model that can understand and generate natural language and has powerful capabilities for understanding and processing complex information. In this embodiment, the large language model is pre-trained on a historical system information data set for the initial large language model, which means that the large language model has learned a large number of non-functional test rules and scenarios related to software systems and can more accurately understand the input rules and generate specific non-functional test details. The analysis process of the large language model can transform abstract non-functional test rules into specific test requirements and steps. For example, for the non-functional test rule of "high availability", the large language model will analyze that the tests to be carried out include specific test points such as cluster switching, fault recovery, and load balancing, as well as how to design test cases to verify these test points.

[0060] After being analyzed by the large language model, non-functional test details are finally generated. The non-functional test details list in detail the non-functional test requirements for the software to be tested, including installation tests, interface tests, performance tests, etc. The non-functional test details are used to guide the generation of subsequent non-functional test cases and the test execution process to ensure that the non-functional characteristics of the software to be tested are comprehensively and deeply verified.

[0061] By adopting a similarity algorithm for rule matching and a large language model for rule analysis, this embodiment can intelligently generate non-functional test details, which not only improves the pertinence and efficiency of non-functional testing but also reduces the dependence on the professional knowledge of testers, providing a more automated and intelligent solution for the non-functional testing of the software to be tested.

[0062] In some embodiments of this application, the generation step of generating non-functional test cases according to the above non-functional test details includes: inputting the above non-functional test details and a preset test case format as prompt words into the large language model to generate the above non-functional test cases. The above non-functional test cases include test steps, input data, and expected outputs, where the above large language model is pre-trained on a historical system information data set for the initial large language model.

[0063] The above content further elaborates on the generation steps. Specifically, first, the non-functional test details and the preset test case format are input into the large language model as prompt words. The non-functional test details include the specific test points and requirements of the software to be tested in terms of high availability, robustness, performance, etc., while the preset test case format provides the structural template of the test case, such as the framework and descriptions of parts like test steps, input data, expected output, etc. The role of the prompt words is to provide the context and guidance for the large language model to generate non-functional test cases, helping the large language model understand the non-functional test objectives and format requirements.

[0064] Before generating non-functional test cases, the large language model has been pre-trained with a historical system information dataset. The pre-training process enables the large language model to learn a large amount of information about software systems, non-functional test rules, non-functional test case instances, etc., thus acquiring the ability to understand and generate non-functional test cases. The large language model will generate non-functional test cases that meet the non-functional test requirements based on the input prompt words, combined with the knowledge it has learned. The generated non-functional test cases describe in detail the test steps (specific operation processes), input data (data or environmental configurations required during non-functional testing), and expected output (expected results or metrics when non-functional testing is successful), ensuring the accurate execution of non-functional testing.

[0065] Through the above process, this embodiment utilizes the generation ability of the large language model, combines non-functional test details and the preset test case format, and realizes the automated generation of non-functional test cases. This not only improves the efficiency and quality of generating non-functional test cases but also reduces the dependence on the professional knowledge of testers, making the non-functional testing process more intelligent and precise, contributing to the comprehensive coverage of the non-functional characteristics of the software to be tested and enhancing the reliability, performance, and user experience of the software to be tested.

[0066] To ensure that the non-functional testing process can comprehensively cover the non-functional characteristics of the software to be tested, the test coverage is collected during the execution of non-functional testing, including: performing non-functional testing on the software to be tested according to the above non-functional test cases; using a test coverage tool to collect the above test coverage during the execution of the above non-functional testing. Specifically, during the execution of non-functional testing, a dedicated test coverage tool is used to collect the test coverage in real-time. The test coverage tool can monitor and record the coverage of the code of the software to be tested by non-functional test cases during execution, including line coverage, branch coverage, path coverage, etc. The test coverage tool can be integrated with the non-functional testing execution environment to automatically collect and report the test coverage, and the test coverage will provide data support for the generation of subsequent supplementary test cases to ensure that the non-functional characteristics of the software to be tested are fully tested and improve the software quality and reliability.

[0067] Through the above steps, this embodiment realizes the automated execution of non-functional test cases and the accurate collection of test coverage. It not only improves the efficiency and accuracy of non-functional testing, but also ensures that the non-functional testing process can comprehensively cover the non-functional characteristics of the software to be tested, which is of great significance for software quality control and performance optimization. The collection and analysis of test coverage provide a basis for the generation of subsequent supplementary test cases, forming a continuously optimized non-functional testing closed-loop and further enhancing the effect of non-functional testing.

[0068] In this embodiment, a dynamic learning mechanism is also introduced. This dynamic learning mechanism allows the large language model and the RAG knowledge base to self-optimize and update based on newly collected test results and feedback data. Specifically, after each test execution, the test results, discovered defects, and corresponding solutions are fed back to the large language model. The large language model learns this new information and gradually improves the accuracy and efficiency of generating non-functional test cases. At the same time, the RAG knowledge base will also be updated according to newly discovered non-functional test rules and related test points to ensure the comprehensiveness and timeliness of the rule base. This dynamic learning and updating mechanism can further improve the intelligent level of non-functional testing and reduce omissions and deviations in future testing.

[0069] In some specific embodiments of this application, supplementary test cases are generated based on the above test coverage and added to the above non-functional test cases, including: parsing the above test coverage to extract the uncovered test points of the above non-functional test cases; matching the above uncovered test points with the above non-functional test rules in the above RAG knowledge base to obtain supplementary non-functional test details; inputting the above supplementary non-functional test details into the large language model to generate the above supplementary test cases, and adding the above supplementary test cases to the above non-functional test cases.

[0070] Specifically, first, parse the test coverage collected during the non-functional test execution. The test coverage tool generates data showing which code segments or configuration items have been covered by the test and which are still blank areas. By parsing this data, the parts not touched by the non-functional test cases, that is, the uncovered test points, can be accurately located. The extraction of uncovered test points is based on the analysis of the test coverage report to identify code paths, logical branches, configuration options, etc. not covered by the current non-functional test cases. After identifying the uncovered test points, find the non-functional test rules related to these uncovered test points in the RAG knowledge base. This step is similar to the matching process of the initial non-functional test rules but focuses on the uncovered test points. Through matching, supplementary non-functional test details can be obtained, which indicate the test points and requirements that still need to be concerned to completely test the non-functional characteristics of the software.

[0071] Finally, input the supplementary non-functional test details into the large language model. Utilize the generation ability of the large language model to generate supplementary test cases for the uncovered test points. The generated supplementary test cases also include specific test steps, input data, and expected outputs, aiming to fill the previous test gaps and ensure that every non-functional feature of the software under test has been fully examined. Add the generated supplementary test cases to the existing non-functional test cases for the next round of non-functional testing.

[0072] Through the above supplementary steps, this embodiment can continuously iterate and generate supplementary test cases based on the original non-functional tests, ensuring that the non-functional feature testing of the software under test achieves the goal of comprehensive coverage, improving the depth and breadth of non-functional testing, and playing an important role in enhancing the quality of the software under test and ensuring the stable operation of the system.

[0073] In addition to generating and executing non-functional test cases, this embodiment also introduces an intelligent analysis and feedback mechanism for non-functional test results. After the non-functional testing is completed, use the large language model or other data analysis tools to deeply analyze the test results, identify potential software problems, performance bottlenecks, or design defects. Feed back the analysis results to the development team and testers to help understand the test coverage, quickly locate the problem, and make necessary code modifications and system optimizations. In addition, intelligent analysis can also automatically adjust the priorities and scopes of subsequent tests according to the severity and frequency of the test results, ensuring that high-risk areas are more fully tested. This mechanism improves the efficiency of non-functional testing and its contribution to software quality improvement, and helps to build a more robust, efficient, and user-friendly software system.

[0074] To enable those skilled in the art to more clearly understand the technical solution of this application, the implementation process of the non-functional test method for the software system of this application will be described in detail below in combination with specific embodiments.

[0075] This embodiment relates to a specific non-functional test method for a software system, such as Figure 3As shown in the figure, it includes four parts, namely the system data acquisition part, the rule library part, the intelligent generation part, and the precise testing part. Among them, for the system data acquisition part, the system configuration information and code change information of the software to be tested are obtained through the system configuration platform or the specified configuration file, the differences between the current code under test and the baseline code are compared, and the change points are recorded, such as newly added modules, modified functions, etc. For the rule library part, the expert team defines the rules related to non-functional testing such as high availability, robustness, and performance, including the characteristics at the code level (such as exception handling, resource management, etc.), the characteristics at the configuration level (such as concurrency settings, timeout configurations, etc.), and the corresponding test points. The rules are stored in the RAG knowledge base according to the above logic, which is used to generate non-functional test details by combining search and retrieval-augmented generation technology, and then generate non-functional test cases. The construction of the RAG knowledge base is achieved by uploading the files of the above rules to the vectorized knowledge base and generating data segments through the chunking method. When matching rules, a similarity algorithm is used for matching and screening. Examples of the rules in the rule library are as follows: 1. Characteristics at the code level: for loop, characteristics at the configuration level: none, corresponding test point: when the data scale of the list object traversed by the for loop increases, perform performance time-consuming testing. 2. Characteristics at the code level: redisClient, characteristics at the configuration level: redis service, corresponding test point: according to the system deployment information, perform redis high availability testing.

[0076] The intelligent generation part then generates specific non-functional test cases based on the non-functional test point details and the algorithm based on machine learning, including test steps, input data, expected output, etc. The detailed process of generating non-functional test cases is to first form a prompt according to the test points, combined with the retrieved augmented known knowledge and the specific format requirements of non-functional test cases, and request the large language model for inference generation. Examples of known knowledge: Performance test cases need to consider at least four dimensions, namely single-transaction baseline testing, single-transaction load testing, mixed-transaction capacity testing, and mixed-transaction fatigue testing. Among them, single-transaction baseline testing needs to cover each transaction, etc. Examples of the specific format requirements of non-functional test cases: Non-functional test cases need to contain at least three fields: test steps, input data, and expected output.

[0077] Regarding the precise testing part, during the test execution process, a code line coverage tool is used to collect precise test data such as code coverage. The collected test coverage is deeply analyzed, and the analysis results are fed back to the intelligent generation part, that is, the uncovered code parts are extracted, and the RAG knowledge base is recombined to generate the code to be tested, the program to be tested, the involved interfaces, and the non-functional test details of non-functional test requirements. According to the non-functional test details, the large language model intelligently infers and generates non-functional test cases, continuously and intelligently expanding the non-functional test cases until the test coverage requirement is met. The intelligent generation part continues to generate new test cases according to the feedback until the test coverage is higher than the preset test coverage.

[0078] This embodiment proposes a non-functional test method for a software system, which combines and applies open-source large language foundation models (such as Qwen series, chatGLM series, etc.), RAG knowledge base and other technologies. First, a rule library for high availability, robustness, and performance testing is established. The rule library contains code and configuration features, and corresponding test key points. The rule library is sorted into a RAG knowledge base. The large language model generates the code to be tested, the program to be tested, the involved interfaces, and the test key point details of non-functional test requirements by analyzing the differences between the code to be tested and the benchmark code, and the configuration of the program to be tested, in combination with the RAG knowledge base. According to the test key point details, the large language model intelligently infers and generates test cases. According to the coverage data of precise testing, that is, the coverage data obtained after executing the test cases by instrumenting the source program once. The uncovered code parts are extracted, and the RAG knowledge base is recombined to generate the code to be tested, the program to be tested, the involved interfaces, and the test key point details of non-functional test requirements. According to the test key point details, the large language model intelligently infers and generates test cases, continuously and intelligently expanding the test cases until the test coverage requirement is met.

[0079] In summary, this embodiment utilizes the learning and understanding ability of the large language model to intelligently analyze code and configuration features. It reduces the problem of incomplete test coverage caused by differences in personnel experience and improves the efficiency of test case generation. By sorting the non-functional test rule library into a RAG knowledge base, the storage of rules is realized. The code and configuration features related to non-functional testing, the code and configuration features prone to problems, and the corresponding test key points are summarized to form a RAG knowledge base, and the rules are queried, inferred, and verified through algorithms such as vectors for retrieval to generate test key point details. It improves the accuracy and coverage of non-functional testing. In addition, this embodiment proposes an intelligent expansion mechanism based on precise test coverage. By extracting the uncovered code parts and combining the configuration, the non-functional test key point details are iteratively generated, and new test cases are intelligently generated and expanded to improve the test coverage, ensure the comprehensiveness and in-depthness of the test, reduce the risk of missed testing, and improve the software quality.

[0080] The embodiments of the present application also provide a non-functional testing device for a software system. It should be noted that the non-functional testing device for the software system in the embodiments of the present application can be used to execute the non-functional testing method for the software system provided in the embodiments of the present application. The device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be repeated here. As used hereinafter, the term "module" can be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0081] The following introduces the non-functional testing device for the software system provided in the embodiments of the present application.

[0082] Figure 4 is a structural block diagram of the non-functional testing device for the software system according to the embodiments of the present application. As Figure 4 shown, the device includes an acquisition unit 10, a matching unit 20, a first generation unit 30, an execution unit 40, a supplement unit 50, and a repetition unit 60. Among them, the acquisition unit is used to execute the acquisition step: the acquisition step of acquiring the system information of the software to be tested, where the system information includes the system configuration information and code change information of the software to be tested; the matching unit is used to execute the matching step: the matching step of matching the system information of the software to be tested with the non-functional testing rules in the RAG knowledge base to obtain non-functional test details, where the non-functional testing rules are stored in the RAG knowledge base in the form of vectors, and the non-functional test details include at least one of installation testing, interface testing, and performance testing; the first generation unit is used to execute the generation step: the generation step of generating non-functional test cases according to the non-functional test details; the execution unit is used to execute the execution step: the execution step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting the test coverage rate during the non-functional testing process, where the test coverage rate represents the coverage degree of the code of the software to be tested when the non-functional test cases are executed; the supplement unit is used to execute the supplement step: the supplement step of generating supplement test cases based on the test coverage rate and supplementing the supplement test cases to the non-functional test cases; the repetition unit is used to execute the repetition step: the repetition step of repeatedly executing the execution step and the supplement step at least once until the test coverage rate is higher than the preset test coverage rate.

[0083] Specifically, the acquisition unit is used to comprehensively collect information related to the software to be tested for subsequent testing. The system information of the software to be tested includes the system configuration information and code change information of the software to be tested. The system configuration information and code change information of the software to be tested are obtained through the system configuration platform or the specified configuration file. The differences between the current code to be tested and the baseline code are compared, and the change points are recorded, such as newly added modules, modified functions, etc. The system configuration information reveals the external conditions for the software to run, while the code change information focuses on the internal changes of the software. The combination of the two provides a solid foundation for subsequent non-functional test rule matching and non-functional test case generation, helps to ensure the comprehensiveness and effectiveness of the test, and avoids missing non-functional performance problems in important test scenarios or configurations.

[0084] The acquisition unit ensures that all subsequent test activities (such as non-functional rule matching, non-functional test case generation, non-functional test execution, etc.) are carried out based on the latest and most comprehensive system information, thereby improving the reliability of the test results and the efficiency of the test process.

[0085] In this embodiment, the RAG knowledge base is used to store non-functional test rules, which are represented in vector form. Vector storage is a method of representing text information in numerical form, which can directly utilize numerical calculations for matching and subsequent processing, improving efficiency and accuracy. The system information of the software to be tested obtained is matched with the non-functional test rules in the RAG knowledge base to obtain non-functional test details. Non-functional test details refer to specific non-functional test items determined according to the matching results of the system information of the software to be tested and the non-functional test rules in the RAG knowledge base for the characteristics of the software to be tested. Non-functional tests cover multiple aspects, such as installation testing (checking the installation and configuration process of the software), interface testing (verifying the compatibility and stability of the software interface), performance testing (evaluating the performance of the software under specific loads), etc. Through matching, non-functional test points related to the specific configuration and code changes of the software to be tested are identified, and non-functional test details are generated.

[0086] The matching unit is used to identify which non-functional characteristics need to be focused on and tested, so as to generate highly targeted non-functional test details and ensure the coverage and effectiveness of the test. The matching unit automates the rule matching process that originally required manual judgment, improving the efficiency and accuracy of test design.

[0087] After determining the non-functional test details, the large language model is used to convert these non-functional test details into executable non-functional test cases. The large language model takes the non-functional test details as input, considers the software characteristics to be tested, test objectives, and environmental configurations, and generates a series of instructions and data that meet the test requirements. These instructions and data will guide how to execute the test, including how to set up the test environment, how to construct test data, and how to determine whether the test results meet expectations.

[0088] Through the first generation unit, non-functional tests are transformed from abstract rules and test points into specific operation guides and data inputs, greatly improving the automation level and efficiency of testing. The generation of non-functional test cases determines the coverage and depth of testing, and is also the key to the accuracy and reliability of test results.

[0089] In the non-functional test execution phase, specific test operations are carried out according to the previously generated non-functional test cases. These non-functional test cases include a series of test scenarios designed to cover various non-functional characteristics of the software, such as performance, security, compatibility, maintainability, etc. During the test execution, test coverage data is collected, which is a key indicator for evaluating the test effect. Test coverage refers to the percentage of the code of the software to be tested covered by the executed non-functional test cases, usually including various indicators such as line coverage, branch coverage, and condition coverage. A higher test coverage means that more software code and logical paths have been tested and verified, which helps to reduce potential undetected defects and errors and improve software quality.

[0090] Non-functional test cases are generated based on the specific non-functional requirements of the software to be tested and the non-functional test rules in the RAG knowledge base. This means that the design of non-functional test cases directly targets the possible non-functional problems of the software to be tested, such as performance bottlenecks, security vulnerabilities, compatibility issues, etc. Therefore, it can test these characteristics deeply and accurately. Therefore, the execution of non-functional test cases realizes the comprehensive verification of the non-functional characteristics of the software to be tested. During the non-functional test execution process, the real-time collected test coverage data can reflect the coverage of the current non-functional test cases on the code of the software to be tested. By collecting test coverage, it provides a data basis for the generation of subsequent supplementary test cases. This is because test coverage intuitively shows which code, logical paths, and configuration items have been tested and which parts have not been covered. According to the current coverage situation, more targeted new test cases (i.e., the subsequent mentioned supplementary test cases) can be generated to fill the test gaps and further improve the integrity and effectiveness of testing.

[0091] By collecting test coverage, evaluate whether each code block, configuration item, and interface has been adequately covered by test cases. If the test coverage of a certain part does not meet the test coverage requirements, then this part will be regarded as the key area that needs to be supplemented with tests. Based on the identified key areas that need to be supplemented with tests, and according to the non-functional test rules in the RAG knowledge base and the specific attributes of the software under test, generate a series of new test cases (i.e., supplementary test cases) specifically for detecting the uncovered or low-coverage code and configurations. These supplementary test cases cover more hidden input situations or explore different execution paths to ensure that every part of the software under test can be tested. Add the supplementary test cases to the original non-functional test cases for the next round of non-functional testing.

[0092] Through the supplementary unit, the test cases can be continuously optimized, specifically making up for the omissions in testing, so as to achieve a comprehensive verification of the non-functional characteristics of the software under test. This not only reduces the workload of manually designing and adjusting test cases, but also significantly improves the test efficiency and the level of software quality assurance. This automated supplementary test case generation process can flexibly cope with the rapid iteration of software versions, ensuring that each test is as comprehensive and effective as possible.

[0093] According to the existing non-functional test cases, conduct non-functional testing on the software under test. During the test execution process, continuously monitor and record the test coverage to understand the coverage degree of the current non-functional test cases on the code of the software under test. After the test is completed, analyze the test coverage to determine whether the preset test coverage threshold (i.e., the preset test coverage) has been reached. If the test coverage is lower than the preset test coverage, it is considered that the test is not comprehensive enough and there are undetected problems. Then, according to the RAG knowledge base and the currently collected test coverage, generate new test cases (i.e., supplementary test cases). These supplementary test cases are designed to cover the software code and configuration points that have not been tested or have been insufficiently tested to improve the test coverage. Add the supplementary test cases to the non-functional test cases and prepare for the next round of non-functional test execution. This round of testing will focus on the supplementary test cases to ensure that the remaining part of the software under test can be effectively covered. The repetition unit is used to execute the repeated steps, that is, repeat the execution step and the supplementary step at least once in sequence until the test coverage is higher than the preset test coverage.

[0094] Through the repetition unit, the non-functional test process forms a closed-loop optimization mechanism that can continuously detect and improve the non-functional characteristics of the software under test. This iterative test strategy not only helps to discover and fix potential defects in the software under test, but also improves the test efficiency and accuracy, avoiding test omissions caused by imperfect one-time design, and ensuring that the software under test can meet the expected functional and performance standards before release.

[0095] In a specific implementation process, the above-mentioned device further includes a second generation unit, a storage unit, a segmentation unit, and a conversion unit. Among them, the above-mentioned second generation unit is used to generate a rule file according to a preset non-functional test rule before matching the system information of the above-mentioned software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details. The above-mentioned preset non-functional test rule includes characteristics at the system configuration level and characteristics at the code level; the above-mentioned storage unit is used to store the above-mentioned rule file in the initial RAG knowledge base; the above-mentioned segmentation unit is used to segment the above-mentioned rule file in the above-mentioned initial RAG knowledge base by using a text segmentation method to obtain a set of data segments including non-functional test rule information; the above-mentioned conversion unit is used to use a vectorization technique to convert each data segment in the above-mentioned set of data segments into a data segment feature vector to obtain the above-mentioned RAG knowledge base composed of multiple above-mentioned data segment feature vectors.

[0096] Specifically, the above-mentioned second generation unit is used to generate a rule file according to a preset non-functional test rule. The preset non-functional test rule includes characteristics at the system configuration level (such as service deployment method, concurrent processing ability, etc.) and characteristics at the code level (such as exception handling mechanism, resource management code, etc.). The rule file is based on the analysis of industry standards and historical test data to cover all non-functional test requirements that the software may encounter. The generated preset rule file is stored in an initial RAG knowledge base. The RAG knowledge base is a database for storing and managing non-functional test rules, providing basic data for subsequent rule matching. The rule file in the initial RAG knowledge base needs to be further processed to improve the efficiency and accuracy of subsequent matching. Specifically, a text segmentation method is used to process the rule file. In this embodiment, the chunk segmentation method can be used to segment the rule file into smaller sets of data segments, and each data segment contains one or more non-functional test rule information. This segmentation is beneficial for subsequent vectorization processing and rule matching. Using a vectorization technique, each data segment in the set of data segments is converted into a data segment feature vector to form an optimized RAG knowledge base composed of multiple data segment feature vectors. The vectorization technique converts text information into numerical vectors, which is not only convenient for computer storage and processing, but also enables the rule matching process to use machine learning algorithms for efficient and accurate retrieval, improving the efficiency and accuracy of non-functional test rule matching.

[0097] Through the above-mentioned second generation unit, a structured, efficient, and easily retrievable RAG knowledge base is constructed, providing a solid data foundation for subsequent non-functional test rule matching and non-functional test case generation. It not only reduces the consumption of computing resources in the rule matching process, but also improves the intelligence and automation level of the entire test process.

[0098] In some embodiments of the present application, the above-mentioned matching unit includes a first matching module and an input module. Among them, the above-mentioned first matching module is used to match the above-mentioned system information with the above-mentioned non-functional test rules in the RAG knowledge base by using a similarity algorithm, and obtain the non-functional test rules that match the above-mentioned system information; the above-mentioned input module is used to input the non-functional test rules that match the above-mentioned system information into the large language model for parsing to obtain the above-mentioned non-functional test details, and the above-mentioned large language model is pre-trained from the initial large language model using the historical system information data set.

[0099] Specifically, the above-mentioned first matching module includes a generation sub-module, a calculation sub-module, and a determination sub-module. Among them, the above-mentioned generation sub-module is used to analyze and process the system information (including the system configuration information and code change information of the software to be tested) through a feature extraction algorithm, and generate a feature vector that can represent the key characteristics of the system information. The feature vector can capture important elements in the system information, such as code structure, configuration parameters, performance indicators, etc., and is the basis for subsequent similarity calculation.

[0100] The above-mentioned calculation sub-module is used to calculate the similarity between the generated system information feature vector and the non-functional test rule feature vector stored in the RAG knowledge base by using a similarity algorithm (such as cosine similarity, Euclidean distance, etc.). The similarity algorithm quantifies the similarity degree between two vectors and generates a similarity value. By calculating the similarity between the feature vector of the system information and the non-functional test rules in the RAG knowledge base, multiple similarity results are obtained. The above-mentioned determination sub-module is used to select the non-functional test rule with the maximum similarity value as the non-functional test rule that best matches the system information. This means that the non-functional test rule that best matches the system information is most suitable for the characteristics of the software to be tested and can guide the generation of the most relevant and effective non-functional test cases.

[0101] Through feature extraction, similarity calculation, and selection of the maximum similarity result, this embodiment can intelligently and accurately select non-functional test rules for specific system information from the RAG knowledge base, providing a solid foundation for subsequent non-functional test case generation and non-functional test execution. This matching strategy based on feature vectors and similarity algorithms not only greatly improves the efficiency and accuracy of matching, but also reduces the influence of human factors, ensuring the comprehensiveness and in-depthness of testing, and is of great significance for improving software quality and test automation level.

[0102] In some embodiments of the present application, the above input module is used to input these non-functional test rules into a pre-trained large language model for in-depth parsing after determining the non-functional test rules that match the system information. In this embodiment, the large language model is pre-trained based on a historical system information dataset for the initial large language model, which means that the large language model has learned a large number of non-functional test rules and scenarios related to software systems and can more accurately understand the input rules and generate specific non-functional test details. The parsing process of the large language model can transform abstract non-functional test rules into specific test requirements and steps. For example, for the non-functional test rule of "high availability", the large language model will parse out specific test points that need to be tested, including cluster switching, fault recovery, load balancing, etc., and how to design test cases to verify these test points.

[0103] After being parsed by the large language model, non-functional test details are finally generated. The non-functional test details list in detail the non-functional test requirements that the software to be tested needs to undergo, including installation tests, interface tests, performance tests, etc. The non-functional test details are used to guide the generation of subsequent non-functional test cases and the test execution process to ensure that the non-functional characteristics of the software to be tested are comprehensively and deeply verified.

[0104] By adopting a similarity algorithm for rule matching and a large language model for rule parsing, this embodiment can intelligently generate non-functional test details, which not only improves the pertinence and efficiency of non-functional testing but also reduces the dependence on the professional knowledge of testers, providing a more automated and intelligent solution for the non-functional testing of the software to be tested.

[0105] In some embodiments of the present application, the above first generation unit is used to input the above non-functional test details and a preset test case format as prompt words into the large language model to generate the above non-functional test cases. The non-functional test cases include test steps, input data, and expected outputs. Among them, the above large language model is obtained by pre-training the initial large language model using a historical system information dataset.

[0106] Specifically, the non-functional test details and the preset test case format are input into the large language model as prompt words. The non-functional test details contain specific test points and requirements for the software to be tested in terms of high availability, robustness, performance, etc., while the preset test case format gives the structural template of the test case, such as the framework and description of parts such as test steps, input data, and expected outputs. The role of the prompt words is to provide the context and guidance for the large language model to generate non-functional test cases, helping the large language model understand the non-functional test objectives and format requirements.

[0107] Before generating non-functional test cases, the large language model has been pre-trained on a historical system information dataset. The pre-training process enables the large language model to learn a large amount of information about software systems, non-functional test rules, non-functional test case examples, etc., thus acquiring the ability to understand and generate non-functional test cases. The large language model will generate non-functional test cases that meet the requirements of non-functional testing based on the input prompts, combining the knowledge it has learned. The generated non-functional test cases describe in detail the test steps (specific operation processes), input data (data or environmental configurations required during non-functional testing), and expected outputs (expected results or metrics when non-functional testing is successful), ensuring that non-functional testing can be accurately executed.

[0108] This embodiment utilizes the generation ability of the large language model and combines non-functional test details and a preset test case format to achieve the automated generation of non-functional test cases. This not only improves the efficiency and quality of non-functional test case generation but also reduces the dependence on testers' professional knowledge, making the non-functional testing process more intelligent and precise, contributing to a comprehensive coverage of the non-functional characteristics of the software under test and enhancing the reliability, performance, and user experience of the software under test.

[0109] To ensure that the non-functional testing process can comprehensively cover the non-functional characteristics of the software under test, the above-mentioned execution unit includes a non-functional testing module and a collection module. Among them, the above-mentioned non-functional testing module is used to perform non-functional testing on the above-mentioned software under test according to the above-mentioned non-functional test cases; the above-mentioned collection module is used to collect the above-mentioned test coverage using a test coverage tool during the execution of the above-mentioned non-functional testing. Specifically, during the execution of non-functional testing, a dedicated test coverage tool is used to collect the test coverage in real-time. The test coverage tool can monitor and record the coverage of the code of the software under test by non-functional test cases during execution, including line coverage, branch coverage, path coverage, etc. The test coverage tool can be integrated with the non-functional testing execution environment to automatically collect and report the test coverage, which will provide data support for the generation of subsequent supplementary test cases to ensure that the non-functional characteristics of the software under test are fully tested and improve software quality and reliability.

[0110] This embodiment realizes the automated execution of non-functional test cases and the accurate collection of test coverage, not only improving the efficiency and accuracy of non-functional testing but also ensuring that the non-functional testing process can comprehensively cover the non-functional characteristics of the software under test, which is of great significance for software quality control and performance optimization. The collection and analysis of test coverage provide a basis for the generation of subsequent supplementary test cases, forming a continuously optimized non-functional testing closed-loop and further enhancing the effect of non-functional testing.

[0111] In the specific implementation process, the above supplementary unit includes an extraction module, a second matching module, and a supplementary module. Among them, the above extraction module is used to parse the above test coverage rate and extract the uncovered test points of the above non-functional test cases; the above second matching module is used to match the above uncovered test points with the above non-functional test rules in the above RAG knowledge base to obtain supplementary non-functional test details; the above supplementary module is used to input the above supplementary non-functional test details into the large language model, generate the above supplementary test cases, and supplement the above supplementary test cases into the above non-functional test cases.

[0112] Specifically, parse the test coverage rate collected during the execution of non-functional tests. The test coverage rate tool generates data showing which code segments or configuration items have been covered by tests and which are still blank areas. By parsing this data, the parts that have not been touched by non-functional test cases can be accurately located, that is, the uncovered test points. The extraction of uncovered test points is based on the analysis of the test coverage rate report to identify code paths, logical branches, configuration options, etc. that have not been covered by the current non-functional test cases. After identifying the uncovered test points, find the non-functional test rules related to these uncovered test points in the RAG knowledge base. Through matching, supplementary non-functional test details can be obtained, which point out the test points and requirements that still need to be concerned in order to fully test the non-functional characteristics of the software.

[0113] Input the supplementary non-functional test details into the large language model and utilize the generation ability of the large language model to generate supplementary test cases for the uncovered test points. The generated supplementary test cases also include specific test steps, input data, and expected outputs, aiming to fill the previous test gaps and ensure that every non-functional characteristic of the software to be tested has been fully verified. Add the generated supplementary test cases to the existing non-functional test cases for the next round of non-functional tests.

[0114] This embodiment can continuously iterate and generate supplementary test cases on the basis of the original non-functional tests, ensuring that the non-functional characteristic tests of the software to be tested reach the goal of comprehensive coverage, improving the depth and breadth of non-functional tests, and playing an important role in improving the quality of the software to be tested and ensuring the stable operation of the system.

[0115] The non-functional test device of the above software system includes a processor and a memory. The above acquisition unit, matching unit, first generation unit, execution unit, supplementary unit, and repetition unit are all stored in the memory as program units, and the corresponding functions are implemented by the processor executing the above program units stored in the memory. The above modules are all located in the same processor; or, the above modules are respectively located in different processors in any combination form.

[0116] The memory may include non-permanent memory in the form of computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory includes at least one memory chip.

[0117] An embodiment of the present invention provides a computer-readable storage medium. The computer-readable storage medium includes a stored program. When the program runs, it controls a device where the computer-readable storage medium is located to execute a non-functional testing method of the software system.

[0118] An embodiment of the present invention provides an electronic device. The electronic device includes a processor, a memory, and a program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the non-functional testing method of the software system.

[0119] This application also provides a computer program product. When executed on a data processing device, it is adapted to execute a program for initializing the steps of the non-functional testing method of the software system.

[0120] Obviously, those skilled in the art should understand that the various modules or steps of the present invention described above can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. They can be implemented by program code executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them can be fabricated into a single integrated circuit module to implement. In this way, the present invention is not limited to any specific combination of hardware and software.

[0121] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0123] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operational steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or means for implementing the functions specified in one or more of the blocks.

[0125] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0126] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0127] A computer-readable medium includes both permanent and non-permanent, removable and non-removable media and can implement information storage by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer-readable medium does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0128] It should also be noted that the term "comprising", "including" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article or apparatus comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or apparatus. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising the element.

[0129] The above are only the preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A non-functional testing method for a software system, characterized in that, Including: An obtaining step of obtaining the system information of the software to be tested, where the system information includes the system configuration information and code change information of the software to be tested; A matching step of matching the system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details. Among them, the non-functional test rules are stored in the RAG knowledge base in the form of vectors, and the non-functional test details include at least one of installation test, interface test, and performance test; A generating step of generating non-functional test cases according to the non-functional test details; An executing step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing process, where the test coverage characterizes the coverage degree of the code of the software to be tested when the non-functional test cases are executed; A supplementing step of generating supplementary test cases based on the test coverage and supplementing the supplementary test cases into the non-functional test cases; A repeating step of repeating the executing step and the supplementing step in sequence at least once until the test coverage is higher than the preset test coverage.

2. The method according to claim 1, wherein Before the matching step of matching the system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details, the method further includes: Generating a rule file according to the preset non-functional test rules, where the preset non-functional test rules include features at the system configuration level and features at the code level; Storing the rule file into the initial RAG knowledge base; Using a text segmentation method to segment the rule file in the initial RAG knowledge base to obtain a set of data segments including non-functional test rule information; Using a vectorization technique to convert each data segment in the set of data segments into a data segment feature vector to obtain the RAG knowledge base composed of multiple data segment feature vectors.

3. The method according to claim 1, characterized in that, The matching step of matching the system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details includes: Using a similarity algorithm to match the system information with the non-functional test rules in the RAG knowledge base to obtain non-functional test rules that match the system information; Inputting the non-functional test rules that match the system information into a large language model for parsing to obtain the non-functional test details, and the large language model is obtained by pre-training the initial large language model using a historical system information data set.

4. The method according to claim 3, characterized in that, Using a similarity algorithm to match the system information with the non-functional test rules in the RAG knowledge base to obtain non-functional test rules that match the system information, including: Analyzing the system information through a feature extraction algorithm to generate a feature vector of the system information; Using a similarity algorithm to calculate the similarity between the feature vector and the non-functional test rules in the RAG knowledge base to obtain multiple similarity results; Determine the non-functional test rule corresponding to the maximum similarity result among the multiple similarity results as the non-functional test rule matching the system information.

5. The method according to claim 1, wherein A generation step of generating non-functional test cases according to the non-functional test details, including: Input the non-functional test details and a preset test case format as prompt words into a large language model to generate the non-functional test cases. The non-functional test cases include test steps, input data, and expected outputs, where the large language model is pre-trained from an initial large language model using a historical system information dataset.

6. The method according to claim 1, wherein An execution step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing process, including: Perform non-functional testing on the software to be tested according to the non-functional test cases; During the execution of the non-functional testing, use a test coverage tool to collect the test coverage.

7. The method according to any one of claims 1 to 6, characterized in that, A supplementation step of generating supplementary test cases based on the test coverage and supplementing the supplementary test cases into the non-functional test cases, including: Analyze the test coverage and extract the uncovered test points of the non-functional test cases; Match the uncovered test points with the non-functional test rules in the RAG knowledge base to obtain supplementary non-functional test details; Input the supplementary non-functional test details into the large language model to generate the supplementary test cases and supplement the supplementary test cases into the non-functional test cases.

8. A non-functional testing device for a software system, characterized in that, Including: An acquisition unit for performing an acquisition step: an acquisition step of acquiring the system information of the software to be tested, where the system information includes the system configuration information and code change information of the software to be tested; A matching unit for performing a matching step: a matching step of matching the system information of the software to be tested with the non-functional test rules in the RAG knowledge base to obtain non-functional test details, where the non-functional test rules are stored in the RAG knowledge base in vector form, and the non-functional test details include at least one of installation testing, interface testing, and performance testing; A first generation unit for performing a generation step: a generation step of generating non-functional test cases according to the non-functional test details; An execution unit for performing an execution step: an execution step of executing the non-functional test cases to perform non-functional testing on the software to be tested, and collecting test coverage during the non-functional testing process, where the test coverage represents the coverage degree of the code of the software to be tested when the non-functional test cases are executed; A supplementation unit for performing a supplementation step: a supplementation step of generating supplementary test cases based on the test coverage and supplementing the supplementary test cases into the non-functional test cases; A repetition unit for performing a repetition step: a repetition step of repeating the execution step and the supplementation step in sequence at least once until the test coverage is higher than a preset test coverage.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the non-functional testing method of the software system according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing non-functional testing of the software system according to any one of claims 1 to 7.

Images