Test method, system and equipment and storage medium

Abstract

The invention discloses a test method, system and device and a storage medium, and is applied to the technical field of computers, and the method comprises the steps: generating a test case set comprising a plurality of test cases through a language model after the language model learns a background knowledge document; testing the tested object based on the test cases in the test case set; when any test case fails to be executed, on the basis of the execution information of the test case and the background knowledge document, a replacement case for the test case is regenerated through the language model, and the replacement case is executed; when the replacement case is successfully executed, it is determined that the tested object passes the test of the replacement case, and when the replacement case fails to be executed, it is determined that the tested object does not pass the test of the replacement case, the problems that the generation efficiency of the test case is low and the comprehensiveness is insufficient are solved, the generation efficiency and comprehensiveness of the test case can be effectively improved, and the user experience is improved. And the test reliability is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a test method, system, device, and storage medium. Background Art

[0002] The number and complexity of functions of computer software systems are increasing, making the requirements for software testing higher and higher. Currently, the efficiency of software testing is relatively low. When generating test cases, developers usually manually write a rule library based on traditional tools, resulting in low test case generation efficiency. Moreover, the generated test cases are not comprehensive enough, leading to a long time to find the cause of problems and a greatly increased possibility of problems being hidden.

[0003] In summary, how to improve the generation efficiency of test cases and the comprehensiveness of the generated test cases is an urgent technical problem that those skilled in the art need to solve currently. Summary of the Invention

[0004] This application provides a test method, system, device, and storage medium to improve the generation efficiency of test cases and the comprehensiveness of the generated test cases.

[0005] This application provides a test method, including:

[0006] Input a background knowledge document reflecting the requirements for test case generation into a preset language model, so that the language model learns the background knowledge document;

[0007] Generate a test case set including multiple test cases through the language model;

[0008] Test the object under test based on the test cases in the test case set;

[0009] When any one of the test cases fails, based on the execution information of the test case and the background knowledge document, regenerate a replacement case for the test case through the language model and execute the replacement case;

[0010] When the replacement case is executed successfully, determine that the object under test has passed the test of the replacement case. When the replacement case fails, determine that the object under test has not passed the test of the replacement case.

[0011] This application also provides a test system, including:

[0012] A background knowledge document learning module, configured to input a background knowledge document reflecting the requirements for test case generation into a preset language model, so that the language model learns the background knowledge document;

[0013] A test case generation module, configured to generate a test case set including multiple test cases through the language model;

[0014] A test execution module, configured to perform tests on the object under test based on the test cases in the test case set;

[0015] A test case re-generation execution module, configured to, when any one of the test cases fails to execute, re-generate a replacement case for the test case based on the execution information of the test case and the background knowledge document through the language model, and execute the replacement case;

[0016] A replacement case execution result determination module, configured to, when the replacement case executes successfully, determine that the object under test passes the test of the replacement case, and when the replacement case fails to execute, determine that the object under test fails the test of the replacement case.

[0017] This application also provides a test device, including:

[0018] A memory, configured to store a computer program;

[0019] A processor, configured to execute the computer program to implement the steps of the test method as described above.

[0020] This application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the test method as described above are implemented.

[0021] Through this application, after inputting the background knowledge document used to reflect the requirements for test case generation into a preset language model, the language model can learn the background knowledge document. Since a test case set including multiple test cases is generated by the language model, the generation efficiency is very high, and a large number of test cases can be generated quickly, thus ensuring the comprehensiveness of the generated test cases, and this process does not require manual intervention. Further, after testing the object under test based on the test cases in the test case set, when a certain test case fails to execute, instead of directly ending and determining that the object under test fails the test of this test case, a replacement case for the test case will be regenerated through the language model based on the execution information of the test case and the background knowledge document, and the replacement case will be executed. This is because when a certain test case fails to execute, it may be that the test case itself is unreasonable, or it is due to factors such as version updates of the object under test or other accidental factors that cause the failure, rather than a problem with the object under test. Therefore, regenerating a replacement case for the test case through the language model is equivalent to regenerating a test case to repeat the verification. If the replacement case can be executed successfully, it can be determined that the object under test passes the test of the replacement case. Of course, if the replacement case also fails to execute, it can be considered that there is indeed a problem with the object under test, that is, it is determined that the object under test fails the test of the replacement case. It can be seen that this application will re-execute with a new case when a test case fails to execute, which can effectively reduce the failure rate and improve the test reliability. And it should be noted that the above process of re-executing once also does not require manual intervention, ensuring the implementation efficiency of the solution.

[0022] In summary, the solution of this application can effectively improve the generation efficiency of test cases, improve the comprehensiveness of the generated test cases, and can effectively improve the reliability of the test. Brief Description of the Drawings

[0023] In order to more clearly illustrate the embodiments of this application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of this application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0024] Figure 1 It is the flowchart of the implementation of the test method provided by a specific embodiment of the present invention;

[0025] Figure 2 It is the schematic diagram of the functional principle of testing in a specific embodiment of the present invention;

[0026] Figure 3 It is the schematic diagram of the structure of the test system provided by a specific embodiment of the present invention;

[0027] Figure 4 Schematic diagram of the test equipment provided by a specific embodiment of the present invention;

[0028] Figure 5 Schematic diagram of a computer-readable storage medium of the present invention. Specific embodiments

[0029] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the protection scope of the present application.

[0030] It should be noted that in the description of the present application, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in the present application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0031] In order to enable those skilled in the art of the present technology to better understand the solution of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0032] Please refer to Figure 1 , Figure 1 Flowchart of the implementation of the test method provided by a specific embodiment of the present invention. The test method may include the following steps:

[0033] Step S101: Input the background knowledge document reflecting the requirements for generating test cases into a preset language model so that the language model can learn the background knowledge document.

[0034] The specific model type used by the language model can be selected according to actual needs. For example, it can be specifically an LLM (Large Language Model). An LLM is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of language text. It has excellent performance in natural language processing.

[0035] The background knowledge document needs to reflect the requirements for test case generation. For example, it can specifically include content such as the requirements document, design document, interface document, project code library, etc. The requirements document usually carries the functional requirements for the object under test. The design document usually carries the requirements for the format, input, output, etc. of the test cases. The interface document usually carries the information of each interface of the object under test. The project code library usually carries the relevant code information that the generated test cases are allowed to use.

[0036] Of course, in other specific scenarios, the background knowledge document can also carry other aspects of content, as long as it can help the language model learn the relevant knowledge content for test case generation.

[0037] In addition, when the language model learns the background knowledge document, in order to ensure the learning efficiency and quality, the RAG (Retrieval-Augmented Generation) technology is usually used to implement the learning of the background knowledge document. RAG is a method that enhances the performance of the language model (LLM) by combining retrieval and generation technologies. RAG can retrieve relevant information from an external knowledge base and combine it with the base model to generate more accurate and reliable answers.

[0038] Step S102: Generate a test case set including multiple test cases through the language model.

[0039] After the language model learns the background knowledge document, a large number of test cases can be automatically generated through the language model. These generated test cases form the obtained test case set. And it can be seen that since a large number of test cases are automatically generated in this process, it effectively solves the disadvantages of low efficiency in test case generation and relying on an artificial rule base to construct test cases in the traditional solution.

[0040] In a specific implementation manner of the present invention, step S102 may specifically include:

[0041] Generate a test case set including multiple test cases through the language model, and the generated test case set satisfies the set coverage constraint rules;

[0042] Among them, the coverage constraint rule reflects the coverage range of the test cases in the test case set.

[0043] This implementation manner takes into account that through the language model, a large number of test cases can be automatically generated, which can effectively ensure the comprehensiveness of the generated test cases. However, in order to ensure that the generated test cases can cover various situations, in this implementation manner, it is required that the language model satisfy the set coverage constraint rules when generating the test case set.

[0044] Since the coverage constraint rules reflect the coverage scope of the test cases in the test case set, if the test case set meets the set coverage constraint rules, it can effectively ensure that the generated test cases can cover various situations, that is, this implementation further guarantees the comprehensiveness of the generated test cases.

[0045] Of course, the specific content of the coverage constraint rules can be set and adjusted according to actual needs, as long as it can make the test cases in the test case set cover various situations. For example, in a specific implementation of the present invention, the coverage constraint rules can specifically include: the test cases in the test case set cover each interface of the object under test, cover each status code of the object under test, and cover the boundary values of the business rules of each business of the object under test.

[0046] The coverage constraint rules in this implementation require that the test cases in the test case set can cover each interface of the object under test, so as to avoid the situation of interface omission. It also requires covering each status code of the object under test. The status code is usually the HTTP (Hypertext Transfer Protocol) status code, so as to avoid the situation of undefined status. It also requires covering the boundary values of the business rules of each business of the object under test, so as to avoid the situation that some values are not tested. It can be seen that the coverage constraint rules of this implementation can effectively guarantee the comprehensiveness of the generated test cases.

[0047] In addition, in actual applications, when generating a test case set including multiple test cases through a language model, in addition to the coverage constraint rules, it can also be required that the generated test case set meets other set constraint rules. For example, it can be required that each test case uses a unified template, so as to guarantee the reliability of the generated test cases, reduce the occurrence probability of situations such as the test cases themselves being incorrect and unable to be executed, and realize the standardization of the test cases. It can also be required that each test case has its precondition information, so as to guarantee the smooth execution of the corresponding test case. Of course, when a test case has no precondition, its precondition information can be filled in as "no precondition". In addition, according to needs, number information, priority information and other contents can be added to the test cases.

[0048] In a specific implementation of the present invention, step S102 may specifically include:

[0049] Generating a test case set including multiple test cases through a language model, and the test cases in the test case set include interface test cases for interface testing of the object under test and business test cases for business testing of the object under test.

[0050] This implementation mode takes into account that when testing the object under test, it is usually necessary to conduct interface testing and business function testing of the object under test. Therefore, the test cases in the test case set should include two types, so as to effectively implement the interface testing of the object under test and the testing of the business functions of the object under test, and ensure the comprehensiveness of the testing.

[0051] In this implementation mode, the test cases in the test case set include test cases for interface testing of the object under test, which are called interface test cases, and also include test cases for business testing of the object under test, which are called business test cases.

[0052] See Figure 2 , which is a schematic diagram of the functional principle of testing in a specific implementation mode of the present invention. The use case generation shown in Figure 2 means generating a test case set including multiple test cases through a language model. Each interface test case in the test case set can be stored in the interface test case library shown in Figure 2 , and each business test case in the test case set can be stored in the business test case library shown in Figure 2 .

[0053] When generating business test cases, the Playwright tool can usually be used as the implementation solution. Playwright is relatively mature and can be well used for UI operation testing. For example, in a specific scenario, when generating business test cases through a language model, python code can be generated by generating Playwright code, so that it can be executed and verified through python later.

[0054] In addition, it should be noted that for interface test cases, since the content is small, the execution parameters required to execute the interface test cases are usually directly written in the content of the interface test cases, that is, the content of the interface test cases contains the execution parameters required by them. For business test cases, the execution parameters required by them can be directly written in the content of the business test cases according to actual needs, or can be generated and stored separately.

[0055] For example, in a certain scenario, business test cases are generated by a language model, and one or more sets of execution parameters for the business test cases are generated by the language model. Each set of execution parameters may include content such as a function description, the name of the business test case corresponding to this set of execution parameters, a parameter list, an expected execution result, permissions, etc. Then, the business test cases and their corresponding sets of execution parameters are stored in a business test case library. Subsequently, during execution, a set of execution parameters can be obtained from the business test case library, and then the business test case corresponding to this set of execution parameters can be obtained and executed. When each set of execution parameters corresponding to the business test case has been obtained in sequence and the business test case has been executed according to the corresponding execution parameters, the business test case is considered to have been executed completely.

[0056] Step S103: Test the object under test based on the test cases in the test case set.

[0057] When testing the object under test based on the test cases in the test case set, for example, a simple implementation method is to sequentially execute each test case in the test case set, and then obtain the test results of each test case respectively.

[0058] Step S104: When any test case fails to execute, based on the execution information of the test case and the background knowledge document, regenerate a replacement test case for the test case through a language model, and execute the replacement test case.

[0059] It can be understood that when a certain test case is successfully executed, it can be determined that the object under test has passed the test of this test case, so the next test case can be continued to be executed.

[0060] And it should be noted that in the traditional solution, the test execution process is rigid, using fixed script testing, and it is unable to dynamically handle unexpected responses such as interface timeouts and UI element changes. When a certain test case fails to execute, it will be determined that the object under test has not passed this test case, resulting in a high test failure rate.

[0061] In the solution of this application, when a certain test case fails to execute, it will not immediately determine that the object under test has not passed this test case. Instead, based on the execution information of the test case and the background knowledge document, a test case will be regenerated through a language model, which is called a replacement test case for this test case, and this replacement test case will be executed.

[0062] An intelligent agent is an entity that can perceive the environment through sensors or data input, and make autonomous decisions and execute actions based on internal algorithms or learning models to achieve specific goals. In the Figure 2 example, specifically, the interface test case execution agent automatically executes the interface test cases, and the business test case execution agent automatically executes the business test cases.

[0063] Step S105: When the replacement test case is executed successfully, it is determined that the object under test passes the test of the replacement test case; when the replacement test case is executed unsuccessfully, it is determined that the object under test fails the test of the replacement test case.

[0064] For ease of description, a certain test case in the test case set is referred to as test case A. In the solution of this application, after executing test case A and determining that test case A is executed unsuccessfully, a replacement test case for test case A will be regenerated through a language model.

[0065] It can be understood that in some cases, the generated replacement test case may have the same content as the original test case A. Then, after executing the replacement test case, the result is the same as that of test case A, which is also a failure. At this time, it can be regarded that the object under test has a fault, that is, it can be determined that the object under test fails the test of the replacement test case, which is equivalent to the object under test failing the test of test case A.

[0066] In some other cases, even if the background knowledge document is not adjusted, when the language model regenerates a replacement test case based on the execution information of the test case and the background knowledge document, the content of the generated replacement test case may also be different from that of test case A. And after executing the replacement test case, the replacement test case may be executed successfully. Then, at this time, it can be determined that the object under test passes the test of the replacement test case, which is equivalent to the object under test passing the test of test case A.

[0067] And in many cases, the object under test does not have a fault, but the test case fails due to a version update, and the staff has also updated the background knowledge document. Therefore, when the language model generates a replacement test case based on the execution information of the test case and the background knowledge document, since the currently used background knowledge document is already the updated background knowledge document, the obtained replacement test case can be executed successfully, which is equivalent to the object under test passing the test of test case A, thereby effectively reducing the test failure rate of the solution of this application.

[0068] In addition, some accidental situations may also cause test failures. In the solution of this application, through two tests (the test of the original test case A and the test of its corresponding replacement test case), the interference of accidental situations can be effectively reduced, thereby effectively reducing the test failure rate of the solution of this application.

[0069] In practical applications, when the replacement use case is successfully executed, the original test case corresponding to the replacement use case can be deleted, that is, the replacement use case is used to replace the corresponding test case in the test case set. Of course, when the replacement use case also fails, it can be set to retain the original test case corresponding to the replacement use case in the test case set, or it can be set to use the replacement use case to replace the corresponding test case in the test case set, which does not affect the implementation of the present invention.

[0070] In addition, for the test cases in the test case set, the staff is allowed to perform maintenance management according to actual needs, including management such as modification, deletion, and addition of test cases.

[0071] In a specific implementation manner of the present invention, it may further include:

[0072] When the replacement use case fails and the replacement use case belongs to the business test use case for the object under test, analyze the execution result of the replacement use case to obtain the failure analysis result.

[0073] In this implementation manner, it is considered that for the case where the object under test fails the interface test case, usually there is a problem with the corresponding interface of the object under test, so usually no detailed failure analysis is required. If the failed replacement use case is a business test case, since there are multiple abnormal reasons, in order to assist the staff in subsequent abnormal reason analysis, in this implementation manner, the execution result of the replacement use case will be automatically analyzed to obtain the failure analysis result.

[0074] Of course, the specific analysis method can be set and adjusted according to actual needs. For example, in a specific implementation manner of the present invention, analyzing the execution result of the replacement use case to obtain the failure analysis result may specifically include:

[0075] Based on the execution result of the replacement use case, determine the relationship matrix of the execution result;

[0076] Among them, the column dimension of the relationship matrix represents different operation events during the execution of the replacement use case, and the row dimension of the relationship matrix represents the call information items under the corresponding operation events during the execution of the replacement use case;

[0077] Based on the relationship matrix, determine the correlation degree between each interface and the failure of the replacement use case execution, and take the interface with the highest correlation degree as the abnormal interface that causes the replacement use case execution to fail;

[0078] Take the abnormal interface as the obtained failure analysis result.

[0079] Compared with the single and inefficient manual fault analysis method in the traditional solution, in this implementation method, analysis is realized based on the relationship matrix. Considering the correlation degree between different interfaces and execution failures, a more comprehensive analysis can be achieved, and the abnormal interfaces can be located more accurately.

[0080] Specifically, the row dimension of the relationship matrix represents the call information under the corresponding operation events during the execution of the replacement use case. For example, in one row, the call information can specifically include metadata such as the API path, response status code, and elapsed time under a certain operation event. The column dimension of the relationship matrix represents different operation events during the execution of the replacement use case, that is, different rows correspond to different operation events.

[0081] After obtaining the relationship matrix, it is necessary to determine the correlation degree between each interface and the execution failure of the replacement use case. For example, a more convenient and reliable implementation method is to match through a time window. The sliding window algorithm (such as a window size of 500 ms) is used to align the timestamps of the interface response and the operation event, and then methods such as the Pearson correlation coefficient can be used to evaluate the statistical correlation between the interface status and the execution failure of the replacement use case, so as to obtain the correlation degree between each interface and the execution failure of the replacement use case. Finally, the interface with the highest correlation degree is used as the abnormal interface that causes the execution failure of the replacement use case, that is, the abnormal interface is used as the obtained failure analysis result.

[0082] In practical applications, the log information of the abnormal interface can also be obtained, together with the above failure analysis results, as the diagnostic report output for this replacement use case to assist the subsequent work of the staff.

[0083] In addition, for the determined abnormal interface, horizontal verification and vertical verification can also be carried out. Horizontal verification refers to checking and counting the abnormal performance of this interface in other test cases and incorporating them into the above diagnostic report. Vertical verification refers to comparing the relationship matrix of this replacement use case in the historical normal scenario and putting the comparative analysis results into the above diagnostic report.

[0084] In Figure 2 the example of , specifically, the failure analysis of this implementation method is based on the intelligent analysis agent, and the abnormal interface can be located more accurately, that is, a more accurate failure reason is provided. <s

[0085] In a specific implementation manner of the present invention, testing the object to be tested based on the test cases in the test case set may specifically include:

[0086] Testing the object to be tested based on the test cases in the test case set in the development environment, test environment, and production environment respectively.

[0087] This implementation mode takes into account the environmental adaptation defects of traditional test schemes and the lack of in-depth monitoring capabilities for interface behavior differences. In this implementation mode of the present application, when testing the object under test based on the test cases in the test case set, the tests will be carried out in different environments respectively. Specifically, the tests of the object under test will be carried out based on the test cases in the test case set in the development environment, test environment, and production environment respectively, which is conducive to discovering some problem situations in a specific environment.

[0088] For example, if a certain test case is executed successfully in both the test environment and the production environment, but in the development environment, both the test case and the regenerated alternative test case fail to execute, it may be that some problems that are difficult to discover and have a relatively deep hidden degree occur after the object under test is deployed in the development environment.

[0089] In practical applications, after testing the object under test based on the test cases in the test case set in the development environment, test environment, and production environment respectively, the test results of the same test case in different environments can be analyzed. When the situation occurs that the same test case is executed successfully in some environments but fails to execute in some environments, comparative analysis can be carried out and the analysis results can be output to assist the staff in subsequent processing. In Figure 2 the example, specifically, the comparative analysis is implemented based on a comparative analysis agent.

[0090] In a specific implementation mode of the present invention, testing the object under test based on the test cases in the test case set includes:

[0091] Determine the execution period;

[0092] After any execution period arrives, determine the current weight of each test case;

[0093] In the current execution period, based on the current weight of each test case, select a specified number of test cases from the test case set, and sequentially execute each of the selected test cases on the object under test.

[0094] This implementation mode takes into account that since the test case set includes a large number of test cases, if each test case in the test case set is executed once, it will take a longer time and is not conducive to conveniently and timely determining the abnormal situation of the object under test.

[0095] For this implementation, it is considered that an execution period can be set, and of course, the value of the execution period can be set and adjusted as needed. In each execution period, not all test cases in the test case set will be executed, but only a part of them, that is, a specified number of test cases are selected for execution. The specified number can be a value or a percentage.

[0096] Moreover, which specific test cases to execute depends on the current weights of each test case in the test case set.

[0097] That is to say, in the current execution period, based on the current weights of each test case, a specified number of test cases are selected from the test case set, and then the selected test cases are executed. And it can be understood that since these test cases are selected according to the current weights, it shows that the currently selected test cases are relatively reasonable choices, which is conducive to testing the abnormal conditions of the object under test.

[0098] In addition, it should be noted that, in combination with the above description, for the implementation of testing the object under test based on the test cases in the test case set in different environments, it can be understood that in different environments, different execution periods can be configured according to needs. For example, in the development environment, the execution period is set to 8 hours, that is, every 8 hours, in the development environment, the selected test cases in this round are executed. For example, in the test environment, the execution period is set to 12 hours, that is, every 12 hours, in the test environment, the selected test cases in this round are executed. That is to say, in different environments, the test cases in the test case set can be executed respectively, and their own execution periods can be set without affecting each other.

[0099] Furthermore, in a specific implementation manner of the present invention, for the test cases in the test case set, the current weight of a test case is a weight determined based on the historical success rate of the test case, the importance coefficient of the test case, and the most recent modification time of the test case.

[0100] This implementation considers that the historical success rate of the test case should be set as one of the factors affecting the weight, so that it is possible to set, according to needs, to preferentially execute test cases with a high historical success rate or test cases with a low historical success rate. In addition, whether the test case has been recently modified and the importance of the test case itself should also be set as factors affecting the weight, so as to effectively ensure the rationality of the obtained test cases.

[0101] Furthermore, in a specific implementation manner of the present invention, for the test cases in the test case set, the current weight of a test case is expressed as:

[0102] W = a×(1 - SR) + b×F + c×T;

[0103] Wherein, W is the current weight of the test case, a, b, and c are all set coefficients, and a + b + c = 1, SR is the historical success rate of the test case, F is the importance coefficient of the test case, and T is the time coefficient obtained based on the most recent modification time of the test case.

[0104] This implementation mode gives a specific implementation method for determining the current weight of the test case. It can be seen that the larger the historical success rate SR of the test case, the lower the current weight of the test case. This is because the larger the historical success rate SR of the test case, the higher the probability that the object under test passes the test of this test case. Therefore, the necessity of repeatedly executing this test case is lower, and thus the weight of this test case is lower.

[0105] The importance coefficient of the test case reflects the importance of the test case itself. The larger its value, the more important the test case is. Therefore, the weight of this test case is higher, that is, this test case needs to be executed more frequently.

[0106] T is the time coefficient obtained based on the most recent modification time of the test case. And it can be understood that when the test case has been recently modified, that is, the most recent modification time is short, then the value of T should be large, so that the weight of this test case is higher. That is, for the test case that has been recently modified, the test of this test case should be carried out in a timely manner.

[0107] In a specific implementation mode of the present invention, it may further include:

[0108] According to the first instruction, select each test case pointed to by the first instruction from the test case set as the obtained single execution use case set;

[0109] Whenever the set trigger condition is satisfied, sequentially execute each test case in the single execution use case set.

[0110] This implementation mode takes into account that according to the above execution cycle, executing a certain number of test cases in each round may cause some test cases to be executed less frequently, or even not executed for a long time. For some test cases, the staff hopes that they need to be executed every once in a while, or whenever a certain condition is met. In this regard, in this implementation mode, it is allowed to receive the first instruction, and then select each test case pointed to by the first instruction from the test case set as the obtained single execution use case set. Subsequently, as long as the set trigger condition is satisfied, each test case in the single execution use case set will be sequentially executed. For example, the set trigger condition is 8:00 am on Monday. At this time, it is equivalent to executing each test case in the single execution use case set every once in a while.

[0111] Of course, other triggering conditions can also be set in other specific embodiments. For example, the triggering condition can be manually triggered by the staff, which means that whenever the staff has a need, each test case in the single-execution test case set will be immediately executed in sequence.

[0112] In a specific embodiment of the present invention, it may further include:

[0113] When it is detected that the background knowledge document has changed, delete each test case in the test case set associated with the changed part of the background knowledge document, and regenerate the corresponding test cases based on the language model;

[0114] Execute each of the regenerated test cases in sequence.

[0115] This embodiment takes into account that the change situation of the background knowledge document can be monitored, and if there is a change in the background knowledge document, many of the originally generated test cases will become invalid. Therefore, each test case in the test case set associated with the changed part of the background knowledge document will be immediately deleted, and the corresponding test cases will be regenerated based on the language model and then executed.

[0116] In addition, in some cases, when it is detected that the background knowledge document has changed, considering that the language model can be used to conveniently and quickly generate test cases, it is also possible to choose to delete all the test cases in the test case set, and regenerate a test case set based on the changed background knowledge document based on the language model.

[0117] Through this application, after inputting the background knowledge document used to reflect the requirements for test case generation into a preset language model, the language model can learn the background knowledge document. Since the test case set including multiple test cases is generated by the language model, the generation efficiency is very high, and a large number of test cases can be generated quickly, thereby ensuring the comprehensiveness of the generated test cases, and this process does not require manual intervention. Further, after testing the object under test based on the test cases in the test case set, when a certain test case fails to execute, it does not directly end and determine that the object under test fails the test of this test case. Instead, based on the execution information of the test case and the background knowledge document, a replacement case for the test case will be regenerated through the language model and the replacement case will be executed. This is because when a certain test case fails to execute, it may be that the test case itself is unreasonable, or it is due to the version update of the object under test, other accidental factors, etc. rather than a problem with the object under test. Therefore, regenerating a replacement case for the test case through the language model is equivalent to regenerating a test case to repeat the verification. If the replacement case can be executed successfully, it can be determined that the object under test has passed the test of the replacement case. Of course, if the replacement case also fails to execute, it can be considered that there is indeed a problem with the object under test, that is, it is determined that the object under test fails the test of the replacement case. It can be seen that this application will re-execute with a new case when a test case fails to execute, which can effectively reduce the failure rate and improve the test reliability. And it should be noted that the above process of re-executing once also does not require manual intervention, ensuring the implementation efficiency of the solution.

[0118] In summary, the solution of this application can effectively improve the generation efficiency of test cases, improve the comprehensiveness of the generated test cases, and effectively improve the reliability of the test.

[0119] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0120] The embodiment of this application also provides a test system, which can be referred to Figure 3 , including:

[0121] A background knowledge document learning module 301, configured to input a background knowledge document used to reflect the requirements for test case generation into a preset language model, so that the language model learns the background knowledge document;

[0122] [[ID=*17]]A test case generation module 302, configured to generate a test case set including multiple test cases through the language model;

[0123] The test execution module 303 is used to test the object under test based on the test cases in the test case set;

[0124] The test case regeneration execution module 304 is used to, when any test case fails to execute, regenerate a replacement case for the test case through a language model based on the execution information of the test case and the background knowledge document, and execute the replacement case;

[0125] The replacement case execution result determination module 305 is used to, when the replacement case executes successfully, determine that the object under test has passed the test of the replacement case, and when the replacement case fails to execute, determine that the object under test has not passed the test of the replacement case.

[0126] In a specific embodiment of the present invention, the test case generation module 302 is specifically used for:

[0127] Generating a test case set including multiple test cases through a language model, and the generated test case set satisfies the set coverage constraint rules;

[0128] Among them, the coverage constraint rules reflect the coverage range of the test cases in the test case set.

[0129] In a specific embodiment of the present invention, the coverage constraint rules include: the test cases in the test case set cover each interface of the object under test, cover each status code of the object under test, and cover the business rule boundary values of each business of the object under test.

[0130] In a specific embodiment of the present invention, the test case generation module 302 is specifically used for:

[0131] Generating a test case set including multiple test cases through a language model, and the test cases in the test case set include interface test cases for interface testing of the object under test and business test cases for business testing of the object under test.

[0132] In a specific embodiment of the present invention, it further includes a failure analysis module, which is used for:

[0133] When the replacement case fails to execute and the replacement case belongs to the business test case for business testing of the object under test, analyzing the execution result of the replacement case and obtaining a failure analysis result.

[0134] In a specific embodiment of the present invention, the failure analysis module is specifically used for:

[0135] When the replacement case fails to execute and the replacement case belongs to the business test case for business testing of the object under test, determining a relationship matrix of the execution result based on the execution result of the replacement case;

[0136] Among them, the column dimension of the relationship matrix represents different operation events during the execution of the replacement use case, and the row dimension of the relationship matrix represents each call information under the corresponding operation event during the execution of the replacement use case;

[0137] Based on the relationship matrix, determine the correlation degree between each interface and the failure of the replacement use case execution, and use the interface with the highest correlation degree as the abnormal interface that causes the replacement use case execution to fail;

[0138] Use the abnormal interface as the obtained failure analysis result.

[0139] In a specific embodiment of the present invention, the test execution module 303 is specifically used for:

[0140] Based on the test cases in the test case set, test the object to be tested in the development environment, test environment, and production environment respectively.

[0141] In a specific embodiment of the present invention, the test execution module 303 is specifically used for:

[0142] Determine the execution period;

[0143] After any execution period arrives, determine the current weight of each test case;

[0144] In the current execution period, based on the current weight of each test case, select a specified number of test cases from the test case set, and sequentially execute each of the selected test cases on the object to be tested.

[0145] In a specific embodiment of the present invention, for the test cases in the test case set, the current weight of the test case is the weight determined based on the historical success rate of the test case, the importance coefficient of the test case, and the most recent modification time of the test case.

[0146] In a specific embodiment of the present invention, for the test cases in the test case set, the current weight of the test case is expressed as:

[0147] W = a×(1 - SR) + b×F + c×T;

[0148] Among them, W is the current weight of the test case, a, b, and c are all set coefficients, and a + b + c = 1, SR is the historical success rate of the test case, F is the importance coefficient of the test case, and T is the time coefficient obtained based on the most recent modification time of the test case.

[0149] In a specific embodiment of the present invention, it further includes a single execution use case set execution module for:

[0150] According to the first instruction, each test case pointed to by the first instruction is selected from the test case set as the obtained single-execution use case set;

[0151] Whenever the set trigger condition is satisfied, each test case in the single-execution use case set is executed in sequence.

[0152] In a specific embodiment of the present invention, it further includes a background knowledge document change processing module for:

[0153] When it is detected that the background knowledge document has changed, each test case associated with the changed part of the background knowledge document in the test case set is deleted, and corresponding test cases are regenerated based on the language model;

[0154] Each regenerated test case is executed in sequence.

[0155] Corresponding to the above method and system embodiments, the embodiments of the present invention also provide a test device, a computer-readable storage medium, and a computer program product, which can be correspondingly referred to with the above.

[0156] See Figure 4 As shown, the device may include:

[0157] A memory 401 for storing computer programs;

[0158] A processor 402 for executing the computer program to implement the steps of the test method in any of the above embodiments.

[0159] The computer program product includes computer programs / instructions, and when the computer programs / instructions are executed by the processor, the steps of the test method in any of the above embodiments are implemented.

[0160] Refer to Figure 5 , a computer program 51 is stored on the computer-readable storage medium 50, and when the computer program 51 is executed by the processor, the steps of the test method in any of the above embodiments are implemented. The computer-readable storage medium 50 mentioned here includes RAM (Random Access Memory), memory, ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), registers, hard disks, removable disks, or any other form of storage medium known in the technical field.

[0161] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0162] The above has introduced in detail a testing method provided by this application. Specific examples are used herein to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A testing method, characterized in that: Including: Input a background knowledge document for reflecting test case generation requirements into a preset language model so that the language model learns the background knowledge document; Generate a test case set including multiple test cases through the language model; Test the object under test based on the test cases in the test case set; When any one of the test cases fails, based on the execution information of the test case and the background knowledge document, regenerate a replacement case for the test case through the language model and execute the replacement case; When the replacement case is successfully executed, determine that the object under test has passed the test of the replacement case. When the replacement case fails, determine that the object under test has not passed the test of the replacement case.

2. The test method according to claim 1, characterized in that, Generating a test case set including multiple test cases through the language model includes: Generate a test case set including multiple test cases through the language model, and the generated test case set satisfies the set coverage constraint rules; Wherein, the coverage constraint rules reflect the coverage range of the test cases in the test case set.

3. The testing method according to claim 2, characterized in that, The coverage constraint rules include: the test cases in the test case set cover each interface of the object under test, cover each status code of the object under test, and cover the business rule boundary values of each business of the object under test.

4. The test method according to claim 1, wherein, Generating a test case set including multiple test cases through the language model includes: Generate a test case set including multiple test cases through the language model, and the test cases in the test case set include interface test cases for interface testing of the object under test and business test cases for business testing of the object under test.

5. The test method according to claim 4, wherein Also including: When the replacement case fails and the replacement case belongs to the business test case for the business test of the object under test, analyze the execution result of the replacement case and obtain a failure analysis result.

6. The test method according to claim 5, characterized in that, Analyzing the execution result of the replacement case and obtaining a failure analysis result includes: Based on the execution result of the replacement case, determine the relationship matrix of the execution result; Wherein, the column dimension of the relationship matrix represents different operation events during the execution of the replacement case, and the row dimension of the relationship matrix represents each call information under the corresponding operation event during the execution of the replacement case; Based on the relationship matrix, determine the association degree between each interface and the failure of the replacement case execution, and use the interface with the highest association degree as the abnormal interface causing the failure of the replacement case execution; Use the abnormal interface as the obtained failure analysis result.

7. The testing method according to claim 1, wherein Testing the object under test based on the test cases in the test case set includes: Test the object under test based on the test cases in the test case set in the development environment, test environment, and production environment respectively.

8. The test method according to any one of claims 1 to 7, characterized in that Testing the object under test based on the test cases in the test case set includes: Determine the execution cycle; After any one of the execution cycles arrives, determine the current weight of each of the test cases. In the current execution cycle, based on the current weight of each of the test cases, select a specified number of the test cases from the test case set, and sequentially execute each of the selected test cases on the object under test.

9. The test method according to claim 8, characterized in that, For the test cases in the test case set, the current weight of a test case is a weight determined based on the historical success rate of the test case, the importance coefficient of the test case, and the most recent modification time of the test case.

10. The test method according to claim 9, wherein For the test cases in the test case set, the current weight of a test case is expressed as: W = a×(1 - SR) + b×F + c×T; where W is the current weight of the test case, a, b, and c are all set coefficients, and a + b + c = 1, SR is the historical success rate of the test case, F is the importance coefficient of the test case, and T is the time coefficient obtained based on the most recent modification time of the test case.

11. The test method according to claim 8, wherein It further includes: According to a first instruction, select each of the test cases pointed to by the first instruction from the test case set as the obtained single execution use case set. Whenever a set trigger condition is satisfied, sequentially execute each of the test cases in the single execution use case set.

12. The test method according to claim 8, characterized in that It further includes: When it is detected that the background knowledge document has changed, delete each of the test cases in the test case set that is associated with the changed part of the background knowledge document, and regenerate corresponding test cases based on the language model. Sequentially execute each of the regenerated test cases.

13. A test system, characterized in that, It includes: A background knowledge document learning module for inputting a background knowledge document for reflecting test case generation requirements into a preset language model so that the language model learns the background knowledge document. A test case generation module for generating a test case set including multiple test cases through the language model. A test execution module for testing the object under test based on the test cases in the test case set. A test case regeneration execution module for, when any one of the test cases fails to execute, regenerating a replacement use case for the test case through the language model based on the execution information of the test case and the background knowledge document, and executing the replacement use case. A replacement use case execution result determination module for, when the replacement use case executes successfully, determining that the object under test has passed the test of the replacement use case, and when the replacement use case fails to execute, determining that the object under test has not passed the test of the replacement use case.

14. A testing device, characterized in that, It includes: A memory for storing a computer program. A processor for executing the computer program to implement the steps of the test method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of the test method according to any one of claims 1 to 12.

Images