Test case analysis method, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a test case analysis method, electronic equipment and a storage medium. The test case analysis method comprises the following steps: acquiring test failure case information; and sending test failure case information to a preset large model, performing prompt interaction with the large model, and issuing a failure analysis task to the large model, so that the large model analyzes a failure reason of the test failure case based on the test failure case information, and obtains an analysis result. According to the scheme of the embodiment, the labor cost is saved, the analysis speed is increased, the analysis efficiency is improved, and the comprehensiveness and accuracy of analysis can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of automation testing, and in particular, to a test case analysis method, an electronic device, and a storage medium. BACKGROUND

[0002] In the field of wireless communication network management, performing test case automation testing is an important means to verify the deliverability of functions. Due to the continuous change of the wireless network management test environment, the test modules are numerous, the test cases are large, and the integration needs to be performed multiple times a day. Therefore, a large number of failed test cases need to be analyzed multiple times a day. The previous method mainly relies on manual analysis. Manual analysis is time-consuming, the test environment cannot be retained for a long time, the test data cannot be persisted, the test experience cannot be automatically inherited, and it is impossible to complete the analysis every time. Therefore, a method is needed to quickly analyze the reasons for test case failure, improve the accuracy of the analysis results, reduce the workload of manual analysis, quickly locate the failure reasons, and quickly solve the problems. SUMMARY

[0003] Embodiments of the present disclosure provide a test case analysis method, an electronic device, and a storage medium.

[0004] In a first aspect, embodiments of the present disclosure provide a test case analysis method, which includes:

[0005] obtaining test failure case information;

[0006] sending the test failure case information to a preset large model, performing prompt interaction with the large model, issuing a failure analysis task to the large model, and analyzing the failure reasons of the test failure case based on the test failure case information by the large model to obtain an analysis result.

[0007] In a second aspect, embodiments of the present disclosure further provide an electronic device, which includes:

[0008] one or more processors;

[0009] a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the test case analysis method;

[0010] one or more input / output (I / O) interfaces connected between the processor and the memory and configured to implement information interaction between the processor and the memory.

[0011] In a third aspect, embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the test case analysis method.

[0012] In a fourth aspect, the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the test case analysis method.

[0013] The embodiment of the present disclosure sends the test failure case information to the preset large model, interacts with the large model for prompting, issues a failure analysis task to the large model, and analyzes the failure reason of the test failure case based on the test failure case information by the large model to obtain an analysis result. The embodiment of the present disclosure realizes the analysis of the test failure case by the large model instead of manual analysis, saves the labor cost, accelerates the analysis speed, improves the analysis efficiency, can quickly locate the failure reason, and thus quickly solves the problem of test case failure. In addition, the analysis of the failure reason by the large model can improve the comprehensiveness and accuracy of the analysis. BRIEF DESCRIPTION OF DRAWINGS

[0014] In the drawings of the embodiments of the present disclosure:

[0015] Figure 1 The test case analysis method flowchart provided by the embodiments of the present disclosure;

[0016] Figure 2 The test case analysis method schematic diagram provided by the embodiments of the present disclosure;

[0017] Figure 3 The block diagram of the electronic device provided by the embodiments of the present disclosure. DETAILED DESCRIPTION

[0018] In order for those skilled in the art to better understand the technical solutions of the present disclosure, the communication perception data processing method and the computer readable storage medium provided by the embodiments of the present disclosure will be described in detail below with reference to the drawings.

[0019] The embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings, but the embodiments shown can be embodied in different forms and the present disclosure should not be interpreted as being limited to the embodiments set forth below. On the contrary, the purpose of providing these embodiments is to make the present disclosure thorough and complete and to enable those skilled in the art to fully understand the scope of the present disclosure.

[0020] The drawings of the embodiments of the present disclosure are used to provide further understanding of the embodiments of the present disclosure and constitute a part of the specification, which is used to explain the present disclosure together with the detailed embodiments and does not constitute a limitation of the present disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing the detailed embodiments with reference to the drawings.

[0021] The present disclosure can be described with reference to plan views and / or sectional views by means of ideal schematic views of the present disclosure. Therefore, the example views can be modified according to manufacturing techniques and / or tolerances.

[0022] In the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0023] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. The term "and / or" as used in the present disclosure includes any and all combinations of one or more of the associated listed items. The singular forms "a" and "the" as used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising", "made of" as used in the present disclosure specify the presence of the features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0024] Unless otherwise defined, all terms used in the present disclosure, including technical and scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal sense, unless the present disclosure explicitly so defines.

[0025] In the field of wireless communication network management, performing use case automatic test is an important means to verify the deliverability of functions. Due to the continuous change of the wireless network management test environment, the test module is multiple, the use case amount is large, and the daily integration needs to be performed multiple times. Therefore, a large number of failed use cases need to be analyzed every day. The previous method mainly relies on manual analysis. Relying on manual analysis is time-consuming, the test environment cannot be kept for a long time, the test data cannot be persisted, the test experience cannot be automatically inherited, and it is impossible to complete all analysis every time.

[0026] The embodiment of the present disclosure provides a test case analysis method, which can send test failure use case information to a preset large model, interact with the large model, issue a failure analysis task to the large model, and analyze the failure reason of the test failure use case based on the test failure use case information by the large model to obtain an analysis result. The embodiment scheme realizes the analysis of the test failure use case by the large model instead of manual analysis, saves the labor cost, accelerates the analysis speed, improves the analysis efficiency, can quickly locate the failure reason, and thus quickly solves the problem of test case failure. In addition, the failure reason analysis by the large model can improve the comprehensiveness and accuracy of the analysis.

[0027] The embodiment scheme of the present disclosure can be applied to, but is not limited to, the automatic test of software in the field of wireless communication network management, and can be applied to wireless network management products. The embodiment scheme of the present disclosure can be implemented by a virtual machine and a container,

[0028] The embodiment scheme of the present disclosure will be described in detail below.

[0029] The embodiment of the present disclosure provides a test case analysis method, as shown in Figure 1 、 Figure 2 The embodiment of the present disclosure provides a test case analysis method, as shown in

[0030] S11, obtaining test failure case information;

[0031] S12, sending the test failure case information to a preset large model, prompting interaction with the large model, issuing a failure analysis task to the large model, and analyzing the failure reason of the test failure case based on the test failure case information by the large model to obtain an analysis result.

[0032] The large model of the embodiment of the present application refers to a large language model (LLM). The large language model (LLM) is a deep learning model trained based on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and process various natural language tasks such as text summarization, question answering, and translation.

[0033] In the embodiment of the present disclosure, a test module, a task storage module and a large model analysis module can be set to respectively execute case testing, manage test task information and analyze the failure reason of the test failure case.

[0034] In the embodiment of the present disclosure, the test module can provide a test environment and an automated test tool.

[0035] In the embodiment of the present disclosure, the test module installs the operating system, platform and application required for testing based on the test environment. The prerequisite for executing the test case is to update the test environment to the current latest version, so that the test environment is available.

[0036] In the embodiment of the present disclosure, the automated test tool stores and schedules test tasks, which can include the necessary conditions for implementing testing: selecting a test environment, updating a test object, selecting an executed test case, and customizing an execution plan of a test task. The test case needs to be executed on the test environment, and the test case is stored in a preset case library and written by a product team. The test case needs test data, which is also prepared by the product team. After the test case is completely executed, the task storage module is notified to collect test-related information, which can be information snapshot saved to the task storage module.

[0037] In the embodiment of the present disclosure, the task storage module is mainly used for managing test task information, for example, including but not limited to task execution time (such as case execution start time, case execution completion time, etc.), execution result, test content, used environment information and any information in the task execution process. The task storage module can manage the test task information of all test cases.

[0038] In the embodiments of the present disclosure, the test failure case information can include any one or more of the following: test failure case name, operation log information, failure keyword information, failure message information, service information, version information, change information, and environment information, etc.

[0039] In the embodiments of the present disclosure, the operation log information can include but is not limited to: operation object, operation time and operation person, and the operation log information can be used to analyze the case failure caused by concurrent operation. The environment information can include but is not limited to: environment IP (network protocol) information, used PAAS (Platform as a Service) version, used platform version, and installed service information on the environment. The service information can include but is not limited to: service name, service version, and service exception log information of the current test, and the service name and service version are used for service change analysis, and the service exception log information such as service restart and service suspension is used to analyze the influence of service exception on the test and record the time point of service exception occurrence. The change information can include but is not limited to: changed service, changed test case, and changed version.

[0040] In the embodiments of the present disclosure, the user can view all case information in the interface of the task storage module, including case execution result, case name, case execution start time, case execution completion time, etc., and if it is a test failure case, the failure keyword information and failure message information (i.e. failure Message information) can be viewed.

[0041] In the embodiments of the present disclosure, through the setting of the task storage module, all case information can be saved persistently, case statistical analysis can be performed, and historical analysis records can be traced back.

[0042] In the embodiments of the present disclosure, the large model analysis module can call the test failure case information corresponding to the test failure case to be analyzed from the preset test set in the task storage module; the test set can include the test failure case information of all test failure cases to be analyzed, and can also include the expected analysis result (i.e., the expected failure cause analysis result) corresponding to each test failure case. The expected analysis result can be the failure cause analysis result given by the test expert according to experience for each test failure case, and the expected analysis result can be used to detect whether the currently selected large model is reasonable when the large model analyzes the failure cause of the test failure case according to the content of the expert knowledge base, and whether the content of the expert knowledge base is correct and complete. In order to reselect the large model in the case where the selected large model is unreasonable, and update and improve the content of the expert knowledge base in the case where the content of the expert knowledge base is incorrect and incomplete.

[0043] In the embodiments of the present disclosure, the test set can be maintained according to the changes of the test object, and the test set is responsible for the test expert, which is a transmission method of the test analysis experience of the test expert.

[0044] In the embodiments of the present disclosure, the expert knowledge base can include a plurality of historical analysis experience information; each historical analysis experience information includes a historical test failure case and a failure cause corresponding to the historical test failure case.

[0045] In the embodiments of the present disclosure, the expert knowledge base includes historical test failure cases with accurate failure cause analysis and historical analysis experience information thereof, which is the experience collection of the test expert. The content of the expert knowledge base is saved in a format that can be understood by the large model, so as to facilitate the large model to directly call and use the content of the expert knowledge base.

[0046] In the embodiments of the present disclosure, the large model analysis module can maintain two agents, i.e., a first agent and a second agent. The first agent can call the test failure case information to be analyzed from the test set in the task storage module, and the large model issues a failure analysis task, so that the large model completes the failure cause analysis of the test failure case. The second agent can verify the analysis result of the large model, so as to ensure the comprehensiveness and correctness of the failure cause analysis of each test failure case.

[0047] In the embodiments of the present disclosure, the execution processes of the test module, the task storage module and the large model analysis module are introduced in detail as follows.

[0048] In the embodiments of the present disclosure, the running process of the test module includes:

[0049] Step 21, start the test task.

[0050] The test task is started in a timed manner, for example, once per hour, to ensure that the task is uninterrupted.

[0051] The test task includes test service updates, test cases, test execution strategies, and test environment configurations. The test task manages the start and stop of each part, and does not allow execution if the conditions are not met.

[0052] Step 22: The test task execution needs to match the available test environment.

[0053] The initialized test environment is prepared in advance and matches the current test. If the test task cannot find a matching test environment, the test task cannot be started until there is a free available test environment. After each test task is executed, the test environment is released, and the queued test task is scheduled to continue execution on the released test environment.

[0054] Step 23: Update the test environment.

[0055] This step is to ensure that each test execution is performed on the latest service. The test environment update is actively checked, and the difference between the service version on the environment and the available test version is compared. If there is a new one, update it. If there is no update, do not perform this step. This can save the time of environment update.

[0056] Step 24: Execute the test case.

[0057] After the test environment update step is successfully executed, the test case can be executed. Before executing the test case, the latest test case and the data required by the test case are updated from the code repository to ensure that each test case executed is the current latest version of the test case.

[0058] Step 25: After the test case execution is completed, notify the task storage module to obtain the test task information of the test task execution.

[0059] In the embodiments of the present disclosure, the running process of the task storage module includes:

[0060] Step 31: Start the collection task.

[0061] After the task storage module receives the message from the test module, the collection task is started. The collection task is also a pipeline composed of multiple tasks, and the collection task can be executed on a virtual machine, responsible for collecting relevant information of the test execution.

[0062] Step 32: Collect test task information.

[0063] After the test module completes the test task, there are the name of the task, the execution result of the task, time information of the task execution, and various information. The next step will collect the data corresponding to these information. The data corresponding to the test task is equivalent to the parent data of all data, and can concatenate the data snapshots of the entire test task.

[0064] In step 33, the environment information, operation log information, service information, and change information of the test task are collected.

[0065] The environment information includes but is not limited to the IP of the test, the version of the PAAS used, and the version of the platform used.

[0066] The operation log information corresponds to the data derived from the execution result of the test case. The test case execution will operate the function on the test environment, and obtain the operation log information in the test execution period, including the operation object, operation time, operator, etc. This part of data is used to analyze the failure of the use case caused by the operation object conflict in the use case concurrent scenario.

[0067] Collecting service information is to take a snapshot of the test service, save the service name and service version of this test, etc. The service information can be used for service change analysis. The service execution exception log will be saved together with the service information. The service restart and service suspension information are used to analyze the impact of service exceptions on the test. The time point of service exception occurrence needs to be recorded.

[0068] After all the information is collected, the test task information can be used to compare the test environment IP and other information as keywords with the last execution. The changes of the PAAS version, platform version, and service version (which can be regarded as version change information in the change information) are compared. The change content can be matched to the requirement change, which is used to analyze whether the failed use case is caused by the version change.

[0069] In step 34, other information related to the test is collected.

[0070] The other information can include but is not limited to keyword information, such as failure keyword information.

[0071] The use case test of the embodiment of the present disclosure can be executed using the RobotFramework (automation framework) framework. After the test case execution is completed, an xml format test report file is generated. The report data needs to be stored in the task storage module. The execution of the test case can also include the execution log of the keyword developed using the Python language. This part of the log is stored in the test module. Based on this part of the log, the failure part (i.e. failure keyword information) can be filtered out and stored in the task storage module.

[0072] The test data required by the test case and the use case execution is obtained from the code library, and the difference part (which can be regarded as use case change information and / or use case data change information in change information) between the data obtained in this test and the data obtained in the last test is saved to the task storage module, which is used to analyze whether the failed use case is caused by use case or use case data change.

[0073] In step 35, the first agent (which can be represented as Agent-1) can call the test failure case information from the task storage module and send it to the large model. The large model analyzes the failure reason of the test case in combination with the expert knowledge base and the prompt word, and the failure reason schedules the second agent (which can be represented as Agent-2) to verify again.

[0074] The word "Agent" originates from the Latin word "Agere", which means "to do". In the context of LLM, Agent can be understood as an intelligent agent that can independently understand, plan decisions, and execute complex tasks.

[0075] In step 36, after the large model analyzes the failure reason of the test case, it sends the analyzed failure reason to the task storage module for saving. The failure reason sent includes what causes the failure, whether Agent-2 verifies successfully, if the verification fails, the test expert needs to analyze manually, and after the analysis, the expert knowledge base needs to be updated. These contents will be described in detail in the large model application and returned to the task storage module. After returning the failure reason, the task of the entire task storage module ends.

[0076] In the embodiments of the present disclosure, the running process of the large model analysis module is introduced as follows.

[0077] In the embodiments of the present disclosure, the running process of the large model analysis module includes the aforementioned steps S11-S12.

[0078] In the embodiments of the present disclosure, obtaining the test failure case information can include: calling the test failure case information corresponding to the test failure case to be analyzed from the preset test set.

[0079] In the embodiments of the present disclosure, the test set can include, but is not limited to, test failure case information of a plurality of test failure cases to be analyzed. The foregoing has described the test set in detail, which will not be repeated here.

[0080] In the embodiments of the present disclosure, the method can further include:

[0081] obtaining a preset selection use case and at least one large model to be selected; the selection use case is a test failure case used for large model selection;

[0082] Input the selection use case into the large model to be selected, and analyze the failure reason of the selection use case by the large model to be selected;

[0083] Compare the failure reason analyzed by the large model to be selected with the correct failure reason corresponding to the selection use case;

[0084] In the case where the failure reason analyzed by the large model to be selected is consistent with the correct failure reason, the large model to be selected is selected for failure reason analysis of the failure use case.

[0085] In the embodiment of the present disclosure, the scheme can be regarded as a selection scheme of the large model. Since there are many types of large models at present, before the large model is used in the embodiment of the present disclosure, one large model suitable for the embodiment of the present disclosure can be selected from many large models for failure reason analysis. It can be verified by executing the large model application automatic evaluation which large model in the alternative large model has the condition of effectively analyzing the failure use case.

[0086] In the embodiment of the present disclosure, in the preset test set, the test failure use case used in the selection scheme of the large model can be recorded as a selection use case, and the selection use case can include test failure use case information and its corresponding correct failure reason.

[0087] In the embodiment of the present disclosure, the selection of the large model can be performed once before implementing the embodiment of the present disclosure, and the same large model can be used for each failure reason analysis unless it is detected that the large model is no longer suitable for failure reason analysis.

[0088] In the embodiment of the present disclosure, the selection scheme of the large model does not involve training and optimization of the large model, and the time cost and economic cost required for training and optimization are relatively large. The large model itself is a result of long-term training, so a commercially released large model or a self-trained large model will be used. The effects obtained by using different large models may not be the same, so it is necessary to evaluate which large model is suitable for the scene required by the embodiment of the present disclosure. Even if the selected large model, with the evolution of the test object, it is necessary to verify whether it meets the condition of analyzing the failure use case.

[0089] In the embodiment of the present disclosure, the method can further include:

[0090] Obtain a preset effectiveness detection use case; the effectiveness detection use case is a test failure use case for detecting the effectiveness of the large model;

[0091] Input the effectiveness detection use case into the large model, and analyze the failure reason of the effectiveness detection use case by the large model;

[0092] The failure cause analyzed by the large model is compared with the standard failure cause corresponding to the effectiveness detection case;

[0093] In a case where the failure cause analyzed by the large model is consistent with the standard failure cause, it is determined that the large model is effective; in a case where the failure cause analyzed by the large model is inconsistent with the standard failure cause, it is determined that the large model is ineffective.

[0094] In the embodiments of the present disclosure, the scheme can be regarded as an effectiveness detection scheme of the large model. Based on the effectiveness detection scheme, the selected large model can be detected for effectiveness in a timely manner, if it is detected that the large model is no longer suitable for failure cause analysis, the large model is reselected, and if it is detected that the large model is still suitable for failure cause analysis, the selected large model can be continuously used.

[0095] In the embodiments of the present disclosure, in the preset test set, the test failure cases used in the effectiveness detection scheme of the large model can be contained, which can be recorded as effectiveness detection cases. The effectiveness detection cases can contain test failure case information and the corresponding standard failure cause.

[0096] In the embodiments of the present disclosure, the implementation steps in the large model selection scheme or the effectiveness detection scheme are the same as the implementation process of the large model for analyzing the failure cause of the test failure case to be tested. The only difference is that the test cases input to the large model are selection cases or effectiveness detection cases. In the scheme, the first agent (Agent-1) also needs to send the selection cases or the effectiveness detection cases to the large model, and the large model matches at least one similar test case and its failure cause from the expert knowledge base, analyzes in combination with the prompt word, and finally the large model gives an analysis result. Another difference is that, for the analysis result, in the selection scheme, the failure cause in the analysis result is compared with the correct failure cause corresponding to the selection case, in a case where the failure cause in the analysis result is consistent with the correct failure cause, the large model to be selected is selected for failure cause analysis of the test failure case, and in a case where the failure cause in the analysis result is inconsistent with the correct failure cause, the large model to be selected can be discarded. In the effectiveness detection scheme, the failure cause in the analysis result is compared with the standard failure cause corresponding to the effectiveness detection case, in a case where the failure cause in the analysis result is consistent with the standard failure cause, it is determined that the large model is effective and can be continuously used, and in a case where the failure cause in the analysis result is inconsistent with the standard failure cause, it is determined that the large model is ineffective and can be reselected.

[0097] In the embodiments of the present disclosure, the selection use case and the effectiveness detection use case can be the same batch of test failure use cases or different test failure use cases, wherein the correct failure reason and the standard failure reason both refer to the verified accurate failure reason, and only the test failure use cases for different functions are called differently.

[0098] In the embodiments of the present disclosure, the detailed scheme for the large model to analyze the failure reason of the test failure use case to be detected is introduced below.

[0099] In the embodiments of the present disclosure, the prompt interaction with the large model is performed, and the failure analysis task is issued to the large model, so that the large model analyzes the failure reason of the test failure use case based on the test failure use case information, which can include:

[0100] calling a preset first intelligent agent;

[0101] sending a prompt word to the large model by the first intelligent agent, instructing the large model to execute the failure analysis task based on the test failure use case information, so that the large model calls a preset expert knowledge base and analyzes the failure reason of the test failure use case based on the test failure use case information and the expert knowledge base.

[0102] In the embodiments of the present disclosure, the analysis of the failure reason of the test failure use case is to use the capability of the large model combined with the historical analysis experience information of the expert knowledge base. The reason for using this way is that the large model has been trained and will always give an answer to the proposed question, but in the scene of analyzing the failure reason of the test failure use case, it is not a possible answer that is needed, but an answer with a certain accuracy is needed. In addition, the test object is constantly changing, and the expert knowledge base is only a summary of analysis experience. The content in the expert knowledge base cannot be determined to reach the accurate failure reason of all test failure use cases in real time. Since the large model has the ability of self-learning, the large model can inherit the experience of test experts and intelligently analyze the current test failure use case to be analyzed based on the experience, so as to give a more accurate failure reason.

[0103] In the embodiments of the present disclosure, the prompt word is a condition for the analysis of the large model, which can limit the analysis range and analysis result of the large model. The prompt word does not have a general format and needs to be continuously adjusted according to different test scenes. The prompt word can include but is not limited to the role played by the large model and the content of the failure analysis task.

[0104] In the embodiments of the present disclosure, the first intelligent agent can log in to the large model to be called according to a preset username and password, and send a prompt word to the large model after logging in, for example: “You are a test expert with 10 years of test experience. Please analyze the failure cause of the test failure case according to the following test failure case information”, and provide the large model with the test failure case information. The large model can analyze the test failure case information according to the prompt word and call the content in the expert knowledge base, so as to realize the failure cause analysis.

[0105] In the embodiments of the present disclosure, the expert knowledge base can be used as a database externally connected to the large model.

[0106] In the embodiments of the present disclosure, the large model calls the preset expert knowledge base, and analyzes the failure cause of the test failure case based on the test failure case information and the expert knowledge base, which can include:

[0107] Obtaining at least one historical analysis experience information with a similarity greater than or equal to a preset similarity threshold to the test failure case information from the expert knowledge base;

[0108] Determining the failure cause corresponding to the test failure case information based on the at least one historical analysis experience information.

[0109] In the embodiments of the present disclosure, since the expert knowledge base includes a large number of historical test failure cases and their corresponding failure causes that have been accurately verified, after the large model obtains any test failure case information sent by the first intelligent agent, it can find one or more similar test cases from the expert knowledge base, obtain the failure causes of these test cases, and analyze the test failure case information of the current test failure case to be analyzed, determine which test case information in the expert knowledge base is closest to the current test failure case to be analyzed, and give the possible failure information of the current test failure case to be analyzed.

[0110] In the embodiments of the present disclosure, after the large model calls the preset expert knowledge base and analyzes the failure cause of the test failure case based on the test failure case information and the expert knowledge base, the method can further include:

[0111] The large model compares the analysis result with an expected analysis result corresponding to the test failure case;

[0112] In the case where the analysis result is consistent with the expected analysis result, the large model outputs the analysis result;

[0113] In the case where the analysis result is inconsistent with the expected analysis result, the reason for the inconsistency is that the selection of the large model is unreasonable or the expert knowledge base is imperfect;

[0114] In the case that the inconsistency is caused by unreasonable selection of the large model, the large model is reselected; in the case that the inconsistency is caused by the incomplete expert knowledge base, the content of the expert knowledge base is increased.

[0115] In the embodiments of the present disclosure, in order to verify whether the analysis result of the large model is correct, the expected analysis result of the test failure case to be analyzed can be given by the test expert in advance and saved in the test set. After the large model outputs the analysis result, the analysis result can be compared with the expected analysis result. If the two are consistent, it means that the analysis result of the large model is consistent with the result analyzed by the expert according to experience, and then the analysis result of the large model can be directly output as the final failure cause. If the two are inconsistent, it can be detected whether the selection of the large model is unreasonable or the expert knowledge base is not perfect. In the detection of whether the large model is reasonable, another set or multiple sets of test failure cases can be input into the large model for analysis, and whether the large model is reasonable can be determined according to the analysis result. If the failure cause analysis of the large model for most (for example, 90%) of the test failure cases is consistent with the expected analysis result, it means that the selection of the large model is reasonable. At this time, it can be determined that there are few similar test cases in the expert knowledge base, so that the large model is difficult to analyze the accurate result according to the test failure cases of this type in the current expert knowledge base. Therefore, at this time, the test failure cases of this type in the expert knowledge base can be enriched and improved. If the failure cause analysis of the large model for most (for example, 70%) of the test failure cases is inconsistent with the expected analysis result, it means that the selection of the large model is unreasonable, and the large model can be reselected.

[0116] In the embodiments of the present disclosure, after the analysis result is obtained, the method can further include:

[0117] verifying the analysis result, and determining the failure cause of the test failure case according to the verification result.

[0118] In the embodiments of the present disclosure, verifying the analysis result, and determining the failure cause of the test failure case according to the verification result can include:

[0119] calling a preset second intelligent agent;

[0120] calling a verification strategy corresponding to the analysis result by the second intelligent agent, and verifying the analysis result based on the verification strategy;

[0121] in the case that the verification result is a verification pass, taking the analysis result of the large model as the failure cause of the test failure case;

[0122] in the case that the verification result is a verification fail, prompting the test expert to manually analyze the failure cause of the test failure case.

[0123] In the embodiments of the present disclosure, the test expert can also prepare a verification strategy corresponding to the failure reason of each test failure use case in the test set in advance, which can be included in the verification interface and verification process corresponding to the failure reason output by the large model. The second agent (Agent-2) can call the verification interface and implement the verification process to verify whether the failure reason is correct. For example, if the failure reason is service restart, the verification strategy can include the calling interface of the service, and the verification process can include a service log query process. The second agent (Agent-2) can call the service interface, query the service log, and determine whether there is a service restart operation in the service process based on the service log query. If there is a service restart operation, the verification is passed, indicating that the failure reason is correct. If there is no service restart operation, the verification is failed, indicating that the failure reason is incorrect. The verification process in the embodiments can also include a process of re-executing the test failure use case based on the restarted service. If the re-executed process shows that the use case is successful, it indicates that the original failure reason is indeed due to service restart. If the re-executed process still shows that the use case is failed, it indicates that the original failure reason is not only due to service restart.

[0124] In the embodiments of the present disclosure, after the second agent (Agent-2) verifies, the second agent (Agent-2) can return the analysis conclusion and the verification conclusion to the task storage module. In the case that the verification result is verification failed, the test expert can intervene to manually analyze the conclusion of the verification failed and give the final failure reason.

[0125] In the embodiments of the present disclosure, the analysis result of the large model, the verification result of the Agent-2, and the intervention of the test expert make the experience data and the real-time data cooperate with the analysis, which can ensure the accuracy of the analysis.

[0126] In the embodiments of the present disclosure, the method can further include:

[0127] After the second agent verifies the analysis result based on the verification strategy and obtains a verification result, the verification result is compared with an expected analysis result corresponding to the test failure use case;

[0128] In the case that the verification result is consistent with the expected analysis result, it is determined that the second agent is effective;

[0129] In the case that the verification result is inconsistent with the expected analysis result, it is determined to update the second agent.

[0130] In the embodiments of the present disclosure, based on the expected analysis result stored in the test set, the effectiveness of the second agent can be detected, so as to realize automatic evaluation of the second agent. The scheme realizes automatic evaluation of the large model, the second agent and the expert knowledge base together with the aforementioned effectiveness detection scheme of the large model and the analysis result detection scheme of the large model, thereby improving the effectiveness of the large model, the second agent and the expert knowledge base. After the end of the current round of evaluation verification, the next evaluation process can be started at a regular time.

[0131] In the embodiments of the present disclosure, two agents, Agent-1 and Agent-2, are used. Agent-1 is responsible for complex task management and assigns analysis tasks to the large model for failure cause analysis. Agent-2 is responsible for verification and verifies the analysis conclusion of the large model by scheduling the corresponding content of the test set, scheduling different verification interfaces (such as application program interface API), and combining the verification data provided by the task storage module, thereby ensuring the accuracy of the given failure cause.

[0132] In the embodiments of the present disclosure, a detailed embodiment of the scheme of the embodiments of the present disclosure is given below.

[0133] The test failure case information of a test failure case stored in the task storage module includes:

[0134] Execution task table:

[0135]

[0136]

[0137] Environment information table:

[0138] Environment IP 1.1.1.1 Deployed PAAS version x.Y.Y.X_8.2 Service 1 version Service 2 version

[0139] Failure case information:

[0140] Operation log table:

[0141] Operation object Security module account login Operation start time 20240531 16:14:45.094 Operation end 20240531 16:14:45.179 Operation person admin Information Login failure

[0142] Abnormal service table:

[0143]

[0144] Service change table:

[0145] This execution 2024-06-03 11:40:01 Last execution 2024-06-03 03:40:01 Change service Service 1, Service 2

[0146] Case change table:

[0147] Test set:

[0148]

[0149] According to the embodiment scheme of the present disclosure, the execution scheme of the large model and the agent is as follows:

[0150] The first agent (Agent-1) sends the failed task to the large model and gives the following prompt words:

[0151] "You are a test expert with 10 years of testing experience. Please analyze the reasons for the test case failure according to the following information:

[0152] Test case name: Close model analysis switch item;

[0153] Failure keyword: configcenter.Edit Item Value;

[0154] Failure information: query auth code failed. detail info is {"code":4,"message":"The user is disabled"}, url: http: / / a.a.a.a:b / b / b / b / b / access_token, body: {"username":"admin","password":"Ze_1","grant_type":"password"}

[0155] There is no exception in the service, no change in the associated service, and no exception information."

[0156] The large model analyzes the similar knowledge obtained from the expert knowledge base and returns the failure reason (the expert knowledge base is a vector library and cannot be expressed in a table):

[0157] The admin account is disabled and cannot be used.

[0158] The second agent (Agent-2) obtains the verification interface and verification parameters according to the failure reason: http: / / a.a.a.a:b / b / b / b / b / access_token, "username":"admin","password":"Ze_1" is sent to the server for verification. The server information is obtained from the task, and the service environment IP is 1.1.1.1. The verification interface and verification parameters are sent to the server, and the server will return the state of the account according to the latest information. In this example, the server returns the result that the admin account is disabled, which is consistent with the result analyzed by the large model according to the historical experience data, and the second agent (Agent-2) verification is passed

[0159] Return the result to the task storage module: the large model analysis conclusion is that the admin account is disabled and cannot be used, and the verification conclusion of the second agent (Agent-2) is that the verification is passed.

[0160] The embodiment scheme of the present disclosure at least includes the following advantages:

[0161] 1. After the automatic case execution is completed, all test case related information (such as test itself information, related test environment information, and logs) and the like can be saved to the task storage module, the test execution data is persisted, the data can be viewed, the analysis conclusion can be modified, and the data can be statistically and compared in various dimensions.

[0162] 2. The expert knowledge base is iterated, the successful analysis experience data is saved, the test case analysis capability can be flattened after the test experience is inherited, and the cause of the test case failure can be determined according to the analysis result of the large model after the test case fails.

[0163] 3. The test set is maintained by the test expert, is used to evaluate the effectiveness of the analysis result of the large model, and is updated along with the change of the test object to timely match the difference between the analysis capability of the large model and the test object.

[0164] 4. The large model is used to analyze the failed test case, and the existing algorithm is used to analyze the data, so that the algorithm does not need to be designed again.

[0165] 5. The agent is used to make secondary verification of the analysis result, avoids the scene that the analysis result completely depends on the historical data analysis, and can combine the real-time test scene and the expert experience data to jointly give the analysis conclusion.

[0166] The embodiment of the present disclosure further provides an electronic device 100, as shown in the accompanying drawings, comprising: Figure 3

[0167] one or more processors 101;

[0168] a memory 102, one or more programs are stored on the memory, when the one or more programs are executed by the one or more processors 101, the one or more processors implement the test case analysis method;

[0169] one or more input / output I / O interfaces 103 connected between the processor and the memory, configured to realize the information interaction of the processor and the memory.

[0170] The embodiment of the present disclosure further provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the test case analysis method.

[0171] ​The embodiments of the present disclosure further provide a computer program product comprising a computer program which, when executed by a processor, implements the test case analysis method.

[0172] Those skilled in the art can understand that all or some of the functional modules / units disclosed above can be implemented as software, firmware, hardware and appropriate combinations thereof.

[0173] In the hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component can have multiple functions, or one function or step can be performed by several physical components in cooperation.

[0174] Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), FLASH memory or other solid state memory; compact disc read-only memory (CD-ROM), digital versatile discs (DVDs) or other optical disk storage; magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices; any other medium that can be used to store the desired information and that can be accessed by a computer. Furthermore, it is common knowledge to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and can include any information delivery media.

[0175] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used in the broadest descriptive sense only and should not be construed to limit the disclosure. In some instances, it will be apparent to those skilled in the art that features, characteristics or / and elements described in connection with a particular embodiment can be used in conjunction with other embodiments unless otherwise explicitly stated. Accordingly, those skilled in the art will appreciate that various modifications can be made to the described embodiments without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A test case analysis method, comprising: obtaining test failure case information; sending the test failure case information to a preset large model, prompting interaction with the large model, issuing a failure analysis task to the large model, and analyzing the failure cause of the test failure case based on the test failure case information by the large model to obtain an analysis result.

2. The test case analysis method of claim 1, wherein, After obtaining the analysis result, the method further comprises: verifying the analysis result and determining the failure cause of the test failure case according to the verification result.

3. The test case analysis method of claim 1, wherein, The prompting interaction with the large model, the issuing of the failure analysis task to the large model, and the analysis of the failure cause of the test failure case based on the test failure case information by the large model comprise: calling a preset first agent; sending prompt words to the large model by the first agent to instruct the large model to execute the failure analysis task based on the test failure case information, so that the large model retrieves a preset expert knowledge base, analyzes the failure cause of the test failure case based on the test failure case information and the expert knowledge base; The expert knowledge base contains a plurality of historical analysis experience information; each historical analysis experience information contains a historical test failure case and a failure cause corresponding to the historical test failure case.

4. The test case analysis method of claim 3, wherein, The large model retrieves a preset expert knowledge base, analyzes the failure cause of the test failure case based on the test failure case information and the expert knowledge base, comprising: obtaining at least one historical analysis experience information with a similarity greater than or equal to a preset similarity threshold from the expert knowledge base; determining the failure cause corresponding to the test failure case information based on the at least one historical analysis experience information.

5. The test case analysis method according to any one of claims 1 to 4, wherein, The test failure case information includes any one or more of the following: test failure case name, operation log information, failure keyword information, failure message information, service information, version information, change information, and environment information.

6. The test case analysis method of claim 3, wherein, After the large model retrieves a preset expert knowledge base, analyzes the failure cause of the test failure case based on the test failure case information and the expert knowledge base, the method further comprises: The large model compares the analysis result with the expected analysis result corresponding to the test failure case; In the case where the analysis result is consistent with the expected analysis result, the large model outputs the analysis result; In the case where the analysis result is inconsistent with the expected analysis result, detecting the reason for inconsistency is that the selection of the large model is unreasonable or the expert knowledge base is imperfect; In the case where the reason for inconsistency is that the selection of the large model is unreasonable, reselecting the large model; in the case where the reason for inconsistency is that the expert knowledge base is imperfect, increasing the content of the expert knowledge base.

7. The test case analysis method of claim 2, wherein, The verification of the analysis result and the determination of the failure cause of the test failure case according to the verification result comprise: calling a preset second agent; invoke a verification strategy corresponding to the analysis result by the second agent, and verify the analysis result based on the verification strategy; in a case where the verification result is a verification pass, take the analysis result of the large model as a failure cause of the test failure case; in a case where the verification result is a verification fail, remind a test expert to perform manual analysis of the test failure case.

8. The test case analysis method of claim 7, wherein, The method further comprises: after the second agent verifies the analysis result based on the verification strategy, compare the verification result with an expected analysis result corresponding to the test failure case; in a case where the verification result is consistent with the expected analysis result, determine that the second agent is valid; in a case where the verification result is inconsistent with the expected analysis result, determine to update the second agent.

9. The test case analysis method of claim 1, wherein, The method further comprises: obtain a preset selection case and at least one large model to be selected; the selection case is a test failure case for large model selection; input the selection case into the large model to be selected, and analyze a failure cause of the selection case by the large model to be selected; compare the failure cause analyzed by the large model to be selected with a correct failure cause corresponding to the selection case; in a case where the failure cause analyzed by the large model to be selected is consistent with the correct failure cause, select the large model to be selected for analysis of a failure cause of a test failure case.

10. The test case analysis method of claim 1, wherein, The method further comprises: obtain a preset validity detection case; the validity detection case is a test failure case for detecting validity of a large model; input the validity detection case into the large model, and analyze a failure cause of the validity detection case by the large model; compare the failure cause analyzed by the large model with a standard failure cause corresponding to the validity detection case; in a case where the failure cause analyzed by the large model is consistent with the standard failure cause, determine that the large model is valid; in a case where the failure cause analyzed by the large model is inconsistent with the standard failure cause, determine that the large model is invalid. 11.An electronic device, comprising: one or more processors; a memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the test case analysis method of any one of claims 1-10; one or more input / output (I / O) interfaces connected between the processor and the memory, configured to implement information interaction of the processor and the memory. 12.A computer readable storage medium having a computer program stored thereon, when the computer program is executed by a processor, the computer program implements the test case analysis method of any one of claims 1-10. 13.A computer program product comprising a computer program, when the computer program is executed by a processor, the computer program implements the test case analysis method of any one of claims 1-10.

Images