Automated software testing method, system and apparatus, and storage medium

Abstract

Provided in the embodiments of the present application are an automated software testing method, system and apparatus, and a storage medium. The method comprises: in response to acquiring a function requirement of software or a target function requirement selected by a user, processing the function requirement or the target function requirement on the basis of a test case generation model, so as to generate at least one test case expressed by a natural language (S310); on the basis of a preset condition, obtaining test information of the test case from the test case (S320); on the basis of the test information, constructing a test script corresponding to the test case, and executing the test script to obtain an execution result (S330); and processing the execution result on the basis of an execution result determination model, and determining an execution state of the test case corresponding to the execution result (S340).

Description

Software automated testing method, system, device and storage medium

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202311443437.8, filed on October 31, 2023, entitled “A Software Automation Testing Method, System and Device,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of artificial intelligence, and in particular to a software automation testing method, system, device and storage medium. Background Art

[0004] Software testing can be achieved by using testing tools or scripts to perform performance testing on the software. However, the current software testing process requires manual design of test cases and detection of code changes to test the software. Furthermore, each function (such as requirements, design, coding, testing, and execution) is completed by multiple independent third-party software, which is highly dependent on the capabilities of each third-party software.

[0005] Therefore, the present application provides a software automation testing method.

[0006] Summary of the Invention

[0007] One of the embodiments of the present application provides a software automation testing method, which includes: in response to obtaining the target functional requirements of the software, processing the target functional requirements based on a test case generation model to generate at least one test case expressed in a natural language; based on preset conditions, obtaining test information of the test case from the test case; based on the test information, constructing a test script corresponding to the test case and executing it to obtain an execution result; and processing the execution result based on an execution result judgment model to judge the execution status of the test case corresponding to the execution result.

[0008] One of the embodiments of the present application provides a software automation testing system, which includes: a generation module for processing the target functional requirements based on a test case generation model in response to obtaining the target functional requirements of the software in natural language, and generating at least one test case expressed in the natural language; a conversion and identification module for obtaining the test information of the test case from the test case based on preset conditions; an execution module for constructing a test script corresponding to the test case based on the test information and executing it to obtain an execution result; and a judgment module for processing the execution result based on an execution result judgment model, and judging the execution status of the test case corresponding to the execution result.

[0009] One embodiment of the present application provides a software automation testing device, which includes: at least one storage medium for storing computer instructions; and at least one processor for executing the computer instructions to implement the software automation testing method.

[0010] One embodiment of the present application provides a storage medium, wherein the storage medium stores computer instructions. When the computer instructions are executed by at least one processor, the software automation testing method is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0012] FIG1 is a schematic diagram of an application scenario of a software automation testing system according to some embodiments of the present application.

[0013] FIG2 is an exemplary structural diagram of a software automation testing system according to some embodiments of the present application.

[0014] FIG3 is an exemplary flowchart of a software automation testing method according to some embodiments of the present application.

[0015] FIG4 is a schematic diagram of test case generation model training according to some embodiments of the present application.

[0016] FIG5 is a schematic diagram of execution result judgment model training according to some embodiments of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] It should be understood that the terms "system," "device," "unit," and / or "module" used herein are a method for distinguishing different components, elements, parts, portions, or assemblies at different levels. However, if other terms can achieve the same purpose, the terms may be replaced by other expressions.

[0019] As used in this application and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "comprises" and "include" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include other steps or elements.

[0020] Flowcharts are used in this application to illustrate the operations performed by the systems according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps may be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0021] Software testing is the process of evaluating and verifying that a software product or application operates as expected. The benefits of software testing include, but are not limited to, reducing errors, lowering development costs, and improving performance. Automated software testing is a key branch and component of software testing activities. Automated software testing can utilize testing tools or scripts to perform performance testing on software. However, the current software testing process requires manual design of test cases and detection of code changes for software testing. Furthermore, each function (e.g., requirements, design, coding, testing, and execution) is performed by multiple independent third-party software programs, which is highly dependent on the capabilities of each third-party software program.

[0022] Therefore, the present application provides a software automation testing method that can coordinate various functions in the software testing process, has low reliance on manual operations in the entire software automation testing process, and realizes a highly automated software testing process through artificial intelligence.

[0023] FIG1 is a schematic diagram of an application scenario of a software automation testing system according to some embodiments of the present application.

[0024] As shown in FIG. 1 , an application scenario 100 of a software automation testing system may include a storage device 110 , a processing device 120 , a terminal 130 , and a network 140 .

[0025] The storage device 110 can store data, instructions, and / or any other information. In some embodiments, the storage device 110 can store data obtained from the terminal 130 and / or the processing device 120. For example, the storage device 110 can store target functional requirements and test cases of the software obtained from the terminal 130 in natural language. The target functional requirements can be requirements obtained by feeding back to the user based on the functional requirements obtained in natural language. In some embodiments, the storage device 110 can store data and / or instructions executed or used by the processing device 120 to perform the exemplary methods described herein. For example, the storage device 110 can store data related to the test case generation model and the execution result judgment model. In some embodiments, the storage device 110 can include one or a combination of mass storage, removable memory, volatile read-write memory, read-only memory (ROM), etc. In some embodiments, the storage device 110 can be implemented via the cloud platform described herein. For example, the cloud platform can include one or a combination of private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, cross-cloud, multi-cloud, etc.

[0026] In some embodiments, storage device 110 may be connected to network 140 to enable communication with one or more components (e.g., processing device 120, terminal 130, etc.). One or more components may access data or instructions stored in storage device 110 via network 140. In some embodiments, storage device 110 may be part of processing device 120 or independent and directly or indirectly connected to the processing device.

[0027] The processing device 120 can process data and / or information obtained from the terminal 130 and / or the storage device 110. For example, in response to obtaining a target functional requirement of the software in natural language, the processing device 120 can, through the terminal 130, process the target functional requirement based on a test case generation model to generate at least one test case expressed in natural language; convert the test case into a structured data representation; identify test information of the test case from the structured data representation based on preset conditions; construct a test script corresponding to the test case based on the test information and execute it to obtain an execution result; and process the execution result based on an execution result judgment model to determine the execution status of the test case corresponding to the execution result. The test information is information in the test case that is critical to the test objectives and test requirements. The test information may include the positioning of page elements, operation types, and input data. In some examples, the generated test case expressed in natural language can be fed back to the user to facilitate fine-tuning of the target functional requirement. In some embodiments, the processing device 120 can be a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processing device 120 can be local or remote. For example, processing device 120 may access information and / or data from terminal 130 and / or storage device 110 via network 140. For another example, processing device 120 may directly connect to terminal 130 and / or storage device 110 to access information and / or data. In some embodiments, processing device 120 may be implemented on a cloud platform.

[0028] Terminal 130 may include a mobile device 131, a tablet computer 132, a laptop computer 133, or any combination thereof. Terminal 130 may acquire the target functional requirements of the software in natural language input by the user. In some embodiments, terminal 130 may interact with other components via network 140. For example, terminal 130 may transmit the acquired target functional requirements of the software in natural language to processing device 120 via network 140. For another example, terminal 130 may also present execution results and execution status to the user. In some embodiments, terminal 130 may be part of processing device 120. The user may be an operator of the software automated testing system, such as a tester.

[0029] The network 140 may include any suitable network capable of facilitating the exchange of information and / or data. In some embodiments, one or more components (e.g., the storage device 110, the processing device 120, the terminal 130, etc.) may exchange information and / or data with one or more components via the network 140. In some embodiments, target functional requirements of the software in a natural language may be obtained via the network 140. The network 140 may include one or a combination of a public network (e.g., the Internet), a private network (e.g., a local area network (LAN), a wide area network (WAN), etc.), a wired network (e.g., Ethernet), a wireless network, a cellular network, a frame relay network, a virtual private network, a satellite network, a telephone network, a router, a hub, a server computer, and the like.

[0030] It should be noted that the application scenario 100 of the software automated testing system is provided for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art will appreciate that various modifications or variations can be made based on the description of this application. For example, the application scenario 100 of the software automated testing system may further include a database. For another example, the application scenario 100 of the software automated testing system may implement similar or different functions on other devices. However, such variations and modifications do not deviate from the scope of this application.

[0031] FIG2 is an exemplary structural diagram of a software automation testing system according to some embodiments of the present application.

[0032] In some embodiments, as shown in FIG. 2 , the software automated testing system 200 may include a generation module 210 , a conversion identification module 220 , an execution module 230 , and a determination module 240 .

[0033] The generation module 210 is configured to, in response to obtaining the target functional requirements of the software, process the target functional requirements based on the test case generation model to generate at least one test case expressed in a natural language.

[0034] The conversion and identification module 220 is used to obtain test information for a test case based on preset conditions. Test information is information in a test case that is crucial to the test objectives and requirements. Test information may include the positioning method of page elements, operation types, and input data. The conversion and identification module 220 may include a conversion module and an identification module. The conversion module is used to convert the test case into a structured data representation. The identification module is used to identify the test information for the test case from the structured data representation based on preset conditions.

[0035] The execution module 230 is used to construct a test script corresponding to the test case based on the test information and execute the test script to obtain an execution result.

[0036] The judgment module 240 is used to process the execution result based on the execution result judgment model and judge the execution status of the test case corresponding to the execution result.

[0037] The modules involved in the above-mentioned software automated testing system can be implemented by hardware or software. In particular, each block in the block diagram showing the software automated testing system, as well as the combination of blocks in the block diagram, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions. It is understood that the name of a module does not, in some cases, constitute a limitation on the module itself. For example, a "generation module" can also be described as a "module for generating test cases."

[0038] FIG3 is an exemplary flow chart of a software automation testing method according to some embodiments of the present application. As shown in FIG3, process 300 includes the following steps. In some embodiments, process 300 can be performed by a processing device (such as processing device 120) and its components.

[0039] In step 310 , in response to obtaining the target functional requirements of the software in natural language, the target functional requirements are processed based on the test case generation model to generate at least one test case expressed in natural language. In some embodiments, step 310 may be performed by the generation module 210 .

[0040] Functional requirements are functions or features that the software under test should possess. For example, functional requirements may include, but are not limited to, functional requirements, user interface requirements, data requirements, security requirements, performance requirements, reliability requirements, maintainability requirements, compatibility requirements, usability requirements, and testability requirements. In some embodiments, functional requirements can be obtained through user input. For example, a user can input one or more functional requirements through terminal 130 based on the functional requirements required by the software under test in actual application scenarios. The user can input one or more functional requirements through voice input, text input, or other means. In some embodiments, processing device 120 can convert the functional requirements input by the user into natural language functional requirements and identify these functional requirements through natural language processing (NLP). For example, processing device 120 can use natural language processing technology to convert the user's voice input into corresponding functional requirements, which can be recognized by a computer. In some embodiments, functional requirements can be obtained via the internet (such as network 140). For example, a functional requirement may include generating a verification code based on a mobile phone number input by the user and sending it to the user. In some embodiments, the target functional requirements are obtained based on the acquired natural language functional requirements of the software. For example, the functional requirements of the software obtained in natural language can be directly used as the target functional requirements, or feedback can be provided to users based on the functional requirements of the software obtained in natural language, and the target functional requirements can be further determined based on the user feedback to achieve adjustment of the target functional requirements.

[0041] In some embodiments, the processing device 120 may also determine reference functional requirements based on the acquired software natural language functional requirements; feed back the reference functional requirements to the user for selection to obtain a user selection result; and determine the target functional requirements based on the user selection result.

[0042] Reference functional requirements are alternative functional requirements provided by processing device 120 for user confirmation. For example, a reference functional requirement could generate a verification code based on the user's input mobile phone number and send it to the user, determine whether the user input is a mobile phone number, or determine whether the user has entered a registered mobile phone number. Because the functional requirements entered by the user are unstructured, meaning that the user input is not directly recognizable by processing device 120, to reduce the variability in the processing device 120's understanding of user input, processing device 120 can generate a series of related reference functional requirements based on the acquired software's natural language functional requirements, and then submit them to the user for selection. In some embodiments, the reference functional requirements can be obtained from a preset functional requirements database or through artificial intelligence. For example, the preset functional requirements database may include multiple related functional requirements. When the user enters one of these functional requirements, processing device 120 may use the other related functional requirements as reference functional requirements. For another example, processing device 120 may input the acquired software's natural language functional requirements into a machine learning model to generate corresponding reference functional requirements. Processing device 120 can provide the reference functional requirements to the user for selection via a display screen, voice, or other means.

[0043] The target functional requirement is a functional requirement selected by the user. It is understandable that the processing device 120 can also directly use the functional requirement of the software in natural language input by the user as the target functional requirement.

[0044] A test case is a set of test specifications, including inputs, execution conditions, and expected results, used to verify the functionality, performance, and reliability of a software system. A test case describes the actions that a tester should perform and the expected outputs to ensure the correctness and quality of the software being tested. In some embodiments, a test case may include at least one of the following elements: test objective, input data, execution steps, expected results, actual results, a pass indicator, etc. In some embodiments, test cases can be obtained via the internet (e.g., network 140) based on the acquired functional requirements (or target functional requirements). For example, test cases related to the functional requirements can be obtained via network 140. In some embodiments, test cases can be obtained by analyzing the acquired functional requirements (or target functional requirements) using a test case generation model. In some embodiments, test cases can be expressed in natural language. Based on the above functional requirement: generating a verification code based on the user's input mobile phone number and sending it to the user. Test cases may include: testing the correct format of the mobile phone number, testing the verification code generation and sending function, testing the accuracy of the verification code, testing the validity period of the verification code, testing the frequency limit for sending the verification code, etc.

[0045] The test case generation model is a model used to output test cases. In some embodiments, the test case generation model may be a machine learning model, such as a neural network model or a deep neural network model. The input to the test case generation model may include target functional requirements, and the output may include test cases corresponding to the target functional requirements. The test case generation model may be obtained through model training. For a detailed description of model training for the test case generation model, see Figure 4 and its related description.

[0046] Step 320 , based on preset conditions, obtain test information of the test case from the test case; in some embodiments, step 320 may be performed by the conversion identification module 220 .

[0047] Step 320 includes converting the test case into a structured data representation. Test cases written in natural language need to be converted into a computer-readable data representation. Therefore, the processing device 120 can use natural language processing (NLP) technology to convert the test case into a structured data representation that is easy to read and write. The structured data representation can include JSON (JavaScript Object Notation) and XML (Extensible Markup Language).

[0048] Step 320 also includes: identifying test information of the test case from the structured data representation based on preset conditions.

[0049] Test information is information in a test case that is crucial to the test purpose and test requirements. Test information may include the positioning method of page elements, operation type, and input data. The positioning method of page elements may be ID positioning, XPath positioning, CSS selector positioning, Name positioning, tag name positioning, link text positioning, etc. The operation type may be click, input, selection, etc. Input data may include normal input data, erroneous input data, empty input data, boundary condition data, random input data, etc. In some embodiments, the test information of a test case may be identified by the processing device 120 based on preset conditions. The preset conditions may be preset fields, etc.

[0050] In step 330 , based on the test information, a test script corresponding to the test case is constructed and executed to obtain an execution result. In some embodiments, step 330 may be executed by the execution module 230 .

[0051] A test script is a code or instruction used to automatically verify the various operations and steps in the software execution test process. In some embodiments, the test script can simulate user behavior, interact with the tested software, and verify its functionality, performance, security, etc. In some embodiments, an automated testing application programming interface (API) can be used to construct an automated test script based on the identified test information. For example, the test information includes at least one of the positioning method of the page element, the operation type, and the input data. Based on the test information, the test script can simulate the user's operations on the interface, such as clicking a button, entering text, selecting a drop-down box, etc., thereby improving test efficiency, reducing errors that may occur during manual testing, and quickly verifying whether the software functions normally.

[0052] In some embodiments, test scripts can be constructed using various programming languages, such as Python, Java, C#, etc. Test scripts can contain information such as test cases, test data (input data), test steps, expected results, etc., and can be executed using command lines, integrated development environments (IDEs), test frameworks, or custom tools. By using test scripts, testers can perform tests more efficiently and promptly discover and fix problems in the software, thereby improving the quality and reliability of the software. In some embodiments, test scripts can be executed on a local environment or distributed using a cloud-based testing platform.

[0053] In some embodiments, the execution result includes at least one of the following: a flag indicating whether the test passed or failed, an error message, and a log record.

[0054] Execution results reflect the status of the software testing process. Test pass / fail indicators include "pass," "fail," and "error." "Pass" indicates a successful test; "fail" indicates a test failure; and "error" indicates a test error. Error information is collected when a test error occurs. Examples include error messages, stack traces, and error rates. Log records are detailed records generated during the software testing process. Log records include execution steps, debugging information, timestamps, log levels, test coverage, and execution time. Timestamps indicate the time an event occurred, while execution time includes the actual time it took for an operation, request, or event to execute.

[0055] Step 340 : Process the execution result based on the execution result judgment model to judge the execution status of the test case corresponding to the execution result. In some embodiments, step 340 may be performed by the judgment module 240 .

[0056] The execution status reflects the current execution status of the software being tested. The execution status includes "normal" and "abnormal." In some embodiments, the execution status can be determined by manual judgment of the execution results, artificial intelligence recognition, or preset rule judgment. In some embodiments, the execution status can be determined based on a preset relationship. For example, when the execution result includes preset text or the attribute value of an element meets preset conditions, the execution status is judged to be "normal." In some embodiments, the execution status can be obtained by processing the execution results based on an execution result judgment model.

[0057] The execution result judgment model is a model for generating the execution status. In some embodiments, the execution result judgment model can be a machine learning model. For example, the execution result judgment model can be a classification model, a regression model, a clustering model, a decision tree, a random forest, a support vector machine, etc. The input of the execution result judgment model may include the execution result, and the output may include the corresponding execution status. Specifically, the output can be a value between 0 and 1, where the closer to 0, the higher the degree of abnormality of the execution status, and the closer to 1, the higher the degree of normality of the execution status. The execution result judgment model can be obtained through model training. For a specific description of the model training of the execution result judgment model, see Figure 5 and its related description.

[0058] In some embodiments, the method further includes generating a test report. The contents of the test report include, but are not limited to, execution results, execution status, error messages, etc. The test report can be presented in the form of text, charts, etc. In some embodiments, the test report can be fed back to the user via a terminal (such as terminal 130). The generated test report can visualize the execution results and status of the software test, facilitating user understanding and decision-making.

[0059] Through the software automation testing method described in some embodiments of the present application, the efficiency of generating software test cases and the efficiency of software testing can be improved through machine learning models, thereby saving labor costs; in addition, by converting test cases into structured data representations, computers can accurately identify them; various functions of the software testing process can be integrated and implemented through a local automated software testing system without relying on various third-party software.

[0060] Figure 4 is a schematic diagram of test case generation model training according to some embodiments of the present application. As shown in Figure 4, in some embodiments, process 400 can be performed by a processing device (such as processing device 120) or its components.

[0061] In some embodiments, processing device 120 may train an initial test case generation model based on sample functional requirements and test case labels to obtain a test case generation model. The test case labels are test cases corresponding to the sample functional requirements. A sample function may have multiple test case labels. Accordingly, the test case generation model can generate multiple test cases based on a functional requirement expressed in natural language.

[0062] Sample functional requirements 410 are historical functional requirements. For example, the sample functional requirements are user functional requirements at a certain time in history. In some embodiments, the sample functional requirements can be obtained through a network (such as network 140), a local storage device (such as storage device 110), or manually input.

[0063] In some embodiments, the sample functional requirements may be obtained by preprocessing the obtained functional requirements, wherein the preprocessing includes at least one of the following: data cleaning, data standardization, data segmentation, and data encoding.

[0064] Data cleaning involves processing data required for functional purposes, detecting and repairing errors, inconsistencies, duplications, and missing values ​​to improve data quality and reliability. Data cleaning can include removing duplicate data, filling missing values, correcting erroneous data, and converting data formats.

[0065] Data standardization refers to converting functional requirement data into a unified format and standard to facilitate data comparison and analysis. Data standardization can include converting data to the same unit, unifying date formats, converting uppercase and lowercase letters, etc.

[0066] Data segmentation is the process of breaking down functional requirement data into words or phrases. Data segmentation can be achieved using a data segmentation algorithm. Examples include rule-based segmentation, statistics-based segmentation, and machine learning-based segmentation.

[0067] Data encoding refers to converting functional requirement data into a format that can be recognized and processed by computers. Data encoding can include converting functional requirement data into ASCII code, binary code, etc. Data encoding allows functional requirement data to be recognized by computers or other models.

[0068] In some embodiments, the preprocessing of functional requirements may also include other methods, which are not limited in this application.

[0069] Test case label 420 is a test case corresponding to a sample functional requirement. For example, a test case label may be a test case designed for a historical functional requirement. In some embodiments, the test case label may be obtained via a network (e.g., network 140), a local storage device (e.g., storage device 110), or manually input.

[0070] In some embodiments, in 430 of FIG. 4 , the test case generation model can be trained using multiple test case labels and sample functional requirements. For example, multiple sample functional requirements with test case labels can be input into the initial test case generation model, a loss function can be constructed using the test case labels and the output of the initial test case generation model, and the parameters of the initial test case generation model can be iteratively updated based on the loss function using gradient descent or other methods. When preset training conditions are met, model training is completed, resulting in a trained test case generation model. The preset training conditions may include convergence of the loss function, the number of iterations reaching a threshold, and the like.

[0071] In some embodiments, the processing device 120 may perform a performance evaluation on the test case generation model to obtain a first performance evaluation result.

[0072] The first performance evaluation result 440 reflects the performance of the test case generation model. The test case generation model needs to be evaluated, and the first performance evaluation result can well reflect the performance of the model.

[0073] In some embodiments, the first performance evaluation result includes at least one of the following indicators: model accuracy, model recall, and F1 score. Among them, model accuracy refers to the proportion of samples predicted by the model to be positive samples that are actually positive samples; model recall refers to the proportion of samples predicted by the model to be positive samples that are actually positive samples; F1 score is the harmonic mean of model accuracy and model recall. The value range of F1 score is 0 to 1, where 0 indicates the worst model performance and 1 indicates the best model performance. The first performance evaluation result may also include other parameters, which are used to determine the performance of the model. Positive samples are samples whose performance evaluation results meet the preset performance requirements, and negative samples are samples whose performance evaluation results do not meet the preset performance requirements.

[0074] In some embodiments, as shown in 450 in FIG. 4 , the processing device 120 may determine whether the first performance evaluation result meets a preset performance requirement.

[0075] The preset performance requirement is a preset performance requirement critical value. For example, the preset performance requirement may be an F1 score equal to 0.8. When the current F1 score is greater than or equal to 0.8, it is determined that the first performance evaluation result meets the preset performance requirement; when the current F1 score is less than 0.8, it is determined that the first performance evaluation result does not meet the preset performance requirement. For example, the preset performance requirement may be that the accuracy of the test case generation model reaches 95%. In some examples, the callback function mechanism of the deep learning framework can be used to check the accuracy of the test case generation model at the end of each epoch (when a complete data set undergoes one forward propagation and one backpropagation process in the neural network), and manual confirmation can also be used. When the accuracy of the test case generation model is greater than or equal to 95%, it is determined that the first performance evaluation result meets the preset performance requirement; when the accuracy of the test case generation model is less than 95%, it is determined that the first performance evaluation result does not meet the preset performance requirement.

[0076] In some embodiments, in response to the first performance evaluation result meeting the preset performance requirement, the processing device 120 can complete the update iteration of the test case generation model. Further, the processing device 120 can use the new test case generation model to generate test cases based on the new functional requirements input by the user.

[0077] In some embodiments, in response to the first performance evaluation result not meeting the preset performance requirement, processing device 120 may generate a first discrepancy reason and, based on the first discrepancy reason, adjust model parameters of the test case generation model to obtain a new test case generation model. The new test case generation model is used for the next software automated test. In this way, the test case generation model can be iteratively updated to improve its accuracy.

[0078] The first discrepancy reason 460 is the reason for the discrepancy in the test case generation model. For example, the first discrepancy reason may include data quality issues, inaccurate feature extraction, inappropriate model selection, etc. In some embodiments, the first discrepancy reason may be generated by analyzing the first discrepancy based on a predetermined relationship between the performance evaluation result and the discrepancy reason.

[0079] Specifically, the processing device 120 may optimize the test case generation model based on the first difference reason. For example, the processing device 120 may adjust parameters of the test case generation model, add features, improve data preprocessing, etc. through a preset model or program to improve the accuracy and robustness of the test case generation model.

[0080] Optimizing the test case generation model involves using the optimized data and model configuration to optimize the parameters of the test case generation model. For example, processing device 120 may use methods such as cross-validation and grid search to select the optimal model configuration. In some embodiments, processing device 120 may perform the aforementioned performance evaluation on the optimized test case generation model, which will not be further described here.

[0081] In this embodiment, the efficiency of generating software test cases is improved by generating a test case model. By performing performance evaluation on the model, a model that meets actual usage requirements can be obtained.

[0082] Figure 5 is a schematic diagram of training a model for judging execution results according to some embodiments of the present application. As shown in Figure 5, in some embodiments, process 500 may be executed by a processing device (such as processing device 120) or a component thereof.

[0083] In some embodiments, the processing device 120 may train an initial execution result judgment model based on the sample execution results and execution result labels corresponding to the sample test cases to obtain an execution result judgment model. The execution result label indicates whether the sample execution result is normal or abnormal, and the execution result label is used to characterize the execution status of the sample execution result.

[0084] Sample execution results 510 are historical execution results. For example, the sample execution results are execution results at a certain time in history. In some embodiments, the sample execution results can be obtained via a network (such as network 140), a local storage device (such as storage device 110), or manually input.

[0085] In some embodiments, the sample execution results corresponding to the sample test cases are obtained by preprocessing the obtained execution results. The preprocessing includes at least one of the following: data cleaning, data denoising, and missing value processing. Data cleaning refers to processing the obtained execution results to remove invalid, duplicate, erroneous, or inconsistent execution results to make the data set more accurate and complete. Data denoising refers to removing unnecessary noise or outliers from the obtained execution results to improve data quality and reliability. Missing value processing refers to filling or deleting missing values ​​in the obtained execution results so that they are not affected in subsequent analysis.

[0086] Execution result label 520 indicates whether the sample execution result is normal or abnormal. For example, the execution result label may indicate whether a historical execution result is normal or abnormal. In some embodiments, the execution result label may be obtained via a network (e.g., network 140), a local storage device (e.g., storage device 110), or manually input.

[0087] In some embodiments, in 530 of FIG. 5 , the execution result judgment model can be trained using multiple execution result labels and sample execution results. For example, multiple sample functional requirements with execution result labels can be input into the initial execution result judgment model, and a loss function can be constructed using the execution result labels and the judgment results of the initial execution result judgment model. Based on the loss function, the parameters of the initial execution result judgment model are iteratively updated using gradient descent or other methods. When the preset training conditions are met, the model training is completed, and a trained execution result judgment model is obtained. The preset training conditions can include convergence of the loss function, the number of iterations reaching a threshold, etc.

[0088] In some embodiments, the processing device 120 may perform a performance evaluation on the execution result judgment model to obtain a second performance evaluation result.

[0089] The second performance evaluation result 540 reflects the performance of the execution result judgment model. The execution result judgment model requires performance evaluation, and the second performance evaluation result can well reflect the performance of the model.

[0090] In some embodiments, the second performance evaluation result includes at least one of the following indicators: model accuracy, model recall, and F1 score. For detailed descriptions of the above parameters, see FIG4 and its related descriptions.

[0091] In some embodiments, as shown in 550 in FIG. 5 , the processing device 120 may determine whether the second performance evaluation result meets a preset performance requirement.

[0092] In some embodiments, in response to the second performance evaluation result meeting the preset performance requirement, the processing device 120 may complete an update iteration of the execution result judgment model. Further, the processing device 120 may use the new execution result judgment model to judge the execution status of the corresponding test case based on the newly input execution result.

[0093] In some embodiments, in response to the second performance evaluation result not meeting the preset performance requirement, processing device 120 may generate a second discrepancy reason and adjust model parameters of the execution result judgment model to obtain a new execution result judgment model. The new execution result judgment model is used in the next software automated test. In this way, the execution result judgment model can be iteratively updated to improve its accuracy.

[0094] The processing device 120 can obtain the log information of the test case execution and classify the reasons for not meeting the performance requirements, for example: the test purpose in the test case is not clearly stated and no suitable test case is generated; the format of the input data in the test case is not specified, resulting in invalid data being entered and wasting the test process; the test case does not have clear execution steps, etc.

[0095] Based on different classification reasons, different adjustments are automatically performed. For example, when the test objective is unclear in the test case, a reminder is given to re-enter the target functional requirements; when invalid data is entered, a reminder is given to restrict the input data format. Second discrepancy reason 560 is the reason for the discrepancy in the execution result judgment model. For example, the second discrepancy reason may include data quality issues, inaccurate feature extraction, inappropriate model selection, etc. In some embodiments, the second discrepancy reason can be generated by analyzing the second discrepancy based on a preset execution result-execution status relationship.

[0096] Specifically, the processing device 120 may optimize the execution result judgment model based on the second difference reason. For example, the processing device 120 may adjust parameters of the execution result judgment model, add features, improve data preprocessing, etc. through a preset model or program to improve the accuracy and robustness of the execution result judgment model.

[0097] Optimizing the execution result judgment model involves using the optimized data and model configuration to optimize the parameters of the execution result judgment model. For example, processing device 120 may use methods such as cross-validation and grid search to select the optimal model configuration. In some embodiments, processing device 120 may perform the aforementioned performance evaluation on the optimized execution result judgment model, which will not be further described here.

[0098] In this embodiment, the execution status generation efficiency is improved by judging the model through execution results, and a model that meets actual usage needs can be obtained by performing performance evaluation on the model; in addition, the model can be iteratively optimized based on the reasons for the differences, and the model parameters can be updated so that each updated model has better accuracy and robustness.

[0099] Through the software automation testing method described in some embodiments of the present application, at least the following beneficial effects can be achieved: 1) Through machine learning models, the efficiency of generating software test cases and the efficiency of software testing are improved, saving labor costs; 2) By converting test cases into structured data representations, computers can accurately identify them; 3) By performing performance evaluation on the model, a model that meets actual usage needs can be obtained; 4) The model can be iteratively optimized based on the reasons for the differences, and the model parameters can be updated so that the model has better accuracy and robustness after each update; 5) The various functions of the software testing process can be integrated and implemented through a local automated software testing system without relying on various third-party software; based on the target functional requirements expressed by it, the complete software test can be automatically completed, and the test method can be automatically updated based on the test results, reducing the workload of testers.

[0100] Some embodiments of the present application also provide a software automation testing device, which includes at least one processor and at least one memory; the at least one memory is used to store computer instructions; and the at least one processor is used to execute at least part of the computer instructions to implement the above-mentioned software automation testing method.

[0101] One embodiment of the present application provides a storage medium storing computer instructions. When the computer instructions are executed by at least one processor, the above-mentioned software automation testing method is implemented.

[0102] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0103] The basic concepts have been described above. It will be apparent to those skilled in the art that the detailed disclosure above is merely illustrative and does not limit the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and amendments to the present application. Such modifications, improvements, and amendments are suggested in the present application and remain within the spirit and scope of the exemplary embodiments of the present application.

[0104] At the same time, this application uses specific terms to describe the embodiments of this application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a certain feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "one embodiment," "an embodiment," or "an alternative embodiment" mentioned twice or multiple times in different locations in this application does not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application may be appropriately combined.

[0105] In addition, unless expressly stated in the claims, the order of the processing elements and sequences described in this application, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of this application. Although the above disclosure discusses some of the invention embodiments currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this application. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing server or mobile device.

[0106] Similarly, it should be noted that, in order to simplify the presentation of this application and thus facilitate understanding of one or more embodiments of the invention, the foregoing descriptions of the embodiments of this application sometimes combine multiple features into a single embodiment, figure, or description thereof. However, this disclosure method does not mean that the subject matter of this application requires more features than those recited in the claims. In fact, an embodiment may have fewer features than all of the features of a single embodiment disclosed above.

[0107] In some embodiments, numbers are used to describe the quantity of components and attributes. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise stated, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the description and claims are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining digits. Although the numerical domains and parameters used to confirm the breadth of their range in some embodiments of the present application are approximate values, in specific embodiments, the settings of such numerical values ​​are as accurate as possible within the feasible range.

[0108] Each patent, patent application, patent application disclosure, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this application is hereby incorporated by reference in its entirety. This includes application history documents that are inconsistent with or conflict with the content of this application, as well as documents (currently or subsequently attached to this application) that limit the broadest scope of the claims of this application. It should be noted that if the descriptions, definitions, and / or use of terms in the accompanying materials of this application are inconsistent or conflicting with the content of this application, the descriptions, definitions, and / or use of terms in this application shall prevail.

[0109] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A software automated testing method, the method comprising: In response to obtaining the target functional requirements of the software in natural language, processing the target functional requirements based on a test case generation model to generate at least one test case expressed in the natural language; Based on a preset condition, obtaining test information of the test case from the test case; Based on the test information, construct a test script corresponding to the test case and execute it to obtain an execution result; The execution result is processed based on an execution result judgment model to judge the execution status of the test case corresponding to the execution result.

2. The method of claim 1, wherein: The obtaining the test information of the test case from the test case based on the preset condition includes: Converting the test case into a structured data representation; Based on the preset condition, the test information is identified from the structured data representation.

3. The method of claim 1, wherein: The test case generation model includes a machine learning model.

4. The method of claim 1, wherein: The test information includes at least one of a positioning mode, an operation type, and input data of a page element.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: Performing a performance evaluation on the test case generation model to obtain a first performance evaluation result; In response to the first performance evaluation result not meeting a preset performance requirement, generating a first difference reason; Based on the first difference reason, the model parameters of the test case generation model are adjusted to obtain a new test case generation model, wherein the new test case generation model is used for the next software automation test.

6. The method according to any one of claims 1 to 5, wherein: The test case generation model is trained through the following process: Based on the sample functional requirements and the test case labels, the initial test case generation model is trained to obtain the test case generation model; wherein the test case labels are the test cases corresponding to the sample functional requirements.

7. The method of claim 6, wherein: The training of the initial test case generation model to obtain the test case generation model includes: Inputting a plurality of the sample functional requirements with the test case labels into the initial test case generation model to obtain an output result; Constructing a loss function through the test case label and the output result; Based on the loss function, iteratively update the parameters of the initial test case generation model; When a preset training condition is met, the test case generation model is obtained; the preset training condition is that the loss function converges or the number of iterations reaches a threshold.

8. The method of claim 5, wherein: The first performance evaluation result includes at least one of the following indicators: model accuracy, model recall, and a harmonic mean of the model accuracy and the model recall; The preset performance requirement includes a preset performance requirement critical value.

9. The method according to claim 6 or 7, wherein: The sample functional requirements are obtained by preprocessing the acquired target functional requirements, and the preprocessing includes at least one of the following: data cleaning, data standardization, data segmentation, and data encoding.

10. The method according to any one of claims 1 to 9, further comprising: Performing a performance evaluation on the execution result judgment model to obtain a second performance evaluation result; In response to the second performance evaluation result not meeting the preset performance requirement, generating a second difference reason; Based on the second difference reason, the model parameters of the execution result judgment model are adjusted to obtain a new execution result judgment model, wherein the new execution result judgment model is used for the next software automation test.

11. The method according to any one of claims 1 to 10, wherein the execution result judgment model is trained by the following process: Based on the sample execution results and execution result labels corresponding to the sample test cases, the initial execution result judgment model is trained to obtain the execution result judgment model; wherein, The execution result label indicates whether the sample execution result is normal or abnormal.

12. The method of claim 11, wherein: The initial execution result judgment model is trained based on the sample execution results and execution result labels corresponding to the sample test cases to obtain the execution result judgment model, including: Inputting a plurality of the sample functional requirements with the execution result labels into the initial execution result judgment model to obtain a judgment result; Constructing a loss function through the execution result label and the judgment result; Iteratively updating the parameters of the initial execution result judgment model based on the loss function; When a preset training condition is met, the execution result judgment model is obtained; the preset training condition is that the loss function converges or the number of iterations reaches a threshold.

13. The method of claim 10, wherein: The second performance evaluation result includes at least one of the following indicators: model accuracy, model recall, and a harmonic mean of the model accuracy and the model recall; The preset performance requirement includes a preset performance requirement critical value.

14. The method according to claim 11 or 12, wherein: The sample execution result corresponding to the sample test case is obtained by preprocessing the acquired execution result, and the preprocessing includes at least one of the following: data cleaning, data denoising, and processing missing values.

15. The method according to any one of claims 1 to 14, wherein: The method further comprises: Determining reference functional requirements based on the obtained functional requirements of the software in natural language; Feeding back the reference function requirement to the user for selection, and obtaining the user selection result; Based on the user selection result, the target function requirement is determined.

16. The method according to any one of claims 1 to 15, wherein: The execution result includes at least one of the following: a flag indicating whether the test passed or failed, an error message, and a log record.

17. The method according to any one of claims 1 to 16, wherein: The preset condition includes a preset field.

18. A software automated testing system, the system comprising: A generation module, configured to, in response to obtaining a target functional requirement of the software in natural language, process the target functional requirement based on a test case generation model to generate at least one test case expressed in the natural language; A conversion identification module, used for obtaining the test information of the test case from the test case based on a preset condition; An execution module, used to construct a test script corresponding to the test case based on the test information and execute the test script to obtain an execution result; The judgment module is used to process the execution result based on the execution result judgment model to judge the execution status of the test case corresponding to the execution result.

19. A software automated testing device, comprising: at least one storage medium storing computer instructions; At least one processor executes the computer instructions to implement the method according to any one of claims 1 to 17.

20. A storage medium storing computer instructions, wherein when the computer instructions are executed by at least one processor, the method according to any one of claims 1 to 17 is implemented.

Images