Method, computer system, storage medium and program product for automated testing of software

Abstract

This disclosure relates to methods, computer systems, storage media, and program products for automating software testing. The method for automating software testing includes: receiving test requirement information and a test initiation command for the software; determining a test execution mode based on the test initiation command; determining software test instructions based on the determined test execution mode and test requirement information; executing the determined software test instructions to obtain software test results; and analyzing the software test results based on the determined test execution mode and outputting the conclusions of the analysis.

Description

Technical Field

[0001] This disclosure relates to the field of automated testing technology, and more particularly to a method, computer system, storage medium, and program product for automated testing of software based on a large model. Background Technology

[0002] As software systems become increasingly larger and more complex, software products are becoming more sophisticated in terms of the number of functions, system architecture, and operating environment, leading to ever-increasing challenges in software quality assurance. Software testing, as a crucial part of the software development process, plays a key role in ensuring system stability and reliability. Summary of the Invention

[0003] This disclosure proposes a method for automating software testing, comprising: receiving test requirement information and a test start instruction for the software; determining a test execution mode based on the test start instruction; determining software test instructions according to the determined test execution mode and test requirement information; executing the determined software test instructions to obtain software test results; and analyzing the software test results according to the determined test execution mode and outputting the conclusions of the analysis.

[0004] According to another aspect of this disclosure, a computer system is provided, comprising: a memory having instructions stored thereon; and a processor configured to execute the instructions stored in the memory to perform the method according to this disclosure.

[0005] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to perform the method according to this disclosure.

[0006] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the method according to this disclosure.

[0007] Other features and advantages of the invention will become clearer from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0008] The present disclosure will now be described in detail below with reference to the accompanying drawings, wherein the same reference numerals throughout the drawings denote the same or similar components. It should be understood that the drawings are not necessarily drawn to scale and are only used to illustrate exemplary embodiments of the present disclosure and should not be considered as limiting the scope of the disclosure. Wherein:

[0009] Figure 1A system environment 100 is shown for an apparatus for automating software testing according to some embodiments of the present disclosure;

[0010] Figure 2 It shows Figure 1 An exemplary block diagram of the automated testing apparatus 130 in the diagram;

[0011] Figure 3 An exemplary flowchart of a method 300 for automating software testing according to some embodiments of the present disclosure is shown;

[0012] Figure 4 Exemplary flowcharts of automated test operations in various test execution modes according to some embodiments of the present disclosure are shown;

[0013] Figure 5 An exemplary flowchart of a method 500 for determining software test instructions using a semantic anchoring scheme according to some embodiments of the present disclosure is shown;

[0014] Figure 6 An exemplary flowchart of a method 600 for calculating semantic similarity according to some embodiments of the present disclosure is shown;

[0015] Figure 7 An exemplary flowchart of a method 700 for semantic assertion verification of software test results according to some embodiments of the present disclosure is shown;

[0016] Figure 8 An exemplary flowchart of a method 800 for determining software test instructions using a constraint-guided random exploration scheme according to some embodiments of the present disclosure is shown;

[0017] Figure 9 An exemplary flowchart of a method 900 for general anomaly detection of software test results according to some embodiments of the present disclosure is shown; and

[0018] Figure 10 An exemplary configuration for implementing a computing device 1000 according to an embodiment of the present invention is shown. Detailed Implementation

[0019] Software testing, as a crucial part of the software development process, plays a key role in ensuring system stability and reliability. However, existing software testing methods have many shortcomings. Currently, software testing mainly employs a combination of manual testing and traditional automated testing. Manual testing relies on the experience of testers for test case design and execution, resulting in low efficiency, high subjectivity, and difficulty in covering diverse and complex scenarios, requiring frequent adjustments to test scripts. While traditional automated testing has improved testing efficiency to some extent, it also has many shortcomings that need to be addressed. For example, traditional automated testing: i) still heavily relies on manual labor, typically only covering the test execution phase, while test requirement analysis, test case design, result analysis, and defect localization still require manual completion, making it difficult to achieve full automation of the test process; ii) test case maintenance costs are high. When the requirements or interfaces of the software under test change, testing the software requires manual adjustment of test scripts, resulting in low flexibility, testing efficiency, and high maintenance costs. This is particularly evident in continuous integration and continuous delivery scenarios; iii) it lacks the ability to understand complex business logic and context. Most existing automated testing tools execute tests based on predefined rules or templates, lacking a deep understanding of requirement descriptions, business processes, and code semantics, thus making it difficult to generate test cases that cover complex business scenarios and abnormal paths; iv) the level of intelligence in test result analysis and defect feedback is low. It lacks intelligent attribution and feedback mechanisms for test failure reasons, making it difficult to guide personnel to optimize tests and fix defects in a timely manner.

[0020] Therefore, it is desirable to provide a technical solution that combines artificial intelligence technology to intelligently and automatically process the entire software testing process, thereby promoting the automation, intelligence, and flexibility of specific testing processes, as well as accurate and intelligent analysis and feedback of test results, so as to better adapt to the needs of rapid iteration and high-quality delivery of modern software systems.

[0021] This disclosure provides methods, computer systems, storage media, and program products for automating software testing based on large models.

[0022] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. However, it should be understood that the descriptions of the various exemplary embodiments are merely illustrative and are not intended to limit the technology of the present disclosure in any way. Unless otherwise specifically stated, the relative arrangement, expressions, and values ​​of components and steps in the exemplary embodiments do not limit the scope of the present disclosure.

[0023] Figure 1 A system environment 100 is shown for an apparatus for automating software testing according to some embodiments of the present disclosure.

[0024] like Figure 1As shown, the automated testing device 130 can communicatively connect to the database 110 and the user 120, for obtaining test requirement information for the software under test from the database 110 and test start instructions from the user 120, and optionally, feeding back the test conclusions of the automated test to the database 110 for storage. The automated testing device 130 can also communicatively connect to the display device 150, for visually displaying the test conclusions of the automated test on the display device 150, thereby intuitively showing the user 120 whether the automated test meets expectations, and optionally, supplementary information associated with the test conclusions (e.g., error messages in system logs or result page screenshots when the automated test does not meet expectations) can be displayed together with the test conclusions. The automated testing device 130 can also communicatively connect to the large model 140, for interacting with the large model 140 during automated testing to complete some steps in the method for automating software testing described in this disclosure that can be performed using the large model.

[0025] Figure 2 It shows Figure 1 An exemplary block diagram of the automated testing apparatus 130 is shown. Figure 2 As shown, the automated testing apparatus 130 may include a processor 131, which provides various functions of the automated testing apparatus 130. In some embodiments, the processor 131 may be configured to execute the method 300 for automating software testing disclosed herein (hereinafter referred to in detail). Figure 3 (Describe it).

[0026] Specifically, processor 131 may include a data acquisition module 131-1, a test execution mode determination module 131-2, a software test instruction determination module 131-3, a software test instruction execution module 131-4, and a test result analysis module 131-5. These modules are configured to execute the following description. Figure 3 Steps S301-S305 of the method 300 for automating software testing shown. It should be understood that... Figure 3 The modules shown are logically divided according to their specific functions, and are not intended to limit the specific implementation method. In actual implementation, the above modules can be implemented as independent physical entities, or they can be implemented by a single entity (e.g., a processor (CPU or DSP, etc.), integrated circuit, etc.). Furthermore, it should be noted that these modules can be divided or further subdivided in different ways.

[0027] The processor 131 of the automated test apparatus 130 can be an implementation of a digital circuit system, an analog circuit system, or a mixed-signal (analog and digital combination) circuit system that performs functions in a computing system. The processing circuitry implementing the processor 131 can include, for example, circuits such as integrated circuits (ICs), application-specific integrated circuits (ASICs), portions or circuits of a single processor core, the entire processor core, a single processor, programmable hardware devices such as field-programmable gate arrays (FPGAs), and / or systems including multiple processors.

[0028] In some embodiments, the automated testing apparatus 130 may further include a memory (not shown). The memory of the automated testing apparatus 130 may store information generated by the processor, as well as programs and data used for processor operations. The memory may include volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Furthermore, the memory of the automated testing apparatus 130 may be implemented at the chip level, or it may be implemented at the device level by including other external components.

[0029] Next, combine Figure 3 A detailed description of the software automation testing method according to this disclosure is provided.

[0030] Figure 3 An exemplary flowchart of a method 300 for automating software testing according to some embodiments of the present disclosure is shown.

[0031] like Figure 3 As shown, in step S301, test requirement information and a test start instruction for the software are received. Step S301 can be executed by the data acquisition module 131-1 of the automated testing device 130. The test requirement information may include at least one of a requirement description document in natural language form, a product design sketch in visual form, and a test plan document written by testers. The test plan document written by testers may include prohibition conditions for software testing (e.g., software page elements that are prohibited from access during testing) and / or termination conditions for software testing. The test start instruction can be a real-time execution instruction issued by the user. The test start instruction may include a test start instruction associated with a specific business path, such as containing a specification of a specific business scenario or use case description. The test start instruction may also include a test start instruction associated with random exploration testing, such as "Please help me perform random exploration testing on this software," without containing any information associated with a specific business path, specific business scenario, or use case description.

[0032] In step S302, the test execution mode is determined based on the test initiation command. Step S302 can be executed by the test execution mode determination module 131-2 of the automated testing device 130. The test execution mode can be selected from a specific path execution mode and a random exploration execution mode. The selection of the test execution mode can be based on the test intent parsed from the semantics of the test initiation command, which can be completed by a large model (e.g., large model 140). When the test intent indicates that the test initiation command received from the user is associated with a specific business path, the test execution mode is determined to be the specific path execution mode; when the test intent indicates that the user wants to perform random exploration testing on the software under test, the test execution mode is determined to be the random exploration execution mode.

[0033] In step S303, software test instructions are determined based on the determined test execution mode and test requirement information. Differentiated software test instruction determination operations can be performed based on the test requirement information, depending on whether the test execution mode is determined to be a specific path execution mode or a random exploration execution mode. Specific details of the differentiated software test instruction determination operations will be provided below. Figures 4-6 and Figure 8 Provide a detailed description.

[0034] In step S304, the determined software test instructions are executed to obtain software test results. The determined software test instructions can be executed by calling a browser automation interface (e.g., Playwright / Selenium) to manipulate the pages of the software under test. The software test results obtained by executing the software test instructions may include a result page and result logs. Data related to the result page may include screenshots of the result page and the document object model (DOM) structure of the result page. The result logs may include console logs and / or network logs.

[0035] In step S305, the software test results are analyzed according to the determined test execution mode, and the analysis conclusions are output. Differentiated software test result analysis operations can be performed when the test execution mode is determined to be a specific path execution mode and when the test execution mode is determined to be a random exploration execution mode, respectively, and the conclusions derived from the corresponding analysis operations are output. Specific differentiated software test result analysis operations will be discussed below. Figure 4 , Figure 7 and Figure 9 Provide a detailed description.

[0036] Next, combine Figure 4 A more detailed description of the automated test operations under each test execution mode. Figure 4Exemplary flowcharts of automated test operations under various test execution modes according to some embodiments of this disclosure are shown. For the sake of brevity, the flowcharts have already been combined with... Figures 1-3 The content described will not be elaborated or discussed in detail below.

[0037] In step S401, a large model is used to perform semantic parsing on the test initiation command to determine the test intent. In step S402, it is determined whether the test intent indicates that the test initiation command is associated with a specific business path. When the test intent indicates that the test initiation command is associated with a specific business path, the process proceeds to step S403, where the test execution mode is determined to be the specific path execution mode; when the test intent indicates that the test initiation command is not associated with a specific business path, the process proceeds to step S407, where it is determined whether the test intent indicates a random exploration test. When the test intent indicates a random exploration test, the process proceeds to step S408, where the test execution mode is determined to be the random exploration execution mode; when the test intent does not indicate a random exploration test, the process ends. Note that the order of steps S402 and S407 can be interchanged, or steps S402 and S407 can be executed in parallel.

[0038] In some embodiments, determining the test execution mode can also take into account test requirement information, such as information related to specific business scenario descriptions or use case descriptions within the test requirement information. The process of determining the test execution mode can be defined as a conditional probability classification problem as represented by equation (1):

[0039] ,

[0040] in: The current context of the test, including test requirement information; The test start command is input by the user. The elements in the candidate set of test execution modes can be either specific (specific path execution mode) or random (random exploration execution mode). This is the finalized test execution mode. This equation is intended to calculate in the current context. and test startup command The probability of belonging to a specific path execution mode or a random exploration execution mode The test execution patterns that correspond to the highest probability are extracted as the final test execution patterns. .

[0041] When the test execution mode is determined to be a specific path execution mode, in step S404, test steps and expected behaviors described in natural language are generated, and software test instructions are determined using a semantic anchoring scheme. Test steps and expected behaviors can be generated through multimodal semantic parsing of test requirement information and test initiation instructions. Test requirement information can include multimodal information such as requirement description documents in natural language, product design sketches in visual form, and test plan documents written by testers. For example, a test step described in natural language could be "Click the login button," and an expected behavior described in natural language could be "The result page should display login success." Next, the test steps and expected behaviors described in natural language are converted into software test instructions that can be executed by the processor. The operation target of each test step described in natural language can be located based on the page information of the software under test, and a software test instruction matching the step described in natural language and targeting that operation target is generated. The location of the operation target can be achieved using the semantic anchoring scheme described herein. This semantic anchoring scheme calculates the semantic similarity between the interactive page elements of the software page and the natural language description of the test steps. The generation of test steps and expected behaviors described in natural language, the location of operational targets using semantic anchoring schemes, and the generation of software test instructions can be performed using a multimodal large model (e.g., large model 140).

[0042] Subsequently, the process proceeds to step S405, where the determined software test instructions are executed to obtain the software test results. The software test results may include the software's result page and result logs after executing the software test instructions. For specific path execution modes, in some embodiments, screenshots of the result page and / or the DOM structure of the result page may be obtained.

[0043] The process then proceeds to step S406, where semantic assertion verification is performed on the software test results, and the conclusion of the semantic assertion verification is output. The expected behavior described in natural language (e.g., "The result page should display a registration success message") generated from step S404 can be received. Semantic assertion verification can be performed between the software test results and the expected behavior described in natural language, and a conclusion is output based on the result of the semantic assertion verification. Traditional testing methods may suffer from "assertion fragility" when verifying test success. For example, when the expected behavior is "The result page should display a registration success message," the test script typically monitors for the presence of an explicit `id="success_msg"` element on the page. However, if the page is redesigned, even with minor adjustments that do not affect the business logic, the original test script may still report an error because the standard for judging test success is too explicit and specific. The semantic assertion verification described in this disclosure can accurately determine whether the software test is successful even without an explicit `id="success_msg"` element, improving the test's robustness and resistance to interference.

[0044] In some embodiments, the semantic assertion verification described herein can employ multimodal semantic comparison. Multimodal semantic comparison can be performed using software page screenshots and natural language descriptions of the expected behavior. Specific operations of the multimodal semantic comparison scheme used for semantic assertion verification will be described in detail below. The process ends after step S406 is completed.

[0045] When the test execution mode is determined to be a random exploration execution mode, in step S409, software test instructions can be determined using the constraint-guided random exploration scheme described in this disclosure. Specifically, semantic parsing can be performed on the test plan file in the test requirements information to determine the constraints of the random exploration test. The constraints of the random exploration test may include prohibition conditions, which indicate prohibited actions that are not expected to be performed during software testing. For example, prohibition conditions may include prohibiting clicking delete buttons on the page to prevent accidental deletion of data. Then, interactive page elements of the software pages that can be accessed next can be determined based on the determined constraints of the random exploration test. Selecting the next interactive page element to be accessed from the accessible interactive page elements can be based on the current number of times the interactive page element has been accessed. The selection of the next interactive page element to be accessed can also utilize the common sense reasoning ability of a large language model. After selecting the next interactive page element to be accessed, software test instructions for accessing the interactive page element can be generated based on the attributes of the interactive page element. For example, when the interactive page element to be accessed is an input box element, a series of software test instructions for entering data into the input box element are generated. This constraint-oriented random exploration scheme runs before each round of random testing steps to determine the specific operation to be performed in that round and the software testing instructions for that operation. This constraint-oriented random exploration scheme is not a random, unordered click (e.g., monkey testing), but a real-time decision-making and planning method. While testing the robustness of the software, it improves the coverage and efficiency of effective paths compared to random clicks, and reduces the time and resource waste required for random testing.

[0046] Subsequently, the process proceeds to step S410, where the determined software test instructions are executed to obtain the software test results. In some embodiments, for the random exploration execution mode, screenshots of the results page and result logs may be obtained.

[0047] The process then proceeds to step S411, where general anomaly detection is performed on the software test results, and a conclusion is output. General anomaly detection can detect common failure modes in the software's web application pages, independent of specific business logic. General anomaly detection can execute visual anomaly detection and log anomaly detection in parallel or sequentially. When visual anomalies are detected in the screenshots of the results page and / or error records are detected in the results logs, a conclusion can be output that the test results do not conform to expected behavior. After completing step S411, the process ends.

[0048] Figure 5 An exemplary flowchart of a method 500 for determining software test instructions using a semantic anchoring scheme according to some embodiments of the present disclosure is shown.

[0049] like Figure 5 As shown, in step S501, a test step described in natural language is received. In step S502, the semantic similarity between the interactive page elements of the software page and the natural language description of the test step is calculated. The DOM structure of the software page and / or screenshots can be obtained to acquire the interactive page elements present in the page of the software under test, and semantic similarity can be calculated for all or a subset of the interactive page elements. In some embodiments, the calculation of semantic similarity can employ a combination of visual semantic similarity and textual semantic similarity, as described below. Figure 6 As described, as an example, data of interactive page elements and natural language descriptions of test steps can be input into a large model (e.g., large model 140) to calculate visual semantic similarity and textual semantic similarity between them using the multimodal understanding capabilities of the large model. In steps S503-S504, the operation target of each test step in the test steps is determined based on the calculated semantic similarity, and software test instructions for the operation target are generated. The interactive page element with the highest semantic similarity among all interactive page elements can be used as the operation target of the test step. In some embodiments, the coordinates or handle of the operation target can be further derived based on the DOM structure and / or screenshots of the acquired software page, thereby generating atomic and precise browser operation instructions for the operation target.

[0050] Figure 6 An exemplary flowchart of a method 600 for calculating semantic similarity according to some embodiments of the present disclosure is shown.

[0051] like Figure 6 As shown, in step S601, the DOM structure and screenshot of the software page are obtained. In step S602, the textual semantic similarity between the text information corresponding to the interactive page elements in the DOM structure and the natural language description of the test steps is calculated. In step S603, the visual semantic similarity between the visual information corresponding to the interactive page elements in the screenshot and the natural language description of the test steps is calculated. In step S604, the textual semantic similarity and visual semantic similarity are weighted and combined. The weighted combination in step S604 can be implemented by linear weighted combination. As mentioned above, the calculation and combination of these semantic similarities can be accomplished by a large model (e.g., large model 140).

[0052] The semantic similarity of the weighted combination calculated in step S604 can be used as the semantic similarity calculated in step S502 and based on step S503. The interactive page element corresponding to the maximum value of the semantic similarity of the weighted combination can be determined as the page element that best matches the natural language description of the test step, as shown in equation (2):

[0053] ,

[0054] in: and The weighting factor used in the weighted portfolio. This is a set of all candidate interactive page elements from the current page. These are the interactive page elements in the set of candidate interactive page elements. A natural language description of the test steps. and elements respectively Text information and image information, For text semantic similarity, and For visual semantic similarity, and The interactive page element that corresponds to the maximum value of the weighted combined semantic similarity.

[0055] By using a semantic anchoring scheme to locate the operational targets corresponding to the test steps described in natural language, multimodal information of the software page is effectively integrated. This method does not rely on traditional hard-coded anchors (such as XPath or CSSSelector) and remains applicable even when the software page is redesigned, demonstrating good flexibility and scalability.

[0056] Figure 7 An exemplary flowchart of a method 700 for semantic assertion verification of software test results according to some embodiments of the present disclosure is shown. As described above, the method may receive software test results generated from step S404, which are expected behaviors described in natural language, and obtained from step S405, which include the DOM structure and / or screenshots of the result page; perform semantic assertion verification between the software test results and the expected behaviors described in natural language; and output a conclusion based on the results of the semantic assertion verification.

[0057] Specifically, in step S701, the semantic matching degree between the natural language description of the expected behavior and the result page is calculated. In some embodiments, the semantic assertion verification described herein may include semantic matching between the DOM structure of the result page and the natural language description of the expected behavior. As an example, when the natural language description of the expected behavior is "a registration success message should be displayed," semantic matching can be used to detect whether there is text in the DOM structure of the result page that is the same as or similar to "registration successful." In some embodiments, the semantic assertion verification described herein may employ multimodal semantic comparison. A screenshot of the result page and the natural language description of the expected behavior can be used for multimodal semantic comparison. As an example, a pre-trained multimodal encoder can be applied to the screenshot of the result page and the natural language description of the expected behavior to extract a first feature vector for the result page, respectively. and the second feature vector of the natural language description of the expected behavior ,in This is a screenshot of the results page. This represents the pre-trained multimodal encoder. Then, the semantic matching degree can be calculated as the cosine similarity between the first feature vector and the second feature vector to achieve multimodal semantic comparison of visual and natural language descriptive text, as shown in equation (3), where... This represents the semantic matching degree determined by multimodal semantic comparison between the screenshot of the results page and the natural language description of the expected behavior:

[0058] .

[0059] In step S702, the semantic matching degree is compared with a predetermined confidence threshold. In step S703, it is determined whether the semantic matching degree is greater than the predetermined confidence threshold. In response to the semantic matching degree being greater than the predetermined confidence threshold, in step S704, a conclusion is output that the test result conforms to the expected behavior; in response to the semantic matching degree being less than the predetermined confidence threshold, in step S705, a conclusion is output that the test result does not conform to the expected behavior. As an example, the predetermined confidence threshold can be 0.8.

[0060] The following section describes in detail the methods for determining software test instructions and analyzing software test results in the randomized exploratory execution mode. Figure 8 An exemplary flowchart of a method 800 for determining software test instructions using a constraint-guided random exploration scheme according to some embodiments of the present disclosure is shown. Figure 8 The illustrated process can be run before each round of random test steps to determine the specific operation to be performed in that round of random test steps and the software test instructions for that operation.

[0061] like Figure 8 As shown, in step S801, the test plan file is semantically parsed using a large model to determine the constraints for random exploratory testing. The constraints for random exploratory testing may include prohibition conditions, which indicate actions that are not expected to be performed during software testing. For example, the test plan file may instruct that clicking delete buttons on a page be prohibited to prevent accidental deletion of data.

[0062] In step S802, interactive page elements of the software page are filtered based on the determined constraints. For example, delete buttons on interactive page elements can be filtered out if the constraints include prohibiting clicking delete buttons on the page.

[0063] In step S803, operation weights are assigned to the filtered interactive page elements. Operation weights can be based on the exploration value of each interactive page element. In some embodiments, the operation weights assigned to the filtered interactive page elements are based on the current number of times the filtered interactive page elements have been accessed. Interactive page elements that have not been accessed or have been accessed less frequently can be considered to have higher exploration value, and therefore, higher operation weights can be assigned to such interactive page elements. Alternatively, operation weights can be assigned based on the common sense reasoning ability of the large model. As an example, based on the common sense of the large model that "login usually involves first clicking the input box to enter the username and password, and then clicking the 'login' button," a higher operation weight can be assigned to the "login" button after the current input box has been accessed to fill in the username and password. In some embodiments, the operation weights of the filtered interactive page elements can be quantified as the corresponding access probability, as shown in equation (4):

[0064] ,

[0065] in, This represents the i-th access action during the random exploration test. In the constraints, as an example, if the access action... If it falls under the prohibited condition in the constraint, the probability of it being accessed is mapped to 0. Indicates that the software under test is The system state after the previous i-1 access actions. express The operation weights are calculated using Equation (4). The Softmax function is used to convert the operation weights into access probabilities.

[0066] In step S804, an operation target and a software test instruction for that operation target are determined based on the assigned operation weights. The interactive page element with the highest operation weight can be selected as the operation target, and a software test instruction for that operation target can be determined. Alternatively, the access probability of each interactive page element can be derived based on the operation weights, and the operation target can be selected based on the access probability. After selecting the operation target, a software test instruction for accessing the operation target can be generated based on its attributes. For example, when the operation target to be accessed is the "Login" button, a software test instruction for clicking that button can be generated.

[0067] Constraints for randomized exploratory testing may also include termination conditions. Termination conditions for randomized exploratory testing may include one or more of the following: the number of steps in the randomized exploratory test has reached a threshold, the software's result page crashes after executing the software test instructions, and the result page times out and fails to respond after executing the software test instructions. Checking whether the software under test currently meets the termination conditions can be performed after each round of randomized test steps.

[0068] Figure 9 An exemplary flowchart of a method for general anomaly detection of software test results according to some embodiments of this disclosure is shown. General anomaly detection can detect common failure modes in software page web applications, without relying on specific business logic.

[0069] like Figure 9 As shown, in steps S901-S902, visual anomalies are detected and determined in the screenshot of the result page. As an example, visual anomalies may include one or more of the following: blank screen, UI misalignment, overlapping or occlusion of page elements, and crash pop-ups. Visual anomalies can be detected using large models or computer vision techniques. When a visual anomaly is detected in the screenshot of the result page, the process proceeds to step S905, outputting a conclusion that the test result does not conform to expected behavior; when no visual anomaly is detected in the screenshot of the result page, the process proceeds to steps S903-S904, where it is detected and determined whether there are error records in the result log. The result log may include console logs and / or network logs after executing the test execution command. Error records in the result log may include one or more of the following: HTTP 4xx / 5xx error codes, JavaScript uncaught exceptions, or timeout errors. When an error is detected in the result log, the process proceeds to step S905, outputting a conclusion that the test result does not meet the expected behavior; when no error is detected in the result log, the process proceeds to step S906, outputting a conclusion that the test result meets the expected behavior. Note that the order of steps S901-S902 and S903-S904 can be interchanged, or steps S901-S902 and S903-S904 can be executed in parallel.

[0070] In some embodiments, detected error logs and visual anomalies can be output along with conclusions that the test results do not conform to expected behavior. For example, they can be output to a display device 150 for viewing by a user 120.

[0071] By using the technical solution described in this disclosure, the entire software testing process can be intelligently and automatically processed. Based on the multimodal semantic analysis capabilities of a large model, multimodal semantic parsing of multimodal test requirements and user instructions can be performed to identify the user's desired test execution mode and understand complex business logic and context. By developing different automated test operations under different test execution modes, the technical solution described in this disclosure can be applied to both specific business path testing and random exploration testing, flexibly adapting to different user needs. Furthermore, regarding identifying operation targets and generating test execution instructions for those targets, the logical reasoning and multimodal semantic analysis capabilities of the large model effectively improve the accuracy of identifying operation targets in specific path execution mode, and in random exploration execution mode, help the testing system obtain the operation weights of page elements based on common-sense reasoning, thereby improving the efficiency of random exploration testing and reducing ineffective resource consumption.

[0072] Figure 10 An exemplary configuration of a computing device that can implement an embodiment of the present invention is shown.

[0073] Computing device 1000 is an example of a hardware device capable of applying the above aspects of the present invention. Computing device 1000 can be any machine configured to perform processing and / or calculations. Computing device 1000 can be, but is not limited to, a workstation, server, desktop computer, laptop computer, tablet computer, personal data assistant (PDA), smartphone, in-vehicle computer, or a combination thereof.

[0074] like Figure 10 As shown, computing device 1000 may include one or more components that can be connected to or communicate with bus 1002 via one or more interfaces. Bus 1002 may include, but is not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus. Computing device 1000 may include, for example, one or more processors 1004, one or more input devices 1006, and one or more output devices 1008. The one or more processors 1004 may be any type of processor and may include, but is not limited to, one or more general-purpose processors or dedicated processors (such as dedicated processing chips). Processor 1004 may, for example, correspond to... Figure 2The processor 131 of the automated testing apparatus 130 is configured to implement the functions of the various modules of the apparatus 130 for automated testing of software disclosed herein. The input device 1006 can be any type of input device capable of inputting information to a computing device, and may include, but is not limited to, a mouse, keyboard, touchscreen, microphone, and / or remote controller. The output device 1008 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer.

[0075] The computing device 1000 may also include or be connected to a non-transitory storage device 1014, which may be any non-transitory storage device capable of storing data, and may include, but is not limited to, disk drives, optical storage devices, solid-state storage, floppy disks, flexible disks, hard disks, magnetic tapes or any other magnetic media, compressed disks or any other optical media, cache memory and / or any other storage chip or module, and / or any other medium from which a computer may read data, instructions and / or code. The computing device 1000 may also include random access memory (RAM) 1010 and read-only memory (ROM) 1012. ROM 1012 may store executable programs, utilities, or processes in a non-volatile manner. RAM 1010 provides volatile data storage and stores instructions related to the operation of the computing device 1000. The computing device 1000 may also include a network / bus interface 1016 coupled to a data link 1018. The network / bus interface 1016 can be any kind of device or system capable of enabling communication with external devices and / or networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication devices and / or chipsets (such as Bluetooth). TM Equipment, IEEE 802.11 equipment, WiFi equipment, WiMax equipment, mobile cellular communication facilities, etc.

[0076] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to limit the scope of this disclosure. Unless the context explicitly indicates otherwise, the singular forms “a” and “the” as used herein are intended to include the plural forms as well. It should also be understood that the word “comprising”, as used herein, indicates the presence of the indicated feature, integral, step, operation, unit, and / or component, but does not preclude the presence or addition of one or more other features, integrals, steps, operations, units, and / or components, and / or combinations thereof. Furthermore, in the description of this disclosure, the terms “first,” “second,” etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or order. Additionally, in the description of this disclosure, unless otherwise stated, “a plurality of” means two or more.

[0077] In this specification, references to "embodiment" or similar expressions mean that a specific feature, structure, or characteristic described in connection with that embodiment is included in at least one specific embodiment of this disclosure. Therefore, the use of phrases such as "in an embodiment of this disclosure" and similar expressions in this specification does not necessarily refer to the same embodiment.

[0078] Those skilled in the art will understand that this disclosure can be implemented in various forms, such as a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microprogram code, etc.), or a software and hardware embodiment, hereinafter referred to as a "circuit," "module," "unit," or "system." Furthermore, this disclosure can also be implemented in any tangible media as a computer program product having computer-usable program code stored thereon.

[0079] The description herein is based on flowcharts and / or block diagrams of systems, apparatuses, methods, and computer program products according to specific embodiments of this disclosure. It will be understood that each block in each flowchart and / or block diagram, and any combination of blocks in the flowcharts and / or block diagrams, can be implemented using computer program instructions. These computer program instructions are executable by a machine comprising a processor of a general-purpose computer or a special-purpose computer, or other programmable data processing means, and are processed by the computer or other programmable data processing means to perform the functions or operations described in the flowcharts and / or block diagrams.

[0080] The accompanying drawings illustrate flowcharts and block diagrams showing the architecture, functionality, and operation of systems, apparatuses, methods, and computer program products achievable according to various embodiments of the present disclosure. Thus, each block in a flowchart or block diagram may represent a module, segment, or portion of program code, including one or more executable instructions to implement a specified logical function. It should also be noted that in some other embodiments, the functions described in a block may not be performed in the order shown in the figures. For example, two blocks illustrated as connected may actually be executed simultaneously, or in some cases, depending on the functions involved, they may be executed in the reverse order shown in the figures. Furthermore, it should be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, may be implemented by a system based on dedicated hardware, or by a combination of dedicated hardware and computer instructions, to perform specific functions or operations.

[0081] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to market technology of the embodiments, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for automating software testing, comprising: Receive test requirement information and test start instructions for the software; The test execution mode is determined based on the test initiation command; Based on the determined test execution mode and test requirement information, determine the software test instructions; Execute the determined software test instructions to obtain software test results; as well as Analyze the software test results based on the determined test execution mode, and output the conclusions of the analysis.

2. The method according to claim 1, wherein the test requirement information includes at least one of a requirement description document in natural language form, a product design sketch in visual form, and a test plan document.

3. The method according to claim 1, wherein determining the test execution mode based on the test initiation command includes: Use a large model to perform semantic parsing on test initiation commands to determine test intent; In response to the test intent indicating that the test initiation command is associated with a specific business path, the test execution mode is determined to be the specific path execution mode; and In response to the test intent instruction to conduct random exploration testing, the test execution mode is determined to be random exploration execution mode.

4. The method of claim 3, wherein the test requirement information includes a requirement description document in natural language form, a product design sketch in visual form, and a test plan document, the test execution mode is determined to be a specific path execution mode, and wherein determining software test instructions based on the determined test execution mode and the test requirement information includes using the large model to: Multimodal semantic parsing is performed on requirements description documents in natural language form, product design sketches in visual form, test plan documents, and test initiation instructions to generate test steps and expected behaviors described in natural language; Calculate the semantic similarity between interactive page elements in the software's page and the natural language descriptions of the test steps; and Based on the calculated semantic similarity, the operational objective of each test step in the test steps is determined, thereby generating software test instructions for the operational objective.

5. The method according to claim 4, wherein calculating the semantic similarity comprises: Obtain the document object model (DOM) structure and screenshots of the software's pages; Calculate the text semantic similarity between the text information corresponding to interactive page elements in the DOM structure and the natural language description of the test steps; Calculate the visual semantic similarity between the visual information corresponding to the interactive page elements in the screenshot and the natural language description of the test steps; as well as The semantic similarity is obtained by weighting and combining the textual semantic similarity and the visual semantic similarity.

6. The method according to claim 4, wherein the software test results include a result page of the software after executing the software test instructions, and wherein analyzing the software test results according to the determined test execution mode and outputting the conclusions of the analysis includes: Calculate the semantic matching degree between the natural language description of the expected behavior and the result page; The semantic matching degree is compared with a predetermined confidence threshold; In response to the semantic matching degree being greater than a predetermined confidence threshold, the conclusion that the test result conforms to the expected behavior is output; as well as In response to the semantic matching degree being less than a predetermined confidence threshold, the conclusion that the test result does not conform to the expected behavior is output.

7. The method of claim 6, wherein calculating the semantic matching degree comprises: A pre-trained multimodal encoder is applied to the result page and the natural language description of the expected behavior to extract a first feature vector for the result page and a second feature vector for the natural language description of the expected behavior, respectively; and The semantic matching degree is calculated as the cosine similarity between the first feature vector and the second feature vector.

8. The method of claim 3, wherein the test requirement information includes a test plan document, the test execution mode is determined to be a random exploration execution mode, and wherein determining the software test instructions based on the determined test execution mode and the test requirement information includes: Use a large model to perform semantic parsing on the test plan file to determine the constraints of the random exploration test; Based on the determined constraints of the random exploratory test, the interactive page elements of the software's pages are filtered. Assign operation weights to the filtered interactive page elements; as well as The operational objectives and software test instructions for those operational objectives are determined based on the assigned operational weights.

9. The method of claim 8, wherein the operation weights assigned to the filtered interactive page elements are based on the current number of times the filtered interactive page elements have been accessed.

10. The method of claim 8, wherein assigning operation weights to the filtered interactive page elements includes using the large model: Perform commonsense inference on the currently accessed interactive page elements; and Based on the inferences made from common sense, operation weights are assigned to the filtered interactive page elements.

11. The method of claim 8, wherein the constraints of the random exploration test include a termination condition for the random exploration test, the termination condition including one or more of the following: the number of steps in the random exploration test has reached a threshold, the result page of the software crashes after the software test instruction is executed, and the result page times out and does not respond.

12. The method according to claim 8, wherein the software test results include the result page and result log of the software after executing the software test instructions, and wherein analyzing the software test results according to the determined test execution mode and outputting the conclusion of the analysis includes: Check if there are any visual anomalies in the screenshot of the results page; Check if there are any error records in the result log; as well as In response to the detection of visual anomalies in the screenshot of the results page and / or the presence of error records in the results log, the system outputs a conclusion that the test results do not conform to the expected behavior.

13. The method of claim 12, wherein the visual anomaly includes one or more of the following: a blank screen, UI misalignment, overlapping or occlusion of page elements, and a crash pop-up.

14. The method of claim 12, wherein the error record comprises one or more of the following: HTTP 4xx / 5xx error codes, JavaScript uncaught exceptions, or timeout errors.

15. A computer system comprising: A memory that stores instructions; as well as The processor is configured to execute instructions stored in the memory to perform the method according to any one of claims 1 to 14.

16. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to cause the processor to perform the method as described in any one of claims 1 to 14.

17. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1 to 14.

Images