A Software Testing Method Combining LSTM Ensemble Learning

Abstract

The present invention discloses a software testing method combining LSTM ensemble learning. First, screenshots of different user interfaces are collected and input into an object detection model for recognition and positioning to obtain the classification and positioning results of UI elements. Then, text extraction is performed on the UI elements with text descriptions to obtain the text content associated with the UI elements. Next, test cases are generated based on the classification and positioning results of the UI elements and the associated text content. Then, the test cases are optimized and screened based on the LSTM model to obtain optimized test cases. Finally, the priorities of the optimized test cases are determined based on the DeepOrder model, and software testing is performed according to the priorities. The present invention solves problems such as testers needing to perform manual testing at the UI level, the poor effectiveness of existing regression testing, and the prioritization of test cases, improves the fault detection rate in the CI environment, and shortens the test execution time.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and particularly relates to the design of a software testing method combining LSTM ensemble learning. Background Art

[0002] Software testing is a crucial process to ensure the quality and reliability of software products. It involves evaluating a software system or component to verify whether it meets the specified requirements and ensuring its proper operation under different conditions. The main purpose of software testing is to identify errors, defects, or vulnerabilities in the software, which may affect the performance, functionality, or security of the software. Moreover, by continuously testing early in the development cycle, the overall quality of the software can be improved, and errors and problems in the final product can be reduced. For software that needs to comply with industry standards or regulatory requirements, testing ensures that the software meets all relevant laws and standards. Good testing practices provide a basis for software maintenance and future upgrades, making it easier to make changes and add new features throughout the software life cycle.

[0003] Traditional software testing involves manually writing software test cases and cannot automatically generate test cases. The analysis of test results and the determination of defect priorities largely rely on the experience and judgment of testers. Especially in UI (User Interface) software testing, most UIs are composed of complex CSS (Cascading Style Sheets) scripts, so it is difficult for human test designers to identify and classify such UI elements.

[0004] Regarding CI (Continuous Integration) testing, one of the important tasks performed in CI testing is regression testing, which tests new code changes in each CI cycle. Relevant test cases must be selected from the entire test suite to detect potential errors introduced by the new changes. How to shorten the regression test execution time and improve the fault detection rate is very important. Currently, test case selection (TCS), test case prioritization (TCP), and test case reduction (TCR) are effective techniques. However, traditional software testing methods may have a high cost due to the long test cycle and the involvement of a large amount of manpower and resources. And with the increase in requirements, the software delivery time will also be shortened. Frequent software updates and releases will result in multiple integrations per day, and traditional testing methods may have difficulty quickly adapting to such changes. Summary of the Invention

[0005] The object of the present invention is to propose a software testing method combining LSTM ensemble learning, which solves problems such as testers needing to perform manual testing at the UI level, the poor effectiveness of existing regression tests, and the prioritization of test cases, improves the fault detection rate in the CI environment, and shortens the test execution time by automatically reducing obsolete test cases.

[0006] The technical solution of the present invention is: a software testing method combining LSTM ensemble learning, including the following steps:

[0007] S1. Collect screenshots of different user interfaces, classify and label the screenshots, and obtain an image data set with accurate labels.

[0008] S2. Input the screenshots in the image data set into an object detection model based on the Transformer architecture for recognition and positioning, and obtain the classification and positioning results of UI elements.

[0009] S3. Use an optical character recognition engine to extract text from UI elements containing text descriptions, and obtain the text content associated with the UI elements.

[0010] S4. Based on the classification and positioning results of UI elements and the associated text content, parse the test flow through the Transformer model to generate test cases.

[0011] S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases.

[0012] S6. Determine the priorities of the optimized test cases based on the DeepOrder model, and perform software testing according to the priorities.

[0013] Furthermore, step S2 includes the following sub-steps:

[0014] S21. Input the screenshots in the image data set into an object detection model based on the Transformer architecture, and convert the screenshots into feature maps containing hierarchical information through the convolutional neural network in the object detection model.

[0015] S22. Deep-process the feature maps through the encoder in the object detection model to generate feature representations.

[0016] S23. Precisely classify and position the feature representations through the decoder in the object detection model, and assign a unique class label and precise position coordinates to each detected UI element to obtain the classification and positioning results of the UI elements.

[0017] Furthermore, step S5 includes the following sub-steps:

[0018] S51. Optimize the test cases according to the optimization equation, and screen out the exceptional test cases with the same judgment.

[0019] S52. Iteratively optimize the hidden layer of the LSTM model based on the exceptional test cases, and output the optimized test cases.

[0020] Further, the optimization equation in step S51 is specifically:

[0021]

[0022] Where t represents the test case, F(t) represents the ratio of the test cases with the execution result of 1 to the total test cases, F ′ (t) represents the ratio of the test cases with the execution result of 0 to the total test cases, num represents the total execution times of the test cases in the previous n loops, s represents the number of test cases with the execution result of 1, m represents the number of test cases with the execution result of 0, i represents the test case with the execution result of 1, and j represents the test case with the execution result of 0.

[0023] Further, the double-objective function of the DeepOrder model in step S6 is:

[0024] f(maximize(p), maximize(|q|))

[0025] Where f(·) represents the double-objective function, maximize(p) represents maximizing the fault detection ability of the test suite under the guidance of the historical fault detection ability of the test cases, p represents the priority of the test cases, maximize(|q|) represents maximizing the number of executed test cases within the given time budget, and q represents the subset of the test cases.

[0026] Further, the calculation formula for the priority of the optimized test cases in step S6 is:

[0027]

[0028] Where p(t a ) represents the priority of the a-th optimized test case t a , ω b is the weight assigned in the interval (0, 1) in the CI cycle b, such that ∑ b∈1,2,…,z ω b = 1, z represents the number of historical CI cycles, and ES (a,b) represents the fault detection ability of the a-th optimized test case in the CI cycle b.

[0029] Further, the activation function of the DeepOrder model in step S6 is:

[0030] h(x) = x × tanh(softplus(x)) = x × tanh(ln(1 + e x ))

[0031] where h(·) represents the activation function, x represents the input activation value, softplus(·) represents the smoothed ReLU function, and e represents the natural constant.

[0032] The beneficial effects of the present invention are as follows:

[0033] (1) The present invention uses the LSTM deep learning model to optimize and screen test cases. By analyzing information such as historical test data, application change records, and user feedback, it can automatically identify test cases that have become irrelevant or invalid due to product updates, reducing the time for manual inspection and cleaning of outdated test cases, enabling the test team to focus more on effective test scenarios.

[0034] (2) Based on the prediction of the DeepOrder model, the present invention can automatically sort the priorities of test cases to ensure that the test cases most likely to discover problems are executed first, which improves test efficiency and software quality because critical or high-risk functions will be verified first.

[0035] (3) During the test execution process, if the model discovers that there are high risks or defect densities in certain areas, the present invention can dynamically adjust the test plan to increase the test coverage of such areas, thereby further improving the effectiveness of the test.

[0036] (4) When performing UI testing, the present invention uses the Transformer model to effectively identify the complex relationships between different UI elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure shows a flowchart of a software testing method combining LSTM ensemble learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Now, exemplary embodiments of the present invention will be described in detail with reference to the drawings. It should be understood that the embodiments shown and described in the drawings are merely exemplary, intended to illustrate the principles and spirit of the present invention, and not to limit the scope of the present invention.

[0039] An embodiment of the present invention provides a software testing method combining LSTM ensemble learning, as Figure 1 shown, including the following steps S1 to S6:

[0040] S1. Collect screenshots of different user interfaces and classify and label the screenshots to obtain an image data set with accurate labels.

[0041] In the embodiments of the present invention, screenshots involving 10,000 different user interfaces were widely collected in the initial research preparation stage. Such screenshots cover diverse test scenarios to ensure the comprehensiveness and generalization ability of the test results. Subsequently, the embodiments of the present invention used the tzutalin image tagging tool to conduct meticulous marking work on the collected image dataset.

[0042] During the marking process, the embodiments of the present invention follow strict classification criteria and subdivide the UI elements in the images into 25 categories, including but not limited to key interface elements such as buttons, text boxes, drop-down menus, icons, sliders, tab pages, and dialog boxes. This classification system not only covers common UI components but also incorporates elements with specific functions or appearance characteristics to comprehensively reflect the complexity of modern software interfaces.

[0043] To enhance the richness and representativeness of the dataset, the embodiments of the present invention paid special attention to diversity when selecting screenshots, ensuring that they cover UI interfaces of different operating systems, browsers, device types (such as mobile phones, tablets, and computers), and different design styles (such as flat and skeuomorphic). At the same time, the embodiments of the present invention also considered various factors such as the layout, size, color, and state (such as enabled, disabled, and hover) of interface elements to ensure that the dataset can comprehensively reflect the challenges in actual tests.

[0044] After the classification and marking of the screenshots are completed, an image dataset with a clear structure and accurate labels is obtained. This dataset not only contains a large number of positive samples (i.e., UI elements that are correctly displayed and easy to identify) but also covers a small number of negative samples (such as blurred, occluded, incorrectly displayed, or difficult-to-recognize UI elements) to simulate various complex situations that may be encountered in the actual test process. Through such a carefully designed dataset, the embodiments of the present invention will lay a solid foundation for subsequent UI element detection and recognition research.

[0045] S2. Input the screenshots in the image dataset into an object detection model based on the Transformer architecture for recognition and localization to obtain the classification and localization results of UI elements.

[0046] In the embodiments of the present invention, the object detection model based on the Transformer architecture has been deeply customized and optimized, designed specifically for the UI element detection task, and can efficiently process complex image data and extract key information.

[0047] Step S2 includes the following sub-steps S21 to S23:

[0048] S21. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture, and convert the screenshots into feature maps containing hierarchical information through the convolutional neural network (CNN) in the object detection model.

[0049] S22. Deep-process the feature maps through the encoder in the object detection model to generate feature representations.

[0050] In the embodiment of the present invention, the encoder captures the complex relationships between UI elements through the multi-head self-attention mechanism to generate more abstract and rich feature representations.

[0051] S23. Precisely classify and locate the feature representations through the decoder in the object detection model, assign a unique class label and precise position coordinates to each detected UI element, and obtain the classification and localization results of the UI elements.

[0052] In the embodiment of the present invention, the decoder can not only identify basic UI elements such as buttons, text boxes, drop-down boxes, check boxes, etc. in the screen screenshot, but also accurately detect interface components with specific functions and complex structures such as shopping cart widgets, profile widgets, etc. By assigning a unique class label and precise position coordinates (such as bounding boxes) to each detected UI element, it provides valuable data support for subsequent test script writing, user behavior simulation, and interface interaction verification.

[0053] S3. Use an optical character recognition engine to extract text from the UI elements containing text descriptions to obtain the text content associated with the UI elements.

[0054] In the embodiment of the present invention, text extraction aims to capture and parse specific text content from UI elements containing text descriptions for subsequent operations such as text verification, search, or data analysis.

[0055] Since the precise boundary coordinates of each UI element have been determined in step S2, such information can be used to precisely crop the images of each UI component from the original UI screenshot. This step ensures the accuracy and efficiency of text extraction and avoids interference from irrelevant information. The cropped UI element screenshots are then saved to the temporary file system for use in this step and subsequent steps. To perform the text extraction task, the embodiment of the present invention uses the Google Tesseract open-source OCR (optical character recognition) engine, and can easily meet the text extraction requirements in different language environments by integrating the Tesseract framework.

[0056] S4. Based on the classification and localization results of the UI elements and the associated text content, parse the test flow through the Transformer model to generate test cases.

[0057] In the embodiments of the present invention, the parser is based on the Transformer model and can understand and parse the test flow descriptions written by testers. Such test flows adopt an easy-to-understand text format and clearly define test scenarios, steps, and expected results. The core logic of the parser closely depends on the structure of the test flow generated in the previous steps. By deeply analyzing such a structure, the parser can accurately identify each test step in the test flow and generate corresponding test scripts accordingly.

[0058] The embodiments of the present invention fully consider the diversity of mobile application UIs and also notice the relatively limited nature of UI elements. Taking the Android platform as an example, although there are a wide variety of UI components, the elements frequently used by developers in actual applications are relatively limited, such as TextView, EditText, Button, ImageButton, ToggleButton, etc. Based on this, the parser is built with the ability to recognize such common elements, and a standard operation set that can be performed on each element is defined. For example, for the Button element, the "click" operation is supported; for the EditText element, operations such as "SendKeys" are supported, thereby simplifying the implementation complexity of the parser and improving the generality and maintainability of test cases.

[0059] S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases.

[0060] Step S5 includes the following sub-steps S51 to S52:

[0061] S51. Optimize the test cases according to the optimization equation and screen out the exceptional test cases with the same judgment.

[0062] In the embodiments of the present invention, each test case has a unique execution result (pass: 0, fail: 1, not executed: -1) in each cycle. If a test case is not executed, -1 is assigned as the test history information.

[0063] If a test case has detected an error in the past, then it is more likely to fail in subsequent tests. However, the execution history of the test case is always the same value in all execution cycles, and it is difficult to obtain ideal prediction results using the LSTM network at this time. Therefore, before network training, it is first necessary to screen out the exceptional test cases with the same judgment, and the optimization equation is as follows:

[0064]

[0065] Where t represents the test case, F(t) represents the ratio of the test cases with the execution result of 1 to the total number of test cases, and F ′ (t) represents the ratio of the test cases with the execution result of 0 to the total number of test cases, num represents the total number of executions of the test cases in the first n loops, s represents the number of test cases with the execution result of 1, m represents the number of test cases with the execution result of 0, i represents the test case with the execution result of 1, and j represents the test case with the execution result of 0.

[0066] S52. Iteratively optimize the hidden layer of the LSTM model based on the exception test cases, and output the optimized test cases.

[0067] S6. Determine the priority of the optimized test cases based on the DeepOrder model, and perform software testing according to the priority. In the embodiments of the present invention, the dual-objective function of the DeepOrder model is:

[0068] f(maximize(p), maximize(|q|))

[0069] Where f(·) represents the dual-objective function, maximize(p) represents maximizing the fault detection ability of the test suite under the guidance of the historical fault detection ability of the test cases, p represents the priority of the test cases, maximize(|q|) represents maximizing the number of test cases executed within the given time budget, and q represents a subset of the test cases. The optimization of the dual-objective function can ensure that the test cases that are more likely to discover faults are executed first.

[0070] In the embodiments of the present invention, the priority calculation formula of the optimized test cases is:

[0071]

[0072] Where p(t a ) represents the priority of the a-th optimized test case t a , ω b is the weight assigned in the interval (0, 1) in the CI cycle b, such that ∑ b∈1,2,…,z ω b = 1, z represents the number of historical CI cycles, and ES (a,b) represents the fault detection ability of the a-th optimized test case in the CI cycle b.

[0073] In the embodiments of the present invention, the activation function of the DeepOrder model is:

[0074] h(x) = x × tanh(softplus(x)) = x × tanh(ln(1 + e x ))

[0075] Among them, h(·) represents the activation function, x represents the input activation value, softplus(·) represents the smoothed ReLU function, and e represents the natural constant. The activation function allows the neural network of the DeepOrder model to learn complex non-linear relationships, enabling the model to better fit the priority distribution of test cases, thereby improving the accuracy of sorting. This activation function can provide a more abundant non-linear transformation than ReLU for positive inputs, which helps the network capture more complex features. Among them, the transformation of x multiplied by x is adopted, thus retaining a small amount of negative information, which helps the smooth flow of information.

[0076] The software testing method provided by the embodiments of the present invention aims to enhance the automated detection and verification capabilities for specific UI elements (such as pillar numbers, key buttons, or information fields in software) in software interface testing, especially showing excellent performance when dealing with complex test scenarios containing unbalanced data sets. Its core role is to automatically and accurately identify and verify the status and attributes of target UI elements from software interface screenshots through intelligent image recognition and deep learning technologies, thereby accelerating the test process and improving test coverage and accuracy.

[0077] Those of ordinary skill in the art will realize that the embodiments described herein are for helping readers understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on these technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A software testing method combining LSTM ensemble learning, characterized in that It includes the following steps: S1. Collect screenshots of different user interfaces, and classify and label the screenshots to obtain an image dataset with accurate labels; S2. Input the screenshots in the image dataset into an object detection model based on the Transformer architecture for recognition and localization to obtain the classification and localization results of UI elements; S3. Use an optical character recognition engine to extract text from the UI elements containing text descriptions to obtain the text content associated with the UI elements; S4. Based on the classification and localization results of UI elements and the associated text content, parse the test flow through the Transformer model to generate test cases; S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases; S6. Determine the priorities of the optimized test cases based on the DeepOrder model and perform software testing according to the priorities; The dual-objective function of the DeepOrder model in step S6 is: wherein represents a bi-objective function represents maximizing the fault detection ability of the test suite under the guidance of the historical fault detection ability of the test cases represents the priority of the test cases represents maximizing the number of test cases executed within a given time budget represents a subset of the test cases The priority calculation formula of the optimized test cases in step S6 is: Among them represents the a priority of the nth optimization test case, is the weight assigned in the CI cycle b in the interval (0, 1), such that , represents the number of historical CI cycles, represents the a nth optimization test case's fault detection ability in the CI cycle b ; The activation function of the DeepOrder model in step S6 is: wherein represents an activation function, represents the activation value of the input, represents the smoothed ReLU function, represents the natural constant.

2. The software testing method combining LSTM ensemble learning according to claim 1, wherein Step S2 includes the following sub-steps: S21. Input the screenshots in the image dataset into an object detection model based on the Transformer architecture, and convert the screenshots into feature maps containing hierarchical information through the convolutional neural network in the object detection model; S22. Deep-process the feature maps through the encoder in the object detection model to generate feature representations; S23. Accurately classify and localize the feature representations through the decoder in the object detection model, and assign a unique class label and accurate position coordinates to each detected UI element to obtain the classification and localization results of the UI elements.

3. The software testing method combining LSTM ensemble learning according to claim 1, characterized in that, Step S5 includes the following sub-steps: S51. Optimize the test cases according to the optimization equation and screen out the exceptional test cases with the same judgment; S52. Iteratively optimize the hidden layer of the LSTM model based on the exceptional test cases and output the optimized test cases.

Images