Software testing method combined with LSTM (Long Short Term Memory) ensemble learning

Abstract

The invention discloses a software testing method combined with LSTM (Long Short Term Memory) ensemble learning, which comprises the following steps of: firstly collecting screenshots of different user interfaces, inputting the screenshots into an object detection model for identifying and positioning to obtain a classification and positioning result of UI (User Interface) elements, then carrying out text extraction on the UI elements containing text description to obtain text contents associated with the UI elements, and finally carrying out LSTM ensemble learning. The method comprises the following steps: classifying and positioning UI elements, generating test cases based on classification and positioning results of the UI elements and associated text contents, optimizing and screening the test cases based on an LSTM model to obtain optimized test cases, determining the priority of the optimized test cases based on a DeepOrder model, and executing software testing according to the priority. The problems that a tester needs to execute a manual test at a UI level, the effectiveness of an existing regression test is poor, and test case priority ranking is carried out are solved, the fault detection rate in the CI environment is increased, and the test execution time is shortened.

Description

Technical Field

[0001] The present invention belongs to the technical field of software testing, and in particular relates to the design of a software testing method combined with LSTM ensemble learning. Background Art

[0002] Software testing is a critical process to ensure the quality and reliability of software products. It involves evaluating a software system or component to verify that it meets the specified requirements and ensures that it functions properly under different conditions. The main purpose of software testing is to identify errors, defects or vulnerabilities in the software, which may affect the performance, functionality or security of the software. Not only that, by continuously testing early in the development cycle, the overall quality of the software can be improved and errors and problems in the final product can be reduced. For software that needs to comply with industry standards or regulatory requirements, testing ensures that the software meets all relevant laws and standards. Good testing practices provide a basis for the maintenance and future upgrades of the software, making it easier to make changes and add new features during the software life cycle.

[0003] Traditional software testing is done by manually writing software test cases, which cannot be automatically generated. The analysis of test results and the determination of defect priorities largely rely on the experience and judgment of testers. Especially in UI (User Interface) software testing, most UIs are composed of complex CSS (Cascading Style Sheets) scripts, so it is difficult for human test designers to identify and classify such UI elements.

[0004] As for CI (Continuous Integration) testing, one of the important tasks performed in CI testing is regression testing, which tests new code changes in each CI cycle. Relevant test cases must be selected from the entire test suite in order to detect potential errors introduced by new changes. How to shorten the regression test execution time and improve the fault detection rate is very important. At present, test case selection (TCS), test case prioritization (TCP) and test case reduction (TCR) are effective technologies. However, traditional software testing methods may be costly due to the long testing cycle, more manpower and resources involved. And as demand increases, the delivery time of software will also be shortened. Frequent software updates and releases will result in multiple integrations every day, and traditional testing methods may find it difficult to quickly adapt to such changes. Summary of the invention

[0005] The purpose of the present invention is to propose a software testing method combined with LSTM ensemble learning to solve the problems of testers needing to perform manual testing at the UI level, the poor effectiveness of existing regression testing, and test case prioritization, improve the fault detection rate in the CI environment, and shorten the test execution time by automatically reducing obsolete test cases.

[0006] The technical solution of the present invention is: a software testing method combined with LSTM ensemble learning, comprising the following steps:

[0007] S1. Collect screenshots of different user interfaces and classify and label them to obtain an image dataset with accurate labels.

[0008] S2. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture for identification and positioning, and obtain the classification and positioning results of UI elements.

[0009] S3. Use an optical character recognition engine to extract text from UI elements containing text descriptions to obtain text content associated with the UI elements.

[0010] S4. Based on the classification and positioning results of UI elements and the associated text content, the test flow is parsed through the Transformer model to generate test cases.

[0011] S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases.

[0012] S6. Determine the priority of optimized test cases based on the DeepOrder model and perform software testing according to the priority.

[0013] Further, step S2 includes the following sub-steps:

[0014] S21. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture, and convert the screenshots into feature maps containing hierarchical information through the convolutional neural network in the object detection model.

[0015] S22. Deeply process the feature map through the encoder in the object detection model to generate feature representation.

[0016] S23. Accurately classify and locate the feature representation through the decoder in the object detection model, assign a unique category label and precise location coordinates to each detected UI element, and obtain the classification and positioning results of the UI element.

[0017] Further, step S5 includes the following sub-steps:

[0018] S51. Optimize the test cases according to the optimization equation, and screen out the exception test cases with the same judgment.

[0019] S52. Iteratively optimize the hidden layer of the LSTM model based on the exception test case, and output the optimized test case.

[0020] Furthermore, the optimization equation in step S51 is specifically:

[0021]

[0022] Where t represents the test case, F(t) represents the ratio of the test cases with execution result 1 to the total test cases, and F ′ (t) represents the ratio of test cases with an execution result of 0 to the total test cases, num represents the total number of executions of test cases in the first n cycles, s represents the number of test cases with an execution result of 1, m represents the number of test cases with an execution result of 0, i represents the test cases with an execution result of 1, and j represents the test cases with an execution result of 0.

[0023] Furthermore, the dual objective function of the DeepOrder model in step S6 is:

[0024] f(maximize(p),maximize(|q|))

[0025] where f(·) represents a dual objective function, maximize(p) means maximizing the fault detection capability of the test suite under the guidance of the historical fault detection capability of the test cases, p represents the priority of the test cases, maximize(|q|) means maximizing the number of test cases executed within a given time budget, and q represents a subset of the test cases.

[0026] Furthermore, the priority calculation formula for optimizing the test case in step S6 is:

[0027]

[0028] Where p(t a ) represents the ath optimized test case t a The priority of b Assign a weight in the interval (0,1) in CI period b, so that ∑ b∈1,2,…,z ω b =1, z represents the number of historical CI cycles, ES (a,b) It represents the fault detection capability of the ath optimized test case in CI cycle b.

[0029] Furthermore, the activation function of the DeepOrder model in step S6 is:

[0030] h(x)=x×tanh(softplus(x))=x×tanh(ln(1+e x ))

[0031] Where h(·) represents the activation function, x represents the input activation value, softplus(·) represents the smoothed ReLU function, and e represents a natural constant.

[0032] The beneficial effects of the present invention are:

[0033] (1) The present invention uses the LSTM deep learning model to optimize and screen test cases. By analyzing historical test data, application change records, user feedback and other information, it can automatically identify test cases that are no longer relevant or invalid due to product updates, thereby reducing the time for manual inspection and cleaning of obsolete test cases, allowing the test team to focus more on effective test scenarios.

[0034] (2) Based on the DeepOrder model prediction, the present invention can automatically sort the priorities of test cases to ensure that the test cases that are most likely to find problems are executed first, which improves testing efficiency and software quality because critical or high-risk functions will be verified first.

[0035] (3) During the test execution process of the present invention, if the model finds that certain areas have higher risks or defect densities, the test plan can be dynamically adjusted to increase the test coverage of such areas, thereby further improving the effectiveness of the test.

[0036] (4) The present invention uses the Transformer model when performing UI testing to effectively identify the complex connections between different UI elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Shown is a flow chart of a software testing method combined with LSTM ensemble learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0039] The embodiment of the present invention provides a software testing method combined with LSTM ensemble learning, such as Figure 1 As shown, the following steps S1 to S6 are included:

[0040] S1. Collect screenshots of different user interfaces and classify and label them to obtain an image dataset with accurate labels.

[0041] In the embodiment of the present invention, screenshots involving 10,000 different user interfaces were extensively collected in the initial research preparation stage, and such screenshots covered a variety of test scenarios to ensure the comprehensiveness and generalization of the test results. Subsequently, the embodiment of the present invention used the tzutalin image labeling tool to perform meticulous labeling work on the collected image dataset.

[0042] During the labeling process, the embodiment of the present invention follows strict classification standards and subdivides the UI elements in the image into 25 categories, including but not limited to key interface elements such as buttons, text boxes, drop-down menus, icons, sliders, tabs, dialog boxes, etc. This classification system not only covers common UI components, but also incorporates elements with specific functions or appearance characteristics to fully reflect the complexity of modern software interfaces.

[0043] In order to improve the richness and representativeness of the data set, the embodiment of the present invention pays special attention to diversity when selecting screenshots, ensuring that different operating systems, browsers, device types (such as mobile phones, tablets, computers) and UI interfaces of different design styles (such as flat and skeuomorphic) are covered. At the same time, the embodiment of the present invention also takes into account the layout, size, color, state (such as enabled, disabled, hover) of interface elements and other factors to ensure that the data set can fully reflect the challenges in actual testing.

[0044] After the screenshots are classified and labeled, a clearly structured and accurately labeled image dataset is obtained. This dataset not only contains a large number of positive samples (i.e., UI elements that are correctly displayed and easy to identify), but also covers a small number of negative samples (such as blurred, occluded, incorrectly displayed, or illegible UI elements) to simulate various complex situations that may be encountered during actual testing. Through such a carefully designed dataset, the embodiments of the present invention will lay a solid foundation for subsequent research on UI element detection and recognition.

[0045] S2. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture for identification and positioning, and obtain the classification and positioning results of UI elements.

[0046] In the embodiment of the present invention, the object detection model based on the Transformer architecture is deeply customized and optimized, and is specially designed for UI element detection tasks. It can efficiently process complex image data and extract key information.

[0047] Step S2 includes the following sub-steps S21 to S23:

[0048] S21. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture, and convert the screenshots into feature maps containing hierarchical information through the convolutional neural network (CNN) in the object detection model.

[0049] S22. Deeply process the feature map through the encoder in the object detection model to generate feature representation.

[0050] In an embodiment of the present invention, the encoder captures the complex relationships between UI elements through a multi-head self-attention mechanism to generate a more abstract and rich feature representation.

[0051] S23. Accurately classify and locate the feature representation through the decoder in the object detection model, assign a unique category label and precise location coordinates to each detected UI element, and obtain the classification and positioning results of the UI element.

[0052] In the embodiment of the present invention, the decoder can not only identify basic UI elements such as buttons, text boxes, drop-down boxes, check boxes, etc. in the screenshot, but also accurately detect interface components with specific functions and complex structures such as shopping cart widgets and profile widgets. By assigning a unique category label and precise location coordinates (such as a bounding box) to each detected UI element, it provides valuable data support for subsequent test script writing, user behavior simulation, and interface interaction verification.

[0053] S3. Use an optical character recognition engine to extract text from UI elements containing text descriptions to obtain text content associated with the UI elements.

[0054] In the embodiment of the present invention, text extraction aims to capture and parse specific text content from UI elements containing text descriptions, so as to perform subsequent text verification, search or data analysis operations.

[0055] Since the precise boundary coordinates of each UI element have been determined in step S2, such information can be used to accurately crop the image of each UI component from the original UI screenshot. This step ensures the accuracy and efficiency of text extraction and avoids interference from irrelevant information. The cropped UI element screenshots are then saved in a temporary file system for use in this step and subsequent steps. In order to perform the text extraction task, the embodiment of the present invention uses the Google Tesseract open source OCR (optical character recognition) engine, which can easily cope with text extraction requirements in different language environments by integrating the Tesseract framework.

[0056] S4. Based on the classification and positioning results of UI elements and the associated text content, the test flow is parsed through the Transformer model to generate test cases.

[0057] In an embodiment of the present invention, the parser is based on the Transformer model and can understand and parse the test flow description written by the tester. Such test flow adopts an easy-to-understand text format and clearly defines the test scenario, steps, and expected results. The core logic of the parser is closely dependent on the structure of the test flow generated in the previous steps. By deeply analyzing such structure, the parser can accurately identify each test step in the test flow and generate the corresponding test script accordingly.

[0058] The embodiments of the present invention fully consider the diversity of mobile application UIs, and also pay attention to the relative limitation of UI elements. Taking the Android platform as an example, although there are many kinds of UI components, the elements frequently used by developers in actual applications are relatively limited, such as TextView, EditText, Button, ImageButton, ToggleButton, etc. Based on this, the parser has built-in recognition capabilities for such commonly used elements, and defines a set of standard operations that can be performed on each element. For example, for the Button element, the "click" operation is supported; for the EditText element, operations such as "input text (SendKeys)" are supported, thereby simplifying the implementation complexity of the parser and improving the versatility and maintainability of test cases.

[0059] S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases.

[0060] Step S5 includes the following sub-steps S51-S52:

[0061] S51. Optimize the test cases according to the optimization equation, and screen out the exception test cases with the same judgment.

[0062] In the embodiment of the present invention, each test case has a unique execution result in each cycle (pass: 0, fail: 1, not executed: -1). If the test case is not executed, a value of -1 is assigned as the test history information.

[0063] If a test case detected an error in the past, it is more likely to fail in subsequent tests. However, the execution history of the test case is always the same value in all execution cycles, so it is difficult to obtain ideal prediction results using the LSTM network. Therefore, before network training, it is necessary to first filter out the exceptional test cases with the same verdict. The optimization equation is as follows:

[0064]

[0065] Where t represents the test case, F(t) represents the ratio of the test cases with execution result 1 to the total test cases, and F ′ (t) represents the ratio of test cases with an execution result of 0 to the total test cases, num represents the total number of executions of test cases in the first n cycles, s represents the number of test cases with an execution result of 1, m represents the number of test cases with an execution result of 0, i represents the test cases with an execution result of 1, and j represents the test cases with an execution result of 0.

[0066] S52. Iteratively optimize the hidden layer of the LSTM model based on the exception test case, and output the optimized test case.

[0067] S6. Determine the priority of the optimized test cases based on the DeepOrder model, and perform software testing according to the priority. In the embodiment of the present invention, the dual objective function of the DeepOrder model is:

[0068] f(maximize(p),maximize(|q|))

[0069] Where f(·) represents a dual objective function, maximize(p) represents maximizing the fault detection capability of the test suite under the guidance of the historical fault detection capability of the test case, p represents the priority of the test case, maximize(|q|) represents maximizing the number of test cases executed within a given time budget, and q represents a subset of test cases. The optimization of the dual objective function can ensure that test cases that are more likely to find faults are executed first.

[0070] In the embodiment of the present invention, the priority calculation formula for optimizing the test case is:

[0071]

[0072] Where p(t a ) represents the ath optimized test case t a The priority of b Assign a weight in the interval (0,1) in CI period b, so that ∑ b∈1,2,…,z ω b =1, z represents the number of historical CI cycles, ES (a,b) It represents the fault detection capability of the ath optimized test case in CI cycle b.

[0073] In the embodiment of the present invention, the activation function of the DeepOrder model is:

[0074] h(x)=x×tanh(softplus(x))=x×tanh(ln(1+e x ))

[0075] Where h(·) represents the activation function, x represents the input activation value, softplus(·) represents the smoothed ReLU function, and e represents the natural constant. The activation function allows the neural network of the DeepOrder model to learn complex nonlinear relationships. The model can better fit the priority distribution of test cases, thereby improving the accuracy of sorting. This activation function can provide richer nonlinear transformations than ReLU when the input is positive, which helps the network capture more complex features. The transformation of x multiplied by x is used, so that a small amount of negative information is retained, which helps the smooth flow of information.

[0076] The software testing method provided by the embodiment of the present invention aims to improve the automatic detection and verification capabilities of specific UI elements (such as pillar numbers, key buttons or information fields in the software) in software interface testing, especially showing excellent performance when dealing with complex test scenarios containing unbalanced data sets. Its core function is to automatically and accurately identify and verify the status and properties of target UI elements from software interface screenshots through intelligent image recognition and deep learning technology, thereby accelerating the test process and improving test coverage and accuracy.

[0077] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A software testing method combined with LSTM ensemble learning, characterized in that: The following steps are involved: S1. Collect screenshots of different user interfaces, and classify and label the screenshots to obtain an image dataset with accurate labels; S2. Input the screenshots in the image dataset into the object detection model based on the Transformer architecture for identification and positioning, and obtain the classification and positioning results of the UI elements; S3, using an optical character recognition engine to extract text from UI elements containing text descriptions to obtain text content associated with the UI elements; S4, based on the classification and positioning results of UI elements and the associated text content, the test flow is parsed through the Transformer model to generate test cases; S5. Optimize and screen the test cases based on the LSTM model to obtain optimized test cases; S6. Determine the priority of optimized test cases based on the DeepOrder model and perform software testing according to the priority.

2. The software testing method combined with LSTM ensemble learning according to claim 1, characterized in that: The step S2 comprises the following sub-steps: S21, inputting the screenshots in the image dataset into an object detection model based on the Transformer architecture, and converting the screenshots into feature maps containing hierarchical information through the convolutional neural network in the object detection model; S22, deep processing the feature map through the encoder in the object detection model to generate feature representation; S23. Accurately classify and locate the feature representation through the decoder in the object detection model, assign a unique category label and precise location coordinates to each detected UI element, and obtain the classification and positioning results of the UI element.

3. The software testing method combined with LSTM ensemble learning according to claim 1, characterized in that: The step S5 comprises the following sub-steps: S51, optimizing the test cases according to the optimization equation, and screening out the exception test cases with the same judgment; S52. Iteratively optimize the hidden layer of the LSTM model based on the exception test case, and output the optimized test case.

4. The software testing method combined with LSTM ensemble learning according to claim 3 is characterized in that: The optimization equation in step S51 is specifically: Where t represents the test case, F(t) represents the ratio of the test cases with execution result 1 to the total test cases, and F ′ (t) represents the ratio of test cases with an execution result of 0 to the total test cases, num represents the total number of executions of test cases in the first n cycles, s represents the number of test cases with an execution result of 1, m represents the number of test cases with an execution result of 0, i represents the test cases with an execution result of 1, and j represents the test cases with an execution result of 0.

5. The software testing method combined with LSTM ensemble learning according to claim 1, characterized in that: The dual objective function of the DeepOrder model in step S6 is: f(maximize(p),maximize(|q|)) where f(·) represents a dual objective function, maximize(p) means maximizing the fault detection capability of the test suite under the guidance of the historical fault detection capability of the test cases, p represents the priority of the test cases, maximize(|q|) means maximizing the number of test cases executed within a given time budget, and q represents a subset of the test cases.

6. The software testing method combined with LSTM ensemble learning according to claim 1, characterized in that: The priority calculation formula for optimizing the test case in step S6 is: Where p(t a ) represents the ath optimized test case t a The priority of b Assign a weight in the interval (0,1) in CI period b, so that ∑ b∈1,2,…,z ω b =1, z represents the number of historical CI cycles, ES (a,b) It represents the fault detection capability of the ath optimized test case in CI cycle b.

7. The software testing method combined with LSTM ensemble learning according to claim 1, characterized in that: The activation function of the DeepOrder model in step S6 is: h(x)=x×tanh(softplus(x))=x×tanh(ln(1+e x )) Where h(·) represents the activation function, x represents the input activation value, softplus(·) represents the smoothed ReLU function, and e represents a natural constant.

Images