A UI function automatic test tool and method

Abstract

The application discloses a UI function automatic test tool and method, and relates to the technical field of automatic test.The application extracts prompt words by using a prompt word generation module; when a return result of a LLM large model is not obtained, a processing module carrying the LLM large model ends test; when the return result of the LLM large model is obtained, operation suggestions are generated based on the return result of the LLM model, the operation suggestions are input into an execution engine, and the execution result of the execution engine is used as input information of the prompt word generation module for next round test.The application uses the LLM large model with language understanding ability, extensive knowledge reserve and continuous learning and optimization ability to solve the problem of manual operation of a UI interface, can intelligently infer operation logic, provides accurate operation suggestions, greatly improves operation efficiency and accuracy, and brings convenience and intelligent interface test experience to users.

Description

Technical Field

[0001] This invention relates to the field of automated testing technology, and in particular to a UI function automated testing tool and method. Background Technology

[0002] Currently, mainstream user interface testing (GUI testing or UI testing) requires manual operation of the interface of the software under test. A single test function requires multiple manual clicks, which is very inefficient. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides an automated UI function testing tool and method.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A UI functionality automated testing tool, comprising:

[0006] The input module is used to acquire input data; the input data includes: UI element information, test case description information, and prompt word rule information;

[0007] A prompt word generation module, connected to the input module, is used to filter the input data to obtain prompt words;

[0008] The processing module, equipped with an LLM large model, is connected to both the prompt word generation module and the prompt word generation module. It determines whether a return result from the LLM large model is obtained based on the prompt word. If no return result from the LLM large model is obtained, the test ends. If a return result from the LLM large model is obtained, operation suggestions are generated based on the LLM model's return result, and these suggestions are input into the execution engine. The execution result of the execution engine is then used as input information for the prompt word generation module for the next round of testing.

[0009] Optionally, the large LLM model is a trained and tuned LLM model;

[0010] The LLM model is a language generation model built using the Transformer model, self-attention mechanism, and feedforward neural network.

[0011] Optionally, the training and tuning process of the LLM model includes:

[0012] Data in the UI testing domain is pre-labeled and cleaned to obtain labeled data, and a training set and a validation set are generated based on the labeled data; the training set includes training sample pairs; the training sample pairs take prompt words as input information and sample labels as output information;

[0013] Build an initial LLM model, freeze some parameters of the initial LLM model, and set the learning rate;

[0014] The initial LLM model is trained using the training set to obtain the trained LLM model;

[0015] The performance of the trained LLM model is validated using a validation set, and the learning rate is adjusted based on this performance to complete the tuning.

[0016] Optionally, the adamw optimizer can be used to perform initial adjustments to the LLM model;

[0017] After the initial adjustment is completed, the SGD with momentum is used to perform a secondary adjustment of the LLM model to obtain the large LLM model.

[0018] Optionally, the data processing in the LLM model includes:

[0019] Convert the input data into index values;

[0020] The index value is mapped to a vector space to generate a first feature vector;

[0021] A position encoding vector is introduced, and the position encoding vector is superimposed with the first feature vector to obtain a new vector;

[0022] A self-attention mechanism is used to focus attention on different positions of the first feature vector to obtain the second feature vector;

[0023] The new vector and the second feature vector are added together and then layer normalization is performed to obtain the normalized vector;

[0024] The normalized vector is used as the input to the feedforward neural network to obtain the output result.

[0025] Optionally, the process by which the prompt word generation module filters the input data to obtain prompt words includes:

[0026] Invalid UI element information is removed from the input data, and the description information of the test cases and the prompt word rule information are concatenated to obtain the prompt words.

[0027] A method for automated UI functionality testing, applied to the aforementioned automated UI functionality testing tool; the method includes:

[0028] Obtain input data; the input data includes: UI element information, test case description information, and prompt word rule information;

[0029] The input data is filtered to obtain prompt words;

[0030] Input the prompt word into the LLM large model to determine whether the LLM large model returns a result;

[0031] The test ends when no results are returned from the large LLM model.

[0032] When the return result of the LLM model is obtained, operation suggestions are generated based on the return result of the LLM model, and the operation suggestions are input into the execution engine. The execution result of the execution engine is used as the input information of the prompt word generation module for the next round of testing.

[0033] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0034] This invention utilizes a large LLM model with language understanding capabilities, extensive knowledge reserves, and continuous learning and optimization capabilities to solve the problem of manual UI interface operation. It can intelligently infer operation logic and provide accurate operation suggestions, greatly improving the efficiency and accuracy of operation and bringing users a convenient and intelligent interface testing experience. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This invention provides a data processing flowchart for the UI function automated testing tool.

[0037] Figure 2 A schematic diagram of the workflow of the LLM model provided by this invention;

[0038] Figure 3 The flowchart of the UI function automated testing method provided by the present invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] The purpose of this invention is to provide a UI function automated testing tool and method that does not require manual intervention and can improve the efficiency of UI testing.

[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0042] The UI function automated testing tool provided by this invention includes: an input module, a prompt word generation module, and a processing module.

[0043] The input module is used to acquire input data. Input data includes: UI element information (i.e., page information), test case descriptions, and prompt rule information. The UI element information can be the DOM (Domain of Documentation) tree information of the software under test. The test case descriptions can be descriptions of bugs in the software under test. The prompt rule information defines rules for the entire task; for example, if the program stops, the task is considered complete.

[0044] The prompt generation module connects to the input module and is primarily used to filter the input data to obtain prompts. The filtering rules can be set to: remove invalid UI element information, and concatenate the test case description and prompts into a string. For example, the format could be: UI element information + test case description + prompt rules. Invalid UI elements refer to elements with the `image` attribute in the software's DOM structure. These elements only contain the image's address and lack detailed image content information; for example, the `image` tag only contains the image address and URL, not the image content.

[0045] The processing module is equipped with an LLM (Large Language Model). It connects to both the prompt word generation module and the prompt word input module, primarily used to determine whether to obtain a return result from the LLM model based on the prompt words. If no LLM model return result is obtained, the test ends. If an LLM model return result is obtained, action suggestions are generated based on the LLM model's response. These suggestions are then input into the execution engine, and the execution engine's result is used as input to the prompt word generation module for the next round of testing. In other words, action suggestions are obtained based on the LLM model's response. An action engine (e.g., a page execution engine) is then executed based on these suggestions. The results are analyzed, and information from the current page is used as input for the next round of testing. This testing process is repeated continuously until the task ends.

[0046] The LLM large model can begin replicating errors by understanding the current state of the test cases and the application through cue words. The cue words describe the goal and workflow of the task. The LLM large model provides a detailed explanation of the workflow, including supported UI actions and their purpose, the expected response for each iteration, and the conditions for terminating replication. With this detailed cue, the approach of using LLM to automate test case implementation gains the advantage of supporting advanced actions, such as suggesting when to perform a rotation operation in a specific direction, suggesting the direction of a swipe or scroll operation, and intelligently filling multiple text fields.

[0047] The specific implementation process of the UI function automated testing tool provided above is as follows: Figure 1 As shown.

[0048] The large LLM model is a trained and tuned version of the LLM model. The LLM model is a language generation model based on the Transformer model. It employs self-supervised learning for pre-training, enabling the model to learn a large amount of linguistic knowledge and semantic understanding. Fine-tuning this model structure on data from specific vertical domains can achieve better results in those domains.

[0049] The core component of the LLM model is the Transformer model, which consists of multiple encoder-decoder layers. Each layer contains a multi-head self-attention mechanism and a feed-forward neural network.

[0050] Based on this, the workflow of the LLM model is as follows:

[0051] Step 1: Input and Output. During training, the input is the cue word, and the output is the sample label. For example, given the phrase "The capital of China is Beijing," the input is "The capital of China is," and the label "Beijing is" is the output for training. Figure 2 The outputs (shifted right) in the text.

[0052] Step 2: Convert the input and label text information into index values ​​in a dictionary, and map these index values ​​to a vector space using inputEmbedding / outputEmbedding to generate a feature vector. The purpose of embedding is to map words or characters into continuous vector representations.

[0053] Step 3: To introduce sequence order information into the model, positional encoding is used. Positional encoding adds a vector to each position in the input sequence, giving each position a unique encoded representation. Positional encoding plays a crucial role in introducing sequence order information in the Transformer model. It enables the model to perceive the relative positional relationships between different positions in the input sequence, better understand semantics and context, and thus improve the model's performance in natural language processing tasks. The positional encoding vector is added to the input vector to obtain a new sequence. Positional encoding PE (*) The formula is as follows:

[0054]

[0055]

[0056] Where pos is the position, i is the dimension index, and d is the position. model This refers to the dimension of the model. The positional encoding input is the sequence index. The encoded result is added to the input embedding / output embedding result from step 2.

[0057] Step 4: Self-Attention Mechanism: The self-attention mechanism allows the model to focus its attention on different positions in the input sequence during encoding and decoding. The self-attention mechanism in an LLM model consists of multiple attention heads, each learning a different attention pattern. The calculation formula for the self-attention mechanism is as follows:

[0058]

[0059] Where Q is the query vector, K is the key vector, V is the value vector, and d kThis is the dimension of the attention head. The result obtained in step 3 is used as the input to the self-attention mechanism, and three copies are made, namely Q, K, and V, to obtain the encoded feature vector through the self-attention mechanism.

[0060] Step 5: Add the feature vector obtained after step 4 to the output of step 3, and then perform layer normalization to normalize the feature of each dimension to a vector with a mean of 0 and a variance of 1.

[0061] Step 6: The vector obtained in Step 5 is used as input to the Feed-Forward Neural Network. The Feed-Forward Neural Network processes the output of the self-attention mechanism in each encoder-decoder layer. It consists of a fully connected layer activation function, which enhances the model's ability to represent the input sequence.

[0062] The calculation formula for a feedforward neural network is as follows:

[0063] FFN(x)=RELU(xw1+b1)w2+b2.

[0064] Here, x is the output of the self-attention mechanism, and w1, b1, w2, and b2 are all learnable parameters.

[0065] Step 7: The above six steps represent the processes of the encoding and decoding modules. Among them, Figure 2 The left side represents the encoding module, and the right side represents the decoding module. Since this encoding-decoding process is existing technology, it will not be described in detail here. Throughout the process, the output of the encoding module serves as the Q, K, and V of the decoding module's self-attention, and is the input of the decoding module. The attention mechanism in the decoding process is Masked Multihead Attention. The role of this attention mechanism is to ensure that when the decoder processes self-attention, it can only see the previously generated parts at each position of the generated sequence, and not the subsequent content. This is to prevent information leakage and the use of future information, thereby ensuring that the model generates outputs in the correct order. (Note that this is only needed during training, not during actual inference). During inference, the outputs are the prediction results from the previous round.

[0066] In summary, LLM utilizes the Transformer model, self-attention mechanism, and feedforward neural network to construct a powerful language generation model. Through pre-training and fine-tuning, it can generate coherent and semantically reasonable text output, demonstrating a certain level of language understanding and generation capabilities.

[0067] The LLM model's pre-training phase utilizes large-scale unsupervised text data, trained through self-supervised learning. The LLM model predicts the next word or a segment of text based on a portion of the input text, thereby learning the statistical patterns and semantic information of language. Sample data from the UI testing domain is collected and labeled within a defined scenario to further improve the LLM's performance. Specifically, the pre-training and fine-tuning process is as follows:

[0068] 1. Data Preparation. Data in the UI testing domain is pre-labeled using other open-source models, followed by a small amount of manual cleaning to obtain relatively clean labeled data.

[0069] 2. Freeze some parameters. For large models, it's common practice to freeze some parameters, keeping their weights unchanged and not participating in the fine-tuning process. This reduces the number of parameters to be fine-tuned, speeds up training, and preserves the knowledge of the pre-trained model. Experiments show that freezing the encoding part while fine-tuning the decoding part yields the best overall performance.

[0070] 3. Learning Rate Adjustment. When fine-tuning the model, it's often necessary to reduce the learning rate to avoid over-adjusting the weights of the pre-trained model. A smaller learning rate allows the model to learn task-specific features more stably. Consider using a lower learning rate and adjusting it based on the performance on the validation set. Throughout the fine-tuning process, the learning rate is set to 2e-5 and decays with each iteration, multiplying by 0.1 every 100 iterations.

[0071] 4. Optimizer selection. In the early stages of iteration, the Adamw optimizer is used to accelerate convergence, and in the later stages, the SGD optimizer with momentum is used to fine-tune the performance.

[0072] 5. Training batch size. Using a larger batch size can generally improve training efficiency, but it may also lead to insufficient GPU memory or unstable training. When fine-tuning the model, the batch size can be increased appropriately, but it is necessary to ensure a stable training process within the constraints of available resources.

[0073] Furthermore, after implementing the LLM large model response recommendations, it is necessary to provide feedback on the implementation.

[0074] Given the probabilistic nature of language models, providing feedback is crucial, as they may sometimes fail to deliver output in the desired pattern. For example, if an LLM big model wants to suggest an action to complete the "open menu" step but incorrectly interprets "open" as an action that should be "clicked," a feedback mechanism can inform it that "open" is an invalid action, allowing the LLM big model to correct its response. However, without feedback, the LLM big model might simply assume the action was executed successfully, and the reason the page remains unchanged afterward could be that the "menu" button was unresponsive. Therefore, by incorporating feedback into subsequent prompts, the LLM big model can recognize errors, correct its suggestions, and validate the accuracy of its responses in future interactions, making the iterative process more efficient.

[0075] The feedback mechanism is the process of obtaining information from the current page after an operation, generating new prompts, and inputting them into the LLM large model.

[0076] Feedback and updated application UI information (which may be the same as or different from before) are sent as follow-up hints to the LLM big model, enabling it to make the next appropriate suggestion for the next iteration. ReBL iterates through this process until the LLM model has successfully triggered an error, or until the termination criteria are met.

[0077] Termination criteria were defined in the prompt word rules, such as the program having stopped or all pages having been traversed. Specifically, crash verification is straightforward, as most crashes will display a message like "Application has stopped." By capturing application UI information, ReBL can verify non-crash errors related to UI display, such as duplicate text in the search bar or inconsistent display of added notes.

[0078] The language processing capabilities of LLM big data have proven valuable in identifying incorrect language changes. For example, when the language settings change but the display remains the same. Once an LLM big model validation error has been triggered, it notifies the execution to terminate the replication process. On the other hand, if no error behavior is triggered in the current iteration, the LLM big model is instructed to continue to the next iteration until a timeout occurs.

[0079] Based on the above description, the present invention uses an LLM large model to solve the problem of manual UI interface operation, which has many main advantages:

[0080] First, the LLM large model possesses powerful language understanding and generation capabilities, enabling it to parse and understand complex task descriptions and instructions. Furthermore, it can intelligently infer the next step's operation logic based on target and page element information, combined with historical operation records, significantly reducing the burden of manual operation.

[0081] Secondly, the LLM large model possesses a broad knowledge base, enabling it to understand the operational logic of various applications and interfaces. Based on specific goals and page element information, it can extract relevant operational steps from existing knowledge and generate accurate operational suggestions. This knowledge-based reasoning capability makes the large model more flexible and intelligent when handling complex interface testing.

[0082] Furthermore, the LLM large model can continuously optimize its recommendation strategy by learning from historical operation records. It can analyze the correctness of historical operations and use feedback information to adjust and improve itself, thereby increasing the accuracy and efficiency of operations. This ability to continuously learn and optimize allows the large model to gradually adapt to the operational needs of different applications and interfaces, providing more personalized and intelligent operation suggestions.

[0083] Furthermore, this invention provides a method for automated UI function testing, applicable to the aforementioned automated UI function testing tool. For example... Figure 3 As shown, the method includes:

[0084] Step 100: Obtain input data. Input data includes: UI element information, test case descriptions, and prompt word rules.

[0085] Step 101: Filter the input data to obtain prompt words.

[0086] Step 102: Input the prompt word into the LLM large model to determine whether the LLM large model return result is obtained.

[0087] Step 103: End the test if no LLM large model return result is obtained.

[0088] Step 104: When the return result of the LLM model is obtained, operation suggestions are generated based on the return result of the LLM model, and the operation suggestions are input into the execution engine. The execution result of the execution engine is used as the input information of the prompt word generation module for the next round of testing.

[0089] As can be seen, this invention utilizes a large LLM model to solve the problem of manually operated UI interfaces, possessing language understanding capabilities, extensive knowledge reserves, and the ability to continuously learn and optimize. It can intelligently infer operational logic and provide accurate operational suggestions, greatly improving operational efficiency and accuracy, and bringing users a convenient and intelligent interface testing experience.

[0090] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0091] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A UI function automated testing tool, characterized in that, include: The input module is used to acquire input data; The input data includes: UI element information, test case description information, and prompt word rule information; the UI element information is the DOM tree information of the software under test, the test case description information is the BUG description of the software under test, and the prompt word rule information includes the rule definition and termination criteria of the task, the termination criteria include the occurrence of "the program has stopped" or "all pages have been traversed"; The prompt word generation module, connected to the input module, is used to filter the input data to obtain prompt words; remove invalid UI element information in the DOM tree information that has the attribute "image" and only contains image address and URL, but does not contain image content, and concatenate the UI element information after removing invalid UI element information, the description information of the test case, and the prompt word rule information into a string; The processing module, equipped with an LLM large model, is connected to the prompt word generation module and the execution engine respectively. It is used to determine whether to obtain the return result of the LLM large model based on the prompt word. When no return result of the LLM large model is obtained, the test ends. When the return result of the LLM large model is obtained, operation suggestions are generated based on the return result of the LLM large model, and the operation suggestions are input into the execution engine. The execution result of the execution engine is used as the input information of the prompt word generation module for the next round of testing. The processing module is further configured to: after the execution engine executes the operation suggestion, obtain the UI element information of the current page, generate execution feedback based on the execution result of the execution engine, and use the execution feedback and the updated UI element information as input to the prompt word generation module to generate subsequent prompts and send them to the LLM large model, so that the LLM large model corrects the returned result when the operation suggestion is invalid; when the UI element information of the current page verifies that a BUG has been triggered, notify the execution engine to terminate the test; when the BUG has not been triggered, continue to the next round of testing until the termination standard is reached or a timeout occurs.

2. The UI function automated testing tool according to claim 1, characterized in that, The large LLM model is a trained and tuned LLM model; The LLM model is a language generation model built using the Transformer model, self-attention mechanism, and feedforward neural network.

3. The UI function automated testing tool according to claim 2, characterized in that, The training and tuning process of the LLM model includes: Data in the UI testing domain is pre-labeled and cleaned to obtain labeled data, and a training set and a validation set are generated based on the labeled data; the training set includes training sample pairs; the training sample pairs take prompt words as input information and sample labels as output information; Build an initial LLM model, freeze some parameters of the initial LLM model, and set the learning rate; The initial LLM model is trained using the training set to obtain the trained LLM model; The performance of the trained LLM model is validated using a validation set, and the learning rate is adjusted based on this performance to complete the tuning.

4. The UI function automated testing tool according to claim 3, characterized in that, The Adamw optimizer was used to perform initial adjustments to the LLM model; After the initial adjustment is completed, the SGD with momentum is used to perform a secondary adjustment of the LLM model to obtain the large LLM model.

5. The UI function automated testing tool according to claim 2, characterized in that, The data processing in the LLM model includes: Convert the input data into index values; The index value is mapped to a vector space to generate a first feature vector; A position encoding vector is introduced, and the position encoding vector is superimposed with the first feature vector to obtain a new vector; A self-attention mechanism is used to focus attention on different positions of the first feature vector to obtain the second feature vector; The new vector and the second feature vector are added together and then layer normalization is performed to obtain the normalized vector; The normalized vector is used as the input to the feedforward neural network to obtain the output result.

6. A method for automated testing of UI functions, characterized in that, The method is applied to the UI function automated testing tool as described in any one of claims 1-5; the method includes: Obtain input data; the input data includes: UI element information, test case description information, and prompt word rule information; The input data is filtered to obtain prompt words; Input the prompt word into the LLM large model to determine whether the LLM large model returns a result; The test ends when no results are returned from the large LLM model. When the return result of the LLM model is obtained, operation suggestions are generated based on the return result of the LLM model, and the operation suggestions are input into the execution engine. The execution result of the execution engine is used as the input information of the prompt word generation module for the next round of testing.

Images