User interface test method and device, equipment, storage medium and product

Abstract

The invention relates to the technical field of user interface automatic testing, and discloses a user interface testing method and device, equipment, a storage medium and a product, and the method comprises the steps: inputting the image information of a user interface into a preset image processing model, obtaining the feature vector and coordinate information of each interface element, the preset image processing model is constructed by combining the optical character recognition model and the image classification model, and is used for synchronously extracting the characteristics of the character type elements and the icon type elements; retrieving the feature vectors based on a preset picture classification vector library, and determining annotation category information corresponding to the feature vectors; and testing the user interface according to the labeled category information and the coordinate information. The text and icon features are synchronously extracted through the merging model, the feature vectors and the coordinate information of the interface elements are obtained at a time, semantic retrieval is conducted in combination with the vector library, the labeling information corresponding to the vectors is output, and automatic, high-precision and cross-device user interface testing is achieved.

Description

Technical Field

[0001] The present application relates to the technical field of user interface automated testing, and in particular to a user interface testing method, apparatus, device, storage medium, and product. Background Art

[0002] Traditional UI (User Interface) testing requires manual selection of source and target images. This lacks specific standards and cannot guarantee that the selected area is most suitable for subsequent image comparison. For example, whether the area has a border or a background is important. If a large number of repeated and very similar icons are stacked together, this may lead to poor recognition and unsatisfactory test results. Summary of the Invention

[0003] The main purpose of this application is to provide a user interface testing method, device, equipment, storage medium and product, aiming to solve the technical problem that traditional UI testing requires manual selection of source target images, resulting in poor recognition effect and unsatisfactory test results.

[0004] To achieve the above objectives, the present application proposes a user interface testing method, which includes:

[0005] Inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract the features of text elements and icon elements;

[0006] Searching the feature vector based on a preset image classification vector library to determine the labeling category information corresponding to the feature vector;

[0007] The user interface is tested according to the annotation category information and the coordinate information.

[0008] Optionally, before the step of inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element, the step further includes:

[0009] Obtaining a preset training image set, and enhancing images in the preset training image set to obtain an initial training image set;

[0010] Training a subject detection model based on the initial training image set to obtain a preset image processing model, wherein the subject detection model is a detection model obtained by combining an optical character recognition model and an image classification model;

[0011] The images in the initial training image set are vectorized using the preset image processing model to construct a preset image classification vector library.

[0012] Optionally, the step of training the subject detection model based on the initial training image set to obtain a preset image processing model includes:

[0013] Labeling the images in the initial training image set to generate a label file containing category labels and location information;

[0014] Converting the initial training image set into a preset annotation format based on the label file to obtain a target training image set;

[0015] The subject detection model is trained according to the target training image set to obtain a preset image processing model.

[0016] Optionally, the step of vectorizing the images in the initial training image set using the preset image processing model to construct a preset image classification vector library includes:

[0017] Establishing a picture database based on the initial training image set;

[0018] Extracting feature information of images in the image database using the preset image processing model to obtain a feature vector database;

[0019] Establishing an index relationship between the feature vector database and the image database, and defining a classification name for the index relationship;

[0020] The category name and the index relationship are stored in the feature vector database to obtain a preset picture category vector library.

[0021] Optionally, the step of searching the feature vector based on a preset image classification vector library to determine the labeled category information corresponding to the feature vector includes:

[0022] Calculating the similarity between the feature vector and the index vector in the preset image classification vector library;

[0023] The index vector with the greatest similarity is determined, and the labeled category information of the feature vector is returned through the index vector.

[0024] Optionally, the step of testing the user interface according to the annotation category information and the coordinate information includes:

[0025] Determining the operation logic of each of the interface elements based on the annotation category information;

[0026] Generate a corresponding test script according to the operation logic;

[0027] The processing script is executed based on the coordinate information to obtain a test result of each interface element in the user interface.

[0028] In addition, to achieve the above-mentioned purpose, the present application also proposes a user interface testing device, which includes:

[0029] A feature extraction module, configured to input the user interface image information into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract the features of text elements and icon elements;

[0030] A category retrieval module is used to search the feature vector based on a preset image classification vector library to determine the labeled category information corresponding to the feature vector;

[0031] An interface testing module is used to test the user interface according to the annotation category information and the coordinate information.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a user interface testing device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the user interface testing method described above.

[0033] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the steps of the user interface testing method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps of the user interface testing method described above.

[0035] This application discloses inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by merging an optical character recognition model and an image classification model, and is used to synchronously extract the features of text elements and icon elements; the feature vector is retrieved based on a preset image classification vector library to determine the annotation category information corresponding to the feature vector; and the user interface is tested based on the annotation category information and the coordinate information. This application synchronously extracts text and icon features by merging models, obtains the feature vector and coordinate information of each interface element at one time, and performs semantic retrieval in combination with the vector library, outputs the annotation information corresponding to the vector, and realizes automated, high-precision, cross-device user interface testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0037] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 This is a flowchart of the first embodiment of the user interface testing method of the present application;

[0039] Figure 2 This is a flow chart of the second embodiment of the user interface testing method of the present application;

[0040] Figure 3 A flowchart of the model training for the user interface testing method of this application;

[0041] Figure 4 This is a flowchart of the third embodiment of the user interface testing method of the present application;

[0042] Figure 5 This is a schematic diagram of the module structure of the user interface testing device according to an embodiment of the present application;

[0043] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the user interface testing method in the embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The main solution of the embodiment of the present application is: inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element, wherein the preset image processing model is constructed by combining an optical character recognition model and an image classification model, and is used to simultaneously extract the features of text elements and icon elements; searching the feature vector based on a preset image classification vector library to determine the labeling category information corresponding to the feature vector; testing the user interface based on the labeling category information and the coordinate information

[0048] UI testing, as a crucial component of functional testing, has long been a focus of automated testing. This has spawned a variety of technologies, such as Appium and Airtest, for static regression testing of UI interfaces. These tools integrate image comparison algorithms to address requirements such as image verification codes and automated assertions that cannot be met using XPath.

[0049] Appium provides several methods for locating elements using image recognition, such as the images-plugin. This plugin supports the -image locator strategy, allowing Appium to specify an image file of the element you want to locate. If Appium can find a screen area that matches your image, it packages the information about that area into a standard WebElement and sends it back to the Appium client. Airtest supports multiple methods for identifying image controls. Essentially, these methods are similar to Appium's. Both methods manually capture an image of a specific area as the source image to be searched. When the test case is executed, the element object is found across the entire interface through image matching. However, it is more user-friendly than Appium in terms of UI interaction. Appium and Airtest are representative examples of element object handling in the field of UI testing. Appium has more methods for obtaining element objects, while Airtest was originally developed for game testing, so it prefers image-based element recognition. The two methods share similar image recognition logic.

[0050] Traditional solutions don't completely solve the element recognition problem in UI testing. Both Appium and Airtest require manual processing of the target source image, making them inconvenient. Manual selection of the selected area is inconsistent, sometimes too large and sometimes too small, and there's no guarantee that the selected area will be optimal for subsequent image comparison. For example, the presence of a border or background can affect the area. Recognition can be poor when a large number of duplicate or very similar icons are stacked together. Image recognition, due to limitations, can't achieve 100% accuracy. Often, image recognition methods search for corresponding images on the current device screen, but they can't match the accuracy of human recognition. Logically, they can only try to find the best possible result. This often leads to recognition errors. Traditional image recognition methods are even less accurate in complex backgrounds with multiple colors. Existing technologies also suffer from generalization issues. When a target image element captured on one interface is searched on another, accuracy can't be guaranteed. For example, an icon captured on phone A for the same app might fail on phone B. This is due to factors like phone resolution and color.

[0051] Therefore, this application provides an image classification model trained using visual deep learning, and combines this model to develop a UI testing method based on image classification, which achieves accurate and efficient recognition of UI elements and optimizes the testing process.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, feature extraction, vector retrieval, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. This embodiment and the following embodiments will be described below using a user interface testing device as an example.

[0053] Based on this, the embodiment of the present application provides a user interface testing method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the user interface testing method of the present application.

[0054] In this embodiment, the user interface testing method includes:

[0055] In step S10, the image information of the user interface is input into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract the features of text elements and icon elements.

[0056] It should be noted that user interface image information refers to the visual images presented by the interface when a user uses software, a website, or an application, which are converted into image data that can be processed by a computer. This image information can be screenshots, frame images of interface recordings, etc., and includes the visual features of various elements on the interface (such as text, icons, buttons, etc.). The preset image processing model is a pre-trained and configured model for processing image data. It is constructed by combining an OCR (Optical Character Recognition) model and an image classification model. It has a specific structure and parameters and is capable of analyzing and processing input user interface image information.

[0057] It's also important to note that a feature vector is a mathematical representation of the characteristics of an interface element. The appearance, shape, color, and other features of each interface element (such as text or icons) are quantified into a set of numerical values, forming a vector. This feature vector allows for comparison, identification, and classification of interface elements in a computer. Coordinate information represents the location of an interface element within an image.

[0058] It should be understood that the algorithm suitable for identifying buttons in image recognition is the convolutional neural network. A convolutional neural network is a deep learning algorithm particularly well-suited for image recognition tasks. It can automatically learn image features and has achieved excellent performance in areas such as image classification, object detection, and face recognition. UI testing requires the recognition of interface elements such as icons, buttons, text, and input boxes, all at once, without the need to recognize text separately. When selecting a model architecture, consider improving the pre-trained CRNN (Convolutional Recurrent Neural Network) model and adding a set of feature detection models to complete the model construction.

[0059] It can be understood that the image information of the user interface is input into the preset image processing model. The image file can be loaded into the model through the image reading interface, or the interface image can be directly captured from the screen and input into the model. When the preset image processing model processes the input image, the OCR model part will recognize the text area in the image and extract the features of the text. The image classification model part will analyze elements such as icons and extract the features of the icons.

[0060] It is understood that during the feature extraction process, the model simultaneously determines the position of each interface element in the image and records its coordinate information. In a feasible example, the bounding box of each element can be found by segmenting and locating the image, thereby obtaining its coordinate information.

[0061] Step S20: searching the feature vector based on a preset image classification vector library to determine the labeling category information corresponding to the feature vector.

[0062] It should be noted that the preset image classification vector library is a pre-built database that stores feature vectors of a large number of images and the corresponding annotated category information for these vectors. These feature vectors are obtained by extracting features from various types of images, and the annotated category information clarifies the category to which each image belongs, such as "button," "icon," "text box," etc. The annotated category information is a category label with clear semantics assigned to images or image elements. It is used to describe the type or function of the element and help identify and distinguish different interface elements.

[0063] It should be understood that determining the labeled category information corresponding to the feature vector may involve calculating a similarity between the feature vector extracted from the user interface image and each index vector in a preset image classification vector library, such as cosine similarity or Euclidean distance. Based on the calculated similarity values, the index vector with the highest similarity to the input feature vector is found.

[0064] It is understandable that since the preset image classification vector library is obtained through a large amount of data training and annotation, the feature vectors and annotation categories therein are highly accurate and representative. Through similarity matching, the corresponding annotation category can be found more accurately for the input feature vector, thereby improving the accuracy of interface element recognition. At the same time, the calculation process of vector retrieval is relatively simple and fast, and it can complete the comparison and matching of a large number of vectors in a relatively short period of time, thereby improving the efficiency of the entire user interface testing process. It can also adapt to different types and styles of user interfaces. As long as the vector library contains the corresponding feature vectors and annotation categories, it can accurately identify new interface elements and avoid secondary training of the model.

[0065] Step S30: testing the user interface according to the annotation category information and the coordinate information.

[0066] It is understood that testing the user interface based on the annotation category information and the coordinate information can be performed by recording a test script based on the annotation category information. For example, if the annotation category is "button", the test script may be click; if it is "input box", the test script may be input text; if it is "drop-down menu", the test script may be expansion, selection of options, etc.

[0067] It should be understood that the generated test script is executed in conjunction with the coordinate information. The coordinate information accurately locates the position of the interface element on the screen. The test program can simulate user operations such as clicks and inputs based on these coordinates. After performing the operation, the actual result is checked to see if it matches the expected result, thereby determining whether the interface element is working properly.

[0068] In one example, the annotation category is "Search Input Box" and the coordinate information is (x2, y2). The test logic is to enter a keyword in the input box, and click the search button to display relevant search results. In this case, record a test script, first simulate entering a specific keyword, such as "mobile phone," at the coordinates (x2, y2), then click the search button to check whether the page displays a response related to "mobile phone." If the displayed results meet the expectations, the test passes; if the results are empty or irrelevant, it indicates a problem with the input box or search function.

[0069] In this embodiment, the image information of the user interface is input into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by merging an optical character recognition model and an image classification model, and is used to simultaneously extract the features of text elements and icon elements; the feature vector is searched based on a preset image classification vector library to determine the annotation category information corresponding to the feature vector; and the user interface is tested based on the annotation category information and the coordinate information. By merging the model to simultaneously extract text and icon features, the feature vector and coordinate information of each interface element are obtained at one time, and semantic retrieval is performed in combination with the vector library, and the annotation information corresponding to the vector is output, thereby realizing automated, high-precision, and cross-device user interface testing.

[0070] Reference Figure 2 , Figure 2 This is a flow chart of the second embodiment of the user interface testing method of the present application. Based on the above-mentioned first embodiment, the second embodiment of the user interface testing method of the present application is proposed.

[0071] In the second embodiment, before step S10, the method further includes:

[0072] Step S01: obtaining a preset training image set, and enhancing images in the preset training image set to obtain an initial training image set.

[0073] It's important to note that a pre-set training image set refers to a collection of image data prepared in advance for model training. This data is typically relevant to the target task. In user interface testing scenarios, this set can include user interface images of varying styles, layouts, and content, from which the model can learn features. The initial training image set is an enhanced set of images, containing more diverse image data to improve the model's generalization capabilities.

[0074] It's understandable that the images in the preset training image set can be screenshots of user interfaces from different applications and versions, or they can be collected from public datasets or actual business systems. Furthermore, to improve the model's generalization capabilities, image enhancement operations such as scaling and noise addition are required on the original images.

[0075] In one example, deep learning is used to solve the problem of UI element recognition. First, a UI dataset containing element information is obtained. Currently, two open and widely used datasets in the industry are Rico and ReDraw. These two datasets can be merged and supplemented with other custom data image collections to generate a training image set for training. ReDraw is a collection of Android screenshots containing GUI (Graphical User Interface) metadata and annotated GUI component images. It includes 15 categories, including RadioButton, ProgressBar, Switch, Button, and CheckBox, with 14,382 UI images and 191,300 labeled GUI components. Rico is the largest mobile UI dataset to date, supporting five data-driven application categories: design search, UI layout generation, UI code generation, user interaction modeling, and user perception prediction. The pre-defined training image set is obtained by unifying the Rico and ReDraw datasets, defining their size ranges, and re-annotating them. The merged dataset is then supplemented with a target application-related dataset to create the pre-defined training image set. The image set is then enhanced using techniques such as geometric transformation, color adjustment, and noise addition. By generating diversified image data through image enhancement, the model can learn more features and patterns in different situations, so that it can better identify and process new and unseen images, thereby improving the generalization ability of the model.

[0076] Step S02: training a subject detection model based on the initial training image set to obtain a preset image processing model, wherein the subject detection model is a detection model obtained by combining an optical character recognition model and an image classification model.

[0077] It should be noted that the subject detection model is a combination of an optical character recognition model and an image classification model. The optical character recognition model primarily identifies and extracts textual elements from images, while the image classification model primarily extracts icon elements from images.

[0078] In one example, the combined OCR model and image classification model are deployed on the server side and provided through an API (Application Programming Interface). The client parser pre-processes the image and compiles it into base64 encoding; after calling the API interface, formatted data is returned. The formatted data contains the boundary coordinates of the element object, which are the normalized coordinates of the four vertices of a rectangle. When the tester records the script, as long as the mouse is placed on an element, the element will be highlighted, so there is no need to manually process the target source image. The trained image classification model can distinguish similar images more accurately, and because the element object is judged based on image feature extraction, it has better adaptability to different resolutions.

[0079] Furthermore, in order to enable the model to learn more features in different situations, ensure the quality of training data, and improve the generalization ability of the model, the step S02 may include:

[0080] The images in the initial training image set are labeled to generate a label file containing category labels and location information; based on the label file, the initial training image set is converted into a preset labeling format to obtain a target training image set; and a subject detection model is trained according to the target training image set to obtain a preset image processing model.

[0081] It should be noted that the category label is used to describe the category to which the elements in the image belong. It is an identifier for classifying elements and helps the model learn the characteristics of different types of elements. Position information refers to the specific location of the interface element in the image. The label file refers to a file that stores annotation information, which contains the category label and position information of each element in the image, providing data support for subsequent model training. The preset annotation format is a predefined and unified annotation data storage format that facilitates the model to read and process annotation information, such as the COCO (Common Objects in Context) format. The target training image set is a training image set that has been annotated and format converted. Its annotation information conforms to the preset annotation format and can be directly used for model training.

[0082] It is understandable that when annotating the images in the initial training image set and generating a label file containing category labels and location information, you can manually view the image and use professional annotation tools (such as LabelImg, RectLabel, etc.) to draw a bounding box for each interface element in the image and assign a category label to it. This method has high annotation accuracy but low efficiency and is suitable for small-scale data sets or situations where high annotation accuracy is required. You can also combine manual annotation and automatic algorithms, first use automatic algorithms to perform preliminary element detection and classification on the image, and then manually correct and improve the results. This method can improve annotation efficiency while ensuring a certain level of annotation accuracy. Of course, you can also use existing pre-trained models or rule engines to automatically annotate images.

[0083] In one example, reference Figure 3 , Figure 3 This is a flowchart for model training of the user interface testing method of this application. First, prepare the dataset, during which enhancement operations such as scaling and adding noise are performed on the data to enrich the data features. Subsequently, convert the prepared dataset into COCO format to provide standardized data input for model training. Then enter the "training subject detection model" stage to train the model so that it learns the data features and patterns. After training is completed, the model performance is checked through the "verification subject detection model" to confirm whether it has achieved the expected results. Finally, the process reaches the "end" node, completing the entire process from data preparation, format conversion, model training to verification.

[0084] Step S03: vectorize the images in the initial training image set using the preset image processing model to construct a preset image classification vector library.

[0085] Furthermore, in order to use the training model to vectorize the image and label the vector to form an image classification vector library, when the user encounters an unrecognizable image during the use of the model, they only need to maintain the correspondence between the vector and the classification, thereby meeting the ease of use and the robustness of the model for image recognition. The step S03 may include:

[0086] A picture database is established based on the initial training image set; feature information of the pictures in the picture database is extracted using the preset image processing model to obtain a feature vector database; an index relationship is established between the feature vector database and the picture database, and a classification name of the index relationship is defined; the classification name and the index relationship are stored in the feature vector database to obtain a preset picture classification vector library.

[0087] It should be noted that the image database is a database formed by organizing and storing the initial training image set. The images are organized according to certain rules to facilitate subsequent image management and access. Feature information is the unique characteristics of interface elements (text, icons, etc.) in user interface testing. The feature vector database stores feature information extracted from the image database and converts it into vector form.

[0088] Additionally, it's important to note that the index relationship is the corresponding link between the feature vector database and the image database. This allows you to quickly find the corresponding original image using the feature vector, or find the corresponding feature vector using the image. The category name defines the index relationship and describes the category to which the feature vector and image belong. In user interface testing, category names might be "button," "text box," "icon," and so on.

[0089] Specifically, the initial training image set is first imported into a specific database management system, such as MySQL or MongoDB. A simple file system can also be used for storage, but the image file naming conventions must be clear for easy subsequent management. The images are then classified and stored according to certain rules. Next, a preset image processing model is used to process each image in the image database, extracting feature information. Based on its own algorithm and training results, the model converts the image features into a set of numerical values, known as feature vectors. The extracted feature vectors are then stored in another database, forming a feature vector database. An index is established between the feature vector database and the image database, using the correspondence between the image's unique identifier and the feature vector. For example, this can be achieved by using the database's indexing functionality or by writing code to implement a custom index structure. Furthermore, a category name is defined for each index relationship based on the image's actual content and purpose. Finally, the category name and index relationship are stored in the feature vector database, forming the final preset image classification vector library. At this point, each feature vector in the library is associated with the corresponding image, category name, and index information, facilitating subsequent retrieval and use. When identifying an object, we first extract its features, then use them to perform a vector search in the feature library. This allows us to return the image's category name, such as an icon or text box, through indexing. This enhancement method maintains high recognition accuracy and speed without requiring retraining, making it an effective complement to image classification models.

[0090] This embodiment discloses obtaining a preset training image set and enhancing images in the preset training image set to obtain an initial training image set; training a subject detection model based on the initial training image set to obtain a preset image processing model, wherein the subject detection model is a detection model that is a combination of an optical character recognition model and an image classification model; and vectorizing the images in the initial training image set using the preset image processing model to construct a preset image classification vector library. By training the combined subject detection model, multi-element joint recognition is achieved, avoiding the multiple calls of independent models in traditional solutions, significantly improving interface efficiency. Constructing a vector library also provides a foundation for subsequent zero-shot enhancement, reducing model iteration costs.

[0091] Reference Figure 4 , Figure 4 This is a flow chart of the third embodiment of the user interface testing method of the present application. Based on the above embodiments, the third embodiment of the user interface testing method of the present application is proposed.

[0092] In the third embodiment, step S20 includes:

[0093] Step S201 : calculating the similarity between the feature vector and the index vector in the preset image classification vector library.

[0094] It can be understood that calculating the similarity between the feature vector and the index vector in the preset image classification vector library can be done by measuring the cosine value of the angle between the two vectors to determine the similarity, or by calculating the straight-line distance between the two vectors in the vector space, or by calculating the sum of the absolute values ​​of the numerical differences in each dimension.

[0095] It should be understood that in practical applications, the method can be selected based on needs. For example, cosine similarity focuses on the similarity of vector directions and is suitable for determining the categorical tendency of image features. Euclidean distance and Manhattan distance measure the difference in spatial position or dimension. The smaller the distance, the closer the numerical characteristics of the feature vector and index vector are, thus providing a quantitative basis for image classification and retrieval.

[0096] Step S202 : determining the index vector with the greatest similarity, and returning the labeled category information of the feature vector through the index vector.

[0097] It is understandable that by using the mapping relationship between the index vector and the annotation category information in the preset image classification vector library to return the annotation category information, good recognition accuracy and recognition speed can still be achieved without retraining.

[0098] In one example, Appium and Airtest both use opencv for recognition. The input parameters are the search object image and UI screenshot, and the output result is the coordinate position of the found image. The implementation principle is opencv template matching, which has no obvious advantages. This solution uses model training. The input parameters are also the search object image and UI screenshot. The output result is the image category name and coordinate position. The implementation principle is vector search. Its advantage is that the first usage uses the model to vectorize the image to be searched, and uses this vector to search and output the image category in the vector library. The query is faster than the full-image search, and the output image category has certain semantics, which can be used for branch judgment when the use case is executed. The second usage takes the search object image and UI screenshot as input and can output the coordinates of this image in the entire screen.

[0099] In the third embodiment, step S30 includes:

[0100] Step S301: determining the operation logic of each of the interface elements based on the annotation category information.

[0101] It should be noted that operational logic refers to the specific operations and rules that need to be performed in user interface testing for different types of interface elements. For example, a "button" corresponds to the "click" operation, an "input box" corresponds to the "enter text" operation, and a "drop-down menu" corresponds to the "expand and select an option" operation.

[0102] It is understandable that after obtaining the label category information of the interface element, the corresponding basic operation logic is directly retrieved through the mapping table. For example, if an element is labeled as "Submit Button", the default operation logic is "Click the button and wait for the page to load."

[0103] Step S302: Generate a corresponding test script according to the operation logic.

[0104] It should be understood that the process of generating test scripts based on operational logic can be to convert the annotation categories of interface elements and their corresponding operational rules into automatically executable code. Through the automated mapping of "annotation categories", "operation logic" and "code generation", the process of manually writing test scripts is converted into model-driven intelligent generation, which significantly reduces the cost of script writing and improves testing efficiency. At the same time, scripts generated based on semantic annotation categories (such as "submit button" and "search input box") are more readable and maintainable, and support rapid updates of test logic during user interface iterations. This can reduce the maintenance work of test scripts, improve the generalization ability of scripts, and improve accuracy. It has reference significance for many fields such as quality assurance of large front-end UI.

[0105] Step S303: executing the processing script based on the coordinate information to obtain a test result of each interface element in the user interface.

[0106] It is understandable that executing the processing script based on the coordinate information and obtaining the test results can be to combine the position information of the interface elements with the preset operation logic, simulate user behavior and verify the interface response through automated means.

[0107] It should be understood that the operation simulation and real-time interaction can include the following scenarios. Single-step operation execution: perform operations in sequence according to the steps in the processing script, such as clicking a button, entering text, sliding the screen, etc., and add a waiting mechanism after each operation to ensure that the interface loading is complete; coordinate batch processing: perform batch operations on the coordinate information of multiple interface elements, such as traversing all "input box" elements and filling in test data in sequence. Visual feedback: Synchronously generate screenshots or screen recordings when performing operations, mark the element area corresponding to the operation coordinates, and facilitate manual review or error location.

[0108] In one example, first, the normalized coordinates output by the model (such as the coordinates of the four vertices of the bounding box) are converted into absolute pixel coordinates according to the actual resolution of the current device or screen. Next, the coordinates are subjected to noise reduction processing, such as deduplication and filtering of outliers, to ensure that the operation position accurately points to the center of the target element or the interactive area, such as the clickable area of ​​a button. Select the corresponding execution engine according to the test object type (such as mobile terminal, website page, desktop application, etc.). Execute the script and use the obtained image classification to perform logical judgment. For example, if there is a "close" tag, it triggers a click to close this pop-up window; execute the script and use the obtained coordinates to perform screen operations, such as clicks and inputs. Use the coordinates to reverse-check the element status after the operation, extract the newly generated image information of the interface, input the preset image processing model for re-detection, and verify whether the element changes are as expected.

[0109] Combined with the semantic operation logic of the labeled category information, differentiated verification can be performed for different element types (buttons, input boxes, etc.), significantly improving test coverage and accuracy. It is especially suitable for automated regression testing in scenarios with rapid UI iteration.

[0110] In this embodiment, the method of calculating the similarity between the feature vector and the index vector in the preset image classification vector library is disclosed; the index vector with the greatest similarity is determined, and the labeling category information of the feature vector is returned through the index vector. The operation logic of each of the interface elements is determined based on the labeling category information; a corresponding test script is generated according to the operation logic; the processing script is executed based on the coordinate information to obtain the test results of each of the interface elements in the user interface. The method based on similarity matching can quickly and accurately find the corresponding labeling category for the feature vector, improves the efficiency of determining the labeling category information, and avoids secondary training of the model. The operation logic of each interface element is determined based on the labeling category information, so that the test can be designed in a targeted manner according to the actual function and use of the element. The test process is simple, the recognition form is normalized, and it can adapt to different applications and devices.

[0111] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the user interface testing method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0112] This application also provides a user interface testing device, please refer to Figure 5 , the user interface testing device comprises:

[0113] A feature extraction module 10 is configured to input the user interface image information into a preset image processing model to obtain feature vectors and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract features of text elements and icon elements.

[0114] A category retrieval module 20 is configured to retrieve the feature vector based on a preset image classification vector library to determine the labeled category information corresponding to the feature vector;

[0115] The interface testing module 30 is used to test the user interface according to the annotation category information and the coordinate information.

[0116] The user interface testing device provided in this application, which utilizes the user interface testing method of the aforementioned embodiment, can resolve the technical problem that traditional UI testing requires manual selection of source and target images, resulting in poor recognition and unsatisfactory test results. Compared with the prior art, the beneficial effects of the user interface testing device provided in this application are the same as those of the user interface testing method provided in the aforementioned embodiment, and the other technical features of the user interface testing device are the same as those disclosed in the aforementioned embodiment method, and are not further elaborated here.

[0117] The present application provides a user interface testing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the user interface testing method in the above-mentioned embodiment one.

[0118] Reference below Figure 6 , which shows a schematic diagram of the structure of a user interface test device suitable for implementing the embodiments of the present application. The user interface test device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The user interface testing device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0119] like Figure 6 As shown, the user interface testing device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory 1002 or programs loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the user interface testing device. The processing device 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the user interface test device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a user interface test device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0120] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0121] The user interface testing device provided in this application, which utilizes the user interface testing method of the aforementioned embodiment, can resolve the technical problem that traditional UI testing requires manual selection of source and target images, resulting in poor recognition and unsatisfactory test results. Compared with the prior art, the beneficial effects of the user interface testing device provided in this application are the same as those of the user interface testing method provided in the aforementioned embodiment, and the other technical features of the user interface testing device are the same as those disclosed in the method of the previous embodiment, and are not further described here.

[0122] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0123] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0124] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the user interface testing method in the above embodiment.

[0125] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0126] The computer-readable storage medium may be included in the user interface testing device, or may exist independently without being assembled into the user interface testing device.

[0127] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the user interface testing device, the user interface testing device executes the user interface testing method described above.

[0128] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of code, and the module, program segment or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0130] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0131] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned user interface testing method. This computer-readable storage medium can solve the technical problem that traditional UI testing requires manual selection of source and target images, resulting in poor recognition and unsatisfactory test results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the user interface testing method provided in the above-mentioned embodiment, and will not be elaborated here.

[0132] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned user interface testing method when executed by a processor.

[0133] The computer program product provided in this application can solve the technical problem that traditional UI testing requires manual selection of source and target images, resulting in poor recognition and unsatisfactory test results. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the user interface testing method provided in the above embodiment, and will not be elaborated here.

[0134] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A user interface testing method, characterized in that: The user interface testing method comprises: Inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract the features of text elements and icon elements; Searching the feature vector based on a preset image classification vector library to determine the labeling category information corresponding to the feature vector; The user interface is tested according to the annotation category information and the coordinate information.

2. The user interface testing method according to claim 1, wherein: Before the step of inputting the image information of the user interface into a preset image processing model to obtain the feature vector and coordinate information of each interface element, the method further includes: Obtaining a preset training image set, and enhancing images in the preset training image set to obtain an initial training image set; Training a subject detection model based on the initial training image set to obtain a preset image processing model, wherein the subject detection model is a detection model obtained by combining an optical character recognition model and an image classification model; The images in the initial training image set are vectorized using the preset image processing model to construct a preset image classification vector library.

3. The user interface testing method according to claim 2, wherein: The step of training the subject detection model based on the initial training image set to obtain a preset image processing model includes: Labeling the images in the initial training image set to generate a label file containing category labels and location information; Converting the initial training image set into a preset annotation format based on the label file to obtain a target training image set; The subject detection model is trained according to the target training image set to obtain a preset image processing model.

4. The user interface testing method according to claim 2, wherein: The step of vectorizing the images in the initial training image set using the preset image processing model to construct a preset image classification vector library includes: Establishing a picture database based on the initial training image set; Extracting feature information of images in the image database using the preset image processing model to obtain a feature vector database; Establishing an index relationship between the feature vector database and the image database, and defining a classification name for the index relationship; The category name and the index relationship are stored in the feature vector database to obtain a preset picture category vector library.

5. The user interface testing method according to any one of claims 1 to 4, characterized in that: The step of searching the feature vector based on a preset image classification vector library to determine the labeling category information corresponding to the feature vector includes: Calculating the similarity between the feature vector and the index vector in the preset image classification vector library; The index vector with the greatest similarity is determined, and the labeled category information of the feature vector is returned through the index vector.

6. The user interface testing method according to any one of claims 1 to 4, characterized in that: The step of testing the user interface according to the annotation category information and the coordinate information includes: Determining the operation logic of each of the interface elements based on the annotation category information; Generate a corresponding test script according to the operation logic; The processing script is executed based on the coordinate information to obtain a test result of each interface element in the user interface.

7. A user interface testing device, characterized in that: The device comprises: A feature extraction module, configured to input the user interface image information into a preset image processing model to obtain the feature vector and coordinate information of each interface element. The preset image processing model is constructed by combining an optical character recognition model and an image classification model to simultaneously extract the features of text elements and icon elements; A category retrieval module is used to search the feature vector based on a preset image classification vector library to determine the labeled category information corresponding to the feature vector; An interface testing module is used to test the user interface according to the annotation category information and the coordinate information.

8. A user interface testing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the user interface testing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the user interface testing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the user interface testing method according to any one of claims 1 to 6 are implemented.

Images