Test method, device, computer device and readable storage medium of user interface

Abstract

This application relates to a method, apparatus, computer device, and readable storage medium for testing user interfaces. The method includes: receiving a test instruction for a page to be tested; obtaining historical description information of the elements to be tested from a preset page description model; obtaining the current page content of the page to be tested; locating a target element from the page elements based on the matching degree between the element description information of each page element in the current page content and the historical description information; the target element having the same element function as the element to be tested; performing a test operation on the target element; obtaining the actual page state of the page to be tested after performing the test operation; and generating a page test result for the page to be tested based on the actual page state and the expected page state. This method can improve the accuracy of user interface testing.

Description

Technical Field

[0001] This application relates to the field of software automation testing technology, and in particular to a user interface testing method, apparatus, computer device, and readable storage medium. Background Technology

[0002] As web applications become increasingly complex, automated testing has become a crucial means of ensuring software quality. By using automated testing tools to simulate user actions and verify system functionality, testing efficiency can be significantly improved while reducing manual labor costs.

[0003] In traditional technologies, automated testing of web user interfaces mainly relies on script recording and playback tools or rule-based testing frameworks. The testing system consists of a test script generation module, a test execution engine, and a result verification module. The test script generation module creates test scripts by recording user operations or manually writing test cases; the test execution engine is responsible for running these test scripts; and the result verification module checks whether the test results meet expectations using predefined judgment conditions.

[0004] However, these solutions are poorly adaptable to changes in the user interface and struggle to cope with complex and ever-changing test scenarios. They also fail to accurately test the user interface when dealing with dynamic content or unexpected interface states. Summary of the Invention

[0005] Therefore, it is necessary to provide a user interface testing method, apparatus, computer equipment, and readable storage medium that can improve the accuracy of user interface testing, addressing the aforementioned technical problems.

[0006] Firstly, this application provides a method for testing user interfaces, including:

[0007] Receive test instructions for the page to be tested; the test instructions are used to indicate the element to be tested, to indicate the test operation for the element to be tested, and to indicate the expected page state of the page to be tested after the test operation is performed.

[0008] The historical description information of the element to be tested is obtained from a preset page description model; the page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested.

[0009] The current page content of the page to be tested is obtained, and the target element is located from the page elements according to the matching degree between the element description information of each page element in the current page content and the historical description information; the target element has the same element function as the element to be tested.

[0010] The test operation is performed on the target element to obtain the actual page state of the page under test after the test operation is performed. Based on the actual page state and the expected page state, the page test result of the page under test is generated.

[0011] The technical solution of this application embodiment, upon receiving a test instruction for a page to be tested, can obtain the element to be tested indicated by the test instruction, the test operation for the element to be tested, and the expected page state after the test operation is executed. By constructing a page description model based on historical visual features and historical text semantic features, and using the historical description information of the element to be tested stored in the model to calculate the matching degree with the element description information of the current page element, the target element with the same element function as the element to be tested can be located. Even if the page structure, style class name, or element identifier of the page to be tested changes, the target element with the same function as the element to be tested can still be identified based on the stability of semantic features and visual features. Then, the test operation is executed on the target element, and the test result is generated by comparing the actual page state with the expected page state. This enables accurate testing of the user interface even when processing dynamic content or unexpected interface states, improving the flexibility of user interface testing and its adaptability to interface changes, and improving the accuracy and reliability of user interface testing.

[0012] In an optional embodiment of the first aspect, the historical description information includes the historical location path information, historical semantic features, and historical visual features of the element to be tested; the element description information includes the current location path information, current semantic features, and current visual features of the page element; locating the target element from each of the page elements based on the matching degree between the element description information of each page element in the current page content and the historical description information includes:

[0013] The page elements that match the current location path information with the historical location path information are selected from the page elements and used as candidate elements based on path matching.

[0014] Page elements whose current semantic features match the historical semantic features are selected from the page elements and used as candidate elements based on semantic matching;

[0015] Page elements whose current visual features match the historical visual features are selected from the page elements and used as candidate elements based on visual matching.

[0016] From the selected candidate elements, the target element with the highest matching degree with the element to be tested is determined.

[0017] The technical solution of this embodiment uses a multi-dimensional matching strategy to filter candidate elements from the path dimension, semantic dimension and visual dimension respectively, so as to comprehensively evaluate the matching degree between page elements and the elements to be tested, and improve the accuracy and robustness of target element positioning.

[0018] In an optional embodiment of the first aspect, determining the target element with the highest matching degree to the element to be tested from the selected candidate elements includes:

[0019] Obtain the feature vector corresponding to each candidate element; the feature vector is constructed based on at least two features among the matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability, and layout consistency corresponding to the candidate element;

[0020] The feature vectors of each candidate element are input into a pre-trained confidence evaluation model to obtain the confidence that each candidate element is the element to be tested.

[0021] If there are candidate elements with a confidence level greater than or equal to a preset threshold, the candidate element with the highest confidence level is determined as the target element.

[0022] The technical solution of this embodiment constructs a feature vector containing multi-dimensional features such as matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability and layout consistency, and uses a pre-trained confidence evaluation model to evaluate the confidence of candidate elements as test elements, thereby accurately locating the target element from the candidate elements and improving the accuracy of target element location.

[0023] In an optional embodiment of the first aspect, after inputting the feature vectors of each of the candidate elements into a pre-trained confidence evaluation model to obtain the confidence that each of the candidate elements is the element to be tested, the method further includes:

[0024] If the confidence level of each candidate element is less than the preset threshold, the semantic matching range corresponding to the historical semantic features and the visual matching range corresponding to the historical visual features are expanded.

[0025] Based on the expanded semantic matching range, page elements whose current semantic features match the historical semantic features are selected from each page element and used as updated candidate elements obtained based on semantic matching.

[0026] Based on the expanded visual matching range, page elements whose current visual features match the historical visual features are selected from each page element and used as updated candidate elements obtained based on visual matching.

[0027] The target element is determined from the selected updated candidate elements.

[0028] The technical solution of this embodiment, when the initial matching strategy fails to locate a reliable target element, increases the number and diversity of candidate elements by expanding the semantic role matching range and expanding the visual search area, thereby improving the success rate of target element location and enhancing the robustness and adaptability of the testing method.

[0029] In an alternative embodiment of the first aspect, performing the test operation on the target element includes:

[0030] Monitor the execution status of test operations targeting the target element;

[0031] If the execution status indicates that the test operation has failed, obtain the current page status of the page to be tested;

[0032] The historical successful operation path corresponding to the current page state is obtained from the preset operation path knowledge base and used as the target operation path for the target element; the operation path knowledge base stores the historical successful operation paths corresponding to each page state.

[0033] The test operation is performed on the target element according to the target operation path.

[0034] The technical solution of this embodiment, in the event of a test operation failure, retrieves a historical successful operation path from the operation path knowledge base and re-executes the test operation according to that operation path. This allows it to adapt to complex page states, improve the success rate of test operations, and enhance the reliability of the test method.

[0035] In an optional embodiment of the first aspect, performing the test operation on the target element according to the target operation path includes:

[0036] From the page elements of the current page content, identify the page elements that can respond to keyboard navigation operations and use them as navigation elements;

[0037] From the navigation elements, identify the navigation element that has the same function as the target element, and use it as the replacement element corresponding to the target element;

[0038] Based on the target operation path, generate a keyboard operation sequence to navigate to the alternative element;

[0039] The test operation is performed on the substitute element by executing the keyboard operation sequence.

[0040] The technical solution of this embodiment determines alternative elements that can respond to keyboard navigation operations and generates a keyboard operation sequence to navigate to the alternative elements. This enables test operations to be performed through keyboard navigation when the target element cannot be directly operated, thereby improving the flexibility and reliability of the test method.

[0041] In an optional embodiment of the first aspect, the page test result includes a page state test score, and generating the page test result of the page to be tested based on the actual page state and the expected page state includes:

[0042] Obtain the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state;

[0043] Based on the page type of the page to be tested, determine the weights corresponding to the text semantic matching degree, the interface layout matching degree, and the business rule matching degree, respectively.

[0044] Based on the weights mentioned above, the text semantic matching degree, the interface layout matching degree, and the business rule matching degree are weighted and summed to obtain the page status test score of the page to be tested.

[0045] The technical solution of this embodiment calculates the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state, and determines the weight corresponding to each matching degree according to the page type. The weighted sum is then used to obtain the page state test score, thereby comprehensively evaluating the test quality of the page under test and improving the accuracy and reliability of the test results.

[0046] Secondly, this application also provides a user interface testing apparatus, comprising:

[0047] A receiving module is used to receive test instructions for a page under test; the test instructions are used to indicate the element under test, to indicate the test operation for the element under test, and to indicate the expected page state of the page under test after the test operation is performed.

[0048] The acquisition module is used to acquire historical description information of the element to be tested from a preset page description model; the page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested.

[0049] The positioning module is used to obtain the current page content of the page to be tested, and locate the target element from each page element according to the matching degree between the element description information of each page element in the current page content and the historical description information; the target element has the same element function as the element to be tested.

[0050] The testing module is used to perform the test operation on the target element, obtain the actual page state of the page under test after the test operation is performed, and generate the page test result of the page under test based on the actual page state and the expected page state.

[0051] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0052] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the above aspects.

[0054] Regarding the beneficial effects of any of the technical solutions in the second to fifth aspects mentioned above, refer to the beneficial effects of the corresponding technical solutions in the first aspect; repeated examples will not be listed here. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of an optional application environment for a user interface testing method in one embodiment;

[0057] Figure 2 This is a schematic diagram of an optional process for testing a user interface in one embodiment;

[0058] Figure 3 This is an optional logic diagram of a user interface testing method in one embodiment;

[0059] Figure 4 This is a schematic diagram of an optional process for testing the user interface in another embodiment;

[0060] Figure 5 This is a schematic diagram of an optional structure of a user interface testing device in one embodiment;

[0061] Figure 6This is a schematic diagram of an optional internal structure of a computer device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0063] The terms "first," "second," etc., used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are used only to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0064] The user interface testing method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0065] In the specific implementation, terminal 102 receives test instructions for the page under test. The test instructions are used to indicate the element under test, the test operation on the element under test, and the expected page state after the test operation is performed. Terminal 102 obtains the historical description information of the element under test from a preset page description model. The page description model is pre-constructed based on the historical visual features and historical text semantic features of the page under test. Terminal 102 obtains the current page content of the page under test, and locates the target element from each page element based on the matching degree between the element description information and the historical description information of each page element in the current page content. The target element has the same element function as the element under test. Terminal 102 performs test operations on the target element, obtains the actual page state of the page under test after the test operation is performed, and generates the page test result of the page under test based on the actual page state and the expected page state.

[0066] For example, the preset page description model can be stored directly on the terminal 102 or on the server 104. If the page description model is stored on the server 104, the terminal 102 can call the page description model from the server 104 to test the user interface.

[0067] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc.

[0068] Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0069] In one exemplary embodiment, such as Figure 2 As shown, a user interface testing method is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes:

[0070] Step S201: Receive test instructions for the page to be tested.

[0071] The test instructions are used to instruct the element under test, to instruct the test operation on the element under test, and to instruct the expected page state after the test operation is performed.

[0072] The page to be tested can be any page in a web application, including but not limited to login pages, homepages, product detail pages, and order pages. The page to be tested consists of multiple page elements, including buttons, input boxes, dropdown lists, links, icons, and text labels.

[0073] The element to be tested refers to the page element on the page that needs to be tested. For example, on a login page, the element to be tested could be a username input field, a password input field, or a login button.

[0074] The test operation refers to the action that needs to be performed on the element under test, including click operations, input operations, drag operations, etc. For example, the test operation for the login button can be a click operation, and the test operation for the username input box can be an input operation.

[0075] The expected page state refers to the page state that the page under test should present after the test operation is performed. Page state includes the page's interface layout, the text content displayed on the page, the display status of each page element, and the page's Uniform Resource Locator (URL) address. For example, after clicking the login button on a login page, the expected page state could be that the page redirects to the user's homepage or displays a login success message.

[0076] In practice, the terminal can read test instructions from the test script, which defines the page to be tested, the element to be tested, the test operation, and the expected page state. The test script can be described in natural language or in a structured data format, such as JSON.

[0077] Step S202: Obtain the historical description information of the element to be tested from the preset page description model.

[0078] The page description model is pre-constructed based on the historical visual and semantic features of the page under test. The page description model can be a data model used to describe the feature information of each page element in the page under test.

[0079] In one embodiment, the page description model can be constructed in the form of a Semantic Page Model (SPM). The SPM establishes semantic description information for each page element, ensuring that page elements not only contain structured location information but also business semantic information and visual feature information. In the SPM, the description information for each page element includes the following attributes: element identifier, semantic role, business context, visual feature vector, text content, and Document Object Model (DOM) path.

[0080] The element identifier is an identifier used to uniquely identify page elements and can be represented as a string. For example, the element identifier for the login button can be represented as "login_btn".

[0081] Semantic roles refer to the semantic functional roles that page elements play on a page, including submission, input, navigation, and display operations. Semantic roles describe the functional nature of page elements and are independent of their specific implementation. For example, the semantic role of a login button can be represented as "submit_action".

[0082] In this context, business context refers to the business scenario or process associated with a page element, including user authentication, product browsing, order processing, and payment settlement. Business context describes the role of a page element within the business logic. For example, the business context of a login button can be represented as "user_authentication".

[0083] The visual feature vector can be a numerical vector representing the visual features of page elements. It is extracted from screenshots of page elements using a deep learning model, which can be a convolutional neural network model, a Transformer-based deep learning model with self-attention mechanisms, or a multimodal pre-trained model (Contrastive Language–Image Pre-training, CLIP), etc. The visual feature vector encodes visual information such as the color, shape, texture, and position of page elements.

[0084] Text content refers to the text information contained in page elements, including button text, label text, placeholder text, and tooltip text. For example, the text content of the login button is "Login".

[0085] The DOM path refers to the location path of a page element in the page's DOM tree. The location path can be represented using Extensible Markup Language (XPath) expressions or Cascading Style Sheets (CSS) selector expressions. For example, the DOM path of the login button can be represented as " / / button[@id='btn-login']".

[0086] For example, the description of the login button in a semantic page model can be represented as follows:

[0087] {

[0088] "element_id": "login_btn",

[0089] "semantic_role": "submit_action",

[0090] "business_context": "user_authentication",

[0091] "visual_features": [0.82, -0.31, 0.45, ..., 0.94], / / feature vector

[0092] "text_content": "Login"

[0093] "dom_path": " / / button[@id='btn-login']"

[0094] }

[0095] The historical description information of the element to be tested may include historical location path information, historical semantic features, and historical visual features.

[0096] Historical location path information includes the DOM path of page elements in the history of the page. Historical location path information can be represented using XPath expressions, CSS selector expressions, or other positioning expressions. For example, the historical location path information of the login button can be represented as " / / button[@id='btn-login']".

[0097] Historical semantic features include the textual semantic information of page elements, including the text content, semantic roles, and business context of page elements. Historical semantic features can be extracted using natural language processing techniques. For example, the historical semantic features of a login button include the text content "login", the semantic role "submit_action", and the business context "user_authentication".

[0098] Historical visual features refer to the visual information of page elements, including their position coordinates, size, color, shape, and icon information. These features can be extracted using computer vision techniques, including object detection and image segmentation. For example, the historical visual features of a login button include its position coordinates being at the center of the page, its size being 120 pixels wide and 40 pixels high, and its color being a blue background with white text.

[0099] For example, the construction of a page description model may include: the terminal obtaining the historical page content of the page to be tested, where the historical page content refers to a snapshot of the page to be tested at a historical moment; the terminal extracting the historical visual features and historical text semantic features of each page element from the historical page content; the terminal constructing historical description information for each page element based on the extracted historical visual features and historical text semantic features; and the terminal storing the historical description information of each page element in the page description model.

[0100] When a pre-built page description model is obtained, the terminal can retrieve the historical description information corresponding to the element under test from the page description model according to the element identifier, element name, element type and other attributes of the element under test indicated in the test instruction.

[0101] Step S203: Obtain the current page content of the page to be tested, and locate the target element from each page element based on the matching degree between the element description information and the historical description information of each page element in the current page content.

[0102] Here, "current page content" refers to the actual page content of the page under test at the current moment. Because the page under test may undergo changes such as interface updates, version iterations, or dynamic loading, the page elements in the current page content may differ from the historical page content recorded in the page description model.

[0103] Element description information refers to the feature information of each page element in the current page content, including the current location path information, current semantic features, and current visual features of the page elements. The definitions of current location path information, current semantic features, and current visual features are similar to those of historical location path information, historical semantic features, and historical visual features, and will not be repeated here. The difference is that current location path information, current semantic features, and current visual features are extracted from the current page content, while historical location path information, historical semantic features, and historical visual features are extracted from historical page content.

[0104] In this context, the target element and the element under test have the same element function. Due to page changes, the positioning path, semantic features, or visual features of the element under test in the current page may change. Therefore, it is necessary to locate the target element corresponding to the element under test through matching degree calculation.

[0105] In this context, element functionality refers to the functional role an element plays on a page. For example, the functionality of a login button is to submit a login form, and the functionality of a username input box is to receive username input. If the target element and the element under test have the same functionality, it means that the target element can perform the same operations and implement the same business logic as the element under test.

[0106] In practical applications, the terminal can obtain the current page content of the page under test through the browser engine. Specifically, it can send a Hypertext Transfer Protocol (HTTP) request to the server of the page under test and receive the HyperText Markup Language (HTML) document returned by the server. Then, it can parse the HTML document, construct a DOM tree structure, traverse the DOM tree, and extract the element description information of each page element in the current page content.

[0107] Then, the matching degree between the element description information of each page element in the current page content and the historical description information of the element to be tested is calculated. Optionally, the matching degree calculation can be based on a comprehensive evaluation of location path matching degree, semantic feature matching degree, and visual feature matching degree.

[0108] Location path matching degree refers to the degree of similarity between the current location path information and the historical location path information. Location path matching degree can be obtained by calculating the edit distance, longest common subsequence, or tree structure similarity of the location paths.

[0109] Semantic feature matching degree refers to the degree of similarity between current semantic features and historical semantic features. It can be obtained by calculating text similarity, using methods such as cosine similarity, Jaccard similarity, edit distance similarity, or deep learning-based semantic similarity. Furthermore, semantic feature matching degree can also include semantic role matching degree, which is determined by whether the semantic role of the current page element matches the historical semantic role of the element being tested.

[0110] Visual feature matching degree refers to the degree of similarity between current visual features and historical visual features. Visual feature matching degree can be obtained by calculating the cosine similarity, Euclidean distance, or Manhattan distance between visual feature vectors.

[0111] In practice, the terminal can perform a weighted summation of the location path matching degree, semantic feature matching degree, and visual feature matching degree to obtain the matching degree of each page element. Then, based on the matching degree of each page element, the terminal selects the page element with the highest matching degree as the target element.

[0112] Step S204: Perform test operations on the target element, obtain the actual page state of the page under test after the test operations are performed, and generate the page test results of the page under test based on the actual page state and the expected page state.

[0113] The actual page state refers to the actual state of the page under test after the test operation is performed. The actual page state includes the page's interface layout, the text content displayed on the page, the display status of each page element, and the page's URL address.

[0114] The page test result refers to the test evaluation result of the page under test, indicating whether the page passes the test. The page test result can be a binary result, including pass or fail; or it can be a score result, including a page status test score.

[0115] In practice, the terminal can perform test operations on the target element through an automated testing framework. Automated testing frameworks include Selenium and Playwright. The process of executing test operations includes: determining the type of test operation based on the test instructions, including click operations, input operations, etc.; and calling the corresponding interface of the automated testing framework to perform the corresponding test operation on the target element. For example, in the case of a click operation, the click interface of the automated testing framework can be called to simulate a user clicking the target element.

[0116] After executing the test operation, the terminal obtains the actual page state of the page under test. The process of obtaining the actual page state includes: first, waiting for the page to finish loading. The waiting time can be a fixed waiting time or a dynamic waiting time, which is determined based on the page loading status; after the page is loaded, the display status of each page element in the page's DOM tree structure, the page's URL address, the text content displayed on the page, etc., can be obtained to obtain the actual page state.

[0117] The terminal compares the actual page state with the expected page state to determine if they match. If the actual page state matches the expected page state, the terminal generates a page test result indicating that the test has passed. If the actual page state does not match the expected page state, the terminal generates a page test result indicating that the test has failed and records the difference information. The difference information includes the differences between the actual page state and the expected page state, such as inconsistent page URLs, inconsistent page text content, and inconsistent display states of page elements. The difference information can be used for subsequent test analysis and problem localization.

[0118] In the aforementioned user interface testing method, upon receiving a test instruction for the page under test, the method can obtain the element under test indicated by the test instruction, the test operation for the element under test, and the expected page state after the test operation is executed. By constructing a page description model based on historical visual features and historical text semantic features, and using the historical description information of the element under test stored in the model to calculate the matching degree with the element description information of the current page element, the method can locate the target element with the same element function as the element under test. Even if the page structure, style class name, or element identifier of the page under test changes, the method can still identify the target element with the same function as the element under test based on the stability of semantic and visual features. Then, the method executes the test operation on the target element, and generates the test result by obtaining the actual page state and comparing it with the expected page state. This allows for accurate testing of the user interface even when dealing with dynamic content or unexpected interface states, improving the flexibility and adaptability of user interface testing to interface changes, and enhancing the accuracy and reliability of user interface testing.

[0119] In another embodiment, the historical description information includes the historical location path information, historical semantic features, and historical visual features of the element to be tested, and the element description information includes the current location path information, current semantic features, and current visual features of the page element. Based on the matching degree between the element description information and the historical description information of each page element in the current page content, the target element is located from each page element, including: selecting page elements from each page element whose current location path information matches the historical location path information as candidate elements obtained based on path matching; selecting page elements from each page element whose current semantic features match the historical semantic features as candidate elements obtained based on semantic matching; selecting page elements from each page element whose current visual features match the historical visual features as candidate elements obtained based on visual matching; and determining the target element with the highest matching degree with the element to be tested from the selected candidate elements.

[0120] Candidate elements refer to page elements selected from all page elements in the current page content that may correspond to the element being tested. By employing a multi-dimensional matching strategy, selecting candidate elements from path, semantic, and visual dimensions, the accuracy of target element location can be improved.

[0121] In practice, the terminal uses three matching strategies—path matching, semantic matching, and visual matching—to select candidate elements from the page elements of the current page content.

[0122] Under the path-matching-based matching strategy, the path similarity between the current location path information of each page element and the historical location path information of the element to be tested is calculated. Path similarity can be obtained by calculating the edit distance of the location path; the smaller the edit distance, the higher the path similarity. The terminal can select page elements with a path similarity greater than a first preset threshold from all page elements as candidate elements obtained based on path matching.

[0123] Under a semantic matching-based matching strategy, the semantic similarity between the current semantic features of each page element and the historical semantic features of the element to be tested is calculated. Semantic similarity can be obtained by calculating the cosine similarity between text contents. The terminal can filter page elements whose semantic similarity is greater than a second preset threshold as candidate elements obtained based on semantic matching. Further optionally, semantic matching can also include semantic role matching, where the terminal determines whether the semantic role of each page element is consistent with the historical semantic role of the element to be tested. If the semantic role of a page element is consistent with the historical semantic role of the element to be tested, the terminal identifies that page element as a candidate element obtained based on semantic matching.

[0124] Under a visual matching-based matching strategy, the visual similarity between the current visual features of each page element and the historical visual features of the element to be tested is calculated. Visual similarity can be obtained through the cosine similarity between visual feature vectors. The terminal can select page elements with a visual similarity greater than a third preset threshold as candidate elements obtained based on visual matching. For example, screenshots of page elements can be input into the CLIP model, which outputs the visual feature vectors of the page elements; the cosine similarity between the current page element's visual feature vector and the historical visual feature vectors of the element to be tested is calculated as the visual feature matching degree.

[0125] Then, the candidate elements obtained based on path matching, semantic matching, and visual matching are summarized to obtain a candidate element set.

[0126] Optionally, the terminal can also select the top-1 matching page element from each matching strategy as the candidate element corresponding to that matching strategy. In other words, the terminal can select the page element from each page element that best matches the current location path information with the historical location path information as the candidate element obtained based on path matching; select the page element from each page element that best matches the current semantic features with the historical semantic features as the candidate element obtained based on semantic matching; and select the page element from each page element that best matches the current visual features with the historical visual features as the candidate element obtained based on visual matching.

[0127] The technical solution of this embodiment uses a multi-dimensional matching strategy to filter candidate elements from the path dimension, semantic dimension and visual dimension respectively, so as to comprehensively evaluate the matching degree between page elements and the elements to be tested, and improve the accuracy and robustness of target element positioning.

[0128] In an exemplary embodiment, determining the target element with the highest matching degree with the element to be tested from the selected candidate elements includes: obtaining the feature vector corresponding to each candidate element; the feature vector is constructed based on at least two features among the matching strategy type, semantic matching degree, text similarity, visual similarity, localization path stability, and layout consistency corresponding to the candidate element; inputting the feature vector of each candidate element into a pre-trained confidence evaluation model to obtain the confidence degree of each candidate element as the element to be tested; if there is a candidate element with a confidence degree greater than or equal to a preset threshold, the candidate element with the highest confidence degree is determined as the target element.

[0129] In this context, a feature vector is a vector data used to represent the features of a candidate element. A feature vector consists of multiple feature dimensions, each corresponding to a feature type.

[0130] Among them, the matching strategy type refers to the matching strategy used to filter candidate elements, including path matching, semantic matching and visual matching. The matching strategy type can also be called strategy type encoding. For example, the matching strategy type can be represented by numerical encoding, with 0, 1 and 2 representing different matching strategies respectively.

[0131] Semantic matching degree refers to the degree of matching between the current semantic features of a candidate element and the historical semantic features of the element to be tested, with a value ranging from 0 to 1. Optionally, semantic matching degree can be semantic role matching degree.

[0132] Text similarity refers to the similarity between the text content of a candidate element and the text content of the element to be tested, with a value ranging from 0 to 1. Text similarity can be obtained by calculating edit distance or cosine similarity.

[0133] Visual similarity refers to the similarity between the current visual features of a candidate element and the historical visual features of the element to be tested, with a value ranging from 0 to 1. Visual similarity can be obtained by calculating the cosine similarity between visual feature vectors. When candidate elements are selected through a visual matching strategy, the visual similarity is the actual calculated similarity value. When candidate elements are not selected through a visual matching strategy, the visual similarity can be set to 0 or a default value.

[0134] Location path stability refers to the stability of a candidate element's location path during historical page updates, with a value ranging from 0 to 1. Location path stability can be determined by analyzing historical versions of the page under test and statistically analyzing the frequency of location path changes. Lower frequency of location path changes indicates higher stability. Optionally, location path stability can be scored based on factors such as DOM path depth and the presence of an ID attribute. DOM path depth refers to the depth of the location path within the DOM tree; shallower depth indicates higher stability. The ID attribute refers to whether a page element has a unique ID attribute; page elements with ID attributes have higher location path stability.

[0135] Layout consistency refers to the degree of consistency between the relative position of a candidate element on the page and the relative position of the element under test on historical pages, with a value ranging from 0 to 1. Layout consistency can be obtained by comparing the position coordinates of the candidate element and the element under test on their respective pages, as well as their relative positional relationships with other page elements. Optionally, layout consistency can be obtained by calculating the position offset. Specifically, the position coordinates of the element under test on historical pages and the position coordinates of the candidate element on the current page can be obtained, and the position offset between the two can be calculated. The smaller the position offset, the higher the layout consistency.

[0136] For example, the feature vector of a candidate element can be represented as:

[0137] feature = [

[0138] strategy_type, # Strategy type encoding, path matching strategy=0, semantic matching strategy=1, visual matching strategy=2

[0139] semantic_match_score, # semantic role matching score, 0 or 1

[0140] text_similarity, # Normalized edit distance value between the current text and historical text, 0 to 1

[0141] visual_similarity, # CLIP image similarity, only valid for visual matching strategies, otherwise 0.

[0142] dom_stability_score, # Stability score for DOM path depth, ID existence, etc., ranging from 0 to 1

[0143] layout_consistency # The relative position offset of an element on the page, compared with the historical page description model, ranging from 0 to 1.

[0144] ].

[0145] The confidence assessment model is a machine learning model used to evaluate the confidence that a candidate element is the element to be tested. Optionally, the confidence assessment model can be a decision tree model, a random forest model, a support vector machine model, a neural network model, or other classification models.

[0146] The confidence level refers to the probability that a candidate element is the element to be tested, and its value ranges from 0 to 1. The higher the confidence level, the greater the likelihood that the candidate element corresponds to the element to be tested.

[0147] The preset threshold is used to determine whether a candidate element is the target element. It can be set according to the actual test scenario, such as 0.7, 0.75 or 0.8.

[0148] In practice, the terminal acquires the matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability, and layout consistency corresponding to the candidate elements; these features are then combined into feature vectors. The terminal inputs the feature vectors of each candidate element into a pre-trained confidence evaluation model, which analyzes the feature vectors and outputs the confidence score of each candidate element as the test element.

[0149] The terminal determines whether there are candidate elements with a confidence level greater than or equal to a preset threshold. If there are candidate elements with a confidence level greater than or equal to the preset threshold, the terminal identifies the candidate element with the highest confidence level as the target element. If there are no candidate elements with a confidence level greater than or equal to the preset threshold, the terminal determines that it cannot locate the target element and may adopt other positioning strategies or provide the user with positioning failure information.

[0150] As an optional implementation, the training process of the confidence assessment model is as follows: The terminal collects historical test data, which includes multiple test samples. Each test sample includes a feature vector of a candidate element and a label indicating whether the candidate element is the element to be tested. A label of 1 indicates that the candidate element is the element to be tested, and a label of 0 indicates that the candidate element is not the element to be tested. The terminal divides the historical test data into a training set and a validation set. The terminal uses the training set to train the confidence assessment model. The training process includes forward propagation, loss calculation, and backpropagation. The terminal uses the validation set to validate the trained confidence assessment model, adjusts the model parameters, and obtains the pre-trained confidence assessment model.

[0151] The technical solution of this embodiment constructs a feature vector containing multi-dimensional features such as matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability and layout consistency, and uses a pre-trained confidence evaluation model to evaluate the confidence of candidate elements as test elements, thereby accurately locating the target element from the candidate elements and improving the accuracy of target element location.

[0152] In an exemplary embodiment, after inputting the feature vectors of each candidate element into a pre-trained confidence evaluation model to obtain the confidence of each candidate element as the element to be tested, the method further includes: expanding the semantic matching range corresponding to the historical semantic features and expanding the visual matching range corresponding to the historical visual features when the confidence of each candidate element is less than a preset threshold; based on the expanded semantic matching range, selecting page elements from each page element whose current semantic features match the historical semantic features as updated candidate elements obtained based on semantic matching; based on the expanded visual matching range, selecting page elements from each page element whose current visual features match the historical visual features as updated candidate elements obtained based on visual matching; and determining the target element from the selected updated candidate elements.

[0153] The semantic matching range refers to the scope used to determine whether the current semantic features match historical semantic features. In the initial matching process, the semantic matching range is typically limited to strict semantic role consistency; that is, a match is only determined when the semantic role of a page element is completely consistent with the historical semantic role of the element being tested. Expanding the semantic matching range means relaxing the matching conditions for semantic roles, including page elements with related semantic roles or similar functions in the candidate range.

[0154] The visual matching range refers to the spatial area used to determine whether current visual features match historical visual features. In the initial matching process, the visual matching range is typically limited to a small area near the historical location of the element being tested. Expanding the visual matching range means broadening the spatial area of ​​the visual search to include page elements that are farther from the historical location of the element being tested in the candidate range.

[0155] Among them, the updated candidate elements refer to the candidate elements obtained by re-filtering after expanding the matching range.

[0156] In practice, if the confidence level of each candidate element is less than the preset threshold, the terminal determines that the current matching strategy has failed to locate a reliable target element and needs to expand the matching range to increase the number of candidate elements.

[0157] To expand the semantic matching range corresponding to historical semantic features, one can obtain the historical semantic role of the element to be tested, for example, the historical semantic role is "submit_action"; then, construct an extended set of semantic roles, which includes other semantic roles with similar functions or semantics to the historical semantic roles; based on the extended set of semantic roles, re-select semantically matching page elements from each page element.

[0158] As an optional implementation, the semantic role extension set can be constructed based on predefined semantic role associations. Semantic role associations define the degree of association between different semantic roles; for example, "submit_action" and "button_action" have a high degree of association, while "submit_action" and "navigation_action" have a medium degree of association. The terminal, based on the historical semantic roles of the element under test, searches for other semantic roles with an association degree greater than a preset association threshold from the semantic role associations and adds these semantic roles to the semantic role extension set.

[0159] As an alternative implementation, the semantic role extension set can also include superordinate semantic roles and subordinate semantic roles. Superordinate semantic roles refer to semantic roles with a broader functional scope, while subordinate semantic roles refer to semantic roles with a narrower functional scope. For example, the superordinate semantic role of "submit_action" could be "action," and the subordinate semantic role of "submit_action" could be "login_submit_action." The terminal adds the superordinate and subordinate semantic roles of the historical semantic roles of the element to be tested to the semantic role extension set.

[0160] To expand the visual matching range corresponding to historical visual features, one can obtain the position coordinates of the element to be tested in the historical pages, including horizontal and vertical coordinates; expand the visual search area with the historical position coordinates of the element to be tested as the center; and re-select visually matching page elements from each page element based on the expanded visual search area.

[0161] Optionally, the visual search area can be represented as a rectangular region centered on the historical position of the element to be tested, defined by a search radius. During the initial matching process, the search radius is set to a first radius value, such as 50 pixels. When expanding the visual matching range, the terminal increases the search radius to a second radius value, which is greater than the first radius value, such as 100 or 150 pixels. The terminal filters page elements whose position coordinates fall within the expanded rectangular region as updated candidate elements.

[0162] The terminal aggregates the updated candidate elements obtained based on semantic and visual matching to obtain an updated candidate element set. The number of candidate elements in the updated candidate element set is usually greater than the number of candidate elements in the initial candidate element set.

[0163] The terminal determines the target element from the selected updated candidate elements. The determination process is the same as that in the previous embodiment, including constructing feature vectors for each updated candidate element, inputting the feature vectors into the confidence evaluation model to obtain the confidence of each updated candidate element, and determining the updated candidate element with the highest confidence as the target element.

[0164] The technical solution of this embodiment, when the initial matching strategy fails to locate a reliable target element, increases the number and diversity of candidate elements by expanding the semantic role matching range and expanding the visual search area, thereby improving the success rate of target element location and enhancing the robustness and adaptability of the testing method.

[0165] In an exemplary embodiment, performing a test operation on a target element includes: monitoring the execution status of the test operation on the target element; if the execution status indicates that the test operation has failed, obtaining the current page status of the page to be tested; obtaining the historical successful operation path corresponding to the current page status from a preset operation path knowledge base, and using it as the target operation path for the target element; the operation path knowledge base stores the historical successful operation paths corresponding to each page status; and performing the test operation on the target element according to the target operation path.

[0166] The execution status refers to the execution status of the test operation, including successful execution and execution failure. Execution failure includes situations such as the target element failing to respond to the test operation, the test operation triggering an exception error, and page loading timeout.

[0167] The current page state refers to the page state when the test operation fails, including the page's DOM tree structure, the display state of each page element, the page's URL address, and the page's business context.

[0168] The operation path knowledge base is a database that stores historical successful operation paths corresponding to each page state. A historical successful operation path refers to the operation path in which a test operation was successfully executed for a specific page state during historical testing. The operation path includes the sequence of operation steps, the operation method, the target element, and the operation execution result.

[0169] In the specific implementation, the construction process of the operation path knowledge base is as follows: During the historical testing process, the terminal records the page state and operation path of each test operation; if the test operation is successfully executed, the terminal stores the page state and operation path in the operation path knowledge base; if there are multiple historically successful operation paths for the same page state, the terminal can select the optimal operation path for storage based on factors such as the success rate and execution time of the operation path.

[0170] Optionally, the operation path knowledge base can be stored using a graph database, such as Neo4j. In the graph database, nodes represent page states, and edges represent operations and their execution results. Node attributes include characteristic information about the page state, such as page context, page URL, and page element composition; edge attributes include detailed information about the operation, such as the operation method, the semantic role of the target element, and the operation execution result.

[0171] The target operation path refers to the historical successful operation path retrieved from the operation path knowledge base for the current page state.

[0172] In practice, during the execution of test operations on the target element, the terminal monitors the execution status of the test operations. This can include: calling the corresponding interface of the automated testing framework to execute test operations on the target element; and listening to the execution results of the test operations to determine whether the test operations were successfully executed. The criteria for determining whether a test operation was successfully executed can include: whether the target element produced the expected response, such as whether a click operation triggered a click event, or whether an input operation successfully entered text; whether the test operation triggered any exceptions or errors, such as an element being unclickable, an element being invisible, or an element not existing; and whether the page loaded within a preset time.

[0173] If the execution status indicates that the test operation was successfully executed, the terminal continues to execute subsequent test steps, including obtaining the actual page status and generating page test results.

[0174] When the execution status indicates that the test operation has failed, the terminal obtains the current page state of the page under test, which may include: obtaining the DOM tree structure of the page and analyzing the display state of each page element; obtaining the URL address of the page and determining whether the page has redirected; extracting the feature information of the page state, including page context identifier, page element composition, page display state, etc.; and obtaining the semantic role and business context information of the target element from the page description model.

[0175] Based on the current page state, the terminal retrieves the historical successful operation path corresponding to the current page state from the operation path knowledge base. This may include: matching the current page state with each page state stored in the operation path knowledge base, where the matching method can be exact matching or fuzzy matching; selecting the page state with the highest matching degree, obtaining the historical successful operation path corresponding to that page state, and using it as the target operation path.

[0176] For example, when the operation path knowledge base is stored in a graph database, the process of retrieving the historical successful operation path corresponding to the current page state from the operation path knowledge base includes: The terminal constructs a graph database query statement. The query statement is used to retrieve the page state node that matches the current page state, as well as the operation edge emanating from that page state node. The query statement can be written using the Cypher query language, which is the query language for the Neo4j graph database. For example, for the scenario where clicking the login button on the login page fails, the query statement can be represented as:

[0177] "MATCH (p:Page {context: 'login_page'})-[r:ACTION]->(next) WHEREr.action_type = 'click' AND r.target_role = 'submit_action'".

[0178] The meaning of this query is: find the page state node p with the page context identifier "login_page", find the operation edge originating from node p with the operation type of click and the target element role of "submit_action", and the next page state node pointed to by the operation edge; sort the operation edges in descending order of priority and success rate, and return the operation sequence with the highest priority and the highest success rate to generate the target operation path. For example, the query result returns the operation sequence with the highest priority and the highest success rate in the same type of page (with the same business context) in history: ["Tab", "Tab", "Enter"], which means pressing the Tab key twice to navigate to the login button, and then pressing the Enter key to perform the click operation. This operation sequence ["Tab", "Tab", "Enter"] is used as the target operation path.

[0179] The terminal performs test operations on the target element according to the target operation path. The execution process includes: the terminal parsing the target operation path and obtaining each operation step in the operation path; the terminal executing each operation step sequentially until the test operation is completed.

[0180] The technical solution of this embodiment, in the event of a test operation failure, retrieves a historical successful operation path from the operation path knowledge base and re-executes the test operation according to that operation path. This allows it to adapt to complex page states, improve the success rate of test operations, and enhance the reliability of the test method.

[0181] In an exemplary embodiment, performing a test operation on a target element according to a target operation path includes: determining, from the page elements of the current page content, page elements that can respond to keyboard navigation operations as navigation elements; determining, from the navigation elements, navigation elements that have the same element function as the target element as replacement elements corresponding to the target element; generating a keyboard operation sequence to navigate to the replacement element according to the target operation path; and performing a test operation on the replacement element by executing the keyboard operation sequence.

[0182] Keyboard navigation refers to page navigation operations performed using keyboard keys, including Tab key navigation, arrow key navigation, Enter key confirmation, and Space key selection.

[0183] Navigation elements are page elements within the current page content that respond to keyboard navigation. Navigation elements typically have a `tabindex` property (a property in computer programming used to define the tab navigation order of form controls) or are focusable HTML elements. Focusable HTML elements include button elements, input box elements, selection box elements, and link elements, which can be navigated using the Tab key and interacted with using the Enter or Space key.

[0184] In this context, a substitute element refers to a navigation element that has the same functionality as the target element. Substitute elements can be accessed and manipulated via keyboard navigation, thus achieving the same testing effect as the target element.

[0185] The keyboard operation sequence refers to a series of keyboard key operations required to navigate to the replacement element. The keyboard operation sequence is generated based on the target operation path. Since the target operation path includes a series of operation methods, such as click, input, selection, and submit, different operation methods correspond to different keyboard keys. For example, a click operation can correspond to the Enter key or Space key, an input operation can correspond to text character input, a selection operation can correspond to a combination of the arrow keys and the Enter key, and a submit operation can correspond to the Enter key.

[0186] In practice, the terminal determines the page elements that can respond to keyboard navigation operations from the page elements of the current page content. This may include: traversing the DOM tree of the page and checking the attributes of each page element; determining whether the page element has a tabindex attribute or is a focusable HTML element; and if the page element meets one of the above conditions, determining the page element as a navigation element.

[0187] The terminal identifies navigation elements that have the same function as the target element from among the navigation elements. This may include: obtaining the function of the target element, which can be determined by analyzing information such as the type, text content, and event handler of the target element; traversing each navigation element and determining whether the function of the navigation element is the same as that of the target element; and if the function of the navigation element is the same as that of the target element, identifying the navigation element as the substitute element.

[0188] Based on the target operation path, the terminal generates a keyboard operation sequence to navigate to the alternative element, which may include:

[0189] First, obtain the position of the replacement element within the navigation element sequence, which is a sequence arranged according to the Tab key navigation order of navigation elements on the page. Each navigation element in the sequence has a position index. For example, the first navigation element accessible via the Tab key on the page has a position index of 1, the second navigation element has a position index of 2, and so on.

[0190] Secondly, obtain the current focus position. The current focus position refers to the index of the navigation element that is currently focused within the navigation element sequence. For example, if the page has just finished loading and no operation has been performed, the current focus position can be the position of the first navigation element on the page; or, if other test operations have been performed on the page, the current focus position is the position where the page focus remained after the last test operation was completed; if the test operation fails, the current focus position can be the position where the page focus remained when the test operation failed.

[0191] Next, calculate the number of Tab key presses required to navigate from the current focus position to the replacement element. The formula is: Number of Tab key presses = Index of the replacement element - Index of the current focus position.

[0192] Then, based on the operation methods contained in the target operation path, determine the interactive key operations to be performed upon reaching the replacement element. The operation methods contained in the target operation path define what operation to perform on the replacement element; different operation methods correspond to different keyboard keys.

[0193] Furthermore, navigation key operations and interactive key operations are combined to generate a complete keyboard operation sequence. The keyboard operation sequence consists of two parts: the first part is the navigation part, which consists of several Tab key presses, used to navigate from the current focus position to the replacement element; the second part is the interactive part, which consists of specific interactive key presses, used to perform test operations on the replacement element.

[0194] The terminal performs test operations on the substitute element by executing a keyboard operation sequence. This can be done by calling the keyboard operation interface of the automated testing framework and executing each key operation in the keyboard operation sequence in sequence. At the same time, it listens for the response of the substitute element to determine whether the test operation was successfully executed.

[0195] The technical solution of this embodiment determines alternative elements that can respond to keyboard navigation operations and generates a keyboard operation sequence to navigate to the alternative elements. This enables test operations to be performed through keyboard navigation when the target element cannot be directly operated, thereby improving the flexibility and reliability of the test method.

[0196] In an exemplary embodiment, performing a test operation on the target element according to the target operation path may further include:

[0197] The terminal scans all focusable elements on the current page. Focusable elements are page elements that can be accessed via the Tab key, including elements with a tabindex attribute and a tabindex value greater than or equal to 0, as well as focusable HTML elements such as input elements, button elements, a elements, select elements, and textarea elements.

[0198] The terminal traverses the DOM tree of the page, checking the tabindex attribute value and element type of each page element, and collects the elements that meet the criteria into a list of focusable elements. The list of focusable elements is arranged according to the tab navigation order of the elements on the page, which is determined by the tabindex attribute value and the position of the element in the DOM tree.

[0199] The terminal obtains the semantic features of the target element from the page description model, including the semantic role and text content. For example, if the target element is a login button, the semantic role is "submit_action" and the text content is "login".

[0200] The terminal searches the list of focusable elements for the element whose semantic role is closest to the target element and whose text content is similar, and uses this element as a replacement. The search process includes: the terminal traverses the list of focusable elements, and for each focusable element, determines whether its semantic role is consistent with or similar to that of the target element; the terminal calculates the text similarity between the text content of the focusable element and the text content of the target element; the terminal selects the focusable element with the highest semantic role and text similarity as the replacement element.

[0201] The terminal obtains the focus position of the current page, i.e., the current focus position. The current focus position refers to the index position of the currently focused page element in the list of focusable elements. The terminal queries the active element of the page through the interface of the automated testing framework and determines the current focus position based on the position of the active element in the list of focusable elements.

[0202] The terminal calculates the number of Tab key presses required to navigate from the current focus position to the replacement element. The formula is: Number of Tab key presses = Index position of the replacement element - Current focus position. A positive result indicates forward navigation, requiring the use of the Tab key; a negative result indicates reverse navigation, requiring the use of the Shift+Tab key combination. The number of presses is the absolute value of the calculated result. For example, assuming the current focus position is the first element in the list of focusable elements, and the replacement element is in the third position, then the number of Tab key presses = 3 - 1 = 2, meaning 2 Tab key presses are required.

[0203] The terminal determines the interaction key to be executed upon reaching the replacement element based on the target operation type. In the case of a click operation, the interaction key is the Enter key.

[0204] The terminal combines historical successful operation paths with DOM analysis results to generate alternative operation sequences. The generation logic includes: if a historical successful operation path is retrieved from the knowledge base, the terminal first verifies whether the historical operation path is applicable to the current page state. The verification process includes: parsing the operation steps in the historical operation sequence and determining whether the operation steps can be executed on the current page. For example, the operation sequence ["Tab", "Tab", "Enter"] requires pressing the Tab key twice from the current focus position to reach the target element.

[0205] The terminal analyzes the DOM to calculate the actual number of tab key presses from the current focus position to the replacement element, and compares this actual number with the number of tab key presses in the historical operation sequence. If they match, it means the historical operation path is perfectly suitable for the current page, and the terminal directly uses the historical operation sequence as the replacement operation sequence. If they don't match, the terminal adjusts the historical operation sequence based on the actual number of tab key presses to generate a replacement operation sequence adapted to the current page. Finally, test operations can be performed on the target element according to the replacement operation sequence.

[0206] In practical applications, before executing a test operation, it is necessary to wait for the target element to become interactive. The time it takes for the target element to become interactive can be called the element appearance time. In an exemplary embodiment, a dynamic waiting mechanism can be used to predict the element appearance time. By setting an appropriate element appearance time, the inefficiency or timeout failure caused by a fixed waiting time can be avoided.

[0207] The element appearance time refers to the time interval from when the page navigation is completed to when the target element becomes interactive. The page navigation is completed by triggering the page's load event. The target element is interactive when the element simultaneously meets the two conditions of being visible and operable, that is, the element's visible property is true and its enabled property is true.

[0208] The terminal uses an LSTM neural network model to predict the appearance time of elements. The input features of the LSTM model include: page URL hash value, which converts the page URL address into a fixed-length numerical value through a hash function as a page identification feature; network type, which indicates the network connection type of the current device, such as 4G, 5G, WiFi, etc., and different network types affect page loading speed; device type, which indicates the type of the current device, such as desktop computer, laptop, tablet, smartphone, etc., and the performance of different device types affects page rendering speed; and historical loading time sequence, which indicates the actual time taken for the page to go from navigation completion to element interactivity in the last 5 tests.

[0209] The structure of the LSTM model is as follows: a single-layer LSTM network with an input dimension of 4 (corresponding to the 4 features mentioned above), 32 hidden units, and the output is the predicted waiting time in seconds.

[0210] The training process of the LSTM model is an online incremental update. After each test execution, the terminal records the input features and the actual wait time, forming training samples (features, actual_wait_time). The terminal uses these training samples to incrementally train the LSTM model and update the model parameters. Through continuous online learning, the LSTM model can adapt to changes in page load performance and provide more accurate predictions.

[0211] In practical applications, before executing test operations, the terminal inputs the current page URL hash value, network type, device type, and historical loading time sequence into the LSTM model. The LSTM model outputs the predicted waiting time, for example, 1.8 seconds. The terminal calls the waiting interface of the automated testing framework to set the appearance time of the elements output by the LSTM.

[0212] In an exemplary embodiment, the page test result includes a page state test score. Based on the actual page state and the expected page state, the page test result of the page to be tested is generated, including: obtaining the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state; determining the weights corresponding to the text semantic matching degree, interface layout matching degree, and business rule matching degree according to the page type of the page to be tested; and performing a weighted summation of the text semantic matching degree, interface layout matching degree, and business rule matching degree according to each weight to obtain the page state test score of the page to be tested.

[0213] The page status test score is a quantitative evaluation of the test quality of the page under test, and its value ranges from 0 to 100. The higher the page status test score, the higher the consistency between the actual page status and the expected page status, and the better the test quality.

[0214] Text semantic matching degree refers to the degree of matching between the actual page state and the expected page state at the text semantic level. It is obtained by comparing the text content in the actual page state and the expected page state, using methods such as text similarity calculation, keyword matching, and semantic vector similarity calculation.

[0215] Interface layout matching degree refers to the degree of matching between the actual page state and the expected page state at the interface layout level. It is obtained by comparing layout information such as the position, size, and arrangement of page elements in the actual page state and the expected page state.

[0216] Among them, business rule matching degree refers to the degree of matching between the actual page state and the expected page state at the business rule level. Business rule matching degree is obtained by verifying whether the actual page state meets the predefined business rules, which include page jump rules, data validation rules, state transition rules, etc.

[0217] Page type refers to the category of the page being tested, including login pages, homepages, product detail pages, order pages, and user center pages. Different page types place varying degrees of emphasis on text semantics, interface layout, and business rules; therefore, it is necessary to determine the appropriate weights based on the page type.

[0218] In specific implementation, the terminal obtains the text semantic matching degree between the actual page state and the expected page state. This can be achieved by extracting the text content in the actual page state, including the page title, page text, button text, and prompt information; extracting the text content in the expected page state; calculating the similarity between the text content in the actual page state and the text content in the expected page state, using methods such as cosine similarity, Jaccard similarity, or semantic similarity based on deep learning; and using the calculated similarity as the text semantic matching degree.

[0219] The terminal obtains the interface layout matching degree between the actual page state and the expected page state. This can be achieved by extracting the layout information of each page element in the actual page state, including element position coordinates, element size, element spacing, and element arrangement; extracting the layout information of each page element in the expected page state; comparing the layout information of the actual page state with the layout information of the expected page state and calculating the layout difference; and calculating the interface layout matching degree based on the layout difference. Optionally, the layout difference may include the difference in position coordinates between page elements.

[0220] The terminal obtains the business rule matching degree between the actual page state and the expected page state. This can be done by obtaining the set of business rules corresponding to the page to be tested, which contains multiple business rules; verifying whether the actual page state satisfies each business rule in the set of business rules; counting the number of business rules satisfied by the actual page state; and calculating the business rule matching degree.

[0221] The terminal can obtain a preset weight configuration table, which stores the weights corresponding to each page type; based on the page type of the page to be tested, it determines the weights corresponding to the text semantic matching degree, interface layout matching degree, and business rule matching degree.

[0222] For example, the weight configuration table may include the following page types and their corresponding weight configurations. For instance, page types may include marketing pages, transaction pages, and form pages.

[0223] Marketing pages are used to display marketing content and attract user attention, such as event promotion pages, product introduction pages, and advertising landing pages. For example, the weighting configuration for marketing pages is as follows: text semantic matching score weighting is 0.2, interface layout matching score weighting is 0.6, and business rule matching score weighting is 0.2. Marketing pages have high requirements for visual presentation; the accuracy of the interface layout directly affects user experience and conversion rates, therefore, interface layout matching score has the highest weighting. Text semantics and business rules are relatively less important, but basic content accuracy and functional correctness still need to be ensured.

[0224] The transaction page is used to complete the transaction process, such as the shopping cart page, checkout page, and payment page. For example, the weighting of the transaction page is configured as follows: text semantic matching score weighting is 0.2, interface layout matching score weighting is 0.3, and business rule matching score weighting is 0.5. The transaction page has high requirements for business rule verification, ensuring correct price calculation, complete payment process, and accurate order status, therefore, business rule matching score has the primary weighting. While interface layout and text semantics are also important, their priority is lower than that of business rules.

[0225] Form pages, such as registration pages, information entry pages, and questionnaire pages, contain numerous input fields. For example, the weighting of form pages is configured as follows: text semantic matching score 0.4, interface layout matching score 0.3, and business rule matching score 0.3. Form pages require high accuracy in text content; field labels, prompts, error messages, and other textual information must be clear and accurate. Therefore, text semantic matching score carries significant weight. Interface layout and business rules also need to be considered, but in a relatively balanced manner.

[0226] The terminal performs a weighted sum of the text semantic matching degree, interface layout matching degree, and business rule matching degree according to their respective weights to obtain the page status test score of the page under test. The page status test score is expressed as follows:

[0227] Page status test score = (text semantic matching degree × text semantic weight + interface layout matching degree × interface layout weight + business rule matching degree × business rule weight) × 100.

[0228] If the page status test score is higher than the test threshold, the page test result is determined to be a passed page test; if the page status test score is less than or equal to the test threshold, the page test result is determined to be a failed page test.

[0229] The testing threshold can be dynamically adjusted based on the page type being tested. For example, for marketing pages, the testing threshold can be automatically lowered to 0.6 because marketing pages typically contain a large amount of marketing copy, promotional information, and advertising images, which frequently change during A / B testing—for example, by changing text, adjusting images, or modifying button text. Marketing pages have relatively lenient requirements for the accuracy of element positioning, allowing for a certain degree of content variation, thus the testing threshold can be lowered. On the other hand, for financial transaction pages, the testing threshold can be automatically raised to 0.95 because financial transaction pages involve critical and sensitive information such as fund transactions, account information, and transaction amounts, requiring extremely high accuracy. Any error could lead to serious financial losses or compliance risks, therefore the testing threshold can be increased.

[0230] In this embodiment, the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state are calculated, and the weights corresponding to each matching degree are determined according to the page type. The page state test score is obtained by weighted summation, thereby comprehensively evaluating the test quality of the page under test and improving the accuracy and reliability of the test results.

[0231] For the convenience of those skilled in the art, Figure 3 An exemplary logic diagram of a user interface testing method is provided, which includes the following steps:

[0232] Step 1: Construct a semantic page model. Obtain the Hypertext Markup Language code, Cascading Style Sheets, and page screenshots from historical versions of the page to be tested as input data. Extract the corresponding historical text semantic features and historical visual features to train and output the constructed semantic page model as the page description model.

[0233] Step Two: Intelligent Localization. Obtain the document object model data of the current page and receive the intent (i.e., test instruction) for the page to be tested. Combining the page description model output in Step One, calculate the feature matching degree to accurately locate and output the operable node (i.e., the target element in the aforementioned embodiment) corresponding to the test intent within the current page.

[0234] Step 3: Self-Healing Execution. Perform test operations on the operable nodes identified above according to the test instructions. During this process, the execution status is monitored in real-time using page status data; when a normal operation is blocked, the server triggers a self-healing repair mechanism (e.g., matching the target operation path using the aforementioned operation path knowledge base), thereby bypassing the obstacle to complete the operation and ultimately outputting the operation result or repair record.

[0235] Step 4: Contextual Verification. Obtain the actual state of the page under test after the operation is performed, and compare it with the expected state in the input command. By calculating the matching degree in dimensions such as text semantics and interface layout, the final output includes a verification result containing a test score, or outputs a heatmap to visually demonstrate the differences between the pages.

[0236] Step 5: Continuous Learning. Collect all test data generated during the above testing process (including feature data of successful localizations, path data of failed repairs, etc.) and feed it back into the system as training samples. Use machine learning algorithms to iteratively train the semantic page model using this full dataset, outputting an optimized model. The optimized model will then be reapplied to Step 1, forming a complete technical loop that continuously improves the localization accuracy and self-healing capabilities of the automated testing system over time.

[0237] In another embodiment, such as Figure 4 As shown, a user interface testing method is provided, which is applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0238] Step S401: Receive test instructions for the page to be tested.

[0239] The test instructions are used to instruct the element under test, to instruct the test operation on the element under test, and to instruct the expected page state after the test operation is performed.

[0240] Step S402: Obtain the historical description information of the element to be tested from the preset page description model.

[0241] The page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested.

[0242] Step S403: Obtain the current page content of the page to be tested; select page elements from each page element whose current location path information matches historical location path information as candidate elements based on path matching; select page elements from each page element whose current semantic features match historical semantic features as candidate elements based on semantic matching; select page elements from each page element whose current visual features match historical visual features as candidate elements based on visual matching; and determine the target element with the highest matching degree with the element to be tested from the selected candidate elements.

[0243] The target element and the element to be tested have the same element function.

[0244] In one embodiment, determining the target element with the highest matching degree with the element to be tested from the selected candidate elements includes: obtaining the feature vector corresponding to each candidate element; the feature vector is constructed based on at least two features among the matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability, and layout consistency corresponding to the candidate element; inputting the feature vector of each candidate element into a pre-trained confidence evaluation model to obtain the confidence degree of each candidate element as the element to be tested; if there is a candidate element with a confidence degree greater than or equal to a preset threshold, the candidate element with the highest confidence degree is determined as the target element.

[0245] In one embodiment, after inputting the feature vectors of each candidate element into a pre-trained confidence evaluation model to obtain the confidence of each candidate element as the element to be tested, the method further includes: if the confidence of each candidate element is less than a preset threshold, expanding the semantic matching range corresponding to the historical semantic features and expanding the visual matching range corresponding to the historical visual features; based on the expanded semantic matching range, selecting page elements from each page element whose current semantic features match historical semantic features as updated candidate elements obtained based on semantic matching; based on the expanded visual matching range, selecting page elements from each page element whose current visual features match historical visual features as updated candidate elements obtained based on visual matching; and determining the target element from the selected updated candidate elements.

[0246] Step S404: Monitor the execution status of the test operation against the target element; if the execution status indicates that the test operation has failed, obtain the current page status of the page to be tested; obtain the historical successful operation path corresponding to the current page status from the preset operation path knowledge base, and use it as the target operation path for the target element.

[0247] The operation path knowledge base stores the historical successful operation paths corresponding to each page status.

[0248] Step S405: From the page elements of the current page content, determine the page elements that can respond to keyboard navigation operations and use them as navigation elements; from the navigation elements, determine the navigation elements that have the same element function as the target element and use them as the replacement elements corresponding to the target element; generate a keyboard operation sequence to navigate to the replacement element according to the target operation path; and perform a test operation on the replacement element by executing the keyboard operation sequence.

[0249] Step S406: Obtain the actual page state of the page to be tested after the test operation is performed, and obtain the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state.

[0250] Step S407: Based on the page type of the page to be tested, determine the weights corresponding to the text semantic matching degree, interface layout matching degree, and business rule matching degree; based on each weight, perform a weighted summation of the text semantic matching degree, interface layout matching degree, and business rule matching degree to obtain the page status test score of the page to be tested.

[0251] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a user interface testing method described above.

[0252] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0253] Based on the same inventive concept, this application also provides a user interface testing apparatus for implementing the user interface testing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more user interface testing apparatus embodiments provided below can be found in the limitations of the user interface testing method described above, and will not be repeated here.

[0254] In one exemplary embodiment, such as Figure 5As shown, a user interface testing apparatus is provided, comprising:

[0255] The receiving module 510 is used to receive test instructions for the page to be tested; the test instructions are used to indicate the element to be tested, to indicate the test operation for the element to be tested, and to indicate the expected page state of the page to be tested after the test operation is performed.

[0256] The acquisition module 520 is used to acquire historical description information of the element to be tested from a preset page description model; the page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested.

[0257] The positioning module 530 is used to obtain the current page content of the page to be tested, and locate the target element from each page element according to the matching degree between the element description information of each page element in the current page content and the historical description information; the target element has the same element function as the element to be tested.

[0258] The testing module 540 is used to perform the test operation on the target element, obtain the actual page state of the page under test after the test operation is performed, and generate the page test result of the page under test based on the actual page state and the expected page state.

[0259] The modules in the aforementioned user interface testing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0260] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a user interface testing method.

[0261] Those skilled in the art will understand that Figure 6 The structure shown is a block diagram of a partial structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0262] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0263] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0264] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0265] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0266] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program mentioned can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0267] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0268] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method of testing a user interface, characterized by, The method includes: Receive test instructions for the page to be tested; the test instructions are used to indicate the element to be tested, to indicate the test operation for the element to be tested, and to indicate the expected page state of the page to be tested after the test operation is performed. The historical description information of the element to be tested is obtained from a preset page description model; the page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested. The current page content of the page to be tested is obtained, and the target element is located from the page elements according to the matching degree between the element description information of each page element in the current page content and the historical description information; the target element has the same element function as the element to be tested. The test operation is performed on the target element to obtain the actual page state of the page under test after the test operation is performed. Based on the actual page state and the expected page state, the page test result of the page under test is generated.

2. The method of claim 1, wherein, The historical description information includes the historical location path information, historical semantic features, and historical visual features of the element to be tested; the element description information includes the current location path information, current semantic features, and current visual features of the page element; locating the target element from each of the page elements based on the matching degree between the element description information of each page element in the current page content and the historical description information includes: The page elements that match the current location path information with the historical location path information are selected from the page elements and used as candidate elements based on path matching. Page elements whose current semantic features match the historical semantic features are selected from the page elements and used as candidate elements based on semantic matching; Page elements whose current visual features match the historical visual features are selected from the page elements and used as candidate elements based on visual matching. From the selected candidate elements, the target element with the highest matching degree with the element to be tested is determined.

3. The method of claim 2, wherein, The step of determining the target element with the highest matching degree with the element to be tested from the selected candidate elements includes: Obtain the feature vector corresponding to each candidate element; the feature vector is constructed based on at least two features among the matching strategy type, semantic matching degree, text similarity, visual similarity, positioning path stability, and layout consistency corresponding to the candidate element; The feature vectors of each candidate element are input into a pre-trained confidence evaluation model to obtain the confidence that each candidate element is the element to be tested. If there are candidate elements with a confidence level greater than or equal to a preset threshold, the candidate element with the highest confidence level is determined as the target element.

4. The method according to claim 3, characterized in that, After inputting the feature vectors of each candidate element into a pre-trained confidence evaluation model to obtain the confidence score of each candidate element as the element to be tested, the method further includes: If the confidence level of each candidate element is less than the preset threshold, the semantic matching range corresponding to the historical semantic features and the visual matching range corresponding to the historical visual features are expanded. Based on the expanded semantic matching range, page elements whose current semantic features match the historical semantic features are selected from each page element and used as updated candidate elements obtained based on semantic matching. Based on the expanded visual matching range, page elements whose current visual features match the historical visual features are selected from each page element and used as updated candidate elements obtained based on visual matching. The target element is determined from the selected updated candidate elements.

5. The method according to claim 1, characterized in that, Performing the test operation on the target element includes: Monitor the execution status of test operations targeting the target element; If the execution status indicates that the test operation has failed, obtain the current page status of the page to be tested; The historical successful operation path corresponding to the current page state is obtained from the preset operation path knowledge base and used as the target operation path for the target element; the operation path knowledge base stores the historical successful operation paths corresponding to each page state. The test operation is performed on the target element according to the target operation path.

6. The method according to claim 5, characterized in that, The step of performing the test operation on the target element according to the target operation path includes: From the page elements of the current page content, identify the page elements that can respond to keyboard navigation operations and use them as navigation elements; From the navigation elements, identify the navigation element that has the same function as the target element, and use it as the replacement element corresponding to the target element; Based on the target operation path, generate a keyboard operation sequence to navigate to the alternative element; The test operation is performed on the substitute element by executing the keyboard operation sequence.

7. The method according to claim 1, characterized in that, The page test results include a page state test score. Generating the page test results for the page to be tested based on the actual page state and the expected page state includes: Obtain the text semantic matching degree, interface layout matching degree, and business rule matching degree between the actual page state and the expected page state; Based on the page type of the page to be tested, determine the weights corresponding to the text semantic matching degree, the interface layout matching degree, and the business rule matching degree, respectively. Based on the weights mentioned above, the text semantic matching degree, the interface layout matching degree, and the business rule matching degree are weighted and summed to obtain the page status test score of the page to be tested.

8. A testing device for a user interface, characterized in that, The device includes: A receiving module is used to receive test instructions for a page under test; the test instructions are used to indicate the element under test, to indicate the test operation for the element under test, and to indicate the expected page state of the page under test after the test operation is performed. The acquisition module is used to acquire historical description information of the element to be tested from a preset page description model; the page description model is pre-constructed based on the historical visual features and historical text semantic features of the page to be tested. The positioning module is used to obtain the current page content of the page to be tested, and locate the target element from each page element according to the matching degree between the element description information of each page element in the current page content and the historical description information; the target element has the same element function as the element to be tested. The testing module is used to perform the test operation on the target element, obtain the actual page state of the page under test after the test operation is performed, and generate the page test result of the page under test based on the actual page state and the expected page state.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images