Cloud real machine driven element knowledge base automatic construction and version evolution method for mobile terminal UI test

Abstract

The application relates to the technical field of mobile terminal application testing, and specifically discloses a cloud real-machine driving element knowledge base automatic construction and version evolution method for mobile terminal UI testing, which comprises the following steps: step S1, cloud real-machine UI element automatic scanning; step S2, locator confidence calculation and sorting; step S3, incremental version difference; step S4, cross-platform element alignment; and step S5, knowledge base storage and version management: storing the data generated in each step into a distributed database to realize version management, history tracing and version rollback of the knowledge base. The application realizes full-process automatic construction, intelligent optimization and version synchronous evolution of the element knowledge base, solves the core problems in the prior art, such as low manual maintenance efficiency, blind locator selection, high version update cost and poor cross-platform compatibility, can improve the executable rate of the test script by more than 30%, reduces the version update scanning cost by more than 70%, and the cross-platform element automatic alignment accuracy is more than 95%.

Description

Technical Field

[0001] This invention relates to the field of mobile application testing technology, specifically to a method for automatically constructing and evolving a cloud-based real device-driven element knowledge base for mobile UI testing, which falls under the category of software testing automation and intelligent technology. Background Technology

[0002] With the rapid development of the mobile internet, mobile applications are becoming increasingly complex, their iteration cycles are shortening, and parallel development on both Android and iOS platforms has become the norm. UI testing, as a crucial means of ensuring mobile application quality, directly impacts the efficiency and quality of application delivery due to its level of automation. The core of automated UI testing relies on an element knowledge base, which stores attribute and locator information for each UI element in the mobile application interface, providing data support for the generation and execution of test scripts.

[0003] However, the construction and maintenance of mobile UI testing element knowledge bases in existing technologies have the following prominent problems:

[0004] First, the knowledge base construction relies entirely on manual operation. Testers must manually collect the attributes and locator information of each UI element and enter it into the knowledge base. With the continuous enrichment of application functions and the exponential growth in the number of UI elements, the manual collection method is extremely inefficient and cannot keep up with the application's pace of iterating a version every week or even every day. Moreover, manual operation is prone to problems such as omissions and errors, resulting in inaccurate knowledge base data.

[0005] Second, the selection of locators lacks scientific basis. Testers often choose locators such as resource-id, text, accessibility-id, or XPath based on experience, without considering the differences in historical success rates of each locator under different scenarios and devices. This leads to a high failure rate in test script execution and affects testing efficiency.

[0006] Third, version updates use a full scan, which is too costly. Existing technologies rescan all UI elements during application version updates, regardless of whether the elements have changed, resulting in a large number of redundant operations, severely consuming cloud real device resources, and causing low update efficiency.

[0007] Fourth, cross-platform element management is inconsistent. The element knowledge bases of Android and iOS platforms are independent of each other, and the same logical element cannot be associated across platforms. Testers must maintain two sets of element information separately, test scripts cannot be reused across platforms, and cross-platform testing is highly complex.

[0008] Fifth, there is a lack of a complete version evolution mechanism. Current technology knowledge base updates are mostly one-off operations, unable to record data such as historical changes of elements and changes in locator confidence, unable to support version rollback and historical tracing, and unable to achieve self-iterative optimization of the knowledge base.

[0009] To address the aforementioned technical pain points, there is an urgent need for a method that can automatically construct an element knowledge base, intelligently sort locators, perform incremental version updates, cross-platform element alignment, and version evolution, in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a cloud-based real device-driven method for the automatic construction and version evolution of element knowledge bases for mobile UI testing. Through core technologies such as automatic cloud real device scanning, multi-locator confidence ranking, incremental version differentiation, cross-platform element semantic alignment, and knowledge base version management, the invention achieves fully automated construction, intelligent optimization, and synchronized version evolution of the element knowledge base, solving core problems in existing technologies such as low efficiency of manual maintenance, blind locator selection, high overhead of version updates, and poor cross-platform compatibility.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0012] A method for automatically building and evolving a cloud-based real device driver element knowledge base for mobile UI testing includes the following steps:

[0013] Step S1, Automatic scanning of UI elements on cloud real devices: The mobile devices in the cloud real device cluster are scheduled through the cloud real device management platform to remotely control the target mobile application, traverse all interfaces of the target mobile application, extract the attribute information of each UI element in each interface, and preprocess the attribute information to obtain element attribute data in a standardized format.

[0014] Step S2, Locator Confidence Calculation and Sorting: For each UI element with multiple locators, based on historical test data, a weighted summation algorithm is used to dynamically calculate the confidence of each locator, and the locators are sorted in descending order of confidence to generate a locator priority list;

[0015] Step S3, Incremental Version Differentiation: When the target mobile application releases a new version, the previous version's UI tree snapshot stored in the knowledge base is retrieved. An algorithm combining hierarchical traversal and attribute matching is used to compare the old and new version UI tree snapshots, and corresponding processing operations are triggered only for the identified changed elements.

[0016] Step S4, cross-platform element alignment: Extract the core feature vectors of UI elements from Android and iOS platforms, use the weighted cosine similarity algorithm to calculate the cross-platform element similarity, and generate a unified abstract identifier and unified abstract attribute for the same logical element;

[0017] Step S5, Knowledge Base Storage and Version Management: Store the data generated in each step into a distributed database to realize version management, historical tracing, and version rollback of the knowledge base.

[0018] Preferably, steps S1-S5 specifically include:

[0019] Step S1, Automatic Scanning of UI Elements on Cloud Real Devices: The mobile devices in the cloud real device cluster are scheduled through the cloud real device management platform to remotely control the target mobile application. All interfaces of the target mobile application are traversed, and the attribute information of each UI element in each interface is extracted. The attribute information includes at least element type, resource-id, text, accessibility-id, and xpath. A UI tree snapshot is generated based on the attribute information and the hierarchical relationship of each UI element, and the attribute information is preprocessed to obtain element attribute data in a standardized format.

[0020] Step S2, Locator Confidence Calculation and Ranking: For each UI element with multiple locators, based on historical test data, a weighted summation algorithm is used to dynamically calculate the confidence of each locator. The formula for calculating the confidence is as follows: Where S is the historical successful positioning rate, F is the failure rate, R is the cross-device compatibility coefficient, and α, β, and γ are weighting coefficients that satisfy α+β+γ=1; the confidence level is corrected based on the type characteristics of each locator, and a locator priority list is generated by arranging them in descending order of confidence level.

[0021] Step S3, Incremental Version Differentiation: When the target mobile application releases a new version, the previous version UI tree snapshot stored in the knowledge base is retrieved. An algorithm combining hierarchical traversal and attribute matching is used to compare the new and old version UI tree snapshots, identify added elements, changed elements and deleted elements, generate an element change list, and trigger corresponding processing operations only for the changed elements in the element change list.

[0022] Step S4, cross-platform element alignment: Extract core feature vectors containing semantic features, element type, relative position features and parent element features from UI elements on Android and iOS platforms respectively. Use the weighted cosine similarity algorithm to calculate the similarity between corresponding elements on the two platforms. Elements with similarity greater than or equal to a set threshold are judged as the same logical element. Generate a unified abstract identifier and a unified abstract attribute containing cross-platform locator mapping relationship for the same logical element after alignment.

[0023] Step S5, Knowledge Base Storage and Version Management: Store the element attribute data, locator priority list, cross-platform association relationship and UI tree snapshot generated in steps S1 to S4 into a distributed database, create a corresponding knowledge base version number for each application version, record version update content, retain historical version data, and support version rollback and historical traceability.

[0024] Preferably, the preprocessing in step S1 includes: removing duplicate elements and invalid attributes, deduplicating and standardizing text attributes, simplifying and optimizing XPath paths, and formatting the preprocessed data into a uniform JSON format;

[0025] The cloud-based real device management platform supports parallel scanning of multiple devices and versions of Android and iOS applications, and monitors the operating status of cloud-based real devices in real time to avoid device overload.

[0026] Preferably, in step S2, the rule for correcting the confidence level based on the type characteristics of each locator is as follows:

[0027] For the resource-id locator, when it exists, add an additional correction value of 0.1 to the calculated result;

[0028] For the accessibility-id locator, adjust the weight coefficient γ to 0.3, α to 0.5, keep β at 0.2, and add an additional correction value of 0.05 when both platforms are available;

[0029] For the text locator, when the text property has a history of changes, a correction value of 0.05 to 0.1 is subtracted based on the frequency of changes;

[0030] For XPath locators, when the UI tree structure changes, subtract a correction value of 0.05; when the XPath path depth exceeds 10 levels, subtract an additional correction value of 0.03.

[0031] For new elements scanned for the first time, the initial confidence scores for each locator on the Android platform are: resource-id 0.7, accessibility-id 0.6, xpath 0.5, and text 0.4; the initial confidence scores for each locator on the iOS platform are: accessibility-id 0.7, xpath 0.5, and text 0.4.

[0032] and / or

[0033] In step S2, the sorting rule for generating the locator priority list is as follows:

[0034] First, sort by confidence level in descending order;

[0035] When the confidence difference between two locators is less than 0.05, the locator with better cross-platform compatibility is prioritized, with accessibility-id preferred over xpath, and xpath preferred over text.

[0036] In the Android platform, if the resource-id locator exists, the resource-id will be ranked first regardless of whether the confidence difference with the accessibility-id is less than 0.05.

[0037] Preferably, step S2 further includes a real-time confidence update step: when a locator is used for UI testing and produces a successful or failed location result, the historical usage count, successful location count, and failed location count of the locator are updated, the confidence is recalculated, and the locator priority list is adjusted; a batch update of the confidence of all locators is performed every preset period; when the application version is updated, the UI tree structure undergoes significant changes, or a new device type is added, a full recalculation of the confidence is triggered.

[0038] Preferably, in step S3, the algorithm combining hierarchical traversal and attribute matching includes:

[0039] A depth-first traversal algorithm is used to traverse the UI tree snapshots of the old and new versions respectively, and a unique identifier is generated for each element node. On the Android platform, a combination of element type and resource-id is used, and on the iOS platform, a combination of element type and accessibility-id is used. When the corresponding attribute is missing, a combination of element type, text and relative position is used.

[0040] For each element node, perform core attribute matching on resource-id, accessibility-id, text, and element type, and determine whether the element is a newly added element, a modified element, a deleted element, or an unchanged element based on the matching results;

[0041] Set an attribute matching threshold. When the change in the text attribute is less than the set threshold and the coordinate position deviation is less than 5 pixels, it is judged as an unchanged element. When the XPath path changes but the core attribute does not change, it is judged as a changed element and only the XPath attribute is updated.

[0042] Preferably, in step S3, the processing operations triggered for different types of changed elements in the element change list include:

[0043] For newly added elements, a supplementary scan of the interface containing the element is triggered, and only the attribute information of the newly added element is scanned and extracted;

[0044] For a changed element, step S2 is triggered to recalculate the confidence and priority list of each locator for the changed element. When the text attribute of the changed element changes, an additional negative correction is applied to the confidence of the text locator according to the frequency of change.

[0045] For deleted elements, the element is marked as deleted in the knowledge base and its historical data is retained, but it is not physically deleted.

[0046] Preferably, in step S4, the semantic features in the core feature vector are extracted by semantic analysis of the element's text attribute and accessibility-id attribute using natural language processing technology; the relative position features are obtained by converting the element's absolute coordinates into a ratio relative to the interface width and height; the similarity calculation formula of the weighted cosine similarity algorithm is:

[0047]

[0048] Wherein, ω1 to ω5 are weight coefficients, satisfying ω1+ω2+ω3+ω4+ω5=1, with default values ​​of ω1=0.4, ω2=0.2, ω3=0.15, ω4=0.15, and ω5=0.1; the default threshold is 0.8; when the similarity between two elements is between 0.7 and 0.8, the element pair is marked as pending manual confirmation.

[0049] Preferably, step S4 further includes an association optimization step: using the manual alignment results as training data, the gradient descent algorithm is used to iteratively optimize the weight coefficients of the weighted cosine similarity algorithm; in step S4, the unified abstract identifier is generated in a format combining application identifier and logical function identifier, and the unified abstract attribute includes a unified function description, a unified element type and a cross-platform locator mapping table, and the cross-platform locator mapping table records the locator type, locator value and corresponding confidence of the same logical element on the Android platform and the iOS platform respectively.

[0050] Preferably, in step S5, the knowledge base version number adopts the format of major version number.minor version number.revision number; the retention of historical version data supports version rollback and comparison of differences between any two versions; the historical tracing includes recording the trajectory of element attribute changes, the trajectory of locator confidence changes, and the trajectory of cross-platform relationship changes; steps S1 to S5 realize the initial automatic construction of the element knowledge base during the first scan of the application, and during application version iteration, the incremental difference in step S3 only processes the changed elements, realizing the automatic version evolution of the element knowledge base.

[0051] The beneficial effects of this invention are:

[0052] (1) It realizes the automatic construction of the entire element knowledge base, completely eliminating the dependence on manual maintenance. In the scenario of rapid iteration of application versions, it can synchronize the knowledge base with the application version in real time, ensuring the accuracy and timeliness of the knowledge base data.

[0053] (2) By using the multi-locator confidence ranking algorithm, the locator is scientifically selected based on historical test data. Compared with existing technologies, the test script execution rate can be increased by more than 30%, and the test rework cost can be greatly reduced.

[0054] (3) By using incremental version differential technology, only the changed elements are processed, which can reduce the scanning overhead by more than 70% compared with the full scan method, and greatly save cloud real machine computing resources and network resources.

[0055] (4) Through the cross-platform element semantic alignment algorithm, the same logical element on Android and iOS platforms can be managed in a unified manner. The automatic alignment accuracy can reach more than 95%. Test scripts can be directly reused across platforms, reducing the complexity of cross-platform testing.

[0056] (5) A complete knowledge base version evolution system has been constructed, which supports version rollback and historical tracing, and the knowledge base can be self-optimized through dynamic updates of locator confidence. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] I. Overall Architecture

[0060] This invention provides a cloud-based real device-driven method for automatically building and evolving a knowledge base of elements for mobile UI testing. The overall architecture consists of five core functional modules that work together to form a complete technical loop through data flow and API calls. Specifically, these modules include: a cloud-based real device UI element automatic scanning module, an element locator confidence calculation and sorting module, an incremental version differential module, a cross-platform element alignment module, and a knowledge base storage and version management module. These modules collaborate sequentially to automate the entire process from UI element scanning to knowledge base version evolution.

[0061] II. Step S1 – Automatic Scanning of UI Elements on Cloud-based Real Devices

[0062] The cloud-based real device UI element automatic scanning module is the data acquisition foundation for the automatic construction of the element knowledge base. It is responsible for automatically scanning the UI of mobile applications based on the cloud-based real device cluster, extracting UI element attribute information, and generating a UI tree snapshot. This module consists of a cloud-based real device scheduling unit, an interface control unit, an element extraction unit, a UI tree generation unit, and a data preprocessing unit.

[0063] The cloud real device scheduling unit schedules suitable Android or iOS cloud real device devices from the cloud real device cluster according to the application version and device type requirements, establishes remote connections, supports parallel scheduling of multiple devices, and monitors the CPU utilization, memory utilization and network status of cloud real devices in real time to avoid device overload and ensure stable scanning.

[0064] The interface control unit remotely controls the target application on the cloud-based real device, simulating user operations (including launching the application, swiping the interface, clicking buttons, and switching pages), traversing all accessible interfaces of the application. For complex applications containing pop-ups, drop-down menus, and pagination, the unit automatically identifies pop-ups and triggers their closing, and automatically loads pagination content to ensure comprehensive scanning.

[0065] The element extraction unit extracts the core attributes of each UI element for each interface, including element type, resource-id (Android-specific), text, accessibility-id, and XPath, as well as coordinate position, hierarchical relationship, and interactivity. For the Android platform, attributes are extracted using the UIAutomator tool; for the iOS platform, attributes are extracted using the XCUITest tool. For elements whose accessibility-id cannot be directly extracted, a temporary identifier is automatically generated for subsequent cross-platform alignment.

[0066] The UI tree generation unit organizes all extracted UI elements into a tree structure according to the interface hierarchy (parent element - child element), generating a UI tree snapshot in XML format. The snapshot node attributes contain all extracted information for that element and record metadata such as the interface's unique identifier (interface package name plus interface class name), scan time, and device information.

[0067] The data preprocessing unit preprocesses element attributes and UI tree snapshots: removing duplicate elements and invalid attributes, correcting erroneous data; deduplicating and standardizing text attributes (removing spaces and unifying capitalization); simplifying and optimizing XPath; and formatting the preprocessed data into a unified JSON format to ensure data consistency and usability.

[0068] III. Step S2 – Calculation and Ranking of Element Locator Confidence

[0069] The element locator confidence calculation and sorting module is the first core algorithm innovation module of this invention, which implements the "multi-locator confidence sorting" algorithm. This module consists of a historical data retrieval unit, a confidence calculation unit, a locator sorting unit, and a confidence update unit.

[0070] 3.1 Confidence Calculation Formula

[0071] The confidence level is calculated using the following weighted summation formula:

[0072]

[0073] Among them, Confidence is the confidence level of the locator, ranging from 0 to 1. The higher the value, the higher the positioning success rate; S is the historical successful positioning rate, calculated by the formula S = number of successful positioning / total number of uses. When the total number of uses is 0 (new elements scanned for the first time), S takes the default value of 0.5; F is the failure rate, calculated by the formula F = number of failed positioning / total number of uses, where (1-F) represents positioning stability; R is the cross-device compatibility coefficient, which is the average of the successful positioning rate of the locator under different models and system versions, ranging from 0 to 1; α, β, and γ are weighting coefficients, satisfying α+β+γ=1, with default values ​​of α=0.6, β=0.2, and γ=0.2.

[0074] 3.2 Positioner Type Characteristic Correction Rules

[0075] To ensure that the confidence level more accurately reflects the actual availability of each locator, differentiated correction rules are set for the type characteristics of the four locators:

[0076] (1) Resource-id locator: It is unique to the Android platform and has the characteristics of strong uniqueness and high stability. When it exists, an additional confidence correction value of 0.1 is added to the basic calculation result.

[0077] (2) Accessibility-id locator: It is cross-platform for both Android and iOS, with strong versatility. The weight coefficients are adjusted to α=0.5, β=0.2, and γ=0.3 to increase the weight of the cross-device compatibility coefficient R, and an additional correction value of 0.05 is added when it is available on both platforms.

[0078] (3) text locator: It is easily affected by text modification and language switching, and the positioning stability is poor. When the text attribute has historical change records, the correction value of 0.05 to 0.1 is subtracted according to the change frequency; when the text attribute is empty or has a vague description, an additional correction value of 0.1 is subtracted.

[0079] (4) XPath locator: Based on the UI tree path, it is easily affected by changes in the interface hierarchy. When the UI tree structure changes, a correction value of 0.05 is subtracted; when the XPath path depth exceeds 10 levels, an additional correction value of 0.03 is subtracted.

[0080] For new elements scanned for the first time (without historical test data), the default initial confidence values ​​for the four locators are: Android platform - resource-id = 0.7, accessibility-id = 0.6, xpath = 0.5, and text = 0.4; iOS platform - accessibility-id = 0.7, xpath = 0.5, and text = 0.4.

[0081] 3.3 Locator Sorting Rules

[0082] The locator sorting unit generates a locator priority list according to the following rules: the primary sorting criterion is descending order of confidence value; when the confidence difference between two locators is less than 0.05, the locator with better cross-platform compatibility is prioritized, in the order of accessibility-id over xpath, and xpath over text; on the Android platform, if the resource-id locator exists, it is ranked first regardless of the confidence difference.

[0083] 3.4 Confidence Level Dynamic Update Mechanism

[0084] The confidence update unit implements a three-level update mechanism: real-time update—the confidence score is updated immediately when the locator is used for UI testing and generates location results; batch update—the confidence score is recalculated in batches every preset period (e.g., daily) based on the test data of the past 7 days; and abnormal update—a full confidence score recalculation is triggered when the application version is updated, the UI tree structure undergoes significant changes, or a new device type is added. Through this dynamic update mechanism, the locator priority can continuously adapt to changes in application version and device environment.

[0085] IV. Step S3 – Incremental Version Differentiation

[0086] The incremental version difference module is the second core algorithm innovation module of this invention, which implements the "incremental version difference" algorithm. This module consists of a version snapshot calling unit, a difference comparison unit, a changed element identification unit, and a scan triggering unit.

[0087] Version 4.1 Snapshot Retrieval and Verification

[0088] The version snapshot retrieval unit retrieves the UI tree snapshot of the previous version of the application from the knowledge base storage and version management module. It performs precise matching by using the application identifier and the unique identifier of the interface (the interface package name plus the interface class name) to verify the integrity and validity of the retrieved snapshot. If the snapshot is corrupted or the data is missing, a rescan of the interface is immediately triggered. If the application is being scanned for the first time, all elements in the new version's UI tree snapshot are treated as newly added elements and processed directly.

[0089] 4.2 Incremental Differential Algorithm of "Hierarchical Traversal + Attribute Matching"

[0090] The differential comparison unit adopts the incremental differential algorithm of "hierarchical traversal + attribute matching", and the specific implementation is as follows:

[0091] The first step is hierarchical traversal: A depth-first traversal is used for both the old and new UI tree snapshots, traversing each element node level by level from the root node. During the traversal, a unique identifier is generated for each element node: "element type + resource-id" is used on the Android platform; "element type + accessibility-id" is used on the iOS platform; when the corresponding attribute is missing, "element type + text + relative position" is used as the identifier. Simultaneously, the hierarchical position and parent node identifier of each node are recorded to capture changes in the element's hierarchy.

[0092] The second step is attribute matching: the four core attributes of each element node—resource-id, accessibility-id, text, and element type—are compared, and the element status is determined according to the following rules: if the unique identifier of the corresponding element node is the same in the old and new versions and all core attributes have not changed, it is determined as an "unchanged element"; if the unique identifier is the same but at least one core attribute has changed, it is determined as a "changed element," and the specific attribute change is recorded; if an element node exists in the UI tree of the new version but there is no corresponding node in the old version, it is determined as a "new element"; if an element node exists in the UI tree of the old version but there is no corresponding node in the new version, it is determined as a "deleted element."

[0093] The third step is optimization: set attribute matching thresholds. For elements with minor modifications to the text attribute (such as adding punctuation or adjusting spaces) and coordinate position deviations of less than 5 pixels, they are judged as "unchanged elements" to avoid triggering unnecessary rescanning due to minor changes. For elements whose XPath path changes but whose core attributes do not change, they are judged as "changed elements" and only the XPath attribute is updated. The difference results are then validated a second time to remove elements that are misjudged due to device differences or scanning errors.

[0094] Compared to the full scan method, this incremental differential algorithm can reduce scanning overhead by more than 70%, and the accuracy of element change recognition can reach more than 98%.

[0095] 4.3 Triggering of Change Element Processing

[0096] The scanning trigger unit triggers processing for different types of changed elements based on the element change list: for newly added elements, the cloud real device is triggered to perform a supplementary scan of the interface where the element is located, and only the attributes of the newly added element are scanned and extracted; for changed elements, the element locator confidence calculation and sorting module is triggered to recalculate its confidence, and when the text attribute changes, an additional negative correction is applied to the text locator; for deleted elements, they are marked as deleted in the knowledge base (no physical deletion is performed, and historical data is retained for traceability).

[0097] 5. Step S4 – Cross-platform element alignment

[0098] The cross-platform element alignment module is the third core algorithm innovation module of this invention, which implements the "cross-platform element alignment" algorithm. This module consists of a cross-platform data retrieval unit, an element feature extraction unit, a similarity matching unit, a unified abstract representation unit, and a relational storage unit.

[0099] 5.1 Core Feature Extraction and Standardization

[0100] The element feature extraction unit extracts five core features from various UI elements on Android and iOS platforms: semantic features (based on the text and accessibility-id attributes, using natural language processing technology for semantic analysis to extract core semantics, such as extracting "Log in now" and "Log in" as "Log in" semantics); element type features (unifying the cross-platform element types, such as unifying Android's Button and iOS's UIButton as "Button" type); relative position features (converting absolute coordinates to a ratio relative to the width and height of the interface to eliminate the impact of device size differences); interaction logic features (clicking to jump to the interface, triggered functions, and whether they are interactive, etc.); and parent element features (the type and semantic features of the parent node).

[0101] 5.2 Weighted Cosine Similarity Matching Algorithm

[0102] The similarity matching unit uses the following weighted cosine similarity calculation formula:

[0103]

[0104] Among them, ω1 to ω5 are weight coefficients that satisfy ω1+ω2+ω3+ω4+ω5=1, with default values ​​of ω1=0.4, ω2=0.2, ω3=0.15, ω4=0.15, and ω5=0.1. The value ranges from 0 to 1, calculated through NLP semantic analysis. If the two elements are of the same type, the value is 1; otherwise, the value is 0. It is obtained by calculating the Euclidean distance between relative coordinates; If the interaction logic is consistent, the value is 1; if it is similar, the value is 0.5; if it is completely different, the value is 0. Calculate the feature similarity of parent elements recursively.

[0105] The default similarity threshold is set to 0.8: when the similarity is greater than or equal to 0.8, they are determined to be the same logical element and cross-platform alignment is automatically completed; when the similarity is between 0.7 and 0.8, it is marked as awaiting manual confirmation, prompting testers to manually align it; when the similarity is less than 0.7, they are determined to be different logical elements and stored independently. The automatic alignment accuracy can reach over 95%.

[0106] 5.3 Generation of Unified Abstract Representation

[0107] The unified abstract representation unit generates the following unified representation for the same logical element that has been aligned: the unified abstract identifier adopts the format of "application identifier + logical function identifier" (e.g., "com.test.app+login_button"), which is globally unique; the unified abstract attributes include unified function description, unified element type, cross-platform locator mapping table (recording the locator type, locator value and confidence of the logical element on Android and iOS platforms respectively) and alignment metadata (alignment time, alignment method and similarity value).

[0108] The relationship storage unit transmits cross-platform element relationships and unified abstract attributes to the knowledge base storage and version management module, and uses the manual alignment results as training data. It then uses the gradient descent algorithm to iteratively optimize the weight coefficients ω1 to ω5 to improve the accuracy of subsequent automatic alignment.

[0109] VI. Step S5 – Knowledge Base Storage and Version Management

[0110] The knowledge base storage and version management module is the core supporting module of the element knowledge base, consisting of a data storage unit, a version management unit, a data query unit, a data update unit, and a history tracing unit.

[0111] The data storage unit uses a distributed database (such as MongoDB) to store the following five types of data in separate tables: UI element basic data (unique element identifier, element type, core attributes, interface identifier, and scan time); locator-related data (confidence values, priority lists, and historical test data for each locator); cross-platform related data (Android and iOS element associations, unified abstract identifiers, and cross-platform locator mapping tables); UI tree snapshot data (XML format snapshots and metadata for each version and interface); and knowledge base version data (version number, update time, and list of updated content). Storage employs multi-dimensional indexes (unique element identifier, unified abstract identifier, version number, and interface identifier) ​​and is equipped with data backup and encryption mechanisms.

[0112] The version management unit assigns a unique version number (in the format of "major version number.minor version number.revision number") to each version of each application, records the update content and triggering method, retains old version data, and supports version rollback and comparison of differences between any two versions.

[0113] The data update unit adopts a transaction management mechanism to ensure the atomicity and consistency of data updates. It performs corresponding entry, update, and mark-delete operations for newly added, modified, and deleted elements, and records each update log.

[0114] The historical traceability unit records the historical change trajectory of all data, including the trajectory of element attribute changes, the trajectory of locator confidence changes, and the trajectory of cross-platform relationship changes. It supports querying historical data at any point in time based on element identifier or version number.

[0115] VII. Complete Workflow

[0116] 7.1 Initial Scan – Automatic Initial Construction of the Knowledge Base

[0117] Step 1: Testers submit scanning tasks through the cloud-based real device management platform, configuring the Android and iOS dual-platform application installation packages, target device type, and scanning range.

[0118] Step 2: The cloud real device scheduling unit schedules suitable Android and iOS cloud real device devices to install and launch the target application.

[0119] Step 3: The interface control unit traverses all accessible interfaces, the element extraction unit extracts the UI element attributes of each interface simultaneously, the UI tree generation unit generates an XML format UI tree snapshot according to the hierarchical relationship, and the data preprocessing unit completes the standardization process.

[0120] Step 4: Transfer the preprocessed element attribute data to the element locator confidence calculation and sorting module. The historical data retrieval unit returns the default initial confidence parameters. The confidence calculation unit calculates the confidence of each locator according to the initial parameters. The locator sorting unit generates a locator priority list.

[0121] Step 5: The cross-platform element alignment module extracts core features from elements on Android and iOS platforms, calculates similarity using a weighted cosine similarity algorithm, and automatically aligns elements with a similarity greater than or equal to 0.8, generating a unified abstract identifier and unified abstract attributes.

[0122] Step 6: The knowledge base storage and version management module categorizes all data and stores it in a distributed database. The version management unit initializes the knowledge base version (e.g., 1.0.0) to complete the initial construction of the knowledge base.

[0123] 7.2 Application Version Updates – Automatic Version Evolution of the Knowledge Base

[0124] Step 1: Testers submit a new version scanning task, and the cloud real device scheduling unit schedules the device to install and start the new version application, completing the UI tree snapshot collection and preprocessing.

[0125] Step 2: The incremental version difference module retrieves the snapshot of the old version UI tree, executes the "hierarchical traversal + attribute matching" incremental difference algorithm, generates a list of element changes, and identifies newly added, modified, and deleted elements.

[0126] Step 3: The scan trigger unit triggers processing according to the change list: for newly added elements, it triggers supplementary scanning and calculates the initial confidence level; for changed elements, it triggers confidence level recalculation; for deleted elements, it triggers a mark deletion operation.

[0127] Step 4: The cross-platform element alignment module performs cross-platform similarity matching on newly added and changed elements, and updates or creates unified abstract identifiers and unified abstract attributes.

[0128] Step 5: The knowledge base storage and version management module completes the data update, the version management unit creates a new version number (e.g., 1.0.1), records the content of this update, and retains the old version data.

[0129] Step 6: The confidence update unit synchronizes the latest confidence data to the knowledge base, and the history tracking unit records all changes in this version's evolution. The entire process is fully automated and requires no manual intervention.

[0130] VIII. Implementation Results

[0131] This embodiment uses a mobile e-commerce application as an example. This application has versions on both Android and iOS platforms, contains over 3000 UI elements, and is updated weekly. After adopting the method of this invention:

[0132] (1) The initial construction of the knowledge base was shortened from about 2 weeks for manual input to about 4 hours for automatic scanning, which improved efficiency by about 80 times, and the data accuracy increased from about 85% for manual operation to more than 98%.

[0133] (2) During version updates, the incremental difference algorithm identifies about 150 changed elements (accounting for 5% of all elements). Only the changed elements are processed, and the version update time is shortened from about 3 hours for full scan to about 20 minutes, and the scanning overhead is reduced by about 78%.

[0134] (3) The multi-locator confidence ranking algorithm increases the first-time execution success rate of the test script from about 67% to about 95%, and significantly reduces the rework rate of the test script;

[0135] (4) The cross-platform element alignment algorithm successfully and automatically aligns about 92% of the same logical elements across platforms in this application, with only about 8% requiring manual confirmation. Based on the unified abstract identifier, the cross-platform reuse of test scripts is realized, and the workload of cross-platform test script maintenance is reduced by about 60%.

[0136] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for automatically building and versioning a cloud-based real device-driven element knowledge base for mobile UI testing, characterized in that... Includes the following steps: Step S1, Automatic scanning of UI elements on cloud real devices: The mobile devices in the cloud real device cluster are scheduled through the cloud real device management platform to remotely control the target mobile application, traverse all interfaces of the target mobile application, extract the attribute information of each UI element in each interface, and preprocess the attribute information to obtain element attribute data in a standardized format. Step S2, Locator Confidence Calculation and Sorting: For each UI element with multiple locators, based on historical test data, a weighted summation algorithm is used to dynamically calculate the confidence of each locator, and the locators are sorted in descending order of confidence to generate a locator priority list; Step S3, Incremental Version Differentiation: When the target mobile application releases a new version, the previous version's UI tree snapshot stored in the knowledge base is retrieved. An algorithm combining hierarchical traversal and attribute matching is used to compare the old and new version UI tree snapshots, and corresponding processing operations are triggered only for the identified changed elements. Step S4, cross-platform element alignment: Extract the core feature vectors of UI elements from Android and iOS platforms, use the weighted cosine similarity algorithm to calculate the cross-platform element similarity, and generate a unified abstract identifier and unified abstract attribute for the same logical element; Step S5, Knowledge Base Storage and Version Management: Store the data generated in each step into a distributed database to realize version management, historical tracing, and version rollback of the knowledge base.

2. The method according to claim 1, characterized in that, The specific steps S1-S5 are as follows: Step S1, Automatic Scanning of UI Elements on Cloud Real Devices: The mobile devices in the cloud real device cluster are scheduled through the cloud real device management platform to remotely control the target mobile application. All interfaces of the target mobile application are traversed, and the attribute information of each UI element in each interface is extracted. The attribute information includes at least element type, resource-id, text, accessibility-id, and xpath. A UI tree snapshot is generated based on the attribute information and the hierarchical relationship of each UI element, and the attribute information is preprocessed to obtain element attribute data in a standardized format. Step S2, Locator Confidence Calculation and Ranking: For each UI element with multiple locators, based on historical test data, a weighted summation algorithm is used to dynamically calculate the confidence of each locator. The formula for calculating the confidence is as follows: Where S is the historical successful positioning rate, F is the failure rate, R is the cross-device compatibility coefficient, and α, β, and γ are weighting coefficients that satisfy α+β+γ=1; the confidence level is corrected based on the type characteristics of each locator, and a locator priority list is generated by arranging them in descending order of confidence level. Step S3, Incremental Version Differentiation: When the target mobile application releases a new version, the previous version UI tree snapshot stored in the knowledge base is retrieved. An algorithm combining hierarchical traversal and attribute matching is used to compare the new and old version UI tree snapshots, identify added elements, changed elements and deleted elements, generate an element change list, and trigger corresponding processing operations only for the changed elements in the element change list. Step S4, cross-platform element alignment: Extract core feature vectors containing semantic features, element type, relative position features and parent element features from UI elements on Android and iOS platforms respectively. Use the weighted cosine similarity algorithm to calculate the similarity between corresponding elements on the two platforms. Elements with similarity greater than or equal to a set threshold are judged as the same logical element. Generate a unified abstract identifier and a unified abstract attribute containing cross-platform locator mapping relationship for the same logical element after alignment. Step S5, Knowledge Base Storage and Version Management: Store the element attribute data, locator priority list, cross-platform association relationship and UI tree snapshot generated in steps S1 to S4 into a distributed database, create a corresponding knowledge base version number for each application version, record version update content, retain historical version data, and support version rollback and historical traceability.

3. The method according to claim 2, characterized in that, The preprocessing in step S1 includes: removing duplicate elements and invalid attributes, deduplicating and standardizing text attributes, simplifying and optimizing XPath paths, and formatting the preprocessed data into a unified JSON format; The cloud-based real device management platform supports parallel scanning of multiple devices and versions of Android and iOS applications, and monitors the operating status of cloud-based real devices in real time to avoid device overload.

4. The method according to claim 2, characterized in that, In step S2, the rule for correcting the confidence level based on the type characteristics of each locator is as follows: For the resource-id locator, when it exists, add an additional correction value of 0.1 to the calculated result; For the accessibility-id locator, adjust the weight coefficient γ to 0.3, α to 0.5, keep β at 0.2, and add an additional correction value of 0.05 when both platforms are available; For the text locator, when the text property has a history of changes, a correction value of 0.05 to 0.1 is subtracted based on the frequency of changes; For XPath locators, when the UI tree structure changes, subtract a correction value of 0.05; when the XPath path depth exceeds 10 levels, subtract an additional correction value of 0.

03. For new elements scanned for the first time, the initial confidence scores for each locator on the Android platform are: resource-id 0.7, accessibility-id 0.6, xpath 0.5, and text 0.4; the initial confidence scores for each locator on the iOS platform are: accessibility-id 0.7, xpath 0.5, and text 0.

4. and / or In step S2, the sorting rule for generating the locator priority list is as follows: First, sort by confidence level in descending order; When the confidence difference between two locators is less than 0.05, the locator with better cross-platform compatibility is prioritized, with accessibility-id preferred over xpath, and xpath preferred over text. In the Android platform, if the resource-id locator exists, the resource-id will be ranked first regardless of whether the confidence difference with the accessibility-id is less than 0.

05.

5. The method according to claim 2, characterized in that, Step S2 also includes a real-time confidence update step: when a locator is used for UI testing and produces a successful or failed location result, the historical usage count, successful location count, and failed location count of the locator are updated, the confidence is recalculated, and the locator priority list is adjusted; a batch update of the confidence of all locators is performed every preset period; when the application version is updated, the UI tree structure undergoes significant changes, or a new device type is added, a full recalculation of the confidence is triggered.

6. The method according to claim 2, characterized in that, In step S3, the algorithm combining hierarchical traversal and attribute matching includes: A depth-first traversal algorithm is used to traverse the UI tree snapshots of the old and new versions respectively, and a unique identifier is generated for each element node. On the Android platform, a combination of element type and resource-id is used, and on the iOS platform, a combination of element type and accessibility-id is used. When the corresponding attribute is missing, a combination of element type, text and relative position is used. For each element node, perform core attribute matching on resource-id, accessibility-id, text, and element type, and determine whether the element is a newly added element, a modified element, a deleted element, or an unchanged element based on the matching results; Set an attribute matching threshold. When the change in the text attribute is less than the set threshold and the coordinate position deviation is less than 5 pixels, it is judged as an unchanged element. When the XPath path changes but the core attribute does not change, it is judged as a changed element and only the XPath attribute is updated.

7. The method according to claim 2, characterized in that, In step S3, the processing operations triggered for different types of changed elements in the element change list include: For newly added elements, a supplementary scan of the interface containing the element is triggered, and only the attribute information of the newly added element is scanned and extracted; For a changed element, step S2 is triggered to recalculate the confidence and priority list of each locator for the changed element. When the text attribute of the changed element changes, an additional negative correction is applied to the confidence of the text locator according to the frequency of change. For deleted elements, the element is marked as deleted in the knowledge base and its historical data is retained, but it is not physically deleted.

8. The method according to claim 2, characterized in that, In step S4, the semantic features in the core feature vector are extracted by semantic analysis of the element's text attribute and accessibility-id attribute using natural language processing techniques; the relative position features are obtained by converting the element's absolute coordinates into a ratio relative to the interface width and height; the similarity calculation formula of the weighted cosine similarity algorithm is as follows: ； Wherein, ω1 to ω5 are weight coefficients, satisfying ω1+ω2+ω3+ω4+ω5=1, with default values ​​of ω1=0.4, ω2=0.2, ω3=0.15, ω4=0.15, and ω5=0.1; the default threshold is 0.8; when the similarity between two elements is between 0.7 and 0.8, the element pair is marked as pending manual confirmation.

9. The method according to claim 2, characterized in that, Step S4 also includes an association optimization step: using the manual alignment results as training data, the gradient descent algorithm is used to iteratively optimize the weight coefficients of the weighted cosine similarity algorithm; in step S4, the unified abstract identifier is generated in a format combining application identifier and logical function identifier, and the unified abstract attribute includes a unified function description, a unified element type and a cross-platform locator mapping table, and the cross-platform locator mapping table records the locator type, locator value and corresponding confidence of the same logical element on the Android platform and the iOS platform respectively.

10. The method according to claim 2, characterized in that, In step S5, the knowledge base version number adopts the format of major version number.minor version number.revision number; the retention of historical version data supports version rollback and comparison of differences between any two versions; the historical tracing includes recording the trajectory of element attribute changes, the trajectory of locator confidence changes, and the trajectory of cross-platform relationship changes; steps S1 to S5 realize the initial automatic construction of the element knowledge base during the first scan of the application, and during application version iteration, the incremental difference in step S3 only processes the changed elements, realizing the automatic version evolution of the element knowledge base.

Images