Data acquisition device and method for mobile terminal page, medium and equipment

Abstract

The invention provides a data collection device and method for a mobile terminal page, a medium and equipment, and the method comprises the steps: determining an operable control set of a first page based on a first accessibility tree and a first page image of the first page of a mobile terminal; the operable control set comprises operable controls on the first page; based on the operable control set, determining an operated second page corresponding to the operable control; based on the first page, the operable control and the second page, performing incremental updating on the first page relation graph to obtain a second page relation graph; the first page relation graph is an initialized graph, or the first page relation graph is obtained by previous incremental updating; and determining page data of the mobile terminal based on the first accessibility tree of the first page, the first page image, the operable control set and the second page relation graph. Therefore, the comprehensiveness of the collected pages and controls can be effectively improved, and the test coverage rate can be increased.

Description

Technical Field

[0001] This disclosure relates to testing techniques, and in particular to a data acquisition device, method, medium, and apparatus for mobile web pages. Background Technology

[0002] As mobile applications become increasingly complex, their testing and data collection become both crucial and challenging. In related technologies, automated testing and data collection of mobile user interfaces (UI) typically relies on the page's accessibility tree (AT) to identify interactive (or operable) controls. However, the accessibility tree only includes standard UI controls, while the page may also contain non-standard, custom-rendered UI controls. Since these non-standard, custom UI controls—for example, UI controls in pages rendered using game engines, the Flutter framework, or custom views—are not declared in the accessibility tree, they cannot be identified during testing and data collection. This results in incomplete UI control collection, which in turn prevents testing from covering these controls, leading to low test coverage. Summary of the Invention

[0003] The embodiments of this disclosure provide a data acquisition device, method, medium, and apparatus for mobile web pages to improve the comprehensiveness of identified controls and facilitate increased test coverage.

[0004] A first aspect of this disclosure provides a data acquisition device for a mobile terminal page, comprising: at least one processor configured to: acquire a first accessibility tree and a first page image of a first page of a mobile terminal; determine a set of operable controls for the first page based on the first accessibility tree and the first page image; the set of operable controls includes operable controls on the first page; determine a second page corresponding to the operable controls after an operation based on the set of operable controls; incrementally update a first page relationship graph based on the first page, the operable controls, and the second page to obtain a second page relationship graph; the first page relationship graph is an initialized graph, or the first page relationship graph was obtained from a previous incremental update; and determine page data of the mobile terminal based on the first accessibility tree, the first page image, the set of operable controls, and the second page relationship graph of the first page.

[0005] A second aspect of this disclosure provides a method for collecting data from a mobile terminal page, comprising: acquiring a first accessibility tree and a first page image of a first page on a mobile terminal; determining a set of operable controls for the first page based on the first accessibility tree and the first page image; the set of operable controls including operable controls on the first page; determining a second page corresponding to the operable controls after an operation based on the set of operable controls; incrementally updating a first page relationship graph based on the first page, the operable controls, and the second page to obtain a second page relationship graph; the first page relationship graph being an initialized graph, or the first page relationship graph being obtained from a previous incremental update; and determining page data of the mobile terminal based on the first accessibility tree, the first page image, the set of operable controls, and the second page relationship graph of the first page.

[0006] A third aspect of this disclosure is to provide a computer-readable storage medium storing a computer program that is executed by a processor to perform the data collection method for mobile web pages described in any of the above embodiments of this disclosure.

[0007] A fourth aspect of this disclosure provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory, the processor executing the executable instructions to implement the mobile terminal page data acquisition method described in any of the above embodiments of this disclosure; or, the electronic device comprising the mobile terminal page data acquisition device described in any of the above embodiments.

[0008] A fifth aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs a data acquisition method for a mobile terminal page provided in any of the above embodiments of this disclosure.

[0009] The mobile terminal page data acquisition apparatus, method, medium, and device provided in the above embodiments of this disclosure acquire a first accessibility tree and a first page image of a mobile terminal. Based on the first accessibility tree and the first page image, the set of operable controls for the first page is determined. Then, based on the set of operable controls, the second page corresponding to the operable controls after operation is determined. Based on the first page, operable controls, and the second page, the relationship graph of the first page is incrementally updated to obtain the relationship graph of the second page. Based on the first accessibility tree, the first page image, the set of operable controls, and the relationship graph of the second page, the page data of the mobile terminal is determined. Since the set of operable controls for the page is determined by combining the page accessibility tree and the page image, other non-standard and custom controls can be obtained from the page image on the basis of obtaining standard controls, which effectively improves the comprehensiveness of the collected page controls. In addition, during the traversal of page controls, the page relationship graph is incrementally generated. The page relationship graph represents the association between the page and the page after the operation jump of the controls in the page, which further improves the richness of the collected page data. Moreover, the page relationship graph can provide an effective reference for the testing of mobile terminal pages, which is conducive to improving the efficiency of test coverage. Attached Figure Description

[0010] Figure 1 This is an exemplary application scenario of the mobile terminal page data collection device provided in this disclosure;

[0011] Figure 2 This is a schematic diagram of the structure of a data acquisition device for a mobile terminal page provided in an exemplary embodiment of this disclosure;

[0012] Figure 3 This is a schematic diagram of the structure of a data collection device for a mobile terminal page provided in another exemplary embodiment of this disclosure;

[0013] Figure 4 This is a flowchart illustrating the data acquisition process of a mobile terminal page data acquisition device provided in an exemplary embodiment of this disclosure;

[0014] Figure 5 This is a schematic diagram illustrating the principle of establishing a page relationship graph provided by an exemplary embodiment of this disclosure;

[0015] Figure 6 This is a flowchart illustrating a data collection method for a mobile webpage provided in an exemplary embodiment of this disclosure;

[0016] Figure 7 This is a flowchart illustrating a data collection method for a mobile terminal page provided in another exemplary embodiment of this disclosure;

[0017] Figure 8This is a flowchart illustrating a data collection method for a mobile webpage provided in yet another exemplary embodiment of this disclosure;

[0018] Figure 9 This is a flowchart illustrating a data collection method for a mobile webpage provided in yet another exemplary embodiment of this disclosure;

[0019] Figure 10 This is a structural diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0021] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0022] This disclosure outlines

[0023] In developing this disclosure, the inventors discovered that as mobile applications become increasingly complex, their testing and data collection become both more important and more difficult. In related technologies, the automated testing and data collection of mobile user interfaces typically relies on the page's accessibility tree to identify interactive (or operable) controls. However, the accessibility tree only includes standard UI controls, while the page may also include non-standard, custom-rendered UI controls. Because these non-standard, custom UI controls are not included in the accessibility tree—for example, UI controls in pages rendered using game engines, Flutter frameworks, or custom views are not declared in the accessibility tree—these UI controls cannot be identified during testing and data collection, resulting in incomplete UI control collection. This also means that these UI controls cannot be covered during testing, leading to low test coverage.

[0024] Exemplary Overview

[0025] Figure 1 This is an exemplary application scenario of the mobile webpage data collection device provided in this disclosure. For example... Figure 1As shown, the mobile terminal page data collection device of this disclosure can be installed in electronic device 11. Electronic device 11 includes, but is not limited to, servers, PCs, tablets, and other page data collection tools with relevant functions. Mobile terminal device 12 is the device for collecting page data. Mobile terminal device 12 is, for example, but not limited to, a device with a preset type of operating system (OS). The preset type of operating system is, for example, but not limited to, Android or other types of systems. Connecting mobile terminal device 12 to electronic device 11 enables electronic device 11 to interact with mobile terminal device 12. Electronic device 11 can control the operation of the mobile terminal to traverse all or part of the pages of the mobile terminal software, taking any traversed page as the first page. The mobile terminal software may include, but is not limited to, one or more of the mobile terminal operating system, application program (APP), etc. Electronic device 11 can acquire the first accessibility tree and the first page image of the first page of the mobile terminal; based on the first accessibility tree and the first page image, determine the set of operable controls for the first page; based on the set of operable controls, determine the second page corresponding to the operable controls after the operation; based on the first page, operable controls, and the second page, incrementally update the relationship graph of the first page to obtain the relationship graph of the second page; based on the first accessibility tree, the first page image, the set of operable controls, and the relationship graph of the second page, determine the page data of the mobile terminal. Because the set of operable controls for the page is determined by combining the accessibility tree and the page image of the mobile terminal, it is possible to obtain other non-standard or custom controls from the page image, in addition to obtaining standard controls, effectively improving the comprehensiveness of the collected page controls. Furthermore, during the traversal of page controls, an incremental page relationship graph is generated. The page relationship graph represents the association between the page and the page after the operation jump of the controls within the page, further improving the richness of the collected mobile terminal page data. Moreover, the page relationship graph can provide an effective reference for testing mobile terminal pages, which is beneficial for improving test coverage efficiency.

[0026] Exemplary device

[0027] Figure 2 This is a schematic diagram of the structure of a mobile webpage data acquisition device provided in an exemplary embodiment of this disclosure. The mobile webpage data acquisition device provided in this embodiment can be applied to electronic devices, which may include, but are not limited to, servers, PCs, mobile phones, tablets, etc. Figure 2 As shown in the figure, the data acquisition device 20 for a mobile terminal page in this embodiment may include at least one processor 21. The figure uses multiple processors 21 as an example, but in practical applications, it is not limited to multiple processors.

[0028] At least one processor 21 is configured to: acquire a first accessibility tree and a first page image of a first page of a mobile device; determine a set of operable controls for the first page based on the first accessibility tree and the first page image; the set of operable controls includes operable controls on the first page; determine a second page after operation corresponding to the operable controls based on the set of operable controls; incrementally update a relationship graph of the first page based on the first page, the operable controls, and the second page to obtain a relationship graph of the second page; the relationship graph of the first page is an initialized graph, or the relationship graph of the first page is obtained from a previous incremental update; and determine page data of the mobile device based on the first accessibility tree, the first page image, the set of operable controls, and the relationship graph of the second page.

[0029] At least one processor 21 may include, but is not limited to, one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and other types of processors. A mobile device is a device with a pre-defined operating system installed. The pre-defined operating system may be, but is not limited to, Android or other types of systems. The first page of a mobile device is the UI page that can be displayed during the operation of the mobile device's software. The mobile device's software may include, but is not limited to, one or more of the mobile device's operating system, applications, etc.

[0030] An accessibility tree is a tree-like structured data representation of controls and their logical structure that are meaningful for user interaction, built upon the UI view hierarchy. The accessibility tree provides a complete description of the hierarchical relationships of the user interface. Each node in the accessibility tree is an accessibility node information object, corresponding to a control on the page. Controls can include, but are not limited to, buttons, input boxes, list items, etc. The first accessibility tree is the accessibility tree corresponding to the first page. The page image is a screenshot of the page. The first page image is a screenshot of the first page.

[0031] In embodiments of this disclosure, the mobile page data acquisition device (hereinafter referred to as the device) can be connected to a mobile device to be tested or to collect page data. The device communicates with the mobile device to obtain a first accessibility tree and a first page image of the mobile device. The mobile device includes, but is not limited to, smartphones, tablets, and in-vehicle central control systems running a preset type of operating system. Optionally, the first accessibility tree of the first page can be obtained by calling native interfaces such as the mobile device's Accessibility Services or UI Automator framework. Optionally, a screenshot of the first page can be obtained as the first page image by calling the mobile device's page screenshot service.

[0032] In some optional embodiments, the software for collecting page data may include, but is not limited to, all or part of the mobile device's operating system and various applications. For example, the software for collecting page data may be an operating system; or, the software for collecting page data may be one or more apps running on the operating system; or the software for collecting page data may include the operating system and all apps. Specific settings can be configured according to actual needs. Access to the relevant page can be achieved by controlling the mobile device to run the operating system and / or applications. For example, by sending instructions simulating user operations to the mobile device, the mobile device can be controlled to access the relevant page.

[0033] Optionally, the pages to be collected can be traversed according to the page data collection requirements, with each traversed page serving as the first page. That is, the first page can be the first page, any intermediate page, or the last page. For example, when a mobile device enters the first page from which page data is to be collected, this first page is used as the first page to collect relevant page data. After navigating to the next page by interacting with controls on the first page, this next page becomes the first page to collect relevant page data, and so on, thus achieving page traversal. Based on the first page, subsequent pages can be accessed through control interactions, thereby achieving automated page traversal of the operating system and / or application.

[0034] After obtaining the first accessibility tree and the first page image, the set of operable controls for the first page can be determined based on these two data points. The set of operable controls includes all operable controls on the first page. Operable controls refer to controls that can be manipulated by the user, including, but not limited to, clickable controls. Standard controls can be obtained from the first accessibility tree, and standard controls as well as other non-standard or custom controls can be obtained from the first page image. Therefore, the set of operable controls obtained by combining the first accessibility tree and the first page image includes all operable controls on the first page.

[0035] After obtaining the set of operable controls on the first page, the second page following the operation of each operable control can be determined based on this set. Optionally, the operable controls in the set can be traversed according to a preset traversal method, and the second page following the operation of each traversed control can be determined. Optionally, an operation command for the operable control can be sent to the mobile device. This operation command simulates the user's operation on the operable control, and the mobile device responds to the operation command, performs the operation on the operable control, and enters the second page. Optionally, after entering the second page, the second accessibility tree and the second page image can be obtained.

[0036] In some optional embodiments, after obtaining the set of operable controls for the first page, each operable control in the set can be traversed using a breadth-first search (BFS) approach. For each operable control encountered, the second page following the operation corresponding to that control is determined. Alternatively, the operable controls in the set can be traversed using a depth-first search (DFS) approach. In this case, because it is depth-first, unlike breadth-first search, for each operable control in the set, after determining the second page following the operation corresponding to that control, it is necessary to traverse along the depth direction based on the second page, following the depth-first traversal rules, and finally complete the traversal of the operable controls for each page.

[0037] A page relationship graph is a directed graph describing the user interface flow of the software, representing the entire page data to be collected. The page relationship graph consists of multiple nodes and directed edges between them. Each node represents a unique page and can store page identifiers, structured data of the accessibility tree, page images, and page description information such as the set of operable controls for that page. Each directed edge represents an action that transitions from a source node to a target node, representing an operation on a control on the source page corresponding to the source node. For example, if the source page corresponding to the source node is page 1 and the target page corresponding to the target node is page 2, the directed edge from the source node to the target node represents the operation on an operable control on page 1 that causes the jump to page 2. Each edge can store description information of the operation that caused this transition, including, but not limited to, one or more of the following: operation type, control information, and input text content. Control information includes, but is not limited to, the control name and control type.

[0038] The complete page relationship graph of the software collecting page data needs to be built incrementally. For example, each time a page is traversed, a new node for the current page and the edges between the current page and historical pages are added to the previously updated page relationship graph. When all pages have been traversed, the page relationship graph is complete. The first page relationship graph is the page relationship graph after the previous update. After navigating to the second page, the first page relationship graph needs to be incrementally updated based on the first page, the operable controls on the first page, and the second page to obtain the updated second page relationship graph. For example, nodes corresponding to the second page are added to the first page relationship graph, directed edges are established between the first and second pages, and the attributes of nodes and edges are recorded. The attributes of nodes are page description information, and the attributes of edges are operation description information of operable controls.

[0039] Optionally, if the first page is the first page in the hierarchy, then the first page relationship graph is an initialized graph. The first page relationship graph may include one or more initialized nodes. For example, the first page relationship graph may include an initialized node, which can be the node corresponding to the first page. To jump to the second page by performing an operation on an operable control on the first page, based on the first page, the operable control, and the second page, a node corresponding to the second page can be added to the first page relationship graph, establishing a directed edge between the first and second pages. The attribute of this directed edge is the operation description information of the operable control that caused the jump to the second page.

[0040] Optionally, if the first page is not the first page, the first page relationship graph is the historical page relationship graph obtained from the previous update. For example, after navigating to the first page, the previous relationship page graph is incrementally updated based on the pages associated with the first page, the first page, and the operable controls that caused the navigation, to obtain the first page relationship graph. The pages associated with the first page refer to the source page before navigation to the first page, i.e., navigation to the first page occurs by operating a certain operable control on that source page. After navigating to the second page, the first page relationship graph is incrementally updated based on the first page, operable controls, and the second page, to obtain the second page relationship graph. Optionally, for each operable control in the first page, the page relationship graph can be incrementally updated in the above manner, adding the nodes of the second page corresponding to each operable control to the page relationship graph, until all pages and all operable controls have been traversed. The page relationship graph can include nodes of all pages and directed edges between nodes. For example, if all operable controls on all pages have been traversed, and the second page corresponding to the last operable control traversed is the last page, the second page does not contain operable controls, or the second page is a historical page that has already been traversed, then the updated second page relationship graph can represent the page relationship graph of the entire software.

[0041] Optionally, the first accessibility tree, the first page image, and the set of operable controls for the first page can be stored to form part of the page data on the mobile device. The page relationship graph can be stored in a preset storage space, and incremental updates to the page relationship graph are achieved by incrementally writing to the preset storage space. In response to the completion of traversing all operable controls on all pages, the second page relationship graph is included as part of the page data on the mobile device, and together with the accessibility tree, page image, and set of operable controls for each page, constitutes the page data on the mobile device. Optionally, the page relationship graph and the related data of each page can be stored in the same memory or database, or the page relationship graph and the related data of each page can be stored separately. For example, the page relationship graph can be stored in a first database, and the related data of each page can be stored in a second database. The page relationship graph and the related data of each page can be associated through page identifiers. Thus, the attributes of the nodes and edges of the page relationship graph can be simply described, and the actual data can be obtained from the recorded related data of the pages. For example, the attributes of the nodes can simply record the page identifier of the corresponding page, and the attributes of the edges can record the identifier of the controls. The page's accessible node tree, page images, and set of operable controls are stored separately to save storage resources. Related page data includes the page's accessibility tree, page images, and set of operable controls.

[0042] The mobile terminal page data acquisition device provided in this embodiment acquires a first accessibility tree and a first page image of a first page on the mobile terminal. Based on the first accessibility tree and the first page image, it determines the set of operable controls for the first page. Then, based on the set of operable controls, it determines the second page corresponding to the operable controls after the operation. Based on the first page, operable controls, and the second page, it incrementally updates the relationship graph of the first page to obtain the relationship graph of the second page. Based on the first accessibility tree, the first page image, the set of operable controls, and the relationship graph of the second page, it determines the page data of the mobile terminal. Since the set of operable controls for the page is determined by combining the page accessibility tree and the page image, it is possible to obtain other non-standard or custom controls from the page image in addition to obtaining standard controls, effectively improving the comprehensiveness of the collected page controls. In addition, during the traversal of page controls, the page relationship graph is generated incrementally. The page relationship graph represents the association between the page and the page after the operation jump of the controls in the page, further improving the richness of the collected page data. Moreover, the page relationship graph can provide an effective reference for the testing of mobile terminal pages, which is conducive to improving the efficiency of test coverage.

[0043] In some optional embodiments, based on any of the above embodiments, the first page relationship graph includes the operation flow relationship between the collected pages; at least one processor is specifically configured to, or to acquire the first accessibility tree and the first page image of the mobile terminal, include:

[0044] Based on the first page relationship graph, a pre-configured page traversal algorithm is used to determine the first page; the page traversal algorithm can be any one of breadth-first search algorithm, depth-first search algorithm, or reinforcement learning algorithm; the mobile terminal's interface is called to obtain the first accessibility tree and the first page image.

[0045] In the first page relationship graph, some nodes may correspond to pages with untraversed operable controls. The first page can be determined from the pages corresponding to each node in the graph using a pre-configured page traversal algorithm. Breadth-first search employs a horizontal expansion strategy, prioritizing the access of all adjacent nodes of the current node before expanding outwards layer by layer. Depth-first search employs a vertical exploration strategy, delving deep along a path and then backtracking to branch points to continue exploration. Reinforcement learning algorithms determine the next action based on the state of the current node, using a specific exploration strategy and reward design.

[0046] Optionally, if the page traversal algorithm is a breadth-first search algorithm, the first page relationship graph includes nodes A, B, and C, where there is a directed edge between nodes A and B, and between nodes A and C. Node C is a node added in the previous update. If the operable controls in the pages corresponding to nodes B and C have not been traversed, then page b corresponding to node B can be used as the first page. The operable controls in page b and their corresponding second pages after operation can be added to the first page relationship graph, resulting in an updated page relationship graph. After traversing page b, page c corresponding to node C can be used as the first page, and the operable controls in page c and their corresponding second pages after operation can be added to the previously updated relationship page graph. After page c has been traversed, since new nodes have been added, based on the updated relationship page graph, the pages corresponding to the expanded nodes are sequentially used as the first pages according to the breadth-first search algorithm, and so on, until all operable controls of all pages have been traversed, resulting in a complete page relationship graph.

[0047] Alternatively, for the breadth-first search algorithm, a queue of pages to be visited can be maintained, prioritizing the exploration of pages closer to the starting page. The breadth-first search algorithm is suitable for scenarios requiring fast and broad coverage of the software's shallow core functionalities.

[0048] Optionally, if the page traversal algorithm is a depth-first traversal algorithm, and if the first page relationship graph includes node A and node B, where node B is a node added in the previous update, and the operable controls in page b corresponding to node B have not yet been traversed, then page b is taken as the first page, an operable control in page b is operated, and the user jumps to page c. The node corresponding to page c is added to the first page relationship graph, and then page c is taken as the first page. An operable control in page c is operated, and the user jumps to page d. If the path reaches the end point, the user backtracks to the next operable control in page c after the operation, and explores along the depth direction until the end point of the path is reached. The user backtracks again in the above manner, and so on, until all operable controls of all pages are traversed, and a complete page relationship graph is obtained.

[0049] Alternatively, for depth-first traversal algorithms, a stack of pages to be visited can be maintained, and a path can be explored first until the end before backtracking. Depth-first traversal algorithms are suitable for scenarios that require in-depth testing of specific functions.

[0050] Optionally, for depth-first traversal algorithms, it is also possible to support setting an upper limit on the traversal depth to prevent getting stuck in an infinitely deep path for too long.

[0051] Optionally, if the page traversal algorithm is a reinforcement learning algorithm, the traversal process is modeled as a reinforcement learning problem. The agent's state is the state of the node that is the current node in the first page relationship graph. The selectable actions are the operable controls on the page corresponding to the current node. The reward can be set according to whether a new page or new control has been discovered on the page corresponding to the current node. Through training, the agent can learn the optimal traversal strategy, effectively determine the next action based on the current node's state, and execute the action to jump to the first page, which can effectively improve traversal efficiency. Optionally, during testing, traversing using a reinforcement learning algorithm can efficiently improve coverage, and the traversal behavior is more purposeful and adaptive, intelligently prioritizing the exploration of more valuable paths, which helps improve testing efficiency.

[0052] The mobile interface is a native interface for mobile devices. This interface can include accessibility service interfaces and page screenshot service interfaces, allowing you to call these native mobile interfaces to obtain the page's accessibility tree and page images.

[0053] In the embodiments of this disclosure, based on the first page relationship graph, the first page is determined by a preset page traversal algorithm, and the accessibility tree and page image of the page are obtained by calling the interface of the mobile terminal. Then, based on the accessibility tree and page image, the set of operable controls of the page, the updated page relationship graph, etc. are obtained, thereby realizing automated page traversal and page data collection and improving the comprehensiveness of page data.

[0054] In some alternative embodiments, based on any of the above embodiments, at least one processor 21 is specifically configured to, or based on a first accessibility tree and a first page image, determine a set of operable controls for a first page, including:

[0055] Based on the attribute information of the nodes in the first accessibility tree, a first set of controls is determined; based on the first page image, a second set of controls is determined; based on the first set of controls and the second set of controls, an operable set of controls is determined.

[0056] The first accessibility tree comprises multiple nodes, each corresponding to a control. The attribute information of each node includes one or more of the following: the control's unique identifier (or control identifier), the control's text content (such as text on a button), whether the control is clickable, whether the control is long-clickable, the control's coordinates and size, the control's class name, and the application to which the control belongs. Based on the attribute information of each node in the first accessibility tree, the first set of controls can be determined.

[0057] In some optional embodiments, clickable nodes can be determined based on the clickability attribute of each node in the first accessibility tree, and a first control set can be obtained based on the clickable nodes and their attribute information. The first control set may include operable controls declared in the first accessibility tree and their attribute information.

[0058] In some optional embodiments, a second set of controls on the first page can be identified based on the first page image using a pre-trained control recognition model. Optionally, the second set of controls can be obtained by identifying operable controls on the first page. The second set of controls includes operable controls identified based on the first page image and their attribute information. The control recognition model can be implemented using any real-time neural network model. Optionally, the control recognition model can be, but is not limited to, a multimodal large language model with visual understanding capabilities. Multimodal large language models include, but are not limited to, Gemini, GPT-4o (Generative Pre-trained Transformer 4Omni), etc. The multimodal large language model can identify visually interactive regions, such as shadows, borders, and preset icon styles, by using pre-designed prompt text and leveraging powerful visual and common-sense reasoning capabilities. For example, the prompt text may be, but is not limited to, "Find all clickable areas in this image, including buttons, icons, and menu items, and return their bounding box coordinates." Because visual features do not depend on the declarations in the accessibility tree of controls and are not constrained by the accessibility tree, controls not declared in the accessibility tree can be identified. Optionally, the control recognition model can incorporate an Optical Character Recognition (OCR) sub-model to recognize and understand text within the control area.

[0059] Optionally, the second set of controls may include non-standard operable controls. Alternatively, the second set of controls may include both non-standard and standard operable controls. Non-standard operable controls are controls that are not declared in the accessibility tree.

[0060] In some optional embodiments, after determining the first control set and the second control set, the operable control set of the first page can be obtained by merging the first control set and the second control set. The merging method may include, but is not limited to, deduplication and merging.

[0061] In the embodiments of this disclosure, standard operable controls declared in the accessibility tree are obtained by performing UI structure analysis on the node attribute information of the accessibility tree of the first page. Non-standard operable controls can be obtained by performing visual analysis on the page screenshot of the first page. The operable control set of the first page is determined by combining the first control set and the second control set. On the basis of obtaining standard operable controls, non-standard operable controls can be obtained, which improves the comprehensiveness of operable controls. The obtained operable control set is used for testing, which helps to improve test coverage.

[0062] In some alternative embodiments, at least one processor 21 is specifically configured to, or based on the first page image, determine a second set of controls, including:

[0063] Based on the first page image, preset prompt text, and a pre-trained multimodal control recognition model, the region bounding box information, control type, and confidence level of candidate controls in the first page image are determined; based on the region bounding box information, control type, and confidence level of the candidate controls, the second control set is determined.

[0064] The preset prompt text is designed in advance based on the control recognition requirements; its specific content will not be elaborated upon. The multimodal control recognition model can be any large language model with multimodal data processing capabilities. The first page image and the preset prompt text are input into the multimodal control recognition model. The model performs inference and outputs control information that may be operable controls in the first page image, serving as candidate controls. The control information output by the model may include, but is not limited to, the region bounding box information, control type, and confidence level of the candidate controls. The region bounding box information is a descriptive information of the region bounding box of the candidate control in the first page image. This information may include, but is not limited to, one or more of the following: center point, size, corner coordinates, etc. The control type is the classification type of the control, which may include, but is not limited to, buttons, text input boxes, drop-down menus, radio buttons, checkboxes, etc., specifically set according to the actual needs of the page. The confidence level is the degree of trustworthiness of the candidate control. Based on the control information output by the model, a second set of controls for recognition is determined. For example, candidate controls can be filtered based on confidence level, selecting those with higher confidence (e.g., confidence level greater than a preset threshold) as the recognized operable controls. The second set of controls may include information such as the bounding box of each operable control, the control type, and the confidence level.

[0065] In the embodiments of this disclosure, a multimodal control recognition model is used to reason about and understand the first page image and preset prompt text to determine the second set of controls included in the first page image, which can improve the comprehensiveness and accuracy of the visual recognition results.

[0066] In some alternative embodiments, at least one processor 21 is specifically configured to determine an operable set of controls, either based on a first set of controls or a second set of controls, including:

[0067] The first position information of the first control in the first control set on the first page is matched with the second position information of the second control in the second control set on the first page to obtain the matching result; based on the matching result, the first control set and the second control set are deduplicated and merged to obtain the operable control set.

[0068] Specifically, for any first control in the first control set, the first position information of the first control on the first page may include, but is not limited to, the coordinates and size of the first control on the first page. For any second control in the second control set, the second position information of the second control on the first page may include, but is not limited to, the coordinates and size of the second control on the first page. The second position information is obtained based on the position information of the second control in the image of the first page.

[0069] The first position information of each first control in the first control set can be matched with the second position information of each second control in the second control set to obtain the matching result for each first control and each second control. The matching result can include whether the first control and the second control match or not, or it can include the similarity or overlap between the first control and the second control. Based on the matching result, the first control set and the second control set are deduplicated and merged to obtain a comprehensive and accurate set of operable controls. Deduplication refers to removing operable controls that are duplicated in the second control set and the first control set to ensure the uniqueness of the controls. For example, if control k1 in the first control set and control k2 in the second control set belong to the same control, then only one copy of the control identifier and other attributes of that control needs to be retained. This can be done by selecting the best attribute through preset rules, or by averaging or weighting according to other weight ratios to obtain one set of attributes. Merging can include merging non-duplicate first controls and second controls.

[0070] In the embodiments of this disclosure, the first control set and the second control set are deduplicated and merged by position matching to obtain a comprehensive and accurate set of operable controls, thereby improving the comprehensiveness and effectiveness of operable controls on the page and providing comprehensive control information for page testing.

[0071] In some optional embodiments, at least one processor 21 is specifically configured to, or based on the matching result, deduplicate and merge the first control set and the second control set to obtain an operable control set, including:

[0072] For any first control in the first control set, in response to the existence of a target second control in the second control set that matches the first control, the target second control and the first control are determined as a single target control, and the control identifier of the first control is determined as the control identifier of the merged target control; or, in response to the existence of no second control in the second control set that matches the first control, preset marking information is added to the first control to obtain a marked control; and / or, based on a second control in the second control set that does not match a first control in the first control set, the second control is determined as a new control; based on the target control, the marked control, and / or the new control, an operable control set is determined.

[0073] Specifically, for any first control in the first control set, it can be determined whether there is a target second control in the second control set that matches the first control based on the matching result.

[0074] If a target second control exists in the second control set that matches the first control, it means that the target second control and the first control belong to the same control (referred to as the target control). The standard control identifier of the first control is then determined as the control identifier of the target control, ensuring the uniqueness of the control identifier. Optionally, the attributes of the target second control and the attributes of the first control can be retained selectively or after weighting. For example, the attributes of the first control can be prioritized. If the target second control has an attribute that the first control does not have, then that attribute of the target second control can be retained. For example, if the attribute of the target second control includes confidence level, while the attribute of the first control does not include confidence level, then confidence level is retained.

[0075] If a second control that matches the first control does not exist in the second control set, it indicates that the first control may be visually invisible, for example, if the first control is obscured by other views. In this case, a preset marker is added to the first control to obtain a marked control. The preset marker is used to indicate the invisibility of the first control. Optionally, the preset marker may include, but is not limited to, one of the following: invalid, invisible, etc.

[0076] If there is no matching first control in the first control set for the second control, it means that the second control is not declared in the first accessibility tree, and the second control is identified as a new control.

[0077] Based on the above processing method, the number of deduplicated target controls can be 0, one, or more; the number of labeled controls can be 0, one, or more; and the number of new controls can be 0, one, or more. The resulting target controls, labeled controls, and / or new controls are merged to obtain a set of operable controls. Optionally, labeled controls can be considered invalid, in which case the resulting target controls and new controls are merged to obtain a set of operable controls.

[0078] In the embodiments of this disclosure, based on the matching results of the first control and the second control, the target control after deduplication and merging, invisible controls, and new controls not declared in the accessibility tree can be effectively determined, thereby effectively improving the comprehensiveness, accuracy, and effectiveness of the operable control set.

[0079] In some optional embodiments, based on any of the above embodiments, at least one processor 21 is specifically configured to, or based on a set of operable controls, determine a second page corresponding to an operable control after an operation, including:

[0080] Take any operable control from the set of operable controls as the current control, and control the mobile device to perform the operation corresponding to the current control, so that the mobile device jumps from the first page to the second page after the operation corresponding to the current control.

[0081] The current control is determined based on a page traversal algorithm. If the page traversal algorithm is a breadth-first search algorithm, each operable control is traversed in the order of the operable controls in the operable control set. The operable control encountered is taken as the current control, and the mobile device is controlled to perform the operation corresponding to the current control. This allows the mobile device to jump from the first page to the second page after the operation corresponding to the current control. Then, based on the first page, the current control, and the second page, the relationship graph of the first page is incrementally updated to obtain the updated relationship graph of the second page. After traversing all the operable controls in the operable control set of the first page, the updated page relationship graph is obtained. Based on the updated page relationship graph, the first page is re-determined according to the page traversal algorithm. The above data collection process is iteratively executed to complete the construction of the overall page relationship graph and the collection of the accessibility tree, page image, and operable control set of all pages.

[0082] When the page traversal algorithm is depth-first search, if the first page is the first page visited, after obtaining the set of operable controls for the first page, one operable control in the set is selected as the current control. For example, the first operable control in the set is selected as the current control, and an operation is performed on the current control. The process then jumps to the second page, completing the update of the first page's relationship graph and storing the first page's first accessibility tree, first page image, and set of operable controls. Then, according to the depth-first principle, the second page can be identified as the new first page, and the above data collection process is iteratively executed. If the first page is a page backtracked to using the depth-first traversal algorithm, an operable control is selected from the operable controls in the first page's set that have not yet been visited as the current control, and the above data collection process is iteratively executed. This process continues until the complete page relationship graph is constructed and the accessibility trees, page images, and sets of operable controls for all pages are collected. Optionally, if the first page is a page backtracked to using a depth-first traversal algorithm, then the first accessibility tree, the first page image, and the set of operable controls for the first page have already been obtained and stored during the first traversal. They can be retrieved directly from the corresponding storage area without having to repeat the operation of determining the set of operable controls based on the first accessibility tree and the first page image.

[0083] When the page traversal algorithm is a reinforcement learning algorithm, based on the state of the node corresponding to the first page in the first page relationship graph, the next action (i.e., the operable control) is selected through the reinforcement learning algorithm as the current control. The operation on the current control is executed to jump to the second page. Data is collected according to the above data collection process until the construction of the complete page relationship graph and the collection of accessibility trees, page images and operable control sets of all pages are completed.

[0084] Optionally, the operation corresponding to the current control can be determined from the control information of the current control. For example, if the current control is clickable, the operation on the current control is a click operation; if the current control is long-pressable, the operation on the current control is a long-press operation. Optionally, for controls that require user input, preset content can be entered to trigger the operation of the current control.

[0085] In some alternative implementations, accessibility APIs can be preferentially used to perform operations on controls. Accessibility APIs include, but are not limited to, `AccessibilityNodeInfo.performAction()`. `AccessibilityNodeInfo` represents a data object containing rich information about a control on the page. `performAction()` indicates that a specified operation is performed on the control represented by `AccessibilityNodeInfo`. Optionally, for controls that can only be identified through visual analysis, or controls identified through accessibility APIs, the Android Debug Bridge (ADB) shell commands can be used to simulate click or swipe events at specified coordinates to trigger operations on the control. For text input fields, preset text (such as username or password) or randomly generated test data can be entered.

[0086] In the embodiments of this disclosure, by simulating user operation of operable controls, the mobile terminal is controlled to perform the operation corresponding to the current control, thereby realizing the jump of the mobile terminal page and providing effective technical support for the automated collection of page data.

[0087] In some alternative embodiments, at least one processor 21 is further configured to:

[0088] After obtaining the first accessibility tree and the first page image of the first page on the mobile device, the first fingerprint information of the first page is generated based on the first accessibility tree, and / or the second fingerprint information of the first page is generated based on the first page image; the first page fingerprint information of the first page is determined based on the first fingerprint information and / or the second fingerprint information.

[0089] The first fingerprint information is generated based on the first accessibility tree and can identify the first page. The second fingerprint information is generated based on the image of the first page and can identify the first page.

[0090] In some optional embodiments, a hash value can be calculated based on the first accessibility tree using a preset hash algorithm, and the hash value can be used as the first fingerprint information. Optionally, the first accessibility tree can be normalized to obtain a processed accessibility tree, and a hash value can be calculated on the processed accessibility tree to obtain the first fingerprint information. The normalization process can be implemented based on preset normalization rules, and the normalization process includes, but is not limited to, removing dynamic text and sorting child nodes.

[0091] In some optional embodiments, a hash value can be calculated based on the first page image using a preset perceptual hash algorithm, and the hash value can be used as the second fingerprint information. Perceptual hash algorithms include, but are not limited to, image hash algorithms such as pHash (perceptual hash algorithm), aHash (average hash algorithm), and dHash (differential hash algorithm).

[0092] In some optional embodiments, after obtaining the first accessibility tree and the first page image of the first page on the mobile device, the first fingerprint information of the first page is generated based on the first accessibility tree; and the first page fingerprint information is determined based on the first fingerprint information. For example, the first fingerprint information is determined as the first page fingerprint information.

[0093] In some optional embodiments, after obtaining the first accessibility tree and the first page image of the first page on the mobile device, a second fingerprint information of the first page is generated based on the first page image; and the first page fingerprint information is determined based on the second fingerprint information. For example, the second fingerprint information is determined as the first page fingerprint information.

[0094] In some optional embodiments, after obtaining the first accessibility tree and the first page image of the first page on the mobile device, a first fingerprint information of the first page is generated based on the first accessibility tree; a second fingerprint information of the first page is generated based on the first page image; and a first page fingerprint information of the first page is determined based on the first fingerprint information and the second fingerprint information. Optionally, the first fingerprint information and the second fingerprint information can be determined as the first page fingerprint information, that is, the first page fingerprint information includes the first fingerprint information and the second fingerprint information.

[0095] At least one processor is specifically configured to determine, based on the accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph, page data for the mobile device, including:

[0096] Based on the first accessibility tree of the first page, the first page image, the first page fingerprint information, the set of operable controls, and the second page relationship graph, the page data of the mobile terminal is determined.

[0097] The first page fingerprint information can be used to uniquely identify and distinguish each page, serving as a type of information within the page data. The page fingerprint information of each page, along with the page accessibility tree, page image, set of operable controls, and page relationship graph, together constitute the page data for the mobile device.

[0098] In the embodiments of this disclosure, page fingerprint information is generated using the page's accessibility tree and / or page image to uniquely identify and distinguish the page, ensuring its uniqueness. Furthermore, determining a composite page fingerprint using both first and second fingerprint information further enhances the robustness of the page fingerprint.

[0099] In some alternative embodiments, at least one processor 21 is specifically configured to, or based on a first accessibility tree, generate first fingerprint information of a first page, including:

[0100] The first accessibility tree is preprocessed to obtain the preprocessed second accessibility tree; based on the second accessibility tree, the first hash value is determined and used as the first fingerprint information.

[0101] The first accessibility tree can be preprocessed based on preset processing rules. Preprocessing includes, but is not limited to, removing dynamic text and sorting child nodes. The purpose of preprocessing is to normalize the accessibility tree. Based on the second accessibility tree, a first hash value is calculated using a preset hash algorithm, and this first hash value is used as the first fingerprint information. The hash algorithm is, for example, but not limited to, cryptographic hash algorithms such as SHA-256 (Secure Hash Algorithm 256-bit). The SHA-256 algorithm converts an input of arbitrary length into a fixed-length (256 bits, i.e., 32 bytes) hash value.

[0102] In the embodiments of this disclosure, a first hash value is generated based on a preprocessed second accessibility tree after preprocessing of the first accessibility tree, serving as the first fingerprint information, thereby ensuring the accuracy and validity of the first fingerprint information.

[0103] In some optional embodiments, at least one processor 21 is specifically configured to generate, or based on the first page image, a second fingerprint of the first page, including:

[0104] Based on a pre-configured image hashing algorithm, a second hash value of the first page image is determined, and the second hash value is used as the second fingerprint information.

[0105] Image hashing algorithms include, but are not limited to, pHash (perceptual hashing), aHash (average hashing), and dHash (differential hashing). aHash generates fingerprint information by calculating the average pixel value of an image, for example, by comparing the difference between each pixel and the average value. pHash extracts image frequency features using discrete cosine transform, retaining low-frequency information to generate a hash value. dHash generates fingerprint information based on the grayscale differences between adjacent pixels in an image, for example, by comparing the differences between horizontally adjacent pixels. Based on the image hashing algorithm, a hash value is calculated for the first page image to obtain a second hash value, which is then used as the second fingerprint information.

[0106] In the embodiments of this disclosure, a hash value of the page image is generated based on an image hashing algorithm, which serves as fingerprint information and provides a valid reference for the generation of page fingerprint information.

[0107] Figure 3 This is a schematic diagram of the structure of a data collection device for a mobile terminal page provided in another exemplary embodiment of this disclosure.

[0108] In some alternative embodiments, based on any of the above embodiments, such as Figure 3 As shown, the apparatus 20 of this disclosure further includes: a memory 22 configured to store acquired page data; the page data includes page fingerprint information of at least one acquired page.

[0109] At least one processor 21 is specifically configured to, or based on the first page, operable controls, and the second page, update the relationship graph of the first page to obtain the relationship graph of the second page, including:

[0110] Determine the fingerprint information of the second page; based on the fingerprint information of the second page and the page fingerprint information of the collected pages, determine the first matching result between the second page and the collected pages; based on the first page, operable controls, the second page and the first matching result, update the relationship graph of the first page to obtain the relationship graph of the second page.

[0111] The fingerprint information of the second page can be determined based on the accessibility tree and / or page image of the second page. The specific operation is similar to that of the fingerprint information of the first page and will not be elaborated here. The page fingerprint information of the collected page is the page fingerprint information of the page obtained before the processing flow of the first page. The fingerprint information of the second page is matched with the page fingerprint information of the collected page to obtain the first matching result. The first matching result can include the matching status of the second page and the collected page, which includes two cases: matching and not matching. Alternatively, the first matching result includes information indicating that there is no page in the collected pages that matches the second page. Alternatively, the first matching result includes the page fingerprint information of the collected page that matches the second page. The first matching result indicates whether the second page is a collected page. Therefore, based on the first matching result, the update method of the first page relationship graph can be determined. For example, if the second page is a collected page, it means that the first page relationship graph already includes the node corresponding to the second page. That is to say, the second page jumped by the current control operation has created a loop. In this case, it is not necessary to add the node corresponding to the second page in the first page relationship graph; a loop edge between the first page and the second page can be added. If the second page is not an already collected page, it means that the second page is a new page, and there is no corresponding node for the second page in the first page relationship graph. Therefore, it is necessary to add the corresponding node for the second page to the first page relationship graph and establish a directed edge between the first and second pages. This updates the page relationship graph, resulting in the second page relationship graph.

[0112] In the embodiments of this disclosure, the page relationship graph is updated based on the matching results by matching the page fingerprint information of the second page with the page fingerprint information of the collected pages, thereby ensuring the accuracy of the page relationship graph.

[0113] In some optional embodiments, the first page relationship graph includes the operational flow relationships between the collected pages; at least one processor 21 is specifically configured to, based on the first page, operable controls, the second page, and the first matching result, incrementally update the first page relationship graph to obtain a second page relationship graph, including:

[0114] In response to the first matching result indicating that the second page is the target page among the collected pages, a loop edge between the first page and the target page is added to the first page relationship graph; based on the operable controls, the operation attributes of the loop edge are determined to obtain the second page relationship graph; or, in response to the first matching result indicating that the second page is not a collected page, the node corresponding to the second page and the target edge between the first page and the second page are added to the first page relationship graph; based on the operable controls, the operation attributes of the target edge are determined to obtain the second page relationship graph.

[0115] If the first matching result indicates that the second page is a page among the collected pages (referred to as the target page), it means that the first page relationship graph already includes the node corresponding to the second page. That is, the node corresponding to the target page is also the node corresponding to the second page. It also means that operating the operable control on the first page redirects back to the collected page. This situation is called a loop. Therefore, a loop edge between the first page and the target page is added to the first page relationship graph, that is, a loop edge is added between the node corresponding to the first page and the node corresponding to the target page. Based on the operable control that caused the loop, the operation attribute of the loop edge is determined. The operation attribute is the operation description information of the operable control; the operation description information is described above and will not be repeated here.

[0116] If the first matching result indicates that the second page is not a collected page, it means that the second page is a new page. Since there is no node corresponding to the second page in the first page relationship graph, a node corresponding to the second page can be added to the first page relationship graph, and a directed edge (called the target edge) can be established between the node on the first page and the node on the second page. Based on the operable controls that cause the user to jump from the first page to the second page, the operation attributes of the target edge are determined, thus obtaining the second page relationship graph.

[0117] In the embodiments of this disclosure, the second page can be determined as a collected page based on the first matching result between the second page and the collected page, thus realizing loop closure detection of the page. The page relationship graph is updated according to different situations of whether the second page is a collected page, avoiding the situation where the same page corresponds to two nodes in the page relationship graph, and ensuring the accuracy and effectiveness of the page relationship graph.

[0118] In some optional embodiments, at least one processor 21 is specifically configured to determine a first matching result between the second page and the collected page, based on the second page fingerprint information and the page fingerprint information of the collected page, including:

[0119] The second page fingerprint information is matched with the page fingerprint information of the collected page to determine a second matching result; in response to the second matching result indicating that there is no page fingerprint information in the page fingerprint information of the collected page that matches the second page fingerprint information, the tree edit distance between the second page and the collected page is determined based on the accessibility tree of the second page and the accessibility tree of the collected page; the visual similarity between the second page and the collected page is determined based on the second page image of the second page and the page image of the collected page; and the first matching result between the second page and the collected page is determined based on the tree edit distance and / or visual similarity between the second page and the collected page.

[0120] Specifically, for any collected page, the fingerprint information of the second page can be compared with the page fingerprint information of the collected page to obtain a second matching result. The second matching result includes whether the second page matches or does not match the collected page. If the fingerprint information of the second page matches the page fingerprint information of the collected page, the second matching result is that the second page matches the collected page. If the fingerprint information of the second page does not match the page fingerprint information of the collected page, the second matching result is that the second page does not match the collected page.

[0121] If the second matching result indicates that there is no matching page fingerprint information among the page fingerprint information of the collected pages, a fine-grained matching can be further performed based on the accessibility tree of the second page, the second page image, and the accessibility tree and page image of the collected pages. The fine-grained matching result is then determined as the first matching result. Specifically, for any collected page, the tree edit distance between the second page and the collected page can be determined based on the accessibility tree of the second page and the page image of the collected page; the visual similarity between the second page and the collected page can be determined based on the tree edit distance and / or visual similarity between the second page and the collected page; and the first matching result between the second page and the collected page can be determined based on the tree edit distance and / or visual similarity between the second page and the collected page. By traversing all collected pages, the first matching result between the second page and each collected page can be obtained.

[0122] In some optional embodiments, the tree edit distance between the accessibility tree of the second page and the accessibility tree of the already captured page can be determined based on a preset tree edit distance algorithm. Tree edit distance algorithms include, but are not limited to, the Zhang-Shasha algorithm and the APTED (Approximate Tree Edit Distance) algorithm. Tree edit distance characterizes the degree of difference between two accessibility trees. The Zhang-Shasha algorithm calculates the edit distance between two trees using dynamic programming. The APTED algorithm accelerates the calculation of tree edit distance by decomposing the tree structure into subproblems and combining heuristic pruning and approximation strategies.

[0123] In some optional embodiments, the visual similarity between the second page image and the captured page image can be determined based on a deep learning-based image similarity model. Deep learning-based image similarity models include, but are not limited to, Learned Perceptual Image Patch Similarity (LPIPS) models. These models can better simulate human visual perception, ignoring minute pixel-level differences, such as ad refreshes, and focusing on the similarity of the overall page layout and content, thus obtaining more robust similarity scores. The Learned Perceptual Image Patch Similarity (LPIPS) model is a deep learning-based image similarity metric that extracts image features through a pre-trained deep neural network model and calculates the Euclidean distance between feature patches to quantify the perceptual differences between two images.

[0124] The first matching result between the second page and the collected page is determined based on at least one of the tree edit distance and visual similarity between the second page and the collected page.

[0125] In some optional embodiments, at least one processor 21 is specifically configured to: in response to a second matching result indicating that no page fingerprint information matching the second page fingerprint information exists in the page fingerprint information of the collected page, determine the tree edit distance between the second page and the collected page based on the accessibility tree of the second page and the accessibility tree of the collected page; and determine a first matching result between the second page and the collected page based on the tree edit distance between the second page and the collected page. Optionally, the first matching result between the second page and the collected page can be determined based on the tree edit distance and a preset distance threshold. For example, if the tree edit distance is less than the preset distance threshold, it is determined that the second page and the collected page are structurally similar, indicating that the second page is not a new page.

[0126] In some optional embodiments, at least one processor 21 is specifically configured to: in response to a second matching result indicating that there is no page fingerprint information matching the second page fingerprint information in the page fingerprint information of the acquired page, determine the visual similarity between the second page and the acquired page based on the second page image of the second page and the page image of the acquired page, wherein the visual similarity represents the visual difference between the second page and the acquired page; and determine a first matching result between the second page and the acquired page based on the visual similarity between the second page and the acquired page. Optionally, the first matching result between the second page and the acquired page can be determined based on the visual similarity and a preset similarity threshold. For example, if the visual similarity is less than the preset similarity threshold, it indicates that the second page and the acquired page are visually similar, and that the second page is not a new page.

[0127] In some optional embodiments, at least one processor 21 is specifically configured to: in response to a second matching result indicating that no page fingerprint information matching the second page fingerprint information exists in the page fingerprint information of the collected page, determine the tree edit distance between the second page and the collected page based on the accessibility tree of the second page and the accessibility tree of the collected page; determine the visual similarity between the second page and the collected page based on the second page image of the second page and the page image of the collected page; and determine a first matching result between the second page and the collected page based on the tree edit distance and the visual similarity. Optionally, the tree edit distance and the visual similarity can be weighted to obtain a comprehensive similarity, and the first matching result between the second page and the collected page can be determined based on the comprehensive similarity. For example, if the comprehensive similarity is less than a preset similarity threshold, it indicates that the second page and the collected page are visually similar, and that the second page is not a new page.

[0128] In the embodiments of this disclosure, if it is determined that the second page is not a collected page based on the matching result of the fingerprint information of the second page and the page fingerprint information of the collected page, a fine matching is performed on the second page and the collected page by further comparing the accessibility tree of the second page with the accessibility tree of the collected page, and / or the second page image of the second page with the page image of the collected page. This can ignore minor differences between pages and focus on the similarity of the overall layout and content of the page, ensuring the accuracy and robustness of the judgment result of whether the second page is a new page.

[0129] In some alternative embodiments, based on any of the above embodiments, at least one processor 21 is further configured to:

[0130] Based on the expected page after the operation corresponding to the second page and the operable controls, determine the test results of the operable controls; based on the test results of the operable controls, determine the test results of the mobile terminal.

[0131] The expected page is obtained in advance based on the page design. The second page represents the actual page displayed after the mobile device controls are interacted with, while the expected page represents the page anticipated during the design phase. The test results of the operable controls can be determined by comparing the second page with the expected page. The test results can include the accuracy of the operable controls' functionality.

[0132] Optionally, during the testing process, the operable controls on all pages to be tested are traversed according to the page and operable control traversal method of the aforementioned embodiments. Based on the second page after the operation corresponding to each operable control and the expected page, the test results of each operable control are obtained. Based on the test results of each operable control, the test results of the operating system and / or application to be tested on the mobile terminal are obtained.

[0133] Optionally, after obtaining the test results, the operating system and / or application can be updated based on the test results to obtain an updated version. Then, page data can be collected and tested according to the above-mentioned page and control traversal method, effectively improving test efficiency and coverage.

[0134] In the embodiments of this disclosure, the set of operable controls determined based on the accessibility tree and page image is used for testing mobile devices, which can effectively test the functionality of controls not declared in the accessibility tree and effectively improve test coverage.

[0135] In some optional embodiments, at least one processor 21 is further configured to determine page data of the mobile device, based on a first accessibility tree of the first page, a first page image, a set of operable controls, and a second page relationship graph, including:

[0136] In response to the first page being the last page traversed, the mobile page data is obtained based on the first accessibility tree of the first page, the first page image, the first page's set of operable controls, the previous accessibility tree, the previous page image, and the previous set of operable controls of the previously traversed pages, and the second page relationship graph.

[0137] Specifically, whether the first page is the last page can be determined based on the second page following the actions taken on the operable controls on the first page. For example, if the first page is the last page navigated to after an action taken on an operable control among all the pages already traversed, and there are no operable controls on the first page, or if the second pages following actions on the operable controls on the first page are all loop pages, indicating that there will be no new pages, then the first page is the last page. Optionally, other methods can be used to determine whether the first page is the last page traversed. If the first page is the last page traversed, it means that the pages for which data needs to be collected have been traversed, and the second page relationship graph is a complete page relationship graph. The accessibility tree, page images, and operable control sets of all traversed pages, along with the second page relationship graph, constitute comprehensive and rich page data for the mobile device.

[0138] In the embodiments of this disclosure, by performing corresponding operations on the operable controls in the mobile terminal page, the mobile terminal page is automatically traversed, and various related data of the page are collected and a complete page relationship graph is constructed, thereby improving the efficiency of page data collection.

[0139] In some optional application scenarios, taking a preset e-commerce application as an example, the system can control a mobile device to run the preset e-commerce application, traverse the pages of the preset e-commerce application, and construct a page relationship graph. Specifically: 1. On the starting page (i.e., the homepage of the preset application), the starting page is taken as the first page. Based on the first accessibility tree and the first page image, the set of operable controls on the starting page is determined, such as "Home", "Categories", "Shopping Cart", search box, product list, etc. 2. Traversal execution: Taking breadth-first traversal as an example, the "Categories" navigation button is selected first. 3. New page: Enter the product category page, calculate the page fingerprint information of the product category page, determine the product category page as the new page based on the page fingerprint information, add it as a new node to the page relationship graph, and add an edge from "Home" to "Categories Page", with the edge attribute marked as "Clicking the Category Button". The page relationship graph can be stored in a database. 4. In-depth exploration: On the category page, it can be determined that the set of operable controls includes multiple product categories, such as "Electronics", "Daily Necessities", "Clothing", etc. Taking "Electronic Products" as an example, clicking on "Electronic Products" leads to the category list page, and the new page is recorded. 5. Visual Recognition: On a product list page, a "New Product Launch" activity entry is a pure image design, not declared in the accessibility tree. This control can be recognized based on a multimodal control recognition model. Clicking on this control successfully leads to the activity page, revealing a hidden path, and adding the activity page to the page relationship graph.

[0140] In some optional application scenarios, taking a preset game application as an example, it is possible to control a mobile device to run the preset game application, traverse the pages of the preset game application, and construct a page relationship graph. Specifically: 1. Start Page: The main menu interface of the game is rendered by a rendering engine (such as Unity). The entire interface is a separate view from the operating system, and the accessibility tree information is empty. 2. Visual Recognition: By analyzing the screenshot of the main menu page through a multimodal control recognition model, the three visually similar button areas of the operable controls "Start Game," "Settings," and "Leaderboard" can be identified, and their text content and coordinates can be extracted. 3. Action Generation: Since there is no accessibility tree information, the set of operable controls can be determined directly based on the visually recognized operable controls. 4. Traversal Execution: By clicking on each operable control, the corresponding page is entered. Taking the "Settings" control as an example, clicking on "Settings" enters the settings page, adding the settings page to the page relationship graph. The accessibility tree, page screenshot, and set of operable controls for the settings page can be collected.

[0141] In some optional application scenarios, taking the handling of the login process and detection of loops as an example, the processing flow includes: 1. Scenario: The application starts and enters the "My" page. Upon detecting an unlogged-in state, it automatically redirects to the "Login Page." 2. Traversal: On the "Login Page," a preset test account and password are entered, and the login operation is executed. Successfully entering the "My" page, i.e., entering the logged-in state. 3. Loop: On the "My" page, the "Logout" button is recognized and clicked, and the page redirects back to the "Login Page." 4. Detection and Recording: By comparing page fingerprint information, it is identified that the "Login Page" and the "Login Page" in step 1 are the same page. Therefore, an edge from the "My" node to the "Login Page" node is added to the page relationship graph, thus accurately recording the "Login-Logout" business loop.

[0142] In some alternative embodiments, at least one processor 21 is further configured to:

[0143] The page images in the mobile page data are labeled to obtain the label information of the page images; the label information includes the correct / incorrect status labels of the page images and / or the correct / incorrect status labels of the controls in the page images; based on the page images and label information, the pre-trained multimodal control recognition model is fine-tuned to obtain the fine-tuned target multimodal control recognition model; based on the label information and the second page relationship graph, the location path of the error page and / or error control is determined; based on the location path and the target multimodal control recognition model, the error page and / or error control in the mobile software corresponding to the mobile page data are retested and stress tested, and a test report is generated.

[0144] The page image's status label indicates whether the page image is correct. The control's status label indicates whether the control is correct. Optionally, the control's status label may include labels indicating whether the control's display is correct and whether the control's text content is correct. Optionally, the label information may also include information about the control area on the page, control type information, etc., and the specific label information can be set according to actual needs. Optionally, but not limited to, manually annotating the page images in the mobile terminal's page data can be used to obtain the page image label information.

[0145] Multimodal control recognition models can be, but are not limited to, multimodal large language models. Multimodal large language models can identify operable controls within page images. By fine-tuning the multimodal control recognition model using page images and their corresponding label information, the model can not only output operable controls but also identify the correct / incorrect state of the page and the correct / incorrect state of the controls.

[0146] In some optional embodiments, retesting refers to repeated testing, while stress testing refers to testing the object under test many times in a short period of time, such as testing a page or control 10,000 times in a short period of time to determine the stress test results. The location path of the faulty page and / or faulty control can be determined based on the tag information of the page image and the second page relationship graph. The location path is the path of the faulty page and / or faulty control in the page relationship graph. Based on the location path, the faulty page and / or faulty control can be located. Then, based on the fine-tuned target multimodal control recognition model, retesting and stress testing are performed on the faulty page and / or faulty control in the mobile software, and a test report is generated based on the test results. Optionally, the test report can be generated according to a preset test report format. For example, the test report can include test result areas corresponding to multiple applications. The test result area corresponding to each application records the test results of various controls of that application, and can record information on whether each path passed or failed during the test. The specific content of the test report can be set according to the test requirements.

[0147] In the embodiments of this disclosure, after collecting page data from the mobile device, the page images in the page data are labeled. Based on the labeled page images and tag information, the multimodal control recognition model is fine-tuned, enabling the model to identify the correct and incorrect states of the page and controls, facilitating the rapid discovery of erroneous pages and controls. Furthermore, by retesting and stress-testing erroneous pages and / or controls, test reports can be automatically generated, effectively improving testing efficiency.

[0148] In some alternative embodiments, based on any of the above embodiments, at least one processor 21 is further configured to:

[0149] Obtain the pre-configured test scope of the mobile software corresponding to the mobile page data; based on the pre-configured test scope, perform control traversal tests on the pages within the test scope in the mobile software and generate a test report.

[0150] The mobile software corresponding to the mobile page data refers to the operating system and / or application capable of displaying the corresponding page during runtime. The test scope represents the range of sub-software to be tested within the mobile software. The test scope is set according to testing requirements. For example, if the mobile software includes an operating system and multiple apps, the user can set the test scope to include the operating system or a specific app within the test scope as a sub-software. Control traversal testing is performed on the pages of the sub-software within the test scope, generating a test report. Control traversal testing refers to traversing the controls on the sub-software's page according to a preset traversal algorithm, performing operations on the traversed controls, obtaining test results based on the correct state of the page and the correct state of the controls after the operations, and generating a test report based on the test results according to a preset test report format.

[0151] Optionally, when the user has identified the sub-software involved in the software update, a test scope can be set based on these sub-software components. Precise testing can then be performed directly on the sub-software within this scope, generating a test report. Test items may include, but are not limited to, retesting and load testing.

[0152] In some optional embodiments, the computationally intensive multimodal control recognition model can be deployed on a cloud server. The device of this disclosure uploads the acquired accessibility tree and page image to the cloud server. Based on the accessibility tree and page image, the cloud server determines the set of operable controls and returns the set of operable controls to the device of this disclosure. The device of this disclosure then performs subsequent operations, thereby achieving automated collection of page data through cloud-based collaborative analysis. This reduces the local computing performance requirements of electronic devices, including the device, and further improves recognition accuracy by leveraging a more powerful and accurate large model deployed in the cloud.

[0153] In some optional embodiments, if the device cannot make a decision on a certain page—for example, if it cannot determine whether the second page is a new page, or if the multimodal control recognition model has low confidence in recognizing a certain area—the traversal can be paused, and a test assistance request can be output so that the relevant user can specify the next operation or indicate whether a certain area is clickable. User input can be further used as training data for the model to further improve its performance. Thus, by combining the efficiency of computers with human intelligence, extreme and difficult scenarios that may be encountered in automated traversal can be solved, and the level of automation can be continuously improved through continuous learning.

[0154] In some alternative embodiments, Figure 4 This is a flowchart illustrating the data acquisition process of a mobile terminal page data acquisition device provided in an exemplary embodiment of this disclosure. For example... Figure 4 As shown, the data acquisition process includes the following steps:

[0155] S310, enter page P_i. Page P_i can be called the first page.

[0156] S320, obtain the accessibility tree and page screenshot of page P_i. The accessibility tree of page P_i is used as the first accessibility tree, and the page screenshot of page P_i is used as the first page image.

[0157] S3310, determine the first control set C_list based on the accessibility tree.

[0158] S3410 determines the region bounding box information, control type, and confidence level of candidate controls in the page screenshot based on the page screenshot, the preset enhanced text, and the pre-trained multimodal control recognition model.

[0159] S3420, determine the second control set V_list based on the region box information, control type and confidence level of the candidate controls.

[0160] S350, the first control set C_list and the second control set V_list are deduplicated and merged to obtain the operable control set A_list.

[0161] S360 determines the current control according to preset traversal rules. Preset traversal rules are also known as page traversal algorithms.

[0162] S370 controls the mobile device to perform operations on the current control.

[0163] On S380, the mobile device navigates to page P_i+1. Page P_i+1 serves as the second page.

[0164] S390: Determine if page P_i+1 is a new page. If it is, execute S400; otherwise, execute S410. This can be determined by obtaining the accessibility tree and a screenshot of page P_i+1.

[0165] S400, update the page relationship graph in the database, add the new node corresponding to page P_i+1, and the target edge between the node of page P_i and the new node of page P_i+1. The page relationship graph in the database corresponds to the first page relationship graph mentioned above.

[0166] S410, Update the page relationship graph in the database, and add a loop edge between the node corresponding to page P_i and the node corresponding to P_i+1 in the graph.

[0167] S420 writes the accessibility tree, page screenshot, and set of operable controls for page P_i into the database.

[0168] For details on the specific operations of each step from S310 to S420, please refer to the aforementioned embodiments, which will not be repeated here.

[0169] Optionally, a first database for storing the page relationship graph and a second database for storing other page data can be set up. The accessibility tree, page screenshot, and set of operable controls for page P_i are then written to the second database. The page relationship graph and other page data can be associated through page identifiers and control identifiers. For example, each node in the page relationship graph can be labeled with corresponding node attributes, which may include, but are not limited to, the page identifier corresponding to that node. Edges between nodes can be labeled with corresponding edge attributes, which may include, but are not limited to, the control identifier, control type, and operation type associated with the two nodes connected by the edge. For example, if a user navigates from page P_i to page P_i+1 using control a, the page relationship graph includes the node Node_i corresponding to page P_i and the node Node_i+1 corresponding to page P_i+1, as well as the edge between node Node_i and node Node_i+1. The attributes of this edge include the control identifier, control type, and operation type of the operable control a.

[0170] Optionally, if the preset traversal rule is breadth-first traversal, the current control is determined to be the control in page P_i. After all operable controls in page P_i have been traversed, the next page is traversed. If the preset traversal rule is depth-first traversal, the current control may or may not be the control in page P_i. For example, after obtaining the set of operable controls in page P_i, the first control traversed is the control in page P_i, such as the first control K1(i) in the set of operable controls in page P_i. After performing the operation of the current control, the page will jump to page P_i+1. According to the depth-first traversal rule, it will continue to traverse an operable control in page P_i+1, such as the first control K1(i+1) in page P_i+1, until the page ends or loops back. Then, the second control K2(1) of the first page P_1 is traversed, and so on, until all pages and controls of the operating system and / or application to be collected on the mobile terminal have been traversed, and the required comprehensive page data is obtained.

[0171] Figure 5 This is a schematic diagram illustrating the principle of establishing a page relationship graph provided by an exemplary embodiment of this disclosure. For example... Figure 5As shown, a simple page transition on a mobile device is used as an example for illustration. The page transition is shown in the upper part of the figure. P_ij represents pages i = 1, 2, 3, j = 1, 2, 3, 4, 5. Kik(s) represents operable controls, k = 1, 2, 3, 5, s = 1, 2, 3. The arrows indicate page jumps after performing operations on the controls at the tail of the arrow. Different controls can jump to different pages after operation. After operating on the control K35(1) in page P_35, the page returns to P_11. Based on the page transition process, the method disclosed in this paper is used to traverse each page and the controls in the page, and the page relationship graph is updated incrementally to obtain the page relationship graph corresponding to the entire transition process, as shown in the lower half of the figure. The circular nodes represent the nodes corresponding to the pages, the node attribute ij is the page identifier, and the edges between the nodes represent the controls that cause the page transition and their related information. For example, the edge between node 11 and node 21 is K11(1), which means that page P_11 will jump to page P_21 by operating the control K11(1). Therefore, a page relationship graph can be used to describe the overall page flow relationship of a mobile operating system and / or application.

[0172] The mobile page data acquisition device provided in this disclosure, by integrating the UI structure and visual information of the accessibility tree, can identify and manipulate all interactive elements, including custom rendering controls. This solves the fundamental defect in related technologies that tools cannot identify custom controls, effectively improving the depth and breadth of UI traversal and enhancing the comprehensiveness of the collected page data. When used for testing, it can effectively improve test coverage. Secondly, it can automatically explore and traverse unknown software without writing test scripts, effectively reducing the manpower and maintenance costs of testing. Thirdly, by combining the first fingerprint information generated by the accessibility tree and the second fingerprint information generated by the page image, it generates the matching fingerprint information of the page. Furthermore, by combining the tree edit distance of the accessibility tree and the visual similarity of the page image, it achieves high-precision page deduplication and loop closure detection, ensuring the validity of the traversal results and the accuracy of the page relationship graph. Finally, this disclosure does not rely on a specific UI framework and has good applicability to native applications, hybrid applications, Flutter applications, and even game applications, solving the pain points of non-standard UI automated testing in the industry. Furthermore, the page relationship graph generated by the traversal process disclosed herein is a dynamically updated and most realistic "living document" of the application, which can be used in various scenarios such as product analysis and user behavior path research.

[0173] The device disclosed herein can be installed in electronic devices such as PCs and servers, serving as a control terminal device. The processor, memory, and other resources of the control terminal device must meet the computational and storage requirements for page data acquisition. The mobile device, as the device under test, can support accessibility services and maintain a stable network connection with the control terminal, for example, via USB, Wi-Fi, etc., for communication. The control terminal's software stack includes an operating system, a core programming language, a deep learning framework, a multimodal model, a mobile automation framework, an image processing library, a database, etc. The control terminal's operating system includes, but is not limited to, Linux, macOS, and Windows. The device under test's software stack includes core services, auxiliary tools, etc. Core services include, for example, accessibility services. Auxiliary tools include, for example, a lightweight auxiliary installation package installed on the device under test, used to more efficiently obtain accessibility tree information or perform specific operations.

[0174] In the technical solutions disclosed herein, if the collection of user personal information is involved, the collection, storage, use, processing, transmission, provision, and disclosure of such user personal information comply with relevant laws and regulations and do not violate public order and good morals. Furthermore, in the technical solutions disclosed herein, the collection and use of user personal information are conducted with the user's knowledge and authorization, and do not involve the illegal collection or use of user personal information.

[0175] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0176] Exemplary methods

[0177] Figure 6 This is a flowchart illustrating a mobile page data collection method provided in an exemplary embodiment of this disclosure. The mobile page data collection method provided in this embodiment can be implemented using any of the mobile page data collection devices provided in the above embodiments of this disclosure, such as... Figure 6 The data collection method for the mobile page shown may include the following steps:

[0178] Step 510: Obtain the first accessibility tree and the first page image of the first page on the mobile device.

[0179] Step 520: Based on the first accessibility tree and the first page image, determine the set of operable controls for the first page.

[0180] The set of operable controls includes the operable controls on the first page.

[0181] Step 530: Based on the set of operable controls, determine the second page after the operation corresponding to the operable control.

[0182] Step 540: Based on the first page, operable controls, and the second page, incrementally update the relationship graph of the first page to obtain the relationship graph of the second page.

[0183] The first page relationship graph is either an initialized graph or it is obtained from the previous incremental update.

[0184] Step 550: Determine the page data for the mobile device based on the first accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph.

[0185] Figure 7 This is a flowchart illustrating a data collection method for a mobile webpage provided in another exemplary embodiment of this disclosure.

[0186] In some optional embodiments, based on any of the above embodiments, the first page relationship graph includes the operation flow relationships between the collected pages. For example... Figure 7 As shown, step 510, obtaining the first accessibility tree and the first page image of the mobile device's first page, may include:

[0187] Step 5110: Based on the first page relationship graph, the first page is determined using a pre-configured page traversal algorithm.

[0188] The page traversal algorithm can be any one of breadth-first search, depth-first search, or reinforcement learning.

[0189] Step 5120: Call the mobile terminal's interface to obtain the first accessibility tree and the first page image.

[0190] Figure 8 This is a flowchart illustrating a data collection method for a mobile webpage provided in another exemplary embodiment of this disclosure.

[0191] In some alternative embodiments, based on any of the above embodiments, such as Figure 8 As shown, in step 520, determining the set of operable controls for the first page based on the first accessibility tree and the first page image may include:

[0192] Step 5210: Determine the first control set based on the attribute information of the nodes in the first accessibility tree.

[0193] Step 5220: Determine the second set of controls based on the first page image.

[0194] Step 5230: Determine the operable control set based on the first control set and the second control set.

[0195] In some optional embodiments, determining the second control set based on the first page image in step 5220 may include:

[0196] Based on the first page image, preset prompt text, and a pre-trained multimodal control recognition model, the region bounding box information, control type, and confidence level of candidate controls in the first page image are determined; based on the region bounding box information, control type, and confidence level of the candidate controls, the second control set is determined.

[0197] In some optional embodiments, determining the operable control set based on the first control set and the second control set in step 5230 may include:

[0198] The first position information of the first control in the first control set on the first page is matched with the second position information of the second control in the second control set on the first page to obtain the matching result; based on the matching result, the first control set and the second control set are deduplicated and merged to obtain the operable control set.

[0199] In some optional embodiments, based on the matching results, the first control set and the second control set are deduplicated and merged to obtain an operable control set, including:

[0200] For any first control in the first control set, in response to the existence of a target second control in the second control set that matches the first control, the target second control and the first control are determined as a single target control, and the control identifier of the first control is determined as the control identifier of the merged target control; or, in response to the existence of no second control in the second control set that matches the first control, preset marking information is added to the first control to obtain a marked control; and / or, based on a second control in the second control set that does not match a first control in the first control set, the second control is determined as a new control; based on the target control, the marked control, and / or the new control, an operable control set is determined.

[0201] In some optional embodiments, based on any of the above embodiments, step 530, which determines the second page after the operation corresponding to the operable control based on the set of operable controls, may include:

[0202] Take any operable control from the set of operable controls as the current control, and control the mobile device to perform the operation corresponding to the current control, so that the mobile device jumps from the first page to the second page after the operation corresponding to the current control.

[0203] Figure 9 This is a flowchart illustrating a data collection method for a mobile webpage provided in yet another exemplary embodiment of this disclosure.

[0204] In some alternative embodiments, based on any of the above embodiments, such as Figure 9As shown, after obtaining the first accessibility tree and the first page image of the mobile device, the method disclosed herein further includes:

[0205] Step 610: Based on the first accessibility tree, generate the first fingerprint information of the first page, and / or, based on the image of the first page, generate the second fingerprint information of the first page.

[0206] Step 620: Determine the first page fingerprint information of the first page based on the first fingerprint information and / or the second fingerprint information.

[0207] Step 550, based on the accessibility tree of the first page, the image of the first page, the set of operable controls, and the relationship graph of the second page, determines the page data for the mobile device, which may include:

[0208] Step 5501: Based on the first accessibility tree of the first page, the first page image, the first page fingerprint information, the set of operable controls, and the second page relationship graph, determine the page data of the mobile terminal.

[0209] In some optional embodiments, generating the first fingerprint information of the first page based on the first accessibility tree in step 610 may include:

[0210] The first accessibility tree is preprocessed to obtain the preprocessed second accessibility tree; based on the second accessibility tree, the first hash value is determined and used as the first fingerprint information.

[0211] In some optional embodiments, step 610, which generates the second fingerprint information of the first page based on the first page image, may include:

[0212] Based on a pre-configured image hashing algorithm, a second hash value of the first page image is determined, and the second hash value is used as the second fingerprint information.

[0213] In some optional embodiments, based on any of the above embodiments, the method of this disclosure further includes: storing the collected page data in a memory; the page data includes page fingerprint information of at least one collected page.

[0214] Step 540 involves updating the relationship graph of the first page based on the first page, operable controls, and the second page to obtain the relationship graph of the second page. This may include:

[0215] Determine the fingerprint information of the second page; based on the fingerprint information of the second page and the page fingerprint information of the collected pages, determine the first matching result between the second page and the collected pages; based on the first page, operable controls, the second page and the first matching result, update the relationship graph of the first page to obtain the relationship graph of the second page.

[0216] In some optional embodiments, the first page relationship graph includes the operation flow relationships between the collected pages; in step 540, based on the first page, operable controls, the second page, and the first matching result, the first page relationship graph is incrementally updated to obtain the second page relationship graph, including:

[0217] In response to the first matching result indicating that the second page is the target page among the collected pages, a loop edge between the first page and the target page is added to the first page relationship graph; based on the operable controls, the operation attributes of the loop edge are determined to obtain the second page relationship graph; or, in response to the first matching result indicating that the second page is not a collected page, the node corresponding to the second page and the target edge between the first page and the second page are added to the first page relationship graph; based on the operable controls, the operation attributes of the target edge are determined to obtain the second page relationship graph.

[0218] In some optional embodiments, determining a first matching result between the second page and the collected page based on the fingerprint information of the second page and the page fingerprint information of the collected page includes:

[0219] The second page fingerprint information is matched with the page fingerprint information of the collected page to determine a second matching result; in response to the second matching result indicating that there is no page fingerprint information in the page fingerprint information of the collected page that matches the second page fingerprint information, the tree edit distance between the second page and the collected page is determined based on the accessibility tree of the second page and the accessibility tree of the collected page; the visual similarity between the second page and the collected page is determined based on the second page image of the second page and the page image of the collected page; and the first matching result between the second page and the collected page is determined based on the tree edit distance and / or visual similarity between the second page and the collected page.

[0220] In some optional embodiments, based on any of the above embodiments, the method of this disclosure further includes:

[0221] Based on the expected page after the operation corresponding to the second page and the operable controls, determine the test results of the operable controls; based on the test results of the operable controls, determine the test results of the mobile terminal.

[0222] In some optional embodiments, step 550, based on the first accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph, determines the page data of the mobile device, including:

[0223] In response to the first page being the last page traversed, the mobile page data is obtained based on the first accessibility tree of the first page, the first page image, the first page's set of operable controls, the previous accessibility tree, the previous page image, and the previous set of operable controls of the previously traversed pages, and the second page relationship graph.

[0224] In some optional embodiments, the method of this disclosure may further include:

[0225] The page images in the mobile page data are labeled to obtain the label information of the page images; the label information includes the correct / incorrect status labels of the page images and / or the correct / incorrect status labels of the controls in the page images; based on the page images and label information, the pre-trained multimodal control recognition model is fine-tuned to obtain the fine-tuned target multimodal control recognition model; based on the label information and the second page relationship graph, the location path of the error page and / or error control is determined; based on the location path and the target multimodal control recognition model, the error page and / or error control in the mobile software corresponding to the mobile page data are retested and stress tested, and a test report is generated.

[0226] In some optional embodiments, based on any of the above embodiments, the method of this disclosure may further include:

[0227] Obtain the pre-configured test scope of the mobile software corresponding to the mobile page data; based on the pre-configured test scope, perform control traversal tests on the pages within the test scope in the mobile software and generate a test report.

[0228] The embodiments described above can be implemented individually or in any combination without conflict. The specific implementation can be set according to actual needs, and this disclosure does not limit them.

[0229] The beneficial technical effects corresponding to the exemplary embodiments of this method can be found in the corresponding beneficial technical effects of the exemplary device section above, and will not be repeated here.

[0230] Any of the mobile page data collection methods provided in this disclosure can be executed by any suitable electronic device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the mobile page data collection methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the mobile page data collection methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0231] Exemplary electronic devices

[0232] Figure 10This is a structural diagram of an electronic device provided in an embodiment of the present disclosure, including at least one processor 91 and a memory 92.

[0233] The processor 91 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0234] The memory 92 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 91 may execute one or more computer program instructions to implement the mobile terminal page data acquisition method and / or other desired functions of the various embodiments of this disclosure described above.

[0235] In one example, the electronic device 90 may also include an input device 93 and an output device 94, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0236] The input device 93 may also include, for example, a touch screen, a microphone, various sensors, etc.

[0237] The output device 94 can output various information to the outside, including, for example, a display, a speaker, a communication network and its connected remote output devices, etc.

[0238] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 90 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 90 may include any other suitable components depending on the specific application.

[0239] Exemplary computer program products and computer-readable storage media

[0240] In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the mobile page data acquisition methods of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0241] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0242] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the mobile page data acquisition methods of various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0243] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0244] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0245] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A data collection device for a mobile webpage, comprising: At least one processor is configured as follows: Get the first accessibility tree and the first page image of the mobile device's first page; Based on the first accessibility tree and the first page image, a set of operable controls for the first page is determined; the set of operable controls includes operable controls on the first page. Based on the set of operable controls, determine the second page after the operation corresponding to the operable control; Based on the first page, the operable controls, and the second page, the relationship graph of the first page is incrementally updated to obtain the relationship graph of the second page; the relationship graph of the first page is an initialized graph, or the relationship graph of the first page is obtained from the previous incremental update; Based on the first accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph, the page data of the mobile terminal is determined.

2. The apparatus according to claim 1, wherein, The step of determining the set of operable controls for the first page based on the first accessibility tree and the first page image includes: Based on the attribute information of the nodes in the first accessibility tree, determine the first control set; Based on the image of the first page, determine the second set of controls; The operable control set is determined based on the first control set and the second control set.

3. The apparatus according to claim 2, wherein, The step of determining the second control set based on the first page image includes: Based on the first page image, the preset prompt text, and the pre-trained multimodal control recognition model, the region bounding box information, control type, and confidence level of the candidate controls in the first page image are determined. The second set of controls is determined based on the region box information, the control type, and the confidence level of the candidate controls.

4. The apparatus according to claim 2, wherein, The step of determining the operable control set based on the first control set and the second control set includes: The first position information of the first control in the first control set on the first page is matched with the second position information of the second control in the second control set on the first page to obtain a matching result; Based on the matching results, the first control set and the second control set are deduplicated and merged to obtain the operable control set.

5. The apparatus according to claim 4, wherein, Based on the matching result, the first control set and the second control set are deduplicated and merged to obtain the operable control set, including: For any first control in the first control set, in response to the existence of a target second control in the second control set that matches the first control, the target second control and the first control are determined as a single target control, and the control identifier of the first control is determined as the control identifier of the merged target control; or, in response to the existence of no second control in the second control set that matches the first control, preset marking information is added to the first control to obtain a marked control; and / or, Based on the second control in the second control set that does not match the first control in the first control set, the second control is identified as a new control; The set of operable controls is determined based on the target control, the labeled control, and / or the new control.

6. The apparatus according to claim 1, wherein, The step of determining the second page after the operation corresponding to the operable control based on the set of operable controls includes: Take any operable control in the set of operable controls as the current control, and control the mobile terminal to perform the operation corresponding to the current control, so that the mobile terminal jumps from the first page to the second page after the operation corresponding to the current control.

7. The apparatus according to claim 1, wherein, The at least one processor is further configured to: After obtaining the first accessibility tree and the first page image of the first page on the mobile device, the first fingerprint information of the first page is generated based on the first accessibility tree, and / or the second fingerprint information of the first page is generated based on the first page image. Based on the first fingerprint information and / or the second fingerprint information, determine the first page fingerprint information of the first page; The process of determining the page data of the mobile device based on the accessibility tree of the first page, the image of the first page, the set of operable controls, and the relationship graph of the second page includes: Based on the first accessibility tree of the first page, the first page image, the first page fingerprint information, the set of operable controls, and the second page relationship graph, the page data of the mobile terminal is determined.

8. The apparatus according to claim 7, wherein, The step of generating the first fingerprint information of the first page based on the first accessibility tree includes: The first accessibility tree is preprocessed to obtain the preprocessed second accessibility tree; Based on the second accessibility tree, a first hash value is determined, and the first hash value is used as the first fingerprint information; and / or, The step of generating the second fingerprint information of the first page based on the first page image includes: Based on a pre-configured image hashing algorithm, a second hash value of the first page image is determined, and the second hash value is used as the second fingerprint information.

9. The apparatus according to claim 1, wherein, Also includes: The memory is configured to store acquired page data; the page data includes page fingerprint information of at least one acquired page; The step of updating the relationship graph of the first page based on the first page, the operable control, and the second page to obtain the relationship graph of the second page includes: Determine the fingerprint information of the second page of the second page; Based on the fingerprint information of the second page and the page fingerprint information of the collected page, a first matching result between the second page and the collected page is determined; Based on the first page, the operable controls, the second page, and the first matching result, the relationship graph of the first page is updated to obtain the relationship graph of the second page.

10. The apparatus according to claim 9, wherein, The first page relationship graph includes the operation flow relationships between the collected pages; the incremental update of the first page relationship graph based on the first page, the operable controls, the second page, and the first matching result to obtain the second page relationship graph includes: In response to the first matching result indicating that the second page is the target page among the collected pages, a loop edge between the first page and the target page is added to the first page relationship graph; based on the operable control, the operation attribute of the loop edge is determined to obtain the second page relationship graph; or... In response to the first matching result indicating that the second page is not a collected page, a node corresponding to the second page and a target edge between the first page and the second page are added to the first page relationship graph; based on the operable control, the operation attribute of the target edge is determined to obtain the second page relationship graph.

11. The apparatus according to claim 9, wherein, The step of determining the first matching result between the second page and the collected page based on the fingerprint information of the second page and the page fingerprint information of the collected page includes: The second page fingerprint information is matched with the page fingerprint information of the collected page to determine the second matching result; In response to the second matching result indicating that there is no page fingerprint information matching the second page fingerprint information in the page fingerprint information of the collected page, the tree edit distance between the second page and the collected page is determined based on the accessibility tree of the second page and the accessibility tree of the collected page. Based on the second page image of the second page and the page image of the captured page, the visual similarity between the second page and the captured page is determined; Based on the tree edit distance and / or visual similarity between the second page and the collected page, the first matching result between the second page and the collected page is determined.

12. The apparatus according to any one of claims 1-11, wherein, The first page relationship graph includes the operation flow relationships between the collected pages; The step of obtaining the first accessibility tree and the first page image of the mobile device includes: Based on the first page relationship graph, the first page is determined using a pre-configured page traversal algorithm; the page traversal algorithm is any one of breadth-first search algorithm, depth-first search algorithm, and reinforcement learning algorithm. Call the interface of the mobile terminal to obtain the first accessibility tree and the first page image of the first page.

13. The apparatus according to any one of claims 1-11, wherein, The at least one processor is further configured to: Based on the second page and the expected page after the operation corresponding to the operable control, the test result of the operable control is determined; Based on the test results of the operable controls, the test results of the mobile device are determined.

14. The apparatus according to any one of claims 1-11, wherein, The process of determining the page data of the mobile device based on the first accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph includes: In response to the first page being the last page traversed, the page data of the mobile device is obtained based on the first accessibility tree of the first page, the first page image, the operable control set of the first page, the previous accessibility tree, the previous page image, and the previous operable control set of the previously traversed pages, and the second page relationship graph.

15. The apparatus according to claim 14, wherein, The at least one processor is further configured to: The page images in the mobile terminal page data are labeled to obtain the tag information of the page images; the tag information includes the correct / incorrect status tags of the page images and / or the correct / incorrect status tags of the controls in the page images; Based on the page image and the label information, the pre-trained multimodal control recognition model is fine-tuned to obtain the fine-tuned target multimodal control recognition model. Based on the tag information and the second page relationship graph, the location path of the error page and / or error control is determined; Based on the positioning path and the target multimodal control recognition model, the error pages and / or error controls in the mobile software corresponding to the mobile page data are retested and stress tested, and a test report is generated.

16. The apparatus according to claim 14, wherein, The at least one processor is further configured to: Obtain the pre-configured test range of the mobile software corresponding to the mobile page data; Based on the pre-configured test scope, control traversal tests are performed on the pages within the test scope in the mobile software, and a test report is generated.

17. A method for collecting data from a mobile webpage, comprising: Get the first accessibility tree and the first page image of the mobile device's first page; Based on the first accessibility tree and the first page image, a set of operable controls for the first page is determined; the set of operable controls includes operable controls on the first page. Based on the set of operable controls, determine the second page after the operation corresponding to the operable control; Based on the first page, the operable controls, and the second page, the relationship graph of the first page is incrementally updated to obtain the relationship graph of the second page; the relationship graph of the first page is an initialized graph, or the relationship graph of the first page is obtained from the previous incremental update; Based on the first accessibility tree of the first page, the first page image, the set of operable controls, and the second page relationship graph, the page data of the mobile terminal is determined.

18. A computer-readable storage medium storing a computer program that is executed by a processor to perform the method of claim 17.

19. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory, and to execute the executable instructions to implement the method of claim 17. Alternatively, the electronic device may include the apparatus described in any one of claims 1-16.

Images