Mobile application barrier-free operability testing method based on task semantic driving

Abstract

A mobile application barrier-free operability testing method based on task semantic driving comprises the steps that interface exploration is conducted on an application in the closed state of screen reading software, and a to-be-detected task is generated based on page function semantics; performing semantic disassembly on the detection task to obtain a multi-level task step; constructing a barrier-free perception view in the screen reading mode to obtain interaction information which can be actually perceived by the blind user; according to task step semantics, candidate nodes are retrieved in a perception view, semantic distinguishability judgment is carried out, and a detection path which can be theoretically completed is deduced; step-by-step navigation verification is carried out based on focus sliding operation of screen reading software, and whether an interface generates expected state change after click is detected, so that whether a target control is real and reachable on an operation level is judged. Compared with an existing barrier-free detection mode which only depends on a control attribute rule or a traversal strategy, the interaction process of a real blind user can be simulated from the perspective of tasks, and the authenticity and effectiveness of barrier-free testing are improved.

Description

Technical Field

[0001] This invention relates to the field of mobile application accessibility testing technology, and in particular to a task semantic-driven mobile application accessibility and operability testing method, used to test whether a mobile application can support visually impaired users to complete functional operations through screen reader software. Background Technology

[0002] With the widespread adoption of mobile internet applications, accessibility has gradually become an important indicator for measuring the usability and social equity of mobile applications. Visually impaired users often need to complete tasks through a combination of voice reading and step-by-step navigation when using screen reader software. Some mobile applications on the market exhibit issues such as "missed focus," "invalid focus," and "unclickable components" when the screen reader is activated, preventing visually impaired users from completing functional operations. Existing accessibility testing methods primarily rely on simple rules (such as depth-first search, breadth-first search, and random search) during testing, failing to realistically simulate the usage of visually impaired users from a functional perspective. Summary of the Invention

[0003] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a method for testing the accessibility and operability of mobile applications based on task semantics.

[0004] The present invention provides a task-semantic driven accessibility testing method, comprising: S110: Detection Task Generation. Without screen reader software enabled, the interface of the mobile application under test is explored, page functions and navigation relationships are summarized, and several accessibility tasks to be tested are generated. S120: Task Step Analysis. The natural language form of the detection task generated in step S110 is semantically structured, transforming the overall task into a multi-level interactive step sequence with execution meaning; S130: Construction of Accessibility Perception View. After screen reader mode is enabled, a unified model is constructed for the information that can be obtained from the interface under accessibility conditions, in order to represent the content perceived by blind users; S140: Detection Path Derivation. Based on the accessibility perception view, determine whether the user can make the correct interaction decision based on the information available from the screen reader, and deduce the detection path required to complete the task accordingly; S150: Dynamic Screen Reader Navigation Mapping and Evaluation. After determining the feasibility of the decision, based on the actual operation methods of blind users in screen reading scenarios, the execution operation is converted into a screen reader software interaction process involving focus movement; and the feasibility of focus attainment and whether click operations redirect correctly are evaluated.

[0005] Specifically, step S110 includes: S1101: Perform random traversal page exploration on the application interface, record the interface switching relationships and the operable controls that trigger these switching relationships, in order to construct an interface state transition graph (i.e., the transition path between pages).

[0006] S1102: For each explored page in the interface state diagram, extract its interface screenshot and hierarchical structure information, and perform functional semantic summarization of the page content based on the multimodal large model to obtain the functional description information of the page.

[0007] S1103: Based on the functional description information, cluster or merge pages with similar functional meanings to form a set of functionally equivalent pages, thereby semantically compressing the interface state space and reducing redundancy in subsequent path deduction.

[0008] S1104: Combining the set of functionally equivalent pages after semantic compression and their mutual transition relationships in the state graph, derive the functional links that the user may complete, and transform the functional links into accessibility tasks to be detected for subsequent accessibility detection in screen reader mode.

[0009] Specifically, step S120 includes: S1201: Perform semantic analysis on natural language task text, identify the target behavior corresponding to the task and the key objects involved, and obtain a task intent representation consisting of "action + target entity".

[0010] S1202: Based on the task intent representation, the overall task is decomposed into several steps with an execution sequence relationship, where each step corresponds to an independent user interaction intent.

[0011] Specifically, step S130 includes: S1301: Collect accessibility node information of the current page through screen reader software interface or accessibility assistance service, including focusable nodes, node text to be read aloud, node control type, and whether the node can be clicked.

[0012] S1302: Obtain the semantic adjacency relationship of nodes based on the traversal order of the screen reader output, and obtain the actual reading order and focus switching rules of the nodes presented to the user in the accessibility mode.

[0013] S1303: Extract the information that can be read aloud from the accessible node into a semantic representation that is perceptible to the user, in order to characterize "what the node is understood from the perspective of a blind person".

[0014] S1304: Based on the screen reading order, the text to be read, and the roles of controls, the page is constructed as a "perceptible view" that represents the semantic boundaries that the user can grasp under screen reading conditions, that is, the effective range of information that blind users can use for behavioral decisions.

[0015] Specifically, step S140 includes: S1401: Based on the task step semantics obtained in step S120, retrieve the set of candidate barrier-free nodes related to the step semantics in the perceptible view obtained in S130 to form a list of possible next interaction targets.

[0016] S1402: Evaluate the discriminative power of the candidate node's text or semantic description. If there is insufficient semantic difference between the nodes, and the user cannot determine which node to select, then this step is deemed "indecisive" under screen reading conditions.

[0017] S1403: If at least one candidate node satisfies the condition that "the user can uniquely and correctly select the node based on the read-aloud information", then the step is considered to be decisable under barrier-free conditions; otherwise, the step is marked as "semantically unreachable".

[0018] S1404: For steps that pass the decision feasibility assessment, the uniquely identifiable nodes corresponding to each step are combined according to the task execution order to form the "semantic feasible path" of the task. This path represents the task completion link that a blind user can theoretically correctly identify.

[0019] Specifically, step S150 includes: S1501: Based on the screen reading sequence information and focusable attributes of the accessibility-aware view, construct the reachability and connectivity between nodes to characterize the navigation path that the user may take under screen reading conditions.

[0020] S1502: For each target node in the task path, calculate the shortest navigation path from the current focus to the target node; if there is no valid focus connection link, the node is considered to be unreachable at the operational level.

[0021] S1503: Translates the navigation path into a sequence of screen reading operations (e.g., move forward N focus points and then click), and simulates gestures via Bluetooth to simulate the behavior of a real blind person performing this operation.

[0022] S1504: Check if there is a navigation path from the current focus to the target node. If there is no path, the operation is considered unreachable.

[0023] S1505: After performing the click operation, confirm whether the interface has changed as expected. If the page does not redirect or the target state does not appear, the operation is considered unreachable.

[0024] The working principle of this invention is as follows: First, with the screen reader software off, the application interface is explored, and a multimodal large model is used to summarize page functions and jump relationships, generating several detection tasks; then, the detection tasks are semantically decomposed to obtain multi-level task steps; an accessible perception view is constructed in screen reader mode to obtain interactive information that blind users can actually perceive; based on the semantics of the task steps, candidate nodes are retrieved in the perception view and semantic distinguishability is determined, deriving the theoretically achievable detection path; based on the focus sliding operation of the screen reader software, step-by-step navigation verification is performed, and it is detected whether the interface produces the expected state change after clicking, thereby determining whether the target control is truly reachable at the operational level.

[0025] The innovation of this invention lies in the following: First, the application's interface is explored, and a multimodal large model is used to summarize page functions and generate detection tasks. Accessibility detection is then performed through task-guided methods. By constructing an accessibility perception view, a user scenario for the blind is simulated, and then, based on semantic distinguishability, a feasible detection path is derived.

[0026] The advantages of this invention are: compared to other accessibility detection methods that explore pages according to rules, this invention is task-driven and can better simulate the usage process of blind users. During the simulation test, an accessibility perception view was constructed, and the semantic distinguishability of candidate nodes was determined, which can detect the problem of insufficient information from screen reader software. Attached Figure Description

[0027] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0028] The technical solution of the present invention will be described below with reference to the accompanying drawings.

[0029] The following describes one possible implementation of the present invention. This embodiment uses the Android system as an example, utilizing UIAutomator and AccessibilityService to automate control and accessibility information collection for the application under test, verifying the detection of the accessibility and operability of the mobile application in a task-driven manner.

[0030] A task-semantic driven accessibility and operability testing method includes the following steps: (1) S110: Generate detection task Before enabling screen reader mode, the mobile application under test is explored to automatically extract functional semantics and generate detection tasks. UIAutomator is used to perform a deep traversal of the application, sequentially clicking on controls with `clickable=true` within the page, and collecting the current Activity name or window identifier after each click. If the Activity changes, the state transition (source_page → target_page) is recorded. A page state graph is generated after the traversal is complete. Screen images are captured for each page, and the interface control hierarchy is collected. A page functional summary is generated using a multimodal large model (e.g., Qwen-VL). The functional summaries of different pages are vectorized and semantic similarity is calculated. Pages with summary content similarity higher than a preset threshold are grouped into the same category, forming a set of functionally equivalent pages to reduce the impact of duplicate pages on task extraction. Based on the connectivity relationships in the state graph, semantically complete page sequences are identified and transformed into user target-level tasks. For example, "food delivery homepage → store details → shopping cart → checkout page" corresponds to the task "complete an order".

[0031] (2) S120: Task Step Analysis

[0032] The detection task is broken down into steps using a large language model, and action types (such as "enter / select / submit") and their objects (such as "takeout entry / store / order") are extracted. Based on the semantic dependencies of the task, the overall task is broken down into a multi-step operation process, such as: [enter takeout entry] → [select store] → [add to cart] → [submit order].

[0033] (3) S130: Construction of barrier-free perception view

[0034] The AccessibilityService is invoked to retrieve all focusable nodes in the current UI hierarchy, and their text, content description, type, and other attributes are read. The system's TYPE_VIEW_FOCUSED event is listened for, and the actual focus traversal sequence is recorded to form a reachable list of nodes. For each node, three key pieces of information are abstracted: (node ​​type, text to be read aloud, and whether focus is available); if a node is neither readable nor focusable, it is considered imperceptible. These nodes and their order are combined to obtain a list of content that the user can truly perceive on the page, serving as the perceptual view for subsequent decisions on whether a decision can be made.

[0035] (4) S140: Detection path derivation

[0036] Based on the target semantics of the current task step (e.g., "enter the takeout entrance" or "select a store"), retrieve all nodes in the perceptible view that meet at least one of the following conditions: the text read aloud contains key semantic keywords of the task (e.g., "takeout" or "store"); the control role type matches the step action (e.g., if action="enter", it matches Button / Tab / jump nodes). Form a candidate node set C = {c1, c2, …}. For the text read aloud of each node in set C, extract the text semantic vector (e.g., based on the Sentence-BERT model) and calculate its similarity score (ci) with the task target semantic text. If max(score(ci)) is less than a set threshold, it is considered "semantically insufficient". For all steps with decision-making capability, sequentially accumulate the corresponding target nodes according to the task step sequence to form a "semantically feasible path" P = {a1 → a2 → … → an}.

[0037] (5) S150: Dynamic screen reading navigation verification

[0038] The AccessibilityService is used to obtain the currently focused node. If the current page has no focus, an ACTION_FOCUS_FORWARD is sent to initialize the focus to the first node of the accessible sequence. By continuously calling ACTION_FOCUS_FORWARD (or ACTION_FOCUS_BACKWARD in a fallback scenario), the focus is moved to subsequent nodes in sequence until: a) the target control is found, indicating that the path is reachable; b) the target has not appeared after a full cycle, indicating that the path is unreachable. When the focus is on the target node, a simulated click action is used to trigger the node behavior. After the click, the accessibility node tree or Activity name is retrieved again. If the page state changes as expected (entering the next page), it indicates that the node is truly operable; otherwise, it is recorded as "operation failed".

Claims

1. A task-semantic driven accessibility testing method, comprising: S110: Detection task generation; Without screen reader software enabled, the interface of the mobile application under test is explored, page functions and navigation relationships are summarized, and several accessibility tasks to be tested are generated. S120: Task step analysis; Semantic structuring processing is performed on the detection task in natural language form generated in step S110, transforming the overall task into a multi-level interactive step sequence with execution meaning; S130: Accessibility Perception View Construction; After the screen reader mode is enabled, the information that can be obtained from the interface under the accessibility channel is uniformly modeled to represent the content perceived by blind users. S140: Detection path derivation; Based on the accessibility perception view, determine whether the user can make the correct interaction decision based on the information available from the screen, and deduce the detection path required to complete the task accordingly. S150: Dynamic screen reader navigation mapping and evaluation; Under the premise of completing the feasibility assessment of the decision, based on the actual operation mode of blind users in the screen reading scenario, the execution operation is converted into a screen reading software interaction process of focus movement; It also assesses whether the focus is reachable and whether the click operation redirects normally.

2. The accessibility and operability testing method based on task semantics as described in claim 1, characterized in that, Step S110 specifically includes: S1101: Perform random traversal page exploration on the application interface, record the interface switching relationships and the operable controls that trigger these switching relationships, in order to construct an interface state transition graph (i.e., the transition path between pages). S1102: For each explored page in the interface state diagram, extract its interface screenshot and hierarchical structure information, and perform functional semantic summarization of the page content based on the multimodal large model to obtain the functional description information of the page. S1103: Based on the functional description information, cluster or merge pages with similar functional meanings to form a set of functionally equivalent pages, thereby semantically compressing the interface state space and reducing redundancy in subsequent path deduction. S1104: Combining the set of functionally equivalent pages after semantic compression and their mutual transition relationships in the state graph, derive the functional links that the user may complete, and transform the functional links into accessibility tasks to be detected for subsequent accessibility detection in screen reader mode.

3. The accessibility and operability testing method based on task semantics as described in claim 1, characterized in that, Step S120 specifically includes: S1201: Perform semantic analysis on natural language task text, identify the target behavior corresponding to the task and the key objects involved, and obtain a task intent representation consisting of "action + target entity"; S1202: Based on the task intent representation, the overall task is decomposed into several steps with an execution sequence relationship, where each step corresponds to an independent user interaction intent.

4. The accessibility and operability testing method based on task semantics as described in claim 1, characterized in that, Step S130 specifically includes: S1301: Collect accessibility node information of the current page through screen reader software interface or accessibility assistance service, including focusable nodes, node text to be read aloud, node control type, and whether the node can be clicked; S1302: Obtain the semantic adjacency relationship of nodes based on the traversal order of the screen reader output, and obtain the actual reading order and focus switching rules of the nodes presented to the user in the accessibility mode; S1303: Extract the information that can be read aloud from the accessible node into a semantic representation that is perceptible to the user, in order to characterize "what the node is understood as from the perspective of a blind person"; S1304: Based on the screen reading order, the text to be read, and the roles of controls, the page is constructed as a "perceptible view" that represents the semantic boundaries that the user can grasp under screen reading conditions, that is, the effective range of information that blind users can use for behavioral decisions.

5. The accessibility and operability testing method based on task semantics as described in claim 1, characterized in that, Step S140 specifically includes: S1401: Based on the task step semantics obtained in step S120, retrieve the set of candidate barrier-free nodes related to the step semantics in the perceptible view obtained in S130 to form a list of possible next interaction targets; S1402: Evaluate the discriminative power of the candidate node’s text or semantic description. If there is a lack of sufficient semantic difference between the nodes and the user cannot determine which node to select, then this step is deemed “indecisive” under screen reading conditions. S1403: If at least one candidate node satisfies the condition that "the user can uniquely and correctly select the node based on the read-aloud information", then the step is considered to be decisable under barrier-free conditions; otherwise, the step is marked as "semantically unreachable". S1404: For steps that pass the decision feasibility judgment, the uniquely identifiable nodes corresponding to each step are combined according to the task execution order to form the "semantic feasible path" of the task. This path represents the task completion link that a blind user can theoretically correctly identify.

6. The accessibility and operability testing method based on task semantics as described in claim 1, characterized in that, Step S150 specifically includes: S1501: Based on the screen reading sequence information and focusable attributes of the accessibility-aware view, construct the reachability and connectivity between nodes to characterize the navigation path that the user may take under screen reading conditions; S1502: For each target node in the task path, calculate the shortest navigation path from the current focus to the target node; if there is no valid focus connection link, the node is considered to be unreachable at the operational level. S1503: Translates the navigation path into a sequence of screen reading operations (e.g., move forward N focus points and then click), and simulates gestures via Bluetooth to simulate the behavior of a real blind person completing the operation; S1504: Check if there is a navigation path from the current focus to the target node. If there is no path, the operation is considered unreachable. S1505: After performing the click operation, confirm whether the interface has changed as expected. If the page does not redirect or the target state does not appear, the operation is considered unreachable.

Images