Method for automatically generating accessibility descriptions of android application icons

Abstract

The application discloses an automatic generation method for barrier-free description of Android application icons. The Android application package is taken as the input, and the Android application package embedded with barrier-free description is taken as the output. When the user starts the barrier-free service, the operating system will use its built-in screen reader to read the barrier-free description of the icon in the application page, thereby helping the user to understand the current page content. The application first automatically explores the key page screenshots and document object models in the Android application as the context, and then requires the large model to generate the corresponding barrier-free description for the icon components in the page based on the questioning method of the thinking chain and write it into the Android application package to achieve the purpose of automatically generating the barrier-free description of the icons. The method provided by the application is effective and efficient, and can generate accurate barrier-free description for the icon components in the Android software at a relatively fast speed.

Description

Technical Field

[0001] This application relates to the field of software optimization technology, and in particular to a method for automatically generating accessibility descriptions for Android application icons. Background Technology

[0002] With the widespread adoption of smartphones, mobile applications have become an integral part of people's daily lives. However, enabling these applications to be used more effectively by people with disabilities (such as the blind and visually impaired) has become a significant challenge. Android provides accessibility services (such as screen readers), allowing blind and visually impaired users to understand the content on the interface through voice feedback. Screen readers read aloud the text content of each component on the page. For icon components without text attributes, developers need to write content description attributes for the screen reader to read aloud. Therefore, for each Android application, the accuracy of the icon component content description directly impacts the user experience for visually impaired users.

[0003] Traditional accessibility description generation methods involved training machine learning models and inputting icon components to generate content descriptions. This not only required additional training resources, but the accuracy of the generated descriptions also depended on the richness of the machine learning model's training dataset. When developers used custom icon components, the performance of the machine learning model was unsatisfactory. More importantly, past methods could not automatically obtain the required output, requiring developers to perform data preprocessing beforehand, wasting their time and effort.

[0004] Therefore, automatically generating accessibility descriptions for icons in Android applications to improve application accessibility has become a current research focus. Summary of the Invention

[0005] This application provides a method for automatically generating accessibility descriptions for Android application icons, which can be used to solve the technical problem that developers need to do a lot of preliminary work for accessibility descriptions.

[0006] This application provides a method for automatically generating accessibility descriptions for Android application icons. The method takes an Android application package as input and an Android application package containing icon accessibility descriptions as output. The method includes:

[0007] Step 1: Use automated exploration to extract page screenshots from the input Android application and the corresponding document object model;

[0008] Step 2: Process the page and document object model obtained in Step 1, construct prompt words for input into the visual big model, and use the thinking chain method to ask questions to the big model to obtain the icon accessibility description information;

[0009] Step 3, repackage the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing icon accessibility description.

[0010] Compared with the prior art, the present application has the following advantages: (1) process automation, Android application developers do not need to perform any manual operation to obtain an Android application package containing icon accessibility description; (2) more context information is provided during the generation process, and the generated accessibility description is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a flowchart of the Android application icon accessibility description automatic generation method provided by the present application;

[0012] Figure 2 is a screenshot of the page after annotation;

[0013] Figure 3 is a screenshot of a comma-separated file containing resource identifier and content description mapping. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

[0015] First, the embodiments of the present application will be introduced below with reference to the accompanying drawings.

[0016] The present application is based on Android automatic dynamic exploration technology and large models. First, the key pages in the Android application are automatically explored, and the corresponding screenshots and document object models are intercepted as context. Then, the corresponding accessibility description for the icon components in the page is generated based on the large model question method of the thinking chain and written into the input Android software to achieve the purpose of automatically generating icon accessibility description. The method proposed in the present application has the advantages of effectiveness and efficiency, and can generate accurate accessibility description for icon components in Android software at a faster speed.

[0017] In combination with examples, the method comprises:

[0018] Step 1, use automatic exploration to extract the page screenshots and document object models corresponding to the screenshots in the input Android application.

[0019] Step 1, the specific steps are as follows:

[0020] Step 1-1, decompile the input Android application package to obtain the manifest file corresponding to the Android application package;

[0021] Step 1-2, parse the manifest file to extract the names of all activities in the manifest file;

[0022] Step 1-3, run the Android Debug Bridge in the command line, start all activities in the manifest file, for the successfully started activity page, intercept the page snapshot of the activity, and capture the document object model file corresponding to the page.

[0023] Step 2, process the page and screenshot document object model obtained in step 1 to construct the prompt word input into the visual large model, and use the thinking chain method to ask the large model to obtain the icon accessibility description information.

[0024] Step 2, the specific steps are as follows:

[0025] Step 2-1, traverse each page screenshot and document object model in the collection formed by the page and screenshot document object model;

[0026] Step 2-2, pruning each document object model to shorten the file length, so that the final constructed input is within the input window size limit of the large model;

[0027] Use depth-first search to parse the user interface hierarchy structure corresponding to the document object model, get all leaf nodes, for all leaf nodes, if the node's Android component type is any one of ImageView, ImageButton, FrameLayout, LinearLayout, Button, TextView, View, then keep the node, otherwise delete the node, get the pruned document object model;

[0028] Step 2-3, traverse the pruned document object model node set obtained in step 2-2, get the coordinate attribute of each node, and locate the corresponding component on the page snapshot through the coordinate attribute;

[0029] Extract the RGB value of the color with the largest area proportion of the component, and calculate the complementary color of the color with the largest proportion by the following formula; combine the coordinate attribute, and use the complementary color to draw a visual prompt box for the icon component on the page snapshot:

[0030] (R',G',B')=(255-R,255-G,255-B)

[0031] In the above formula, R, G, B correspond to the RGB value of the color with the largest proportion in the component area; R', B', G' correspond to the RGB value of the calculated contrast color;

[0032] Step 2-4, call the large model application programming interface to start a new round of session, and use the document object model obtained in step 2-2 and step 2-3 and the labeled page screenshot as the first session request;

[0033] Step 2-5, continue to send a session request in the current session round, send the unlabeled page snapshot and the corresponding activity name of the page to the visual large model, and the prompt word "please infer the basic function of the page according to the above page snapshot of the Android application and the corresponding document object model", require the large model to infer and output the page function;

[0034] Step 2-6, continue to send a session request in the current session round, send the page snapshot after drawing the visual prompt to the visual large model, and the prompt word "please identify how many icons in the above Android application page screenshot are highlighted by the square frame" to ask the large model the number of components labeled in the page, if the large model answers correctly, go to the next step, if the large model answers incorrectly, repeat the question until the large model answers correctly, to ensure that no icon component in the page is missed by the large model; then, for each node in the pruned document object model, perform steps 2-7 and 2-8.

[0035] Step 2-7, read the node in the pruned document object model, continue to send a session request in the current session round, send all attribute information of the node to the large model, and the prompt word "this is the basic attribute information of a component, please identify its visual features according to the boundary information of the component, and infer the function of the component according to the visual features and attribute information of the icon component", require the visual large model to locate the icon component according to the boundary attribute of the node, and read the visual features of the icon component, and infer the component function according to the visual features and attribute information of the icon component and output;

[0036] Step 2-8, finally, continue to send a session request in the current session round, send the page function description obtained in steps 2-5 and 2-6, the component function description to the large model, and the prompt word "please generate a concise and easy-to-understand accessibility description for the component in the above question according to the function description of the current page and the function description of the component" to require the large model to write appropriate accessibility description and output; and write the content description output by the large model and the resource identifier of the component into a comma-separated file.

[0037] Step 3, parse the comma-separated file obtained in step 2-8, and pack the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing icon accessibility description.

[0038] The specific steps are as follows:

[0039] Step 3-1, parse each line in the comma-separated file obtained in step 2-8, parse the resource file in the decompiled Android application package, and set the content description attribute of the corresponding component to the stored icon accessibility description according to the resource identifier of the component in the comma-separated file;

[0040] Step 3-2, repackaging and signing the directory of the decompiled Android file using the Android Debug Bridge in the command line to obtain an Android application package containing the icon accessibility description.

[0041] The method provided by the application not only improves the generation efficiency, but also guarantees the accuracy and consistency of the description, and can better serve the accessibility population.

[0042] The above-mentioned embodiments of the application do not constitute a limitation on the protection scope of the application.

Claims

1. An automated method for generating accessibility descriptions for Android application icons, characterized in that, The method takes an Android application package as input and an Android application package containing icon accessibility descriptions as output. The method includes: Step 1: Use automated exploration to extract page screenshots from the input Android application and the corresponding document object model; Step 2: Process the page and document object model obtained in Step 1, construct prompt words for input into the visual big model, and use the thinking chain method to ask questions to the big model to obtain the icon accessibility description information; Step 2 involves processing the page and document object model obtained in Step 1, including: Step 2-1: Traverse the collection of screenshots and document object models for each page; Step 2-2: Prune each document object model to shorten the file length so that the final input is within the input window size limit of the large model. The document object model is parsed using a depth-first search to obtain all leaf nodes. For each leaf node, if the Android component type of the node is any one of ImageView, ImageButton, FrameLayout, LinearLayout, Button, TextView, or View, the node is kept; otherwise, the node is deleted, resulting in the pruned document object model. Step 2-3: Traverse the set of pruned document object model nodes obtained in Step 2-2, obtain the coordinate attributes of each node, and locate the corresponding component on the page snapshot using the coordinate attributes; Extract the RGB value of the color with the largest component area, and calculate its complementary color using the following formula; combine this with coordinate attributes, and use the complementary color to draw a visual tooltip for the icon component on the page snapshot: (R′,G′,B′)=(255-R,255-G,255-B) In the above formula, R, G, B correspond to the RGB values ​​of the colors that account for the largest proportion of the component area; R', B', G' correspond to the RGB values ​​of the calculated contrasting colors. Step 3: Repackage the icon accessibility description information obtained in Step 2 into the input Android application to obtain an Android application containing icon accessibility descriptions.

2. The method according to claim 1, characterized in that, Step 1: Use automated exploration to extract page screenshots from the input Android application and the corresponding Document Object Model (DOM); including: Step 1-1: Decompile the input Android application package to obtain the manifest file corresponding to the Android application package; Steps 1-2: Parse the manifest file and extract the names of all activities in the manifest file; Steps 1-3: Run the Android Debug Bridge in the command line, start all activities in the manifest file, capture a snapshot of the activity page for successfully started activities, and capture the corresponding Document Object Model file of the page.

3. The method according to claim 2, characterized in that, Step 2 involves processing the page and document object model obtained in Step 1 to construct prompts for input into the visual big model. It also uses a thought chain approach to ask questions of the big model to obtain accessibility descriptions of icons, including: Step 2-4: Call the large model application programming interface to start a new session, using the document object model obtained in Step 2-2 and Step 2-3, along with the annotated page screenshots, as the first session request. Steps 2-5: In the current session round, continue to send session requests, sending unlabeled page snapshots and the corresponding activity names and prompts to the visual big model: "Please infer the basic functions of the page based on the page screenshots of the Android application and the corresponding document object model." The big model is required to infer and output the page functions. Steps 2-6: In the current session round, continue sending session requests, sending a snapshot of the page after visual cues have been drawn and the prompt "Please identify how many icons are highlighted in the above Android application page screenshot" to the visual model to ask the model the number of labeled components on the page. If the model answers correctly, proceed to the next step; if the model answers incorrectly, repeat the question until the model answers correctly to ensure that no icon components on the page are missed by the model. Subsequently, for each node in the pruned document object model, execute steps 2-7 and 2-8. Steps 2-7: Read the nodes in the pruned Document Object Model. In the current session round, continue to send session requests, sending all attribute information of the nodes and the prompt "This is the basic attribute information of a component. Please identify its visual features based on the component's boundary information and infer the function of the component based on the component's attributes" to the large visual model. The large visual model is required to locate the icon component based on the node's boundary attributes, read the visual features of the icon component, infer the component's function based on the icon component's visual features and attribute information, and output the result. Steps 2-8: Finally, in the current session round, continue sending session requests, sending the page function description, component function description, and prompt "Please generate a concise and easy-to-understand accessibility description for the component mentioned above based on the current page function description and component function description" to the large model, requiring the large model to write an appropriate accessibility description and output it; and write the content description output by the large model and the resource identifier of the component into a comma-separated file.

4. The method according to claim 3, characterized in that, Step 3: Repackage the icon accessibility description information obtained in Step 2 into the input Android application to obtain an Android application containing icon accessibility descriptions, including: Step 3-1: parse each line of the comma-separated file obtained in Step 2-8, parse the resource files in the decompiled Android application package, and set the content description attribute of the corresponding component to the stored icon accessibility description according to the resource identifier of the component in the comma-separated file. Step 3-2: Use the Android Debug Bridge in the command line to repackage and sign the decompiled Android file directory to obtain an Android application package containing icons and accessibility descriptions.

Images