Android application icon barrier-free description automatic generation method

Abstract

The invention discloses an automatic generation method for barrier-free description of Android application icons. The Android application program package is used as input, and the Android application program package embedded with the barrier-free description is used as output. After the user starts the barrier-free service, the operating system can read the barrier-free description of the icon in the application page by using the built-in screen reader, so that the user is helped to understand the current page content. The method comprises the following steps: firstly, automatically exploring a key page screenshot and a document object model in an Android application as a context; and then, according to the questioning method based on the thinking chain, the large model is required to generate corresponding barrier-free descriptions for the icon components in the page and write the barrier-free descriptions into an Android application package so as to achieve the purpose of automatically generating the barrier-free descriptions of the icons. The method provided by the invention is effective and efficient, and accurate barrier-free description can be generated for the icon component in the Android software at a relatively high speed.

Description

Technical Field

[0001] The present application relates to the field of software optimization technology, and in particular to a method for automatically generating accessible descriptions for Android application icons. Background Art

[0002] With the popularity of smart phones, mobile applications have become an important part of people's daily lives. However, for people with disabilities (such as the blind and visually impaired), how to make these applications better usable has become an important issue. The Android system provides barrier-free services (such as screen readers) so that blind and visually impaired users can understand the content on the interface through voice feedback. The screen reader reads the text content of each component on the page to the user. For icon components that do not have text attributes, developers are required to write content description attributes for the screen reader to read. Therefore, for each Android application, the accuracy of the icon component content description directly affects the user experience of visually impaired users.

[0003] In the past, the method of generating accessible descriptions was to train machine learning models and input icon components to generate content descriptions for them. This not only requires additional training resources, but the accuracy of the generated content descriptions depends on the richness of the machine learning model training data set. In the scenario where developers use customized icon components, the performance of the machine learning model is not satisfactory. More importantly, the previous methods cannot automatically obtain the required outputs, requiring developers to perform data preprocessing in advance, wasting developers' energy.

[0004] Therefore, automatically generating accessible descriptions of icons in Android applications to improve the accessibility of applications has become the current research focus. Summary of the invention

[0005] This application provides a method for automatically generating accessible descriptions for Android application icons, which can be used to solve the technical problem that accessible descriptions require developers to perform a lot of preliminary preparation work.

[0006] The present application provides a method for automatically generating an accessibility description of an Android application icon. The method takes an Android application package as input and takes an Android application package containing an accessibility description of an icon as output. The method comprises:

[0007] Step 1: Use automated exploration to extract page screenshots input into the Android application and the document object model corresponding to the screenshots;

[0008] Step 2: Process the page and document object model obtained in step 1, construct prompt words to be input into the visual big model, and use the thinking chain method to ask the big model to obtain the icon accessibility description information;

[0009] Step 3: repackage the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing the icon accessibility description.

[0010] Compared with the prior art, the present invention has the following significant advantages: (1) the process is automated, and Android application developers can obtain an Android application package containing icon accessibility descriptions without having to perform any manual operations; (2) more contextual information is available during the generation process, and the generated accessibility descriptions are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a flow chart of the method for automatically generating accessible descriptions of Android application icons provided by the present invention;

[0012] Figure 2 This is a screenshot of the page after annotation;

[0013] Figure 3 is a comma-delimited file containing a mapping of resource identifiers and content descriptions. DETAILED DESCRIPTION

[0014] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0015] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0016] The present invention is based on Android automated dynamic exploration technology and a large model. First, the key pages in the Android application are automatically explored and the corresponding screenshots and document object models are captured as contexts. Then, based on the large model questioning method of the thinking chain, corresponding accessibility descriptions are generated for the icon components in the page and written into the input Android software to achieve the purpose of automatically generating icon accessibility descriptions. The method proposed by the present invention has the advantages of being effective and efficient, and can generate accurate accessibility descriptions for icon components in Android software at a relatively fast speed.

[0017] In conjunction with the example, the method includes:

[0018] Step 1: Use automated exploration to extract page screenshots input into the Android application and the document object model corresponding to the screenshots.

[0019] Step 1, the specific steps are as follows:

[0020] Step 1-1, decompile the input Android application package and obtain the manifest file corresponding to the Android application package;

[0021] Step 1-2, parse the manifest file and extract the names of all activities in the manifest file;

[0022] Steps 1-3: Run the Android Debug Bridge in the command line, start all activities in the manifest file, take a snapshot of the successfully started activity page, and capture the document object model file corresponding to the page.

[0023] Step 2, process the page and screenshot document object model obtained in step 1, construct prompt words to be input into the visual large model, and use the thinking chain method to ask questions to the large model to obtain icon accessibility description information.

[0024] Step 2: The specific steps are as follows:

[0025] Step 2-1, traversing each page screenshot and document object model in the collection formed by the page and screenshot document object model;

[0026] Step 2-2, pruning each document object model to shorten the file length so that the final constructed input is within the input window size limit of the large model;

[0027] Use depth-first search to parse the user interface hierarchy structure corresponding to the document object model and obtain all leaf nodes. For all leaf nodes, if the Android component type of the node is any one of ImageView, ImageButton, FrameLayout, LinearLayout, Button, TextView, and View, then keep the node, otherwise delete the node to obtain the pruned document object model;

[0028] Step 2-3, traverse the pruned document object model node set obtained in step 2-2, obtain the coordinate attribute of each node, and locate the corresponding component on the page snapshot through the coordinate attribute;

[0029] Extract the RGB value corresponding to the color with the largest component area, and calculate the complementary color of the color with the largest area using the following formula; combine the coordinate attributes and use the complementary color to draw a visual prompt box for the icon component on the page snapshot:

[0030] (R',G',B')=(255-R,255-G,255-B)

[0031] In the above formula, R, G, B correspond to the RGB values ​​of the color that occupies the largest area of ​​the component; R', B', G' correspond to the RGB values ​​of the calculated contrast color;

[0032] Step 2-4, calling the large model application programming interface to start a new round of session, using the document object model obtained in step 2-2 and step 2-3 and the annotated page screenshot as the first session request;

[0033] Step 2-5, continue to send session requests in the current session round, send the unlabeled page snapshot and the activity name corresponding to the page and the prompt "Please infer the basic functions of the page based on the above Android application page screenshot and the corresponding document object model" to the visual big model, and ask the big model to infer and output the page function;

[0034] Step 2-6, continue to send session requests in the current session round, send a page snapshot with visual prompts drawn and the prompt "Please identify how many icons are highlighted in the above Android application page screenshot" to the visual big model to ask the big model the number of components marked in the page. If the big model answers correctly, proceed to the next step. If the big model answers incorrectly, repeat the question until the big model answers correctly to ensure that no icon components in the page are missed by the big model; then, for each node in the pruned document object model, execute steps 2-7 and 2-8.

[0035] Step 2-7, read the nodes in the pruned document object model, continue to send session requests in the current session round, send all the attribute information of the provided nodes and the prompt "This is the basic attribute information of a component. Please identify its visual features based on the boundary information of the component, and infer the function of the component based on the component attributes" to the large model, and require the visual large model to locate the icon component based on the boundary attributes of the node, read the visual features of the icon component, and infer the component function based on the visual features and attribute information of the icon component and output it;

[0036] Step 2-8. Finally, continue to send session requests in the current session round, and send the page function description, component function description and prompt words "Please generate a concise and easy-to-understand accessibility description for the component in the above question based on the function description of the current page and the function description of the component" obtained in step 2-5 and step 2-6 to the big model to ask the big model to write a suitable accessibility description and output it; and write the content description and component resource identifier output by the big model into a comma-delimited file.

[0037] Step 3, parsing the comma-delimited file obtained in steps 2-8, and repackaging the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing the icon accessibility description.

[0038] The specific steps are as follows:

[0039] Step 3-1, parsing each line in the comma-delimited file obtained in step 2-8, parsing the resource file in the decompiled Android application package, and setting the content description attribute of the corresponding component to the stored icon accessibility description according to the resource identifier of the component in the comma-delimited file;

[0040] Step 3-2: Use the Android Debug Bridge in the command line to repackage and sign the decompiled Android file directory to obtain an Android application package containing an icon accessibility description.

[0041] The method provided in this application not only improves the generation efficiency, but also ensures the accuracy and consistency of the description, and can better serve the barrier-free population.

[0042] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A method for automatically generating accessible descriptions of Android application icons, characterized in that: The method takes an Android application package as input and takes an Android application package containing an icon accessibility description as output, and the method comprises: Step 1, using automated exploration to extract page screenshots input into the Android application and the document object model corresponding to the screenshots; Step 2: Process the page and document object model obtained in step 1, construct prompt words to be input into the visual big model, and use the thinking chain method to ask the big model to obtain the icon accessibility description information; Step 3: repackage the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing the icon accessibility description.

2. The method according to claim 1, characterized in that Step 1: Use automated exploration to extract page screenshots input into the Android application and the document object model corresponding to the screenshots; including: Step 1-1, decompile the input Android application package and obtain the manifest file corresponding to the Android application package; Step 1-2, parse the manifest file and extract the names of all activities in the manifest file; Steps 1-3: Run the Android Debug Bridge in the command line, start all activities in the manifest file, take a snapshot of the successfully started activity page, and capture the document object model file corresponding to the page. Step 2, process the page and screenshot document object model obtained in step 1, construct prompt words to be input into the visual large model, and use the thinking chain method to ask questions to the large model to obtain icon accessibility description information.

3. The method according to claim 2, characterized in that Step 2: Process the page and document object model obtained in step 1, construct prompt words to be input into the visual big model, and use the thinking chain method to ask the big model to obtain icon accessibility description information, including: Step 2-1, traversing each page screenshot and document object model in the collection formed by the page and screenshot document object model; Step 2-2, pruning each document object model to shorten the file length so that the final constructed input is within the input window size limit of the large model; Use depth-first search to parse the user interface hierarchy structure corresponding to the document object model and obtain all leaf nodes. For all leaf nodes, if the Android component type of the node is any one of ImageView, ImageButton, FrameLayout, LinearLayout, Button, TextView, and View, then keep the node, otherwise delete the node to obtain the pruned document object model; Step 2-3, traverse the pruned document object model node set obtained in step 2-2, obtain the coordinate attribute of each node, and locate the corresponding component on the page snapshot through the coordinate attribute; Extract the RGB value corresponding to the color with the largest component area, and calculate the complementary color of the color with the largest area using the following formula; combine the coordinate attributes and use the complementary color to draw a visual prompt box for the icon component on the page snapshot: (R',G',B')=(255-R,255-G,255-B) In the above formula, R, G, B correspond to the RGB values ​​of the color that occupies the largest area of ​​the component; R', B', G' correspond to the RGB values ​​of the calculated contrast color; Step 2-4, calling the large model application programming interface to start a new round of conversation, using the document object model obtained in step 2-2 and step 2-3 and the annotated page screenshot as the first conversation request; Step 2-5, continue to send session requests in the current session round, send the unlabeled page snapshot and the activity name corresponding to the page and the prompt word "Please infer the basic functions of the page based on the page screenshot of the above Android application and the corresponding document object model" to the visual big model, and ask the big model to infer and output the page function; Step 2-6, continue to send session requests in the current session round, send a page snapshot with visual prompts drawn and the prompt word "Please identify how many icons are highlighted in the screenshot of the Android application page above" to the visual big model to ask the big model the number of components marked in the page. If the big model answers correctly, proceed to the next step. If the big model answers incorrectly, repeat the question until the big model answers correctly to ensure that no icon components in the page are missed by the big model; then, for each node in the pruned document object model, execute steps 2-7 and 2-8; Step 2-7, read the nodes in the pruned document object model, continue to send session requests in the current session round, send all the attribute information of the provided nodes and the prompt word "This is the basic attribute information of a component. Please identify its visual features based on the boundary information of the component, and infer the function of the component based on the component attributes" to the large model, and require the visual large model to locate the icon component based on the boundary attributes of the node, read the visual features of the icon component, and infer the component function based on the visual features and attribute information of the icon component and output it; Step 2-8. Finally, continue to send session requests in the current session round, and send the page function description, component function description and prompt words "Please generate a concise and easy-to-understand accessibility description for the component in the above question based on the function description of the current page and the function description of the component" obtained in step 2-5 and step 2-6 to the big model, requiring the big model to write a suitable accessibility description and output it; and write the content description and component resource identifier output by the big model into a comma-delimited file.

4. The method according to claim 3, characterized in that Step 3, repackaging the icon accessibility description information obtained in step 2 into the input Android application to obtain an Android application containing the icon accessibility description, including: Step 3-1, parsing each line in the comma-delimited file obtained in step 2-8, parsing the resource file in the decompiled Android application package, and setting the content description attribute of the corresponding component to the stored icon accessibility description according to the resource identifier of the component in the comma-delimited file; Step 3-2: Use the Android Debug Bridge in the command line to repackage and sign the decompiled Android file directory to obtain an Android application package containing an icon accessibility description.

Images