Determining the attributes of elements in the viewable content and adding them to the accessibility tree

Abstract

The method may receive an image representing displayable content for display by an application. The method may execute a layout extraction model that uses the image as input and generates as output a list of elements of the image. The list of elements includes at least a bounding box defining a portion of the image and a role attribute. The method may use the list of elements to add the role attribute to a node in an accessibility tree.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Application No. 18 / 046,898, filed October 14, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This description describes how to add attributes or nodes to the accessibility tree to provide access to additional viewable content. [Background technology]

[0003] This description generally relates to methods for creating roles and / or nodes for elements of displayable content and adding those roles and / or nodes to an accessibility tree. The accessibility tree is used to support accessible features in software, such as screen readers, that provide access to content to people with different sets of capabilities. Traditionally, many documents, such as web documents, have been strongly oriented toward presenting content to users visually. Accessibility software, such as screen readers, may be used, for example, by people with no vision or low vision, to provide content in an alternative format. The accessibility software relies on the accessibility tree being complete to facilitate providing content in an accessible format. Summary of the Invention

[0004] This disclosure describes a method for adding information representing elements from displayable content that were not previously in the accessibility tree to an accessibility tree. This disclosure describes an accessibility infrastructure that receives an image representing displayable content for display by an application. A layout extraction model generates a list of elements of the image. Each element includes a bounding box that defines the portion of the image in which the element is located and a role attribute associated with the element. The accessibility infrastructure then adds the role attribute to a node in the accessibility tree. In some examples, the accessibility infrastructure may also add nodes or other attributes to the accessibility tree.

[0005] In some aspects, the technology described herein relates to a method that includes receiving an image representing displayable content for display by an application, and executing a layout extraction model that uses the image as input and generates as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute, the method further including adding the role attribute to a node in an accessibility tree using the list of elements.

[0006] In some aspects, the technology described herein relates to a system including: an image receiving module configured to receive an image representing displayable content for display by an application; and a layout extraction model configured to use the image as input and to generate as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute; and an accessibility tree enrichment module configured to use the list of elements to add role attributes to nodes in an accessibility tree.

[0007] In some aspects, the technology described herein relates to a computing device including a processor and a memory configured with instructions for receiving an image representing displayable content for display by an application, executing a layout extraction model using the image as input, and generating as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute, the instructions further being for adding the role attribute to a node in an accessibility tree using the list of elements. [Brief explanation of the drawings]

[0008] [Figure 1A] Document Accessibility Scenario 100A is shown. [Figure 1B] 1 shows document accessibility scenario 100B. [Figure 1C] 10 shows images for a document accessibility scenario 100C, according to embodiments described throughout this disclosure. [Figure 1D] 1 illustrates displayable content including text according to an embodiment described in this disclosure. [Figure 1E] 1 illustrates an accessibility tree according to an embodiment described in this disclosure. [Figure 2] 2 illustrates a user device 202 according to an embodiment described throughout this disclosure. [Figure 3] 3 shows a flowchart 300 according to an embodiment described throughout this disclosure. [Figure 4A] 4 illustrates a method 400A according to embodiments described throughout this disclosure. [Figure 4B] 4 illustrates a method 400B according to embodiments described throughout this disclosure. [Figure 4C] 4 illustrates a method 400C according to embodiments described throughout this disclosure. [Figure 4D] 4 illustrates a method 400D according to embodiments described throughout this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0009] This disclosure describes a method for adding information representing elements from displayable content that were not previously present in the accessibility tree to an accessibility tree. The accessibility tree is used by accessibility software to provide access to the displayable content. If the accessibility tree associated with the displayable content is incomplete, the accessibility software cannot provide complete access to the displayable content. For example, image elements in web-viewable content may not contain information about their semantic function or content that can be grafted into the accessibility tree. In another example, a PDF file may not contain text that can be grafted into the accessibility tree. In either situation, the accessibility software cannot provide access to these image elements / documents.

[0010] This disclosure describes an accessibility infrastructure that receives an image representing displayable content for display by an application. A layout extraction model generates a list of elements of the image. Each element includes a bounding box that defines the portion of the image in which the element is located and a role attribute associated with the element. The accessibility infrastructure then adds the role attribute to a node in an accessibility tree. In some examples, the accessibility infrastructure may also add nodes or other attributes to the accessibility tree.

[0011] Many documents and types of viewable content are primarily designed to be presented to users in a visual way via software. Visual cues in many documents help users navigate the content. For example, subheadings, changes in text formatting, tables, or buttons all carry semantic meaning that helps users navigate the information. If this information is missing from the accessibility tree, navigating the viewable content becomes very difficult.

[0012] Content creators may need to take additional steps to create viewable content or documents that contain the necessary information needed to generate a complete accessibility tree. For example, in HTML, creators may include HTML attributes that are fully understood by the accessibility tree generation functions in the browser. Alternatively, creators may include Accessible Rich Internet Applications (ARIA) tags in the HTML to specify nodes or roles in the accessibility tree. However, many creators are unaware that they may need to take additional steps to ensure that their content contains the necessary information to generate a complete accessibility tree. Unfortunately, this means that some viewable content and documents are inaccessible to users via accessibility software.

[0013] 1A illustrates a document accessibility scenario 100A. The exemplary document accessibility scenario 100A shows two branches: a first branch provides displayable content 110 for visual display, and a second branch provides an accessible version of displayable content 110 via accessibility software 112.

[0014] Document accessibility scenario 100A begins with HTML code 102 that includes displayable content. In an embodiment, HTML code 102 may be executed on any web browser. Exemplary HTML code 102 generates an image of a puppy with a title, a button with the word "Donate!", and an image of a letter icon that is a link to an email address. However, HTML code 102 is not intended to be limiting. In an embodiment, HTML code 102 may display any possible content. Instead of HTML code 102, document accessibility scenario 100A may receive displayable content and any other format designed for display in an application running on a user device.

[0015] The browser may use the HTML code 102 to generate a document object model, DOM 104. DOM 104 is a data structure that represents the content indicated by the HTML code 102. In document accessibility scenario 100A, DOM 104 includes nodes for images, buttons, and links.

[0016] When presenting information visually, a browser may generate a cascading style sheet, CSS 103, from HTML code 102. CSS 103 may be used to generate a CCS object model that provides information about the layout of the displayed content (not shown).

[0017] The visual content processing module 106 may receive the DOM 104 and CSS 103 as input and generate a render tree. Layout software may then calculate the exact position and size of each object on the render tree. Finally, paint software may take the final render tree and generate pixels to visually display the content. In the example of FIG. 1A, a visible version of the displayable content 110 is displayed, including an image of a dog, a button with the text "Donate!", and an icon representing a letter that links to an email address.

[0018] The accessibility software may alternatively provide an accessible version of the content via the second branch of document accessibility scenario 100 A. The application providing access to the content (a browser in the example of document accessibility scenario 100 A) may make an API call to the accessibility infrastructure to provide an accessible version of the displayable content.

[0019] An application that provides access to content may use DOM 104 to generate an accessibility tree 108. Accessibility tree 108 is a data structure that maps content that can be navigated by accessibility software. Accessibility tree 108 includes nodes 114. In an embodiment, 114 may include more than one node. In an embodiment of document accessibility scenario 100A, accessibility tree 108 includes three nodes that all share a single parent node, the root node. However, this is not intended to be limiting. In an embodiment, an accessibility tree may include any amount of nodes 114 connected via any possible configuration of parent and child nodes. In an embodiment, the root node may include a document node.

[0020] Each node 114 of the accessibility tree 108 includes attributes 116. In embodiments, the attributes 116 may include any number of attributes 116. For example, each node 114 of the exemplary accessibility tree 108 shown in FIG. 1A includes two attributes: role and name. However, this is not intended to be limiting. In embodiments, each node 114 may include any amount of attributes 116. In embodiments, each node 114 may include a role attribute, a name attribute, a status attribute, and a description attribute. In embodiments, the nodes 114 of the accessibility tree 108 may include additional attributes, such as a parent node ID attribute, a child node ID attribute, a position attribute, etc. The nodes 114 may also include other metadata, including details about the text, position, how various nodes are represented, or any other characteristics.

[0021] The role attribute may describe a semantic role, for example, heading, button, table, paragraph, text box, combo box, list box, image, etc. In examples, the role attribute may include roles defined by any accessibility tree standard or definition, such as those defined by the ARIA standard. In further examples, the role attribute may include any classification of a group of elements in a display that has any semantic meaning.

[0022] In embodiments, the name attribute may relate to the content represented by the node. For example, the name may represent the text on a button or a clue as to what is depicted in an image. In embodiments where the role attribute of a node relates to text, the name field may contain the text.

[0023] The accessibility software 112 uses the accessibility tree 108 to generate an accessible version of the displayable content. In an example of the document accessibility scenario 100A, the accessibility software 112 generates an audible version of the content, which includes reading out the name attribute of each node followed by the role attribute, "puppy picture," "donate button," and "email link." However, the example of FIG. 1A is not intended to be limiting. In an example, other types of accessibility software may be used. In the case of a screen reader type of accessibility software 112, the screen reader may use additional attributes attached to each node 114 to provide additional audible cues.

[0024] Document accessibility scenario 100A provides an example of turning displayable content in the form of HTML code into visible and accessible content, but this is not intended to be limiting. In an example, document accessibility scenario 100A may convert any other format of displayable content into visible and / or accessible content.

[0025] The accessibility challenge when generating viewable content for display on a web browser is that developers need to use HTML tags and ARIA attributes in a way that matches the intent of the code. Lack of awareness and prioritization among developers, and failure to make accessibility a priority, are the main barriers to an accessible web and accessible content. Traditional assistive technologies often have problems accessing custom controls (e.g., a button with a custom class) that are used as a Layout elements such as the <html> tag) and common icon controls (e.g., the back button in a browser) are often overlooked. Traditional assistive technologies also do not generate accessibility trees for image-based documents such as PDFs.

[0026] This problem is illustrated in FIG. 1B by document accessibility scenario 100B. Document accessibility scenario 100B uses different HTML code 120 to generate substantially the same displayable content 110. HTML code 120 is an example of HTML code that does not include HTML tags or ARIA attributes to facilitate the creation of a complete accessibility tree by content creators. For example, HTML code 120 provides an unnamed image "abc.jpg" that provides a clue as to what the image file's contents are. Additionally, instead of an HTML button tag, HTML code 120 provides a button link that is an image of the button "def.gif." Finally, HTML code 120 provides an image "ghi.jpg" that is a link to an email address, without a link HTML tag.

[0027] As a result, DOM 122 generated from HTML code 120 looks very different from DOM 104. DOM 104 includes an image node, a button node, and a link node, while DOM 122 includes three image nodes. When DOM 122 is converted into accessibility tree 124, each of the three example elements in HTML code 120 has a role attribute of image and the name of each image file, which in this case is arbitrarily assigned. As a result, when accessibility software 112 generates an audible version of HTML code 120, it speaks "image," "image," and "image." Thus, the accessible version of HTML code 120 does not include all of the information available in viewable content 110. Therefore, document accessibility scenario 100A does not provide adequate accessibility to viewable content 110.

[0028] The document accessibility scenario 100C shown in FIG. 1C also provides access to the HTML code 120 from the document accessibility scenario 100B. However, in the document accessibility scenario 100C, access to the HTML code 120 is significantly improved by the inclusion of an accessibility infrastructure 126. The accessibility infrastructure 126 receives the image, generates bounding boxes around portions of the image that contain individual elements, and identifies role attributes of the individual components. This is visually illustrated in FIG. 1C. In the example of FIG. 1C, displayable content 110, including a web page display, is output from the visual content processing module 106 and received by the accessibility infrastructure 126. The displayable content 110 includes the same three elements shown in FIGS. 1A and 1B: an image of a dog, a contribution button, and a letter icon representing an email link. The accessibility infrastructure 126 generates at least one bounding box 130 within the image that includes the displayable content 110. In an example of document accessibility scenario 100C, accessibility infrastructure 126 identifies three instances of bounding box 130, represented by dotted lines, one bounding box 130 for each of the three elements. Next, accessibility infrastructure 126 determines a role attribute for each element defined by the bounding box. Finally, accessibility infrastructure 126 adds the role attribute to node 114 in accessibility tree 128. However, this is not intended to be limiting. In an example, adding the role attribute to node 114 may include modifying the role attribute. For example, a role attribute of "image" may be modified to "button."

[0029] 1B , thereby making the displayable content accessible to accessibility software 112. Accessibility software 112, which may be a screen reader, may receive accessibility tree 128 and list that the image includes a "puppy image," a "donate! button," and an "email link." Accessibility infrastructure 126 is described in further detail below.

[0030] 2 illustrates a user device 202 operable to perform the methods described herein. In embodiments, the user device 202 may include a laptop computer, a desktop computer, a handheld device such as a tablet, a mobile phone, a wrist-worn device such as a smartwatch, or any other user device that provides access to content, including images.

[0031] User device 202 includes a processor 204, a communication interface 206, and a memory 208. In embodiments, processor 204 may include multiple processors and memory 208 may include multiple memories. Processor 204 may be configured with instructions to execute the accessibility infrastructure described in this disclosure. The instructions may include non-transitory computer-readable instructions stored in and retrieved from memory 208.

[0032] The communication interface 206 of the user device 202 may be operable to facilitate communication between the user device 202 and a server or other computing device. In embodiments, the communication interface 206 may utilize a short-range wireless communication protocol such as BLUETOOTH, Wi-Fi, Zigbee, or any other wireless or wired communication method.

[0033] Memory 208 includes applications 210. In an example, the applications may include a web browser, such as Google Chrome, Firefox, Microsoft Edge, or any other web browser. In an example, the applications may include any application operable to display documents to a user, such as a PDF viewer, a word processor, etc. In an example, the applications may include any application operable to provide a user access to documents or viewable content.

[0034] The application 210 includes an accessibility infrastructure 212 that is operable to receive an image, identify elements in the image via bounding boxes, generate role attributes for the elements, and update the role attributes of nodes in an accessibility tree.

[0035] In an embodiment, accessibility infrastructure 212 may execute one or more software modules. In an embodiment, accessibility infrastructure 212 may execute any combination of image receiving module 222, layout extraction model 224, OCR model 226, icon model 228, other models 230, and / or accessibility tree expansion module 232. In the embodiment of FIG. 2, image receiving module 222, layout extraction model 224, OCR model 226, icon model 228, other models 230, and accessibility tree expansion module 232 are included in library 220 separate from application 210. In an embodiment, library 220 may include separate software that is available and executable by accessibility infrastructure 212, for example, via one or more API calls. However, the embodiment of FIG. 2 is not intended to be limiting. In an embodiment, any of image receiving module 222, layout extraction model 224, OCR model 226, icon model 228, other models 230, and accessibility tree expansion module 232 may be incorporated into the object code or executable code of accessibility infrastructure 212 or library 220.

[0036] In embodiments, any of image receiving module 222, layout extraction model 224, OCR model 226, icon model 228, other models 230, and accessibility tree expansion module 232 may execute on a server (not shown) available to accessibility infrastructure 212 via network and communication interface 206. If any of image receiving module 222, layout extraction model 224, OCR model 226, icon model 228, other models 230, and accessibility tree expansion module 232 may execute on a server, a user may be provided with controls that allow the user to make selections about which image data, among other displayable content data, may be sent to the server. Additionally, certain data may be processed in one or more ways before being stored or used such that user information is removed. For example, a user's identification information may be processed so that user information cannot be determined for the user, or, if location information is available, the user's geographic location may be generalized (such as to the city, zip code, or state level) so that the user's specific location cannot be determined from the image data or displayable content data submitted to the server. In this way, users can control what information is collected about them, how that information is used, and what information is provided to them.

[0037] 3 illustrates a flowchart 300. The flowchart 300 illustrates the execution of the accessibility infrastructure 126. The flowchart 300 begins when the application 210 sends an image 302 to the accessibility infrastructure 126. In an example embodiment, the image 302 may be received in response to determining that the accessibility tree of the displayable content 110 has a node 114 with a role of image or document. In an example embodiment, the image 302 may be received in response to determining that the attribute 116 is missing from the node 114 or that the node 114 has a null value for the attribute 116. For example, as described above with respect to the example accessibility tree 124, each of the three nodes includes an image without a name attribute. In an example embodiment in which a single document is received for display, such as a PDF document, the accessibility tree may include only a root node representing the document without any attributes. In an example embodiment, the image 302 may be received in response to a user command. For example, a user using a screen reader to navigate the displayable content 110 within the application 210 may suspect that the screen reader is missing information and may initiate the steps of flowchart 300 by user command.

[0038] The flowchart 300 illustrates the layout extraction model 224 receiving the image 302. For example, the layout extraction model 224 may receive the image 302 via the image receiving module 222. The image receiving module 222 is operable to receive the image from the application 210.

[0039] Image 302 may represent displayable content 110 for display by application 210. In an embodiment, image 302 may include any type of image file, including, but not limited to, PDF, JPEG, JPG, GIF, TIF, BMP, or any other image format that goes through at least one of rendering, layout, or painting processes for display on a computer display via visual content processing module 106. In an embodiment, displayable content 110 may include any combination of HTML elements designed for display within an application. For example, displayable content may include custom controls (e.g., having a custom class used as a button) Custom controls may include common icon controls (e.g., tags) and common icon controls (e.g., a browser's back button). Custom controls may include images, buttons, text boxes, tables, radio buttons, headings, paragraphs, links, etc. The examples provided are not intended to be limiting. In an embodiment, an image may contain any type of displayable content.

[0040] The layout extraction model 224 is operable to use the image 302 as input and to generate as output a list of elements 306 for the image 302. The list of elements 306 includes at least a bounding box 130 that defines a portion of the image 302 and a role attribute associated with the bounding box 130.

[0041] The layout extraction model 224 may include a machine learning model trained, for example, using supervised or semi-supervised training, to take an image 302 and identify a list of elements from the image 302, each element including a bounding box 130 and an associated role attribute.

[0042] In an embodiment, layout extraction model 224 may further receive DOM 304, and executing the layout extraction model may further include using the document object model as input. In an embodiment, DOM 304 may be generated by application 210. In an embodiment, DOM 304 may be used by layout extraction model 224 to generate list of elements 306.

[0043] In an embodiment, a role attribute may be determined to be text-related. Text-related attributes may include headings, paragraphs, words, combo boxes, list boxes, text boxes, static text, or any other text-related displayable content that primarily includes formatted text.

[0044] In response to determining that a portion of the image 302 includes a role attribute related to text, the layout extraction model 224 may run the OCR model 226 on the portion of the image 302 and analyze the output of the OCR model 226 to generate a text span. The text span may include a text string of characters.

[0045] In an embodiment, OCR model 226 may include a machine learning model trained, for example, using supervised or semi-supervised training, to receive a portion of image 302 and identify text within the portion of image 302.

[0046] In an embodiment, the OCR model 226 may output an OCR text tree. The OCR text tree may include a root node representing the image document. Child nodes of the root node may include paragraph nodes. Child nodes of the paragraph nodes may include sentence nodes. Child nodes of the sentence nodes may include word nodes. Child nodes of the word nodes may include character nodes. Metadata may also be included in the nodes of the OCR text tree, including information about the position of the text and the formatting of the text.

[0047] The OCR text tree is not in a format that can be spliced ​​into the accessibility tree 312. Therefore, in an embodiment, the layout extraction model 224 may perform further processing on the OCR text tree to generate text spans. In an embodiment, analyzing the OCR text tree output by the OCR model 226 to generate text spans may further include determining formatting to apply to consecutive character groups in the output of the OCR model. Text spans may then be set for consecutive character groups that have the same formatting.

[0048] 1D shows exemplary displayable content 140. Displayable content 140 may include a PDF file displayed on a web browser. Displayable content 140 includes a title, three headings, and respective paragraphs under each of the three headings. Layout extraction model 224 has drawn bounding box 130 around the entire region of displayable content 140 that includes text, thereby capturing substantially all of displayable content 140. However, in an embodiment, layout extraction model 224 may instead draw bounding boxes around each or any adjacent combination of the title, the three headings, and the three paragraphs.

[0049] In an embodiment, OCR model 226 may attempt to identify text within bounding box 130 by searching for text starting in the upper left corner and sweeping to the right along a surface region, such as a line, and repeating this action for successive lines below until OCR model 226 reaches the bottom of bounding box 130. In this manner, OCR model 226 may emulate the reading of English, although this is not intended to be limiting. For other languages, OCR model 226 may evaluate the regions of bounding box 130 in a different order, for example, from right to left for Hebrew. In an embodiment, OCR model 226 may evaluate the region represented by bounding box 130 for text using any other method.

[0050] The OCR text tree output by OCR model 226 for displayable content 140 may include the text represented by displayable content 140 along with metadata identifying the formatting of the text. The formatting may include, for example, font size, font type, font color, font style such as bold, italic, underline, strikethrough, subscript, and superscript. Thus, by traversing the OCR text tree, layout extraction model 224 may evaluate all text found within bounding box 130, starting from the upper left corner and sweeping to the right, on subsequent lines to the bottom of bounding box 130.

[0051] In an embodiment, layout extraction model 224 may determine the formatting to apply to consecutive groups of characters as it traverses the OCR text tree. For example, the title of displayable content 140, "A Guide to British Cheese Varieties," includes consecutive groups of characters that all have the same formatting. Subsequent lines of text in displayable content 140 have different formatting. Thus, the characters of the title may be grouped together by layout extraction model 224 into a single span of text for inclusion in accessibility tree 312, separate from other text found in displayable content 140.

[0052] In an embodiment, layout extraction model 224 may set a name attribute on the text span. In an embodiment, accessibility infrastructure 212 may access the name attribute to share the content of the text span with a user. However, in other embodiments, layout extraction model 224 may generate a child node from node 310. Node 310 may have a role attribute of heading, paragraph, or any other attribute associated with the text. The child node may have a role attribute of static text and a name attribute set on the text span.

[0053] 1E shows an exemplary accessibility tree 132 generated by accessibility infrastructure 212 based on displayable content 140. Accessibility tree 132 includes a root node 152 with a role attribute of document. In an embodiment, root node 152 may instead include a role attribute of image.

[0054] Root node 152 includes child nodes 154. Child nodes 154 may include text-related role attributes. In an example, child node 154 may have a role attribute of title or static text. In an example, child node 154 may have a name attribute of "A Guide to British Cheese Varieties."

[0055] Child node 154 includes three further child nodes 156. Each child node 156 may include a respective text-related role attribute, such as heading or static text. In an example, child nodes 156 may include name attributes of "Stilton," "Cornish Yarg," and "Red Leicester."

[0056] Each child node 156 may include a respective child node 158. In an example, each child node 158 may include a respective text-related role attribute, such as paragraph or static text, in an example. In an example, each name attribute of child nodes 158 may include the contents of the paragraph found under the heading of the respective parent node in FIG. 1D .

[0057] In an embodiment, child nodes 158 may each include a respective child node 160. In an embodiment, child nodes 160 may each include a respective role attribute of static text and a respective name attribute that includes the content of the paragraph under the heading of the respective parent node in Figure 1D. If child node 160 includes a name attribute with the content of the paragraph in Figure 1D, child node 158 may not include a name attribute with the content of the paragraph in Figure 1D.

[0058] In an example, OCR model 226 may determine text-related role attributes of displayable content 140, such as title, heading, or paragraph. In an example, layout extraction model 224 may determine text-related role attributes of displayable content 140 based on image 302. In an example, layout extraction model 224 may determine text-related role attributes of displayable content 140 based on metadata found in the OCR text tree.

[0059] In embodiments, the order and / or placement of nodes in accessibility tree 132 may be determined based on at least one of the order in which each node is encountered when traversing the OCR text tree, the role attribute assigned to each node, and / or the OCR model metadata associated with each node.

[0060] In an embodiment, the role attributes may be determined to include icon-related attributes. The icon-related attributes may include, for example, an envelope symbol for an email link, a back arrow for navigating back to a previous website, the letter "i" for a link to information, etc. In response to determining that a portion of an image includes icon-related attributes, the layout extraction model 224 may run the icon model 228 on that portion of the image 302 to identify role attributes associated with the icon-related attributes. For example, FIGS. 1A and 1B provide an example of an envelope icon that is a link to an email address.

[0061] In an embodiment, the layout extraction model 224 may determine that the role attribute includes a button. In an embodiment, the layout extraction model 224 may identify text within a portion of the image. The text may be used to set the name attribute 308.

[0062] In an embodiment, the layout extraction model 224 may determine that the attribute includes a navigation bar, a toolbar, or any other control or widget.

[0063] In an embodiment, the layout extraction model 224 may call other models 230 to identify other elements in the image 302 and add those elements to the list of elements.

[0064] In an embodiment, the role attribute may be a first role attribute, and the list of elements may further include a second role attribute. In an embodiment, the layout extraction model 224 may generate a node 310. In an embodiment, the node may be a first node associated with a first role attribute, and the accessibility tree may further include a second node associated with a second role attribute.

[0065] In some embodiments, layout extraction model 224 may be unable to determine the role attribute of an element. In some embodiments where there is no role attribute for an element identified in image 302, layout extraction model 224 may not add the element to list of elements 306 or may not add the element to accessibility tree 312.

[0066] After the layout extraction model 224 is executed, the accessibility tree extension module 232 receives the list of elements 306. In an embodiment, the accessibility tree extension module 232 may further receive any combination of name attributes 308 and / or nodes 310. The accessibility tree extension module 232 adds a role attribute to at least one node in the accessibility tree 312. In an embodiment, the accessibility tree extension module 232 may also add the nodes 310 generated by the layout extraction model 224 to the accessibility tree 312.

[0067] In embodiments, the accessibility tree extension module 232 may splice the node 310 into the accessibility tree 312 based on a role attribute of the node 310. In embodiments, the location at which the node 310 is spliced ​​into the accessibility tree 312 relative to other nodes may be determined based on one or more attributes of the other nodes. In embodiments, the location at which the node 310 is spliced ​​into the accessibility tree 312 relative to other nodes may be determined based on metadata associated with the list of elements 306. In embodiments, the metadata may be received from the OCR model 226 or the layout extraction model 224. In embodiments, the metadata may indicate the location and / or dimensions of the bounding box 130 or text associated with the element. In embodiments, the metadata may include font and formatting information associated with the element, or any other clues operable to provide semantic and / or layout information related to relationships between elements in the image 302.

[0068] In an embodiment, the layout extraction model 224 may evaluate the image 302 and generate a new accessibility tree 132 associated with the image 302. The new accessibility tree 132 may then be spliced ​​into the previous accessibility tree at the image node associated with the image 302.

[0069] 4A illustrates a method 400A according to an embodiment. The method 400A is operable to determine a list of elements of an image, the list of elements including a bounding box defining a portion of the image and a role attribute. The method 400A is further operable to add the role attribute to a node in an accessibility tree.

[0070] Method 400A begins at step 402. In step 402, an image 302 representing displayable content 110 is received for display by application 210, as described above.

[0071] The method 400A continues with step 406, where the layout extraction model 224 is executed using the image 302 as input, as described above, to generate a list 306 of elements that includes at least a bounding box 130 defining a portion of the image 302 and a role attribute.

[0072] The method 400A continues at step 408. At step 408, a role attribute is added to the node in the accessibility tree 312, as described above.

[0073] In an embodiment, the method 400A may further include steps 404 and 410. In step 404, the DOM 304 is received, as described above.

[0074] In step 410, nodes 310 are added to the accessibility tree 312 based on role attributes, as described above.

[0075] Figure 4B illustrates method 400B, Figure 4C illustrates method 400C, and Figure 4D illustrates method 400D. In an embodiment, step 406 of method 400A may further include any of the steps of methods 400B, 400C, and 400D.

[0076] Method 400B begins at step 420. At step 420, it is determined whether a role attribute is associated with the text, as described above.

[0077] If the answer to step 420 is yes, then step 422 is performed, in which an OCR model 226 is run on a portion of the image 302, as described above.

[0078] The method 400B may continue at step 424. At step 424, the output of the OCR model is analyzed to generate text spans, as described above.

[0079] Method 400B may continue at step 426, where formatting to be applied to groups of consecutive characters in the output of the OCR model is determined, as described above.

[0080] The method 400B may continue at step 428. At step 428, a text span may be set to a contiguous group of characters, as described above.

[0081] The method 400B may continue at step 430. At step 430, a name attribute may be set for the text span, as described above.

[0082] Method 400C begins at step 430. In step 430, it is determined whether the role attributes include icon-related attributes, as described above.

[0083] If the answer to step 430 is yes, then step 432 is performed, in which the icon model 228 is run against a portion of the image to generate role attributes associated with the icon-related attributes, as described above.

[0084] Method 400D begins at step 440. In step 440, it is determined whether the role attribute includes a button, as described above.

[0085] If step 440 evaluates to yes, method 400D may continue to step 442. In step 442, text within the portion of the image may be identified, as described above.

[0086] In an embodiment, method 400D may continue to step 444. In step 444, text may be added to the name attribute of node 310 in accessibility tree 312, as described above.

[0087] In embodiments, the methods described herein may include additional steps. For example, after adding at least one of a role attribute or a node associated with an element to accessibility tree 312, accessibility infrastructure 212 may execute user commands using accessibility tree 312, such as finding an element within a page, selecting an element, copying an element, etc. In embodiments, accessibility infrastructure 212 may use the enhanced accessibility tree 312 to perform any function that accessibility software typically performs.

[0088] The methods described herein may enable displayable content to be included in the accessibility tree where it was not previously available. The described methods are independent of any particular user application to run. In particular, when incorporated into a library accessible to user applications, the methods may enable any accessibility software to improve access to displayable content over previous rule-based models.

[0089] Various embodiments of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Various embodiments of the systems and techniques described herein may be implemented as, and / or generally referred to herein as, circuits, modules, blocks, or systems that may combine software and hardware aspects. For example, a module may include functions / acts / computer program instructions executed on a processor or some other programmable data processing apparatus.

[0090] Some of the above examples are described as a process or method that is depicted as a flowchart. While the flowchart describes operations as sequential, many of the operations may be performed in parallel, concurrently, or simultaneously. The order of operations may also be rearranged. A process may be terminated when its operations are completed, but may have additional steps not included in the drawings. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0091] The above methods, some of which are illustrated by flowcharts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) may perform the necessary tasks.

[0092] Specific structural and functional details disclosed herein are presented solely for purposes of illustrating examples, however, examples may be embodied in many alternative forms and should not be construed as limited to only the embodiments set forth herein.

[0093] Terms such as first, second, etc. may be used herein to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of exemplary embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0094] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms a, an, and the are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will be further understood that the terms includes, comprising, includes, and / or including, when used herein, specify the presence of stated features, components, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, components, steps, operations, elements, components, and / or groups thereof.

[0095] It should also be noted that in some alternative implementations, the functions / acts shown may occur out of the order noted in the figures. For example, two figures shown in succession may, in fact, be executed concurrently or may sometimes be executed in the reverse order, depending upon the functions / acts involved.

[0096] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary embodiments belong. For example, terms such as those defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly defined as such herein.

[0097] Portions of the above exemplary embodiments and corresponding Detailed Description are presented in terms of software, or algorithms and symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the ways in which those skilled in the art effectively convey the substance of their work to others skilled in the art. An algorithm, as the term is used herein, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0098] In the exemplary embodiments described above, references to symbolic representations of actions and operations that may be implemented as program modules or functional processes (e.g., in the form of flowcharts) include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types, which may be described and / or implemented in existing structural elements and using existing hardware. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application specific integrated circuits, field programmable gate array (FPGA) computers, etc.

[0099] It should be recognized, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise specified, or as will be apparent from the description, terms such as processing or computing or determining to calculate or display refer to the actions and processes of a computer system or similar electronic computing device that manipulates and converts data, represented as physical quantities, electronic quantities in the computer system's registers and memory, into other data that is similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0100] It should also be noted that the software-implemented aspects of the exemplary embodiments are typically encoded on some form of non-transitory program storage medium or implemented via some type of transmission medium. The program storage medium may be magnetic (e.g., a floppy disk or hard drive) or optical (e.g., a compact disk read-only memory, or CD ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cable, optical fiber, or some other suitable transmission medium known in the art. Examples are not limited by these aspects of any given example.

[0101] Finally, it should also be noted that while the appended claims set forth particular combinations of features described herein, the scope of the disclosure is not limited to the particular combinations claimed below, but rather is extended to encompass any combination of features or embodiments disclosed herein, regardless of whether that particular combination is specifically recited in the appended claims at this time.

[0102] In some aspects, the technology described herein relates to a method that further includes determining that a role attribute is associated with text, and in response to determining that a portion of the image is associated with text, running an OCR model on the portion of the image, analyzing the output of the OCR model to generate a text span, and setting a name attribute on the text span.

[0103] In some aspects, the technology described herein relates to a method in which analyzing the output of an OCR model to generate a text span includes determining formatting to apply to consecutive character groups in the output of the OCR model and setting the text span to the consecutive character groups.

[0104] In some aspects, the technology described herein relates to a method further including determining that role attributes include icon-related attributes, and, in response to determining that a portion of the image includes the icon-related attributes, executing an icon model on the portion of the image to generate role attributes associated with the icon-related attributes.

[0105] In some aspects, the technology described herein relates to a method further including determining that the role attribute includes a button.

[0106] In some aspects, the technology described herein relates to a method further including identifying text within a portion of an image and adding the text to a name attribute of a node in an accessibility tree.

[0107] In some aspects, the technology described herein relates to a method, wherein the role attribute is a first role attribute and the list of elements further includes a second role attribute.

[0108] In some aspects, the technology described herein relates to a method further including adding nodes based on a role attribute.

[0109] In some aspects, the technology described herein relates to a method in which an image is received in response to determining that an accessibility tree for displayable content displayed by an application has an image node or a document node that is missing an attribute.

[0110] In some aspects, the technology described herein relates to a method, wherein the image comprises an image-based Portable Document Format document.

[0111] In some aspects, the technology described herein relates to a method in which an image is received in response to a user command.

[0112] In some aspects, the techniques described herein relate to a method further comprising receiving a document object model, and wherein executing the layout extraction model further comprises using the document object model as input.

[0113] In some aspects, the technology described herein relates to a method, wherein the method is performed on a browser extension or a browser plug-in.

[0114] In some aspects, the technology described herein relates to a system where the layout extraction model determines that a role attribute is associated with text, and where, in response to determining that a portion of the image is associated with text, the system is further configured to run an OCR model on the portion of the image, analyze the output of the OCR model to generate a text span, and set a name attribute on the text span.

[0115] In some aspects, the technology described herein relates to a system in which the layout extraction model is further configured to determine formatting to be applied to consecutive character groups in the output of the OCR model and to set text spans to consecutive character groups.

[0116] In some aspects, the technology described herein relates to a system where the layout extraction model determines that the role attributes include icon-related attributes, and in response to determining that a portion of the image includes icon-related attributes, is further configured to run an icon model on the portion of the image to generate role attributes associated with the icon-related attributes.

[0117] In some aspects, the technology described herein relates to a system, wherein the layout extraction model is further configured to determine that the role attribute includes a button.

[0118] In some aspects, the technology described herein relates to a system in which the layout extraction model is further configured to identify text within a portion of the image and add the text to a name attribute of a node in an accessibility tree.

[0119] In some aspects, the technology described herein relates to a system in which the role attribute is a first role attribute and the list of elements further includes a second role attribute.

[0120] In some aspects, the technology described herein relates to a system in which the accessibility tree enrichment module is further configured to add nodes based on role attributes.

[0121] In some aspects, the technology described herein relates to a system in which an image is received in response to determining that an accessibility tree of displayable content displayed by an application has an image node or a document node that is missing an attribute.

[0122] In some aspects, the technology described herein relates to a system in which the image comprises an image-based Portable Document Format document.

[0123] In some aspects, the technology described herein relates to a system in which an image is received in response to a user command.

[0124] In some aspects, the technology described herein relates to a system, wherein the system further includes a document object model receiving module, and wherein the layout extraction model further uses the document object model to determine the list of elements.

[0125] In some aspects, the technology described herein relates to a system in which the layout extraction model runs on a browser extension or browser plug-in.

[0126] In some aspects, the technology described herein relates to a computing device, wherein the memory is further configured with instructions to determine that a role attribute is associated with text, and in response to determining that a portion of the image is associated with text, run an OCR model on the portion of the image, analyze the output of the OCR model to generate a text span, and set a name attribute on the text span.

[0127] In some aspects, the techniques described herein relate to a computing device where analyzing the output of the OCR model to generate a text span includes determining formatting to apply to consecutive character groups in the output of the OCR model, and the memory is further configured with instructions for setting the text span to the consecutive character groups.

[0128] In some aspects, the technology described herein relates to a computing device, wherein the memory is further configured with instructions to determine that the role attributes include icon-related attributes, and in response to determining that the portion of the image includes the icon-related attributes, execute an icon model on the portion of the image to generate role attributes associated with the icon-related attributes.

[0129] In some aspects, the technology described herein relates to a computing device, wherein the memory is further configured with instructions for determining that the role attribute includes a button.

[0130] In some aspects, the technology described herein relates to a computing device, wherein the memory is further configured with instructions to identify text within a portion of an image and add the text to a name attribute of a node in an accessibility tree.

[0131] In some aspects, the technology described herein relates to a computing device, wherein the role attribute is a first role attribute and the list of elements further includes a second role attribute.

[0132] In some aspects, the technology described herein relates to a computing device, wherein the memory is further configured with instructions for adding nodes based on role attributes.

[0133] In some aspects, the techniques described herein relate to a computing device in which an image is received in response to determining that an accessibility tree of displayable content displayed by an application has an image node or a document node that is missing an attribute.

[0134] In some aspects, the technology described herein relates to a computing device, wherein the image comprises an image-based Portable Document Format document.

[0135] In some aspects, the technology described herein relates to a computing device in which an image is received in response to a user command.

[0136] In some aspects, the techniques described herein relate to a computing device, wherein the memory is further configured with instructions to receive a document object model, and wherein executing the layout extraction model further includes using the document object model as input.

[0137] In some aspects, the technology described herein relates to a computing device in which the instructions are executed in a browser extension or a browser plug-in.

Claims

1. 1. A method comprising: receiving an image representing displayable content for display by an application; and executing a layout extraction model using the image as input and generating as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute, the method further comprising: adding the role attribute to a node in an accessibility tree using the list of elements.

2. determining that the role attribute is associated with text; 10. The method of claim 1, further comprising: in response to determining that the portion of the image is associated with text, running an OCR model on the portion of the image; analyzing the output of the OCR model to generate a text span; and setting a name attribute on the text span.

3. Analyzing the output of the OCR model to generate the text spans includes: determining formatting to be applied to groups of consecutive characters in the output of the OCR model; setting said text span to said contiguous group of characters; The method of claim 2 , comprising:

4. determining that the role attributes include icon-related attributes; responsive to determining that the portion of the image includes the icon-related attribute, executing an icon model on the portion of the image to generate the role attribute associated with the icon-related attribute; The method of claim 1 further comprising:

5. determining that the role attribute includes a button; identifying text within the portion of the image; adding the text to a name attribute of the node in the accessibility tree; The method of claim 1 further comprising:

6. The method of any one of claims 1 to 5, further comprising adding the nodes based on the role attribute.

7. 7. The method of claim 1, wherein the image is received in response to determining that the accessibility tree of the displayable content displayed by the application has an image node or a document node that is missing an attribute.

8. The method of claim 7 , wherein the image comprises an image-based Portable Document Format document.

9. A method according to any preceding claim, wherein the image is received in response to a user command.

10. The method of any one of claims 1 to 9, further comprising receiving a document object model, and wherein executing the layout extraction model further comprises using the document object model as input.

11. The method according to any one of claims 1 to 10, wherein the method is executed on a browser extension or a browser plug-in.

12. 1. A system comprising: an image receiving module configured to receive an image representing displayable content for display by the application; a layout extraction model configured to use the image as input and to generate as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute; an accessibility screen module configured to execute the layout extraction model; an accessibility tree enrichment module configured to add the role attributes to nodes in an accessibility tree using the list of elements; A system comprising:

13. 13. The system of claim 12, wherein the layout extraction model is further configured to, in response to determining that the role attribute is associated with text and determining that the portion of the image is associated with text, run an OCR model on the portion of the image, analyze output of the OCR model to generate a text span, and set a name attribute on the text span.

14. 14. The system of claim 13, wherein the layout extraction model is further configured to determine formatting to be applied to groups of consecutive characters in the output of the OCR model and to set the text spans to the groups of consecutive characters.

15. 13. The system of claim 12, wherein the layout extraction model is further configured to: determine that the role attributes include icon-related attributes; and, in response to determining that the portion of the image includes the icon-related attributes, run an icon model on the portion of the image to generate the role attributes associated with the icon-related attributes.

16. 13. The system of claim 12, wherein the layout extraction model is further configured to determine that the role attribute includes a button, identify text within the portion of the image, and add the text to a name attribute of the node in the accessibility tree.

17. The system of any one of claims 12 to 16, wherein the role attribute is a first role attribute and the list of elements further includes a second role attribute.

18. The system of any one of claims 12 to 17, wherein the accessibility tree enrichment module is further configured to populate the nodes based on the role attribute.

19. 19. The system of claim 12, wherein the image is received in response to determining that the accessibility tree of the displayable content displayed by the application has an image node or a document node that is missing an attribute.

20. 20. The system of claim 19, wherein the image comprises an image-based Portable Document Format document.

21. A system according to any one of claims 12 to 20, wherein the image is received in response to a user command.

22. 22. The system of claim 12, further comprising a document object model receiving module, wherein the layout extraction model further uses the document object model to determine the list of elements.

23. 1. A computing device comprising: a processor; a memory configured with instructions, the instructions comprising: receiving an image representing displayable content for display by the application; instructions for executing a layout extraction model using the image as input to generate as output a list of elements of the image, the list of elements including at least a bounding box defining a portion of the image and a role attribute, the instructions further comprising: instructions for adding the role attribute to a node in an accessibility tree using the list of elements.

Images