Generating view-based accessibility information using large language models

Abstract

A computing system uses an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface associated with an application executing at a computing device. The information associated with the plurality of user interface elements may be generated by the application programming interface. The computing system converts the information into structured text. The computing system generates a summary of content included in the graphical user interface by applying at least the machine learning model to the structured text using a machine learning model, where the machine learning model is a large language model. The computing system then sends the summary of the content included in the graphical user interface to the computing device, where the summary of the content includes a description including a general view structure and semantic information associated with the graphical user interface.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 587,018, filed on September 29, 2023, which is incorporated herein by reference in its entirety. Background Technology

[0003] Accessibility services are designed to assist users with disabilities in interacting with devices and applications. For example, accessibility services can provide visually impaired users with the ability to easily understand content presented to them on a screen. However, for many use cases within any given device space, accessibility information is not always readily available. For instance, users often need to interact with every individual element across the entire screen to receive information about the content presented to them, which can be time-consuming and difficult, especially when the screen layout is unknown or unintuitive. Therefore, there is a need to improve accessibility services so that users with disabilities can receive a more user-friendly experience when interacting with devices and applications. Summary of the Invention

[0004] In general, the technology disclosed herein relates to techniques for providing users with a summary of content included in a graphical user interface by applying a large language model to accessibility information associated with an application. A computing device (e.g., a smartphone or other user computing device) may include an accessibility service (e.g., an accessibility service) that includes an application programming interface (API) configured to collect view accessibility information from one or more applications executing on the computing device. In some examples, the accessibility service may analyze content displayed by the computing device in response to triggered user input (e.g., mechanically (such as by pressing a button), by gesture recognition / control (such as triple-tapping on the screen), by audio input (such as verbal commands), etc.) or automatically (so that no user input is required, provided that the user has given explicit permission for the accessibility service to analyze the content of the screen without such user input). In some examples, the accessibility information may be generated by the API, such that the application remains user-accessible even when the developer does not provide the accessibility information. A computing system communicating with the computing device may use the API to retrieve the accessibility information. Accessibility information can be associated with multiple user interface elements included in a graphical user interface (GUI) that is associated with an application running on a computing device. The computing system can then convert the accessibility information into structured text (e.g., an Extensible Markup Language (XML) structure) and apply a machine learning model (e.g., a Large Language Model (LLM)) to this structured text. The machine learning model can interpret the structured text and generate a summary of the content included in the GUI associated with the application running on the computing device. This content summary can be sent from the computing system to the computing device. In some examples, the content summary can be presented to the user via text-to-speech synthesis technology, allowing the user to quickly and easily understand the elements presented to them.

[0005] In one example, this disclosure relates to a method comprising retrieving information associated with a plurality of user interface elements included in a graphical user interface by a computing system using an application programming interface. The method further comprises: converting the information into structured text by the computing system; and generating a summary of the content included in the graphical user interface by the computing system using a machine learning model, by applying at least the machine learning model to the structured text, wherein the machine learning model is a large language model. The method further comprises outputting a summary of the content included in the graphical user interface by the computing system.

[0006] In another example, this disclosure relates to a computing system including one or more processors and one or more storage devices storing instructions. When executed by the one or more processors, the instructions cause the one or more processors to use an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface. The instructions further cause the one or more processors to: convert the information into structured text and generate a summary of the content included in the graphical user interface by applying a machine learning model, at least to the structured text, wherein the machine learning model is a large language model. The instructions further cause the one or more processors to output a summary of the content included in the graphical user interface.

[0007] In another example, this disclosure relates to a non-transitory computer-readable storage medium encoded with instructions. When executed by one or more processors, these instructions cause the processors to use an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface. The instructions further cause the processors to: convert the information into structured text and generate a summary of the content included in the graphical user interface by applying a machine learning model, at least to the structured text, wherein the machine learning model is a large language model. The instructions further cause the processors to output a summary of the content included in the graphical user interface.

[0008] In another example, this disclosure relates to a computer program product for generating a summary of content included in a graphical user interface, the computer program product comprising one or more instructions. When executed by at least one processor, the one or more instructions cause the at least one processor to use an application programming interface to retrieve information associated with a plurality of user interface elements included in the graphical user interface. The one or more instructions further cause the at least one processor to: convert the information into structured text, and generate a summary of the content included in the graphical user interface by applying a machine learning model, at least to the structured text, wherein the machine learning model is a large language model. The one or more instructions further cause the at least one processor to output a summary of the content included in the graphical user interface.

[0009] Details of one or more examples of this disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0010] Figure 1 This is a conceptual diagram illustrating an example computing system for providing a user with a summary of the content included in a graphical user interface, based on one or more techniques according to this disclosure.

[0011] Figure 2 This is a block diagram illustrating another example computing system configured to apply machine learning modules to accessibility information according to one or more techniques according to this disclosure.

[0012] Figure 3 This is a conceptual diagram illustrating a machine learning module configured to apply a language model to accessibility information according to one or more techniques disclosed herein.

[0013] Figure 4 This is a conceptual diagram illustrating text-to-speech synthesis using one or more techniques according to this disclosure for providing a user with a summary of content included in a graphical user interface.

[0014] Figure 5 This is a flowchart illustrating example operations for providing a user with a summary of content included in a graphical user interface, according to one or more techniques of this disclosure. Detailed Implementation

[0015] Figure 1 This is a conceptual diagram illustrating an example computing system for providing a summary of content included in a graphical user interface to a user, based on one or more technologies according to this disclosure. Figure 1 In one example, user 120 interacts with computing device 112, which communicates with computing system 100. In some examples, some or all of the components and / or functionalities attributable to computing system 100 may be implemented or performed by computing device 112. Although Figure 1 While not explicitly shown in the examples, computing system 100 can be implemented on multiple computing devices, including but not limited to portable devices, mobile devices, or other devices such as mobile phones (including smartphones), laptops, desktop computers, tablet computers, smart TV platforms, server computers, mainframes, etc. In some examples, computing system 100 can represent a cloud computing system that provides one or more services via network 101. That is, in some examples, computing system 100 can be a distributed computing system.

[0016] As described above, some or all of the components and / or functionalities attributable to computing system 100 may be implemented or performed by computing device 112. Computing system 100 may communicate with computing device 112 via network 101. Network 101 may include any public or private communication network, such as a cellular network, Wi-Fi network, direct cellular-to-satellite communication network, or other types of networks used for transmitting data between computing system 100 and computing device 112. In some examples, network 101 may represent one or more packet-switched networks, such as the Internet. Computing device 112 may use any suitable communication technology to send data to and receive data from computing system 100 across network 101. For example, computing system 100 and computing device 112 may each be operatively coupled to network 101 using a corresponding network link. Network 101 may include network hubs, network switches, network routers, etc., which are operatively coupled to each other, thereby providing information exchange between computing device 112 and computing system 100. In some examples, the network link of network 101 may be Ethernet, ATM, or other network connections. Such connections may include wireless and / or wired connections, including satellite network connections.

[0017] like Figure 1 As shown in the example, computing device 112 includes one or more user interface (UI) components (“UI component 102”). The UI component 102 of computing device 112 can be configured to serve as an input device and / or output device of computing device 112. UI component 102 can be implemented using various technologies. For example, UI component 102 can be configured to receive input from user 120 via haptic, audio, and / or video feedback. Examples of input devices include presence-sensitive displays, presence-sensitive or touch-sensitive input devices (such as...) Figure 1 (as shown), a mouse, keyboard, voice response system, video camera, microphone, or any other type of device for detecting commands from user 120. In some examples, presence-sensitive displays include touch-sensitive or presence-sensitive input screens, such as resistive touchscreens, surface acoustic wave touchscreens, capacitive touchscreens, projected capacitive touchscreens, pressure-sensitive screens, acoustic pulse recognition touchscreens, or other presence-sensitive technologies. That is, the UI component 102 of computing device 112 may include presence-sensitive devices that can receive tactile input from user 120. UI component 102 can receive indications of tactile input by detecting one or more gestures from user 120, for example, when user 120 touches or points at one or more locations of UI component 102 with a finger or stylus.

[0018] UI component 102 may additionally or alternatively be configured to function as an output device by providing output to user 120 through tactile, audio, or video stimuli. Examples of output devices include a sound card, a video graphics adapter card, or any of one or more display devices, such as a liquid crystal display (LCD), a dot matrix display, a light-emitting diode (LED) display, a microLED, a miniLED, an organic light-emitting diode (OLED) display, electronic ink, or a similar monochrome or color display capable of outputting visual information to user 120. Additional examples of output devices include speakers, tactile devices, or other devices capable of generating understandable output for the user. For example, UI component 102 may present output to user 120 as a graphical user interface, which may be associated with functionality provided by computing device 112. In this way, UI component 102 may present various user interfaces of applications (e.g., email messaging applications, internet browser applications, etc.) that are executed at or accessible by computing device 112. User 120 may interact with the corresponding user interface of an application to cause computing device 112 to perform operations related to the functionality provided by the application.

[0019] In some examples, the UI component 102 of computing device 112 can detect two-dimensional and / or three-dimensional gestures as input from user 120. For example, sensors of UI component 102 can detect user movement (e.g., movement of hand, arm, pen, stylus, etc.) within a threshold distance of the sensors of UI component 102. UI component 102 can determine a two-dimensional or three-dimensional vector representation of the movement and associate that vector representation with a gesture input having multiple dimensions (e.g., waving, pinching, clapping, swiping, etc.). In other words, in some examples, UI component 102 can detect multidimensional gestures without the user making a gesture on or near the screen or surface where UI component 102 outputs information for display. Instead, UI component 102 can detect multidimensional gestures performed at or near a sensor, which may or may not be located near the screen or surface where UI component 102 outputs information for display.

[0020] exist Figure 1 In the example, computing system 100 includes a user interface (UI) module 104. Module 104 may perform the operations described herein using hardware, software, firmware, or a combination thereof residing in and / or executing on computing system 100. Computing system 100 may utilize one or more processors to execute module 104. In some examples, computing system 100 may execute module 104 as a virtual machine executing on the underlying hardware. Module 104 may execute as one or more services of an operating system or computing platform, or as one or more executable programs at the application layer of the computing platform.

[0021] like Figure 1 As shown in the example, UI module 104 can be operated by computing system 100 to perform one or more functions, such as receiving input and sending instructions on this input to other components associated with computing system 100. UI module 104 can also receive data from components associated with computing system 100. Using the received data, UI module 104 can cause other components associated with computing system 100 (such as UI component 102) to provide output based on that data. For example, UI module 104 can send data to UI component 102 of computing device 112 to display a GUI, such as GUI 116.

[0022] like Figure 1 As shown in the example, GUI 116 is an example representation of a time application. GUI 116 includes multiple user interface elements. For example, GUI 116 includes a button 118A for "Alarm," a button 118B for "Clock," a button 118C for "Timer," and a button 118D for "Stopwatch." User 120 can switch between the various buttons included in GUI 116 to access different functionalities and / or user interfaces provided by the application. Figure 1In the example, GUI 116 represents a user interface for the "clock" functionality, as shown by the highlighted button 118B. GUI 116 includes the word "clock" in the upper left corner, the time "12:17" below "clock," and the date "March 31, Friday (Fri, 31 Mar)" below the time. At the center of GUI 116 is a "Summarize" button 114, which user 120 can interact with to generate a summary of the content included in GUI 116. For example, computing device 112 can receive a gesture indication detected at the summarize button 114 from user 120, where the gesture indication is manually provided by user 120 by tapping the screen. In some examples, the gesture indication can be audible input, whereby the gesture is provided by user 120 via, for example, a voice command. For example, the user can speak the command "summarize." The instruction can then be sent to computing system 100, which can then execute the techniques described herein for providing a summary of the content included in GUI 116 to user 120. In some examples, the gesture instruction is provided by user 120 using gesture controls, such as by providing the aforementioned gestures (e.g., waving, pinching, clapping, swiping, etc.) or by tapping the screen in some way (e.g., triple-tapping the screen). Therefore, the techniques described herein for providing a summary of the content included in GUI 116 to user 120 can be executed by computing system 100 in response to various gesture instructions. In this way, the user does not need to perform a specific gesture to receive the application's content summary, which can make the application more accessible and user-friendly.

[0023] As mentioned above, accessibility for all users, including those with disabilities, is a critical aspect of modern technology because it aims to ensure that digital experiences are inclusive and user-friendly. In most applications, developers must hard-code accessibility features to create products and services suitable for users with visual, hearing, motor, and cognitive impairments. For example, an operating system may have an accessibility framework that provides a full suite of features and services to facilitate interaction for users with disabilities. For instance, an accessibility framework may include services that currently allow users to drag their fingers across the screen and have user interface elements they hover over read aloud to them. While such features can enable users to navigate the interface and read text on the screen, users often need to navigate the entire screen before they can find the specific function or information they might be searching for, which can be time-consuming and difficult when there are many user interface elements. Therefore, providing a summary of the content included in the user interface can provide users with a much easier way to interact with the application. Additionally, as previously mentioned, developers typically need to hard-code accessibility features for users. The technology described in this disclosure employs a novel accessibility service that can run continuously in the background of the device and is configured to use an API to generate accessibility information. In this way, users can still access the application even if the developers do not provide or hardcode accessibility information in their applications.

[0024] According to the technology disclosed herein, computing system 100 may include an accessibility service module 108 that provides a summary of the content included in GUI 116 to user 120 by applying a machine learning model to accessibility information. Specifically, accessibility service module 108 may retrieve accessibility information from computing device 112 via API module 106 and then apply ML module 110 to the retrieved accessibility information. Computing system 100 may then generate a summary of the content included in GUI 116 by at least applying ML module 110 to the accessibility information, and then send the summary of the content included in GUI 116 to computing device 112, wherein GUI 116 is associated with an application executing computing device 112, such as... Figure 1 The example shows the application of time.

[0025] Generally, providing user 120 with the opportunity to provide input to control whether the programs or features of computing device 112 and / or computing system 100 can collect and utilize user information (e.g., user 120's personal data, information about user 114's current location, current speed, movement, location history, etc.) or instruct computing device 112 and / or computing system 100 whether and / or how they can receive content relevant to user 120. Other user information may include data including the user's usage context obtained from the application itself or from other sources. Examples of usage contexts may include the breadth of sharing (public sharing, sharing with a large group, sharing privately, or sharing with specific individuals), shared contexts, etc. When the user permits, additional data may include the device's state, such as the device's location, the app running on the device, etc. Additionally, some data may be processed in one or more ways before it is stored or used by computing device 112 and / or computing system 100, resulting in the deletion of personally identifiable information. For example, user identity can be processed in a way that makes it impossible to determine personally identifiable information about the user, or location information can be generalized (e.g., to the city, zip code, or state level) to make it impossible to determine the user's specific location. Therefore, user 120 can control how computing device 112 and / or computing system 100 collects and uses information about them. For example, computing device 112 can prompt user 120 to provide explicit consent for computing device 112 and / or computing system 100 to retrieve and / or store any or all of user 120's data.

[0026] As described above, with the explicit consent of user 120, accessibility service module 108 can continuously run and be configured to monitor the content of the active user interface of one or more applications. In the example involving computing device 112, with the explicit consent of user 120, accessibility service module 108 can continuously run in the background of computing device 112 and be configured to monitor the content of the active user interface of one or more applications executing on computing device 112. In other words, API 106 receives explicit consent from user 120 to collect view accessibility information from one or more applications. As described above, accessibility service module 108 can analyze the content of the graphical user interface in response to triggered input (e.g., mechanically (such as by pressing a button), by gesture recognition / control (such as triple-clicking on the screen), by audio (such as verbal commands), etc.) or automatically (so that user input is not triggered), provided that the user has given explicit permission for accessibility service to analyze the content of the screen. In some examples, API module 106 can provide information about user interface elements, events, and actions to assistive technologies (e.g., screen readers, zoom gestures, toggling devices, etc.) provided by accessibility service module 108. In some examples, API module 106 can be configured to enable the exchange of data in a standardized format. For example, API module 106 can support REST (Representative State Transfer), a widely used architectural style for building APIs that exchange data between applications using HTTP (Hypertext Transfer Protocol).

[0027] API module 106 can be configured to generate accessibility event streams when a user interacts with computing device 112 and applications running on computing device 112. In some examples, these events can represent actions and changes in the user interface, such as button presses, text changes, and screen transitions (e.g., user 120 toggling between buttons 118A and 118D on GUI 116). With explicit consent from user 120, accessibility service module 108 can receive and analyze these events to better understand how user 120 interacts with applications running on computing device 112.

[0028] In some examples, API module 106 can also be configured to model the user interface elements of GUI 116 as a hierarchy of accessibility nodes. Each node can correspond to a user interface element displayed on GUI 116, such as buttons 118A to 118D, summary button 114, time, date, other text fields, images, etc.

[0029] API module 106 can be configured to retrieve accessibility actions from an application running on computing device 112. "Accessibility actions" can refer to different types of input that can be detected at a location associated with UI component 102, such as mechanical input (e.g., clicking a button, swiping the screen, etc.), audio input (e.g., verbal commands), or gesture control (e.g., triple taps on the screen, waving, assistive gestures, etc.). Therefore, accessibility actions can provide users with the ability to interact with the application in multiple ways according to their needs. For example, in a situation where user 120 cannot provide mechanical input, accessibility service module 108 can provide user 120 with the ability to trigger a "click" action on summary button 114 when user 120 provides a voice command (such as "summary").

[0030] In some examples, API module 106 can be configured to retrieve descriptive text (i.e., content descriptions) for user interface elements that may not have visible text labels (e.g., images or icons). In this way, accessibility service module 108 can communicate the meaning and purpose of these elements to visually impaired users who cannot see them. In other words, API module 106 can be configured to retrieve information relating to each element included in GUI 116, regardless of the type of element or whether accessibility information has been provided by the developer.

[0031] For example, computing system 100 can retrieve information associated with multiple user interface elements included in GUI 116 via API module 106, wherein the information includes one or more of the following: node type, text content associated with the node, actions that can be performed on the node, relationships between one or more nodes, or multiple accessibility features included in the node.

[0032] In some examples, the accessibility service module 108 may also provide users with the ability to configure various accessibility options according to their needs. For example, user 120 may be able to adjust user interface elements of the GUI, such as text size, enable color correction, set zoom gestures, and configure gesture-based navigation for GUI 116.

[0033] In some examples, information retrieved from computing device 112 by API module 106 may be stored by computing system 100 to identify potential accessibility issues and / or to better understand how user 120 interacts with computing device 112.

[0034] As described above, generally, the accessibility service module 108 can only send information (e.g., location information, other context information, etc.) to the ML module 110 when the computing system 100 receives permission from the user of the computing device 112 to send information. For example, in cases where the computing system 100 and / or the computing device 112 can collect, transmit, or utilize personal information about the user (e.g., location information, financial information, etc.), as discussed herein, the user may be given the opportunity to control whether the programs or features of the computing system 100 can collect user information (e.g., information about the user's social networks, the user's social actions or activities, the user's occupation, the user's preferences, or the user's current location), or to control whether and / or how the computing system 100 and / or the computing device 112 can store and share user information.

[0035] Furthermore, some data may be processed in one or more ways before being stored, transmitted, or used, such that personally identifiable information is deleted. For example, a user's identity may be processed to the point that personally identifiable information about the user cannot be determined. Therefore, the user can control how information about the user is collected according to the techniques disclosed herein, and how that information is stored, transmitted, and / or used.

[0036] Figure 2 This is a block diagram illustrating another example computing system configured to apply a machine learning module to accessibility information according to one or more techniques of this disclosure. Figure 2 As shown in the example, computing system 200 includes a processor 224, one or more communication channels 230, one or more user interface components (UICs) 232, one or more communication units 228, and one or more storage devices 238. The storage device 238 of computing system 200 may include a user interface module 204 and an accessibility service module 208. For example... Figure 2 As shown in the example, the accessibility service module 208 further includes an API module 206, a machine learning module 210, an XML conversion module 244, and an accessibility information storage 242. Some or all of the components and / or functionalities attributable to the computing system 200 can be implemented or executed by a computing device communicating with the computing system 200. The computing system 200, user interface module 204, accessibility service module 208, API module 206, machine learning module 210, and user interface (UI) component 232 can respectively communicate with... Figure 1 The computing system 100, user interface module 104, accessibility service module 108, API module 106, machine learning module 110 and user interface (UI) component 102 are similar (or even substantially similar).

[0037] For example, one or more communication units 228 of computing system 200 can communicate with external devices by transmitting and / or receiving data at computing system 200 (such as transmitting and receiving data to and from a remote computer system or computing device). Example communication units 228 include network interface cards (e.g., such as Ethernet cards), optical transceivers, radio frequency transceivers, or any other type of device capable of transmitting and / or receiving information. Other examples of communication units 228 may be devices configured to transmit and receive Ultrawideband®, Bluetooth®, GPS, 3G, 4G, and Wi-Fi®, etc., which can be found in computing devices (such as mobile devices).

[0038] like Figure 2 As shown in the example, communication channel 230 can interconnect each of the components shown to enable inter-component communication (physical, communicative, and / or operational communication). In some examples, communication channel 230 may include a system bus, a network connection (e.g., to a wireless connection as described above), one or more inter-process communication data structures, or any other component for transferring data locally or remotely between hardware and / or software.

[0039] One or more I / O devices 234 of the computing system 200 can receive input and generate output. Examples of input are tactile input, audio input, motion-sensing input, and optical input, to name just a few. In one example, the input device of the I / O device 234 may include a touchscreen, touchpad, mouse, keyboard, voice response system, video camera, button, control panel, microphone, or any other type of device for detecting input from a person or machine. The output device of the I / O device 234 may include a sound card, video graphics adapter card, speaker, display, or any other type of device for generating output to a person or machine.

[0040] User interface module 204, accessibility service module 208, API module 206, machine learning module 210, XML conversion module 244, and accessibility information storage 242 (hereinafter referred to as "modules 204 to 244") can perform the operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and executing on computing system 200 or one or more other computing devices (e.g., cloud-based applications—not shown). For example, some or all of modules 204 to 244 may be included on a local computing device (such as... Figure 1 The technologies described herein can be implemented locally on the computing device (112) and can be executed on the local computing device. Therefore, all the technologies described herein can be implemented locally on the computing device.

[0041] The computing system 200 may utilize one or more processors 224 to execute one or more modules 204 to 244, or may execute as a virtual machine executing on or within the underlying hardware, any one or more modules 204 to 244, or a portion thereof. The one or more modules 204 to 244 may be implemented in various ways, such as being implemented as a downloadable or pre-installed application, remotely implemented as a cloud application, or implemented as part of the operating system of the computing system 200. Other examples of a computing system 200 implementing the techniques of this disclosure may include... Figure 2 Additional components not shown.

[0042] exist Figure 2 In the examples, one or more processors 224 may implement functionality and / or execute instructions within computing system 200. For example, one or more processors 224 may receive and execute instructions that provide functionality to UIC 232, communication unit 228, one or more storage devices 238, and operating system to perform one or more operations as described herein. For example, one or more processors 224 may receive and execute instructions that provide some or all of the functionality to modules 204 to 244 to perform one or more operations or various functions described herein. One or more processors 224 include a central processing unit (CPU). Examples of CPUs include, but are not limited to: digital signal processors (DSPs), general-purpose microprocessors, tensor processing units (TPUs); neural processing units (NPUs); neural processing engines; the core of a CPU, VPU, GPU, TPU, NPU, or other processing device; application-specific integrated circuits (ASICs); field-programmable arrays (FPGAs); or other equivalent integrated or discrete logic circuit systems.

[0043] One or more storage devices 238 within computing system 200 may store information (such as accessibility information or other data discussed herein) for processing during operation of computing system 200. In some examples, one or more storage devices in storage device 238 may be volatile or temporary memory. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, storage device 238 may also include one or more computer-readable storage media. Storage device 238 may be configured to store a larger amount of information in non-volatile memory for a longer period than in volatile memory. Examples of non-volatile memory include magnetic hard disks, optical disks, floppy disks, flash memory, or in the form of electrically programmable memory (EPROM) or electrically erasable and programmable (EEPROM) memory. Storage device 238 may store information related to… Figure 2 The program instructions and / or data associated with modules 204 to 244.

[0044] Such as about Figure 1 As described, computing system 200 can use API module 206 to retrieve information associated with multiple user interface elements included in a graphical user interface (GUI) associated with an application running on the computing device. API module 206 can retrieve information in response to receiving an instruction for a gesture detected on the computing device. UI module 204 can interpret instructions or other input detected on the computing device. UI module 204 can relay information about input detected on the computing device to one or more associated platforms, operating systems, applications, and / or services running on the computing device, causing the computing device to perform functions. UI module 204 can also receive information and instructions from one or more associated platforms, operating systems, applications, and / or services running on the computing device (e.g., accessibility service module 208) to generate and / or output a summary of the content included in the GUI, wherein the GUI is associated with an application running on the computing system. Additionally, UI module 204 can act as an intermediary between one or more associated platforms, operating systems, applications, and / or services executing on the computing device and various output devices of the computing device (e.g., speakers, LED indicators, vibrators, etc.) to utilize the computing device to generate outputs (e.g., graphical outputs, audible outputs, haptic outputs, etc.).

[0045] Accessibility service module 208 can be implemented on a computing device in various ways. For example, accessibility service module 208 can be implemented as a downloadable or pre-installed application or "app". In another example, accessibility service module 208 can be implemented as part of the operating system of the computing device.

[0046] Accessibility information storage 242 is a repository of accessibility information received by computing system 200 from computing device via API module 206. In some examples, accessibility information may be associated with a “view hierarchy” of the GUI of an application running on the computing device, where the graphical user interface may be represented as a GUI view tree. In some examples, this hierarchy may display a hierarchy of information presented via the GUI, such as categories, subcategories, and sub-subcategories. In some examples, information associated with multiple user interface elements may include one or more of the following: node type, text content associated with a node, actions that can be performed on a node, relationships between one or more nodes, or multiple accessibility features included in a node. The retrieved accessibility information may be stored in accessibility information storage 242 for use by other modules of accessibility service module 208 (such as XML conversion module 244). In some examples, accessibility information storage 242 may at least partially function as a cache operation for accessibility information retrieved from computing device 112 (e.g., using one or more communication units 228) or other computing devices. Generally, accessibility information storage 242 can be configured as a database, flat file, table, or other data structure stored within storage device 238. In some examples, accessibility information storage 242 is shared among various modules executing at computing system 200 (e.g., in one or more modules 204 to 244 or...). Figure 2 (Shared among other modules not shown). In other examples, different data stores are configured for modules that require data stores and execute at computing system 200. Each data store can be configured and managed by a different module and can store data in different ways. In some examples, computing system 200 can receive and store accessibility information from computing devices for a specified time period.

[0047] In the example accessibility service, accessibility information can be referred to as "AccessibilityNodeInfo," which can be a class representing a window content node and the actions that can be requested from its source. Figure 2In one example, the accessibility service module 208 can retrieve “AccessibilityNodeInfo” information from the computing device 112, where the information is presented as an accessibility node information tree, which may or may not be mapped one-to-one to the view hierarchy of the application’s user interface. In some examples, it may be difficult to apply machine learning models (such as large language models) to accessibility information written in multiple languages ​​and / or formats. Therefore, the computing system 200 may employ an XML conversion module 244 to convert the accessibility information retrieved from the computing device 112 into structured text. For example, the API module 206 can retrieve accessibility information from the computing device 112 in any structure or programming language. The XML conversion module 244 can then convert the accessibility information into Extensible Markup Language (XML). For example, the XML conversion module 244 can determine the type of accessibility information to be included in the XML representation. More specifically, the XML conversion module 244 can analyze various attributes, such as content descriptions, roles, states, actions, and / or other relevant properties of user interface elements. XML conversion module 244 can further generate structured text representing the hierarchy of user interface elements and their associated accessibility information. XML conversion module 244 can map accessibility information to XML attributes or elements of each UI element. If UI elements have parent-child relationships, this hierarchy can be reflected in the XML structure. In some examples, XML conversion module 244 can determine how to represent accessibility information based on complexity. In some examples, as described above, if accessibility information is not already provided, API module 206 can generate accessibility information associated with the application's user interface elements. In some examples, API module 206 can also generate or otherwise retrieve contextual information that helps interpret the accessibility information. For example, this information may include information about the current screen, the application, or the context in which UI elements are presented. XML conversion module 244 can receive this information and generate a more accurate and effective XML (or other structured text) representation. In some examples, XML conversion module 244 can alternatively convert the accessibility information into other structured text types, such as, but not limited to, HTML, JSON, CSV, INI files, etc. In this way, accessibility information retrieved by API module 206 can be provided to ML module 210 in a standardized format. In some examples, structured text can be stored in accessibility information storage 242.

[0048] In some implementations, accessibility information can be preprocessed, which may include extracting one or more additional features from the raw data. For example, feature extraction techniques can be applied to accessibility information to generate one or more new additional features.

[0049] In some examples, the user interface generated on the computing device may include frequently changing and / or updated user interface elements (e.g., timers, image slideshows, etc.). Therefore, the accessibility information retrieved by API module 206 may need to be updated relative to any changes occurring on the computing device. Thus, in some implementations, accessibility information can be inherently ordered. In some cases, ordered accessibility information can be generated by sampling or otherwise segmenting the accessibility information stream. As an example, frames can be extracted from a video. However, in some implementations, ordered accessibility information can become non-ordered through a summary summary, where the summary of the content generated by computing system 200 includes information about updated and / or changed user interface elements. In some examples, graphical user interface elements can be categorized into static and dynamic objects. Static objects can be attributed to elements of the GUI that do not change their appearance after initial rendering (e.g., company logos, menu options, etc.). However, dynamic objects can be associated with those elements of the GUI whose appearance may change with each rendering (e.g., text fields, animations, etc.).

[0050] The computing system 200 can apply the machine learning module 210 to the structured text generated by the XML transformation module 244. As described herein, the machine learning module 210 can employ a large language model (LLM) that can interpret the structured text and generate a summary of the content included in a GUI associated with an application executed on the computing device and / or computing system. In some examples, the machine learning module 210 can implement other machine learning models that can replace or combine information about... Figure 3 The LLM model described is used. Machine learning module 210 can perform various types of natural language processing (NLP) based on accessibility information or "input data." For example, machine learning module 210 can summarize, translate, or organize the input data. Machine learning module 210 can use recurrent neural networks (RNNs) and / or transformer models (self-attention models), such as GPT-3, BERT, and T5. In some implementations, machine learning module 210 can perform classification, summarization, name generation, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0051] In some implementations, the machine learning module 210 can perform various types of classification based on the input data. For example, the machine learning module 210 can perform binary classification or multi-class classification. In binary classification, the output data may include classifying the input data into one of two distinct classes. In multi-class classification, the output data may include classifying the input data into one (or more) of more than two classes. The classification can be single-label or multi-label. The machine learning module 210 can perform discrete category classification, where the input data is simply classified into one or more classes or categories.

[0052] In the case where the machine learning module 210 performs classification, supervised learning techniques can be used to train the machine learning module 210. For example, the machine learning module 210 can be trained on a training dataset that includes training examples labeled as belonging to (or not belonging to) one or more classes.

[0053] In some implementations, machine learning module 210 can perform regression to provide output data in the form of continuous numerical values. Continuous numerical values ​​can correspond to any number of different measures or numerical representations, including, for example, monetary values, fractions, or other numerical representations. In the examples, machine learning module 210 can perform linear regression, multinomial regression, or nonlinear regression. In the examples, machine learning module 210 can perform simple regression or multiple regression. As mentioned above, in some implementations, the Softmax function or other functions or layers can be used to compress sets of real values ​​associated with two or more possible classes into a set of real values ​​in the range (0, 1) that sums to one.

[0054] Machine learning module 210 can perform various types of clustering. For example, machine learning module 210 can identify one or more clusters that the input data is most likely to correspond to. Machine learning module 210 can identify one or more clusters within the input data. That is, when the input data includes multiple objects, documents, or other entities, machine learning module 210 can classify the multiple entities included in the input data into multiple clusters. In some implementations where machine learning module 210 performs clustering, unsupervised learning techniques can be used to train machine learning module 210.

[0055] In some cases, the machine learning module 210 can act as an agent within the environment. For example, the machine learning module 210 can be trained using reinforcement learning, which will be discussed in more detail below.

[0056] In some implementations, the machine learning module 210 may include a parametric model, while in other implementations, it may include a non-parametric model. In some implementations, the machine learning module 210 may include a linear model, while in other implementations, it may include a non-linear model.

[0057] As described above, machine learning module 210 can be or includes one or more machine learning models of various different types. Examples of such different types of machine learning models are provided below for illustration. One or more of the example models described below can be used (e.g., combined) to provide output data in response to input data. Additional models may be used in addition to the example models provided below.

[0058] In some implementations, the machine learning module 210 may be or include one or more classifier models, such as, for example, a linear classification model; a quadratic classification model, etc. The machine learning module 210 may be or include one or more regression models, such as, for example, a simple linear regression model; a multiple linear regression model; a logistic regression model; a stepwise regression model; a multiple adaptive regression spline; a locally estimated scatter plot smoothing model, etc.

[0059] In some implementations, the machine learning module 210 may be or include one or more artificial neural networks (also simply referred to as neural networks). A neural network may include a set of connected nodes, which may also be referred to as neurons or perceptrons. A neural network may be organized into one or more layers. A neural network comprising multiple layers may be referred to as a "deep" network. A deep network may include an input layer, an output layer, and one or more hidden layers located between the input layer and the output layer. The nodes of a neural network may be connected or not fully connected.

[0060] In some examples, the machine learning module 210 may be or include one or more generative networks, such as, for example, generative adversarial networks. Generative networks can be used to generate new data, such as human feedback text.

[0061] In examples where the input data does not include feature embeddings, one or more neural networks can be used to provide embeddings based on the input data. For example, an embedding can be a knowledge representation abstracted from the input data to one or more learned dimensions. In some cases, embeddings can be a useful source for identifying relevant entities. In some cases, embeddings can be extracted from the network's output, while in others, they can be extracted from any hidden node or layer of the network (e.g., near but not the final layer). Embeddings can be used to perform automatic suggestions such as next video recommendations, product suggestions, entity or object recognition, etc. In some cases, embeddings are useful inputs to downstream models. For example, embeddings can be useful for summarizing input data for downstream models or processing systems (e.g., search queries).

[0062] In some implementations, the machine learning module 210 may perform or be subjected to one or more reinforcement learning techniques, such as Markov decision processes; dynamic programming; Q-functions or Q-learning; value function methods; deep Q-networks; differentiable neural computers; asynchronous advantage actor-critics; deterministic policy gradients, etc.

[0063] In some implementations, the machine learning module 210 can be an autoregressive model. In some cases, an autoregressive model can specify that the output data depends linearly on its own previous values ​​and random terms. In some cases, an autoregressive model can take the form of a stochastic difference equation. An example of an autoregressive model is WaveNet, a generative model used for raw audio.

[0064] In some implementations, the machine learning module 210 may include or form part of a multi-model ensemble. As an example, bootstrap aggregating, also known as "bagging," can be performed. In bootstrap aggregating, the training dataset is divided into multiple subsets (e.g., through random sampling with replacement), and multiple models are trained on each subset separately. During inference, the corresponding outputs of the multiple models can be combined (e.g., through averaging, voting, or other techniques) and used as the output of the ensemble.

[0065] An example ensemble is the random forest, also known as a random decision forest. Random forests are ensemble learning methods used for classification, regression, and other tasks. Random forests are generated by producing multiple decision trees during training. In some cases, during inference, the class predicted as the mode of multiple classes (classification) or the mean of individual trees (regression) can be used as the output of the forest. Random decision forests can correct the tendency for decision trees to overfit their training set.

[0066] Another example of an ensemble technique is stacking, which in some cases can be referred to as stacked generalization. Stacking involves training a combiner model to mix or otherwise combine predictions from several other machine learning models. Thus, multiple machine learning models (e.g., the same or different types) can be trained based on training data. Additionally, a combiner model can be trained to take predictions from other machine learning models as input and, in response, produce a final inference or prediction. In some cases, a single-layer logistic regression model can be used as the combiner model.

[0067] Another example of ensemble techniques is boosting. Boosting can involve progressively building an ensemble by iteratively training weak models and then adding them to a final strong model. For example, in some cases, each new model can be trained to emphasize training examples that were misclassified (e.g., mis-classified) by the previous model. For example, weights associated with each such misclassified example can be increased. A common implementation of boosting is AdaBoost, which is also known as adaptive boosting. Other example boosting techniques include LPBoost; TotalBoost; BrownBoost; xgboost; MadaBoost; LogitBoost; gradient boosting, etc. Furthermore, any of the models mentioned above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. As an example, an ensemble can include a top-level machine learning model or heuristic function to combine and / or weight the outputs of the models forming the ensemble.

[0068] In some implementations, multiple machine learning models (e.g., forming an ensemble model) can be linked and jointly trained (e.g., by backpropagating errors sequentially through model ensemble)). However, in some implementations, only a subset of the jointly trained models (e.g., one of the jointly trained models) is used for inference.

[0069] In some implementations, the machine learning module 210 can be used to preprocess input data for subsequent input into another model. For example, the machine learning module 210 can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal component analysis, singular value decomposition, word2vec / GLOVE, and / or related methods); clustering; and even classification and regression for downstream consumption. Many of these techniques have been discussed above and will be discussed further below.

[0070] In some implementations, input data can be intentionally deformed in any number of ways during training to improve model robustness, generalization, or other qualities. Example techniques for deforming input data include adding noise; changing color, shading, or hue; scaling; segmentation; augmentation; and so on.

[0071] In response to receiving input data, the machine learning module 210 can provide output data. As an example, in various implementations, the output data may include content stored locally on the user device or stored in the cloud, which may be shared in relation to the initial content selection.

[0072] In some implementations, the output data can influence downstream processes or decision-making. As an example, in some implementations, a summary of the output data or content can be interpreted and / or used by a rule-based regulator.

[0073] The technology disclosed herein can be used by one or more computing devices (e.g., Figure 1 The computing device 112 of A is implemented or otherwise executed on the one or more computing devices. Examples of such computing devices include user computing devices (e.g., laptops, desktops, and mobile computing devices such as tablets, smartphones, wearable computing devices, etc.); embedded computing devices (e.g., devices embedded in vehicles, cameras, image sensors, industrial machines, satellites, game consoles or controllers, or home appliances such as refrigerators, thermostats, energy meters, home energy managers, smart home assistants, etc.); other computing devices; or combinations thereof. The computing system 200 that implements the machine learning module 210 or other aspects of this disclosure may include multiple hardware components that implement the performance of the techniques described herein.

[0074] The machine learning module 210 described herein can be trained according to one or more of a variety of different training types or techniques. For example, in some implementations, the machine learning module 210 can be trained using supervised learning, wherein the machine learning module 210 is trained on a training dataset that includes instances or examples with labels. These labels can be applied manually by experts, generated through crowdsourcing, or provided through other techniques (e.g., through physics-based or complex mathematical models). In some implementations, training examples can be provided by the user's computing device if the user has provided consent. In some implementations, this process can be referred to as personalizing the model.

[0075] In some implementations, backpropagation can be combined with optimization techniques (e.g., gradient-based techniques) to train the machine learning module 210 (e.g., when the machine learning model is a multi-layer model such as an artificial neural network). For example, iterative loops of propagation and model parameter (e.g., weight) updates can be performed to train the machine learning module 210. Example backpropagation techniques include truncated time-through backpropagation, Levenberg-Marquardt backpropagation, etc.

[0076] In some implementations, unsupervised learning techniques can be used to train the machine learning module 210 described herein. Unsupervised learning can include inferring functions from unlabeled data to describe hidden structures. For example, the data may not include classification or categorization. Unsupervised learning techniques can be used to generate machine learning models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks.

[0077] The machine learning module 210 can be trained using semi-supervised techniques that combine aspects of supervised and unsupervised learning. The machine learning module 210 can be trained using evolutionary techniques or genetic algorithms, or generated in other ways. In some implementations, reinforcement learning can be used to train the machine learning module 210 described herein. In reinforcement learning, an agent (e.g., a model) can take actions in the environment and learn to maximize rewards and / or minimize penalties resulting from such actions. Reinforcement learning differs from supervised learning problems in that it does not present correct input / output pairs or explicitly correct suboptimal actions.

[0078] In some implementations, one or more generalization techniques can be performed during training to improve the generalization of the machine learning module 210. Generalization techniques can help reduce the overfitting of the machine learning module 210 to the training data. Example generalization techniques include dropout; weight decay; batch normalization; early stopping; subset selection; stepwise selection; label smoothing; etc.

[0079] In some implementations, the machine learning module 210 described herein may include or be influenced by multiple hyperparameters (such as, for example, learning rate, number of layers, number of nodes per layer, number of leaves in the tree, number of clusters, etc.). Hyperparameters can affect model performance. Hyperparameters can be selected manually or automatically by applying techniques such as, for example, grid search, black-box optimization techniques (e.g., Bayesian optimization, random search, etc.), gradient-based optimization, etc. Example techniques and / or tools for performing automatic hyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE), etc.

[0080] In some implementations, various techniques can be used to optimize and / or adjust the learning rate during model training. Example techniques and / or tools used to perform learning rate optimization or adaptation include Adagrad; Adaptive Moment Estimation (ADAM); Adadelta; RMSprop, etc.

[0081] In some implementations, transfer learning techniques can be used to provide an initial model from which the machine learning module 210 described in this paper is trained.

[0082] In some implementations, the machine learning module 210 described herein may be included in different portions of computer-readable code on a computing device. In one example, the machine learning module 210 may be included in a particular application or program and used by such a particular application or program (e.g., for dedicated use). Thus, in one example, the computing device may include multiple applications, and one or more of such applications may contain its own corresponding machine learning libraries and machine learning models.

[0083] In another example, the machine learning module 210 described herein may be included in the operating system of a computing device (e.g., in the central intelligence layer of the operating system) and may be invoked or otherwise used by one or more applications that interact with the operating system. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an application programming interface (API) (e.g., a common public API across all applications).

[0084] In some implementations, the central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing device. The central device data layer can communicate with multiple other components of the computing device, such as one or more sensors, a field manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0085] The technologies discussed in this paper involve servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality between and within components. For example, the processes discussed in this paper can be implemented using a single device or component, or multiple devices or components working in combination.

[0086] Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0087] Furthermore, the machine learning techniques described herein are readily interchangeable and combinable. While some example techniques have been described, many other techniques exist and can be used in conjunction with aspects of this disclosure.

[0088] In some implementations, transfer learning (TL) can be used. Transfer learning involves reusing a model and its parameters learned while solving a problem and applying it to different but related problems. Models trained on very large datasets can be retrained or fine-tuned on additional data. Typically, all model design and their parameters from the source model are copied, except for the output layer. The output layer is often called the head, while the other layers are often called the foundation. The source parameters can be considered to include knowledge learned from the source dataset, and this knowledge can also be applied to the target dataset. Fine-tuning may involve updating the head parameters, while the body parameters are either fixed or updated in later steps.

[0089] Therefore, machine learning module 210 may apply a large language model to structured text, with or without incorporating one or more of the machine learning techniques described above, to generate a summary of the content included in a graphical user interface associated with an application running on the computing device. Computing system 200 may then send the summary included in the graphical user interface, which is associated with an application running on the computing device, to the computing device. In some examples, computing system 200 may output a summary of the content included in the graphical user interface, which is associated with an application running on computing system 200, via an audio output device or other I / O device 234 of computing system 200.

[0090] Specifically, as will be discussed later. Figure 4 As described, a text-to-speech synthesis technique performed by a computing device and / or computing system can be used to convert a summary of content into speech, wherein the summary is read aloud to a user of the computing device and / or computing system.

[0091] Figure 3This is a conceptual diagram illustrating a machine learning module configured to apply a language model to accessibility information according to one or more techniques of this disclosure. As described above, ML module 310 can be or includes one or more transformer-based neural networks, such as large language model module 352. Large language model module 352 can implement, for example, the Pathways language model developed by Google. Transformer-based neural networks can refer to a type of deep learning architecture specifically designed to process ordered data (such as text or time series). In other words, transformer-based neural networks like LLM can be configured to perform natural language processing (NLP) tasks such as question answering, machine translation, text summarization, and sentiment analysis. Language model module 352 can be configured to perform tasks such as classification, sentiment analysis, entity extraction, question answering extraction, summarization, rewriting text in different styles, advertising copy generation, and concept generation.

[0092] Transformer-based neural networks can utilize self-attention mechanisms, which allow the model to weigh the importance of different elements in a given input sequence relative to each other. Self-attention can help language model module 352 effectively capture long-distance dependencies and complex relationships between elements (such as words in a sentence).

[0093] Language model module 352 may include an encoder and a decoder that operate to process and generate ordered data, such as structured text. Both the encoder and decoder may include one or more of the following: self-attention mechanisms, position-wise feedforward networks, layer normalization, or residual connections. In some examples, the encoder may process the input sequence and create representations that capture the relationships and context between elements in the sequence. The decoder may then obtain the representations generated by the encoder and produce an output sequence. In some examples, the decoder may use a process called autoregressive decoding to generate the output element by element at a time (e.g., word at a time), where previously generated elements are used as input to predict the next element in the sequence.

[0094] In some examples, language model module 352 can generate a summary file 356 that includes a summary of the content included in the user interface by determining a set of information types included in structured text. Information types can be, or otherwise include, topics, issues, opinions, themes, purposes, intentions, keywords, etc. In some examples, language model module 352 can determine information types by utilizing a self-attention mechanism to capture relationships and dependencies between words in an input sequence. For example, language model module 352 can lexicalize (e.g., split) a sequence of words or subwords, and then transform that sequence into a vector (e.g., a numerical representation) that language model module 352 can process. Language model module 352 can use a self-attention mechanism to weigh the importance of each lexical relative to other lexicals. In this way, language model module 352 can identify patterns and relationships between lexicals, and further identify the words corresponding to the lexicals, which indicate one or more information types of accessibility information.

[0095] Generally, the language model module 352 may excel at performing NLP tasks such as generating text and other content. However, regarding specific types of content (e.g., specific information types), the language model module 352 may be more likely to generate false or inaccurate information. To address the problem of generating false information, the language model module 352 can be configured to exclude the generation of content associated with a set of excluded information types. For example, the set of excluded information types may include one or more of phone numbers, addresses, URLs, etc. Thus, accessibility information in the form of structured text can be passed in the language model module 352, with certain prerequisites or "rules" that can be stored in the rule store 354. Rules can also be text inputs such as, for example, "What will this XML look like when rendered?", "Have your answer avoid technical terms and class names.", and "Keep your description short." In other words, the rule store 354 can store multiple text inputs that further specify how the language model module 352 should generate the summary file 356. In other words, the language model module 352 is applied to structured text according to one or more predefined rules stored in the rule store 354. These predefined rules may include, for example, unauthorized terms, unauthorized class names, unauthorized length of the content summary, or unauthorized rendering of the content summary. Because the language model module 352 can interpret both the rules and the structured text, the computing system 200 can provide a more accurate and user-friendly summary of the content included in the user interface.

[0096] The language model module 352 can generate a summary file 356 based on one or more information types determined by the view hierarchy and accessibility information of the GUI. For example, the summary file 356 may include a description that includes general view structure and semantic information associated with the graphical user interface. The summary file 356 may include a description that, for example, includes the text of any headings presented on the GUI, but not the text of any body text presented on the GUI. For example, regarding... Figure 1 The summary file 356 may include a text document containing concise descriptions of the time, date, and various buttons / functions provided by the GUI 116, and presenting information related to each element in order from top to bottom of the GUI 116.

[0097] Although this paper primarily describes transformer-based neural networks, the language model module 352 can be or otherwise includes one or more other types of neural networks. For example, the language model module 352 can be or includes an autoencoder. In some examples, the purpose of an autoencoder is to learn representations of a dataset (e.g., low-dimensional encoding), often for dimensionality reduction. For example, in some examples, an autoencoder may attempt to encode input data and provide output data reconstructing the input data from the encoding. Recently, the concept of autoencoders has been increasingly used in generative models for learning data. In some examples, an autoencoder may include an additional loss in addition to reconstructing the input data. The language model module 352 can be or includes one or more other forms of artificial neural networks, such as, for example, deep Boltzmann machines; deep belief networks; stacked autoencoders, etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0098] In some examples, the language model module 352 may be or include one or more feedforward neural networks. In a feedforward network, connections between nodes do not form loops. For example, each connection may connect a node from a previous layer to a node from a subsequent layer. In some examples, the language model module 352 may be or include one or more recurrent neural networks. In some examples, at least some nodes in the recurrent neural network may form loops.

[0099] Recurrent neural networks (RNNs) can be particularly useful for processing inherently ordered input data. For example, RNNs can pass information from the beginning of an input data sequence to later parts of it by using loops or directed loop connections. Ordered input data can include words in a sentence (e.g., for natural language processing, speech detection, or processing). In some examples, ordered input data can include time-series data (e.g., sensor data relative to time or images captured at different times). For example, RNNs can analyze sensor data relative to time to detect or predict swipe direction, perform handwriting recognition, etc. Ordered input data can include words in a sentence (e.g., for natural language processing, speech detection, or processing); notes in a musical piece; ordered actions taken by a user (e.g., to detect or predict ordered application usage); ordered object states, etc.

[0100] Examples of recurrent neural networks may include long short-term (LSTM) recurrent neural networks, gated recurrent units, bidirectional recurrent neural networks, continuous-time recurrent neural networks, neural history compressors, echo-state networks, Elman networks, Jordan networks, recurrent neural networks, Hopfield networks, fully recurrent networks, sequence-to-sequence configurations, etc.

[0101] In some examples, the language model module 352 may be or include one or more convolutional neural networks. In some examples, the convolutional neural network may include one or more convolutional layers that perform convolutions on the input data using learned filters. Filters may also be referred to as kernels. Convolutional neural networks can be particularly useful for vision problems, such as when the input data includes images like still images or videos. However, convolutional neural networks can also be applied to natural language processing.

[0102] Machine learning module 310 may include training module 350, which trains (e.g., pre-trains, fine-tunes, etc.) language model module 352. Training module 350 may pre-train language model module 352 on a large and diverse corpus of text. This dataset may cover a wide range of topics and domains to ensure that language model module 352 learns multiple language patterns and contextual relationships. Training module 350 may train language model module 352 to optimize an objective function. The objective function may be or include a loss function, such as cross-entropy loss, which compares the output data generated by the model from the training data with the labels associated with the training data (e.g., ground truth labels) (e.g., determining the differences between them). For example, the objective function of language model module 352 may correctly predict the next word in a word sequence or fill in missing words as correctly as possible.

[0103] By utilizing the language model module 352, the accessibility services provided by the computing system 200 can reduce the time and / or effort required to summarize the content on the user interface. In other words, by presenting users of the computing device and / or computing system with concise and understandable summaries of their screens (e.g., avoiding technical jargon), the technology of this disclosure can reduce the time and effort required for users with disabilities to understand the content presented to them.

[0104] In some examples, training module 350 can continuously or periodically train language model module 352. In some examples, training module 350 can fine-tune language model module 352 using feedback during the training process. For example, Figure 2UI component 232 can receive user input via a computing device and / or computing system, selecting feedback (e.g., approval, disapproval, etc.) related to the summary of content presented to the user. In some examples, this feedback can indicate whether the summary of the content is accurate or inaccurate, correct or incorrect, high-quality or low-quality, etc. UI module 204 can receive this feedback and can send it to accessibility service module 208. Accessibility service module 208 can transmit the feedback to machine learning module 210 (specifically to training module 350), where training module 350 uses the feedback for training. For example, training module 350 can transform the feedback into labeled data for supervised training. Alternatively or additionally, training module 350 can fine-tune language model module 352 by monitoring the relationship between the performance of language model module 352 and user feedback, and iterate the fine-tuning process as needed (e.g., to receive more positive user feedback and fewer negative user feedback). In this way, the techniques of this disclosure can establish a feedback loop that continuously improves the quality of the output of language model module 352 (i.e., summary file 356).

[0105] Figure 4 This is a conceptual diagram illustrating a text-to-speech synthesis technique that utilizes one or more technologies according to this disclosure to provide a user with an output summarizing content included in a graphical user interface. The computing system 400 can be connected to... Figure 1 The computing system 100 and Figure 2 The computing system 200 is similar (or even substantially similar). The computing device 412 can be connected to... Figure 1 The computing device 112 is similar (or even substantially similar). The user interface (UI) component 402 can be with... Figure 1 UI component 102 and / or Figure 2 The UI component 232 is similar (even essentially similar). Network 401 can be compared to... Figure 1 It is similar to (or even basically similar to) Network 101.

[0106] As described above, the computing system 400 generates a summary of content included in a graphical user interface associated with an application executed at the computing device 412, or in some examples, at the computing system 400. This summary includes a description that includes a general view structure and semantic information associated with the graphical user interface. The computing system 400 can then send the summary of the content to the computing device 412. In some examples, the computing system 400 can output the summary of the content via an audio output device of the computing system 400. Figure 4As shown in the example, Summary 460 is provided to user 420 by UI component 402. Summary 460 reads as follows: "The screen displays a clock with the time 12:17 and the date Friday the 31st of March. There is also a button to add a city. There are also buttons for setting alarms, timers, and stopwatches." In other words, Summary 460 is a graphical user interface (such as...) Figure 1 A concise description of the content included in the GUI 116. In this example, the summary 460 is read aloud to the user 420 using text-to-speech synthesis technology.

[0107] Automatic text-to-speech algorithms utilize narration models to read the text of a document aloud in a manner that mimics a human narrator. In some examples, the computing system 400 can be configured to perform the task... Figure 3 The machine learning module 310 automatically processes the generated summary file 356 into speech to produce a summary 460. In other examples, the computing device 412 may be configured to perform automatic text-to-speech processing to produce the summary 460. Automatic text-to-speech processing may refer to one or more text-to-speech algorithms (e.g., generative adversarial network GAN models) that synthesize text files (such as...) Figure 3 The summary file 356) is used to generate a summary 460. In this way, a summary of the content of any user interface generated on the user's computing device and any given layout can be presented to the user without the user having to manually navigate the entire user interface. Therefore, users with disabilities can receive a more user-friendly experience when interacting with devices and applications, as they may be able to understand the content presented to them more quickly and easily.

[0108] As described above, the techniques described herein can all be implemented natively on a computing device (such as computing device 412). For example, computing device 412 can use an application programming interface to retrieve information associated with multiple user interface elements included in a graphical user interface, which is associated with an application running at computing device 412. Computing device 412 can then convert the information into structured text and generate a summary of the content included in the graphical user interface by applying a machine learning model (e.g., a large language model) to the structured text. Computing device 412 can then output a summary of the content included in the graphical user interface to user 420, wherein the graphical user interface is associated with an application running at computing device 412.

[0109] Figure 5This is a flowchart illustrating example operations for outputting via an application language model according to one or more techniques of this disclosure. The computing system 100 uses an application programming interface generated by the API module 106 to retrieve information associated with multiple user interface elements included in the graphical user interface 116 (582). In some examples, GUI 116 is a GUI associated with an application executed at the computing device 112. In some examples, the computing system 100 receives indications of gestures detected at the location of the computing device 112. In some examples, the gesture indications are provided manually by the user 120 or through gesture control. In some examples, the information associated with the multiple user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed on the node, relationships between one or more nodes, or multiple accessibility features included in the node. In some examples, the information associated with the multiple user interface elements is generated by the API module 106. The XML conversion module 244 of the computing system 100 then converts the information into structured text (584). Then, the machine learning module 210 of the computing system 100 uses the large language model module 352 to generate a summary (586) of the content included in the graphical user interface 116 by applying the large language model module 352 to the structured text. In some examples, the large language model module 352 is applied to the structured text according to one or more predefined rules stored in the rule store 354, which include one or more of unauthorized terms, unauthorized class names, unauthorized length of the content summary, or unauthorized rendering of the content summary. The computing system 100 then outputs a summary file 356 (588) that includes a summary of the content included in the graphical user interface 116. In some examples, the computing system 100 sends the summary file 356 to the computing device 112. In some examples, the summary file 356 includes a description that includes general view structure and semantic information associated with the graphical user interface 116. In some examples, the summary file 356 is processed by a text-to-speech algorithm to produce a summary 460, which is read aloud by the computing device 112 to the user 120. In some examples, computing system 400 outputs summary 460 via audio output device 234 of computing system 400, wherein summary 460 includes a summary of the content included in the graphical user interface associated with an application running at computing system 400.

[0110] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on or transmitted over a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to: a tangible medium, such as a data storage medium; or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0111] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but rather refer to non-transient tangible storage media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital universal discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0112] Instructions can be executed by one or more processors such as digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Therefore, as used herein, the term "processor" can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules. Moreover, the techniques can be fully implemented in one or more circuit or logic elements.

[0113] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but these various components, modules, or units are not necessarily required to be implemented by different hardware units. Rather, as described above, the various units can be combined within a hardware unit, or provided by a collection of interoperable hardware units including one or more processors as described above, combined with suitable software and / or firmware.

[0114] It should be recognized that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all of the described actions or events are necessary for the practice of the technique). Furthermore, in some examples, actions or events may be performed concurrently, for example, through multithreading, interrupt handling, or multiple processors, rather than sequentially.

[0115] In some examples, computer-readable storage media include non-transitory media. The term "non-transitory" indicates that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., stored in RAM or cache).

[0116] Example 1: A method includes: a computing system using an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface; the computing system converting the information into structured text; the computing system using a machine learning model to generate a summary of the content included in the graphical user interface by applying at least the machine learning model to the structured text, wherein the machine learning model is a large language model; and the computing system outputting the summary of the content included in the graphical user interface.

[0117] Example 2: The method as described in Example 1, wherein outputting the summary of the content included in the graphical user interface further comprises: sending the summary of the content included in the graphical user interface to a computing device, wherein the graphical user interface is associated with an application executed on the computing device.

[0118] Example 3: The method as described in Example 1, wherein outputting the summary of the content included in the graphical user interface further comprises: outputting the summary of the content included in the graphical user interface by an audio output device of the computing system, wherein the graphical user interface is associated with an application executed on the computing system.

[0119] Example 4: The method as described in any one of Examples 1 to 3, wherein the information associated with the plurality of user interface elements is generated by the application programming interface.

[0120] Example 5: The method as described in any one of Examples 1 to 4, wherein the information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, the relationship between one or more nodes, or a plurality of accessibility features included in the node.

[0121] Example 6: The method as described in any one of Examples 1 to 5, wherein the machine learning model is applied to the structured text according to one or more predefined rules, the one or more predefined rules including one or more of the following: unauthorized terms, unauthorized class names, unauthorized length of the summary of the content, or unauthorized rendering of the summary of the content.

[0122] Example 7: The method as described in any one of Examples 1 to 6, wherein the summary of the content includes a description, the description including a general view structure and semantic information associated with the graphical user interface.

[0123] Example 8: The method of any one of Examples 1 to 7, wherein the computing system retrieves the information automatically or in response to one or more of mechanical input, audio input, or gesture control.

[0124] Example 9: A computing system includes: one or more processors; and one or more storage devices storing instructions, wherein, when executed by the one or more processors, the instructions cause the one or more processors to: use an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface; convert the information into structured text; use a machine learning model to generate a summary of the content included in the graphical user interface by applying at least the machine learning model to the structured text, wherein the machine learning model is a large language model; and output the summary of the content included in the graphical user interface.

[0125] Example 10: A computing system as described in Example 9, wherein, in order to further output the summary of the content included in the graphical user interface, the one or more processors are further configured to send the summary of the content included in the graphical user interface to a computing device, wherein the graphical user interface is associated with an application executed on the computing device.

[0126] Example 11: A computing system as described in Example 9, wherein, in order to further output the summary of the content included in the graphical user interface, the one or more processors are further configured to output the summary of the content included in the graphical user interface by an audio output device of the computing system, wherein the graphical user interface is associated with an application executed at the computing system.

[0127] Example 12: A computing system as described in any one of Examples 9 to 11, wherein the information associated with the plurality of user interface elements is generated by the application programming interface.

[0128] Example 13: A computing system as described in any one of Examples 9 to 12, wherein the information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, a relationship between one or more nodes, or a plurality of accessibility features included in the node.

[0129] Example 14: A computing system as described in any one of Examples 9 to 13, wherein the machine learning model is applied to the structured text according to one or more predefined rules, the one or more predefined rules including one or more of the following: unauthorized terms, unauthorized class names, unauthorized length of the summary of the content, or unauthorized rendering of the summary of the content.

[0130] Example 15: A computing system as described in any one of Examples 9 to 14, wherein the summary of the content includes a description that includes a general view structure and semantic information associated with the graphical user interface.

[0131] Example 16: A computing system as described in any one of Examples 9 to 15, wherein the one or more processors retrieve the information automatically or in response to one or more of mechanical input, audio input, or gesture control.

[0132] Example 17: A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to: use an application programming interface to retrieve information associated with a plurality of user interface elements included in a graphical user interface; convert the information into structured text; use a machine learning model to generate a summary of the content included in the graphical user interface by at least applying the machine learning model to the structured text, wherein the machine learning model is a large language model; and output the summary of the content included in the graphical user interface.

[0133] Example 18: A non-transitory computer-readable medium as described in Example 17, wherein, in order to further output the summary of the content included in the graphical user interface, the one or more processors are further configured to: send the summary of the content included in the graphical user interface to a computing device, wherein the graphical user interface is associated with an application executed on the computing device.

[0134] Example 19: A non-transitory computer-readable medium as described in Example 17, wherein the information associated with the plurality of user interface elements is generated by the application programming interface.

[0135] Example 20: A non-transitory computer-readable medium as described in any one of Examples 17 to 19, wherein the information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, a relationship between one or more nodes, or a plurality of accessibility features included in the node.

[0136] Example 21: A non-transitory computer-readable medium as described in any one of Examples 17 to 20, wherein the machine learning model is applied to the structured text according to one or more predefined rules, the one or more predefined rules including one or more of the following: unauthorized terms, unauthorized class names, unauthorized length of the summary of the content, or unauthorized rendering of the summary of the content.

[0137] Example 22: A non-transitory computer-readable medium as described in any one of Examples 17 to 21, wherein the summary of the content includes a description that includes a general view structure and semantic information associated with the graphical user interface.

[0138] Example 23: A non-transitory computer-readable medium as described in any one of Examples 17 to 22, wherein the one or more processors retrieve the information automatically or in response to one or more of mechanical input, audio input, or gesture control.

[0139] Example 24: A computer program product for generating a summary of content included in a graphical user interface, the computer program product comprising one or more instructions, which, when executed by at least one processor, cause the at least one processor to: use an application programming interface to retrieve information associated with a plurality of user interface elements included in the graphical user interface; convert the information into structured text; use a machine learning model to generate a summary of the content included in the graphical user interface by at least applying the machine learning model to the structured text, wherein the machine learning model is a large language model; and output the summary of the content included in the graphical user interface.

[0140] Example 25: A computer program product as described in Example 24, wherein, in order to further output the summary of the content included in the graphical user interface, the one or more processors are further configured to: send the summary of the content included in the graphical user interface to a computing device, wherein the graphical user interface is associated with an application executed on the computing device.

[0141] Example 26: A computer program product as described in Example 24, wherein the information associated with the plurality of user interface elements is generated by the application programming interface.

[0142] Example 27: A computer program product as described in any one of Examples 24 to 26, wherein the information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, a relationship between one or more nodes, or a plurality of accessibility features included in the node.

[0143] Example 28: A computer program product as described in any one of Examples 24 to 27, wherein the machine learning model is applied to the structured text according to one or more predefined rules, the one or more predefined rules including one or more of the following: unauthorized terms, unauthorized class names, unauthorized length of the summary of the content, or unauthorized rendering of the summary of the content.

[0144] Example 29: A computer program product as described in any one of Examples 24 to 28, wherein the summary of the content includes a description that includes a general view structure and semantic information associated with the graphical user interface.

[0145] Example 30: A computer program product as described in any one of Examples 24 to 29, wherein the one or more processors automatically or in response to one or more of mechanical input, audio input, or gesture control retrieve the information.

[0146] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method comprising: The computing system uses an application programming interface to retrieve information associated with multiple user interface elements included in the graphical user interface; The computing system converts the information into structured text; The computing system uses a machine learning model to generate a summary of the content included in the graphical user interface by applying at least the machine learning model to the structured text, wherein the machine learning model is a large language model; and The computing system outputs a summary of the content included in the graphical user interface.

2. The method as described in claim 1, wherein, The output of the summary of the content included in the graphical user interface further includes: A summary of the content included in the graphical user interface is sent to the computing device, wherein the graphical user interface is associated with an application running on the computing device.

3. The method as described in claim 1, wherein, The output of the summary of the content included in the graphical user interface further includes: The audio output device of the computing system outputs a summary of the content included in the graphical user interface, wherein the graphical user interface is associated with an application running on the computing system.

4. The method according to any one of claims 1 to 3, wherein, The information associated with the plurality of user interface elements is generated by the application programming interface.

5. The method according to any one of claims 1 to 4, wherein, The information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, the relationship between one or more nodes, or a plurality of accessibility features included in the node.

6. The method according to any one of claims 1 to 5, wherein, The machine learning model is applied to the structured text according to one or more predefined rules, which include one or more of the following: unauthorized terms, unauthorized class names, unauthorized length of the summary of the content, or unauthorized rendering of the summary of the content.

7. The method according to any one of claims 1 to 6, wherein, The summary of the content includes a description, which includes general view structure and semantic information associated with the graphical user interface.

8. The method according to any one of claims 1 to 7, wherein, The computing system retrieves the information automatically or in response to one or more of mechanical input, audio input, or gesture control.

9. A computing system, comprising: One or more processors; and One or more storage devices storing instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: Use application programming interfaces to retrieve information associated with multiple user interface elements included in the graphical user interface; The information is converted into structured text; Using a machine learning model, a summary of the content included in the graphical user interface is generated by applying the machine learning model at least to the structured text, wherein the machine learning model is a large language model; and Output a summary of the content included in the graphical user interface.

10. The computing system of claim 9, wherein, To further output the summary of the content included in the graphical user interface, the one or more processors are further configured to: A summary of the content included in the graphical user interface is sent to the computing device, wherein the graphical user interface is associated with an application running on the computing device.

11. The computing system of claim 9, wherein, To further output the summary of the content included in the graphical user interface, the one or more processors are further configured to: The audio output device of the computing system outputs a summary of the content included in the graphical user interface, wherein the graphical user interface is associated with an application running on the computing system.

12. The computing system as claimed in any one of claims 9 to 11, wherein, The information associated with the plurality of user interface elements is generated by the application programming interface.

13. The computing system as described in any one of claims 9 to 12, wherein, The information associated with the plurality of user interface elements further includes one or more of the following: node type, text content associated with the node, actions that can be performed using the node, the relationship between one or more nodes, or a plurality of accessibility features included in the node.

14. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform any of the methods described in claims 1 to 8.

15. A computer program product for generating a summary of content included in a graphical user interface, the computer program product comprising one or more instructions that, when executed by at least one processor, cause the at least one processor to: Use application programming interfaces to retrieve information associated with multiple user interface elements included in the graphical user interface; The information is converted into structured text; Using a machine learning model, a summary of the content included in the graphical user interface is generated by applying the machine learning model at least to the structured text, wherein... The machine learning model is a large language model; and Output a summary of the content included in the graphical user interface.

Images