Neural network-based multi-modal transformer for multi-task user interface modeling

Abstract

A method includes receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device. The method also includes generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot. The method also includes predicting, by the neural network and based on the generated representation, a modeled task output associated with the graphical user interface. The method also includes providing, by the computing device, the predicted modeled task output.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 221,677, filed July 14, 2021, the entirety of which is incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to methods and apparatuses for multi-task user interface modeling. BACKGROUND

[0004] Neural networks can be trained to predict aspects of modeling tasks related to graphical user interfaces, such as content, functionality, layout, and the like. Modern graphical user interfaces open up a rich problem space for modeling, where inputs are inherently multi-modal and composed of several different types of data. Based on graphical user interfaces, there are a large number of modeling tasks that can directly enhance end-user experiences and advance the development of intelligent user interfaces. SUMMARY

[0005] In one aspect, a computer-implemented method is provided. The method includes receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device. The method further includes generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot. The method further includes predicting, by the neural network and based on the generated representation, a modeling task output associated with the graphical user interface. The method further includes providing, by the computing device, the predicted modeling task output.

[0006] In another aspect, a computing device is provided. The computing device includes one or more processors and a data storage device. Stored on the data storage device are computer- executable instructions that, when executed by the one or more processors, cause the computing device to perform functions. The functions include receiving, via the computing device, a screenshot of a display provided by a graphical user interface of the computing device; generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; predicting, by the neural network and based on the generated representation, a modeling task output associated with the graphical user interface; and providing, by the computing device, the predicted modeling task output.

[0007] In another aspect, a computer program is provided. The computer program includes instructions that, when executed by a computer, cause the computer to perform functions. The functions include receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device; generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; predicting, by the neural network and based on the generated representation, a modeled task output associated with the graphical user interface; and providing, by the computing device, the predicted modeled task output.

[0008] In another aspect, an article of manufacture is provided. The article of manufacture includes one or more computer-readable media having stored thereon computer-readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions. The functions include receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device; generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; predicting, by the neural network and based on the generated representation, a modeled task output associated with the graphical user interface; and providing, by the computing device, the predicted modeled task output.

[0009] In another aspect, a computing device is provided. The computing device includes means for receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device; means for generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; means for predicting, by the neural network and based on the generated representation, a modeled task output associated with the graphical user interface; and means for providing, by the computing device, the predicted modeled task output.

[0010] The foregoing overview is merely illustrative and is not intended to limit the scope of the application in any way. Further aspects, embodiments, and features will become apparent from the following detailed description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a diagram illustrating an example neural network according to example embodiments.

[0012] Figure 2A illustrates example predicted results for a UI object detection task according to example embodiments.

[0013] Figure 2B illustrates additional example predicted results for a UI object detection task according to example embodiments.

[0014] Figure 3 An example of a language command detection task is shown, in accordance with example embodiments.

[0015] Figure 4 An example of a screen summarization task is shown, in accordance with example embodiments.

[0016] Figure 5 An example of a widget captioning task is shown, in accordance with example embodiments.

[0017] Figure 6 An example of a tappability prediction task is shown, in accordance with example embodiments.

[0018] Figure 7 is a diagram showing the training and inference phases of a machine learning model, in accordance with example embodiments.

[0019] Figure 8 A distributed computing architecture is depicted, in accordance with example embodiments.

[0020] Figure 9 is a block diagram of a computing device, in accordance with example embodiments.

[0021] Figure 10 A network of computing clusters arranged as a cloud-based server system is depicted, in accordance with example embodiments.

[0022] Figure 11 is a flowchart of a method, in accordance with example embodiments. DETAILED DESCRIPTION

[0023] The present application relates to a transformer-based neural network that can accept multi-modal input and can simultaneously accomplish multiple modeling tasks for a graphical user interface. Tasks can include, for example, UI object detection, natural language command grounding, widget captioning, screen summariztion, and UI tappability prediction. The model can be configured to process three types of data: images, structure (e.g., view hierarchy), and natural language.

[0024] The flexible architecture of the transformer makes it a “Swiss army knife” for solving a variety of problems. In addition to success in handling individual domains that deal with homogenous input or output (like natural language and vision), the transformer architecture has recently shown promising results in handling problems that involve multi-modal input, multi-task output, or both.

[0025] Described herein are tasks for modeling graphical user interfaces, which are an important medium that underpins almost every aspect of daily human activity. Modern graphical user interfaces open up a rich problem space for modeling, where input is inherently multi-modal, consisting of several different types of data. User interface screens exist in both visual form (i.e., screenshots) and structural representation (i.e., tree view hierarchy). Based on graphical user interfaces, there are a large number of modeling tasks that would directly enhance end-user experience and propel the development of intelligent user interfaces. For example, existing methods develop models and datasets for grounding language commands to executable UI actions, generating language descriptions for accessibility on mobile devices, and understanding the usability of user interfaces, or recognizing objects on the screen. Previous work also starts to learn effective representations of user interface screens, which can potentially benefit downstream tasks.

[0026] A versatile user interface transformer (VUT) is described, which can handle three types of data: image, structure (view hierarchy), and language, and can perform a variety of different tasks, such as UI object detection, natural language command grounding, widget-to-text, screen summarization, and UI tapability prediction.

[0027] The VUT can perform different tasks simultaneously. Typically, using different models for different tasks results in a large amount of computational resources, including memory resources, processing resources, and / or power resources. This is particularly challenging when the tasks must be performed on a mobile device, such as a mobile phone. Therefore, using one model to perform all different tasks can greatly reduce the amount of computational resources required.

[0028] The VUT is a multi-modal model for graphical user interface multi-task modeling, accomplishing a wide range of tasks through one model to enhance mobile user experience.

[0029] The VUT can be based on a dual-tower transformer architecture, one for image structure and the other for language, where each transformer encodes and decodes its own modality with cross-tower attention.

[0030] The image-structure transformer can serve as both an encoder and a decoder. The image-structure transformer of the VUT can perform early fusion across modalities. Instead of operating across language and image regions, the image-structure transformer of the VUT operates on the entire screenshot image and view hierarchy substructure. This is able to improve the efficiency and accuracy of task performance. The image-structure transformer of the VUT is used not only for representation learning, but also for object detection when the view hierarchy information is not present in the input, such as for object detection tasks.

[0031] The image-to-structure converter of VUT is a single tower architecture where both image and object query are input to the same converter, i.e., early fusion, instead of the encoder-decoder architecture used in traditional models.

[0032] The question-and-answer converter of VUT is designed based on an auto-regressive architecture where the question or command is input as a prefix to the model and the response is decoded token by token.

[0033] For the language (command) grounding task, instead of generating a language response as in existing models, the last hidden state of the model is used to retrieve the on-screen UI objects to fulfill the command.

[0034] Using multiple different heads based on the same neural network layers increases efficiency and accuracy and also enables effective separate and / or joint training of one or more tasks.

[0035] Figure 1 is a diagram illustrating an example neural network 100 according to example embodiments. The graphical user interface contains a set of UI elements for completing a coherent set of tasks. There can be various types of data involved in formulating the UI tasks: <S, V, T, Q, A>. S is a screenshot image 102 that describes the visual appearance of the UI screen 102a. V is a view hierarchy 104 tree that represents the underlying structure of the UI screen 102. T is the target object 106 (UI element) to operate or query in the view hierarchy 104. Q is the natural language description of the task, which can be an open-ended question 130 such as “What is the caption of the element?” an is-or-yes question such as “Does the object look clickable?” or a command such as “Click the next button,” etc. When the response form of the task is natural language, the answer A 134 is the natural language answer to the question Q 130.

[0036] In some embodiments, the method includes receiving, via the computing device, a displayed screenshot 102 provided by a graphical user interface of the computing device.

[0037] The method also includes generating, by an image-to-structure transformer 108 of the neural network 100, a representation 120 by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot 102. The image-to-structure model 108 encodes the entire screenshot 102 of the UI and its view hierarchy tree 104 with early fusion of the two modalities.

[0038] In general, the image-to-structure transformer 108 is a dual-modal model that takes in an image such as the screenshot 102 and the corresponding view hierarchy 104 and outputs a hidden representation 120 for each node in the image 102 and the view hierarchy 104. For the image modality, the content embedding 114 CS The image embeddings 110 can be computed, and the content embeddings 114 of the view hierarchy 104 can be computed by the structure embedder 112. Such embeddings can be combined with the pattern and position encodings 116 P S into the image-structure transformer 108. The transformer 118 generally refers to the transformer layer of the neural network 100, where the transformer layer includes the image-structure transformer 108 and the question-answer transformer 126.

[0039] The screenshots can be randomly resized for image augmentation, so there can be different sizes. In some embodiments, a binary non-padding mask S mask can be used for the screenshots S 102. Additionally, for example, tensor reshaping and / or broadcasting can be applied. The content embeddings 114 C S can be determined as:

[0040] C S = Dense(ResNet(S)); P S = PE(S mask ) (Equation 1)

[0041] where, and where M is the number of superpixels after ResNet, and D denotes the dimensionality of

[0042] In some embodiments, the method includes predicting, by the neural network 100, a layout of virtual objects in the screenshots 102. When the view hierarchy 104 is not present in the input, the image-structure model 108 predicts the UI objects. For the view hierarchy modality, when the view hierarchy 104 is not present, the value of the content embedding 114 C v of the view hierarchy modality is set to zero. In some embodiments, the position encoding 116 P v of the view hierarchy modality can be a learned embedding vector for each query position. When the view hierarchy 104 is present in the input, each object in the view hierarchy tree, the view hierarchy 104, is embedded in the context of the entire structure.

[0043] A user interface (UI) object can include a set of attributes, including type, whether clickable, position of the bounding box, document order (DOM) position, textual content, and whether the object is a target. The attributes can be individually embedded into the same dimensionality and then combined by addition to form an embedding for each element.

[0044] These attributes can be individually embedded into the same dimension and then combined by addition to form the embedding of each element E0. Note that for the command landing task 138, information of whether the object is the target can not be present in the input as T will be predicted by the model. C v and P v The method can be similar to the prior art. Whether V is present in the input or not, the embedding of the view hierarchy modality can be two tensors: and

[0045] The method then includes determining, by the neural network 100, a content embedding 114 and a position embedding 116 for each of the screenshots 102 and the view hierarchy 104. Generating the representation 120 by fusion can include generating the representation 120 based on the content embedding 114 and the position embedding 116. Because embeddings from both modalities (the image modality and the view hierarchy modality) will jointly participate in the self-attention of the transformer encoder, it can be advantageous to make their position encodings global to each modality rather than local. To this end, a learnable modality embedding can be added to each of these modality-specific position encodings. The embeddings from both modalities can be concatenated along a first dimension to form the input to the image-structure transformer 108 to output the representation 120.

[0046] For example, P s and P v may be the position encodings within each modality. Typically, the embeddings from both modalities can jointly participate in the self-attention of the transformer encoder. In some embodiments, the position embedding 116 of the screenshots 102 and the position embedding 116 of the view hierarchy 104 can be global embeddings corresponding to the entire screenshot 102. For example, the position encodings can be global rather than local to each modality. To this end, a learnable modality embedding can be added to each of these modality-specific position encodings as follows:

[0047] P′ s = P s + E s ; P′ v = P v + E v (Equation 2)

[0048] where, and are learnable embeddings for the image and view hierarchy modalities, respectively. The embeddings of the two modalities can then be concatenated along a first dimension to form the input to the transformer encoder:

[0049] C = Concat 1st_dim [C s , Cv ] ; P = Concat 1st_dim [P' s , P' v ] (Eq. 3)

[0050] where, and are the final content embedding and position encoding, respectively, which are fed to a multi-layer transformer encoder:

[0051] H = Transformer Encoder (C, P) (Eq. 4)

[0052] hidden representations In some embodiments, H can be separate for the hidden representations of each modality:

[0053] H s = H [:M] ; H v = H [M:] (Eq. 5)

[0054] while resulting in hidden representations for each modality: and

[0055] The image-structure model 108 performs an object detection task 124. For example, the object output layer 122 can be configured to output a UI object detection task 124 based on the representation 120. In some embodiments, the method includes receiving, via a computing device, a view hierarchy 104 that indicates a layout of virtual objects in the screenshot 102.

[0056] The method also includes predicting, by the neural network 100 and based on the generated representation 120, a modeled task output associated with the graphical user interface. As used herein, the term “modeled task output” generally refers to any task associated with a graphical user interface. For example, a graphical user interface can include virtual objects, lists, images, videos, icons, user-selectable icons, user-adjustable controls (e.g., sliders that adjust image features, sound characteristics, etc.), application interfaces, application icons, and input fields for text, images, speech, etc. A graphical user interface can also include these objects, tasks that can be performed, hierarchical structures, and various relationships between display characteristics including color, hue, resolution, etc. Thus, a modeled task output can be the output of any task that identifies various elements, attributes, functions, designs, layouts, etc. of a graphical user interface. Such tasks can include the output of natural language command landing tasks, widget-to-text tasks, screen summarization tasks, object detection tasks, clickable-ability prediction tasks, etc.

[0057] In some embodiments, the modeling of the graphical user interface includes multi-task modeling, and wherein the neural network 100 includes dual transformers, where the image-structure transformer 108 predicts modeling task outputs for image-structure tasks, and the question-answer transformer 126 predicts modeling task outputs for natural language tasks. The question-answer model 126 encodes the question and uses the encoding from the image-structure model 108 to predict the answer. The image-structure transformer 108 and the question-answer transformer 126 are configured with cross tower attention (e.g., encoder-decoder attention 146).

[0058] In some embodiments, the question-answer transformer 126 can be a language model that encodes the question Q 130 and decodes the answer A 134. The process can begin at “BEGIN” 128 and end at “EOS” 132. The input to the model can be X = x 1:t where t denotes the length of the sequence. The input can be a token sequence for the question Q 130 of the command drop task 138, or the concatenation of Q 130 with the decoded answer A’ when the answer A 134 is a language answer to be generated. In some embodiments, during training with a teacher, A = A’ is enforced. During the autoregressive inference process, A’ is a predicted token sequence that depends on the step:

[0059] g 1:t = Transformer Decoder (E(x 1:t ), PE(1:t); H S , H v (Eq. 6)

[0060] where x i is the i-th token in the sequence (1≤i≤t), E(.) and PE(.) compute content embeddings and position encodings for each token in the sequence. H s and H v can be accessed through the encoder-decoder attention 146 in the transformer decoder. The sequence of hidden states, can be used to predict the next token to generate the answer or to retrieve the target UI object in the view hierarchy of the command drop task 138.

[0061] In some embodiments, the modeling task output is for one or more of the following tasks: object detection task 124, natural language command grounding task 138, widget plus text task 140, screen summarization task 142, or tappability prediction task 144. When the task output is language, the question-answer model 126 directly implements the task, e.g., widget plus text task 140, screen summarization task 142, and tappability prediction task 144. For the command grounding task 138, the representation 118 of the image-structure converter 108 can instead be used to locate the UI element on which to act.

[0062] In some embodiments, the neural network 100 includes an object detection head, a text head, and a pointer head. These heads are based on the hidden representation 120. For example, an object detection head such as the object output layer 122 can be used for the UI object detection task 124, where H v can be used as an input layer. Then,

[0063]

[0064] where, is a linear projection of the output object type logits, where represents one more than the number of UI object classes (e.g., 21 + 1 = 22). In some embodiments, an additional padding type can be included on top of the original UI object classes. This function is a multi-layer perceptron parameterized by bbx and is a linear projection to generate coordinates. The logits are used to generate both object predictions and compute the best compound loss using Hungarian Matching during training.

[0065] The text head can be used for tasks with text response components, such as answer A 134, a softmax layer can be applied to the decoder hidden states g 1:t (top of the softmax layer as determined in Equation 6) to generate each answer token a

[0066] a i = argmax(Softmax(g |Q|+i-1 W txt ))(Equation 8)

[0067] where, i is the i-th token in the answer sequence A, and |Q| is the length of the question. Furthermore, represents the learnable weights, and |vocab| is the vocabulary size. For each task with a text response component, the model can be optimized for cross-entropy loss on the predicted and ground truth answer label sequences.

[0068] The indicator head or landing indicator 136 can be used to command the landing task 138, and the last hidden state from the transformer decoder can be used as an "indicator" to match all objects in the view hierarchy 104 based on their hidden representations 120 using the following dot product similarity:

[0069]

[0070] Among them, h j It is H v In the j-th row, H v It is the hidden representation of the j-th object in view hierarchy 104. This term... Let g represent a learnable projection, and g |Q| The last hidden state represents the result from the decoder (as determined using Equation 6), which allows access to the entire question (command) sequence Q130 via decoder self-attention, and this last hidden state can be used as an "indicator" instead of pooling the word packets embedded by span. The model can be optimized by minimizing the cross-entropy loss between the predicted and baseline truth object indices.

[0071] In some embodiments, the modeling task output is used for object detection task 124, and the method includes detecting one or more types of container objects that indicate the layout hierarchy of screenshot 102 via neural network 100. Given screenshot image S 102, object detection task 124 is to detect each UI element on the screen. In some embodiments, the modeling task output is used for object detection task 124, and the method includes detecting one or more of text fields, toggle buttons, or image views via neural network 100. This task is challenging because it requires detecting different types of container objects that determine the layout hierarchy of the screen. In some embodiments, the layout hierarchy includes one of linear layout, frame layout, or list. Detecting these objects is an important step toward providing accessibility features or refactoring or adapting the UI when the view hierarchy structure 104 is unavailable. As a screen understanding task, this task can benefit improvements to other UI modeling tasks. The task is described as:

[0072] V = F(S) (Equation 10)

[0073] In some embodiments, the object detection task 124 can be implemented based on a single-tower image-to-structure converter and does not depend on a question-and-answer model.

[0074] In some embodiments, the modeled task output is for a widget-to-text task 140, and the method includes predicting, by the neural network 100 and for the screenshot 102, a natural language description of a predicted function of a virtual object in a graphical user interface. Generally, generating natural language descriptions for user interface elements is an important task for accessibility and language-based interaction. In some embodiments, given a UI view hierarchy 104, a screenshot image S 102, and a target element T 106, the model predicts a natural language phrase A 134 that describes the function of the object. This relationship can be described as:

[0075] A = F(S, V, T, Q) (Equation 11)

[0076] The model uses information from S 102 and the view hierarchy 104 by the image-to-structure model 108. Examples of questions Q 130 in the widget-to-text task 140 can include “What is the text of the element?” or “What best describes the object?” Examples of answers A 134 can include “retweet” or “shopping cart.” The widget-to-text task 140 extends techniques from classic image-to-text tasks to the UI domain.

[0077] The method also includes providing, via the computing device, the predicted modeled task output.

[0078] In some embodiments, the modeled task output is for a widget-to-text task 140, and the method includes predicting, by the neural network 100 and for the screenshot 102, a natural language description of a predicted function of a virtual object in a graphical user interface.

[0079] In some embodiments, the modeled task output is for a screen summary task 142, and the method includes predicting, by the neural network 100, a summary of the screenshot 102 for a graphical user interface screen summary task 142, which is a task of generating a summary that describes the entire screen, as determined by Equation 3 below, rather than focusing on a single element as in the widget-to-text task 140.

[0080] A = F(S, V, Q) (Equation 12)

[0081] Some examples of questions Q 130 for the screen summary task 142 are “What is the description of the screen?” or “What best summarizes the UI?” This task is broadly related to multi-modal summarization tasks in existing methods, but is specific to the user interface domain.

[0082] A useful feature of modern smartphone interfaces is the interpretation of natural language commands from the user into executable actions, such as voice control. In the language command grounding task 138, given a UI, S, and V, and a language command Q, the model needs to predict which object on the screen can fulfill the language command. This can be determined as:

[0083] T = F(S, V, Q) (Equation 13)

[0084] Thus, the method further includes associating the target virtual object with a natural language command. The method further includes providing the natural language command through the graphical user interface. Note that this task positions the target object T 106 on the screen, rather than generating a natural language response as the widget-plus-text task 140 and the screen summary task 142 do. The possibilities for Q 130 can be infinite, it can be any phrase that the user enters in order to manipulate the UI. Some example questions Q 130 can be “go to the next screen” or “tap the checkout button.” The command can also be indirect to a certain object, for example “click the icon to the right of the search box.”

[0085] In some embodiments, providing the natural language command includes displaying the natural language command at or near the target virtual object. In some embodiments, providing the natural language command includes providing the natural language command as a voice command in response to a user interaction with the target virtual object. An important feature of modern smartphone interfaces is to interpret the user’s natural language commands as actionable actions, for example voice control.

[0086] In some embodiments, the modeling task output is for the tapability prediction task 144, and the method involves: for the graphical user interface, identifying a mismatch between the tapability features that the developer designed and the tapability features that the user perceives. Whether a user perceives a UI object as clickable or not is an important usability issue. A mismatch between the tapability that the user perceives and the tapability that the designer or developer intended can adversely affect the mobile user experience. In the tapability prediction task 144, given the UI S 102 and the view hierarchy 104, the target T 106 of the expanded query, and the asking question Q 130, the model provides a yes or no answer A 134. This can be determined as:

[0087] A = F(S, V, T, Q) (Equation 14)

[0088] The method further involves providing the developer of the graphical user interface with a suggestion to offset the identified mismatch. Typically, the tasks share the image-to-structure transformer 108. In addition to the UI object detection task 124, other tasks also share the question-answer transformer 126. As a result, the natural language input Q is the task indicator for this task. Q also carries the actual task details for the landing task, in order to find the object on the UI.

[0089] For the clickable prediction task, synthetic yes or no questions can be generated based on the following regular expression pattern. The model is trained to decode “yes” or “no” as the answer to the question: “[object | element | widget | control] [clickable | tappable]?”. In some embodiments, examples of questions generated based on the regular expression are, for example, “Is the object clickable?”, “Is the widget clickable?”, “Is the element clickable?”, and so on.

[0090] For the widget plus text task 140 and the screen summary task 142, the model will need to generate open-ended answers. In some embodiments, the following regular expression can be used to generate questions for these tasks. The VUT can be trained to decode a screen summary or widget text that follows the question: “What is the [summary | description] of the [screen | UI]?” or “What is the [text | description] of the [object | element | widget | control]?” Some questions generated based on the regular expression can be: “What is the summary of the screen?” “What is the description of the UI?” “What is the text of the widget?” “What is the description of the object?” and so on.

[0091] For the language command landing task 138, a command involving a particular object in the screen can be fed to the model, which is trained to locate the object involved. Example commands can be generated by human annotators for the target UI objects displayed on the screen. For example, a human annotator can be asked to come up with different commands for each highlighted target object. Commands such as “click on the notification bar above the status option”, “press the back arrow button”, “select the icon above the clock option”, “swipe down the notification bar” can be generated by human annotators.

[0092] The method also involves training the neural network 100 to receive an input screenshot displayed by a particular graphical user interface and to predict a modeling task output associated with modeling of the particular graphical user interface. For the UI object detection task, RICO, a public corpus of mobile user interfaces containing 64462 unique Android screens from 9362 different applications can be used for training. Each screen includes an RGB screenshot and a corresponding view hierarchy. The view hierarchy is a tree structure with 21 unique types of nodes, which can be merged from the Android view class attributes in the original dataset. The nodes in the tree correspond to UI elements on the screen or container elements that manage their child nodes’ layouts. In some embodiments, the view hierarchy can have up to 128 nodes in the dataset. For example, the data can be split into a training set (54611), a validation set (2518), and a test set (2627). Additional and / or alternative datasets are possible, with various distributions for the training set, validation set, and test set.

[0093] For the widget-plus-text task, a public dataset can be used. The released dataset includes more than 200,000 human annotations for more than 46,000 unique UI objects from 17,000 RICO screens. The annotated UI elements can be split for training (39,951), validation (3,436), and testing (3,531). In some embodiments, the dataset can be split by application, such that screens of the same application can only appear in one split.

[0094] A screen summary dataset was collected for 22,301 unique Android screens. Based on the displayed UI screen, human workers were asked to generate 3-5 summaries for the screen. The maximum length of a summary was 10 words. In some embodiments, the dataset can be divided into a training set (17,569 screens), a validation set (2,298), and a test set (2,434).

[0095] The tappability prediction dataset includes tappability annotations for more than 20,000 UI elements from 3,218 Android screens. In data collection, given a target UI element highlighted on the screen, human evaluators were asked to answer "yes" or "no" for whether the target object was clickable to them. In some embodiments, the dataset can be divided into training (14,783), validation (1,854), and testing (2,029).

[0096] The language landing dataset includes 10,000 human annotations for manipulating UI objects of 1,432 unique screens from 26 Android built-in applications, such as Settings. Human evaluators generated commands such as "click the button below the battery information," and the maximum length of a command phrase was 20 words. In some embodiments, the dataset can be divided into training (7,822), validation (1,024), and testing (987).

[0097] When splitting each dataset into training, validation, and test sets, it can be desirable to ensure that there is no application (or screen) overlap between the training sets and any test sets for different tasks. This can be important because, in the multi-task learning case, the VUT learns from all of the training sets. Therefore, it is preferable that the union of applications and screens across all training sets does not overlap with any test set.

[0098] In some embodiments, the training can be performed at a computing device.

[0099] In some embodiments, the prediction modeling task output involves: obtaining, at the computing device, a trained neural network; and applying the obtained trained neural network to a prediction of the modeling task output.

[0100] In some embodiments, predicting the modeling task output involves determining, by the computing device, a request for the predicted modeling task output. The method also involves sending, from the computing device to a second computing device, the request for the predicted modeling task output, the second computing device comprising a trained version of the neural network. After sending the request, the method involves the computing device receiving, from the second computing device, the predicted modeling task output.

[0101] Some example model parameters are provided for illustrative purposes and are not to be construed as limiting the scope of the claims. For the UI object detection task, the VUT can be configured with a 12-layer transformer encoder as the image-structure model, which reaches 48,000,000 trainable parameters, which is slightly less than the 50,000,000 trainable parameters of the DETR with a 6-layer encoder and a 6-layer decoder. For the remaining tasks, the VUT can be configured with a 6-layer transformer encoder for the image-structure model and a 6-layer transformer decoder for the question-answer model. When all tasks are jointly trained, there are 64,000,000 parameters. The heads and the token embeddings and projections for specific tasks are the main factors of the parameter size growth. When only a subset of these tasks is involved in the training, e.g., widget plus text and object detection, there can be fewer trainable parameters involved as only a fraction of the full model is used. All VUT variants use the following configurations: #Attention_Heads = 8, Hidden_Dimension = 2568, Transformer_MLP_Dimension = 20488, Transformer_QKV_Dimension = 256.

[0102] All tasks except for the UI object detection require the model to encode the view hierarchy. To this end, each object in the view hierarchy is represented as a content embedding Cvand a position encoding P v The content embedding embeds the properties of the object, such as the type, the textual content, and the clickable properties. For the textual content, it can treat all the word piece tokens that the object possesses as a “bag of words”. Each token can be assigned a learnable embedding, and then max-pooling can be performed on the collection of embeddings to obtain a fixed-length embedding vector to represent the textual content of the object. The embedding of each content property can be added to form the content embedding of the object.

[0103] Because the flattened view hierarchy is fed to the transformer 118, the positional encodings are configured to capture both spatial position and structural position of the object. Spatial position includes the four coordinate values of the bounding box of the object, i.e., [top, left, bottom, right], and structural position includes three DOM position attributes, including the index position of the object in the pre-order and post-order traversal of the hierarchy, and the depth of the object in the hierarchy. Each type of position can be encoded using a sinusoidal representation. Note that in the image-structure model 108, the positional encodings 116 are added to the input of each layer of the transformer 118. This is in contrast to the question-answer model 126, in which the positional encoding of each token is only added to the input of the first layer. The learned embeddings for positional encodings are used in the question-answer model 126. During training, 10% dropout can be used for the attention and MLP dropout in the question-answer transformer 126, and 20% dropout can be applied to the encoding from the image-structure model 108 before the cross-attention 146. During 5-task joint learning, the attention and MLP dropout rate can be 20% for the image-structure transformer. During the interference autoregressive decoding, the maximum decoding length can be 30, which covers the total length of the question and answer.

[0104] The phrases can be tokenized into sequences of word pieces in a manner similar to that used in BERT, which results in a vocabulary size of 28536. The maximum size of a screenshot image can be 1080 x 1080. Each image can be randomly resized for image augmentation. The maximum number of UI objects and containers on each screen can be capped at 128. The VUTs can be implemented based on the library JAX2 for machine learning. In some embodiments, each VUT model can be trained with a batch size of 64 screens / examples, which is parallelized over 64 TPU v3 cores.

[0105] Example Applications

[0106] In some embodiments, user interface modeling is described. In one example implementation, a cloud-based developer tool can be provided. For example, a platform for designing and / or improving a GUI can be provided to a developer. As described herein, the neural network 100 can output predicted tasks, and an interactive developer tool can be provided to the developer.

[0107] For example, the tapability of on-screen objects can be identified, and a mismatch between the tapability features designed by the developer and the tapability features perceived by the user can be determined. The cloud-based developer tool can then provide such information to the developer to enable improvement of the tapability features. Moreover, for example, in some embodiments, such a cloud-based developer tool can be a substantially real-time developer tool that models a GUI, predicts modeling task outputs, and provides suggestions substantially in real-time.

[0108] In another example implementation, the neural network 100 can predict modeling task outputs that can be used to improve the user experience of an end user of a mobile device. For example, it is described to ground language commands into executable UI actions. This can enable improved user experience.

[0109] Further, for example, it is described to generate language descriptions for accessibility on a mobile device. As another example, a summary of a GUI can be performed and provided to a user. These features can also enable an enhanced user experience, especially with text and / or voice commands to help a user navigate a GUI and / or multiple screens of a GUI.

[0110] In some embodiments, it is described to understand the usability of a user interface, along with identifying objects on a screen.

[0111] Additional and / or alternative applications are possible. For example, one or more features can be made available to developers to help with the task of developing a user platform through a GUI. For example, widget plus text can enable a developer to use the output of the widget plus text task, rather than needing to manually annotate a widget.

[0112] Further, for example, an object detection task can provide a developer with a layout hierarchy of objects in a GUI, an index of objects, their functionality, and so on. Such predictive outputs of the neural network 100 can significantly reduce the time and resources allocated to a development task to be performed by a developer, and also enhance the accuracy of the development task. This can be an important application for the development of mobile platforms (e.g., such as Android-based systems).

[0113] These tasks can be performed by a single neural network model, which can be jointly trained to perform all tasks, a particular subset of all tasks that are jointly trained to perform, and / or independently trained to perform a task. This selection can depend on the platform (e.g., mobile or cloud-based platform), the allocation of available resources to processors, memory, power, network bandwidth, and so on, and can depend on the target audience (e.g., end user, developer, and so on).

[0114] One or more such example features can be provided via a cloud-based platform, as an interactive feature, as a platform-as-a-service (PaaS) platform, a software-as-a-service (SaaS) platform, and / or a machine learning-as-a-service (MLaaS) platform. As described herein, the applications can improve the user experience of a user of a mobile phone, provide ease-of-use features for an end user of a GUI, assist a developer in designing an application based on an operating system of a mobile platform such as a mobile device, assist a developer in troubleshooting various aspects of a GUI, and so on.

[0115] Figure 2A and Figure 2B Example prediction results for UI object detection tasks are shown, according to example embodiments. Examples of the prediction versus ground truth for each task on test data are shown, as implemented by a single model of VUT when it jointly learned all tasks. Reference is made to Figure 2A Image 205 shows a ground truth image for a user interface for a search function, and image 210 shows a predicted image for the same user interface. Reference is made to Figure 2B Image 215 shows a ground truth image for a user interface for a web login page function, and image 220 shows a predicted image for the same user interface.

[0116] Figure 3 Examples of language command detection tasks are shown, according to example embodiments. In each screenshot, the object located by the model is highlighted with a bounding box having a dashed border. For example, in image 305, a search page with a search box and a voice icon is displayed. The bounding box 305a around the voice icon is highlighted, and the command detection task can predict the command “Tap the voice icon.” As another example, in image 310, a page with a list of applications is displayed. The bounding box 310a around the calendar icon (appearing below the calculator icon) is highlighted, and the command detection task can predict the command “Press the icon below the calculator icon.” Further, for example, in image 315, a home page with a weather notification is displayed. The bounding box 315a around the temperature display is highlighted, and the command detection task can predict the command “Select the weather text below the notification bar.”

[0117] Figure 4 Examples of screen summarization tasks are shown, according to example embodiments. The reference (ground truth summary) created by a human annotator for each screen is displayed along with the prediction from the neural network. For example, in image 405, a location-based search page is displayed. The human annotator can have created a ground truth summary such as “Page showing search box in application,” and the screen summarization task can predict that the screen provides “Search bar to search for locations.” As another example, in image 410, a page from a media playback application is displayed. The human annotator can have created a ground truth summary such as “Page shows music playing on application,” and the screen summarization task can predict that the screen provides “Page showing music tracks in music application.” Further, for example, in image 415, a page for account setup is displayed. The human annotator can have created a ground truth summary such as “Pop-up display to set up account details,” and the screen summarization task can predict that the screen provides “Pop-up display to create account.”

[0118] Figure 5Examples of a widget-plus-text task are shown in accordance with example embodiments. The target element is highlighted by a bounding box with a dashed border. One of three references (ground truth text) created by a human annotator for each target element is displayed. For example, in image 505, a page with words and / or phrases to be translated from one language to another language is displayed, with a text entry portion 505a, a list of French words in portion 505b, and a “copy” widget with a bounding box 505c. The ground truth text can be “copy to clipboard option,” and the widget-plus-text task can display a bounding box 505c around the “copy” widget, and the predicted widget text is “copy text.” As another example, in image 510, a page with an emoji application displaying multiple emojis is displayed. The ground truth text can be “select emoji 8,” and the widget-plus-text task can display a bounding box 510c around the “emoji 8” widget, and the predicted widget text is “select emoji.” Further, for example, in image 515, a page with a user profile is displayed. The ground truth text can be “enter and confirm password,” and the widget-plus-text task can display a bounding box 515a around the “confirm password” input field widget, and the predicted widget text is “enter password.”

[0119] Figure 6 Examples of a tapability prediction task are shown in accordance with example embodiments. The element of interest is highlighted by a bounding box with a dashed border. For example, in image 605, the tapability prediction task can be to predict the tapability of the icon of the user image. The ground truth can indicate that the icon is tapable, and the model can predict that the icon is tapable and place a bounding box 605a around the icon. As another example, in image 610, the tapability prediction task can be to predict the tapability of the icon with the text “More about your information.” The ground truth can indicate that the icon is not tapable, and the model can predict that the icon is not tapable and place a bounding box 610a around the icon. Further, for example, in image 615, the tapability prediction task can be to predict the tapability of the download icon of “widget 3.” The ground truth can indicate that the icon is tapable, and the model can predict that the icon is tapable and place a bounding box 615a around the download icon of “widget 3.”

[0120] These and other example applications are contemplated within the scope of the present disclosure.

[0121] Experimental Results

[0122] Comparison of VUT and DETR

[0123] The image-structure model 108 described here shares some aspects with the existing transformer-based model for end-to-end object detection architecture (DETR). Thus, the two models can be compared for the UI object detection task. In this experiment, DETR can be configured to use a 6-layer transformer encoder and a 6-layer transformer decoder, and have a similar number of parameters in the model, and the VUT image-structure can be configured to use a 12-layer transformer encoder in this experiment. DETR (50,000,000 parameters) has slightly more parameters than VUT (48,000,000 parameters) due to the weights associated with the encoder-decoder attention 146. The experiment shows that the image-structure model significantly outperforms the encoder-decoder architecture of DETR. In fact, the DETR experiment found that more encoding layers can significantly improve accuracy. But in the current experiment, the image-structure model of VUT uses an encoder-only architecture and also achieves better accuracy. The experiment shows that the multi-modal encoding method presented here works well for the object detection task.

[0124] Single-task training with VUT

[0125] To understand how well VUT performs when learning multiple tasks jointly, a baseline can be established by training VUT based on each dataset individually. Each model can be trained until it converges. For the UI object detection task, the model can be trained using the default settings of DETR, with a batch size of 64 for 300,000 iterations. The learning rate schedule includes a learning rate decay from 1e -4 to 1e -5 -3 at 200,000 steps. In this experiment, a 6-layer image-structure encoder is used in VUT, with 8-headed attention and a hidden size of 256. This model achieves an AP = 37.0, AP50 = 47.6, and AP75 = 38.8. Note that this accuracy is lower than previously reported results using CenterNet on different UI datasets. However, these results cannot be directly compared. The task for the VUT model is more challenging because the VUT model is trained to detect 21 different UI object types including several container elements, whereas there were 12 objects in the previous work. Furthermore, the previous work used a dataset that was manually labeled by humans and also employed heavy post-processing to improve predictions.

[0126] For the widget-plus-text task, both the 6-layer image-structure model and the 6-layer question-answer model can be used in addition to the text heads. Similarly, the model can be trained with a batch size of 64 until it converges, which can take 45,000 steps. The VUT model achieves comparable accuracy to existing models, despite the model architecture being significantly different from previous work. Table 1 below provides the results for the widget-plus-text task:

[0127]

[0128] Table 1

[0129] For the screen summary task, the same settings can be used as for the model trained for the widget-plus-text, and the model converges at 50,000 steps. The accuracy achieved by the VUT is seen in Table 2:

[0130]

[0131] Table 2

[0132] The language command landing task uses similar model settings as the widget-plus-text and screen summary tasks, except that it uses the landing heads instead of the text heads. The model can take approximately 27,000 steps to converge with a batch size of 64 (see results in Table 3 below). To train the model for each of these tasks, the learning rate can be decayed from 1e -4 to 1e -5 .

[0133]

[0134] Table 3

[0135] To learn the clickable prediction task in isolation, the same model settings are used as for the two text-related tasks (summary and text). We find that the model is very prone to overfitting, despite using a larger dropout rate. Thus, we train the model with a batch size of 64, while early-stopping. Despite using a very different model architecture here, the accuracy (Table 4 below) is comparable to previously published results.

[0136]

[0137] Table 4

[0138] Jointly training the VUT for multiple tasks

[0139] The performance of the VUT on multiple tasks simultaneously can be evaluated. In this experiment, both the 6-layer image-to-structure transformer and the 6-layer question-to-answer transformer can be used with all task heads. Each task head and model component is only used when the particular task requires it. The entire model can be implemented based on Jax.

[0140] Because the UI object detection task requires more iterations than the other tasks, multi-task learning can be initiated by training the VUT for the UI object detection task, and then jointly training the VUT for all tasks by alternating batches from each dataset. This learning strategy is reasonable because by learning from the UI object task, the model can learn useful information about how to encode screen pixels. As consistently shown in the experiments, joint learning involving object detection can generally facilitate the learning of the other four tasks.

[0141] Based on such a multi-task learning strategy, the VUT model can first be trained for the UI object detection for about 300k steps. Then, the model can be trained for an additional 200k steps for all tasks together. While training, the model can alternate between the 5 datasets and tasks, and can use one batch from one dataset at a time. As shown in Tables 1, 2, 3, and 4 above, multi-task learning, while more challenging than single-task learning, can generally perform at the same level as single-task learning. For the widget-plus-text, screen summary, and clickable prediction tasks, multi-task learning seems to consistently outperform single-task learning. When it comes to text generation related tasks, the accuracy of the landing task 138 can decrease. This is consistent with the model architecture because the landing task relies on the last hidden state of the question-answer model, and can compete with the three text generation tasks by “pulling” the hidden representation of the question-answer model in different directions. However, it seems that having the object detection task in multi-task learning often outperforms configurations that do not involve object detection. For the object detection task itself, the accuracy can decrease when the batch changes for multi-task learning begin. However, this gradually recovers its accuracy, especially after the learning rate decay. The accuracy of object detection recovers to AP = 32.5, AP50 = 44.2, and AP75 = 33.7. With careful learning rate scheduling and tuning, the accuracy can be further improved.

[0142] Additional Experimental Results

[0143] As indicated previously, the VUT model can first be trained for the UI object detection task, which helps the model acquire an understanding of the screenshot images and learn to represent the pixels before it is further trained with other tasks. As more tasks participate in the training, the accuracy of the model on the UI object detection task can be affected. Table 5

[0144]

[0145]

[0146] Table 5

[0147] As indicated in Table 5, the accuracy of VUT on the object detection task is mostly preserved when additional tasks are added to the multi-task learning. There is a more pronounced drop in object detection accuracy when all tasks are added to the training, i.e., the last row in the table. Fine-tuning the learning rate schedule and dropout rates for different parts of the model can potentially bring the accuracy to its original level. For example, in all 5 tasks * with smaller dropout in the image structure transformer, no attention dropout, and 10% MLP dropout rate, as it seems that larger dropout would hurt the UI object detection task. At the same time, in this experiment, the MLP and attention dropout rates can be increased to 20% in the question-answer transformer to avoid overfitting to other tasks. In this setting, the accuracy of the UI object detection is better recovered, and the model accuracy on other tasks seems to have a minor impact. These experiments show that it is feasible to not treat object detection as a standalone pre-training task, but as part of multi-task learning, where VUT implements all tasks through a single model.

[0148] Training machine learning model for generating inferences / predictions

[0149] Figure 7 A diagram 700 illustrating a training phase 702 and an inference phase 704 of trained machine learning model(s) 732, in accordance with example embodiments, is shown. Some machine learning techniques involve training one or more machine learning algorithms on an input training dataset to identify patterns in the training data and provide output inferences and / or predictions about the (patterns in the) training data. The resulting trained machine learning algorithm can be referred to as a trained machine learning model. For example, Figure 7 A training phase 702 is shown, in which one or more machine learning algorithms 720 are being trained on training data 710 to become trained machine learning model(s) 732. Then, during an inference phase 704, trained machine learning model(s) 732 can receive input data 730 and one or more inference / prediction requests 740 (possibly as part of the input data 730), and responsively provide one or more inferences and / or predictions 750 as output.

[0150] As such, the trained machine learning model(s) 732 can include one or more models of one or more machine learning algorithms 720. The machine learning algorithm(s) 720 can include, but are not limited to: artificial neural networks (e.g., convolutional neural networks, recurrent neural networks, Bayesian networks, hidden Markov models, Markov decision processes, logistic regression functions, support vector machines, suitable statistical machine learning algorithms, and / or heuristic machine learning systems described herein). The machine learning algorithm(s) 720 can be supervised or unsupervised, and can implement any suitable combination of online and offline learning.

[0151] In some examples, on-device coprocessors such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs) can be used to accelerate the machine learning algorithm(s) 720 and / or the trained machine learning model(s) 732. Such on-device coprocessors can be used to accelerate the machine learning algorithm(s) 720 and / or the trained machine learning model(s) 732. In some examples, the trained machine learning model(s) 732 can be trained, resident, and executed to provide inference on a particular computing device, and / or can otherwise make inferences for a particular computing device.

[0152] During the training phase 702, the machine learning algorithm(s) 720 can be trained by providing at least the training data 710 as training input using unsupervised, supervised, semi-supervised, and / or reinforcement learning techniques. Unsupervised learning involves providing the machine learning algorithm(s) 720 with a portion (or all) of the training data 710, and the machine learning algorithm(s) 720 determine one or more output inferences based on the provided portion (or all) of the training data 710. Supervised learning involves providing the machine learning algorithm(s) 720 with a portion of the training data 710, where the machine learning algorithm(s) 720 determine one or more output inferences based on the provided portion of the training data 710, and accept or correct the output inference(s) based on a correct result associated with the training data 710. In some examples, supervised learning of the machine learning algorithm(s) 720 can be governed by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels can be used to correct inferences of the machine learning algorithm(s) 720.

[0153] Semi-supervised learning involves having a portion, but not all, of the training data 710 with correct results. During semi-supervised learning, supervised learning is used for the portion of the training data 710 with correct results, and unsupervised learning is used for the portion of the training data 710 without correct results. Reinforcement learning involves the machine learning algorithm(s) 720 receiving a reward signal with respect to a prior inference, where the reward signal can be a numerical value. During reinforcement learning, the machine learning algorithm(s) 720 can output an inference and receive a reward signal in response, where the machine learning algorithm(s) 720 are configured to attempt to maximize the numerical value of the reward signal. In some examples, reinforcement learning also utilizes a value function that provides a numerical value representing an expected sum of the numerical values of the reward signal provided over time. In some examples, the machine learning algorithm(s) 720 and / or trained machine learning model(s) 732 can be trained using other machine learning techniques, including but not limited to incremental learning and curriculum learning.

[0154] In some examples, the machine learning algorithm(s) 720 and / or trained machine learning model(s) 732 can use transfer learning techniques. For example, transfer learning techniques can involve a trained machine learning model(s) 732 that is pre-trained on a set of data and additionally trained using the training data 710. More specifically, the machine learning algorithm(s) 720 can be pre-trained on data from one or more computing devices, and the resulting trained machine learning model is provided to the computing device CD1, where CD1 is intended to execute the trained machine learning model during the inference phase 704. Then, during the training phase 702, the pre-trained machine learning model can be additionally trained using the training data 710, which can be derived from the kernel and non-kernel data of the computing device CD1. The further training of the machine learning algorithm(s) 720 and / or pre-trained machine learning model using the training data 710 of CD1 data can be performed using supervised or unsupervised learning. Once the machine learning algorithm(s) 720 and / or pre-trained machine learning model have been trained on at least the training data 710, the training phase 702 can be complete. The resulting trained machine learning model can be used as at least one of the trained machine learning model(s) 732.

[0155] In particular, once the training phase 702 has been completed, the trained machine learning model(s) 732 can be provided to the computing device, if not already on the computing device. The inference phase 704 can begin after the trained machine learning model(s) 732 are provided to the computing device CD1.

[0156] During the inference phase 704, the trained machine learning model(s) 732 can receive input data 730 and generate and output one or more corresponding inferences and / or predictions 750 about the input data 730. As such, the input data 730 can be used as input to the trained machine learning model(s) 732 for providing corresponding inference(s) and / or prediction(s) 750 to the kernel component and the non-kernel component. For example, the trained machine learning model(s) 732 can generate inference(s) and / or prediction(s) 750 in response to one or more inference / prediction requests 740. In some examples, the trained machine learning model(s) 732 can be executed by a portion of other software. For example, the trained machine learning model(s) 732 can be executed by an inference or prediction daemon for ease of providing inferences and / or predictions upon request. The input data 730 can include data from the computing device CD1 that executes the trained machine learning model(s) 732 and / or input data from one or more computing devices other than CD1.

[0157] The input data 730 can include the training data described herein. Other types of input data are also possible.

[0158] The inference(s) and / or prediction(s) 750 can include task outputs, numerical values, and / or other output data produced by the trained machine learning model(s) 732 operating on the input data 730 (and the training data 710). In some examples, the trained machine learning model(s) 732 can use the output inference(s) and / or prediction(s) 750 as input to feedback 760. The trained machine learning model(s) 732 can also rely on past inferences as input for generating new inferences.

[0159] After training, the trained version of the neural network can be an example of the trained machine learning model(s) 732. In this approach, an example of the inference / prediction request(s) 740 can be a request for a predicted modeling task output, and a corresponding example of the inference(s) and / or prediction(s) 750 can be the predicted modeling task output.

[0160] In some examples, a computing device CD_SOLO can include the trained version of the neural network, possibly after training. Then, the computing device CD_SOLO can receive a request for a predicted modeling task output and use the trained version of the neural network to predict the modeling task output.

[0161] In some examples, two or more computing devices CD_CLI and CD_SRV can be used to provide the output image; for example, a first computing device CD_CLI can generate a request for a predicted modeling task output and send it to a second computing device CD_SRV. The CD_SRV can then use a trained version of the neural network to predict the modeling task output and respond to the request for the output class from the CD_CLI. The CD_CLI can then provide the requested output (e.g., using a user interface and / or display, a printed copy, an electronic communication, etc.) upon receiving a response to the request.

[0162] Example data network

[0163] Figure 8 A distributed computing architecture 800 is depicted in accordance with example embodiments. The distributed computing architecture 800 includes server devices 808, 810 that are configured to communicate with programmable devices 804a, 804b, 804c, 804d, 804e via a network 806. The network 806 can correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WW AN, an enterprise intranet, the public Internet, or any other type of network configured to provide a communication path between networked computing devices. The network 806 can also correspond to a combination of one or more LANs, WANs, enterprise intranets, and / or the public Internet.

[0164] Although Figure 8 Although only five programmable devices are shown, the distributed application architecture can serve tens, hundreds, or thousands of programmable devices. Further, the programmable devices 804a, 804b, 804c, 804d, 804e (or any additional programmable devices) can be any kind of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mounted device (HMD), a network terminal, a mobile computing device, etc. In some examples, such as illustrated by the programmable devices 804a, 804b, 804c, 804e, the programmable devices can be directly connected to the network 806. In other examples, such as illustrated by the programmable device 804d, the programmable devices can be indirectly connected to the network 806 via an associated computing device, such as the programmable device 804c. In this example, the programmable device 804c can act as an associated computing device to pass electronic communications between the programmable device 804d and the network 806. In other examples, such as illustrated by the programmable device 804e, the computing device can be part of and / or inside a vehicle, such as a car, truck, bus, boat or ship, airplane, etc. In other examples, not shown in FIG. 8, the programmable devices can be directly and indirectly connected to the network 806. Figure 8 In other examples, not shown in FIG. 8, the programmable devices can be directly and indirectly connected to the network 806.

[0165] The server devices 808, 810 can be configured to perform one or more services requested by the programmable devices 804a-804e. For example, the server devices 808 and / or 810 can provide content to the programmable devices 804a-804e. The content can include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio, and / or video. The content can include compressed and / or uncompressed content. The content can be encrypted and / or unencrypted. Other types of content are also possible.

[0166] As another example, the server devices 808 and / or 810 can provide the programmable devices 804a-804e with access to software for databases, searching, calculations, graphics, audio, video, web / internet applications, and / or other functionality. Many other examples of server devices are also possible.

[0167] Computing device architecture

[0168] Figure 9 is a block diagram of an example computing device 900 in accordance with example embodiments. In particular, Figure 9 The computing device 900 shown in FIG. 10 can be configured to perform at least one function of the neural network 100 and / or the method 1100 and / or perform at least one function related to the neural network 100.

[0169] The computing device 900 can include a user interface module 901, a network communication module 902, one or more processors 903, a data storage 904, one or more cameras 918, one or more sensors 920, and a power system 922, all of which can be linked together via a system bus, network, or other connection mechanism 905.

[0170] The user interface module 901 can be operable to send data to and / or receive data from external user input / output devices. For example, the user interface module 901 can be configured to send data to and / or receive data from user input devices such as a touchscreen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a voice recognition module, and / or other similar devices. The user interface module 901 can also be configured to provide output to user display devices such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices now known or later developed. The user interface module 901 can also be configured to generate audible output utilizing devices such as speakers, speaker jacks, audio output ports, audio output devices, headphones, and / or other similar devices. The user interface module 901 can also be configured with one or more haptic devices that can generate haptic output such as vibrations and / or other output detectable by touch and / or physical contact with the computing device 900. In some examples, the user interface module 901 can be used to provide a graphical user interface (GUI) for utilizing the computing device 900, such as, for example, the graphical user interface of a mobile telephone device.

[0171] The network communication module 902 can include one or more devices that provide one or more wireless interfaces 907 and / or one or more wired interfaces 908 that can be configured to communicate via a network. The wireless interface(s) 907 can include one or more wireless transmitters, receivers, and / or transceivers such as Bluetooth TM transceivers, Zigbee @ transceivers, Wi-Fi TM transceivers, WiMAX TM transceivers, LTE TM transceivers, and / or other types of wireless transceivers that can be configured to communicate via wireless networks. The wired interface(s) 908 can include one or more wired transmitters, receivers, and / or transceivers such as Ethernet transceivers, Universal Serial Bus (USB) transceivers, or similar transceivers that can be configured to communicate via twisted pair wires, coaxial cables, fiber optic links, or similar physical connections to a wired network.

[0172] In some examples, the network communication module 902 can be configured to provide reliable, secure, and / or authenticated communications. For each communication described herein, information for facilitating reliable communications (e.g., guaranteed message delivery) can be provided, possibly as part of a message header and / or trailer (e.g., packet / message sequencing information, encapsulation headers and / or trailers, size / time information, and transmission verification information such as a cyclic redundancy check (CRC) and / or parity value). Communications can be secured (e.g., encoded or encrypted) and / or decrypted / decoded using one or more cryptographic protocols and / or algorithms, such as but not limited to the Data Encryption Standard (DES), the Advanced Encryption Standard (AES), the Rivest-Shamir-Adelman (RSA) algorithm, the Diffie-Hellman algorithm, the Secure Sockets Layer (SSL) or Transport Layer Security (TLS) secure sockets protocol, and / or the Digital Signature Algorithm (DSA). Other cryptographic protocols and / or algorithms can also be used, or in addition to those listed here, to secure (and then decrypt / decode) communications.

[0173] The one or more processors 903 can include one or more general- purpose processors, and / or one or more special-purpose processors (e.g., digital signal processors, tensor processing units (TPUs), graphics processing units (GPUs), application specific integrated circuits, etc.). The one or more processors 903 can be configured to execute computer-readable instructions 906 contained in the data storage 904 and / or other instructions described herein.

[0174] The data storage 904 can include one or more non-transitory computer- readable storage media that can be read and / or accessed by at least one of the one or more processors 903. The one or more computer-readable storage media can include volatile and / or non- volatile storage components, such as optical, magnetic, organic, or other storage or disk storage that can be integrated in whole or in part with at least one of the one or more processors 903. In some examples, data storage 904 can be implemented using a single physical device (e.g., one optical, magnetic, organic, or other storage or disk storage unit), while in other examples data storage 904 can be implemented using two or more physical devices.

[0175] The data storage 904 can include computer-readable instructions 906 and possibly additional data. In some examples, the data storage 904 can include storage required to perform at least a portion of the methods, scenarios, and techniques described herein and / or at least a portion of the functionality of the devices and networks described herein. In some examples, the data storage 904 can include storage for a trained neural network model 912 (e.g., a model of a trained neural network such as the neural network 100). In particular in these examples, the computer-readable instructions 906 can include instructions that, when executed by the one or more processors 903, enable the computing device 900 to provide some or all of the functionality of the trained neural network model 912.

[0176] In some examples, the computing device 900 can include one or more cameras 918. The camera(s) 918 can include one or more image capture devices, such as still and / or video cameras, equipped to capture light and record the captured light in one or more images; that is, the camera(s) 918 can generate an image(s) of the captured light. The one or more images can be one or more still images and / or one or more images used in video imagery. The camera(s) 918 can capture light and / or electromagnetic radiation emitted as visible light, infrared radiation, ultraviolet light, and / or as one or more other frequencies of light.

[0177] In some examples, the computing device 900 can include one or more sensors 920. The sensors 920 can be configured to measure conditions within the computing device 900 and / or conditions in the environment of the computing device 900 and provide data about those conditions. For example, the sensors 920 can include one or more of: (i) sensors for obtaining data about the computing device 900, such as but not limited to a thermometer for measuring the temperature of the computing device 900, a battery sensor for measuring the power of one or more batteries of the power system 922, and / or other sensors that measure conditions of the computing device 900; (ii) identification sensors for identifying other objects and / or devices, such as but not limited to a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode (e.g., a quick response (QR) code) reader, and a laser tracker, where the identification sensors can be configured to read identifiers (such as RFID tags, barcodes, QR codes) and / or other devices and / or objects that are configured to be read and provide at least identification information; (iii) sensors for measuring the position and / or motion of the computing device 900, such as but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a GPS device, a sonar sensor, a radar device, a laser displacement sensor, and a compass; (iv) environmental sensors for obtaining data indicative of the environment of the computing device 900, such as but not limited to, an infrared sensor, an optical sensor, a light sensor, a biological sensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; and / or (v) force sensors for measuring one or more forces (e.g., inertial forces and / or gravitational forces) acting around the computing device 900, such as but not limited to, one or more sensors that measure one or more dimensions of force, torque, ground force, frictional force, and / or a zero moment point (ZMP) sensor that identifies and / or measures the position of the ZMP. Many other examples of sensors 920 are also possible.

[0178] The power system 922 can include one or more batteries 924 and / or one or more external power interfaces 926 for providing power to the computing device 900. Each of the one or more batteries 924 can act as a source of stored power for the computing device 900 when electrically coupled to the computing device 900. The one or more batteries 924 of the power system 922 can be configured to be portable. Some or all of the one or more batteries 924 can be readily removable from the computing device 900. In other examples, some or all of the one or more batteries 924 can be internal to the computing device 900 and thus can not be readily removable from the computing device 900. Some or all of the one or more batteries 924 can be rechargeable. For example, a rechargeable battery can be recharged via a wired connection between the battery and another power source, such as by one or more power sources external to the computing device 900 and connected to the computing device 900 via the one or more external power interfaces. In other examples, some or all of the one or more batteries 924 can be non-rechargeable batteries.

[0179] The one or more external power interfaces 926 of the power system 922 can include one or more wired power interfaces, such as a USB cable and / or a power cord, that enable wired power connections to one or more power sources external to the computing device 900. The one or more external power interfaces 926 can include one or more wireless power interfaces, such as a Qi wireless charger, that enable wireless power connections, such as via the Qi wireless charger, to one or more external power sources. Once a power connection to an external power source is established using the one or more external power interfaces 926, the computing device 900 can draw power from the external power source through the established power connection. In some examples, the power system 922 can include related sensors, such as a battery sensor associated with the one or more batteries or other types of power sensors.

[0180] Cloud-based server

[0181] Figure 10 A cloud-based server system is depicted in accordance with example embodiments. In Figure 10In particular embodiments, the functionality of the neural network and / or computing device can be distributed among the computing clusters 1009a, 1009b, 1009c. Computing cluster 1009a can include one or more computing devices 1000a, cluster storage arrays 1010a, and cluster routers 1011a connected by local cluster network 1012a. Similarly, computing cluster 1009b can include one or more computing devices 1000b, cluster storage arrays 1010b, and cluster routers 1011b connected by local cluster network 1012b. Likewise, computing cluster 1009c can include one or more computing devices 1000c, cluster storage arrays 1010c, and cluster routers 1011c connected by local cluster network 1012c.

[0182] In some embodiments, the computing clusters 1009a, 1009b, 1009c can be a single computing device residing at a single computing center. In other embodiments, the computing clusters 1009a, 1009b, 1009c can include multiple computing devices in a single computing center, or even multiple computing devices in multiple computing centers located in different geographic locations. For example, Figure 10 Each of the computing clusters 1009a, 1009b, 1009c is depicted as residing at a different physical location.

[0183] In some embodiments, the data and services at the computing clusters 1009a, 1009b, 1009c can be encoded as computer-readable information stored in a non-transitory, tangible computer-readable medium (or computer-readable storage medium) and accessible to other computing devices. In some embodiments, the computing clusters 1009a, 1009b, 1009c can be stored on a single disk drive or other tangible storage medium, or can be implemented across multiple disk drives or other tangible storage media located in one or more different geographic locations.

[0184] In some embodiments, each of the computing clusters 1009a, 1009b, and 1009c can have the same number of computing devices, the same number of cluster storage arrays, and the same number of cluster routers. However, in other embodiments, each computing cluster can have a different number of computing devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of computing devices, cluster storage arrays, and cluster routers in each computing cluster can depend on the one or more computing tasks assigned to each computing cluster.

[0185] For example, in computing cluster 1009a, computing device 1000a can be configured to execute a conditioned, axial self-attention based neural network and / or various computing tasks of the computing device. In one embodiment, the neural network and / or various functions of the computing device can be distributed among one or more of computing devices 1000a, 1000b, 1000c. Computing devices 1000b and 1000c in respective computing clusters 1009b and 1009c can be configured similarly to computing device 1000a in computing cluster 1009a. On the other hand, in some embodiments, computing devices 1000a, 1000b, and 1000c can be configured to perform different functions.

[0186] In some embodiments, the computing tasks and stored data associated with the neural network and / or computing device can be distributed among computing devices 1000a, 1000b, and 1000c based at least in part on processing requirements of the neural network and / or computing device, processing capabilities of computing devices 1000a, 1000b, 1000c, latencies of network links between computing devices in each computing cluster and between computing clusters themselves, and / or other factors that can contribute to cost, speed, fault tolerance, resilience, efficiency, and / or other design objectives of the overall system architecture.

[0187] Cluster storage arrays 1010a, 1010b, 1010c of computing clusters 1009a, 1009b, 1009c can be data storage arrays that include disk array controllers configured to manage read and write access to groups of hard disk drives. The disk array controllers, individually or in conjunction with their respective computing devices, can also be configured to manage backup or redundant copies of data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0188] The functionality of the conditioned, axial self-attention based neural network and / or computing device can be distributed across the computing devices 1000a, 1000b, 1000c of the computing clusters 1009a, 1009b, 1009c in a manner similar to the way various active and / or backup portions of the components can be distributed across the cluster storage arrays 1010a, 1010b, 1010c. For example, some cluster storage arrays can be configured to store a first layer of the neural network and / or a portion of the data of the computing device, while other cluster storage arrays can store a second layer of the neural network and / or other portion(s) of the data of the computing device. Also, for example, some cluster storage arrays can be configured to store data of an encoder of the neural network, while other cluster storage arrays can store data of a decoder of the neural network. Also, some cluster storage arrays can be configured to store backup versions of data stored in other cluster storage arrays.

[0189] The cluster routers 1011a, 1011b, 1011c in the computing clusters 1009a, 1009b, 1009c can include networking equipment configured to provide internal and external communications for the computing clusters. For example, the cluster router 1011a in the computing cluster 1009a can include one or more Internet exchange and routing devices configured to provide (i) local area network communications between the computing devices 1000a and the cluster storage arrays 1010a via the local cluster network 1012a, and (ii) wide area network communications between the computing cluster 1009a and the computing clusters 1009b and 1009c via the wide area network link 1013a to the network 806. The cluster routers 1011b and 1011c can include networking equipment similar to the cluster router 1011a, and the cluster routers 1011b and 1011c can perform similar networking functions for the computing clusters 1009b and 1009b as the cluster router 1011a performs for the computing cluster 1009a.

[0190] In some embodiments, the configuration of the cluster routers 1011a, 1011b, 1011c can be based at least in part on data communication needs of the computing devices and cluster storage arrays, data communication capabilities of the networking equipment in the cluster routers 1011a, 1011b, 1011c, latencies and throughputs of the local cluster networks 1012a, 1012b, 1012c, latencies, throughputs, and costs of the wide area network links 1013a, 1013b, 1013c, and / or other factors that can contribute to cost, speed, fault tolerance, resilience, efficiency, and / or other design criteria for a reasonable system architecture.

[0191] Example operational methods

[0192] Figure 11is a flowchart of a method 1100 according to example embodiments. The method 1100 can be performed by a computing device, such as the computing device 900. The method 1100 can begin at block 1110, where the computing device receives a screenshot of a display provided by a graphical user interface of the computing device.

[0193] At block 1120, the computing device generates, by an image-to-structure transformer of the neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot.

[0194] At block 1130, the computing device predicts, by the neural network and based on the generated representation, a modeled task output associated with the graphical user interface.

[0195] At block 1140, the computing device provides, via the computing device, the predicted modeled task output.

[0196] Some embodiments are directed to predicting, by the neural network, the layout of the virtual objects in the screenshot.

[0197] Some embodiments are directed to receiving, via the computing device, a view hierarchy indicating the layout of the virtual objects in the screenshot. Such embodiments are directed to determining, by the neural network, a content embedding and a position embedding for each of the screenshot and the view hierarchy. Generating the representation by fusing includes generating the representation based on the content embedding and the position embedding. In such embodiments, the position embedding of the screenshot and the position embedding of the view hierarchy can be global embeddings corresponding to the entire screenshot.

[0198] In some embodiments, the modeling of the graphical user interface involves multi-task modeling, and wherein the neural network includes a dual transformer, wherein the image-to-structure transformer predicts a modeled task output for an image structure task, the question-to-answer transformer predicts a modeled task output for a natural language task, and wherein the image-to-structure transformer and the question-to-answer transformer are configured to have cross-tower attention.

[0199] In some embodiments, the modeled task output can be used for one or more of the following tasks: an object detection task, a natural language command grounding task, a widget with text task, a screen summarization task, or a flickability prediction task.

[0200] In some embodiments, the modeled task output can be used for a natural language command grounding task. These embodiments are directed to predicting, by the neural network and based on the representation, a target virtual object in the graphical user interface. Such embodiments are further directed to associating the target virtual object with a natural language command. These embodiments are further directed to providing the natural language command via the graphical user interface.

[0201] In some embodiments, providing the natural language command involves displaying the natural language command at or near the target virtual object.

[0202] In some embodiments, providing the natural language command involves providing the natural language command as a voice command in response to the user’s interaction with the target virtual object.

[0203] In some embodiments, the modeled task output can be used for an object detection task. Such embodiments involve detecting, by the neural network, one or more types of container objects indicative of a layout hierarchy of the screenshot. In such embodiments, the layout hierarchy can include one of a linear layout, a frame layout, or a list.

[0204] In some embodiments, the modeled task output can be used for an object detection task. Such embodiments involve detecting, by the neural network, one or more of a text field, a toggle button, or an image view.

[0205] In some embodiments, the modeled task output can be used for a widget-to-text task. Such embodiments involve predicting, by the neural network and for a screenshot, a natural language description of a function of a predicted virtual object in the graphical user interface.

[0206] In some embodiments, the modeled task output can be used for a tappability prediction task. Such embodiments involve identifying a mismatch between a developer-designed tappability feature of a graphical user interface and a user-perceived tappability feature of the graphical user interface. Such embodiments further involve providing a suggestion to a developer of the graphical user interface to compensate for the identified mismatch.

[0207] In some embodiments, the neural network can include an object detection head, a text head, and an indicator head.

[0208] Some embodiments involve training a neural network to receive an input screenshot displayed by a particular graphical user interface and to predict a modeled task output associated with modeling of the particular graphical user interface.

[0209] In some embodiments, the training is performed at a computing device.

[0210] In some embodiments, predicting the modeled task output involves obtaining, at the computing device, the trained neural network. Such embodiments further involve applying the obtained trained neural network to the prediction of the modeled task output.

[0211] In some embodiments, predicting the modeled task output involves determining, by the computing device, a request to predict the modeled task output. Such embodiments involve sending, from the computing device to a second computing device, the request to predict the modeled task output, the second computing device including a trained version of the neural network. Such embodiments further involve, after sending the request, the computing device receiving, from the second computing device, the predicted modeled task output.

[0212] The present disclosure is not limited to the particular embodiments described in this application, which are intended as illustrations of various aspects. Those skilled in the art will be able to devise numerous modifications without departing from the spirit and scope of the application. In addition to methods and apparatuses listed herein, those skilled in the art will be able to devise other methods and apparatuses within the scope of the present application in light of the foregoing description. Such modifications and variations are intended to fall within the scope of the appended claims.

[0213] The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying drawings. In the drawings, like symbols typically identify corresponding components, unless context dictates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments can be utilized, and other changes can be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the Figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are explicitly contemplated herein.

[0214] With respect to any one or all of the ladder diagrams, scenarios, and flowcharts in the drawings, and as discussed herein, each block and / or communication can represent a processing of information and / or a transfer of information. Alternative embodiments include those in which additional blocks and / or functions are added, those in which fewer blocks and / or functions are utilized, those in which the order of blocks and / or functions are changed, and those in which some blocks and / or functions are implemented differently. For example, one or more of the blocks and / or functions shown and / or discussed can be implemented using hardware, software, firmware, or a combination thereof. Figure 1 In addition, more or fewer blocks and / or functions can be utilized, and the ladder diagrams, scenarios, and flowcharts can be combined or separated in part or in whole.

[0215] A block that represents a processing of information can correspond to a circuit that is configured to perform the specific logical function represented by the block. Alternatively or additionally, a block that represents a processing of information can correspond to a module, segment, or portion of program code that comprises one or more instructions executable by a processor for implementing specific logical functions or actions in the methods or techniques. The program code and / or related data can be stored on any type of computer readable medium, including a storage device, such as a disk or hard drive or other storage device.

[0216] Computer-readable media can also include non-transitory computer-readable media, such as non-transitory computer-readable media that store data for short periods of time like register memory, processor cache and Random Access Memories (RAM). Computer-readable media also can include non-transitory computer-readable media that store programs of instructions or data for long periods of time, like secondary or persistent long term storage, e.g., read only memory (ROM), optical or magnetic disks, e.g., CD-ROMs, etc. The computer-readable media can also be any other volatile or non-volatile storage systems. For example, the computer-readable media can be considered computer-readable storage media, or a tangible storage device.

[0217] Furthermore, a block that represents one or more information transmissions can correspond to a software and / or hardware module within the same physical device or separate physical devices. However, other information transmissions can be between software modules and / or hardware modules that are physically located on different physical devices.

[0218] As described herein, a multi-modal transformer VUT for multi-task modeling of user interfaces is described. The model can be configured to accept three types of data, namely UI screenshot images, view hierarchy, and natural language questions. Experiments based on 5 datasets show that VUT simultaneously implements 5 types of UI tasks and demonstrates the prospect of providing unified modeling for the user interface domain. The VUT model supports multi-modal multi-task learning for several benchmark UI tasks, which can ultimately benefit mobile interaction and user experience.

[0219] Although the example tasks described herein solve the UI modeling problem, they can be generalized to different tasks. For example, the input and output modalities are based on generic data types. The input includes images, view hierarchy, and language. The output head is equipped with the ability to generate view hierarchy, object references, and language responses. Thus, many tasks based on these input and output modalities can potentially be learned with this model. For example, UI layout generation can be handled by a question-answer model to generate a token sequence of linearized view hierarchy.

[0220] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are provided by way of example only, and are not intended to limit the true scope of the disclosure, which is expressly set forth in the claims that follow.

Claims

1. A computer-implemented method comprising: receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device; generating, by an image-to-structure transformer of a neural network, a fused representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; generating, by the image-to-structure transformer, a hidden representation based on the fused representation, wherein the neural network further comprises a question- answer transformer, and wherein the question-answer transformer is configured to utilize the hidden representation based on cross-tower attention; predicting, by the neural network, a modeled task output associated with the graphical user interface based on the generated fused representation; and providing, by the computing device, the predicted modeled task output.

2. The computer-implemented method of claim 1, further comprising: predicting, by the neural network, the layout of virtual objects in the screenshot.

3. The computer-implemented method of claim 1, further comprising: receiving, via the computing device, a view hierarchy indicating the layout of virtual objects in the screenshot; and determining, by the neural network, a content embedding and a position embedding for each of the screenshot and the view hierarchy, wherein generating the fused representation by fusing comprises generating the fused representation based on the content embedding and the position embedding.

4. The computer-implemented method of claim 3, wherein the position embedding of the screenshot and the position embedding of the view hierarchy are global embeddings corresponding to the entire screenshot.

5. The computer-implemented method of claim 1, wherein the modeling of the graphical user interface comprises multi-task modeling, wherein the image-to-structure transformer predicts a modeled task output for an image-to-structure task, the question-answer transformer predicts a modeled task output for a natural language task, and wherein the image-to- structure transformer and the question-answer transformer are configured to have cross-tower attention.

6. The computer-implemented method of claim 1, wherein the modeled task output is for one or more of: an object detection task, a natural language command landing task, a widget with text task, a screen summary task, or a tappability prediction task.

7. The computer-implemented method of claim 1, wherein the modeled task output is for a natural language command landing task, and the method further comprises: predicting, by the neural network, a target virtual object in the graphical user interface based on the fused representation; associating the target virtual object with a natural language command; and providing, via the graphical user interface, the natural language command.

8. The computer-implemented method of claim 7, wherein providing the natural language command comprises displaying the natural language command at or near the target virtual object.

9. The computer-implemented method of claim 7, wherein providing the natural language command comprises providing the natural language command as a voice command in response to a user interaction with the target virtual object.

10. The computer-implemented method of claim 1, wherein the modeled task output is for an object detection task, and the method further comprises: detecting, by the neural network, one or more types of container objects indicating a layout hierarchy of the screenshot. ​ ​ 11. The computer-implemented method of claim 10, wherein the layout level comprises one of a linear layout, a frame layout, or a list.

12. The computer-implemented method of claim 1, wherein the modeling task output is for an object detection task, and the method further comprises: detecting, by the neural network, one or more of a text field, a toggle button, or an image view.

13. The computer-implemented method of claim 1, wherein the modeling task output is for a widget-plus-text task, and the method further comprises: predicting, by the neural network, a natural language description of a function of a predicted virtual object in the graphical user interface for the screenshot.

14. The computer-implemented method of claim 1, wherein the modeling task output is for a tappability prediction task, and the method further comprises: identifying, for a graphical user interface, a mismatch between a developer-designed tappability feature and a user-perceived tappability feature; and providing, to a developer of the graphical user interface, a suggestion to compensate for the identified mismatch.

15. The computer-implemented method of claim 1, wherein the neural network comprises an object detection head, a text head, and a pointer head.

16. The computer-implemented method of claim 1, further comprising: training a neural network to receive an input screenshot displayed by a particular graphical user interface and to predict a modeling task output associated with modeling of the particular graphical user interface.

17. The computer-implemented method of claim 16, wherein the training is performed at the computing device.

18. The computer-implemented method of claim 1, wherein predicting a modeling task output further comprises: obtaining, at the computing device, a trained neural network; and applying the obtained trained neural network to prediction of the modeling task output.

19. The computer-implemented method of claim 1, wherein predicting a modeling task output comprises: determining, by the computing device, a request for prediction of a modeling task output; sending, from the computing device to a second computing device, the request for prediction of the modeling task output, the second computing device comprising a trained version of the neural network; and after sending the request, the computing device receiving, from the second computing device, the predicted modeling task output.

20. A computing device comprising: one or more processors; and data storage, wherein the data storage has stored thereon computer-executable instructions that, when executed by the one or more processors, cause the computing device to perform functions comprising: receiving a displayed screenshot provided by a graphical user interface of the computing device; generating, by an image-structure transformer of a neural network, a fused representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; generating, by the image-structure transformer, a hidden representation based on the fused representation, wherein the neural network further comprises a question-answer transformer, and wherein the question-answer transformer is configured to utilize the hidden representation based on cross-tower attention; ​ predict, by a neural network, a modeled task output associated with the graphical user interface based on the generated fused representation; and provide the predicted modeled task output.

21. The computing device of claim 20, wherein, The instructions cause the computing device to perform functions including the method of any of claims 2-19.

22. One or more non-transitory computer-readable media having stored thereon computer- readable instructions that, when executed by one or more processors of a computing device, cause the computing device to perform functions including: receive a screenshot of a display provided by a graphical user interface of the computing device; generate, by an image-structure transformer of a neural network, a fused representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot; generate, by the image-structure transformer, a hidden representation based on the fused representation, wherein the neural network further includes a question-answer transformer, and wherein the question-answer transformer is configured to utilize the hidden representation based on cross-tower attention; predict, by a neural network, a modeled task output associated with the graphical user interface based on the generated fused representation; and provide the predicted modeled task output.

23. The non-transitory computer-readable medium of claim 22, wherein, The instructions cause the computing device to perform functions including the method of any of claims 2-19. The instructions cause the computing device to perform functions including the method of any of claims 2-19.

Images