Adjust the user interface for touch screen and mouse / keyboard environments

Abstract

Aspects of the subject technology relate to dynamically adjusting a UI based on a current modality. Layout features of a UI may be determined based on an input modality of a computing device. The arrangement of UI elements may be determined based on the layout features and the corresponding importance scores of the UI elements. UI elements arranged based on the arrangement may be provided for display by a computing device.

Description

Technical Field

[0001] The present invention relates to adapting user interfaces for touch screen and mouse / keyboard environments. Background Art

[0002] The user interface (UI) of an application running on a computing device may be provided with a plurality of user interface (UI) elements (i.e., icons, buttons, menu options, etc.). An application UI designer may specify rules for the UI, such as the number of UIs to be included in the UI of the application and the arrangement of UI elements relative to each other in the UI. However, computing devices may be laptop computers, desktop computers, tablet computers, and mobile devices. The resolution, dimensions, and size of a computing device display may vary based on the operating environment (e.g., laptop computer, desktop computer, tablet computer, mobile device). Providing rules for UIs for corresponding operating environments would be cumbersome for application UI designers and also present a technical problem of consuming a large amount of data storage of the computing device because more rules need to be stored in the data storage. Summary of the invention

[0003] Aspects of the subject technology relate to generation and management of user interfaces (UIs). Layout features of a UI may be determined based on an input modality of a computing device. The placement of UI elements may be determined based on the layout features and the corresponding importance scores of the UI elements. UI elements arranged based on the placement may be provided for display by a computing device.

[0004] Aspects of the subject technology also relate to a computing system. The system may include one or more processors and a non-transitory computer-readable medium including instructions stored therein, which, when processed by the one or more processors, causes the one or more processors to perform operations. These operations include determining layout features of a user interface (UI) based on an input modality of a computing device. These operations also include determining corresponding click target sizes of UI elements based on layout features and corresponding importance scores of UI elements. The operations further include determining the arrangement of selected UI elements based on layout features, corresponding importance scores of UI elements, and corresponding click target sizes of UI elements. These operations further include providing arranged UI elements for display.

[0005] Aspects of the subject technology also relate to a non-transitory computer-readable medium including instructions stored therein, which, when executed by a processor, cause the machine to perform operations including determining layout features of a user interface (UI) based on an input modality of a computing device. The operations may also include determining corresponding importance scores based on a designation of an application. The operations further include selecting a set of UI elements from a UI model based on the layout features and the corresponding importance scores of the UI elements. The operations also include determining an arrangement of the selected UI elements based on the layout features and the corresponding importance scores of the UI elements. The operations also include providing a UI.

[0006] It will be appreciated that other configurations of the subject technology will become apparent to those skilled in the art from the following detailed description, in which various configurations of the subject technology are illustrated and described in a pictorial manner. It will be appreciated that the subject technology is capable of other and different configurations, and that several of its details are capable of being modified in various other respects, all without departing from the scope of the subject technology. Therefore, the drawings and detailed description should be considered illustrative rather than restrictive in nature. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The accompanying drawings, which are included to provide a further understanding and are incorporated in and constitute a part of this specification, illustrate the disclosed aspects and together with the description serve to explain the principles of the disclosed aspects.

[0008] Figure 1 An example client-server environment is illustrated in accordance with example aspects of the subject technology.

[0009] Figure 2 Illustrated is an example data flow for a system for optimizing a UI based on a current modality according to an example aspect of the subject technology.

[0010] Figure 3 A flow diagram illustrating an example process for adjusting a UI based on a current modality according to an example aspect of the subject technology.

[0011] Figure 4 Illustrated is an example user interface adjusted for different input modalities in accordance with example aspects of the subject technology.

[0012] Figure 5 An example electronic system is conceptually illustrated by which some embodiments of the subject technology may be implemented. DETAILED DESCRIPTION

[0013] The detailed description set forth below is intended to be a description of various configurations of the subject technology, and is not intended to represent the only configuration in which the subject technology can be practiced. The accompanying drawings are incorporated herein and form a part of the detailed description. In order to provide a thorough understanding of the subject technology, the detailed description may include specific details. However, the subject technology is not limited to the specific details set forth herein, and can be practiced without these specific details. In some instances, structures and components are shown in block diagram form to avoid obscuring the subject technology concept.

[0014] The user interface (UI) of an application running on a computing device may be provided with a plurality of user interface (UI) elements (i.e., icons, buttons, menu options, etc.). An application UI designer may specify rules for the UI, such as the number of UIs to be included in the UI of the application and the arrangement of UI elements relative to each other in the UI. However, computing devices may be laptop computers, desktop computers, tablet computers, and mobile devices. The resolution, dimensions, and size of a computing device display may vary based on the operating environment (e.g., laptop computer, desktop computer, tablet computer, mobile device). Providing rules for UIs for corresponding operating environments would be cumbersome for application UI designers and also present a technical problem of consuming a large amount of data storage of the computing device because more rules need to be stored in the data storage.

[0015] In addition, a computing device such as a convertible computing device may be able to switch between a laptop computer mode and a tablet computer mode. Although the resolution, size, and screen size of the display of the computing device may not change between the laptop computer mode and the tablet computer mode, the input method may be different between the laptop computer mode and the tablet computer mode. For example, when the computing device is operated in the laptop computer mode, the operating system of the computing device may receive mouse-based input (e.g., input via a mouse and / or keyboard). In the laptop computer mode, a plurality of UI elements with relatively small click target sizes may be provided in the user interface (UI). The click target size of the UI element may be relatively small because the touch size of the cursor or pointer for selecting the UI element that contacts the UI element is relatively small.

[0016] The computing device may also operate in a tablet mode, in which the operating system of the computing device may receive touch screen-based input (e.g., input via a finger or stylus). The touch size of a finger or stylus may be larger than the touch size of a cursor or pointer manipulated via a mouse-based input. Therefore, it may be desirable to have UI elements with larger click target sizes than the UI in laptop mode to accommodate selection of UI elements using a finger or stylus.

[0017] To accommodate different input modalities of a computing device, a UI designer may need to provide different rules for the UI for the corresponding input modality of the computing device. However, unlike a cursor or pointer in a mouse-based input modality, the size of a finger or stylus may vary from user to user in a touchscreen-based input modality. The touch size of a finger or stylus may vary from user to user based on the various sizes of the user's finger or stylus. Therefore, providing a UI element with a different click target size (e.g., a larger click target size than a laptop mode) in a tablet mode UI may still be small for some users.

[0018] Providing UI elements with a tap target size that does not match the touch size may cause unintended selection of the UI element and may present technical issues of unnecessary data processing that negatively impacts data processing and transmission.

[0019] In order to solve the technical problem, the subject technology provides the following technical solution, namely, providing a user interface (UI) model associated with an application running on a computing device to optimize the UI for a mouse-based input modality or a touch screen-based input modality. The UI model provides a library of user interface (UI) elements (e.g., menu options, icons, buttons) and rules for placing UI elements in the UI of the application. In particular, the layout features of the UI associated with the current input mode are determined based on the UI model associated with the application. The UI elements to be included in the UI for display can be selected from the UI model based on the determined layout features and the importance measure of the UI elements. Based on the rules provided by the UI model, the selected UI elements are arranged in the UI.

[0020] Figure 1 An example environment 100 for automatically adjusting a UI based on a current modality according to the subject technology is illustrated. Environment 100 includes computing devices 102, 104, and 106, and servers 110 and 114. In some aspects, environment 100 may have more than Figure 1 More or fewer computing devices (e.g., 102-106) and / or servers (e.g., 110 and 114) than those shown.

[0021] Each of computing devices 102, 104, and 106 may represent various forms of processing devices having a processor, memory, and communication capabilities. Computing devices 102, 104, and 106 may communicate with each other, with servers 110 and 114, and / or with Figure 1As non-limiting examples, the processing device may include a desktop computer, a laptop computer, a handheld computer, a personal digital assistant (PDA), a cellular phone, a network appliance, a camera, a smart phone, an Enhanced General Packet Radio Service (EGPRS) mobile phone, a media player, a navigation device, an email device, a game console, or any combination of these or other processing devices.

[0022] Each of computing devices 102, 104, and 106 may operate in a laptop computer mode that may allow mouse-based input (e.g., input via a mouse and / or keyboard) or in a tablet computer mode that may allow touchscreen-based input (e.g., input via a finger or stylus). An operating system on each of computing devices 102, 104, and 106 may switch between the laptop computer mode and the tablet computer mode based on changes detected in the hardware configuration of the respective computing device 102, 104, and 106.

[0023] The change in hardware configuration may include detecting a change in the angle between the screen portion and the keyboard portion of the laptop computer, which changes from an angle within a predetermined threshold angle (i.e., 180 degrees) to an angle above the predetermined threshold angle. For example, when the angle between the screen portion and the keyboard portion is within a predetermined threshold angle, the computing device may be in laptop computer mode. However, when the angle between the screen portion and the keyboard portion is above a predetermined threshold angle, the computing device may be in tablet computer mode. In some aspects, a change in hardware configuration may be detected when the screen portion and the keyboard portion of the computing device are detached from each other. When the screen portion and the keyboard portion are detached, the computing device may be in tablet computer mode. On the other hand, when the screen portion and the keyboard portion are attached, the computing device is in laptop computer mode.

[0024] Each of computing devices 102, 104, and 106 may be associated with a user. The user may be the owner of a computing device (e.g., computing device 102, 104, or 106), or a person logged into an account on the computing device. In some aspects, a user may be associated with multiple computing devices (e.g., any combination of computing devices 102, 104, and 106). For example, computing device 102 may be a user's personal laptop, while computing device 104 may be a user's business laptop.

[0025] Each of computing devices 102, 104, and 106 may have an application installed thereon. When an application is launched, the application may provide a user interface (UI) for display via a display of the computing device. Each application installed on a computing device may include at least one user interface (UI) model. The UI of the application is generated based on the UI model and usage data associated with the application.

[0026] The usage data may include usage logs of the application and user interactions with UI elements of the UI associated with the application. For example, the usage logs of the application may be usage logs collected and aggregated over a predetermined period of time (e.g., during a session of the application, a day, a week, etc.). The usage logs may include the time of day that the application was launched on the computing device, the number of hours the application was active, and / or the frequency of use of the application.

[0027] User interaction with UI elements may include a click pattern received at a corresponding UI element within the UI. Clicking may include a selection mechanism for selecting UI elements in the UI. The selection mechanism may be a click (i.e., single click, double click) or input gesture (e.g., drag, pan (span) or slide) received at a UI element via a pointer device (e.g., mouse, stylus, finger, etc.). The click pattern may include the type of pointer device used to click the corresponding UI element, the touch size of the click received at the corresponding UI element, the type of selection mechanism received at the corresponding UI element, the number of clicks received on the corresponding UI element, the number of failed clicks, the number of useless clicks, the layout across applications, and the attention span of the content.

[0028] The type of pointer device used can include a mouse, keyboard, joystick, finger, or stylus for clicking UI elements in the UI. One type of pointer device can be a mouse, keyboard, or joystick in a mouse-based input modality, and a finger or stylus in a touchscreen-based input modality.

[0029] The touch size of the received click can be the input area in contact with the UI element when the click is received at the UI element. The touch size can be the area of ​​the fingertip contacting the touch screen when receiving input on the UI element while the computing device is operating in a touch screen-based input mode. In some other aspects, the touch size of the click can include the intensity of the click. The intensity of the click can be the touch intensity received via the touch screen when the computing device receives input on the UI element from a finger or stylus while operating in a touch screen-based input mode.

[0030] The type of selection mechanism can include click, drag, pan and slide. The UI element can receive a sliding input gesture. In some other aspects, any combination of the type of selection mechanism can be received at the UI element. For example, the UI element can be panned before receiving a click.

[0031] The number of clicks received on the UI element may include successful clicks received on the UI element. For example, a click may be counted as a successful click when a click is received on the UI element, and the click received on the UI element triggers an action associated with the UI element. The frequency of use of the UI element may also be determined based on the number of clicks received on the UI element. In some other aspects, the order in which the UI elements are accessed may be determined. The access order may indicate that the UI elements may be selected continuously.

[0032] When a click is considered to be a failure, it can be counted as the number of failed clicks. A failed click can be two clicks received near a UI element within a predetermined time period (e.g., within 2 seconds), where the second of the two clicks triggers an action associated with the UI element. The number of clicks received near a UI element before being identified as a failed click may not be limited to two clicks, and the number of clicks may be two or more clicks. A failed click may indicate that the click target size of the UI element is too small for the current input modality.

[0033] A click is determined to be a useless click when a successful click is followed by an action that triggers a return to a previous screen indicating that the UI element erroneously received the click within a predetermined time period. The number of useless clicks may not be counted in the number of clicks received on the UI element.

[0034] The attention span may be determined based on scrolling or zooming a particular area on the layout and / or content of the application.In some aspects, the attention span may be determined based on eye tracking data received from an eye tracking device (ie, an eye tracking camera).

[0035] User interactions with corresponding UI elements are recorded, and usage data of the application can be updated based on the recorded user interactions. The UI of the application can be updated based on the updated usage data. The UI of the application can be generated and provided for display. When the usage data is updated based on the user interaction, the UI model of the application can be updated, and the UI of the application can be regenerated based on the updated UI model. In some aspects, the number of UI elements included in the regenerated UI can be reduced when compared to the UI before regeneration. In some other aspects, the position of the UI element may be different in the UI before and after regeneration.

[0036] Usage data may be generated and recorded for the corresponding application. For example, usage data of users associated with one or more computing devices 102, 104, and 106 may be generated and recorded for the application. In some aspects, usage data may be generated for multiple different users of an application on a computing device (i.e., one of computing devices 102, 104, 106) or on multiple computing devices (i.e., computing devices 102, 104, and 106). The usage data may be stored in the corresponding computing devices 102, 104, and 106. In some aspects, the usage data may be transmitted to one or more servers (e.g., servers 110 and 114) via a network 108. Insofar as the system discussed herein collects usage data associated with a user or may utilize usage data, the user is provided with an opportunity to control whether a program or feature collects usage data (e.g., a user's preference) and to control the UI associated with the application based on the collected usage data. The user may also be provided with the option of turning on or off certain features or functions provided by the system. In some aspects, the user may choose to disable the features and functions provided by the system discussed herein (e.g., controlling the UI associated with the application based on the collected usage data). In addition, users can specify that certain data should be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the identity of a user can be processed so that no personally identifiable information can be determined for that user, or the user's geographic location can be summarized (such as to a city, zip code, or state level) in the case of location information so that the user's specific location cannot be determined. Thus, users have control over whether and how the disclosed system collects, stores, and uses user information.

[0037] The network 108 can be a computer network such as, for example, a local area network (LAN), a wide area network (WAN), the Internet, a cellular network, or a combination thereof, connecting any number of mobile clients, fixed clients, and servers. In addition, the network 108 can include, but is not limited to, any one or more of the following network topologies, including bus networks, star networks, ring networks, mesh networks, star bus networks, tree or hierarchical networks, etc. In some aspects, the communication between each client (e.g., computing devices 102, 104, and 106) and a server (e.g., server 110) can occur via a virtual private network (VPN), a secure shell (SSH) tunnel, a secure socket layer (SSL) communication, or other secure network connection. In some aspects, the network 108 can further include a corporate network (e.g., an intranet) and one or more wireless access points.

[0038] Each of servers 110 and 114 can represent a single computing device, such as a computer server including a processor and a memory. The processor can execute computer instructions stored in the memory. Servers 110 and 114 can be geographically co-located and / or servers 110 and 114 can be located in different locations. In some aspects, servers 110 and 114 can collectively represent computer servers. In some aspects, servers 110 and 114 can each be implemented using multiple distributed computing devices. Servers 110 and 114 are configured to communicate with client devices (e.g., computing devices 102, 104, and 106) via network 108.

[0039] Servers 110 and 114 may include data stores 112 and 116, respectively, for storing usage data of users or computing devices 102, 104, and 106. In some aspects, UI models of applications implemented on computing devices 102, 104, and 106 may be stored on servers 110 and 114. For example, the UI models may include UI elements associated with the application, as well as default importance metrics for the UI elements. In one or more embodiments, computing device 102, computing device 104, computing device 106, server 110, or server 114 may be or may include the following: Figure 5 All or part of the electronic system components discussed.

[0040] In some aspects, the subject technology may include or otherwise utilize one or more machine learning models to adjust UI elements based on usage data. The machine learning model 120 may be trained to receive input data 122 (i.e., usage data) and, in response, provide output data 124 (i.e., size, position, dimensions of the UI element). Thus, the machine learning model 120 performs inference.

[0041] The machine learning model can be one or more recurrent neural networks. Some of the nodes of the recurrent neural network may form a cycle. Recurrent neural networks can be particularly useful for processing input data that is sequential in nature. In particular, in some cases, a recurrent neural network can pass or retain information from a previous portion of an input data sequence to a subsequent portion of an input data sequence by using loops or directed recurrent node connections.

[0042] The sequential input data may include time series data (e.g., patterns of clicks received at different times). The recurrent neural network may use the data to compare time to adjust the UI elements to be included in the UI. As another example, the sequential input data may include usage logs of the application and user interactions with UI elements.

[0043] Example recurrent neural networks may include long short-term (LSTM) recurrent neural networks; gated recurrent units; bidirectional recurrent neural networks; continuous-time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recurrent neural networks; Hopfield networks; fully recurrent networks; sequence-to-sequence configurations.

[0044] In some aspects, the machine learning model can be trained or otherwise configured to receive input data and provide output data in response. The input data can include different types, forms, or variations of the input data. The input data can include usage data including, for example, usage logs of the application and user interactions with UI elements of a UI associated with the application.

[0045] In response to the reception of input data, the machine learning model can provide output data. The output data may include different types, forms or variants of the output data. The output data may include data for adjusting UI elements. In some embodiments, the output data may include various types of classification data (e.g., binary classification, multi-class classification, single label, multi-label, discrete classification, regression classification, probability classification, etc.), or may include various types of regression data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multivariate regression, etc.). In other cases, the output data may include any one of clustering data, anomaly detection data, recommendation data or other forms of output data discussed above. In some embodiments, the output data may affect downstream processing or make decisions. As an example, in some embodiments, the output data may be interpreted and / or acted upon by a rule-based regulator.

[0046] Therefore, the subject technology provides systems and methods that include or otherwise utilize one or more machine learning models to adjust UI elements based on usage data. Any of the above-mentioned different types or forms of input data can be combined with any of the above-mentioned different types or forms of machine learning models to provide any of the above-mentioned different types or forms of output data.

[0047] The systems and methods of the subject technology may be implemented by or otherwise executed on one or more computing devices (i.e., computing devices 102, 104, and 106). Thus, in some implementations, the machine learning model may be stored at and / or implemented locally by one or more computing devices (i.e., computing devices 102, 104, and 106). Output data obtained by local implementation of the machine learning model at the computing device may be used to improve the performance of the computing device (e.g., an application implemented by the computing device).

[0048] In other embodiments, the machine learning model may be stored at and / or implemented by a server (e.g., servers 110 and 114). Output data obtained by implementing the machine learning model on the server may be used to improve other server tasks, or may be used by a computing device (i.e., computing devices 102, 104, and 106) to improve services performed by or for the computing device. For example, the output data may improve other downstream processing performed by the server for the computing device (i.e., computing devices 102, 104, and 106). It can be said that the server performs machine learning as a service.

[0049] Figure 2 An example system for optimizing a UI based on a current modality according to an example aspect of the subject technology is illustrated. System 200 includes a display device 210, an input interface 220, and a computing device 230. Display device 210 can display information provided by an operating system. Display device 210 can be a touch screen. In some aspects, display device 210 can be a convertible display, wherein when the computing device is in tablet mode, display device 210 can be a touch screen, and when the computing device is in laptop mode, display device 210 can be a conventional monitor.

[0050] The input interface 220 may provide input signals from an input device (e.g., a mouse, a stylus, a keyboard). For example, when the computing device is in laptop mode, the computing device receives input signals from the input device via the input interface 220. When the computing device is in tablet mode, the display device 210 may be used as part of the input interface 220 and may receive input signals therefrom via a touch screen.

[0051] The computing device 230 may include a processor 240 and a memory 250. The processor 240 may retrieve instructions and data to perform the processing of the computing device 230. The number of processors is not limited to Figure 2 The memory 250 may include a user interface (UI) module 252 and an application 254 .

[0052] The UI module 252 may include instructions and data to generate a UI to be provided to the display device 210 based on a user interface (UI) model 254A associated with the application 254. The application 254 may be an application installed on the computing device 230. In some aspects, the application 254 may include system software, such as an operating system for a computing device. The application 254 may include a UI model 254A, which may include a list of UI elements (i.e., menu options, icons, buttons), a UI layout, and rules for placing UI elements within the UI layout of the application 254, such as layout features of the UI elements and click target size. The rules for placing UI elements may be specified by the designer of the UI, and may indicate the UI elements to be included in the UI, regardless of the current input modality or usage data. The rules for placing UI elements may indicate the minimum click target size of the UI elements specified by the designer of the UI.

[0053] Layout characteristics may include the amount of space available for placing UI elements within the UI and the available options for placing UI elements within the UI. The layout characteristics may be determined based on the current input modality and the configuration of the display device 210 of the computing device 230 (ie, screen size, resolution).

[0054] When the launch of application 254 is detected, UI module 252 can determine the layout features of the UI associated with the application based on the current input modality of the computing device and the configuration of the display device 210 of the computing device. The layout features can be determined for the visible area of ​​the UI provided on the display device 210. For example, the layout features can be the amount of space available for placing UI elements within the UI. In some aspects, the layout features can indicate the number of UI elements included in the UI. In some aspects, UI module 252 can receive a signal to launch the application from the operating system 230 of the computing device 230. UI module 252 can identify the current input modality of the computing device. The signal received from the operating system 230 may include data related to the current input modality of the computing device. UI module 252 can retrieve UI information of application 254 from the UI model based on the current input modality of the identified computing device. UI module 252 can determine the layout features based on the identified current input modality and the retrieved UI information.

[0055] The UI module 252 may select a plurality of UI elements from the UI model 236 based on the determined layout features and the corresponding importance scores of the UI elements. The UI module 252 may determine usage data for UI elements (i.e., menu options, icons, buttons, etc.) associated with the application 254. In some aspects, the UI module 252 may also include a usage log of the application, importance metrics specified by the application UI designer, and / or usage patterns from other applications. The usage log of the application may include the time of day when the application is launched on the computing device, the number of hours the application is active, and / or the frequency of use of the application.

[0056] UI module 252 may refer to UI model 254A, which may include, for example, importance metrics for UI elements specified by a designer of application 254. The importance metrics may indicate the importance of a UI element. An application UI designer may specify UI elements to be included in a UI, regardless of the usage data of the UI elements. A user may specify UI elements to be included in a UI, for example, by setting preferences. In some aspects, user preferences may override the importance metrics specified by an application UI designer.

[0057] UI module 252 can collect usage data from other applications that have similar appearance, content, and interactivity to the subject application and are installed on the computing device. The collected usage data from other applications can be included in the usage data of the subject application so that more data can be used to better optimize the UI. The collected usage data from other applications can be used to update UI model 254A.

[0058] UI module 252 can determine the importance score of the corresponding UI element based on the determined usage data. In some aspects, UI module 252 can determine the importance score based on the determined usage data and the importance metric of the specified UI element. UI elements with a number of clicks that meet a threshold and / or included in a specific area of ​​the layout of the application can have an importance score higher than the threshold score. For example, UI elements can be prioritized based on the importance score of the UI element. UI module 252 can record the usage style of the UI elements of the application, and update the usage style of the UI elements based on the recording results. Update the importance score of the UI element based on the updated usage style.

[0059] UI module 252 can identify UI elements with importance scores that meet the threshold score, and select multiple UI elements from the identified UI elements based on the determined click target size of the UI element and the determined layout features of the UI element. UI module 252 can arrange the selected UI elements according to the UI layout stored in the UI model. The UI layout can indicate the position of the UI element relative to other UI elements on the UI of the application. For example, the UI layout can indicate that the UI element can be provided adjacent to another UI element. UI module 252 can provide the selected UI elements arranged based on the UI layout to the UI for display.

[0060] The UI module 252 can identify the UI layout for the UI from the UI model. The UI designer of the application can specify the layout to be used. The UI module 252 can select a set of UI elements to be included in the UI. The UI module 252 can refer to the importance scores of the UI elements and select UI elements that meet the threshold scores. The UI model may include which UI elements are to be included in the UI layout. A portion of the UI element can be identified by the UI model, and the rest of the UI element can be selected based on the importance scores of the UI elements. The UI module 252 can arrange the selected UI elements based on the UI layout and provide the UI for display.

[0061] UI module 252 can select multiple UI elements with importance scores that meet the threshold score. UI module 252 can arrange the selected UI elements based on the corresponding importance scores. The UI element with the highest importance score can be arranged at the top center of the UI. Relative to the UI element with the highest importance score, the remaining selected UI elements can be arranged in the UI. UI module 252 can arrange the selected UI elements based on the importance scores of the UI elements in the UI, and provide the UI for display.

[0062] UI module 252 may determine a click target size for a corresponding selected UI element based on a current input modality of the computing device. The click target size for a corresponding selected UI element may be determined based on the current input modality of the computing device and the usage data. For example, a UI element having an importance score that meets a threshold score based on the usage data may be considered to have a higher importance metric to the user, and a click target size that meets the threshold click target size may be assigned to the UI element. UI module 252 may provide the arranged UI elements to the UI for display.

[0063] Figure 3A flowchart is shown that illustrates an example process 300A for automatically adjusting a UI based on a current input modality according to an example aspect of the technical subject matter. For illustrative purposes, various blocks of the example process 300A are described herein with reference to components and / or processes described herein. One or more blocks of the process 300A may be, for example, Figure 1 The example process 300A may be implemented by one or more components or processors of the server 110 and / or server 114. In some embodiments, one or more blocks may be implemented separately from other blocks and implemented by one or more different processors or controllers. In addition, for the purpose of illustration, the blocks of the example process 300A are described as occurring in series or linearly. However, multiple blocks of the example process 300A may occur in parallel. In addition, the blocks of the example process 300A do not need to be executed in the order shown and / or one or more of the blocks of the example process 300A do not need to be executed.

[0064] At block 310A, layout characteristics of the UI are determined based on the UI model and the input modality of the computing device. The UI module may determine the layout characteristics of the UI in response to detecting a change in the input modality based on a change in the device configuration of the computing device when the application is active. In some aspects, the UI module may determine the layout characteristics of the UI in response to detecting the launch of the application.

[0065] The UI module can determine the current input modality of the computing device. In some aspects, the current input modality can be determined in response to detecting a change in the input modality or detecting the launch of an application. In some other aspects, the UI module can check the current input modality of the computing device.

[0066] For example, the UI module may determine that the current input modality of the computing device is a touch screen-based input associated with a tablet computer mode. The UI module detects a computing device in tablet computer mode that uses a touch screen-based input and determines the current input modality of the computing device as a touch screen-based input. The UI module may reference a UI model associated with a launched application and determine the layout features of the UI of the launched application associated with tablet computer mode. The layout features may be the amount of space available for placing UI elements in the UI. The layout features may be determined for the visible area of ​​the UI on the display device. In some aspects, the layout features of the UI of the launched application may be different based on a laptop computer mode or a tablet computer mode. The layout features of the UI in laptop computer mode may be larger than the layout features of the UI in tablet computer mode.

[0067] At box 320A, multiple UI elements to be displayed on the UI are selected from the UI model based on the layout features and the corresponding importance scores of the UI elements. The UI module can determine the corresponding importance scores of the UI elements based on the usage data of the UI elements. The space required to display the corresponding UI elements on the UI (e.g., click target size) is determined based on the importance scores of the UI elements. For example, when a UI element is selected more frequently than other UI elements in the UI or is included in an area of ​​the UI that the user appears to be interested in based on the attention span, a higher importance score indicating a high importance metric can be given to the UI element. Therefore, the UI element can be given more space than other UI elements in the UI, causing the UI element to appear larger than other UI elements in the UI. Based on the determined importance scores and the rules in the UI model, the UI module can select UI elements to be included in the UI provided for display.

[0068] At box 330A, the selected UI elements are arranged according to the UI model. For example, the UI module can determine the arrangement of the selected UI elements based on the corresponding importance scores and click target sizes. The UI module can arrange the remaining UI elements on the list of elements to be included in the UI with a click target size that is higher than the minimum threshold click target size. In some aspects, UI elements with importance scores lower than the threshold score can be hidden (i.e., not provided to the UI for display). When the input modality of the computing device changes from touch-screen-based input to mouse-based input, a layout of the UI in which hidden UI elements can be provided to the UI for display can be performed.

[0069] At block 340A, the selected UI element in the determined arrangement is provided to the UI for display.

[0070] Figure 3 A flowchart is shown that illustrates an example process 300B for automatically adjusting a UI based on a current input modality according to an example aspect of the subject technology. For purposes of illustration, various blocks of the example process 300B are described herein with reference to components and / or processes described herein. One or more blocks of the process 300B may be, for example, Figure 1 The example process 300B may be implemented by one or more components or processors of the server 110 and / or server 114. In some embodiments, one or more boxes may be implemented separately from other boxes and may be implemented by one or more different processors or controllers. In addition, for the purpose of illustration, the boxes of the example process 300B are described as occurring in series or linearly. However, multiple boxes of the example process 300B may occur in parallel. In addition, the boxes of the example process 300B do not need to be executed in the order shown and / or one or more of the boxes of the example process 300B do not need to be executed.

[0071] At block 310B, layout characteristics of the UI are determined based on the UI model and the input modality of the computing device.The UI module may determine that the current input modality of the computing device is a touch screen-based input associated with tablet mode.

[0072] At block 320B, a UI layout is identified based on the UI model. The UI module may reference the UI model and identify the UI layout to use. The UI layout may be identified based on the current input modality or based on what the UI designer of the application specifies in the UI model.

[0073] At block 330B, a set of UI elements to be included in the UI may be selected based on the UI layout. Some UI elements may be designated as included in a UI having an identified UI layout as indicated by the UI model. In some cases, UI elements may be selected based on their respective importance scores. UI elements having importance scores that meet a threshold score may be selected for inclusion in the UI.

[0074] At block 340B, the set of UI elements may be arranged within the UI based on the identified UI layout. The positions of the UI elements relative to each other may be specified by the UI layout. In some aspects, the click target size of the UI elements may be considered when arranging the set of UI elements within the UI. The relative positions of the UI elements may be specified by the UI layout, and depending on the click target size of the UI elements, the positions of the UI elements may be set within the UI.

[0075] At block 350B, UI elements arranged within the UI based on the UI layout may be provided for display.

[0076] Figure 3 A flowchart is shown that illustrates an example process 300C for automatically adjusting a UI based on a current input modality according to an example aspect of the technical subject matter. For illustrative purposes, various blocks of the example process 300C are described herein with reference to components and / or processes described herein. One or more blocks of the process 300C may be, for example, Figure 1 The example process 300C may be implemented by one or more components or processors of the server 110 and / or server 114. In some embodiments, one or more boxes may be implemented separately from other boxes and may be implemented by one or more different processors or controllers. In addition, for the purpose of illustration, the boxes of the example process 300CB are described as occurring in series or linearly. However, multiple boxes of the example process 300C may occur in parallel. In addition, the boxes of the example process 300C do not need to be executed in the order shown and / or one or more of the boxes of the example process 300B do not need to be executed.

[0077] At block 310C, layout characteristics of the UI are determined based on the UI model and the input modality of the computing device.The UI module may determine that the current input modality of the computing device is a touch screen-based input associated with tablet mode.

[0078] At block 320C, a plurality of UI elements may be selected based on respective importance scores.The UI elements may be selected based on respective importance scores of the UI elements satisfying a threshold score.

[0079] At block 330C, the set of UI elements may be arranged in the UI based on the UI layout stored in the UI model. The position of the UI element may be determined based on the importance score of the UI element and the UI layout. In some aspects, the click target size of the UI element may also be considered when arranging the set of UI elements in the UI. At block 340C, the UI elements arranged in the UI may be provided for display.

[0080] In some aspects, the UI may include: Figure 4 0046] A scrollable container view (eg, a list of messages in an email inbox) is illustrated in FIG. Figure 4 Illustrated is an example user interface adjusted for different input modalities in accordance with example aspects of the subject technology.

[0081] UI 400 is associated with an application (i.e., an email application) and is generated based on a mouse-based input modality. For example, the UI module detects a computing device using a mouse-based input in laptop mode and determines the current input modality of the computing device as mouse-based input. The UI module determines the layout features of the UI (i.e., UI 400) associated with the mouse-based input modality based on the UI model associated with the application. The UI module determines the number of UI elements to be included in UI 400 based on the layout features and the corresponding importance scores of the UI elements, and selects the UI elements to be included in UI 400.

[0082] As shown in UI 400, UI 400 may include UI elements 402, 404, 406, and 408. UI element 402 may be a group of menu options, including, for example, five email folders (i.e., Inbox, Sent, Draft, Junk, Outbox). UI element 404 may be a group of menu options for a contact list (i.e., Alice, Bob, Charlie, Dave, Ellie, Frank, George, Henry, Isaac). UI element 406 may be a group of menu options for an inbox message list (i.e., 11 messages), which may be a scroll view. UI element 408 may be a search bar for searching for items (e.g., emails, contacts) in an email application.

[0083] The UI module can refer to the UI model of the application and determine that UI element 406 (i.e., the message list in the inbox) should always be included in the UI. The application UI designer of the email application can specify in the UI model that UI element 406 should be included, regardless of the usage data. The UI module can determine that UI elements 402 and 404 can be included in the UI based on the remaining available space according to the layout characteristics. The UI module can determine that the UI elements included in UI elements 402 and 404 can be selected more frequently than other UI elements in the application, and based on the usage data of the UI elements, UI elements 402 and 404 can have an importance score that meets the threshold score. The UI module can select UI element 408 to be included in the UI. The user can specify that UI element 408 is included in the UI. Based on the preferences set by the user, the UI module can determine to include UI element 408 in the UI.

[0084] The UI module can arrange the selected UI elements 402, 404, 406 and 408 based on the UI model of the application and the usage data. The UI module can arrange the UI elements 402, 404, 406 and 408 based on the UI layout specified by the UI designer of the application in the UI model. The UI module can determine the click target size of the UI element based on the corresponding importance score determined based on the usage data. In some aspects, the UI module can determine the appearance size of the UI element based on the click target size of the UI element. For example, the text items in the message list in the UI element 406 are provided to be displayed in a single-line view.

[0085] UI 410 is associated with an application (i.e., an email application) and is generated for a touch screen-based input modality. The UI module detects a computing device using a touch screen-based input in tablet mode and determines the current input modality of the computing device as touch screen-based input. The UI module determines layout features of a UI (i.e., UI 410) associated with the touch screen-based input modality based on a UI model associated with the application. The UI module determines the number of UI elements to be included in UI 410 based on the layout features and the corresponding importance scores of the UI elements, and selects the UI elements to be included in UI 410.

[0086] As shown in UI 410, UI 410 may include UI elements 412A, 412B, 416, and 418. In some aspects, UI elements 412A and 412B may correspond to UI elements 402 and 404 in UI 400. The UI module may determine that an email folder (i.e., UI element 402), a contact list (i.e., UI element 404), an email message (i.e., UI element 406), and a search bar (i.e., UI element 408) should be included in UI 410 (i.e., a touch screen based input modality) based on a UI model of an email application.

[0087] The UI module may arrange UI elements 412A, 412B, 416, and 418 based on the UI model and usage data of the application. The UI module may determine that the click target size of the UI elements in UI 410 for the touch screen-based input modality may be larger than the click target size of the UI elements in UI 400 for the mouse-based input modality. Therefore, the UI module may select a predetermined number of UI elements (e.g., one, three) from the group of UI elements and provide the predetermined number of UI elements for displaying and hiding other UI elements.

[0088] In some aspects, the UI module may determine based on usage data that email folders are accessed more frequently than contact lists, and "Inbox" may be most frequently accessed in email folders. Therefore, the UI module may compress and hide some folders in the email folders in UI element 402 and the contact list in UI element 404 under UI element 412A. The user may access the hidden UI element by selecting UI element 412A. When the user clicks on UI element 412A, a list including the hidden UI element may be displayed in an expanded view, and the user may access the hidden UI element. The UI module may provide the most frequently accessed "Inbox" in the email folder list as UI element 412B in UI410 based on usage data.

[0089] The UI module can determine that the list of email messages is the most frequently accessed UI element in the email application based on the usage data. The UI module can set the click target size of the UI element 416 in UI 410 to be larger than the click target size set for the UI element 406 in UI 400. The UI module can determine that the email message (i.e., element 406 in UI 400, element 416 in UI 410) provided to be displayed as a single-line view in UI 400 based on user preferences can be provided to be displayed as a multi-line view, and the multi-line view includes a part of the content of the email message. In some aspects, the number of items included in the UI element 406 in UI 400 and the number of items included in the UI element 416 in UI 410 can be different. Based on the click target size set for the items in UI element 416 being larger than the click target set for the items in UI element 406, UI element 416 can include fewer items (i.e., messages) than UI element 406. Correspondingly, each item in UI element 416 may take up more space than each item in UI element 406, and therefore, there may not be enough space to accommodate all items included in UI element 406 into UI element 416. In some aspects, the apparent size of the UI element may be related to the click target size. Since the click target size of UI element 416 is set to be larger than UI element 406, the apparent size of UI element 416 can be set to be larger than UI element 406.

[0090] The UI module can determine that UI element 418 (i.e., a search bar) corresponding to UI element 408 in UI 400 should be included in UI 410. For example, the usage indicates that the search bar is used less frequently when the computing device is operating in a touch screen-based input modality than when the computing device is operating in a mouse-based input modality. The UI module can reduce the click target size and apparent size of UI element 418 in UI 410.

[0091] The UI module may arrange the selected group of UI elements 412 and 416 based on the UI model of the application. For example, the UI module may arrange the group of UI elements 412 and 416 based on the UI layout specified by the designer of the application in the UI model. In some aspects, the UI module may determine the click target size of the UI elements in the group of UI elements based on the corresponding importance scores determined based on the usage data. In some other aspects, the UI module may determine the apparent size of the UI element.

[0092] The application can provide a UI model for any of the input modalities of the computing device (i.e., mouse-based input modality, touchscreen-based input modality). For example, in the case where a UI module only for laptop mode is available for the UI model, the arrangement of UI elements to be provided to the UI for display can be converted to an arrangement of UI elements suitable for tablet mode based on the continuous recording of the usage pattern of UI elements by the UI module.

[0093] The UI module can associate the usage pattern and arrangement of the UI element with a corresponding user account. In this case, the usage pattern and arrangement of the UI element can be executed on the associated user account, regardless of the computing device being used.

[0094] The UI module can update the arrangement of the UI elements in response to updating the usage pattern of the UI elements. For example, the click target size can be adjusted according to the updated usage pattern within a session of the application.

[0095] Figure 5 An example electronic system 500 is conceptually illustrated with which some embodiments of the subject technology may be implemented. The electronic system 500 may be a computer, a phone, a personal digital assistant (PDA), or any other kind of electronic device. Such an electronic system includes various types of computer-readable media and interfaces for various other types of computer-readable media. The electronic system 500 includes a bus 508, a processing unit 512, a system memory 504, a read-only memory (ROM) 510, a permanent storage device 502, an input device interface 514, an output device interface 506, and a network interface 516.

[0096] Bus 508 collectively represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of electronic system 500. For example, bus 508 communicatively connects processing unit 512 with ROM 510, system memory 504, and permanent storage device 502.

[0097] Processing unit 512 retrieves instructions to execute and data to process from these various memory units in order to perform the processes of the subject disclosure. In various implementations, the processing unit may be a single processor or a multi-core processor.

[0098] ROM 510 stores static data and instructions required by the processing unit 512 and other modules of the electronic system. On the other hand, permanent storage device 502 is a read-write memory device. This device is a non-volatile memory unit that stores instructions and data even when the electronic system 500 is turned off. Some embodiments of the subject disclosure use a mass storage device (e.g., a magnetic or optical disk or flash memory) as permanent storage device 502.

[0099] Other embodiments use a removable storage device (e.g., a floppy disk, a flash drive) as the permanent storage device 502. Similar to the permanent storage device 502, the system memory 504 is a read-write memory device. However, unlike the storage device 502, the system memory 504 is a volatile read-write memory, such as a random access memory. The system memory 504 stores some instructions and data required by the processor at runtime. In some embodiments, the processes disclosed in the subject matter are stored in the system memory 504, the permanent storage device 502, or the ROM 510. For example, the various memory units include instructions for displaying graphical elements and identifiers associated with the corresponding application, receiving predetermined user input to display a visual representation of a shortcut associated with the corresponding application, and displaying a visual representation of the shortcut. The processing unit 512 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the processes of some embodiments.

[0100] The bus 508 is also connected to input and output device interfaces 514 and 506. The input device interface 514 enables a user to transmit information to the electronic system and select commands. Input devices used with the input device interface 514 include, for example, an alphanumeric keyboard and a pointing device (also referred to as a "cursor control device"). The output device interface 506 enables, for example, the display of images generated by the electronic system 500. Output devices used with the output device interface 506 include, for example, a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices, such as a touch screen that acts as both an input device and an output device.

[0101] Finally, if Figure 5 As shown in , bus 508 also couples electronic system 500 to a network (not shown) via network interface 616. In this manner, the computer can be part of a network of computers (e.g., a LAN, WAN, or intranet or a network of networks, such as the Internet). Any or all components of electronic system 500 may be used in conjunction with the subject disclosure.

[0102] Many of the above features and applications are implemented as software processes specified as sets of instructions recorded on computer-readable storage media (also referred to as computer-readable media). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of processors, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, magnetic media, optical media, electronic media, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0103] In this specification, the term "software" is meant to include, for example, firmware residing in a read-only memory or other form of electronic storage or an application that can be stored in magnetic storage, optical, solid-state, etc., which can be read into a memory for processing by a processor. Moreover, in some embodiments, multiple software aspects of the subject disclosure can be implemented as sub-parts of a larger program while retaining different software aspects of the subject disclosure. In some embodiments, multiple software aspects can also be implemented as separate programs. Finally, any combination of separate programs that implement the software aspects described herein together are within the scope of the subject disclosure. In some embodiments, the software program, when installed to operate on one or more electronic systems, defines one or more specific machine implementations of the operations of running and executing the software program.

[0104] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0105] These functions can be implemented in digital electronic circuits, computer software, firmware or hardware. One or more computer program products can be used to implement these techniques. Programmable processors and computers can be included in mobile devices or packaged as mobile devices. The process and logic flow can be performed by one or more programmable processors and by one or more programmable logic circuits. General and special computing devices and storage devices can be interconnected through communication networks.

[0106] Some embodiments include electronic components, such as a microprocessor, a memory that stores and stores computer program instructions in a machine-readable or computer-readable medium (or otherwise referred to as a computer-readable storage medium, a machine-readable medium, or a machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), digital versatile disk-read only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic or solid-state hard drives, read-only and recordable CDs, ultra-density CDs, any other optical or magnetic media, and floppy disks. The computer readable medium may store a computer program executable by at least one processing unit and including a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files including high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

[0107] Although the above discussion refers primarily to microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits (e.g., application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs)). In some embodiments, such integrated circuits execute instructions stored on the circuits themselves.

[0108] As used in this specification and any claims of this application, the terms "computer", "server", "processor" and "memory" refer to electronic or other technical devices. These terms do not include people or groups of people. For the purpose of explanation, the terms display or displaying refer to displaying on an electronic device. As used in this specification and any claims of this application, the terms "computer-readable medium" and "computer-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient signals.

[0109] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT or LCD monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0110] Aspects of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or a middleware component (e.g., an application server), or a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0111] A computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device (e.g., the result of a user interaction) can be received from the client device at the server.

[0112] It should be understood that any particular order or hierarchy of steps in the disclosed process is an illustration of an example method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged, or all of the steps shown can be performed. Certain steps can be performed simultaneously. For example, in some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in the embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged as multiple software products.

[0113] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but should be given a full scope consistent with the language claims, wherein, unless otherwise specified, reference to an element in the singular is not intended to mean "one and only one", but rather "one or more". Unless otherwise specified, the term "some" refers to one or more. Masculine pronouns (e.g., his) include feminine and neuter genders (e.g., her and its), and vice versa. Titles and subtitles, if any, are for convenience only and do not limit subject disclosure.

[0114] As used herein, the phrase "at least one of" preceding a series of items, where the terms "and" or "or" separate any of the items, modifies the entire list, rather than each member of the list (i.e., each item). The phrase "at least one of" does not require selection of at least one of each item listed; rather, the phrase allows for a meaning that includes at least one of any of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. For example, each of the phrases "at least one of A, B, and C" or "at least one of A, B, or C" refers to only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0115] Phrases such as an aspect, this aspect, another aspect, some aspects, one or more aspects, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, an example, this example, another example, some examples, one or more examples, configuration, this configuration, another configuration, some configurations, one or more configurations, subject technology, the disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not mean that the disclosure involving such phrases is essential to the subject technology, or that such disclosure applies to all configurations of the subject technology. Disclosures involving such phrases may apply to all configurations, or one or more configurations. Disclosures involving such phrases may provide one or more examples. Phrases such as aspects or certain aspects may refer to one or more aspects, and vice versa, and this applies similarly to the other aforementioned phrases.

[0116] To the extent that the systems discussed herein collect usage data associated with a user or can utilize usage data, the user is provided with an opportunity to control whether a program or feature collects usage data (e.g., a user's preference) and to control the UI associated with the application based on the collected usage data. The user may also be provided with the option of turning on or off certain features or functions provided by the system. In some aspects, the user may choose to disable the features and functions provided by the systems discussed herein (e.g., controlling the UI associated with the application based on the collected usage data). In addition, the user may specify that certain data should be processed in one or more ways before being stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that no personally identifiable information can be determined for the user, or the user's geographic location in the case of obtaining location information may summarize the user's geographic location (such as, summarized to the city, zip code, or state level) so that the user's specific location cannot be determined. Therefore, the user can control whether and how the disclosed system collects, stores, and uses user information.

[0117] All structural and functional equivalents of the elements of the various aspects described in this disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the subject technology. In addition, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly referenced in the above description. In addition, with respect to the use of the terms "including", "having", etc. in the description or claims, such terms are intended to be interpreted as inclusive in a manner similar to the term "including" such as "including" when used as a transitional word in the claims.

Claims

1. A method for generating and managing a user interface (UI), comprising: identifying, using one or more processors, an input modality as one or more of a touch screen-based input modality, a mouse-based input modality, and a stylus-based input modality; determining, with the one or more processors, layout characteristics of a user interface of an application on the computing device based on the input modality of the computing device; obtaining usage data including a click pattern from user interactions with the user interface, wherein the click pattern includes corresponding touch sizes of clicks; determining respective importance scores for user interface elements from the input modality and the usage data for the application; determining a corresponding position and size of each of the user interface elements as an arrangement of the user interface elements based on the corresponding importance scores; determining, using the one or more processors, a corresponding click target size for the user interface element; selecting, using the one or more processors, a set of user interface elements from the user interface elements based on the click target sizes of the user interface elements, the layout characteristics, and the respective importance scores of the user interface elements; and The set of user interface elements is provided for display according to the arrangement.

2. The method according to claim 1 further includes determining the corresponding importance score based on one or more of the following: rules provided by the user, rules provided by the designer of the application, usage data of the application, and usage data of similar applications.

3. The method according to claim 1, further comprising: identifying the usage data associated with the user interface element; one or more of rearranging the set of user interface elements based on the usage data and adjusting respective click target sizes of the set of user interface elements based on the usage data to obtain rearranged user interface elements; as well as The rearranged user interface elements are provided to the user interface for display.

4. The method according to claim 1, wherein: Determining the respective positions and the sizes of the user interface elements as the arrangement of the user interface elements includes applying the usage data to a machine learning model.

5. The method according to claim 1, wherein: The usage data further includes a time log of the user's interactions with the application.

6. The method according to claim 1, wherein: The user interaction with the application further includes the user's attention span.

7. The method according to claim 1, wherein: The click pattern further includes one or more of a type of clicking device, a type of selection mechanism, a number of clicks, a number of failed clicks, and a number of useless clicks.

8. A computing system comprising: one or more processors; and A non-transitory machine-readable medium comprising instructions stored therein, the instructions, when executed by the one or more processors, causing the one or more processors to perform operations comprising: identifying an input modality as one or more of a touch screen-based input modality, a mouse-based input modality, and a stylus-based input modality; determining, based on the input modality of the computing device, layout characteristics of a user interface of an application on the computing device; obtaining usage data including a click pattern from user interactions with the user interface, wherein the click pattern includes corresponding touch sizes of clicks; determining respective importance scores for user interface elements from the input modality and the usage data for the application; determining a corresponding position and size of each of the user interface elements as an arrangement of the user interface elements based on the layout features and the corresponding importance scores; determining a corresponding click target size for the user interface element; selecting a set of user interface elements from the user interface elements based on the click target sizes of the user interface elements, the layout characteristics, and the respective importance scores of the user interface elements; and The set of user interface elements is provided for display according to the arrangement.

9. The computing system of claim 8, wherein: The respective importance scores are further based on one or more of: rules provided by the user, rules provided by a designer of the application, usage data of the application, and usage data of similar applications.

10. The computing system of claim 8, wherein: The operations further include: identifying the usage data associated with the user interface element; one or more of rearranging the set of user interface elements based on the usage data and adjusting respective click target sizes of the set of user interface elements based on the usage data to obtain rearranged user interface elements; and The rearranged user interface elements are provided to the user interface for display.

11. The computing system of claim 8, wherein: Determining the respective positions and the sizes of the user interface elements as the arrangement of the user interface elements includes applying the usage data to a machine learning model to determine the arrangement of the user interface elements.

12. The computing system of claim 8, wherein: The usage data further includes a time log of the user's interactions with the application.

13. The computing system of claim 8, wherein: The user interaction with the application further includes attention span.

14. The computing system of claim 8, wherein: The click pattern further includes one or more of a type of clicking device, a type of selection mechanism, a number of clicks, a number of failed clicks, and a number of useless clicks.

15. A non-transitory computer-readable medium comprising instructions stored therein, the instructions, when executed by a processor, causing a computer to perform operations comprising: identifying an input modality as one or more of a touch screen-based input modality, a mouse-based input modality, and a stylus-based input modality; determining, based on the input modality of the computing device, layout characteristics of a user interface of an application on the computing device; obtaining usage data including a click pattern from user interactions with the user interface, wherein the click pattern includes corresponding touch sizes of clicks; determining respective importance scores for user interface elements from the input modality and the usage data for the application; determining a corresponding position and size of each of the user interface elements as an arrangement of the user interface elements based on the corresponding importance scores; determining a corresponding click target size for the user interface element; selecting a set of user interface elements from the user interface elements based on the click target sizes of the user interface elements, the layout characteristics, and the respective importance scores of the user interface elements; and The set of user interface elements is provided for display according to the arrangement.

16. The non-transitory computer readable medium of claim 15, wherein: The respective importance scores are further based on one or more of: rules provided by the user, rules provided by a designer of the application, usage data of the application, and usage data of similar applications.

17. The non-transitory computer readable medium of claim 15, wherein: The operations further include: identifying the usage data associated with the user interface element; one or more of rearranging the set of user interface elements based on the usage data and adjusting respective click target sizes of the set of user interface elements based on the usage data to obtain rearranged user interface elements; and The rearranged user interface elements are provided to the user interface for display.

18. The non-transitory computer readable medium of claim 15, wherein: Determining the respective positions and sizes of the user interface elements as the arrangement of the user interface elements includes applying the usage data to a machine learning model.

19. The non-transitory computer readable medium of claim 15, wherein: The usage data further includes a time log of the user's interactions with the application.

20. The non-transitory computer readable medium of claim 15, wherein: The user interaction with the application further includes attention span.

Images