Generating user interfaces across devices

Abstract

A computing system retrieves information associated with user interface elements included in a first user interface and information associated with a device configured to display a second user interface. The computing system determines constraints of the second user interface. The computing system converts the information associated with the user interface elements into a masked image, and applies an image-to-image diffusion model to the masked image to generate a target image with the constraints of the second user interface. The computing system generates, based on the information associated with the user interface elements, a prompt, and applies a language model to the prompt to generate information indicative of a target user interface layout. The computing system combines the target image and the information indicative of the target user interface layout to generate a data structure. The data structure includes position information for the user interface elements within the second user interface.

Description

GENERATING USER INTERFACES ACROSS DEVICESBACKGROUND

[0001] Some wearable computing devices (e.g., smart watches, activity trackers, headsup display devices, etc.) display graphical content for web and mobile apps. Often, the user interface displayed by these wearable computing devices is directly scaled to the same user interface displayed by larger computing devices, such as smartphones or computers. As such, constraints around readability and clickability of the user interface may be ignored, which may result in a suboptimal visual and interactive experience for users.SUMMARY

[0002] In general, techniques of this disclosure are directed to adapting user interfaces between various computing devices having varying display sizes and properties using generative modeling. Rather than requiring a developer or user interface designer to create a custom user interface layout for every different display resolution, shape, and size, techniques of this disclosure may automatically generate a user interface that is customized for the display on which the user interface will be displayed. A computing system may use information for a first graphical user interface, such as a user interface for an application executed by a first computing device (e.g., a laptop, a desktop, a tablet, a smartphone, a computing device embedded within a vehicle, a computing device embedded within a gaming console, etc.) and information associated with a second, different computing device (e.g., a wearable computing device, such as a smartwatch, smart glasses, etc., a smartphone, a vehicle, etc.). Using the information about the first user interface, the computing system may identify particular user interface elements, such as interactive elements (buttons, text fields, etc.) and non-interactive elements (e.g., background images, text, etc.) and, using one or more machine learning models and the information about the second device, dynamically generate a second user interface having a customized layout for the second device.

[0003] In this respect, various aspects of the techniques described in this disclosure may improve sideways-compatibility between dense mobile or web applications and devices with different screen dimensions and constraints. The use of generative artificial intelligence may further assist in accurately determining optimal sizing and positioningfor user interface elements, and the use of an image-to-image diffusion model may help to maintain overall aesthetics. As such, user interface design and experience may be better maintained when performing user interface translation between devices.

[0004] In some examples, a method includes retrieving, by a computing system, information associated with a plurality of user interface elements included in a first graphical user interface, retrieving, by the computing system, information associated with a target device configured to display a second graphical user interface, and determining, by the computing system, and based on the information associated with the target device, constraints of the second graphical user interface. The method further includes converting, by the computing system, a first subset of the information associated with the plurality of user interface elements into a masked image, applying, by the computing system, a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface, and generating, by the computing system, and based on a second subset of the information associated with the plurality of user interface elements, a prompt. The method further includes applying, by the computing system, a second machine learning model to the prompt to generate information indicative of a target user interface layout, and combining, by the computing system, the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0005] In some examples, a computing system comprises one or more processors, and one or more storage devices that store instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to retrieve information associated with a plurality of user interface elements included in a first graphical user interface, and retrieve information associated with a target device configured to display a second graphical user interface. The one or more processors are further configured to determine, based on the information associated with the target device, constraints of the second graphical user interface, and convert a first subset of the information associated with the plurality of user interface elements into a masked image. The one or more processors are further configured to apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface, and generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt. The one or moreprocessors are further configured to apply a second machine learning model to the prompt to generate information indicative of a target user interface layout, and combine the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0006] In some examples, a non-transitory computer-readable storage medium is encoded with instructions that, when executed by one or more processors, cause the one or more processors to retrieve information associated with a plurality of user interface elements included in a first graphical user interface, and retrieve information associated with a target device configured to display a second graphical user interface. The one or more processors are further configured to determine, based on the information associated with the target device, constraints of the second graphical user interface, and convert a first subset of the information associated with the plurality of user interface elements into a masked image. The one or more processors are further configured to apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface, and generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt. The one or more processors are further configured to apply a second machine learning model to the prompt to generate information indicative of a target user interface layout, and combine the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0007] The details of one or more examples of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1 is a conceptual diagram illustrating an example computing system configured to facilitate user interface translation from a primary device to a target device, in accordance with one or more techniques of the present disclosure.

[0009] FIG. 2 is a block diagram further illustrating an example computing system configured to facilitate user interface translation from a primary device to a target deviceusing generative modeling, in accordance with one or more techniques of the present disclosure.

[0010] FIG. 3 is a block diagram illustrating an example machine learning module configured to generate data structures including position information for a plurality of user interface elements within a second graphical user interface, in accordance with one or more techniques of the present disclosure.

[0011] FIG. 4 is a block diagram illustrating an example process for generating data structures including position information for a plurality of user interface elements within a second graphical user interface, in accordance with one or more techniques of the present disclosure.

[0012] FIG. 5 is a flow chart illustrating an example operation of a computing system that facilitates user interface translation from a primary device to a target device, in accordance with one or more techniques of the present disclosure.DETAILED DESCRIPTION

[0013] FIG. 1 is a conceptual diagram illustrating an example computing system configured to facilitate user interface translation from a primary device to a target device, in accordance with one or more techniques of the present disclosure. Specifically, computing system 100 may facilitate user interface translation from primary device 102, which may be a device configured to support mobile or web applications, to target device 104, which may be a wearable device also configured to support mobile or web applications on a screen with different constraints (e.g., screen dimensions, resolution, etc.).

[0014] As shown in the example of FIG. 1, primary device 102 includes one or more user interface (UI) components 108 for displaying generated user interface (GUI) 112. As further shown in the example of FIG. 1, GUI 112 may include one or more user interface elements, such as elements 114A-114C (which may be referred to collectively herein as “elements 114”), which may be non-clickable user interface elements (e.g., labels, noninteractive icons, headers and footers, static images, read-only text boxes, etc.) or clickable user interface elements (e.g., buttons, dropdown menus, sliders, toggles, tabs, links, interactive icons, etc.). Target device 104 also includes one or more UI components 110, which may operate similarly if not substantially similar to UI components 108. As shown, UI components 110 may generate GUI 116, in which GUI 116 may also include one or more of user interface elements 114. However, as shown in the example of FIG. 1,GUI 112 and GUI 116 may have different screen dimensions or constraints, and thus may require different user interface layouts for a visually pleasing experience for a user.

[0015] The techniques of this disclosure may provide a solution for translating user interface information from primary device 102 to target device 104 in a way such that that the design, aesthetics, and relevant information of elements 114 are maintained across devices. Specifically, as described herein, computing system 100 may retrieve information associated with user interface elements 114 included in GUI 112, retrieve constraints for GUI 116, and apply generative modeling to generate a data structure including position information for user interface elements 114 within GUI 116. As an example, as shown in FIG. 1, GUI 116 may display interface elements 114 with a different layout and / or sizing than GUI 112. Furthermore, GUI 116 may include scrollbar 120, in which a user may interact with GUI 112 to “scroll” through GUI 120 to view or interact with elements 114. As shown in the example of FIG. 1, the constraints of GUI 116 may result in one or more of elements 114, such as element 114C, to not be fully visible within a single frame (e.g., GUI 112 may include elements 114A, 114B, and 114C in a single frame while GUI 116 may include elements 114 A, 114B, and only a portion of element 114C in a single frame). As such, GUI 116 may be configured with scrollbar 120 such that a user can scroll GUI 116 to fully view or interact with element 114C. Thus, rather than directly scaling user interface elements 114 from primary device 102 to target device 104, computing system 100 may resize and / or reposition elements 114 such that elements 114 are still readable and easy to interact with on screens with different constraints (e.g., target device 104, which may be a wearable device that includes a screen with dimensions smaller than that of primary device 102, which may be a computer).

[0016] Other examples of primary device 102 and target device 104 that implement techniques of this disclosure may include additional components not shown in FIG. 1. Examples of primary device 102 and target device 104 may include, but are not limited to, portable, mobile, or other devices, such as mobile phones (including smartphones), wearable computing devices (e.g., smart watches, smart glasses, digital bracelets, etc.) laptop computers, desktop computers, tablet computers, smart television platforms, server computers, mainframes, infotainment systems (e.g., vehicle head units), etc.

[0017] As described above, primary device 102 includes UI components 108 and target device 104 includes UI components 110. UI components 108 and UI components 110 may be configured to function as input devices and / or output devices for primary device 102 and target device 104, respectively. UI components 108 and UI components 110 maybe implemented using various technologies. For instance, UI components 108 and UI components 110 may be configured to receive input from a user through tactile, audio, and / or video feedback. Examples of input devices include a presence-sensitive display, a presence-sensitive or touch-sensitive input device, a voice responsive system, video camera, microphone or any other type of device for detecting input from a user. In some examples, a presence-sensitive display includes a touch-sensitive or presence-sensitive input screen, such as a resistive touchscreen, a surface acoustic wave touchscreen, a capacitive touchscreen, a projective capacitive touchscreen, a pressure sensitive screen, an acoustic pulse recognition touch screen, or another presence-sensitive technology. That is, UI components 108 and UI components 110 may include a presence-sensitive device that may receive tactile input from a user. UI components 108 and UI components 110 may receive indications of the tactile input by detecting one or more gestures from a user (e.g., when a user touches or points to one or more locations of UI components 108 and UI components 110 with a finger or a stylus pen). In some examples, UI components 108 and UI components 110 may detect two-dimensional and / or three-dimensional gestures as input from a user.

[0018] UI components 108 and UI components 110 may additionally or alternatively be configured to function as output devices by providing output to a user using tactile, audio, or video stimuli. Examples of output devices include a sound card, a video graphics adapter card, or any of one or more display devices, such as a liquid crystal display (LCD), dot matrix display, light emitting diode (LED) display, microLED, miniLED, organic light-emitting diode (OLED) display, e-ink, or similar monochrome or color display capable of outputting visible information to a user. Additional examples of an output device include a speaker, a haptic device, or other device that can generate intelligible output to a user. For instance, UI components 108 and UI components 110 may present output to a user as a graphical user interface that may be associated with functionality provided by primary device 102 and target device 104, respectively. In this way, UI components 108 and UI components 110 may present various user interfaces of applications executing at or accessible by primary device 102 and target device 104, respectively (e.g., an electronic message application, an Internet browser application, etc.). A user may interact with a respective user interface of an application to cause primary device 102 and / or target device 104 to perform operations relating to a function.

[0019] As further shown in the example of FIG. 1, primary device 102 and target device 104 may communicate with computing system 100. In some examples, primary device102 and target device 104 may communicate with computing system 100 via a network, such as any public or private communications network, e.g., cellular, Wi-Fi, and / or other types of networks for transmitting data between computing systems, servers, and computing devices. The network may include one or more network hubs, network switches, network routers, or any other network equipment that is operatively intercoupled thereby providing for the exchange of information between computing system 100, primary device 102, and target device 104. The network may include a wide-area network such as the Internet, a local-area network (LAN), a personal area network (PAN) (e.g., Bluetooth®), an enterprise network, a wireless network, a cellular network, a telephony network, a Metropolitan area network (e.g., WIFI, WAN, WiMAX, etc.), one or more other types of networks, or a combination of two or more different types of networks (e.g., a combination of a cellular network and the Internet). Computing system 100 may transmit and receive data across the network using any suitable communication techniques. For example, primary device 102, target device 104, and / or computing system 100 may include wireless communication devices capable of transmitting and / or receiving communication signals using a network, such as a cellular radio, a 3G radio, a 4G radio, a 5G radio, a Bluetooth® radio (or any other PAN radio), an NFC radio, or a WIFI radio (or any other WLAN radio). Additionally, or alternatively, primary device 102, target device 104, and / or computing system 100 may include wired communication devices capable of transmitting and / or receiving communication signals via a direct link over a wired communication medium (e.g., a universal serial bus (“USB”) cable). The links coupling primary device 102, target device 104, and / or computing system 100 to the network may be Ethernet or other types of network connections and such connections may be wireless and / or wired connections.

[0020] Computing system 100 may represent any suitable remote computing system, such as one or more desktop computers, laptop computers, mainframes, servers, cloud computing systems, etc. capable of sending and receiving information both to and from primary device 102 and target device 104, e.g., via a network. In some examples, computing system 100 may represent a cloud computing system that provides one or more services via a network. That is, in some examples, computing system 100 may be a distributed computing system. One or more computing devices, such as primary device 102 and target device 104, may access the services provided by the cloud by communicating with computing system 100. While described herein as being performed at least in part by computing system 100, any or all techniques of the present disclosuremay be performed by one or more other devices, such as primary device 102 and / or target device 104. That is, in some examples, primary device 102 and / or target device 104 may be operable to perform one or more techniques of the present disclosure.

[0021] In the example of FIG. 1, computing system 100 includes user interface (UI) module 106. While UI module 106 is shown in the example of FIG. 1 as being located within computing system 100, in other examples, as described above, all or part of the functionality provided by UI module 106 (and other modules in other figures described herein) may be delegated to primary device 102 and / or target device 104. UI Module 106 may perform operations described herein using hardware, software, firmware, or a mixture thereof residing in and / or executing at computing system 100. Computing system 100 may execute UI module 106 with one processor or with multiple processors. In some examples, computing system 100 may execute UI module 106 as a virtual machine executing on underlying hardware. UI module 106 may execute as one or more services of an operating system or computing platform or may execute as one or more executable programs at an application layer of a computing platform.

[0022] UI module 106, as shown in the example of FIG. 1, may be operable by computing system 100 to perform one or more functions for user interface translation between devices, such as primary device 102 and target device 104. UI module 106 may receive input or data (e.g., from primary device 102 and target device 104) and send indications of such input or data to other components associated with computing system 100. UI module 106 may also receive data from components associated with computing system 100.

[0023] In general, a user operating primary device 102 and / or target device 104 may be provided with an opportunity to provide input to control whether programs or features of computing system 100 can collect and make use of user and / or device information (e.g., information about user’s current location, personal information, information displayed via GUI 112 and / or GUI 116, etc.), or to dictate whether and / or how computing system 100 may receive content that may be relevant to a user operating primary device 102 and / or target device 104. In addition, certain data may be treated in one or more ways before it is stored or used by computing system 100 so that personally identifiable information is censored and / or removed. For example, information pertaining to a user’s identity and displayed via a GUI may be retrieved once the user provides explicit consent for computing system 100 to do so. In some examples, computing system 100 may perform data caching, in which some or all of the information retrieved from primary device 102and / or target device 104 may be cleared or erased after user interface translation is performed. In some examples, computing system 100 may perform data encryption, in which some or all of the information retrieved from primary device 102 and / or target device 104 may be encrypted prior to being stored by computing system 100. In some examples, the information retrieved by computing system 100 may be treated such that no personally identifiable information can be determined about the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, a user operating primary device 102 and / or target device 104 may have control over how information is collected from them and their devices and used by computing system 100. For example, responsive to a user operating primary device 102 providing explicit consent for computing system 100 to retrieve and store their data, computing system 100 may then retrieve data from primary device 102 and store the retrieved data in a memory. In some examples, prior to retrieving data from target device 104 and / or translating user interface information from primary device 102 to target device 104, a user may be prompted to provide explicit consent for computing system 100 to do so. In some examples, a user may be prompted to confirm their association with a device (e.g., via a PIN, password, etc.) prior to computing system 100 retrieving data from primary device 102 and / or target device 104 and performing user interface translation from primary device 102 to target device 104. For example, a user operating primary device 102 may indicate that they would like to have their current user interface translated onto a screen of target device 104. The user operating primary device 102 may be prompted to provide a PIN, password, etc. that is associated with target device 104 to confirm association between primary device 102 and target device 104. The user operating primary device 102 may also be prompted to provide explicit consent for computing system 100 to retrieve data from primary device 102 and / or target device 104. Responsive to receiving explicit consent from the user, computing system 100 may implement the techniques described herein for user interface translation between primary device 102 and target device 104.

[0024] As described herein, UI module 106 may receive, from primary device 102 and responsive to receiving explicit user consent, information associated with elements 114 included in GUI 112. UI module 106 may also receive, responsive to receiving explicit user consent, information associated with target device 104 configured to display GUI 116. For example, UI module 106 may receive, from target device 104, target device 104model information. UI module 106 may determine, based on the target device model information, one or more of the dimensions of GUI 116 and display resolution of GUI 116. As described herein, UI module 106 may further be configured to apply generative modeling for the translation of elements 114 from primary device 102 to target device 104. Specifically, UI module 106 may categorize one or more pixels included the information associated with elements 114 as corresponding to either clickable user interface elements (e.g., buttons, dropdown menus, sliders, toggles, tabs, links, interactive icons, etc.) or non-clickable user interface elements (e.g., labels, noninteractive icons, headers and footers, static images, read-only text boxes, etc.). UI module 106 may generate a first subset of the information associated with elements 114, in which the first subset of the information may include pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements. UI module 106 may then apply a binary mask to the first subset of the information to generate a masked image, in which the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

[0025] UI module 106 may apply a machine learning model (e.g., image-to-image diffusion model) to the masked image to generate a target image with the constraints of GUI 116. UI module 106 may further generate, based on a second subset of the information associated with elements 114 (e.g., information corresponding to clickable user interface elements such as buttons, dropdown menus, sliders, toggles, tabs, links, interactive icons, etc.), a prompt, in which the prompt may include the information associated with target device 104, the second subset of the information associated with elements 114, and one or more identified adjustments to data included in the second subset of the information associated with elements 114. For example, UI module 106 may receive information from primary device 102 that includes the constraints of element 114C. UI module 106 may also receive information from target device 104 that includes the constraints of GUI 116. Based on the target device 104 information, UI module 106 may identify one or more needed adjustments to the constraints of element 114C, such that the readability, design, and interactivity of element 114C is maintained when displayed on GUI 112.

[0026] UI module 106 may generate a prompt (e.g., a natural language prompt) that includes the above information, and apply a second machine learning model to the prompt to generate information indicative of a target user interface layout. UI module 106 may then combine the target image and the information indicative of the target user interfacelayout to generate a data structure, in which the data structure includes position information for elements 114 within GUI 116. Computing system 100 may send the data structure including the position information for elements 114 within GUI 116 to target device 104.

[0027] In some examples, UI module 106 may further receive, from target device 104, user input indicating one or more desired changes to the position information. UI module 106 may then generate a prompt including the one or more desired changes and provide the prompt as input to the second machine learning model to generate information indicative of an updated target user interface layout. In this way, users may be able to edit the layout of elements 114 within GUI 116 in accordance with their needs or preferences.

[0028] As described above, primary device 102 and / or target device 104 may be operable to perform one or more techniques of the present disclosure. As such, in some examples, the techniques described with respect to computing system 100 may be implemented locally by primary device 102 and / or target device 104. For example, in some examples, primary device 102 may be configured to retrieve information associated with target device 104 and determine, based on the information associated with target device 104, the constraints of GUI 116. In these examples, primary device 102 may convert the first subset of the information associated with elements 114 into a masked image and apply a first machine learning model to the masked image to generate a target image with the constraints of GUI 116. Primary device 102 may generate, based on a second subset of the information associated with elements 114, a prompt, and apply a second machine learning model to the prompt to generate information indicative of a target user interface layout. Primary device 102 may further combine the target image and the information indicative of the target user interface layout to generate a data structure that includes position information for elements 114 within GUI 116.

[0029] Although the examples provided herein describe user interface translation from primary device 102 to target device 104 in which target device 104 includes a screen with dimensions smaller than that of primary device 102, in some examples, computing system 100 may facilitate the reverse, e.g., computing system 100 may retrieve information associated with user interface elements 114 included in GUI 116, retrieve constraints for GUI 112, and apply generative modeling to generate a data structure including position information for user interface elements 114 within GUI 112. As such, the techniques described herein may be used to facilitate user interface translation between devices including screens with different constraints (e.g., user interface translation from a largerprimary device to a smaller target device, or user interface translation from a smaller primary device to a larger target device). In some examples, primary device 102 and target device 104 may include screens with the same dimensions, but may have different constraints such as resolution, and / or may have different accessibility features (e.g., target device 104 may include a touchscreen while primary device 102 may not, or vice-versa).

[0030] Thus, as described herein, computing system 100 may improve user interface translation and cross-device compatibility of applications through an automated method of determining the user interface elements to be translated between devices, determining the constraints of each device, and using parallel machine learning models to generate user interface translation instructions that maintain overall design, aesthetics, and interactivity.

[0031] FIG. 2 is a block diagram further illustrating an example computing system configured to facilitate user interface translation from a primary device to a target device using generative modeling, in accordance with one or more techniques of the present disclosure. Computing system 200 may be similar if not substantially similar to computing system 100 of FIG. 1.

[0032] As shown in FIG. 2, computing system 200 includes user interface components 232, processors 224, communication units 228, and one or more storage devices 226. Storage device 226 further includes user interface module 206, communication channels 230, and storage device(s) 226. Storage device 226 further includes application programming interface (API) module 218, operating system (OS) 222, and UI module 206, which may be similar if not substantially similar to UI module 106 of FIG. 1. As shown in the example of FIG. 2, UI module 206 further includes machine learning module 242, training module 244, prompt generation module 240, and data processing module 234 including image processing module 236 and image metadata module 238, all of which may be referred to collectively herein as “modules 234-240.”

[0033] One or more communication units 228 of computing system 200, for example, may communicate with external devices by transmitting and / or receiving data at computing system 200, such as to and from remote computer systems or devices, such as primary device 102 and target device 104 of FIG. 1. Example communication units 228 include a network interface card (e.g., such as an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device that can send and / or receive information. Other examples of communication units 228 may be devices configured totransmit and receive Ultrawideband®, Bluetooth®, GPS, 3G, 4G, and Wi-Fi®, etc. that may be found in computing devices, such as mobile devices and the like.

[0034] As shown in the example of FIG. 2, communication channels 230 may interconnect each of the components as shown for inter-component communications (physically, communicatively, and / or operatively). In some examples, communication channels 230 may include a system bus, a network connection (e.g., to a wireless connection as described above), one or more inter-process communication data structures, or any other components for communicating data between hardware and / or software locally or remotely.

[0035] In the example of FIG. 2, one or more processors 224 may implement functionality and / or execute instructions within computing system 200. For example, one or more processors 224 may receive and execute instructions that provide the functionality of UIC 232, communication units 228, one or more storage devices 226 and an operating system to perform one or more operations as described herein. For example, one or more processors 224 may receive and execute instructions that provide the functionality of some or all of API module 218, OS 222, UI module 206, and / or modules 234-240 to perform one or more operations and various functions described herein. The one or more processors 224 may include a central processing unit (CPU). Examples of CPUs include, but are not limited to, a digital signal processor (DSP), a general-purpose microprocessor, a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU or another processing device, an application specific integrated circuit (ASIC), a field programmable logic array (FPGA), or other equivalent integrated or discrete logic circuitry, or other equivalent integrated or discrete logic circuitry.

[0036] One or more storage devices 226 within computing system 200 may store information for processing during operation of computing system 200 (e.g., computing system 200 may store data that API module 218, OS 222, UI module 206, and / or modules 234-240 may access during execution at computing system 200). In some examples, storage devices 226 is a temporary memory, meaning that a primary purpose of storage device 226 is not long-term storage. Storage devices 226 may be configured for shortterm storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories.

[0037] Storage devices 226, in some examples, may also include one or more computer- readable storage media. Storage devices 226 may be configured to store larger amounts of information than volatile memory. Storage devices 226 may further be configured for long-term storage of information as non-volatile memory space and retain information after power on / off cycles. Examples of non-volatile memories include magnetic hard disks, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Storage devices 226 may store program instructions and / or information (e.g., data) associated with API module 218, OS 222, UI module 206, and / or modules 234-240.

[0038] API module 218, OS 222, UI module 206, and modules 234-240 may perform operations described herein using software, hardware, firmware, or a mixture of hardware, software, and firmware residing in and executing on computing system 200 or at one or more other remote computing devices (e.g., cloud-based application - not shown) or other devices (such as primary device 102 and target device 104 of FIG. 1). Computing system 200 may execute one or more of API module 218, OS 222, UI module 206, and modules 234-240 with one or more processors 224 or may execute any or part of one or more of API module 218, OS 222, UI module 206, and modules 234-240 as or within a virtual machine executing on underlying hardware. One or more of API module 218, OS 222, UI module 206, and modules 234-240 may be implemented in various ways, for example, as a downloadable or pre-installed application, remotely as a cloud application, or as part of the operating system of computing system 200. Other examples of computing system 200 that implement techniques of this disclosure may include additional components not shown in FIG. 2.

[0039] Operating system 222, in some examples, controls the operation of components of computing system 200. For example, operating system 222, in one example, facilitates the communication of API module 218, UI module 206, and modules 234-240 with processors 224, one or more UI components 232, one or more communication units 228, and one or more communication channels 230. API module 218, UI module 206, and / or modules 234-240 may each include program instructions and / or data that are executable by computing system 200 (e.g., by one or more processors 224). As one example, UI module 206 can include instructions that cause computing system 200 to perform one or more of the operations and actions described in the present disclosure.

[0040] As described with respect to FIG. 1, UI module 106 may facilitate user interface translation that incorporates generative modeling to optimize user interface layouts ontarget devices. As described above, with explicit user consent and confirmation of association between devices, computing system 200 may retrieve, via API module 218, information associated with a plurality of user interface elements included in a first graphical user interface (e.g., elements 114 included in GUI 112 of FIG. 1), and retrieve constraints for a second graphical user interface (e.g., screen dimensions and / or resolution for GUI 116 of FIG. 1). UI module 206 may receive the information associated with the first and second GUIs and apply one or more of modules 234-240 to generate a data structure including position information for the plurality of user interface elements within the second graphical user interface. In some examples, UI module 206 may receive information and instructions from one or more associated platforms, operating systems, applications, and / or services executing at computing system 200 for generating the data structure. In addition, UI module 206 may act as an intermediary between the one or more associated platforms, operating systems, applications, and / or services executing at computing system 200 and various output devices of computing system 200 to produce output with computing system 200.

[0041] As shown in the example of FIG. 2, UI module 206 further includes data processing module 234, machine learning module 242, and training module 244. Data processing module 234 further includes image processing module 236 and image metadata module 238, and may be configured to receive data retrieved by API module 218 from external devices (e.g., primary device 102 and target device 104 of FIG. 1). Data processing module 234 may be configured as a module for processing data retrieved by API module 218 prior to UI module 206 implementing prompt generation module 240, training module 244, and / or machine learning module 242. Specifically, data processing module 234 may receive information associated with a plurality of user interface elements included in a first graphical user interface. As described herein, in some examples, data processing module 234 may be configured to censor, encrypt, and / or remove any personal information associated with a user that may be included in the information associated with the plurality of user interface elements.

[0042] Image processing module 236 may be configured to analyze the information associated with the plurality of user interface elements and categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements. For example, image processing module 236 may categorize one or more pixels included in the information that correspond to background images or noninteractive iconsas corresponding to non-clickable user interface elements. Image processing module 236 may then generate the first subset of the information associated with the plurality of user interface elements that includes the categorized pixels. UI module 206 may convert the first subset of the information into a masked image by applying a binary mask, and provide the masked image to machine learning module 242 as input.

[0043] Image metadata module 238 may determine, based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements, such as buttons, interactive icons, etc. The metadata may be indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element. Image metadata module 238 may generate, based on the metadata, a second subset of the information associated with the plurality of user interface elements, which may be provided to prompt generation module 240.

[0044] As described herein, with explicit user consent, API module 218 may retrieve information associated with a target device configured to display a second graphical user interface, in which UI module 206 may determine, based on the information associated with the target device, constraints of the second graphical user interface. In some examples, the information associated with the target device may include target device model information. In these examples, UI module 206 may determine, based on the target device model information, one or more of the dimensions of the second graphical user interface and display resolution of the second graphical user interface, or any other screen constraints associated with the target device.

[0045] UI module 206 and / or data processing module 234 may identify, based on the information associated with the target device, one or more adjustments to data included in the second subset of the information associated with the plurality of user interface elements (e.g., the clickable user interface elements metadata). For example, image metadata module 238 may output metadata indicative of the constraints of a user interface element included in the first GUI (e.g., the GUI displayed by a primary device). UI module 206 may determine the constraints of a second GUI (e.g., a GUI displayed by a target device), and identify, based on the constraints of the second GUI, one or more needed adjustments to the constraints of the user interface.

[0046] Prompt generation module 240 may receive the second subset of the information associated with the plurality of user interface elements (e.g., the clickable user interfaceelements metadata), the information associated with the target device (e.g., the dimensions of the second GUI displayed by the target device), and the one or more identified adjustments. Prompt generation module 240 may apply one or more templates to the second subset of the information, the information associated with the target device, and the one or more adjustments to generate a prompt. As described herein, the prompt may be a natural language prompt that indicates instructions for a machine learning model (e.g., a language model) to generate data indicative of a target user interface layout based on the aforementioned information provided to prompt generation module 240. The prompt generated by prompt generation module 240 may be provided to machine learning module 242 as input.

[0047] As described above, UI module 206 may generate a data structure including position information for the plurality of user interface elements within the second graphical user interface. In some examples, computing system 200 sends, to the target device, the data structure including the position information for the plurality of user interface elements within the second graphical user interface, such that the target device displays the target user interface layout for the plurality of user interface elements. In some examples, computing system 200 receives, from the target device, user input indicating one or more desired changes to the position information. As such, in some examples, prompt generation module 240 may generate a prompt including the one or more desired changes. In some examples, prompt generation module 240 provides the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

[0048] As described herein, computing system 200 may implement parallel machine learning models to generate user interface translation instructions that maintain overall design, aesthetics, and interactivity. Specifically, machine learning module 242 of UI module 206 may apply an image-to-image diffusion model to the target image output by image processing module 236 and a language model to the prompt output by prompt generation module 240.

[0049] In some examples, machine learning module 242 may be implemented by one or more of modules 234-240 for reasons including but not limited to, data processing, data optimization, data validation, prompt generation, etc. For example, in some examples, prompt generation module 240 may implement machine learning module 242 to intelligently generate a prompt.

[0050] As such, in some examples, machine learning module 242 includes one or more machine-learned models trained to receive input data of one or more types and, in response, provide output data of one or more types. The input data may include one or more features that are associated with an instance or an example. In some implementations, the one or more features associated with the instance or example can be organized into a feature vector. In some implementations, the output data can include one or more predictions. Predictions can also be referred to as inferences. Thus, given features associated with a particular instance, machine learning module 242 can output a prediction for such instance based on the features.

[0051] Machine learning module 242 may include an image diffusion model and a language model. In some examples, however, machine learning module 242 can be or include one or more other various different types of machine-learned models. In particular, in some implementations, machine learning module 242 can perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0052] In some implementations, machine learning module 242 can perform various types of classification based on the input data. For example, machine learning module 242 can perform binary classification or multiclass classification. In binary classification, the output data can include a classification of the input data into one of two different classes. In multiclass classification, the output data can include a classification of the input data into one (or more) of more than two classes. The classifications can be single label or multi-label. Machine learning module 242 may perform discrete categorical classification in which the input data is simply classified into one or more classes or categories. As an example, data processing module 234 may implement machine learning module 242 to classify the information associated with the plurality of user interface elements included in the first graphical user interface as corresponding to clickable or non-clickable user interface elements.

[0053] In some implementations, machine learning module 242 can perform classification in which machine learning module 242 provides, for each of one or more classes, a numerical value descriptive of a degree to which it is believed that the input data should be classified into the corresponding class. In some instances, the numerical values provided by machine learning module 242 can be referred to as “confidence scores” that are indicative of a respective confidence associated with classification of the input into the respective class. In some implementations, the confidence scores can becompared to one or more thresholds to render a discrete categorical prediction. In some implementations, only a certain number of classes (e.g., one) with the relatively largest confidence scores can be selected to render a discrete categorical prediction.

[0054] Machine learning module 242 may output a probabilistic classification. For example, machine learning module 242 may predict, given a sample input, a probability distribution over a set of classes. Thus, rather than outputting only the most likely class to which the sample input should belong, machine learning module 242 can output, for each class, a probability that the sample input belongs to such class. In some implementations, the probability distribution over all possible classes can sum to one. In some implementations, a Softmax function, or other type of function or layer can be used to squash a set of real values respectively associated with the possible classes to a set of real values in the range (0, 1) that sum to one.

[0055] In some examples, the probabilities provided by the probability distribution can be compared to one or more thresholds to render a discrete categorical prediction. In some implementations, only a certain number of classes (e.g., one) with the relatively largest predicted probability can be selected to render a discrete categorical prediction.

[0056] In cases in which machine learning module 242 performs classification, machine learning module 242 may be trained using supervised learning techniques. For example, machine learning module 242 may be trained on a training dataset that includes training examples labeled as belonging (or not belonging) to one or more classes.

[0057] In some implementations, machine learning module 242 can perform regression to provide output data in the form of a continuous numeric value. The continuous numeric value can correspond to any number of different metrics or numeric representations, including, for example, currency values, scores, or other numeric representations. As examples, machine learning module 242 can perform linear regression, polynomial regression, or nonlinear regression. As examples, machine learning module 242 can perform simple regression or multiple regression. As described above, in some implementations, a Softmax function or other function or layer can be used to squash a set of real values respectively associated with two or more possible classes to a set of real values in the range (0, 1) that sum to one.

[0058] Machine learning module 242 may perform various types of clustering. For example, machine learning module 242 can identify one or more previously-defined clusters to which the input data most likely corresponds. Machine learning module 242 may identify one or more clusters within the input data. That is, in instances in which theinput data includes multiple objects, documents, or other entities, machine learning module 242 can sort the multiple entities included in the input data into a number of clusters. In some implementations in which machine learning module 242 performs clustering, machine learning module 242 can be trained using unsupervised learning techniques.

[0059] Machine learning module 242 may perform anomaly detection or outlier detection. For example, machine learning module 242 can identify input data that does not conform to an expected pattern or other characteristic (e.g., as previously observed from previous input data). As examples, the anomaly detection can be used for fraud detection or system failure detection.

[0060] In some implementations, machine learning module 242 can provide output data in the form of one or more recommendations. For example, machine learning module 242 can be included in a recommendation system or engine. As an example, given input data that describes previous outcomes for certain entities (e.g., a score, ranking, or rating indicative of an amount of success or enjoyment), machine learning module 242 can output a suggestion or recommendation of one or more additional entities that, based on the previous outcomes, are expected to have a desired outcome (e.g., elicit a score, ranking, or rating indicative of success or enjoyment). As one example, given input data descriptive of a context of a computing device, such as computing system 200, a recommendation system can output a suggestion or recommendation of an application that the user might enjoy or wish to download to computing system 200.

[0061] Machine learning module 242 may, in some cases, act as an agent within an environment. For example, machine learning module 242 can be trained using reinforcement learning, which will be discussed in further detail below.

[0062] In some implementations, machine learning module 242 can be a parametric model while, in other implementations, machine learning module 242 can be a nonparametric model. In some implementations, machine learning module 242 can be a linear model while, in other implementations, machine learning module 242 can be a non-linear model.

[0063] As described above, machine learning module 242 can be or include one or more of various different types of machine-learned models. Examples of such different types of machine-learned models are provided below for illustration. One or more of the example models described below can be used (e.g., combined) to provide the output data inresponse to the input data. Additional models beyond the example models provided below can be used as well.

[0064] In some implementations, machine learning module 242 can be or include one or more classifier models such as, for example, linear classification models; quadratic classification models; etc. Machine learning module 242 may be or include one or more regression models such as, for example, simple linear regression models; multiple linear regression models; logistic regression models; stepwise regression models; multivariate adaptive regression splines; locally estimated scatterplot smoothing models; etc.

[0065] In some examples, machine learning module 242 can be or include one or more decision tree-based models such as, for example, classification and / or regression trees; iterative dichotomiser 3 decision trees; C4.5 decision trees; chi-squared automatic interaction detection decision trees; decision stumps; conditional decision trees; etc.

[0066] Machine learning module 242 may be or include one or more kernel machines. In some implementations, machine learning module 242 can be or include one or more support vector machines. Machine learning module 242 may be or include one or more instance-based learning models such as, for example, learning vector quantization models; self- organizing map models; locally weighted learning models; etc. In some implementations, machine learning module 242 can be or include one or more nearest neighbor models such as, for example, k-nearest neighbor classifications models; k- nearest neighbors regression models; etc. Machine learning module 242 can be or include one or more Bayesian models such as, for example, naive Bayes models; Gaussian naive Bayes models; multinomial naive Bayes models; averaged one-dependence estimators; Bayesian networks; Bayesian belief networks; hidden Markov models; etc.

[0067] In some implementations, machine learning module 242 can be or include one or more artificial neural networks (also referred to simply as neural networks). A neural network can include a group of connected nodes, which also can be referred to as neurons or perceptrons. A neural network can be organized into one or more layers. Neural networks that include multiple layers can be referred to as “deep” networks. A deep network can include an input layer, an output layer, and one or more hidden layers positioned between the input layer and the output layer. The nodes of the neural network can be connected or non-fully connected.

[0068] Machine learning module 242 can be or include one or more feed forward neural networks. In feed forward networks, the connections between nodes do not form a cycle.For example, each connection can connect a node from an earlier layer to a node from a later layer.

[0069] In some instances, machine learning module 242 can be or include one or more recurrent neural networks. In some instances, at least some of the nodes of a recurrent neural network can form a cycle. Recurrent neural networks can be especially useful for processing input data that is sequential in nature. In particular, in some instances, a recurrent neural network can pass or retain information from a previous portion of the input data sequence to a subsequent portion of the input data sequence through the use of recurrent or directed cyclical node connections.

[0070] In some examples, sequential input data can include time-series data (e.g., sensor data versus time or imagery captured at different times). For example, a recurrent neural network can analyze sensor data versus time to detect or predict a swipe direction, to perform handwriting recognition, etc. Sequential input data may include words in a sentence (e.g., for natural language processing, speech detection or processing, etc.); notes in a musical composition; sequential actions taken by a user (e.g., to detect or predict sequential application usage); sequential object states; etc.

[0071] Example recurrent neural networks include long short-term (LSTM) recurrent neural networks; gated recurrent units; bi-direction recurrent neural networks; continuous time recurrent neural networks; neural history compressors; echo state networks; Elman networks; Jordan networks; recursive neural networks; Hopfield networks; fully recurrent networks; sequence-to- sequence configurations; etc.

[0072] In some implementations, machine learning module 242 can be or include one or more convolutional neural networks. In some instances, a convolutional neural network can include one or more convolutional layers that perform convolutions over input data using learned filters.

[0073] Filters can also be referred to as kernels. Convolutional neural networks can be especially useful for vision problems such as when the input data includes imagery such as still images or video. However, convolutional neural networks can also be applied for natural language processing.

[0074] In some examples, machine learning module 242 can be or include one or more generative networks such as, for example, generative adversarial networks. Generative networks can be used to generate new data such as new images or other content.

[0075] Machine learning module 242 may be or include an autoencoder. In some instances, the aim of an autoencoder is to learn a representation (e.g., a lower-dimensional encoding) for a set of data, typically for the purpose of dimensionality reduction. For example, in some instances, an autoencoder can seek to encode the input data and then provide output data that reconstructs the input data from the encoding. As an example, in some examples, image metadata module 238 may implement machine learning module 242 to encode the metadata into text tokens, in which the text tokens may be further passed into the language model described herein along with a prompt. In some instances, the autoencoder can include additional losses beyond reconstructing the input data.

[0076] Machine learning module 242 may be or include one or more other forms of artificial neural networks such as, for example, deep Boltzmann machines; deep belief networks; stacked autoencoders; etc. Any of the neural networks described herein can be combined (e.g., stacked) to form more complex networks.

[0077] One or more neural networks can be used to provide an embedding based on the input data. For example, the embedding can be a representation of knowledge abstracted from the input data into one or more learned dimensions. In some instances, embeddings can be a useful source for identifying related entities. In some instances, embeddings can be extracted from the output of the network, while in other instances embeddings can be extracted from any hidden node or layer of the network (e.g., a close to final but not final layer of the network). Embeddings can be useful for performing auto suggest next video, product suggestion, entity or object recognition, etc. In some instances, embeddings are useful inputs for downstream models. For example, embeddings can be useful to generalize input data (e.g., search queries) for a downstream model or processing system.

[0078] Machine learning module 242 may include one or more clustering models such as, for example, k-means clustering models; k-medians clustering models; expectation maximization models; hierarchical clustering models; etc.

[0079] In some implementations, machine learning module 242 can perform one or more dimensionality reduction techniques such as, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc.

[0080] In some implementations, machine learning module 242 can perform or be subjected to one or more reinforcement learning techniques such as Markov decisionprocesses; dynamic programming; Q functions or Q-leaming; value function approaches; deep Q-networks; differentiable neural computers; asynchronous advantage actor-critics; deterministic policy gradient; etc.

[0081] In some implementations, machine learning module 242 can be an autoregressive model. In some instances, an autoregressive model can specify that the output data depends linearly on its own previous values and on a stochastic term. In some instances, an autoregressive model can take the form of a stochastic difference equation. One example of an autoregressive model is WaveNet, which is a generative model for raw audio.

[0082] In some implementations, machine learning module 242 can include or form part of a multiple model ensemble. As one example, bootstrap aggregating can be performed, which can also be referred to as “bagging.” In bootstrap aggregating, a training dataset is split into a number of subsets (e.g., through random sampling with replacement) and a plurality of models are respectively trained on the number of subsets. At inference time, respective outputs of the plurality of models can be combined (e.g., through averaging, voting, or other techniques) and used as the output of the ensemble.

[0083] One example ensemble is a random forest, which can also be referred to as a random decision forest. Random forests are an ensemble learning technique for classification, regression, and other tasks. Random forests are generated by producing a plurality of decision trees at training time. In some instances, at inference time, the class that is the mode of the classes (classification) or the mean prediction (regression) of the individual trees can be used as the output of the forest. Random decision forests can correct for decision trees' tendency to overfit their training set.

[0084] Another example ensemble technique is stacking, which can, in some instances, be referred to as stacked generalization. Stacking includes training a combiner model to blend or otherwise combine the predictions of several other machine-learned models. Thus, a plurality of machine-learned models (e.g., of same or different type) can be trained based on training data. In addition, a combiner model can be trained to take the predictions from the other machine-learned models as inputs and, in response, produce a final inference or prediction. In some instances, a single-layer logistic regression model can be used as the combiner model.

[0085] Another example ensemble technique is boosting. Boosting can include incrementally building an ensemble by iteratively training weak models and then adding to a final strong model. For example, in some instances, each new model can be trained toemphasize the training examples that previous models misinterpreted (e.g., misclassified). For example, a weight associated with each of such misinterpreted examples can be increased. One common implementation of boosting is AdaBoost, which can also be referred to as Adaptive Boosting. Other example boosting techniques include LPBoost; TotalBoost; BrownBoost; xgboost; MadaBoost, LogitBoost, gradient boosting; etc. Furthermore, any of the models described above (e.g., regression models and artificial neural networks) can be combined to form an ensemble. As an example, an ensemble can include a top level machine-learned model or a heuristic function to combine and / or weight the outputs of the models that form the ensemble.

[0086] In some implementations, multiple machine-learned models (e.g., that form an ensemble can be linked and trained jointly (e.g., through backpropagation of errors sequentially through the model ensemble). However, in some implementations, only a subset (e.g., one) of the jointly trained models is used for inference.

[0087] In some implementations, as described above, machine learning module 242 can be used to preprocess the input data for subsequent input into another model. For example, machine learning module 242 can perform dimensionality reduction techniques and embeddings (e.g., matrix factorization, principal components analysis, singular value decomposition, word2vec / GLOVE, and / or related approaches); clustering; and even classification and regression for downstream consumption. As another example, machine learning module 242 may encode the metadata determined by image metadata module 238 as text tokens for input to a prompt-engineered language model.

[0088] As discussed above, machine learning module 242 can be trained or otherwise be configured to receive the input data and, in response, provide the output data. The input data can include different types, forms, or variations of input data. As examples, in various implementations, the input data can include features that describe the content (or portion of content) of a user interface, device model information, screen constraints (e.g., dimensions, resolution, accessibility, etc.). In some implementations, machine learning module 242 can receive and use the input data in its raw form. For example, in some examples, raw user interface information may be passed through a “background extractor” which can filter out clickable user interface elements and keep information (e.g., pixels) corresponding to non-clickable user interface elements. In some implementations, the raw input data can be preprocessed, such as by data processing module 234. Thus, in addition or alternatively to the raw input data, machine learning module 242 can receive and use the preprocessed input data.

[0089] In some implementations, preprocessing the input data can include extracting one or more additional features from the raw input data. For example, feature extraction techniques can be applied to the input data to generate one or more new, additional features. Example feature extraction techniques include edge detection; corner detection; blob detection; ridge detection; scale-invariant feature transform; motion detection; optical flow; Hough transform; etc.

[0090] In some implementations, the extracted features can include or be derived from transformations of the input data into other domains and / or dimensions. As an example, the extracted features can include or be derived from transformations of the input data into the frequency domain. For example, wavelet transformations and / or fast Fourier transforms can be performed on the input data to generate additional features.

[0091] In some implementations, the extracted features can include statistics calculated from the input data or certain portions or dimensions of the input data. Example statistics include the mode, mean, maximum, minimum, or other metrics of the input data or portions thereof.

[0092] In some implementations, as described above, the input data can be sequential in nature. In some instances, the sequential input data can be generated by sampling or otherwise segmenting a stream of input data. As one example, frames can be extracted from a video. In some implementations, sequential data can be made non-sequential through summarization.

[0093] As another example preprocessing technique, portions of the input data can be imputed. For example, additional synthetic input data can be generated through interpolation and / or extrapolation.

[0094] As another example preprocessing technique, some or all of the input data can be scaled, standardized, normalized, generalized, and / or regularized. Example regularization techniques include ridge regression; least absolute shrinkage and selection operator (LASSO); elastic net; least-angle regression; cross-validation; LI regularization; L2 regularization; etc. As one example, some or all of the input data can be normalized by subtracting the mean across a given dimension’s feature values from each individual feature value and then dividing by the standard deviation or other metric.

[0095] As another example preprocessing technique, some or all or the input data can be quantized or discretized. In some cases, qualitative features or variables included in the input data can be converted to quantitative features or variables. For example, one hot encoding can be performed.

[0096] In some examples, dimensionality reduction techniques can be applied to the input data prior to input into machine learning module 242. Several examples of dimensionality reduction techniques are provided above, including, for example, principal component analysis; kernel principal component analysis; graph-based kernel principal component analysis; principal component regression; partial least squares regression; Sammon mapping; multidimensional scaling; projection pursuit; linear discriminant analysis; mixture discriminant analysis; quadratic discriminant analysis; generalized discriminant analysis; flexible discriminant analysis; autoencoding; etc.

[0097] In some implementations, during training, the input data can be intentionally deformed in any number of ways to increase model robustness, generalization, or other qualities. Example techniques to deform the input data include adding noise; changing color, shade, or hue; magnification; segmentation; amplification; etc.

[0098] In response to receipt of the input data, machine learning module 242 can provide the output data. The output data can include different types, forms, or variations of output data. As examples, in various implementations, the output data can include content, either stored locally on the user device or in the cloud, that is relevantly shareable along with the initial content selection.

[0099] As discussed above, in some implementations, the output data can include various types of classification data (e.g., binary classification, multiclass classification, single label, multi- label, discrete classification, regressive classification, probabilistic classification, etc.) or can include various types of regressive data (e.g., linear regression, polynomial regression, nonlinear regression, simple regression, multiple regression, etc.). In other instances, the output data can include clustering data, anomaly detection data, recommendation data, or any of the other forms of output data discussed above.

[0100] In some implementations, the output data can influence downstream processes or decision making. As one example, in some implementations, the output data can be interpreted and / or acted upon by a rules-based regulator.

[0101] The present disclosure provides techniques that include or otherwise leverage one or more machine-learned models to suggest content, either stored locally on the uses device or in the cloud, that is relevantly shareable along with the initial content selection based on features of the initial content selection. Any of the different types or forms of input data described above can be combined with any of the different types or forms of machine-learned models described above to provide any of the different types or forms of output data described above.

[0102] In some examples, computing system 200 may communicate over a network with an example server computing system that includes a machine-learned model. For example, a server device may store and implement machine learning module 242. In some instances, output data obtained through machine learning module 242 at a server device can be used to improve other server tasks or can be used by other non-user devices to improve services performed by or for such other non-user devices. For example, the output data can improve other downstream processes performed by a server device for a computing device of a user or embedded computing device. In other instances, output data obtained through implementation of machine learning module 242 at a server device can be sent to and used by a user computing device, such as computing system 200, an embedded computing device. For example, the server device can be said to perform machine learning as a service.

[0103] In yet other implementations, different respective portions of machine learning module 242 can be stored at and / or implemented by some combination of a user computing device; an embedded computing device; a server computing device; etc. In other words, portions of machine learning module 242 may be distributed in whole or in part amongst computing system 200 and a server device.

[0104] Computing system 200 and / or the server device may perform graph processing techniques or other machine learning techniques using one or more machine learning platforms, frameworks, and / or libraries, such as, for example, TensorFlow, Caffe / Caffe2, Theano, Torch / Py Torch, MXnet, CNTK, etc. Computing system 200 and / or the server device may be distributed at different physical locations and connected via one or more networks. If configured as distributed computing devices, computing system 200 and / or the server device may operate according to sequential computing architectures, parallel computing architectures, or combinations thereof. In one example, distributed computing devices can be controlled or guided through use of a parameter server.

[0105] In some implementations, multiple instances of machine learning module 242 can be parallelized to provide increased processing throughput. For example, the multiple instances of machine learning module 242 can be parallelized on a single processing device or computing device or parallelized across multiple processing devices or computing devices.

[0106] Each computing device that implements machine learning module 242 or other aspects of the present disclosure can include a number of hardware components that enable performance of the techniques described herein. For example, each computingdevice can include one or more memory devices that store some or all of machine learning module 242. For example, machine learning module 242 can be a structured numerical representation that is stored in memory. The one or more memory devices can also include instructions for implementing machine learning module 242 or performing other operations. Example memory devices include RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof.

[0107] Each computing device can also include one or more processing devices that implement some or all of machine learning module 242 and / or perform other related operations. Example processing devices include one or more of: a central processing unit (CPU); a visual processing unit (VPU); a graphics processing unit (GPU); a tensor processing unit (TPU); a neural processing unit (NPU); a neural processing engine; a core of a CPU, VPU, GPU, TPU, NPU or other processing device; an application specific integrated circuit (ASIC); a field programmable gate array (FPGA); a co-processor; a controller; or combinations of the processing devices described above. Processing devices can be embedded within other hardware components such as, for example, an image sensor, accelerometer, etc.

[0108] Hardware components (e.g., memory devices and / or processing devices) can be spread across multiple physically distributed computing devices and / or virtually distributed computing systems.

[0109] Machine learning module 242 described herein can be trained with training module 244 and then provided for storage and / or implementation at one or more computing devices, such as computing system 200. For example, training module 244 executes locally at computing system 200. However, in some examples, training module 244 can be separate from computing system 200 or any other computing device that implements machine learning module 242.

[0110] In some implementations, machine learning module 242 may be trained in an offline fashion or an online fashion. In offline training (also known as batch learning), machine learning module 242 is trained on the entirety of a static set of training data. In online learning, machine learning module 242 is continuously trained (or re-trained) as new training data becomes available (e.g., while the model is used to perform inference).

[0111] Training module 244 may perform centralized training of machine learning module 242 (e.g., based on a centrally stored dataset). In other implementations, decentralized training techniques such as distributed training, federated learning, or the like can be used to train, update, or personalize machine learning module 242.

[0112] Machine learning module 242 described herein can be trained according to one or more of various different training types or techniques. For example, in some implementations, machine learning module 242 can be trained by training module 244 using supervised learning, in which machine learning module 242 is trained on a training dataset that includes instances or examples that have labels. The labels can be manually applied by experts, generated through crowd-sourcing, or provided by other techniques (e.g., by physics-based or complex mathematical models). In some implementations, if the user has provided consent, the training examples can be provided by the user computing device. In some implementations, this process can be referred to as personalizing the model.

[0113] Training data used by training module 244 can include, upon user permission for use of such data for training, anonymized usage logs of sharing flows, e.g., content items that were shared together, bundled content pieces already identified as belonging together, e.g., from entities in a knowledge graph, etc. In some implementations, training data can include examples of input data that have been assigned labels that correspond to output data.

[0114] In some implementations, machine learning module 242 can be trained by optimizing an objective function. For example, in some implementations, the objective function may be or include a loss function that compares (e.g., determines a difference between) output data generated by the model from the training data and labels (e.g., ground-truth labels) associated with the training data. For example, the loss function can evaluate a sum or mean of squared differences between the output data and the labels. In some examples, the objective function may be or include a cost function that describes a cost of a certain outcome or output data. Other examples of an objective function can include margin-based techniques such as, for example, triplet loss or maximum-margin training.

[0115] One or more of various optimization techniques can be performed to optimize an objective function. For example, the optimization technique(s) can minimize or maximize the objective function. Example optimization techniques include Hessian-based techniques and gradient-based techniques, such as, for example, coordinate descent; gradient descent (e.g., stochastic gradient descent); subgradient techniques; etc. Other optimization techniques include black box optimization techniques and heuristics.

[0116] In some implementations, backward propagation of errors can be used in conjunction with an optimization technique (e.g., gradient based techniques) to trainmachine learning module 242 (e.g., when a machine-learned model is a multi-layer model such as an artificial neural network). For example, an iterative cycle of propagation and model parameter (e.g., weights) update can be performed to train machine learning module 242. Example backpropagation techniques include truncated backpropagation through time, Levenberg- Marquardt backpropagation, etc.

[0117] In some implementations, machine learning module 242 described herein can be trained using unsupervised learning techniques. Unsupervised learning can include inferring a function to describe hidden structure from unlabeled data. For example, a classification or categorization may not be included in the data. Unsupervised learning techniques can be used to produce machine-learned models capable of performing clustering, anomaly detection, learning latent variable models, or other tasks.

[0118] Machine learning module 242 can be trained using semi-supervised techniques which combine aspects of supervised learning and unsupervised learning. Machine learning module 242 can be trained or otherwise generated through evolutionary techniques or genetic algorithms. In some implementations, machine learning module 242 described herein can be trained using reinforcement learning. In reinforcement learning, an agent (e.g., model) can take actions in an environment and learn to maximize rewards and / or minimize penalties that result from such actions. Reinforcement learning can differ from the supervised learning problem in that correct input / output pairs are not presented, nor sub-optimal actions explicitly corrected.

[0119] In some implementations, one or more generalization techniques can be performed during training to improve the generalization of machine learning module 242. Generalization techniques can help reduce overfitting of machine learning module 242 to the training data. Example generalization techniques include dropout techniques; weight decay techniques; batch normalization; early stopping; subset selection; stepwise selection; etc.

[0120] In some implementations, machine learning module 242 described herein can include or otherwise be impacted by a number of hyperparameters, such as, for example, learning rate, number of layers, number of nodes in each layer, number of leaves in a tree, number of clusters; etc. Hyperparameters can affect model performance.Hyperparameters can be hand selected or can be automatically selected through application of techniques such as, for example, grid search; black box optimization techniques (e.g., Bayesian optimization, random search, etc.); gradient-based optimization; etc. Example techniques and / or tools for performing automatichyperparameter optimization include Hyperopt; Auto-WEKA; Spearmint; Metric Optimization Engine (MOE); etc.

[0121] In some implementations, various techniques can be used to optimize and / or adapt the learning rate when the model is trained. Example techniques and / or tools for performing learning rate optimization or adaptation include Adagrad; Adaptive Moment Estimation (ADAM); Adadelta; RMSprop; etc.

[0122] In some implementations, transfer learning techniques can be used to provide an initial model from which to begin training of machine learning module 242 described herein.

[0123] In some implementations, machine learning module 242 described herein can be included in different portions of computer-readable code on a computing device. In one example, machine learning module 242 can be included in a particular application or program and used (e.g., exclusively) by such a particular application or program. Thus, in one example, a computing device can include a number of applications and one or more of such applications can contain its own respective machine learning library and machine- learned model(s).

[0124] In another example, machine learning module 242 described herein can be included in an operating system of a computing device (e.g., in a central intelligence layer of an operating system) and can be called or otherwise used by one or more applications that interact with the operating system. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an application programming interface (API) (e.g., a common, public API across all applications).

[0125] In some implementations, the central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device. The central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0126] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks andfunctionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination.

[0127] Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0128] In addition, the machine learning techniques described herein are readily interchangeable and combinable. Although certain example techniques have been described, many others exist and can be used in conjunction with aspects of the present disclosure.

[0129] FIG. 3 is a block diagram illustrating an example machine learning module configured to generate data structures including position information for a plurality of user interface elements within a second graphical user interface, in accordance with one or more techniques of the present disclosure. Machine learning module 342 may be similar if not substantially similar to machine learning module 242 of FIG. 2. As shown in the example of FIG. 3, machine learning module 342 further includes language model 350 and image diffusion model 352. As described herein, UI module 206 may implement a machine learning architecture including image diffusion model 352 and language model 350 to generate a data structure including position information for elements 114 within GUI 116 of FIG. 1. In some examples, image diffusion model 352 and language model 350 may be implemented in parallel.

[0130] As described above, UI module 206 may output a masked image that includes non-clickable user interface elements. The masked image, which may include masked areas or “holes” around clickable user interface elements, may be provided to image diffusion model 352 as input. Image diffusion model 352 may be a generative model, such as an image-to-image diffusion model, configured to transform an input image (e.g., the masked image described herein) into a new or target image. In some examples, as described in more detail with respect to FIG. 4, image diffusion model 352 may receive the masked image including masked areas or “holes,” and fill in such areas or holes. In some examples, image diffusion model 352 may enhance and / or adjust the elements of an image to fit the specifications of a target device. As described above, the computing system may determine, based on the information associated with a target device, one or more of the dimensions of a second graphical user interface and display resolution of a second graphical user interface. The dimensions of the second graphical user interface may be defined based on the natural viewing size of the target device for which the UI isbeing translated or adapted to. As an example, the generated target image may have M x N dimensions that match up to the natural viewing size of the target device (in which M x N may be much smaller than M’ x N’, or the dimensions of a graphical user interface displayed by a primary device). Furthermore, image diffusion model 352 may resize or adjust the elements included in the masked image to match the target device constraints (e.g., screen dimensions, resolution, etc.), or the constraints of the second graphical user interface.

[0131] Image diffusion model 352 may generate a target image in which the masked areas of the input masked image may be filled with content that is visually coherent with the existing decorative elements. In some examples, image diffusion model 352 may generate the target image through a process of sequential sampling. Specifically, image diffusion model 352 may gradually reduce noise present in a noisy version of the input masked image over multiple steps, in which the image details may be refined with each step. In some examples, image diffusion model 352 may learn patterns and features through a training dataset, and use the learned representations to predict and fill in the areas or holes described above. In some examples, image diffusion model 352 may be trained to maintain certain styles or attributes (e.g., color schemes, patterns, etc.) such that the target image contains similar design and aesthetics as the original UI. In some examples, the target image generated by image diffusion model 352 may undergo additional resizing (e.g., downscaling or upscaling) to fit the specifications of the target device.

[0132] ML module 342 can be or include one or more transformer-based neural networks, such as language model 350. Language model 350 may implement, for example, the Pathways Language Model developed by Google. Transformer-based neural networks may refer to a type of deep learning architecture specifically designed for handling sequential data, such as text or time series. In other words, transformer-based neural networks such as large language models (LLMs) may be configured to perform natural language processing (NLP) tasks, such as question-answering, machine translation, text summarization, and sentiment analysis. Language model 350 may be configured to perform tasks such as classification, sentiment analysis, entity extraction, extractive question answering, summarization, re-writing text in a different style, ad copy generation, concept ideation, etc.

[0133] Transformer-based neural networks may utilize a self-attention mechanism, which allows the model to weigh the importance of different elements in a given input sequencerelative to each other. In some examples, a self-attention mechanism may help language model 350 effectively capture long-range dependencies and complex relationships between elements, such as words in a sentence.

[0134] Language model 350 may include an encoder and a decoder that operate to process and generate sequential data, such as structured text. Both the encoder and decoder may include one or more of self-attention mechanisms, position-wise feedforward networks, layer normalization, or residual connections. In some examples, the encoder may process an input sequence and create a representation that captures the relationships and context among the elements in the sequence. The decoder may then obtain the representation generated by the encoder and produce an output sequence. In some examples, the decoder may generate the output one element at a time (e.g., one word at a time), using a process called autoregressive decoding, in which the previously generated elements are used as input to predict the next element in the sequence.

[0135] In some examples, language model 350 may tokenize (e.g., split) a sequence of words or subwords, which language model 350 may convert into vectors (e.g., numerical representations) that language model 350 can process. Language model 350 may use a self-attention mechanism to weigh the importance of each token in relation to the others. In this way, language model 350 may identify patterns and relationships between the tokens, and in turn the words corresponding to the tokens, that indicate different types of information.

[0136] Specifically, language model 350 may tokenize extracted metadata or receive text tokens encoded from extracted metadata that corresponds to clickable user interface elements (e.g., metadata indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, one or more actions associated with a user interface element, and the like). As described above, language model 350 may be prompt- engineered, in which language model 350 may receive a prompt (e.g., from prompt generation module 240 of FIG. 2) that instructs language model 350 to rearrange UI element locations for optimal interfacing on a target device. In some examples, the prompt engineering can specifically include the target device screen and interface specifications.

[0137] In general, language model 350 may excel at performing NLP tasks, such as generating text and other content. In some examples, language model 350 may be configured with certain prerequisites or “rules” pertaining to a target user interface layout.For example, a prompt passed into language model 350 may include text inputs such as, for example, “Prioritize target device screen size and resolution,” “If not all of the user interface elements can fit in a single frame, make the target user interface scrollable,” etc. In other words, language model 350 may be provided with prompts that further specify how the information indicative of the target user interface layout should be generated by language model 350.

[0138] Language model 350 may generate the information indicative of the target user interface layout, which may be any type of file suitable for generating or defining user interfaces. For example, language model 350 may generate the information indicative of the target user interface layout in the form of one or more of an HTML file, a CSS file, a JavaScript File, an XML (Extensible Markup Language) file, a JSON file, an image file (PNG, JPG, SVG, etc.), a font file (TrueType Font, OpenType Font, etc.), or the like.

[0139] As described herein, computing system 200 may combine the target image and the information indicative of the target user interface layout to generate a data structure, in which the data structure includes position information for the plurality of user interface elements within the graphical user interface displayed by the target device. Specifically, the data structure may be in a form in which it can be received by a display driver of the target device and serve an end application on the target device.

[0140] FIG. 4 is a block diagram illustrating an example process for generating data structures including position information for a plurality of user interface elements within a second graphical user interface, in accordance with one or more techniques of the present disclosure. Data processing module 434, image processing module 436, image metadata module 438, prompt generation module 440, and machine learning module 442 may be similar if not substantially similar to data processing module 234, image processing module 236, image metadata module 238, prompt generation module 240, and machine learning module 242 of FIG. 2, respectively. Image diffusion model 452 and language model 450 may be similar if not substantially similar to image diffusion model 352 and language model 350 of FIG. 3, respectively.

[0141] As described herein, with explicit user consent and confirmation of association between devices, data processing module 434 may receive information associated with a plurality of user interface elements included in a first graphical user interface, and receive information associated with a target device configured to display a second graphical user interface. Data processing module 434 may determine, based on the information associated with the target device (e.g., target device model information), one or more ofthe dimensions of the second graphical user interface and display resolution of the second graphical user interface.

[0142] Image processing module 436 of data processing module 434 may categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements. Specifically, image processing module 436 may implement one or more algorithms to detect user interface elements and categorize them. As an example, image processing module 436 may implement an object detection algorithm configured to identify non-clickable elements such as buttons, text fields, icons, etc. As another example, image processing module 436 may implement semantic segmentation, which may involve classifying each pixel of an image into categories (e.g., button, background, text, etc.). Image processing module 436 may then generate the first subset of the information that includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements. In some examples, image processing module 436 may apply a binary mask to the first subset of the information to generate masked image 460, in which masked image 460 may only include the pixels categorized as corresponding to the non- clickable user interface elements. Specifically, image processing module 436 may generate a first mask for pixels categorized as corresponding to the clickable user interface elements (e.g., image processing module 436 may mark these pixels as white), and generate a second mask for pixels categorized as corresponding to the non-clickable user interface elements (e.g., image processing module 436 may mark these pixels as the background, or black). Image processing module 436 may extract the pixels marked as the background using inverse masking, in which the pixels categorized as corresponding to the clickable user interface elements may be “cut out” from the UI, such that only decorative elements of the UI (e.g., background images, colors, non-interactive icons, etc.) are maintained. In some examples, image processing module 436 may implement interpolation techniques to fill any “holes” left by removed elements. In some examples, image processing module 436 may implement machine learning techniques to predict background content. In some examples, image processing module 436 may implement additional techniques for image enhancement and refining, such as edge smoothing, color correction, brightness and / or contrast adjustments, etc. The final output of image processing module 436 may be masked image 460.

[0143] As described above with respect to FIG. 3, masked image 460 may be provided to image diffusion model 452 of machine learning module 442, in which image diffusion model 452 may output target image 462. Target image 462 may be in any suitable image file format, including, but not limited to, PNG, JPEG, TIFF, GIF, WebP, BMP (Bitmap Image File), SVG (Scalable Vector Graphics), and the like.

[0144] As described herein, in some examples, language model 450 may be simultaneously implemented with image diffusion model 452. As described above with respect to FIG. 2, image metadata module 438 of data processing module 434 may determine, based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements. In some examples, image metadata module 438 may receive data from image processing module 436 that is indicative of the clickable user interface elements. As described herein, image metadata module 438 may generate a second subset of the information that includes the metadata, in which the metadata may be indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element. In some examples, image metadata module 238 may implement machine learning module 442 to encode the metadata into text tokens, in which the text tokens may be further passed into language model 450 along with prompt 464 generated by prompt generation module 440.

[0145] Computing system 200 (and / or data processing module 434) may identify, based on the information associated with the target device, one or more adjustments to data included in the second subset of the information. As shown in the example of FIG. 4, image metadata module 438 may provide the second subset of the information associated with the plurality of user interface elements (e.g., the clickable user interface elements metadata), the information associated with the target device (e.g., the constraints of the second GUI displayed by the target device), and the one or more identified adjustments to prompt generation module 440. Prompt generation module 440 may then apply one or more templates to the aforementioned information to generate prompt 464, which may be a text file including a natural language prompt. As described herein, prompt 464 may indicate instructions for language model 450 to generate data indicative of a target user interface layout based on the template information. Furthermore, prompt 464 may include one or more prerequisites or rules pertaining to the target user interface layout.

[0146] Image diffusion model 452 may be applied to masked image 460 to generate target image 462, in which target image 462 may be an image with the constraints of the second graphical user interface. Language model 450 may be applied to the metadata generated by image metadata module 438 and / or prompt 464 to generate target user interface layout file 466. As shown in the example of FIG. 4, target image 462 and target user interface layout file 466 may then be provided to user interface parameter fusion module 470, which may combine target image 462 and target user interface layout file 466 to generate data structure 472. As described herein, data structure 472 may include position information for the plurality of user interface elements within the second graphical user interface displayed by the target device, and may be in a form in which it can be received by a display driver of the target device and serve an end application on the target device.

[0147] In this way, the techniques described herein may improve user interface translation between devices. Specifically, by implementing the data processing and machine learning techniques described above, the computing system may provide data for a well-fitted, visually pleasing, and readable translated user interface to a target device that also maintains the design and aesthetics of the original user interface as presented by a primary device.

[0148] FIG. 5 is a flow chart illustrating an example operation of a computing system that facilitates user interface translation from a primary device to a target device, in accordance with one or more techniques of the present disclosure. The components of FIG. 5 are described with respect to FIGS. 1-4. Computing system 200 retrieves information associated with elements 114 included in first GUI 112 (580). Computing system 200 retrieves information associated with target device 104 configured to display second GUI 116 (582). UI module 206 determines, based on the information associated with target device 104, constraints of GUI 116 (584). In some examples, the information associated with target device 104 includes target device model information. In some examples, UI module 206 determines, based on the target device model information, one or more of the dimensions of GUI 116 and display resolution of GUI 116.

[0149] In some examples, image processing module 236 of UI module 206 categorizes one or more pixels included in the information associated with elements 114 as corresponding to either clickable user interface elements or non-clickable user interface elements. In some examples, image processing module 236 generates a first subset of the information, in which the first subset of the information includes pixels categorized ascorresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements. In some examples, image processing module 236 applies a binary mask to the first subset of the information to generate masked image 460, in which masked image 460 only includes the pixels categorized as corresponding to the non-clickable user interface elements. UI module 206 converts the first subset of the information associated with elements 114 into masked image 460 (586).

[0150] Machine learning module 242 applies a first machine learning model to masked image 460 to generate target image 462 with the constraints of GUI 116 (588). In some examples, the first machine learning model is image-to-image diffusion model 352.

[0151] In some examples, image metadata module 238 determines, based on the information associated with elements 114, metadata that corresponds to clickable user interface elements, in which the metadata is indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element. In some examples, image metadata module 238 generates, based on the metadata, a second subset of the information. Prompt generation module 240 generates, based on the second subset of the information associated with elements 114, prompt 464 (590).

[0152] In some examples, UI module 206 identifies, based on the information associated with target device 104, one or more adjustments to data included in the second subset of the information. In some examples, prompt generation module 240 applies one or more templates to the information associated with target device 104, the second subset of the information, and the one or more adjustments to generate prompt 464.

[0153] In some examples, prompt generation module 240 provides prompt 464 as input to a second machine learning model. In some examples, the second machine learning model is language model 350. Machine learning module 242 applies the second machine learning model to prompt 464 to generate target user interface layout file 466 (592).

[0154] Parameter fusion module 470 combines target image 462 and target user interface layout file 466 to generate data structure 472, in which data structure 472 includes position information for elements 114 within GUI 116 (594).

[0155] In some examples, computing system 200 sends, to target device 104, data structure 472 including the position information for elements 114 within GUI 116. In some examples, computing system 200 receives, from target device 104, user inputindicating one or more desired changes to the position information for elements 114 within GUI 116. In some examples, prompt generation module 240 generates a prompt including the one or more desired changes. In some examples, prompt generation module 240 provides the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

[0156] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over, as one or more instructions or code, a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media, which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that may be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0157] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are instead directed to non-transient, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0158] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0159] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a hardware unit or provided by a collection of intraoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0160] It is to be recognized that, depending on the example, certain acts or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi -threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0161] In some examples, a computer-readable storage medium comprises a non- transitory medium. The term “non-transitory” indicates that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).

[0162] Example 1 : A method includes retrieving, by a computing system, information associated with a plurality of user interface elements included in a first graphical user interface; retrieving, by the computing system, information associated with a target device configured to display a second graphical user interface; determining, by thecomputing system, and based on the information associated with the target device, constraints of the second graphical user interface; converting, by the computing system, a first subset of the information associated with the plurality of user interface elements into a masked image; applying, by the computing system, a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generating, by the computing system, and based on a second subset of the information associated with the plurality of user interface elements, a prompt; applying, by the computing system, a second machine learning model to the prompt to generate information indicative of a target user interface layout; and combining, by the computing system, the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0163] Example 2: The method of example 1, wherein the method further includes categorizing, by the computing system, one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generating, by the computing system, the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements; and applying, by the computing system, a binary mask to the first subset of the information to generate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

[0164] Example 3: The method of any of examples 1 through 2, wherein the first machine learning model is an image-to-image diffusion model.

[0165] Example 4: The method of any of examples 1 through 3, wherein the method further includes determining, by the computing system, and based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements, wherein the metadata is indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element; and generating, by the computing system, and based on the metadata, the second subset of the information.

[0166] Example 5: The method of any of examples 1 through 4, wherein the second machine learning model is a language model, and wherein the method further includesidentifying, by the computing system, and based on the information associated with the target device, one or more adjustments to data included in the second subset of the information; applying, by the computing system, one or more templates to the information associated with the target device, the second subset of the information, and the one or more adjustments to generate the prompt; and providing, by the computing system, the prompt as input to the second machine learning model.

[0167] Example 6: The method of any of examples 1 through 5, wherein the method further includes sending, by the computing system and to the target device, the data structure including the position information for the plurality of user interface elements within the second graphical user interface; receiving, by the computing system and from the target device, user input indicating one or more desired changes to the position information for the plurality of user interface elements within the second graphical user interface; generating, by the computing system, a prompt including the one or more desired changes; and providing, by the computing system, the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

[0168] Example 7: The method of any of examples 1 through 6, wherein the information associated with the target device includes target device model information, wherein the method further includes determining, by the computing system and based on the target device model information, one or more of the dimensions of the second graphical user interface and display resolution of the second graphical user interface.

[0169] Example 8: A computing system comprising: one or more processors; and one or more storage devices that store instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: retrieve information associated with a plurality of user interface elements included in a first graphical user interface; retrieve information associated with a target device configured to display a second graphical user interface; determine, based on the information associated with the target device, constraints of the second graphical user interface; convert a first subset of the information associated with the plurality of user interface elements into a masked image; apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt; apply a second machine learning model to the prompt to generate information indicative of a target user interface layout; and combine the target image and theinformation indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0170] Example 9. The computing system of example 8, wherein the one or more processors are further configured to: categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generate the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements; and apply a binary mask to the first subset of the information to generate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non- clickable user interface elements.

[0171] Example 10. The computing system of examples 8 through 9, wherein the first machine learning model is an image-to-image diffusion model.

[0172] Example 11. The computing system of examples 8 through 10, wherein the one or more processors are further configured to: determine, based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements, wherein the metadata is indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element; and generate, based on the metadata, the second subset of the information.

[0173] Example 12. The computing system of examples 8 through 11, wherein the second machine learning model is a language model, and wherein the one or more processors are further configured to: identify, based on the information associated with the target device, one or more adjustments to data included in the second subset of the information; apply one or more templates to the information associated with the target device, the second subset of the information, and the one or more adjustments to generate the prompt; and provide the prompt as input to the second machine learning model.

[0174] Example 13. The computing system of examples 8 through 12, wherein the one or more processors are further configured to: send, to the target device, the data structure including the position information for the plurality of user interface elements within the second graphical user interface; receive, from the target device, user input indicating oneor more desired changes to the position information for the plurality of user interface elements within the second graphical user interface; generate a prompt including the one or more desired changes; and provide the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

[0175] Example 14. The computing system of examples 8 through 13, wherein the information associated with the target device includes target device model information, and wherein the one or more processors are further configured to: determine, based on the target device model information, one or more of the dimensions of the second graphical user interface and display resolution of the second graphical user interface.

[0176] Example 15. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to: retrieve information associated with a plurality of user interface elements included in a first graphical user interface; retrieve information associated with a target device configured to display a second graphical user interface; determine, based on the information associated with the target device, constraints of the second graphical user interface; convert a first subset of the information associated with the plurality of user interface elements into a masked image; apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt; apply a second machine learning model to the prompt to generate information indicative of a target user interface layout; and combine the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

[0177] Example 16. The non-transitory computer-readable storage medium of claim 15, wherein the one or more processors are further configured to: categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generate the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements; and apply a binary mask to the first subset of the information togenerate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

[0178] Example 17. The non-transitory computer-readable storage medium of examples 15 through 16, wherein the first machine learning model is an image-to-image diffusion model.

[0179] Example 18. The non-transitory computer-readable storage medium of examples 15 through 17, wherein the one or more processors are further configured to: determine, based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements, wherein the metadata is indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element; and generate, based on the metadata, the second subset of the information.

[0180] Example 19. The non-transitory computer-readable storage medium of examples 15 through 18, wherein the second machine learning model is a language model, and wherein the one or more processors are further configured to: identify, based on the information associated with the target device, one or more adjustments to data included in the second subset of the information; apply one or more templates to the information associated with the target device, the second subset of the information, and the one or more adjustments to generate the prompt; and provide the prompt as input to the second machine learning model.

[0181] Example 20. The non-transitory computer-readable storage medium of examples 15 through 19, wherein the one or more processors are further configured to: send, to the target device, the data structure including the position information for the plurality of user interface elements within the second graphical user interface; receive, from the target device, user input indicating one or more desired changes to the position information for the plurality of user interface elements within the second graphical user interface; generate a prompt including the one or more desired changes; and provide the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

[0182] Example 21. The non-transitory computer-readable storage medium of examples 19 through 20, wherein the information associated with the target device includes target device model information, and wherein the one or more processors are further configuredto: determine, based on the target device model information, one or more of the dimensions of the second graphical user interface and display resolution of the second graphical user interface.

[0183] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: retrieving, by a computing system, information associated with a plurality of user interface elements included in a first graphical user interface; retrieving, by the computing system, information associated with a target device configured to display a second graphical user interface; determining, by the computing system, and based on the information associated with the target device, constraints of the second graphical user interface; converting, by the computing system, a first subset of the information associated with the plurality of user interface elements into a masked image; applying, by the computing system, a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generating, by the computing system, and based on a second subset of the information associated with the plurality of user interface elements, a prompt; applying, by the computing system, a second machine learning model to the prompt to generate information indicative of a target user interface layout; and combining, by the computing system, the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

2. The method of claim 1, further comprising: categorizing, by the computing system, one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generating, by the computing system, the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non- clickable user interface elements; and applying, by the computing system, a binary mask to the first subset of the information to generate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

3. The method of any combination of claims 1 and 2, wherein the first machine learning model is an image-to-image diffusion model.

4. The method of any combination of claims 1-3, further comprising: determining, by the computing system, and based on the information associated with the plurality of user interface elements, metadata that corresponds to clickable user interface elements, wherein the metadata is indicative of one or more of user interface element positioning, user interface element dimensions, user interface element design, user interface element type, user interface element font, and one or more actions associated with a user interface element; and generating, by the computing system, and based on the metadata, the second subset of the information.

5. The method of any combination of claims 1-4, wherein the second machine learning model is a language model, the method further comprising: identifying, by the computing system, and based on the information associated with the target device, one or more adjustments to data included in the second subset of the information; applying, by the computing system, one or more templates to the information associated with the target device, the second subset of the information, and the one or more adjustments to generate the prompt; and providing, by the computing system, the prompt as input to the second machine learning model.

6. The method of any combination of claims 1-5, further comprising: sending, by the computing system and to the target device, the data structure including the position information for the plurality of user interface elements within the second graphical user interface; receiving, by the computing system and from the target device, user input indicating one or more desired changes to the position information for the plurality of user interface elements within the second graphical user interface; generating, by the computing system, a prompt including the one or more desired changes; andproviding, by the computing system, the prompt including the one or more desired changes as input to the second machine learning model to generate information indicative of an updated target user interface layout.

7. The method of any combination of claims 1-6, wherein the information associated with the target device includes target device model information, the method further comprising: determining, by the computing system and based on the target device model information, one or more of the dimensions of the second graphical user interface and display resolution of the second graphical user interface.

8. A computing system comprising: one or more processors; and one or more storage devices that store instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to: retrieve information associated with a plurality of user interface elements included in a first graphical user interface; retrieve information associated with a target device configured to display a second graphical user interface; determine, based on the information associated with the target device, constraints of the second graphical user interface; convert a first subset of the information associated with the plurality of user interface elements into a masked image; apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt; apply a second machine learning model to the prompt to generate information indicative of a target user interface layout; andcombine the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

9. The computing system of claim 8, wherein the instructions further cause the one or more processors to: categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generate the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements; and apply a binary mask to the first subset of the information to generate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

10. The computing system of any combination of claims 8 and 9, wherein the first machine learning model is an image-to-image diffusion model.

11. The computing system of any combination of claims 8-10, comprising means for performing any of the methods of claims 4-7.

12. A non-transitory computer-readable storage medium encoded with instructions that, when executed by one or more processors, cause one or more processors to: retrieve information associated with a plurality of user interface elements included in a first graphical user interface; retrieve information associated with a target device configured to display a second graphical user interface; determine, based on the information associated with the target device, constraints of the second graphical user interface; convert a first subset of the information associated with the plurality of user interface elements into a masked image; apply a first machine learning model to the masked image to generate a target image with the constraints of the second graphical user interface; generate, based on a second subset of the information associated with the plurality of user interface elements, a prompt; apply a second machine learning model to the prompt to generate information indicative of a target user interface layout; and combine the target image and the information indicative of the target user interface layout to generate a data structure, wherein the data structure includes position information for the plurality of user interface elements within the second graphical user interface.

13. The non-transitory computer-readable storage medium of claim 12, wherein the instructions further cause the one or more processors to: categorize one or more pixels included in the information associated with the plurality of user interface elements as corresponding to either clickable user interface elements or non-clickable user interface elements; generate the first subset of the information, wherein the first subset of the information includes pixels categorized as corresponding to the clickable user interface elements and pixels categorized as corresponding to the non-clickable user interface elements; and apply a binary mask to the first subset of the information to generate the masked image, wherein the masked image only includes the pixels categorized as corresponding to the non-clickable user interface elements.

14. The non-transitory computer-readable storage medium of any combination of claims 12 and 13, wherein the first machine learning model is an image-to-image diffusion model.

15. The non-transitory computer-readable storage medium of any combination of claims 12-14, the instructions further configured to cause the one or more processors to perform any of the methods of claims 4-7.

Images