Generation of user interface layout using artificial intelligence

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automated layout generation by an artificial intelligence system. Methods can include obtaining two or more discrete units of content. Based on the two or more discrete units of content a new layout is generated in a canvas. The layout generation can include: generating a bounding box as a presentation space for each given unit of content; generating positioning data specifying locations within the canvas at which each bounding box is located; assigning each bounding box to a corresponding user interface layer; and generating a compressed text representation of the new layout. The new layout can be rendered based on the text representation.

Description

GENERATION OF USER INTERFACE LAYOUT USING ARTIFICIAL INTELLIGENCECROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to Greek Application No. 20240100375, filed on May 20, 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to data processing and artificial intelligence systems that are trained to generate user interface layouts.

[0003] Advances in machine learning are enabling artificial intelligence to be implemented in more applications. For example, large language models have been implemented to allow for a conversational interaction with computers using natural language rather than a restricted set of prompts. This allows for a more natural interaction with the computer. Artificial intelligence is also capable of generating various different types of content including computer code, images, text, and audio.SUMMARY

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of obtaining, by an artificial intelligence (Al) system, two or more discrete units of content; generating, by the Al system and based on the two or more discrete units of content, a new layout of the two or more discrete units of content in a canvas, wherein the canvas represents an amount of display space available for presentation of the two or more discrete units of content, and wherein generating the new layout includes: generating, by the Al system, a bounding box as a presentation space for each given unit of content among the two or more discrete units of content; generating, by the Al system, positioning data specifying locations within the canvas at which the bounding box of each given unit of content are located; assigning, by the Al system, the bounding box for each given unit of content to a corresponding user interface layer; and generating, by the Al system, a text representation of the new layout that represents one or more of the bounding box, the positioning data, or the user interface layer using a compressed token representation; and rendering, based on the text representation ofthe new layout, the two or more discrete units of content according to the new layout, wherein each given unit of content is rendered in (i) the corresponding user interface layer assigned to that given unit of content, and within the bounding box generated for that given unit of content.

[0005] These and other embodiments can each optionally include one or more of the following features. Generating the text representation using the token compressed representation can include generating a representation of layout parameters of the new layout using fewer tokens than full text labels of the layout parameters.

[0006] Methods can include obtaining, by the Al system, a layout personality of a content provider from which the two or more discrete units of content were obtained. The layout personality can include a set of style tags assigned to previous layouts of the content provider. Generating the new layout can include generating the new layout based on the layout personality.

[0007] Methods can include extracting, from a uniform resource locator of the content provider, a set of content presented at the uniform resource locator; evaluating features of the set of content to obtain layout preferences of the content provider; and generating the layout personality of the content provider based on the layout preferences. The layout preferences include one or more text preferences, image preferences, font preferences, logo preferences, or image preferences.

[0008] Methods can include extracting, from one or more content sources; a set of training data; performing one or more data synthesis operations to generate additional training data for the Al system. At least one data synthesis operations can include creating variations of training samples from the set of training data; and training a language model of the Al system based on the set of training data and the generated additional training data.

[0009] Generating the text representation of the new layout can include generating a sequence of discrete tokens specifying bounding boxes, content identifiers specifying which of the two or more discrete units of content that have been assigned to each of the bounding boxes, and layers to which each of the bounding boxes have been assigned.

[0010] Generating the text representation of the new layout can include compressing representations layout parameters in a proto string.

[0011] This specification describes techniques for enabling an artificial intelligence (Al) system to generate user interface layouts in an efficient manner. One of the technical challenges of implementing Al to perform specific tasks is the amount of time that it takes the Al system to perform the task, which is referred to as the Al latency. The Al latency is particularly notable in the context of generative Al models, which are typically implemented using large language models (LLMs). For example, these LLMs can take 20 seconds or more to generate an output after receiving the input to which the LLM is responding.

[0012] One way to reduce the latency of a large language model is to use a smaller language model. However, use of a smaller language model negatively impacts the ability to accurately analyze / understand inputs and generate information that is relevant to the inputs. Therefore, simply using a smaller language model to achieve a target latency (also referred to as a latency constraint) will result in a system that is more likely to generate a less accurate / less desirable (e.g., lower quality ) output, which is counter-productive and wastes computing resources. Generally speaking, the size of a language model is evaluated / determined based on the number of parameters used by the model, with smaller models using fewer parameters than larger models.

[0013] While smaller models (e.g., models with fewer parameters) have a lower latency (e.g., ~10 seconds), continuing to reduce the size of the model eventually results in an output having less than a required level of accuracy / quality. As discussed in more detail below, the present Al system is optimized to reduce the latency of the model (e g., meet a target latency), while retaining sufficient quality by changing the manner in which the model handles inputs. For example, the present Al system uses compressed layout representations that can reduce the Al latency by as much as 80% relative to using non-compressed layout representations. In this way, the present Al system can meet specified latency constraints with a larger LLM than would be possible without the use of the compressed layout representations. In this way, a higher level of model quality / accuracy is maintained while still meeting the specified latency constraint.

[0014] More specifically, the present Al system can be trained using a set of compressed representations of layout parameters, and also trained to generate compressed representations of layout parameters. This reduces the number of tokens generated by the Al system significantly. For example, instead of using the phrase “bounding box” each time a boundingbox parameter is generated by the Al system, the Al system can generate “BBox”, or some other shorthand version of the layout parameter “bounding box.” In this simple example, it is easy to see that generating “BBox” rather than “bounding box” reduces the number of tokens generated by the Al system by more than 60%, which reduces the Al system latency.

[0015] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. l is a block diagram of an example environment in which generative artificial intelligence can be implemented.

[0017] FIG. 2 is a block diagram illustrating the creation of a layout using the language model.

[0018] FIG. 3 is a flow chart of an example process for generating a new layout for a set of assets.

[0019] FIG. 4 a block diagram of an example computer.

[0020] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0021] This specification describes techniques for training an artificial intelligence (Al) system to spatially arrange a set of discrete content units, also referred to as assets. The Al system can also be configured to specify layers of a user interface in which each of the assets is presented, and can also specify other formatting for each of the assets (e.g., color, style, font, etc.). As used throughout this document, parameters specifying spatial arrangement, user interface layer depth, and / or other formatting for the assets is referred to as a layout.

[0022] The Al system can include a language model (e.g., an LLM) that is trained / configured to generate new layouts for a set of discrete units of content / assets. For example, when the Al system obtains two or more assets , the Al system can generate a new layout for the assets in a canvas. The canvas represents an amount of display space available for presentation of the two or more discrete units of content. For example, in the context of aweb page, the canvas may be the dimensions of the web page. Meanwhile, in the context of a native mobile application, the canvas may be the dimensions of the application user interface. The canvas can correspond to the full size of the user interface (e.g., web page or application user interface) or less than the full size. For example, in some situations, third party content is being formatted for presentation in a specified area (e.g., sub-portion) of a web page or application user interface. In these situations, the canvas used for formatting the assets can be based on (e.g., equal to) the size of the area in which the third-party content will be presented.

[0023] The generation / creation of the layout can include the generation / creation of bounding boxes for each of the assets. Each bounding box can correspond to, or be, a presentation space for each given unit of content among the two or more discrete units of content. For example, a bounding box can specify a set of dimensions (e.g., of a rectangle, triangle, or another geometric shape) of the user interface area in which an asset can be presented (e.g., (a) pixels x (b) pixels, where (a) and (b) are numbers).

[0024] The generation / creation of the layout can also include the generation / creation of positioning data. The positioning data indicates where, within the user interface, the bounding boxes will be positioned. More specifically, the positioning data can specify locations within the canvas at which the bounding box of each given unit of content are located.

[0025] The generation / creation of the layout can also include the assignment of each bounding box to a user interface layer. As discussed in more detail later, each layer of a user interface dictates which assets will be overt aid / placed over other assets. For example, an asset that is presented in a bounding box having a deeper layer will be overlaid with an asset presented in a bounding box having a less deep layer (e.g., assuming the bounding boxes overlap in position on the canvas.

[0026] The Al system represents the layout by generating a text representation of the bounding boxes, positioning data, user interface layer, and / or any other formatting generated by the Al system for the assets. As discussed in more detail below, the text representation can be generating using a compressed token representation to reduce the latency of the language model of the Al system.

[0027] The Al system is configured to render the two or more discrete units of content for display according to the new layout. For example, the Al system can interpret the textual representation, and create a visual rendering of the two or more discrete units of content according to the layout. In this way, the layout of the two or more discrete units of content are visually perceptible, and available for presentation to the content provider or a user of a client device to whom the new layout is presented. For example, the layout can be presented to the content provider for approval. The layout can also be stored, and the two or more discrete units of content can be presented according to the layout when content is delivered to a client device, for example, as discussed below with reference to FIG. 1.

[0028] FIG. 1 is a block diagram of an example environment 100 in which generative artificial intelligence can be implemented. In this specification, the generative artificial intelligence is configured to efficiently generate visually appealing layouts for a set of content assets. The example environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and a service apparatus 110. The example environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0029] A client device 106 is an electronic device capable of requesting and receiving online resources over the network 102. Example client devices 106 include personal computers, gaming devices, mobile communication devices, tablet devices, digital assistant devices, augmented reality devices, virtual reality devices, wearable devices, and other devices that can send and receive data over the network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate the sending and receiving of data over the network 102, but native applications (other than browsers) executed by the client device 106 can also facilitate the sending and receiving of data over the network 102.

[0030] A gaming device is a device that enables a user to engage in gaming applications, for example, in which the user has control over one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (either physical or visually rendered) that enables user control over content rendered by the gaming application. Thegaming device can store and execute the gaming application locally, or execute a gaming application that is at least partly stored and / or served by a cloud server (e g., online gaming applications). Similarly, the gaming device can interface with a gaming server that executes the gaming application and “streams” the gaming application to the gaming device. The gaming device may be a tablet device, mobile telecommunications device, a computer, or another device that performs other functions beyond executing the gaming application.

[0031] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input by way of voice, and respond with content using audible feedback, and can present other audible information. In some situations, digital assistant devices also include a visual display or are in communication with a visual display (e.g., by way of a wireless or wired connection).Feedback or other information can also be provided visually when a visual display is present. In some situations, digital assistant devices can also control other devices, such as lights, locks, cameras, climate control devices, alarm systems, and other devices that are registered with the digital assistant device.

[0032] As illustrated, the client device 106 is presenting an electronic document 150. An electronic document is data that presents a set of content at a client device 106. Examples of electronic documents include webpages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e g., “apps” and / or gaming applications), such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents can be provided to client devices 106 by electronic document servers 104 (“Electronic Doc Servers”).

[0033] For example, the electronic document servers 104 can include servers that host publisher websites. In this example, the client device 106 can initiate a request for a given publisher webpage, and the electronic server 104 that hosts the given publisher webpage can respond to the request by sending machine executable instructions that initiate presentation of the given webpage at the client device 106.

[0034] In another example, the electronic document servers 104 can include app servers from which client devices 106 can download apps. In this example, the client device 106 can download fdes required to install an app at the client device 106, and then execute thedownloaded app locally (i.e., on the client device). Alternatively, or additionally, the client device 106 can initiate a request to execute the app, which is transmitted to a cloud server. In response to receiving the request, the cloud server can execute the application and stream a user interface of the application to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 can present the user interface generated by the cloud server’s execution of the app, and communicate any user interactions with the user interface back to the cloud server for processing.

[0035] Electronic documents can include a variety of content. For example, an electronic document 150 can include native content 152 that is within the electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that may change over time or on a per-request basis. For example, a publisher of a given electronic document (e.g., electronic document 150) can maintain a data source that is used to populate portions of the electronic document. In this example, the given electronic document can include a script, such as the script 154, that causes the client device 106 to request content (e.g., a digital component) from the data source when the given electronic document is processed (e.g., rendered or executed) by a client device 106 (or a cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital component) obtained from the data source into the given electronic document to create a composite electronic document including the content obtained from the data source.

[0036] In some situations, a given electronic document (e.g., electronic document 150) can include a digital component script (e.g., script 154) that references the service apparatus 110, or a particular service provided by the service apparatus 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for digital components 112 (referred to as a “component request”), which is transmitted over the network 102 to the service apparatus 110. For example, the digital component script can enable the client device 106 to generate a packetized data request including a header and payload data. The component request 112 can include event data specifying features such as a name (or network location) of a server from which the digital component is being requested, a name (or network location) of the requesting device (e.g., the client device 106), and / or information that theservice apparatus 110 can use to select one or more digital components, or other content, provided in response to the request. The component request 112 is transmitted, by the client device 106, over the network 102 (e.g., a telecommunications network) to a server of the service apparatus 110.

[0037] The component request 112 can include event data specifying other event features, such as the electronic document being requested and characteristics of locations of the electronic document at which digital component can be presented. For example, event data specifying a reference (e.g., URL) to an electronic document (e.g., webpage) in which the digital component will be presented, available locations of the electronic documents that are available to present digital components, sizes of the available locations, and / or media types that are eligible for presentation in the locations can be provided to the service apparatus 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities (e.g., people, places, or things) that are referenced by the electronic document can also be included in the component request 112 (e.g., as payload data) and provided to the service apparatus 110 to facilitate identification of digital components that are eligible for presentation with the electronic document. The event data can also include a search query that was submitted from the client device 106 to obtain a search results page.

[0038] Component requests 112 can also include event data related to other information, such as information that a user of the client device has provided, geographic information indicating a state or region from which the component request was submitted, or other information that provides context for the environment in which the digital component will be displayed (e.g., a time of day of the component request, a day of the week of the component request, a type of device at which the digital component will be displayed, such as a mobile device or tablet device). Component requests 112 can be transmitted, for example, over a packetized network, and the component requests 112 themselves can be formatted as packetized data having a header and payload data. The header can specify a destination of the packet and the payload data can include any of the information discussed above.

[0039] The service apparatus 110 chooses digital components (e.g., third-party content, such as video fdes, audio files, images, text, gaming content, augmented reality content, and combinations thereof, which can all take the form of advertising content or non-advertisingcontent) that will be presented with the given electronic document (e.g., at a location specified by the script 154) in response to receiving the component request 112 and / or using information included in the component request 112.

[0040] In some implementations, a digital component is selected in less than a second to avoid errors that could be caused by delayed selection of the digital component. For example, delays in providing digital components in response to a component request 112 can result in page load errors at the client device 106 or cause portions of the electronic document to remain unpopulated even after other portions of the electronic document are presented at the client device 106.

[0041] Also, as the delay in providing the digital component to the client device 106 increases, it is more likely that the electronic document will no longer be presented at the client device 106 when the digital component is delivered to the client device 106, thereby negatively impacting a user's experience with the electronic document. Further, delays in providing the digital component can result in a failed delivery of the digital component, for example, if the electronic document is no longer presented at the client device 106 when the digital component is provided.

[0042] In some implementations, the service apparatus 110 is implemented in a distributed computing system that includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and distribute digital component in response to requests 112. The set of multiple computing devices 114 operate together to identify a set of digital components that are eligible to be presented in the electronic document from among a corpus of millions of available digital components (DCi-x). The millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference the corresponding digital component and / or include distribution parameters (DPi-DPx) that contribute to (e.g., trigger, condition, or limit) the distribution / transmission of the corresponding digital component. For example, the distribution parameters can contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or with some pre-specified level of similarity) one of the distribution parameters of the digital component.

[0043] In some implementations, the distribution parameters for a particular digital component can include distribution keywords / topics / categories that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) in order for the digital component to be eligible for presentation. Additionally, or alternatively, the distribution parameters can include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewport, user scrolling speed, or other information about the consumption of data). The distribution parameters can also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and / or information specifying that the component request 112 originated at a particular type of client device (e.g., mobile device or tablet device) in order for the digital component to be eligible for presentation. The distribution parameters can also specify an eligibility value (e.g., ranking score, or some other specified value) that is used for evaluating the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0044] The identification of the eligible digital component can be segmented into multiple tasks 117a-l 17c that are then assigned among computing devices within the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze a different portion of the digital component database 116 to identify various digital components having distribution parameters that match information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res 1-Res 3) 118a- 118c of the analysis back to the service apparatus 110. For example, the results 118a-l 18c provided by each of the computing devices in the set 114 may identify a subset of digital components that are eligible for distribution in response to the component request and / or a subset of the digital component that have certain distribution parameters. The identification of the subset of digital components can include, for example, comparing the event data to the distribution parameters, and identifying the subset of digital components having distribution parameters that match at least some features of the event data.

[0045] The service apparatus 110 aggregates the results 118a- 118c received from the set of multiple computing devices 114 and uses information associated with the aggregatedresults to select one or more digital components that will be provided in response to the request 112. For example, the service apparatus 110 can select a set of winning digital components (one or more digital components) based on the outcome of one or more content evaluation processes, as discussed below. In turn, the service apparatus 110 can generate and transmit, over the network 102, reply data 120 (e.g., digital data representing a reply) that enable the client device 106 to integrate the set of winning digital components into the given electronic document, such that the set of winning digital components (e.g., winning third- party content) and the content of the electronic document are presented together at a display of the client device 106.

[0046] In some implementations, the client device 106 executes instructions included in the reply data 120, which configures and enables the client device 106 to obtain the set of winning digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 can include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device 106 to transmit a server request (SR) 121 to the digital component server 108 to obtain a given winning digital component from the digital component server 108. In response to the request, the digital component server 108 will identify the given winning digital component specified in the server request 121 (e.g., within a database storing multiple digital components) and transmit, to the client device 106, digital component data (DC Data) 122 that presents the given winning digital component in the electronic document at the client device 106.

[0047] When the client device 106 receives the digital component data 122, the client device will render the digital component (e.g., third-party content), and present the digital component at a location specified by, or assigned to, the script 154. For example, the script 154 can create a walled garden environment, such as a frame, that is presented within, e.g., beside, the native content 152 of the electronic document 150. In some implementations, the digital component is overlay ed over (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service apparatus 110 can specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service apparatus 110 can specify a location or object within the scene depicted in the video content over which the digital component is to be presented.

[0048] The service apparatus 1 10 can also include an artificial intelligence system 160 configured to autonomously generate new layouts for sets of assets. The service apparatus 110 can also be configured to autonomously generate digital components and / or layouts for digital components, either prior to a request 112 (e.g., offline) and / or in response to a request 112 (e.g., online or real-time). The artificial intelligence (“Al”) system 160 can collect online content about a specific entity (e.g., digital component provider or another entity) and summarize the collected online content using one or more language models 170, which can include large language models of various sizes (e g., having a different number of parameters). Note that the language model 170 is depicted as being separate from the service apparatus 110 and the Al system 160 for purposes of illustration, but the language model 170 can be integrated into the service apparatus 110 and / or the Al system 160.

[0049] A large language model (“LLM”) is a model that is trained to generate and understand human language. LLMs are trained on massive datasets of text and code, and they can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text, such as web site content, search results, news articles, or research papers; answer questions about text, such as “What is the capital of Georgia?”; create chatbots that can have conversations with humans; and generate creative text, such as poems, stories, recommendations, and code.

[0050] The category of “large language models” can vary in size. Larger models generally use a larger number of parameters than smaller models, which enables the larger models to complete a wider range of tasks and generate more accurate responses to input. For example, because of the larger number of parameters, the larger language models are generally configured to understand the context of the input more effectively, and to generate more detailed responses than smaller models that use a smaller number of parameters.

[0051] Although larger language models are generally more flexible and more accurate than smaller models, these additional capabilities also require significantly more computational resources, including more powerful and / or a larger number of graphics processing units (GPUs) or tensor processing units (TPUs) to train and deploy (e.g., operate) than smaller language models. For example, a larger language model can require 20 seconds or more to generate a response to an input, while a smaller language model can be configured to generate a response in less than 5 seconds. Furthermore, larger language models canrequire significantly more memory to train and / or deploy. As described in more detail throughout this specification, a smaller language model (e.g., a medium language model having a standard latency of about 10 seconds) can be used with compression techniques that reduce the latency of the model, for example, by reducing the amount of time required for inference as well as reducing the number of tokens that are required to be generated.

[0052] The language model 170 can be any appropriate language model neural network that receives an input sequence made up of text tokens selected from a vocabulary and auto- regressively generates an output sequence made up of text tokens from the vocabulary. For example, the language model 170 can be a Transformer-based language model neural network or a recurrent neural network-based language model. The language model 170 can also be a multimodal language model that also accepts images, videos, or other media, and is configured to analyze / understand contents of the images, videos, or other media in addition to text input.

[0053] In some situations, the language model 170 can be referred to as an autoregressive neural network when the neural network used to implement the language model 170 auto-regressively generates an output sequence of tokens. More specifically, the auto- regressively generated output is created by generating each particular token in the output sequence conditioned on a current input sequence that includes any tokens that precede the particular text token in the output sequence, i.e., the tokens that have already been generated for any previous positions in the output sequence that precede the particular position of the particular token, and a context input that provides context for the output sequence.

[0054] For example, the current input sequence when generating a token at any given position in the output sequence can include the input sequence and the tokens at any preceding positions that precede the given position in the output sequence. As a particular example, the current input sequence can include the input sequence followed by the tokens at any preceding positions that precede the given position in the output sequence. Optionally, the input and the current output sequence can be separated by one or more predetermined tokens within the current input sequence.

[0055] More specifically, to generate a particular token at a particular position within an output sequence, the neural network of the language model 170 can process the current input sequence to generate a score distribution (e.g., a probability distribution) that assigns arespective score, e g., a respective probability, to each token in the vocabulary of tokens. The neural network of the language model 170 can then select, as the particular token, a token from the vocabulary using the score distribution. For example, the neural network of the language model 170 can greedily select the highest-scoring token or can sample, e.g., using nucleus sampling or another sampling technique, a token from the distribution.

[0056] As a particular example, the language model 170 can be an auto-regressive Transformer-based neural network that includes (i) a plurality of attention blocks that each apply a self-attention operation and (ii) an output subnetwork that processes an output of the last attention block to generate the score distribution.

[0057] The language model 170 can have any of a variety of Transformer-based neural network architectures. Examples of such architectures include those described in J.Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al. Training compute-optimal large language models, arXiv preprint arXiv:2203.15556, 2022; J.W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, H. F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van den Driessche, L. A. Hendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Higgins, A. Creswell, N. McAleese, A.Wu, E. Eisen, S. M. Jayakumar, E. Buchatskaya, D. Budden, E. Sutherland, K. Simonyan, M. Paganini, L. Sifre, L. Martens, X. L. Li, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mensch, J. Lespiau, M. Tsimpoukelli, N. Grigorev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d’Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, B. A. Hechtman, L. Weidinger, I. Gabriel, W. S. Isaac, E. Lockhart, S. Osindero, L. Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs / 2112.11446, 2021; Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019; Daniel Adiwardana, Minh- Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. Towards a human-like open-domain chatbot. CoRR, abs / 2001 .09977, 2020; and Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020; Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, Karen Simonyan. Flamingo: A Visual Language Model for Few-Shot Learning. arXiv preprint arXiv:2204.14198, 2022.

[0058] Generally, however, the Transformer-based neural network includes a sequence of attention blocks, and, during the processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention block then updates each of the hidden states at least in part by applying self-attention to generate a respective output hidden state for each of the input tokens. The input hidden states for the first attention block are embeddings of the input tokens in the input sequence and the input hidden states for each subsequent attention block are the output hidden states generated by the preceding attention block.

[0059] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence for the last input token in the input sequence to generate the score distribution.

[0060] Generally, because the language model is auto-regressive, the service apparatus 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, e.g., by using beam search decoding from score distributions generated by the language model 170, using a Sample-and-Rank decoding strategy, by using different random seeds for the pseudo-random number generator that’s used in sampling for different runs through the language model 170 or using another decoding strategy that leverages the auto-regressive nature of the language model.

[0061] In some implementations, the language model 170 is pre-trained, i.e., trained on a language modeling task that does not require providing evidence in response to user questions, and the service apparatus 110 (e g., using Al system 160) causes the languagemodel 170 to generate output sequences according to the pre-determined syntax through natural language prompts in the input sequence.

[0062] For example, the service apparatus 110 (e.g., Al system 160), or a separate training system, pre-trains the language model 170 (e.g., the neural network) on a language modeling task, e g., a task that requires predicting, given a current sequence of text tokens, the next token that follows the current sequence in the training data and / or a coding task. As a particular example, the language model 170 can be pre-trained on a maximum-likelihood objective on a large dataset of text, e g., text that is publicly available from the Internet or another text corpus.

[0063] The language model 170 used to generate the layouts as described in more detail below can be trained, for example, on a combination of pre-existing corpus’ as well as newly generated content based on available assets (e.g., text, images, videos, logos, etc ). For example, the service apparatus 110 (e.g., using the Al system 160) can generate various combinations of available assets as additional training data for the language model 170.

[0064] In some implementations, at least some of the training data for the language model 170 that generates new layouts can be obtained from a content repository. For example, the process can start with a dump of assets (text and images) from historical requests. In some situations, the image assets may be pointers to the raw image bytes. In these situations, the first step is to add the raw image bytes to the dump. The content repository is then re-called with the historical request data to get fresh rendering results.

[0065] The results can be provided as a Lottie proto which the service apparatus 110 (or another data processing apparatus) can convert to a layout proto. Each Lottie layer maps 1 : 1 to a layout proto layer. The layout proto generated by the service apparatus 110 has a reduced set of features and is more compressed in its representation than a Lottie layer. For example, the layout proto can support fewer geometric shapes (e.g., only rectangles and ellipses in some implementations) than Lottie supports. An even further compressed, and less human readable version of the layout proto is used to train the model to reduce inference time latency. Further optimizations to the layout proto that can be used to reduce latency include compression, including the shortening of proto fields, representing truncated floats as integers, and / or subtracting the modal value from numeric fields to increase the percentage of zero-valued fields in the final proto.

[0066] The training data used to train the language model 170 can be tokenized. For example, the raw image bytes can be replaced with corresponding tokens using a tokenizer similar to that described in Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu. CoCa: Contrastive Captioners are Image-Text Foundation Models. arXiv preprint - arXiv:2205.01917, which is incorporated by reference herein in its entirety. The tokenized data can then be split into a training set and a test set, and used to train the language model 170.

[0067] Once trained, the language model 170 takes as input a prompt 172 and various assets. In some implementations, the assets include a logo image, product image, text headline, text description and call to action (CTA) text and outputs. The output 174 of the language model 170 can be a debug string of the layout proto. The debug string is representation of the layout of the assets (e.g., a user interface or digital component). Generally speaking, the debug string of a layout proto provides a textual representation of the layout hierarchy that developers can read and analyze to understand how the UI is structured. It typically includes information about the types of UI elements (e.g., TextView, Button, LinearLayout), their attributes (e.g., width, height, margins), and their relationships within the hierarchy (e.g., parent-child relationships). As mentioned above, in some implementations, the debug string output by the language model 170 can have compressed / truncated textual representations of the layout hierarchy to reduce the number of tokens required to be generated by the language model 170, thereby reducing the latency of the language model 170 relative to generating the full text descriptions of the layout hierarchy.

[0068] FIG. 2 is a block diagram 200 illustrating the creation of a layout 201 using the language model 170. The task of generating the layout 201 using the language model is approached as a language modeling problem, where the representation of the layout 201 is expressed as a sequence of discrete tokens. For example, the output of the language model 170 can be a series of tokens that define bounding boxes, asset identifiers of assets presented within the bounding boxes, styling applied to the assets, and / or other visual aspects of the assets in the layout 201.

[0069] In operation, the language model 170 obtains / receives a prompt 202 and a set of assets 204. In this example, the set of assets 204 includes a logo 206 of an entity for whichthe layout 201 is being generated, an image 208 that will be presented in the layout 201 , headline text 210 that will be presented in the layout 201, and call to action (CTA) text 212 that will be presented over a CTA element specified / selected by the language model 170.

[0070] The prompt 202 generally instructs, guides, and / or constrains the output generated by the language model 170. For example, the prompt can convey information such as the size of the canvas available for arranging the set of assets 204, how the output of the language model should be formatted, and / or an entity specific style profile. In some implementations, the prompt can also identify the set of assets 204 for which the language model 170 is generating the layout 201. The prompt can take many forms, but in some implementations, the prompt 202 can take the following form:Please prepare a width: 4390 x height: 7680 layout for the following assets:HEADLINE: Your Favorite Spot;LOGO: <image>;IMAGE 0: <image>;CTA TEXT: Reserve Today.The output should be a single sentence in protocol buffer debug string format.

[0071] In this example, the prompt 202 specifies that the size of the canvas for arranging the assets is 4390 pixels by 7680 pixels. The prompt 202 also specifies the assets that will be arranged in the layout. For example, the headline “Your Favorite Spot” will be presented in the layout 201, as will a logo image.

[0072] The prompt 202 also identifies the CTA text, which is the text of and / or associated with a CTA element. The CTA element can be any graphical user interface (GUI) element that is configured to invoke an action in response to interaction with the CTA element. For example, the CTA element can be a button, a slider, a drop down, or any other GUI element. In this example, the CTA text is “Reserve Today.” Finally, the example prompt 202 specifies that the output of the language model should be formatted in a single sentence, and in protocol (proto) buffer debug string format. As discussed above, the language model 170 can be configured (e.g., trained) to generate a layout proto debug string format that has a compressed representation of the layout, thereby reducing the number of tokens the language model 170 must generate, which reduces the latency of the language model 170.

[0073] When entity specific style information is available (e.g., in a style profile), the language model can be informed of that entity specific style information by way of the prompt 202. The following is an example prompt that conveys entity specific style information of the entity for which the layout 201 is being generated.Please prepare a width: 4390 x height: 7680 layout for the following assets:HEADLINE: Your Favorite Spot;LOGO: <image>;IMAGE 0: <image>;CTA TEXT: Reserve Today. You may use these fonts: NotoSans-Regular You may use these colors: r: 114, g: 659, b: 886, a: 0; r: 0, g: 0, b: 0, a: 0; r: 969, g: 820, b: 757, a: 0 The output should be a single sentence in protocol buffer debug string format.

[0074] In this example prompt, the language model is instructed to use specific combinations of colors and a specific font when generating the layout 201. Of course, other styling information can be included as well.

[0075] In some implementations, the service apparatus 110 can determine the entity specific style information based on content previously generated by the entity. For example, the service apparatus 110 can identify a URL of an entity, and extract, from that URL, various visual features, such as text features, image features, contextual information, logos, etc. Furthermore, metadata associated with the URL, such as labels specifying a category of the content and / or a characterization of the content (e.g., outdoors, rugged, refined, calm, or other characterizations) can be obtained by the service apparatus 110. In this way, the service apparatus 110 (including, for example, the language model 170 and / or the Al system 160) can use this information to guide the creation of layouts for the entity. For example, this type of entity specific style information can be used by the language model 170 to select colors, visual effects, geometric areas in which assets will be presented, fonts, etc. for the layouts generated by the language model 170.

[0076] In some implementations, the set of assets 204 can be extracted from an online location (e.g., URL) of the entity for which the layout 201 is being generated. For example, one or more of the logo 206 of an entity for which the layout 201 is being generated, the image 208 that will be presented in the layout 201, the headline text 210 that will be presented in the layout 201, and / or the call to action (CTA) text 212 that will be presentedover a CTA element can be extracted from a website of the entity (e.g., content provider). In some implementations, the set of assets 204 can be uploaded by the entity for which the layout 201 is being generated. For example, the logo 206 of an entity for which the layout201 is being generated, the image 208 that will be presented in the layout 201, the headline text 210 that will be presented in the layout 201, and / or the call to action (CTA) text 212 that will be presented over a CTA element can be uploaded to the service apparatus 110 by the entity (e.g., content provider), and made available to the language model 170.

[0077] When the set of assets 204 are input to the language model 170, the language model will evaluate the set of assets 204 in the context of the prompt 202 to output the layout 201. For example, the language model 170 will use the instructions contained in the prompt202 as guidelines for and / or restraints on the creation of the layout 201.

[0078] As previously discussed, the language model 170 outputs the layout 170 in a protocol buffer debug string format, which is a sequence of tokens that are generated by the language model 170. In this way, the language model 170 can specify the details of the layout 201, also referred to as a new layout, in a text format / representation. As previously discussed, the text representation generated by the language model 170 can utilize a compressed token representation to reduce the number of tokens that the language model 170 must generate to represent the new layout 201, thereby reducing the latency (e.g., time required) to generate / output the new layout 201. Although the layout 201 is output as a text representation, FIG. 2 graphically depicts the layout 201 for purposes of example, and to visually illustrate aspects of the layout 201.

[0079] In the process of generating the new layout, the language model uses the set of assets 204 (e.g., a set of two or more discrete units of content) as input to determine the appropriate spatial arrangement (positioning), layer assignment, size, and style (e.g., color, font, opacity, etc.) for the set of assets 204. In some implementations, the language model 170 generates a bounding box for each of the assets in the set of assets 204. For example, in the layout 201, the language model 170 has generated a headline bounding box 216 in which the headline text 210 will be presented, a logo bounding box 218 in which the logo 206 will be presented, an image bounding box 220 in which the image 208 will be presented, a CTA element bounding box 222 in which the CTA element will be presented, and a CTA text bounding box 224 in which the CTA text 214 will be presented. As such, each of thebounding boxes 216-224 are presentation spaces for each of the individual assets (e.g., discrete units of content) in the set of assets 204.

[0080] The language model 170 can be configured to generate the layout 201 in a content-aware manner. In some implementations, the language model 170 can perform semantic analysis of the set of assets 204 to understand the contents of the assets. For example, the language model 170 can evaluate the image 208 to identify what is depicted by the image 208, which can inform where to place the image, text, etc. In a specific example, assume that the image 208 depicts a person. In this example, the language model 170 can identify the person in the image 208, and where the person’s head is depicted. Using this information, the language model 170 can determine that the person’s head should be visible when placed in the layout 201 and / or that other assets should not be placed over the location where the person’s head will be depicted in the layout 201. In this way, the language model can avoid occluding focal points or cluttering the assets in the layout 201.

[0081] As part of the layout generation process, the language model 170 can also assign each of the bounding boxes to a corresponding user interface layer among multiple user interface layers. As used herein, a layer in a user interface can be a distinct visual plane or container that holds assets among the set of assets 204. For example, each of the headline 210, image 208, CTA element (e.g., a button), CTA text 212, and / or logo 206 can be assigned to a corresponding UI layer. Layers are used in graphical user interfaces (GUIs) to organize and manage the presentation of content, interactions, and information.

[0082] Layers help organize the set of assets 204 into logical groups, making it easier to manage and manipulate them as a single entity. Additionally, layers can be arranged in a hierarchical structure, with some layers nested within others. This allows for more complex UI designs where elements can be stacked and ordered according to their importance or relationship. Furthermore, layers can have properties that control their visibility and opacity, allowing them to be shown or hidden and adjusted for transparency effects. For example, when two bounding boxes overlap, as shown with respect to the logo bounding box 218 and the image bounding box 220, they can be placed in different UI layers that are each assigned independent levels of opacity. In this example, assume that layer 0 is the base layer of the UI layers, and that each layer having a higher number (e.g., 1, 2, 3, etc.) is in a visual plane that is placed over / on top of lower numbered layers. Further assume that each layer has 100%opacity (e.g., no transparency). In this example, assets in higher level (e.g., less deep) layers of the UI will occlude assets that are (i) in lower level (e.g., deeper) layers of the UI and overlapped by the assets in the higher-level layers. Now, assume that the opacity of the higher-level layer is reduced. In this situation, the assets in the lower / deeper layers of the UI will become more visible as the opacity is decreased (e.g., transparency increased). As such, the language model 170 can generate different visual effects by assigning different assets to different UI layers, and manipulating the opacity of the UI layers. Each layer can also have its own styling and formatting properties, such as background color, border, padding, and margin, which enables the language model to further control the visual appearance of the layer and / or assets within the layout 201.

[0083] In some implementations, the service apparatus 110, language model 170, or another data processing apparatus can manipulate the set of assets to fit in the canvas and / or layers available for the layout 201. For example, background content can be removed from images, image resolution can be adjusted (e.g., increased or decreased), and / or resized to fit in bounding boxes to which the assets are assigned. Furthermore, the layer transparency to which the assets are assigned can be adjusted as discussed above.

[0084] In FIG. 2, the CTA text bounding box 224 overlaps the CTA element bounding box 222. As such, the CTA text 212 will be presented over the CTA element when the set of assets 204 are rendered according to the layout 201. If these two bounding boxes (e.g., 222 and 224) are assigned to different UI layers, the visual characteristics of the manner in which the CTA text 214 will appear over the CTA element can be changed by the language model 170 by adjusting the properties of the different UI layers. These properties of the different UI layers can be specified by the textual representation of the layout 201 output by the language model.

[0085] During the layout generation process, the language model 170 also generates positioning data specifying locations within the canvas (e.g., user interface) at which each of the bounding boxes will be located / positioned. For example, when using rectangular bounding boxes, the language model 170 can generate positioning data specifying a pair of coordinates for opposite corners (e.g., diagonal comers) of the rectangle to define the location of the rectangle in the canvas. More specifically, the language model 170 can specify the xmin, Ymin and Xmax, ymax points of the rectangle bounding box, which define thelocation of the bounding box. The point defined by the Xmin, Ymin coordinates will be the lower left corner of the rectangle, and point defined by the Xmax, ymax coordinates will be the top right corner of the rectangle. Once these two corners of the rectangle are defined, the other two corners of the rectangle are readily apparent (e.g Xmin, ymax & Xmax, ymin), such that the, such that they don’t necessarily need to be specified in tokens generated by the language model 170. Rather, the language model 170 can generate a sequence of discrete tokens specifying the different bounding boxes (e.g., using different identifiers for each bounding box), an identifier of the asset that will occupy each of the bounding boxes, layers to which each of the bounding boxes have been assigned, and the positioning data specifying the locations (e.g., coordinates) of the two diagonally opposite corners of the rectangle. Of course, if different geometric shapes are used for the bounding boxes, different positioning data can be used to specify the locations of those bounding boxes in the canvas.

[0086] Another advantage of outputting the layout in the form of a textual representation is that the layout is directly / fully editable by adjusting the text output by the language model 170. For example, assume that the entity for which the layout 201 is being generated would like to move the location (or layer) of the logo bounding box 218 in the layout 201. In this example, the entity could simply adjust the positioning data (or layer assignment) in the textual representation output by the language model 170. Similarly, all other aspects of the layout 201 are represented textually, such that any of the aspects of the layout 201 can similarly be adjusted by adjusting the textual representation output 170. This is an advantage over conventional user interface layout generation tools / techniques because those conventional tools / techniques are not modifiable, rather they are fixed in nature. As such, there is no opportunity to simply adjust a subset of aspects of the layout without regenerating a new layout, which is more time consuming and more resource intensive than being able to simply adjust the text representation of the layout 201 as is possible with the current system.

[0087] Once the textual representation of the new layout 201 is output by the language model 170, it can be passed (e.g., transmitted) to the service apparatus 110 (or another data processing apparatus). Upon receipt, the service apparatus 110 renders the set of assets 204 according to the textual representation of the layout 201 generated by the language model 170. Once rendered, the service apparatus 110 can present the set of assets 204 according tothe layout 201 . For example, the rendered content 230 can be presented to the entity at the client device 106.

[0088] As shown in the rendered content 230, the set of assets are arranged according to the layout 201. For example, the headline 210 “Your Favorite Spot” is presented in the location of the headline bounding box 216, the logo 206 is presented in the location of the logo bounding box 218, and overlaid on the image 208 of two people sitting in beach chairs at the beach, which is presented at the location of the image bounding box 220. Finally, the CTA element, which is depicted as a button, is positioned at the location of the CTA element bounding box 222 and overlaid with the CTA text 212 “Reserve Today,” which is presented at the location of the CTA text bounding box 224.

[0089] In some implementations, the language model 170 is configured to generate the parameters of the layout 201 without generating new assets to populate (e.g., present in) the layout 201. For example, the assets presented in the layout 201 can be limited to those assets uploaded by the entity for which the layout was generated and / or assets extracted from resources (e.g., web pages) of the entity. In this way, the systems described herein can prevent model hallucinations that could otherwise occur. For example, the nature of generative language models can lead these models to generate information that is not accurate and / or false. However, by limiting the assets that populate the layout 201 to content that was supplied by and / or obtained from content provided by the entity for which the layout 201 is generated, the language model 170 is not generating any new assets that will populate the layout 201, such that assets presented in the layout will not suffer from model hallucinations. Of course, the styling of the set of assets 204 can be adjusted as discussed above, but this does not constitute the generation of new text or images that are presented in the layout 201. In some implementations, the textual representation can include pointers to the content assets that enable the service apparatus 110 and / or client device 106 to retrieve the content assets that are used to populate the layout 201.

[0090] FIG. 3 is a flow chart of an example process 300 for generating a new layout for a set of assets (e.g., two or more discrete units of content). Operations of the process 300 can be performed, for example, by the service apparatus 110 of FIG. 1 (e.g., including the Al system 160 and / or language model 170), or another data processing apparatus. The operations of the process 300 can also be implemented as instructions stored on a computerreadable medium, which can be non-transitory. Execution of the instructions, by one or more data processing apparatus, cause the one or more data processing apparatus to perform operations of the process 300. Note that the process 300 depicts operations performed prior to use of a language model to generate a new layout (e.g., training operations and layout personality generation, which is also referred to as an entity specific style profile) as well as operations performed to generate a new layout using the language model. It should be understood that not all operations depicted by FIG. 3 are essential or required, subsets of (e.g., fewer than all) operations can be implemented without performing other operations.

[0091] A set of training data is extracted from one or more content sources (302). The set of training data can be extracted, for example, from online repositories of user interfaces. For example, previously generated arrangements / layouts of text, images, videos, animations, and / or user interface elements (e.g., buttons, drop downs, sliders, or other user interface elements) can be extracted from a central repository of such content, or collected from various online locations. In some implementations, the set of training data that is extracted is in a form output by a user interface development tool, and the set of training data can include information such as the relative locations of each discrete unit of content presented, a UI layer to which each discrete unit of content is assigned, and / or formatting / styling information for each discrete unit of content.

[0092] One or more data synthesis operations are performed (304) In some implementations, the data synthesis operations are performed to generate additional training data beyond that extracted from the one or more content sources. For example, as previously discussed, various combinations of text, images, video, user interface elements can be combined in different ways to create variations of training samples from the set of training data. For example, different combinations of the discrete units of content in the extracted training samples can be combined into new training samples. Similarly, the arrangement of the discrete units of content in the extracted training samples can be rearranged, resized, restyled / reformatted, etc. to obtain additional training samples that differ from the training samples in the set of training data extracted from the one or more content sources. As such, at least one of the data synthesis operations includes creating variations of training samples from the set of training data extracted from the one or more content sources.

[0093] A language model is trained based on the set of training data and / or the additional training data (306). The language model can be trained as previously discussed, or in any appropriate manner. For brevity, the discussion of training the language model is not repeated here, but the result of the training is a language model that is trained / configured to accept, as input, a two or more discrete units of content (e.g., a set of assets), and output a new layout specifying a spatial arrangement of those discrete units of content on a canvas and / or formatting / styling to be applied to the discrete units of content.

[0094] A set of content is extracted from a resource at a uniform resource locator of a content provider (308). The set of content can be content presented on a resource located at the uniform resource locator, which can reference a second level domain of a content provider (e.g., entity for which a new layout will be generated). For example, a website of a content provider can be accessed, and the content presented on the web pages of that website can be extracted, along with formatting information.

[0095] Features of the set of content are evaluated (310). In some implementations, the evaluation of the features (e.g., visual features) is performed to obtain layout preferences of the content provider. For example, the text, images, logos, and other types of content presented by the set of content can be evaluated to identify / infer one or more of text preferences (e.g., font style, font size, text color, etc.), image preferences (e.g., greyscale, color, applied fdters, image sizes, etc.), logo preferences (e.g., logo size, color scheme, logo location, etc ), and / or font preferences for different types of text (e.g., bold, font type, etc. for headlines versus descriptive text). In some implementations, the preferences of the content provider can be performed based on how frequently / infrequently aspects of the content is formatted in specific ways.

[0096] A layout personality of the content provider is generated based on the layout preferences (312). In some implementations, the prevailing preferences for each type of content (e.g., text, images, etc.) can be aggregated into a layout personality for the content provider. For example, assume that the evaluated content of the content provider revealed that the content provider prefers greyscale images, text in NotoSans-Regular with a 12-point font for descriptive text and 18-point font for headlines, with a logo placed in the middle third of the canvas. In this example, the layout personality for this content provider could reflect these preferences. In other implementations, different layout personalities can begenerated for a single content provider. For example, a different layout personality could be generated for different contexts, such as uses of the content. In this way, the layout personality for the content provider can be specified / inferred on a per-use-type basis depending on how the content is used.

[0097] In some implementations, the generated layout personality can include a set of style tags that were assigned to previous layouts of the content provider. For example, an evaluation of the previous layouts of content generated by, provided by, or presented on resources of the content provider can be tagged with style information, such as relaxing, rugged, outdoorsy, adventurous, or other style characterizations. In this way, combinations of formatting / styling applied to content of the content provider can be characterized in a more general sense, and those style tags can be used to guide / constrain a language model to generate a layout that is consistent with those style tags.

[0098] Two or more discrete units of content are obtained (314). In other words, a set of assets are obtained. As previously discussed, the two or more discrete units of content can be obtained form a website (or another online resource) of the content provider (e.g., entity) for whom a new layout will be generated. Alternatively, or additionally, the two or more discrete units of content can be uploaded, or otherwise provided by, the content provider.The two or more discrete units of content can include, for example, sets of text, images, interactive user interface elements (e.g., buttons), videos, or other forms / sets of content. In some implementations, the purpose / type of content can be used to distinguish between discrete units of content. For example, when two sets of text are obtained together and / or from a same source, they can be distinguished as discrete units of text based on descriptor data specifying that one set of text is a headline, while another set of text is description text. Meanwhile, different types of content (e g., image versus text) are considered to be discrete units of text. Units of text can also be considered discrete by being stored as different files.

[0099] A new layout of the two or more discrete units of content (e.g., set of assets) is generated (316). In some implementations, the new layout is generated within a canvas. As previously discussed, the canvas is the space available for arranging the two or more discrete units of content within a user interface. The canvas can be a full size of the user interface that will be presented to a user, or a subsection (e g., a slot in which the resulting layout of the discrete units of content will be presented). In some implementations, the canvas size isspeci fi ed / dictated / con strained by the content provider for whom the new layout will be generated (e.g., a canvas of an application user interface or a website user interface). In some implementations, the canvas size is specified / dictated / constrained by an amount of space available in a third-party resource / application. For example, in situations where the content provider for whom the layout is being generated is creating content (e.g., a public service announcement or an advertisement) for presentation in a third-party application / website, the amount of space available for presenting the new layout of the two or more discrete units of content is specified by the third-party. In these situations, the canvas size can be the amount of space specified by the third-party.

[0100] In some implementations, a layout personality of the content provider is obtained in addition to the two or more discrete units of content. As previously discussed, the layout personality can include a set of style tags assigned to previous layouts of the content provider. The layout personality can also specify fonts, colors, visual effects, image preferences, etc. of the content provider. When the layout personality of the content provider can be used when generating the new layout. For example, the language model can use the layout personality as a basis for selecting specific fonts, color palettes, content locations, image sizes, etc. for the new layout.

[0101] As part of the new layout generation, a bounding box is generated for each given unit of content (e.g., each asset) among the two or more discrete units of content (e.g., set of assets) (318). As previously discussed, each bounding box is a presentation space for a corresponding unit of content (e.g., asset). The generation of the bounding box can be based on the particular unit of content to be presented and / or the layout personality of the content provider. For example, the size of the bounding box will be generated with a sufficient size such that the unit of content will be visually perceivable in the bounding box. The size of the bounding box can also be based, in part, on the layout personality of the content provider. For example, assuming that the logo of the content provider is always presented at a specific size, the bounding box for the logo can be generated based on that specific size.

[0102] As part of the new layout generation, positioning data is generated for each bounding box (320). As previously discussed, the positioning data specifies locations within the canvas at which the bounding box for each given unit of content is located. The positioning data can be a set of coordinates specifying the locations of specific features of thebounding box. For example, the corners of a rectangular bounding box can be the features for which coordinates are generated. These coordinates would constitute positioning data for the rectangular bounding box.

[0103] As part of the new layout generation, each bounding box is assigned to a corresponding UI layer (322). The assigned UI layer of a bounding box dictates the layer characteristics that will be applied to the given unit of content presented in the bounding box. For example, the layer characteristics can include color, opacity, and other visual attributes that can be applied to the given unit of content. The UI layers were previously discussed in more detail. For brevity, that discussion is not repeated here, but is equally applicable.

[0104] As part of the new layout generation, formatting / styling is applied to the bounding boxes and / or UI layers (324). In some implementations, the formatting / styling is defined by the layer characteristics as previously discussed. In some implementations, the formatting / styling is defined by the layout personality of the content provider. In some implementations, the formatting / styling is based on other content with which the new layout of discrete units of content will be presented. For example, in situations where the new layout will be presented with third-party content (e.g., in a slot of a web page or native application), the formatting / styling can be determined / adjusted based on the formatting / styling of the third-party content. In this way, the new layout will be visually more similar to the third-party content, such that the resulting combination of the new layout and the third-party content will be more visually appealing due to the consistencies in formatting / styling.

[0105] A text representation of the new layout is generated (326). In some implementations, the generation of the text representation includes the generation of text representing one or more of the bounding boxes for the discrete units of content, the positioning data for the bounding boxes, the user interface layer to which each bounding box has been assigned, and / or formatting / styling information. As previously discussed, the text representation can be generated using a compressed token representation that reduces the number of tokens required to be generated by a language model, which reduces the amount of time required for the language model to generate the text representation of the new layout. As such, the latency of the language model is reduced relative to generating the full text description of the new layout. In other words, the text representation generated using thetoken compressed representation is a generation of the representation of layout parameters of the new layout using fewer tokens than full text labels of the layout parameters.

[0106] In some implementations, the text representation of the new layout is generated by generating a sequence of discrete tokens specifying bounding boxes, content identifiers specifying which of the two or more discrete units of content that have been assigned to each of the bounding boxes, and layers to which each of the bounding boxes have been assigned. In some implementations, the generation of the text representation of the new layout includes compressing representations of layout parameters in a proto string.

[0107] The new layout is rendered (328). In some implementations, the new layout of the two or more discrete units of content are rendered and presented in a display according to the new layout. Each given unit of content can be rendered in (i) the corresponding user interface layer assigned to that given unit of content, and within the bounding box generated for that given unit of content. The rendered new layout can be presented to the content provided. For example, the content provider can be provided an option of approving or revising the new layout. As previously discussed, modification of the new layout is straight forward because the text representation of the new layout is completely editable.

[0108] FIG. 4 is a block diagram of an example computer system 400 that can be used to perform operations described above. The system 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Each of the components 410, 420, 430, and 440 can be interconnected, for example, using a system bus 450. The processor 410 is capable of processing instructions for execution within the system 400. In one implementation, the processor 410 is a single-threaded processor. In another implementation, the processor 410 is a multi-threaded processor. The processor 410 is capable of processing instructions stored in the memory 420 or on the storage device 430.

[0109] The memory 420 stores information within the system 400. In one implementation, the memory 420 is a computer-readable medium. In one implementation, the memory 420 is a volatile memory unit. In another implementation, the memory 420 is a non-volatile memory unit.

[0110] The storage device 430 is capable of providing mass storage for the system 400. In one implementation, the storage device 430 is a computer-readable medium. In various different implementations, the storage device 430 can include, for example, a hard diskdevice, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0111] The input / output device 440 provides input / output operations for the system 400. In one implementation, the input / output device 440 can include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802. 11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other devices, e.g., keyboard, printer, display, and other peripheral devices 460. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0112] Although an example processing system has been described in FIG. 4, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0113] An electronic document (which for brevity will simply be referred to as a document) does not necessarily correspond to a file. A document may be stored in a portion of a file that holds other documents, in a single file dedicated to the document in question, or in multiple coordinated files.

[0114] For situations in which the systems discussed here collect and / or use personal information about users, the users may be provided with an opportunity to enable / disable or control programs or features that may collect and / or use personal information (e.g., information about a user’s social network, social actions or activities, a user’s preferences, or a user’s current location). In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information associated with the user is removed. For example, a user’s identity may be anonymized so that the no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined.

[0115] Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software,firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0116] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0117] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0118] This document refers to a service apparatus. As used herein, a service apparatus is one or more data processing apparatus that perform operations to facilitate the distribution of content over a network. The service apparatus is depicted as a single block in block diagrams. However, while the service apparatus could be a single device or single set of devices, this disclosure contemplates that the service apparatus could also be a group of devices, or even multiple different systems that communicate in order to provide various content to client devices. For example, the service apparatus could encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0119] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0120] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0121] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random-access memory or both. The essential elements of acomputer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and compact disk read only memory (CD-ROM) and digital video disk read only memory (DVD-ROM) disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0122] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0123] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includesa front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0124] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML page) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server.

[0125] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0126] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may beadvantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0127] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

[0128] What is claimed is:

Claims

CLAIMS1. A method, comprising: obtaining, by an artificial intelligence (Al) system, two or more discrete units of content; generating, by the Al system and based on the two or more discrete units of content, a new layout of the two or more discrete units of content in a canvas, wherein the canvas represents an amount of display space available for presentation of the two or more discrete units of content, and wherein generating the new layout includes: generating, by the Al system, a bounding box as a presentation space for each given unit of content among the two or more discrete units of content; generating, by the Al system, positioning data specifying locations within the canvas at which the bounding box of each given unit of content are located; assigning, by the Al system, the bounding box for each given unit of content to a corresponding user interface layer; and generating, by the Al system, a text representation of the new layout that represents one or more of the bounding box, the positioning data, or the user interface layer using a compressed token representation; and rendering, based on the text representation of the new layout, the two or more discrete units of content according to the new layout, wherein each given unit of content is rendered in (i) the corresponding user interface layer assigned to that given unit of content, and within the bounding box generated for that given unit of content.

2. The method of claim 1, wherein generating the text representation using the token compressed representation comprises generating a representation of layout parameters of the new layout using fewer tokens than full text labels of the layout parameters.

3. The method of claim 1, further comprising obtaining, by the Al system, a layout personality of a content provider from which the two or more discrete units of content were obtained, wherein: the layout personality comprises a set of style tags assigned to previous layouts of the content provider; andgenerating the new layout comprises generating the new layout based on the layout personality.

4. The method of claim 3, further comprising: extracting, from a uniform resource locator of the content provider, a set of content presented at the uniform resource locator; and evaluating features of the set of content to obtain layout preferences of the content provider, wherein the layout preferences include one or more text preferences, image preferences, font preferences, logo preferences, or image preferences; and generating the layout personality of the content provider comprises generating the layout personality based on the layout preferences.

5. The method of claim 1, further comprising: extracting, from one or more content sources; a set of training data; performing one or more data synthesis operations to generate additional training data for the Al system, wherein at least one data synthesis operations includes creating variations of training samples from the set of training data; and training a language model of the Al system based on the set of training data and the generated additional training data.

6. The method of claim 1, wherein generating the text representation of the new layout comprises generating a sequence of discrete tokens specifying bounding boxes, content identifiers specifying which of the two or more discrete units of content that have been assigned to each of the bounding boxes, and layers to which each of the bounding boxes have been assigned.

7. The method of claim 1, wherein generating the text representation of the new layout comprises compressing representations layout parameters in a proto string.

8. An artificial intelligence (“Al”) system comprising: one or more memory devices; and one or more computing devices configured to interface with the one or more memory devices and execute instructions that cause the artificial intelligence system to perform operations comprising: obtaining, by an artificial intelligence (Al) system, two or more discrete units of content; generating, based on the two or more discrete units of content, a new layout of the two or more discrete units of content in a canvas, wherein the canvas represents an amount of display space available for presentation of the two or more discrete units of content, and wherein generating the new layout includes: generating a bounding box as a presentation space for each given unit of content among the two or more discrete units of content; generating positioning data specifying locations within the canvas at which the bounding box of each given unit of content are located; assigning the bounding box for each given unit of content to a corresponding user interface layer; and generating a text representation of the new layout that represents one or more of the bounding box, the positioning data, or the user interface layer using a compressed token representation; and rendering, based on the text representation of the new layout, the two or more discrete units of content according to the new layout, wherein each given unit of content is rendered in (i) the corresponding user interface layer assigned to that given unit of content, and within the bounding box generated for that given unit of content.

9. The Al system of claim 8, wherein generating the text representation using the token compressed representation comprises generating a representation of layout parameters of the new layout using fewer tokens than full text labels of the layout parameters.

10. The Al system of claim 8, wherein the instructions cause the Al system to perform operations further comprising obtaining, by the Al system, a layout personality of a content provider from which the two or more discrete units of content were obtained, wherein: the layout personality comprises a set of style tags assigned to previous layouts of the content provider; and generating the new layout comprises generating the new layout based on the layout personality.

11. The Al system of claim 10, wherein the instructions cause the Al system to perform operations further comprising: extracting, from a uniform resource locator of the content provider, a set of content presented at the uniform resource locator; evaluating features of the set of content to obtain layout preferences of the content provider, wherein the layout preferences include one or more text preferences, image preferences, font preferences, logo preferences, or image preferences; and generating the layout personality of the content provider based on the layout preferences.

12. The Al system of claim 8, wherein the instructions cause the Al system to perform operations further comprising: extracting, from one or more content sources; a set of training data; performing one or more data synthesis operations to generate additional training data for the Al system, wherein at least one data synthesis operations includes creating variations of training samples from the set of training data; and training a language model of the Al system based on the set of training data and the generated additional training data.

13. The Al system of claim 8, wherein generating the text representation of the new layout comprises generating a sequence of discrete tokens specifying bounding boxes, content identifiers specifying which of the two or more discrete units of content that havebeen assigned to each of the bounding boxes, and layers to which each of the bounding boxes have been assigned.

14. The Al system of claim 8, wherein generating the text representation of the new layout comprises compressing representations layout parameters in a proto string.

15. A non-transitory computer readable medium storing instructions that, upon execution by one or more data processing apparatus of an artificial intelligence system, cause the artificial intelligence system to perform operations comprising: obtaining, by an artificial intelligence (Al) system, two or more discrete units of content; generating, based on the two or more discrete units of content, a new layout of the two or more discrete units of content in a canvas, wherein the canvas represents an amount of display space available for presentation of the two or more discrete units of content, and wherein generating the new layout includes: generating a bounding box as a presentation space for each given unit of content among the two or more discrete units of content; generating positioning data specifying locations within the canvas at which the bounding box of each given unit of content are located; assigning the bounding box for each given unit of content to a corresponding user interface layer; and generating a text representation of the new layout that represents one or more of the bounding box, the positioning data, or the user interface layer using a compressed token representation; and rendering, based on the text representation of the new layout, the two or more discrete units of content according to the new layout, wherein each given unit of content is rendered in (i) the corresponding user interface layer assigned to that given unit of content, and within the bounding box generated for that given unit of content.

16. The non-transitory computer readable medium of claim 15, wherein generating the text representation using the token compressed representation comprisesgenerating a representation of layout parameters of the new layout using fewer tokens than full text labels of the layout parameters.

17. The non-transitory computer readable medium of claim 15, wherein the instructions cause the Al system to perform operations further comprising obtaining, by the Al system, a layout personality of a content provider from which the two or more discrete units of content were obtained, wherein: the layout personality comprises a set of style tags assigned to previous layouts of the content provider; and generating the new layout comprises generating the new layout based on the layout personality.

18. The non-transitory computer readable medium of claim 17, wherein the instructions cause the Al system to perform operations further comprising: extracting, from a uniform resource locator of the content provider, a set of content presented at the uniform resource locator; evaluating features of the set of content to obtain layout preferences of the content provider, wherein the layout preferences include one or more text preferences, image preferences, font preferences, logo preferences, or image preferences; and generating the layout personality of the content provider based on the layout preferences.

19. The non-transitory computer readable medium of claim 15, wherein the instructions cause the Al system to perform operations further comprising: extracting, from one or more content sources; a set of training data; performing one or more data synthesis operations to generate additional training data for the Al system, wherein at least one data synthesis operations includes creating variations of training samples from the set of training data; and training a language model of the Al system based on the set of training data and the generated additional training data.

20. The non-transitory computer readable medium of claim 15, wherein generating the text representation of the new layout comprises generating a sequence of discrete tokens specifying bounding boxes, content identifiers specifying which of the two or more discrete units of content that have been assigned to each of the bounding boxes, and layers to which each of the bounding boxes have been assigned.

Images