Conversational Interface for Content Creation and Editing Using Large-Scale Language Models

Abstract

An exemplary embodiment of the present disclosure provides an exemplary method that includes obtaining natural language input via a conversational campaign assistant interface with a custom language model. The method includes generating, by the custom language model, an output including a predicted user intent. The method includes determining an action to perform and determining a natural language response. The method includes sending an action data structure to an action component that includes executable instructions that cause the action component to automatically perform an operation associated with completing the action. The method includes sending a response data structure to the conversational campaign assistant interface that includes a natural language response to be provided for display to a user via the conversational campaign assistant interface. The method includes obtaining user input indicating validation of the action data structure or the response data structure and updating the custom language model based on the user input.

Description

[Technical Field]

[0001] Priority This application claims the benefit of priority to U.S. Non-Provisional Patent Application No. 18 / 322,543, filed May 23, 2023, which claims priority to Non-Provisional Patent Application No. 17 / 968,472, filed October 18, 2022, each of which is incorporated herein by reference in its entirety.

[0002] The present disclosure relates generally to systems and methods for providing a conversational interface for content creation and editing using large-scale language models. More specifically, the present disclosure relates to training and implementing one or more language models to facilitate a conversational interface for content creation and editing. [Background technology]

[0003] The computing device can perform many tasks and provide an interactive interface for content generation. The interactive interface for content generation can include multiple interactive components, including input fields. Summary of the Invention

[0004] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0005] In one exemplary aspect, the present disclosure provides an exemplary system including one or more processors and one or more memory devices storing executable instructions for causing the one or more processors to perform operations. In the exemplary system, the operations include obtaining natural language input via a conversational campaign assistant interface using a custom language model. In the exemplary system, the operations include generating, using the custom language model, an output including a predicted user intent. In the exemplary system, the operations include determining one or more actions to perform by parsing the output generated by the custom language model, determining an action associated with the output, or generating an action data structure including executable instructions for causing a processor to perform an operation associated with completing the action. In the exemplary system, the operations include determining a natural language response by parsing the output generated by the custom language model, generating a response data structure including a natural language response to the obtained natural language input, or sending an action data structure to an action component including executable instructions for causing the action component to automatically perform an operation associated with completing the action. In an exemplary system, the operations include sending a response data structure to the conversational campaign assistant interface, the response data structure including a natural language response to be provided for display to a user via the conversational campaign assistant interface. In an exemplary system, the operations include obtaining user input indicating validation of the action data structure or the response data structure following sending the action data structure and the response data structure. In an exemplary system, the operations include updating the custom language model based on the user input.

[0006] In an exemplary aspect, the present disclosure provides a computer-implemented method, the method including obtaining, via a custom language model, a natural language input via a conversational campaign assistant interface. The exemplary method includes generating, via the custom language model, an output including a predicted user intent. The exemplary method includes determining one or more actions to perform by parsing the output generated by the custom language model, determining an action associated with the output, or generating an action data structure including executable instructions that cause a processor to perform an operation associated with completing the action. The exemplary method includes determining a natural language response by parsing the output generated by the custom language model, generating a response data structure including a natural language response to the obtained natural language input, or sending the action data structure to an action component including executable instructions that cause the action component to automatically perform an operation associated with completing the action. The exemplary method includes sending a response data structure to the conversational campaign assistant interface including a natural language response that is provided for display to a user via the conversational campaign assistant interface. The exemplary method includes obtaining user input indicating validation of the action data structure or the response data structure following sending the action data structure and the response data structure. An exemplary method includes updating a custom language model based on user input.

[0007] In an exemplary non-transitory computer-readable medium, the operations include generating an initial user interface including a content assistant component. In an exemplary non-transitory computer-readable medium, the operations include obtaining natural language input via a conversational campaign assistant interface with a custom language model. In an exemplary non-transitory computer-readable medium, the operations include generating output including a predicted user intent with the custom language model. In an exemplary non-transitory computer-readable medium, the operations include determining one or more actions to perform by parsing the output generated by the custom language model, determining an action associated with the output, or generating an action data structure including executable instructions that cause a processor to perform an operation associated with completing the action. In an exemplary non-transitory computer-readable medium, the operations include determining a natural language response by parsing the output generated by the custom language model, generating a response data structure including a natural language response to the obtained natural language input, or sending an action data structure to an action component including executable instructions that cause the action component to automatically perform an operation associated with completing the action. In an exemplary non-transitory computer-readable medium, the operations include sending a response data structure to a conversational campaign assistant interface, the response data structure including a natural language response to be provided for display to a user via the conversational campaign assistant interface. In an exemplary non-transitory computer-readable medium, the operations include, subsequent to sending the action data structure and the response data structure, obtaining user input indicating validation of the action data structure or the response data structure. In an exemplary non-transitory computer-readable medium, the operations include updating a custom language model based on the user input.

[0008] In one exemplary aspect, the present disclosure provides an exemplary system including one or more processors and one or more memory devices storing executable instructions that cause the one or more processors to perform operations. In the exemplary system, the operations include providing input data to a first machine learning model. In the exemplary system, the operations include obtaining output data indicative of one or more suggested content item components. In the exemplary system, the operations include evaluating the first machine learning model based at least in part on the output data by obtaining data indicative of a quality score associated with the output data of the first machine learning model. In the exemplary system, the operations include evaluating the first machine learning model based at least in part on the output data by comparing the quality score associated with the output data of the first machine learning model to a threshold quality score. In the exemplary system, the operations include determining that the first machine learning model has a quality score above the threshold quality score. In the exemplary system, the operations include implementing the first machine learning model in a content creation flow in response to determining that the first machine learning model has a quality score above the threshold quality score.

[0009] In some embodiments of the exemplary system, the operation includes generating an initial user interface including a content assistant component. In some embodiments of the exemplary system, the operation includes obtaining data indicative of user input. In some embodiments of the exemplary system, the operation includes processing the data indicative of the user input by a first machine learning model. In some embodiments of the exemplary system, the operation includes obtaining output data indicative of one or more content item components from the first machine learning model. In some embodiments of the exemplary system, the operation includes transmitting data that causes the one or more content item components to be provided for display on the user interface. In some embodiments of the exemplary system, the operation includes obtaining data indicative of a user selection of approval of the one or more content item components. In some embodiments of the exemplary system, the operation includes generating one or more content items including the plurality of content item components in response to obtaining the data indicative of a user selection of approval of the one or more content item components.

[0010] In some embodiments of the exemplary system, the content assistant component includes one or more input fields.

[0011] In some embodiments of the exemplary system, the first machine learning model is trained using a knowledge distillation training method.

[0012] In some embodiments of the exemplary system, the first machine learning model is trained based at least in part on output from a pre-trained second machine learning model.

[0013] In some embodiments of the exemplary system, the pre-trained second machine learning model is a large-scale language model.

[0014] In some embodiments of the exemplary system, the pre-trained second machine learning model is tuned using one or more prompts.

[0015] In some embodiments of the exemplary system, the input data includes at least one of freeform input or landing page content.

[0016] In some embodiments of the exemplary system, the input data includes natural language input.

[0017] In some embodiments of the exemplary system, the one or more content item components include at least one of a headline or a description.

[0018] In an exemplary aspect, the present disclosure provides a computer-implemented method, the method including generating an initial user interface including a content assistant component. The exemplary method includes obtaining data indicative of input received from a user. The exemplary method includes processing the data indicative of the input received from the user by a machine learning model interfacing with the content assistant component. The exemplary method includes obtaining output data indicative of one or more content item components from the machine learning model interfacing with the content assistant component. The exemplary method includes transmitting data that causes the one or more content item components to be provided for display via an updated user interface. The exemplary method includes obtaining data indicative of a user selection of approval for the one or more content item components. The exemplary method includes generating one or more content items in response to obtaining the data indicative of the user selection of approval for the one or more content item components.

[0019] In some embodiments of the exemplary method, a knowledge distillation training method is used to train the machine learning model.

[0020] In some embodiments of the exemplary method, the method may include training a machine learning model by inputting labeled data into a first machine learning model. The exemplary method may include training the machine learning model by obtaining output data from the first machine learning model. The exemplary method may include training the machine learning model by comparing the output data from the first machine learning model with output data from a second machine learning model. The exemplary method may include training the machine learning model by adjusting the first machine learning model based on comparing the output data of the first machine learning model and the second machine learning model.

[0021] In some embodiments of the example method, the labeled data includes data output by a second machine learning model, and the second machine learning model is a pre-trained model.

[0022] In some embodiments of the exemplary method, the output by the second machine learning model includes annotated data.

[0023] In some embodiments of the example method, the labeled data includes at least one of: (i) business and product descriptions, (ii) proxy data from websites associated with content creators, (iii) human-curated data, or (iv) free-form input.

[0024] In some embodiments of the exemplary method, the method may include training the machine learning model by inputting unlabeled data to the machine learning model and the pre-trained second machine learning model. In some embodiments of the exemplary method, the method may include training the machine learning model by obtaining output data from the machine learning model and the pre-trained second machine learning model. In some embodiments of the exemplary method, the method may include training the machine learning model by comparing output data from the machine learning model with output data from the pre-trained second machine learning model. In some embodiments of the exemplary method, the method may include training the machine learning model by adjusting the machine learning model based on comparing the output data of the machine learning model with the output of the pre-trained second machine learning model. In some embodiments of the exemplary method, the initial user interface includes a graphical user interface.

[0025] In an exemplary aspect, the present disclosure provides an exemplary non-transitory computer-readable medium embodied in a computer-readable storage device, storing instructions that, when executed by a processor, cause the processor to perform operations. In the exemplary non-transitory computer-readable medium, the operations include providing input data to a first machine learning model. In the exemplary non-transitory computer-readable medium, the operations include obtaining output data indicative of one or more suggested content item components. In the exemplary non-transitory computer-readable medium, the operations include evaluating the first machine learning model based at least in part on the output data by obtaining data indicative of a quality score associated with the output data of the first machine learning model. In the exemplary non-transitory computer-readable medium, the operations include evaluating the first machine learning model based at least in part on the output data by comparing the quality score associated with the output data of the first machine learning model to a threshold quality score. In the exemplary non-transitory computer-readable medium, the operations include determining that the first machine learning model has a quality score above the threshold quality score. In an exemplary non-transitory computer-readable medium, the operations include implementing the first machine learning model in a content creation flow in response to determining that the first machine learning model has a quality score above a threshold quality score.

[0026] In an exemplary non-transitory computer-readable medium, the operations include generating an initial user interface including a content assistant component. In an exemplary non-transitory computer-readable medium, the operations include obtaining data indicative of input received from a user. In an exemplary non-transitory computer-readable medium, the operations include processing the data indicative of input received from the user by a first machine learning model. In an exemplary non-transitory computer-readable medium, the operations include obtaining output data indicative of one or more content item components from the first machine learning model. In an exemplary non-transitory computer-readable medium, the operations include transmitting data that causes the one or more content item components to be provided for display on the user interface. In an exemplary non-transitory computer-readable medium, the operations include obtaining data indicative of a user selection of approval of the one or more content item components. In an exemplary non-transitory computer-readable medium, the operations include generating one or more content items in response to obtaining the data indicative of a user selection of approval of the one or more content item components.

[0027] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1] 1 illustrates a block diagram of an exemplary system for providing a conversational interface for content creation and editing using large-scale language models, according to an exemplary embodiment of the present disclosure. [Figure 2] 1 illustrates a block diagram associated with machine learning model(s), according to an exemplary embodiment of the present disclosure. [Figure 3] 1 illustrates a flowchart of an exemplary method for training machine learning model(s), according to an exemplary embodiment of the present disclosure. [Figure 4]1 shows a flowchart of an exemplary process flow according to an exemplary embodiment of the present disclosure. [Figure 5] 1 shows a flowchart of an exemplary process flow according to an exemplary embodiment of the present disclosure. [Figure 6A] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 6B] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 6C] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 7] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 8] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 9] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 10] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 11] 1 illustrates a block diagram associated with machine learning model(s), according to an exemplary embodiment of the present disclosure. [Figure 12] 1 shows a flowchart of an exemplary process flow according to an exemplary embodiment of the present disclosure. [Figure 13] 1 shows a flowchart of an exemplary process flow according to an exemplary embodiment of the present disclosure. [Figure 14A] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 14B] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 14C] 1 illustrates a flowchart of an exemplary method according to an exemplary embodiment of the present disclosure. [Figure 15] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 16]1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 17] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 18] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 19] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. [Figure 20] 1 illustrates an exemplary user interface according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029] In general, the present disclosure is directed to systems and methods for generating and tuning large-scale language models that receive natural language user input and generate content item components as output. The computing system and method can include providing for displaying an interface for creating and editing content items in response to receiving and processing data indicative of free-form user input. For example, the computing system can initiate display of a content assistant component. Through the content assistant component, the system can receive user input including free-form speech, analyze the speech, extract information from websites about the user, and generate advertisements based on the received input, generating headlines and / or other content items (e.g., advertising components) that typically require the user to manually enter information into multiple fields associated with a structured interface. The structured interface can be associated with a content creation flow. The content creation flow can be used to create content items (e.g., advertisements, advertising campaigns). Conventional structured interfaces require manual entry into multiple (e.g., in some cases, 40 or more) input fields associated with strict requirements.

[0030] Implementations of the present disclosure can use complex machine learning models to provide more efficient and user-friendly generation of content items. In some implementations, analyzing user input can be performed by a machine learning model (e.g., a natural language processing model). In some implementations, systems and methods can include training and / or tuning the machine learning model. By way of example, a computing system can obtain model input from multiple sources. The sources can include, for example, business and product descriptions, proxy data from websites associated with content creators, human-curated data, free-form input, etc. The model output can include content item components (e.g., headlines, descriptions, graphics, videos, color schemes). The computing system can train the machine learning model(s) implemented as part of the content item generation process.

[0031] In some implementations, the training process can include knowledge distillation using one or more teacher and student models. For example, the teacher model can be an existing large-scale language model. In some examples, the student model can be a customized model developed to generate customized content items (e.g., headlines, descriptions, advertisements, videos, images, soundbites). For example, a machine learning model can provide output including content item components (e.g., headlines, descriptions, advertisements, videos, images, soundbites). The output of the student model can be evaluated to determine a quality score associated with the output. For example, the quality score can be associated with automatically generated quality scores, user satisfaction with the content item components, human-administered grading, etc.

[0032] In some implementations, the computing system can collect multiple datasets and employ large-scale language models to generate, annotate, and / or generate data examples to train high-quality, serviceable models. The computing system can evaluate the machine learning model(s). For example, the evaluation can include offline and / or online evaluation. The computing system can also determine whether the machine learning models are faithful.

[0033] The present disclosure provides numerous technical effects and advantages. For example, the present disclosure can reduce processing by reducing the number of manual inputs provided by a user. Additionally, a computing system can provide reduced processing by reducing the number of interface screens that need to be obtained, loaded, interacted with, and updated. For example, a user can provide data indicative of an input (e.g., audio input, text input, etc.). The computing system can intelligently process the data indicative of the input and extract relevant information from the input. In some implementations, the computing system can provide follow-up questions to guide the user in providing input regarding a desired content item to be generated. The computing system can provide reduced user error due to less user input to populate multiple fields that a user would traditionally fill manually via structured user input and less pre-processing of data indicative of the user input.

[0034] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in further detail.

[0035] 1 illustrates a block diagram of an exemplary computing system 100 for generating, training, and / or using complex language models for interacting with a user interface to facilitate an exemplary embodiment of content item generation of the present disclosure. The computing system 100 includes a client computing system 102, a server computing system 104, and a training computing system 106 that are communicatively coupled via a network 108.

[0036] The client computing system 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0037] The client computing system 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 114 can include one or more computer-readable storage media, which can be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 that are executed by the processor 112 to cause the client computing system 102 to perform operations.

[0038] In some implementations, the client computing system 102 may store or include one or more machine learning models 120. For example, the machine learning models 120 may be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models. The neural networks may include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models may utilize attention mechanisms, such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., Transformer models). Exemplary machine learning models 120 are described with reference to FIGS. 2 and 3.

[0039] In some implementations, one or more machine learning models 120 may be received from the server computing system 104 over the network 108, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the client computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel acquisition and composition of a modular application space across multiple instances of user data acquired via a user interface associated with the user device).

[0040] More specifically, the machine learning model can obtain data indicative of user input (e.g., user session data 124A). The user input data can be associated with a current user session and / or can include historical user data. For example, data associated with a current user session can be data obtained in real time via user input component 122. The historical user data can include data associated with a user account, user characteristics, etc. The historical user data can include data associated with a user device (e.g., a device identifier). Additionally or alternatively, the historical user data can include data associated with a user identifier. In some embodiments, the historical user data can include aggregated data associated with multiple user identifiers. In some embodiments, training data 166 can include session data (e.g., for one or more input sessions) associated with one or more input devices, such as session data indexed across types of input interfaces or devices (e.g., mobile devices with touchscreens, mobile devices with keyboards, large touchscreens, small touchscreens, large touchscreens, voice input, or a combination thereof, etc.). In some embodiments, training data 166 can include session data not associated with a user identifier. Using the machine learning models, the computing system can generate, train, and maintain one or more student models 164 (e.g., customized language models). The computing system can use the student models 164 to facilitate a user interface for generating customized content items (e.g., headlines, descriptions, advertisements) based on natural language input obtained from a user. The computing system can obtain, suggest, and / or generate one or more content items in response to obtaining user input (e.g., natural language input).

[0041] Additionally or alternatively, one or more machine learning models 126 may be included in or stored and implemented on a server computing system 104 that communicates with the client computing system 102 according to a client-server relationship. For example, the machine learning models 126 may be implemented by the server computing system 104 as part of a web service (e.g., a content development service, a campaign management service, a content strategy management service). Thus, one or more machine learning models 120 may be stored and implemented on the client computing system 102 and / or one or more models 126 may be stored and implemented on the server computing system 104.

[0042] The client computing system 102 may also include one or more user input components 122 capable of receiving user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0043] The client computing system may include a user data database 124. The user data database 124 may include user session data 124A, user context data 124B, and / or user account data 124C. The user session data 124A may include data obtained via the user input component 122 indicative of a current user session. For example, the user session data 124A may include current search terms and / or other user input received within a threshold time period of the current session. For example, a user may submit a first search and, five minutes later, a second search. The proximity in time of the first and second searches may be user context data 124B. The computing system may use the user context data 124B when processing user queries to determine relevant content items and predicted performance (e.g., predicted performance metrics) improvements to provide in response to data indicative of user input to the structured user interface. Data indicative of the user input can be used by the client computing system 102 to send a request to the server computing system 104 for one or more suggested content item components (e.g., suggested headlines, generated user interface elements, suggested labels). The computing system can retrieve, generate, and / or present the one or more suggested content items to the user via a user interface of a device (e.g., a user device). User context data 124B can include previous session context and / or historical session data. User context data can include location, time, previous campaigns, previous content items used, device used, type of build workflow used, etc. User account data 124C can include data associated with a user account (e.g., login, device identifier, user identifier).

[0044] The server computing system 104 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 134 can include one or more computer-readable storage media, which can be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 can store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 104 to perform operations.

[0045] In some implementations, server computing system 104 includes or is otherwise implemented with one or more server computing devices. When server computing system 104 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0046] As described above, the server computing system 104 may store or otherwise include one or more machine learning models 126. For example, the machine learning models 126 may be or include various machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some exemplary machine learning models may utilize attention mechanisms such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., Transformer models). Exemplary machine learning models 126 are described with reference to FIGS. 2-3.

[0047] The client computing system 102 and / or the server computing system 104 can train the machine learning models 120 and / or 126 by interacting with a training computing system 106 that is communicatively coupled via a network 108. The training computing system 106 can be separate from the server computing system 104 or can be part of the server computing system 104.

[0048] The training computing system 106 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 154 can include one or more computer-readable storage media, which can be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 106 to perform operations. In some implementations, the training computing system 106 includes or is otherwise implemented with one or more server computing devices (e.g., server computing system 104).

[0049] The training computing system 106 may include a model trainer 160 that trains the machine learning models 120 and / or 126 stored on the client computing system 102 and / or the server computing system 104 using various training or learning techniques, such as knowledge distillation. For example, the model trainer 160 may include one or more teacher models 162, one or more student models 164, and / or training data 166. For example, the teacher models 162 may include one or more pre-trained large-scale language models. The large-scale language models may include deep neural networks that require extensive resources and time to train. The large-scale language models may be equipped to determine meaning from natural language input.

[0050] In some implementations, the training data may include labeled data. For example, the labeled data may be input to a teacher model and / or a student model. In some examples, the training data 166 may be data obtained as output from the teacher model 162. A computing system may obtain the output from the teacher model 162 and label the data. This labeled data may be used to train the student model 164. The use of the training data 166 to train the student model 164 is further described with reference to FIGS. 2-3.

[0051] In some implementations, various training or learning techniques can include, for example, backpropagation of errors. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update parameters over several training iterations.

[0052] In some implementations, performing backpropagation may include performing truncated backpropagation over time. Model trainer 160 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0053] In particular, model trainer 160 can train machine learning models 120 and / or 126 based on a set of training data 166. Training data 166 can include, for example, past performance metrics (e.g., predicted performance improvement(s)). In some implementations, one or more student models 164 can be machine learning models 120 and / or 126.

[0054] In some implementations, if the user provides consent, training examples may be provided by the client computing system 102. Thus, in such implementations, the machine learning model 120 provided to the client computing system 102 may be trained by the training computing system 106 based on user-specific data received from the client computing system 102. In some implementations, this process may be referred to as personalizing the model.

[0055] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some embodiments, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0056] Network 108 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 108 may be carried over any type of wired and / or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0057] The machine learning models described herein may be used for a variety of tasks, applications, and / or use cases.

[0058] In some implementations, input to the machine learning model(s) of the present disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to generate an output. As an example, the machine learning model(s) may process the natural language data to generate a language-encoded output. As another example, the machine learning model(s) may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model(s) may process the text or natural language data to generate a translation output. As another example, the machine learning model(s) may process the text or natural language data to generate a classification output. As another example, the machine learning model(s) may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model(s) may process the text or natural language data to generate a semantic intent output. The semantic intent output may include at least one word or phrase determined from the text or natural language data. As another example, the machine learning model(s) can process text or natural language data to generate upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As another example, the machine learning model(s) can process text or natural language data to generate a predicted output.

[0059] In some implementations, input to the machine learning model(s) of the present disclosure can be audio data. The machine learning model(s) can process the audio data to generate an output. As an example, the machine learning model(s) can process the audio data to generate a speech recognition output. As another example, the machine learning model(s) can process the audio data to generate a speech translation output. As another example, the machine learning model(s) can process the audio data to generate a latent embedding output. As another example, the machine learning model(s) can process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning model(s) can process the audio data to generate a predicted output.

[0060] In some implementations, input to the machine learning model(s) of the present disclosure can be latent-coded data (e.g., a latent space representation of the input, etc.). The machine learning model(s) can process the latent-coded data to generate an output. As an example, the machine learning model(s) can process the latent-coded data to generate a recognition output. As another example, the machine learning model(s) can process the latent-coded data to generate a reconstruction output. As another example, the machine learning model(s) can process the latent-coded data to generate a retrieval output. As another example, the machine learning model(s) can process the latent-coded data to generate a reclustered output. As another example, the machine learning model(s) can process the latent-coded data to generate a predicted output.

[0061] In some implementations, input to the machine learning model(s) of the present disclosure can be statistical data. The statistical data can be, represent, or otherwise include data that has been computed and / or calculated from some other data source. The machine learning model(s) can process the statistical data to generate an output. As an example, the machine learning model(s) can process the statistical data to generate a recognition output. As another example, the machine learning model(s) can process the statistical data to generate a prediction output. As another example, the machine learning model(s) can process the statistical data to generate a classification output. As another example, the machine learning model(s) can process the statistical data to generate a segmentation output. As another example, the machine learning model(s) can process the statistical data to generate a visualization output. As another example, the machine learning model(s) can process the statistical data to generate a diagnostic output.

[0062] In some cases, the machine learning model(s) can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task. The input may include audio data, and the output may include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), and the output includes compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or visual data).

[0063] In some cases, the input includes audio data representing a speech utterance and the task is a speech recognition task. The output may include text output that is mapped to the speech utterance. In some cases, the task includes encrypting or decrypting input data. In some cases, the task includes a microprocessor performance task, such as branch prediction or memory address translation.

[0064] In some implementations, machine learning models can be deployed on-device. For example, one or more components of a predictive machine learning model or pipeline can be deployed on-device to avoid uploading potentially sensitive information to a server regarding the type of input, the type of device(s), or the content of the input (e.g., regarding failures, contact information, addresses, etc.). For example, a server computing system can submit a form including a learned context vector that describes one or more input fields associated with a component (e.g., portions of an application associated with performing a processing task). An on-board client model associated with the client computing system 102 can input local client characteristics (e.g., obtained via the user input component 122) and the context vector to generate a configured modular application. This on-device processing can enhance data privacy for users. In some embodiments, it can also reduce bandwidth usage by reducing the amount of data transmitted off-device.

[0065] 2 illustrates one exemplary system 200 for generating predicted performance improvements associated with proposed content items and for providing an updated user interface including the proposed content items and their respective predicted performance improvements, according to an exemplary embodiment of the present disclosure. The exemplary system 200 includes a computing system 202. The computing system 202 can be any type of system of one or more computing devices (e.g., client computing system 102, server computing system 104, etc.). The computing device may be, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a node of a distributed computing device, a virtual instance hosted on a shared server, or any other type of computing device. In some embodiments, the computing system 202 includes multiple computing devices interconnected via a network or distributed in other interoperable manner. For example, the computing system 202 may include a server for serving content over a network (e.g., network 108). For example, computing system 202 may include a web server for hosting web content and for collecting data about the web content (e.g., for receiving, monitoring, generating, or otherwise processing data about the web content, such as usage, downloads, and / or interactions with the web content).

[0066] Computing system 202 may include processor(s) 212 and memory 214. The one or more processors 212 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. Memory 214 may include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which may be non-transitory, and combinations thereof. Memory 214 may store data 216 and instructions 218 that are executed by processor(s) 212 to cause computing system 202 to perform operations.

[0067] In some implementations, the computing system 202 can store or otherwise implement one or more machine learning models of a machine learning model framework. In some embodiments, the content assistant component 220 includes a two-model machine learning model framework. In some embodiments, the content assistant component does not include any combination of a machine learning model framework, a custom language model 222, and / or a large-scale language model 226. The machine learning model framework can include a machine-learned custom language model 222 (e.g., with learnable weights 224) and / or a machine-learned large-scale language model 226 (e.g., with learnable weights 228). In some embodiments, the content assistant component 220 can implement a single model that implements the custom language model 222 (e.g., by combining one or more aspects of the custom language model 222 and / or the large-scale language model 226, training a single model to directly obtain a desired output, etc.) for content suggestion, content ranking, content generation, and / or any combination. One or more of the machine learning models can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models can utilize attention mechanisms, such as self-attention. For example, some exemplary machine learning models can include multi-head self-attention models (e.g., Transformer models).

[0068] An embodiment of the exemplary system 200 may be configured to process data 230, as shown in FIG. 2. In response to processing the data 230, the computing system 202 may provide output 250. The output 250 may include generated suggested content item components 255. The suggested content item components 255 may be provided for display via a user interface 270 of a client device associated with the client computing system 260. The client computing system 260 may include processor(s) 262 and memory 264. The one or more processors 262 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 264 may include one or more computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., which may be non-transitory, and combinations thereof. The memory 264 may store data 266 and instructions 268 that are executed by the processor 262 to cause the client computing system 260 to perform operations.

[0069] The output 250 can include a suggested content item component 255. The suggested content item component 255 can include, for example, a suggested headline, a description, a follow-up communication, a video, an image, a sound bite, etc. For example, the custom language model 222 can take user input and provide follow-up communication to gather additional input and generate further suggested content item components 255.

[0070] In some embodiments, data 230 is obtained via an input interface of a computing system (e.g., computing system 202 and / or client computing system 260). For example, in some embodiments, content assistant component 220 can be configured to process data 230 as input to computing system 202. In some embodiments, data 230 can be implicit in the structure and / or configuration of content assistant component 220.

[0071] In some implementations, the content suggestion generator model is trained to receive a set of input data (e.g., data 230) describing user input and, in response, provide output data (e.g., suggested content item component 255) indicating one or more suggested content items to be rendered via a user interface. For example, Figure 2 shows an instance of a rating.

[0072] For example, the evaluation component 240 can perform offline evaluation and / or online evaluation (e.g., actual traffic evaluation). The evaluation can include offline and online evaluation. The offline evaluation can include using a human evaluation platform. The human evaluation platform can include templates and / or rubrics to standardize the evaluation. The online evaluation can include pilot and / or live traffic experiments to capture online metrics. In some implementations, the evaluation component 240 can include automated evaluation and / or manual evaluation. In some implementations, the evaluation component 240 can include tagging and / or labeling data as training data. The system can determine a quality score for the model. The system can compare the quality score of the model to a threshold quality score. If the quality score of the model is above the threshold quality score, the system can determine that the model is ready to be used. If the quality score of the model is below the threshold quality score, the system can determine that additional training and / or tuning is required before the model can be used.

[0073] In some implementations, the input data may include one or more features associated with an example or example. In some implementations, the one or more features associated with the example or example may be organized into a feature vector. In some implementations, the output data may include one or more predictions. A prediction may also be referred to as an inference. Thus, given features associated with a particular example, a machine learning model may output a prediction for such example based on the features.

[0074] The machine learning model may be or include one or more of a variety of different types of machine learning models. In particular, in some implementations, the machine learning model may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0075] In some implementations, data 230 includes aggregate data 232, proxy data 234, and / or user-input data 238. In some embodiments, data 230 can include instances of virtually any kind or type of data that can describe various phenomena. Generally, an instance refers to a particular subject or a set of one or more data values ​​grouped together to describe a subject. For example, an instance can be a feature vector. An instance can be associated with image data (e.g., a feature vector of an image, a hashed image, etc.). An instance can be associated with a measurement or other data collection event (e.g., at a particular time, or of a particular subject, or using a particular device, or from a particular perspective, etc.). An instance can be associated with a network session, such as a set of interactions with a web server. In some embodiments, an instance can be associated with a user's interactions with web content (e.g., anonymous or identified).

[0076] In some embodiments, the custom language model 222 can be trained using knowledge distillation. For example, the custom language model 222 can be a student model, and the large-scale language model 226 can be a teacher model. The knowledge distillation training process can be performed to train the small-scale custom language model 222 based on the pre-trained large-scale language model 226. For example, the large-scale language model can be a large-scale model that is generated and trained using a large amount of computing resources. To conserve resources, knowledge distillation can be used to train the custom language model 222 to produce results similar to the pre-trained large-scale language model 226.

[0077] In some implementations, the models of the content assistant component 220 can be tuned and / or trained using prompts. For example, the computing system can obtain prompts that indicate sample inputs and outputs. The sample inputs and outputs can be used to tune the models to be familiar with the types of inputs and outputs desired for a particular implementation. For example, a particular implementation can include generating advertisements. Generating advertisements requires a nuanced understanding of the business, business goals, products, etc. In this manner, by prompting one or more models (e.g., custom language model 222 and / or large-scale language model 226), the computing system can tune the models to provide higher quality output (e.g., output 250 that can be evaluated using evaluation component 240). In some examples, after evaluation component 240 performs the evaluation, the data can be aggregated and used as data 230 for further model training.

[0078] In some implementations, the content assistant component 220 can be a machine learning model. In some implementations, training and tuning can include using data 230. Aggregate data 232 can include data obtained from one or more user sessions aggregated by a system (e.g., computing system 202). Proxy data 234 can include, for example, output data obtained from the content assistant component 220. In some implementations, the proxy data 234 can be tagged and / or labeled to represent sample inputs and / or outputs. User input data 238 can include user-generated inputs having inputs and / or outputs associated with desired content items and generated content items.

[0079] The trained model can be used to generate one or more suggested content items and / or update the structured user interface and / or construction workflow. In some embodiments, a simulated user model can be trained using the training data to generate simulated inputs according to patterns learned from the training data. The simulated user model, in some examples, can provide a simulated environment in which a reinforcement learning agent, a notification component subsystem, can be trained.

[0080] 3 shows a flowchart diagram of an example method 300 for training a student model 310 using knowledge distillation. Training the student model 310 may include utilizing a teacher model 315, training data 305, and / or a distillation component 320. For example, a computing system may define a large-scale language model as the teacher model 315. A computing system may define a custom language model as the student model 310. In some implementations, the teacher model 315 may include a large-scale language model with millions and / or billions of parameters. In some implementations, the student model 310 may include a custom language model with thousands of parameters.

[0081] The teacher model 315 can be fully trained. For example, the teacher model can be trained as part of the method 300 and / or prior to convergence with one or more student models 310. Any training method can be used to train the teacher model 315. For example, the teacher model 315 can be trained until full convergence. For example, the loss function can be any loss function based on the problem statement.

[0082] The computing system can intelligently train the student model 310 using the teacher model 315. In some implementations, for example, the student model 310 and the teacher model 315 can include neural networks. For example, the student model 310 can be trained in concert with a fully trained teacher model 315. In some implementations, training data 305 is input to the teacher model 315. Output data 330 of the teacher model 315 can be obtained. The computing system can annotate the training data 305 by running the training data 305 through the teacher model 315. The computing system can obtain annotated training data 325 from the teacher model 315 as output data 330. The student model 310 can obtain the annotated training data 325 as input. The student model 310 can be trained using the annotated training data.

[0083] For example, the teacher model 315 can include a large-scale language model that can be trained with training data 305. The training data 305 can include, for example, a dataset having over 1.56 trillion words of multilingually cleaned public web documents, code, and conversations from the Internet. In some implementations, the training data 305 can include a seed dataset collected from humans. In some implementations, the computing system can acquire data to expand the training dataset. In some implementations, the training data 305 can include proxy data from a content provider's (e.g., advertiser's) website. For example, a content provider can have a website that includes pages such as "About Us," "Company Story," "Blog," and "Services." These pages can include business description content. One or more teacher and / or student models can take the proxy data as input and generate output including one or more content items (e.g., headlines, descriptions). In some implementations, the training data 305 can include human-curated data. Human-curated data can be data acquired through user input. For example, a user can provide a written business description. Human-curated data can be used as seed training data. In some implementations, the computing system can use an existing large-scale language model to generate the data used as training data 305. For example, the large-scale language model can generate descriptions and creative assets. In some implementations, the descriptions and creative assets can include freeform input and / or headlines.

[0084] Additionally or alternatively, the teacher model 315 can include an encoder-decoder network. In some implementations, the encoder-decoder network can be a general-purpose multimodal language model with multi-task learning capabilities. In some implementations, the encoder-decoder network can be pre-trained on text from billions of high-quality web documents and can be highly tailored to a particular subject area (e.g., search results, advertising, a particular business).

[0085] In some examples, the computing system can tune the student model 310. For example, the computing system can input a subject-specific dataset to the student model 310. By providing the subject-specific dataset to the student model 310 during training, the computing system can demonstrate significant improvements in subject-specific applications (e.g., use of the student model). Tuning and / or training the student model 310 to a specific subject can allow for a more nuanced understanding of the input obtained from the user. The more nuanced the understanding of the input obtained from the user, the better the output can be. The better the output, the less processing can be performed by the computing system to handle repetitive input obtained from users who must manually enter data into multiple fields with strict requirements and / or increased touch input. Furthermore, providing a better output can improve user experience and satisfaction.

[0086] Although the teacher model 315 is described as a single model for purposes of illustration, the teacher model 315 can include one or more models. Although the student model 310 is described as a single model for purposes of illustration, the student model 310 can include one or more models.

[0087] In some implementations, the teacher and student setup can include additional and / or alternative means of knowledge distillation. For example, knowledge distillation can include use of a distillation component 320 that includes a distillation loss function 335 and a student loss function 345. The distillation loss function 335 can be used in forward propagation of the teacher model 315 and the student model 310. Distillation can include backpropagation of the student model 310, for example, by converging the losses between the outputs of the student model 310 and the teacher model 315.

[0088] Algorithms used in knowledge distillation of the teacher model 315 and the student model 310 can include, for example, adversarial distillation, multi-supervised distillation, cross-modal distillation, graph-based distillation, attention-based distillation, data-free distillation, quantized distillation, lifetime distillation, and / or neural architecture search-based distillation. Training the student model 310 can include, for example, offline distillation, online distillation, and / or self-distillation. Knowledge-based distillation can include response-based knowledge, feature-based knowledge, and / or relationship-based knowledge. Knowledge distillation can be implemented in a variety of applications. Applications can include, for example, vision, natural language processing, and / or speech.

[0089] Vision applications can include, for example, image classification, face recognition, image segmentation, action recognition, video caption generation, image retrieval, text-to-image synthesis, and / or video classification. NLP applications can include, for example, text generation, text recognition, neural machine translation, question answering, and / or document retrieval. Audio applications can include, for example, speech recognition, language identification, audio classification, speaker recognition, speech synthesis, speech enhancement, and / or acoustic event detection.

[0090] Knowledge distillation can provide various technical benefits and effects. For example, by utilizing a pre-trained model to tune a content-specific model, a computing system can save the computing resources and processing power required to train a model to perform similarly to a pre-trained model without the pre-trained model. Furthermore, in some implementations, the training dataset can include one or more prompts used to tune the teacher model 315 and / or student model 310. By providing input along with the prompts, the computing system can tune the model to produce better output than traditional model training methods.

[0091] 4 shows a flowchart diagram of a process flow 400 for facilitating content generation through a responsive content building process at 420. By way of example, a computing system may include a content item table 405. At 410, the computing system may obtain user input indicating a user selecting (e.g., clicking) an option to generate a new content item (e.g., an advertisement, an advertising campaign). At 415, the computing system may obtain user input indicating a user selecting a content item group. In response to obtaining user input indicating selecting a content item group, the computing system may initiate responsive content building at 420.

[0092] At 420, the computing system initiating the responsive content construction can include providing a takeover promotion for the content assistant for display via a user interface. For example, at 425, the takeover promotion for the content assistant can be provided for display.

[0093] At 430, the computing system can provide a selectable interface element for display that includes a message. For example, the message can include the message "Use content assistant?" In some implementations, the message can be displayed by selectable user interface elements labeled "Yes" and "No."

[0094] As an example, the computing system may obtain data indicating that the user selected "No." In response, at 435, the computing system may cause the takeover promotion for the content assistant to disappear. At 440, the computing system may provide for display a selectable interface including a message. The message may include: "Does the user have a final URL?" In some implementations, the message may be displayed by selectable user interface elements labeled "Yes" and "No." The computing system may obtain data indicating that the user selected "Yes." In response, at 445A, the computing system may pre-fill one or more input fields associated with the user interface for content generation. The computing system may obtain data indicating that the user selected "No." In response, at 445B, the computing system may provide one or more input fields associated with the user interface for content generation without pre-filling them.

[0095] As an example, in response to displaying the message at 430, the computing system can obtain data indicating a user selection of "yes." In response, the computing system can update the user interface and provide a content assistant 447 for display. At 450, the computing system can provide for displaying a content assistant component via the user interface. The content assistant component can be displayed in a variety of formats. For example, the formats can include a set number of free-form text entry fields (e.g., as shown in FIG. 8), one or more progressive disclosure fields (e.g., as shown in FIG. 9), and / or a conversational interface (e.g., as shown in FIG. 10). The computing system can execute a content assistant flow 455. The content assistant flow 455 is described in more detail with reference to FIG. 5.

[0096] Referring to FIG. 5, content assistant flow 455 includes providing an interface for a user to interact with (e.g., a content assistant component of a user interface). As mentioned above, the interface can be an audio interface, a visual interface, etc. For purposes of explanation, content assistant flow 455 is described as a visual interface (e.g., a graphical user interface). At 505, the computing system can provide a message for display: "Tell me about your business?" The message can include any message that prompts a user to provide input data in natural language and / or a free-form format. At 510, the computing system can obtain user input in response to displaying the message. For example, the displayed message can indicate, "Can you describe the product or service you would like to promote and how it would benefit potential customers?" For example, the user can provide information about their business. A user might provide the following as input: "We're a small puppy supply company in Northern Massachusetts. We're called PuppyThings. We sell dog food, treats, toys, and anything puppy-related. Quality products at great value. We're open 9-5, offer free money-back, and ship to all 50 states."

[0097] At 515, the computing system can extract information from the user input. For example, the extracted information can include the business name and / or descriptive terms and phrases. In the example shown in FIG. 10, the system can determine that the business name is PuppyThings. The system can extract other related terms such as puppies, food, treats, toys, high quality, 9-to-5, free money-back, and delivers to all 50 states.

[0098] In some implementations, the computing system may determine that additional information is needed. In response, at 520, the computing system may provide a message for display indicating the need for additional information from the user. For example, the message may include a follow-up question, a question for clarification, a question regarding information not included in the initial user input, etc. For example, as shown in FIG. 10, the system may provide the following follow-up question: "Is there a website that you visit after clicking on the ad?"

[0099] At 522, the computing system may obtain user input in response to the message for additional information. For example, as shown in Figure 10, the user may provide the following as input: "www.puppythings.com." The computing system may extract information from the user input at 515 as described above. This process may be repeated as many times as necessary to gather relevant information.

[0100] At 525, the computing system may generate suggested content. For example, the suggested content may include the input of one or more input fields. In some implementations, the content may include the generated advertisement and / or the entire campaign. For example, the generated advertisement may include a visual representation of the suggested advertisement. As an example, the computing system may provide for displaying a summary of the advertising campaign strategy, including one or more generated advertisements, display requirements, time of display, intended performance, etc.

[0101] For example, with reference to FIG. 10, the computing system may generate the following message: "Your Information: We've created your ad based on 10 keywords, 5 headlines, and 2 descriptions. Contact us if you want to make any revisions." For example, the computing system may generate the following suggested headlines: Anything your puppy needs, puppy food, treats, and toys, quality products for puppies, 9-5 hours and free money-back, and shipping to all states. The computing system may generate the following suggested descriptions: puppy supplies, value, high quality, free shipping. Shop now at puppythings.com and / or get everything your new puppy needs: food, treats, toys, and more. All at puppythings.com. The computing system may determine the following target keywords: puppy supplies, dog food, dog treats, etc.

[0102] At 530, the computing system can obtain user input indicating a desire to modify the suggested content. For example, the user can specify that they mean to offer free money-back guarantees and separately ship to all 50 states. They can also indicate that they do not offer free shipping. The suggested headline and / or description can be modified based on the user input. For example, the computing system can obtain user input in a natural language format. For example, the user can reply, "Please change the first headline and remove free shipping. Thank you."

[0103] At 535, the computing system may modify the content in response to the user input. In some implementations, the computing system may repeat steps 530 and 535 until the computing system obtains input of user satisfaction with the proposed content.

[0104] At 540, the computing system may obtain user input indicating satisfaction. For example, the computing system may obtain data indicating that the user selected “continue,” “accept,” etc.

[0105] At 545, in response to obtaining user input indicating satisfaction, the computing system may terminate the content assistant flow 455. In some implementations, the computing system continues the process described in FIG.

[0106] Returning to FIG. 4 , following the content assistant flow 455, the computing system may obtain user input indicating selection of an option to “apply suggestions” at 460. In response, the computing system may populate one or more input fields (e.g., those associated with generating a content item, advertisement, etc.). The user may review the populated input fields to determine whether any errors or problems exist. At 465, the computing system may provide for displaying a message via a user interface. The message may include: “Are there any errors or problems?” In some implementations, the message may be displayed by selectable user interface elements labeled “Yes” and “No.” The computing system may obtain data indicating the user selected “Yes.” In response, at 470, the computing system may initiate display of an error message and / or a troubleshooting flow. The computing system may obtain data indicating the user selected “No.” In response, at 475, the computing system may determine whether this is the user's first time using the assistant.

[0107] In some embodiments, the computing system can determine that the user is not using the assistant for the first time. In response, at 480, the computing system can render the content assistant invisible and provide for displaying input fields populated by the content assistant. In some embodiments, the computing system can determine that the user is using the assistant for the first time. In response, at 485, the computing system can provide for displaying a feature promotion sequence that includes an alert. The alert can include highlighting one or more input fields populated by the content assistant and / or a notice describing where to find the content item components and / or content items generated by the computing system via the responsive content building at 420.

[0108] 6A-6C show a flowchart diagram of an exemplary method 600 for performing in accordance with an exemplary embodiment of the present disclosure. While FIGS. 6A-6C show steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particularly shown order or arrangement. Various steps of method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0109] At (602), the method 600 may include providing input data to a first machine learning model. For example, a computing system may provide the input data to the first machine learning model. As described herein, the input data may include at least one of free-form input and / or landing page content. The input data may include natural language input.

[0110] In some implementations, the first machine learning model can be trained using a knowledge distillation training method. For example, the knowledge distillation training method can include training the first machine learning model (e.g., a student model) using a pre-trained larger teacher model (e.g., a complex large-scale language model). In some implementations, the first machine learning model can be trained based at least in part on output from a pre-trained second machine learning model.

[0111] In some implementations, the pre-trained second machine learning model can be a large-scale language model. The pre-trained second machine learning model can be tuned using one or more prompts. For example, the system can tune the model to generate suggested assets from free-form input using few-shot learning. For example, the input can include a prompt. The prompt can include a headline and / or a description. For example, a headline can include a new Tycoon game, "Try out a dog-friendly cafe and build a dog paradise," or "The coolest dog app in the US." A description can include "Serve the best drinks at the friendliest dog cafe and Dog Cafe Tycoon," "Food for hungry customers," and "Grow your dog cafe empire."

[0112] At (604), method 600 can include obtaining output data indicating one or more suggested content item components. For example, a computing system can obtain output data indicating one or more suggested content item components. For example, the content item components can include at least one of a headline or a description. For example, the model output can include generated text output. For example, the text could be, "Dog Cafe Tycoon is a simulation app. This is a brief introduction to the app: a new tycoon game where you build a dog paradise. Serve drinks at the dog cafe and help develop a dog cafe empire. The main topic keyword for this application is dog cafe. The target country is the United States." Tuning the large-scale language model can improve the model's ability to generate more nuanced content item suggestions in the future. For example, generating advertisements can include the need to understand the nuances associated with a business, specific user goals, etc. This language processing can differ from other language processing applications.

[0113] At (606), the method 600 may include evaluating the first machine learning model based at least in part on the output data. For example, a computing system may evaluate the first machine learning model based at least in part on the output data. As described herein, the evaluation of the first machine learning model may be performed based at least in part on determining a quality score associated with the output data of the first machine learning model.

[0114] At (606A), method 600 may include obtaining data indicative of a quality score associated with the output data of the first machine learning model. For example, a computing system may obtain the data indicative of a quality score associated with the output data of the first machine learning model. By way of example, the quality score may be automatically generated by the system and / or manually entered by a user.

[0115] At (606B), the method 600 may include comparing a quality score associated with the output data of the first machine learning model to a threshold quality score. For example, the computing system may compare the quality score associated with the output data of the first machine learning model to the threshold quality score. As an example, the computing system may continuously evaluate the model. The computing system may determine that the quality of the output data is acceptable. For example, in a training implementation, the output data may be obtained for input data previously tagged and / or processed by a pre-trained model. The system may compare the actual output data to the intended output data.

[0116] At (608), the method 600 may include determining that the first machine learning model has a quality score that is greater than a threshold quality score. For example, the computing system may determine that the first machine learning model has a quality score that is greater than a threshold quality score. As described herein, the quality score may be associated with a quality of the output data.

[0117] At (610), the method 600 can include implementing the first machine learning model in a content creation flow in response to determining that the first machine learning model has a quality score above a threshold quality score. For example, the computing system can implement the first machine learning model in the content creation flow in response to determining that the first machine learning model has a quality score above a threshold quality score. The content creation flow can be associated with a third party that provides a platform for content creators to generate customized content items (e.g., websites, advertisements, search results for display that link to the generated content items). The content creation flow can include a user interface for generating the content items. The content creation flow can include providing for display a content assistant component of the user interface.

[0118] At (612), method 600 may include generating an initial user interface including a content assistant component. For example, a computing system may generate an initial user interface including a content assistant component. As described herein, the initial user interface may include a content assistant component. The content assistant component may be configured to obtain user input and, in response, generate suggested content (e.g., headlines, descriptions, advertisements, videos, images, sound bites, etc.). In some implementations, the content assistant component may include one or more input fields. For example, the content assistant component may include a set number of predetermined input fields. In some implementations, the content assistant component may include multiple progressively revealing input fields. In some implementations, the content assistant component may be configured to provide a conversational interface.

[0119] In some implementations, the user input may include text snippets, documents, images, handwriting, audio, etc. In some implementations, the user may not have a strong landing page associated with their website. In response, the computing system may initiate an input funnel to obtain the user input. Additionally or alternatively, the user input may include fine-grained control over the content item strategy. For example, the content item may be an advertisement and the strategy may correlate to a campaign. The system may obtain user input indicating the user's selection of one or more fine-grained controls, including requirements through which the system may interact and adapt to the campaign.

[0120] At (614), method 600 can include obtaining data indicative of user input. For example, a computing system can obtain the data indicative of the user input. The user input can include input obtained from a user by any means. The means can include touch input (e.g., via a touch-sensitive keyboard), voice input, etc.

[0121] At (616), the method 600 can include processing the data indicative of the user input with a first machine learning model. For example, the computing system can process the data indicative of the user input with the first machine learning model. Processing the data indicative of the user input can include converting data from the natural language input into a feature vector or other data structure that is ingested by the first machine learning model.

[0122] At (618), the method 600 can include obtaining output data from the first machine learning model that indicates one or more content item components. For example, the computing system can obtain output data from the first machine learning model that indicates one or more content item components. By way of example, the one or more content item components can include a headline, a description, a video, an image, a tagline, a sound bite, etc. The content items can be generated and / or obtained by the content assistant component.

[0123] At (620), the method 600 can include transmitting data that causes one or more content item components to be provided for display via a user interface. For example, the computing system can transmit data that causes one or more content item components to be provided for display via a user interface. For example, the content item components can be a headline, a description, etc., that are populated into one or more input fields associated with a content creation flow form.

[0124] At (622), the method 600 may include obtaining data indicating a user selection of approval of one or more content item components. For example, the computing system may obtain data indicating a user selection of approval of one or more content item components. For example, the computing system may obtain data indicating a user selection of "accept," "confirm," "create advertisement," or some other notification indicating approval of the content item components.

[0125] At (624), method 600 can include generating one or more content items in response to obtaining data indicating a user selection of approval of one or more content item components. For example, a computing system can generate one or more content items in response to obtaining data indicating a user selection of approval of one or more content item components. For example, a content item can be an advertisement that includes multiple content item components. In some implementations, a content item can include multiple headlines, descriptions, videos, images, taglines, etc. Based on the processed user input and / or other user data, the system can generate one or more content items using the one or more content item components.

[0126] 7 shows a flowchart diagram of an exemplary method 700 for performing in accordance with an exemplary embodiment of the present disclosure. While FIG. 7 shows steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particularly shown order or arrangement. Various steps of method 700 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0127] At (702), method 700 can include obtaining data indicative of input received from a user. For example, a computing system can obtain data indicative of input received from a user. The input from the user can include free-form input and / or landing page input. For example, the free-form input can include a user describing themselves and / or an associated business using natural language. The user input can include a uniform resource locator (URL). In some examples, the URL can be associated with a website and / or a landing page for the website associated with the user. In some implementations, the system can obtain data from the landing page to generate one or more content item components and / or content items to suggest to the user.

[0128] At 704, the method 700 can include processing, with the machine learning model, data indicative of the input received from the user. For example, the computing system can process, with the machine learning model, data indicative of the input received from the user.

[0129] In some embodiments, method 700 may include training one or more machine learning models for implementation in the content creation flow. For example, a computing system may train one or more machine learning models for implementation in the content creation flow. By way of example, the one or more machine learning models may be trained in any manner. For example, the one or more machine learning models may be trained using a knowledge distillation training method. In some embodiments, training the one or more machine learning models may include inputting labeled data into a first machine learning model. In some embodiments, the system may obtain output data from the first machine learning model relative to output data from an intended machine learning model. The system may adjust the first machine learning model based on comparing the output data of the first machine learning model and the intended machine learning model.

[0130] In some embodiments, the labeled data can include data output by a second machine learning model, where the second machine learning model is a pre-trained model. The output obtained by the intended machine learning model can include annotated data. For example, the system can annotate the input data with a desired outcome based on the output when the same input data is processed by the intended machine learning model. In some embodiments, the annotated data can be a silver training set. By way of example, the labeled data can include at least one of business and product descriptions, proxy data from websites associated with content creators, human-curated data, or free-form input.

[0131] In some implementations, training one or more machine learning models can include inputting unlabeled data to a first machine learning model and a pre-trained second machine learning model. The training can include obtaining output data from the first machine learning model and the second machine learning model. The training can include adjusting the first machine learning model based on comparing the output data of the first machine learning model with the output of the second machine learning model. For example, the system can perform loss regression on the outputs of the first machine learning model and the second machine learning model. In some examples, the system can continuously update and reprocess the input until a difference between the outputs of the first machine learning model and the second machine learning model is within an acceptable threshold.

[0132] In some implementations, the method can include implementing a machine learning model in a content creation flow. For example, a computing system can implement the machine learning model in the content creation flow. The content creation flow can be associated with a third party that provides a platform for content creators to generate customized content items (e.g., websites, advertisements, search results for display that link to the generated built content items).

[0133] In some implementations, the method can include generating an initial user interface that includes the content assistant component. For example, a computing system can generate an initial user interface that includes the content assistant component. In some implementations, the initial user interface can be a graphical user interface.

[0134] At (706), method 700 may include obtaining output data from the machine learning model indicating one or more content item components. For example, a computing system may obtain output data from the machine learning model indicating one or more content item components. For example, the content item components may include one or more suggestions for content. For example, the content may include suggested headlines, descriptions, images, videos, taglines, etc. In some implementations, the content may include generated advertisements.

[0135] At (708), method 700 can include transmitting data that causes one or more content item components to be provided for display via a user interface. For example, a computing system can transmit data that causes one or more content item components to be provided for display via a user interface. For example, the system can populate existing user input fields with content generated by the model. In some implementations, the system can generate an entire suggested content item (e.g., an advertisement).

[0136] At (710), method 700 can include obtaining data indicating a user selection of approval of one or more content item components. For example, a computing system can obtain data indicating a user selection of approval of one or more content item components. As an example, the system can obtain data indicating a selection of a user interface element indicating a user selection of "accept," "confirm," "create ad," or some other notification indicating approval of the content item components.

[0137] At (712), the method 700 can include generating one or more content items in response to obtaining data indicating a user selection of approval for one or more content item components. For example, a computing system can generate one or more content items in response to obtaining data indicating a user selection of approval for one or more content item components.

[0138] 8-10 illustrate exemplary content assistant components of a user interface according to an exemplary embodiment of the present disclosure. A computing system can be provided for displaying the content assistant component via the user interface. The content assistant component can be displayed in a variety of formats. For example, the formats can include a set number of free-form text entry fields (e.g., as shown in FIG. 8), one or more progressive disclosure fields (e.g., as shown in FIG. 9), and / or a conversational interface (e.g., as shown in FIG. 10).

[0139] FIG. 8 shows an exemplary content assistant component 800 including a set number of freeform text entry fields 805, 810, and 815. For example, the first freeform text entry field 805 may include a prompt for the user saying, "Please describe the product or service you would like to advertise." The computing system may obtain data indicative of the user's input into the freeform text entry field 805. The freeform text entry may be natural language input by the user. The second freeform text entry field 810 may include a prompt for the user saying, "Please describe what makes your product or service unique." The computing system may obtain data indicative of the user's input into the freeform text entry field 810. The third freeform text entry field 815 may include a prompt for the user saying, "What URL should this advertisement link to?" In some implementations, the generated content may be an advertisement. For example, the user may provide a uniform resource locator (URL) that directs the user to a website associated with the user. For example, the user may be associated with a business called Puppy Things. The URL associated with the business can be "http: / / www.puppythings.com".

[0140] 9 shows an example content assistant component 900 that includes one or more staged disclosure fields. For example, the content assistant component 900 can include a first staged disclosure field 905. The first staged disclosure field 905 can be presented when the content assistant component 900 is first started. If a user provides input into the first staged disclosure field 905, the system can provide for displaying a second staged disclosure field 910. After the system determines that the user has entered data into the second staged disclosure field 910, the system can provide for displaying a third staged disclosure field 915.

[0141] In some implementations, the staged disclosure fields 905, 910, and / or 915 can be predetermined fields. Additionally or alternatively, the staged disclosure fields can be determined based on processing information obtained as user input into previously disclosed staged disclosure fields. For example, the first staged disclosure field 905 may indicate, "Tell me about your business." If the user provides a short, one- or two-word, generic sentence answer, the second staged disclosure field 910 may indicate a more targeted prompt than if the user provided a multi-paragraph answer into the first staged disclosure field 905. When the user has finished providing input into one or more staged disclosure fields, the system can obtain data indicating that the user selected the submit element 920. In response, the system can process the user input provided via the staged disclosure fields and provide content suggestions via a content creation flow.

[0142] FIG. 10 illustrates an exemplary content assistant component 1000 that includes a conversational user interface. In some implementations, the content assistant component 1000 can include a round-trip communication session between a user and the content assistant. For example, the content assistant can provide an initial prompt 1005 to the user. The user can provide a response 1010. The computing system can facilitate the transmission of multiple communications between the user and the content assistant. As described with respect to FIG. 5, the computing system can provide follow-up questions, process data indicative of user input associated with the user's response, and generate suggested content (e.g., via a machine-learned natural language processing model) based on processing of the user input.

[0143] 11 illustrates one exemplary system 1100 for generating predicted performance improvements associated with proposed content items and for providing an updated user interface including the proposed content items and their respective predicted performance improvements, according to an exemplary embodiment of the present disclosure. The exemplary system 1100 includes a computing system 1102. The computing system 1102 can be any type of system of one or more computing devices (e.g., client computing system 102, server computing system 104, etc.). The computing device may be, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a node of a distributed computing device, a virtual instance hosted on a shared server, or any other type of computing device. In some embodiments, the computing system 1102 includes multiple computing devices interconnected via a network or distributed in other interoperable manner. For example, the computing system 1102 may include a server for serving content over a network (e.g., network 108). For example, computing system 1102 may include a web server for hosting web content and for collecting data about the web content (e.g., for receiving, monitoring, generating, or otherwise processing data about the web content, such as usage, downloads, and / or interactions with the web content).

[0144] The computing system 1102 may include processor(s) 1112 and memory 1114. The one or more processors 1112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 1114 may include one or more computer-readable storage media, which may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1114 may store data 1116 and instructions 1118 that are executed by the processor 1112 to cause the computing system 1102 to perform operations.

[0145] In some implementations, the computing system 1102 can store or otherwise implement one or more machine learning models of a machine learning model framework. In some embodiments, the content assistant component 1120 includes a multi-model machine learning model framework. In some embodiments, the content assistant component 1120 does not include any combination of a machine learning model framework and / or a custom language model 1122, a language model 1126, or a model 1125. The machine learning model framework can include a machine-learned custom language model 1122 (e.g., with learnable weights 1124), a machine-learned language model 1126 (e.g., with learnable weights 1128), or a model 1125. The model 1125 can include a generative model 1129 or an action model 1127. The generative model 1129 can generate images, audio, text, audiovisual, or any other content. The action model 1127 can be capable of interfacing with one or more other models or components to perform actions (e.g., make recommendations, interact with other websites). The model 1125 may also have associated learnable weights (not shown in FIG. 11).

[0146] In some embodiments, the content assistant component 1120 may implement a single model that implements the custom language model 1122 (e.g., by combining one or more aspects of the custom language model 1122 and / or the language model 1126, training a single model to directly obtain a desired output, etc.) for content suggestions, content ranking, content generation, general advertising content or recommendations, general analytics content or recommendations, and / or any combination. One or more of the machine learning models may be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including nonlinear and / or linear models. The neural networks may include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some exemplary machine learning models may leverage attention mechanisms, such as self-attention. For example, some exemplary machine learning models may include multi-head self-attention models (e.g., Transformer models). In some examples, the model 1125 may include a multimodal model that can understand both image, text, or other input. In some examples, the model 1125 may include multiple models that can use a language model (e.g., a large-scale language model) to better understand the intent of the acquired data (e.g., user input data 1138).

[0147] An embodiment of the exemplary system 1100 may be configured to process data 1130, as shown in FIG. 11 . In response to processing the data 1130, the computing system 1102 may provide output 1150. The output 1150 may include generated suggested content item components 1155, natural language responses 1157, or content campaign recommendations 1159. The suggested content item components 1155 may include multiple components or elements used to generate a content item. For example, the suggested content item components may include text (e.g., a headline or description), images (e.g., retrieved from a database or generated in real time), audiovisual, interactive, or other types of content. The natural language responses 1157 may include text generated by a language model based on determining the intent of the retrieved user input and providing a natural language response that provides an answer, recommendation, or other response as output 1150. The content campaign recommendations 1159 may include recommendations for analyzing historical content performance data (e.g., advertising performance, cost per click, return on advertising spend), recommendations for adjusting content campaign parameters based on the analyzed data (e.g., adjusting costs across different media channels, adjusting maximum or minimum bid parameters), or any other recommendations associated with the content campaign.

[0148] The suggested content item components 1155, natural language responses 1157, or content campaign recommendations 1159 can be provided for display via a user interface 1170 of a client device associated with the client computing system 1160. The client computing system 1160 can include processor(s) 1162 and memory 1164. The one or more processors 1162 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be a single processor or multiple operably connected processors. The memory 1164 can include one or more computer-readable storage media, which can be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1164 can store data 1166 and instructions 1168 that are executed by the processor 1162 to cause the client computing system 1160 to perform operations.

[0149] The output 1150 can include suggested content item components 1155. The suggested content item components 1155 can include, for example, suggested headlines, descriptions, follow-up communications, videos, images, audio, soundbites, etc. In some examples, the suggested content item components 1155 can be creative assets in the form of freeform input or that can be entered into a headline. For example, the custom language model 1122 can obtain user input and can collect additional user input to provide follow-up communications to generate further suggested content item components 1155. By way of example, the custom model can be trained using distillation techniques and a previously trained language model (e.g., a previously trained large-scale language model).

[0150] In some embodiments, the data 1130 is obtained via an input interface of a computing system (e.g., computing system 1102 and / or client computing system 1160). For example, in some embodiments, the content assistant component 1120 can be configured to process the data 1130 as input to the computing system 1102. In some embodiments, the data 1130 can be implicit in the structure and / or configuration of the content assistant component 1120.

[0151] In some implementations, the content suggestion generator model is trained to receive a set of input data (e.g., data 1130) describing user input and, in response, provide output data (e.g., suggested content item component 1155) indicating one or more suggested content items to be rendered via a user interface. For example, Figure 11 shows an instance of a rating.

[0152] For example, the evaluation component 1140 can perform offline evaluation and / or online evaluation (e.g., real traffic evaluation). The offline evaluation can include using a human evaluation platform. The human evaluation platform can include templates and / or rubrics to standardize the evaluation. The online evaluation can include pilot and / or live traffic experiments to capture online metrics. In some implementations, the evaluation component 1140 can include automated evaluation and / or manual evaluation. In some implementations, the evaluation component 1140 can include tagging and / or labeling data as training data. The system can determine a quality score for the model. The system can compare the quality score of the model to a threshold quality score. If the quality score of the model is above the threshold quality score, the system can determine that the model is ready to be used. If the quality score of the model is below the threshold quality score, the system can determine that additional training and / or tuning is required before the model can be used.

[0153] In some implementations, the input data may include one or more features associated with an example or example. In some implementations, the one or more features associated with the example or example may be organized into a feature vector. In some implementations, the output data may include one or more predictions. A prediction may also be referred to as an inference. Thus, given features associated with a particular example, a machine learning model may output a prediction for such example based on the features.

[0154] The machine learning model may be or include one or more of a variety of different types of machine learning models. In particular, in some implementations, the machine learning model may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0155] In some implementations, data 1130 includes aggregate data 1132, proxy data 1134, and / or user-input data 1138. In some embodiments, data 1130 can include instances of virtually any kind or type of data that can describe various phenomena. Generally, an instance refers to a particular subject or a set of one or more data values ​​grouped together to describe a subject. For example, an instance can be a feature vector. An instance can be associated with image data (e.g., a feature vector of an image, a hashed image, etc.). An instance can be associated with a measurement or other data collection event (e.g., at a particular time, or of a particular subject, or using a particular device, or from a particular perspective, etc.). An instance can be associated with a network session, such as a set of interactions with a web server. In some embodiments, an instance can be associated with a user's interactions with web content (e.g., anonymous or identified).

[0156] In some embodiments, the custom language model 1122 can be trained using knowledge distillation. For example, the custom language model 1122 can be a student model, and the large-scale language model 1126 can be a teacher model. A knowledge distillation training process can be performed to train the custom language model 1122 based on the pre-trained large-scale language model 1126. For example, the large-scale language model can be a large-scale model generated and trained using a large amount of computing resources. For example, the large-scale language model can be a language model consisting of a neural network with many parameters (e.g., millions, billions, or trillions of weights). The language model can be trained on a large amount of unlabeled text by self-supervised learning or semi-supervised learning. To save resources, knowledge distillation can be used to train the custom language model 1122 and produce results similar to the pre-trained large-scale language model 1126.

[0157] In some implementations, the models of the content assistant component 1120 can be tuned and / or trained using prompts. For example, the computing system can obtain prompts that indicate sample inputs and outputs. The sample inputs and outputs can be used to tune the models to be familiar with the types of inputs and outputs desired for a particular implementation. For example, a particular implementation can include generating advertisements. Generating advertisements requires a nuanced understanding of the business, business goals, products, etc. In this manner, by prompting one or more models (e.g., custom language model 1122 and / or large-scale language model 1126), the computing system can tune the models to provide higher quality output (e.g., output 1150 that can be evaluated using evaluation component 1140). In some examples, after evaluation component 1140 performs the evaluation, the data can be aggregated and used as data 1130 for further model training.

[0158] In some implementations, the content assistant component 1120 can be a machine learning model. In some implementations, training and tuning can include using data 1130. The aggregated data 1132 can include data obtained from one or more user sessions aggregated by a system (e.g., the computing system 202). The proxy data 1134 can include, for example, output data obtained from the content assistant component 1120. In some implementations, the proxy data 1134 can be tagged and / or labeled to represent sample inputs and / or outputs. The user input data 1138 can include user-generated inputs having inputs and / or outputs associated with desired content items and generated content items.

[0159] The trained model can be used to generate one or more suggested content items and / or update the structured user interface and / or construction workflow. In some embodiments, a simulated user model can be trained using the training data to generate simulated inputs according to patterns learned from the training data. The simulated user model, in some examples, can provide a simulated environment in which a reinforcement learning agent, a notification component subsystem, can be trained.

[0160] 12 shows an example content assistant flow 1200 for determining when to provide a prompt for display to a user to obtain additional user input via a user interface. At 1202, an initial prompt can be provided for display via a user interface. At 1205, the system can determine whether user input has been obtained. If user input has been obtained, the system can proceed to 1210. At 1210, the content assistant can obtain one or more keyword recommendations. At 1215, the content assistant can generate suggested content. Following 1215, the system can proceed to 1220. If user input is not obtained at 1205, the computing system can proceed to 1220.

[0161] At 1220, the computing system may determine whether a prompt for more information needs to be provided for display via the user interface (e.g., via the content assistant component). For example, at 1225, the computing system may determine a content score and compare the content score to a threshold score value. For example, the score may be a numeric score or other form of rating. The score may be determined based on the number of headlines, descriptions, images, or other content elements selected or embedded within the content campaign interface. At 1230, the computing system may determine whether the content score exceeds a threshold. If the content score does not exceed the threshold, the computing system may proceed to step 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0162] If the content score does not exceed the threshold, the system may proceed to 1235. At 1235, the system may determine whether at least X minutes have elapsed since the last prompt was provided for display via the user interface. X may be any number of minutes (or may be a number of seconds) that may be automatically updated. Additionally or alternatively, X may be manually selected or entered. If X minutes have not elapsed, the computing system may proceed to 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0163] If X minutes have passed, the computing system may proceed to 1240. At 1240, the computing system may determine whether question Y has been provided for display via a user interface. For example, the computing system may maintain a log of the current communication session. The computing system may determine the meaning of question Y and compare it to the log of the current communication session. If the computing system determines that question Y has been provided for display (e.g., asked of a user), the computing system may proceed to 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0164] If the computing system determines that question Y is not provided for display, the computing system may proceed to 1245. At 1245, the computing system may determine whether user input indicating typing has been obtained within Z seconds. Z may be any amount of time (e.g., seconds, minutes). For example, Z time may be automatically determined by the computing system based on a determined average amount of time between received inputs. Additionally, or alternatively, Z time may be set (e.g., by a user providing manual input). If the computing system determines that user input indicating typing has been obtained within Z seconds, the computing system may proceed to 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0165] If the computing system determines that user input indicating typing has not been obtained within Z seconds, the computing system may proceed to 1250. At 1250, the computing system may provide a prompt for display via a user interface. The prompt may include a message prompting the user to provide additional user input. The additional user input may be free-form input, a selection of a selectable screen element, or any other user input.

[0166] As discussed herein, at 1255, the content assistant can periodically poll the client device for captured user input. The computing system can proceed to 1260. At 1260, the content assistant component can determine that user input has been captured. In response, the content assistant can determine the intent of the user input. For example, the computing system can determine the semantic intent through natural language processing (e.g., via a large-scale language model).

[0167] The computing system may determine that the content assistant does not have the capability to provide recommendations or perform actions related to the intent. In response, the computing system may proceed to 1265. At 1265, the content assistant may provide a message for display including an indication that it does not have the capability to assist with the determined intent. The computing system may proceed to step 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0168] The computing system can determine that the content assistant has the capability to provide recommendations or perform actions related to the intent. In response, the computing system can proceed to 1215, where the content assistant can generate suggested content as discussed herein.

[0169] 13 shows an example content assistant flow loop 1300 for providing recommendations for updating a content creation interface until the predicted score of the generated content exceeds a threshold. At 1305, the computing system can evaluate the obtained user input to determine whether a prompt to obtain user input needs to be provided for display. For example, the computing system can determine the intent of the obtained user input (e.g., a desire to generate a new content campaign, an inquiry regarding the performance of various settings of the content campaign) and evaluate whether a response needs to be provided for display via the user device.

[0170] At 1310, the computing system may determine whether a prompt to obtain additional user input was provided for display. If a prompt to obtain additional user input was not provided for display, the computing system may proceed to 1315. At 1315, the content assistant may periodically poll the client device for updated user input. If a prompt to obtain additional user input was provided for display, the computing system may proceed to 1320. At 1320, the computing system may determine whether a prerequisite model competency exists for the subject of the obtained user input. For example, the computing system may determine whether an answer or response can be generated in response to the user's input. If a prerequisite model competency does not exist, the computing system may proceed to step 1340. At 1340, the computing system may provide for displaying a general question for more information to be obtained via a user input component of the user interface.

[0171] If the requisite model competence exists for the subject matter, the computing system may proceed to 1325. At 1325, the computing system may determine whether one or more salient properties are missing. If the computing system determines that missing salient properties exist, the computing system may proceed to 1330. At 1330, the computing system may provide for displaying a question directed regarding the missing properties that is obtained via a user input component of the user interface. For example, the missing salient properties may include a headline, a business name, a location, or other relevant information related to the subject matter of the content item to be generated. The computing system may proceed to 1345 to obtain additional user input in response to the question provided for display via the user interface.

[0172] If the computing system determines that there are no missing salient properties, the computing system may proceed to 1340. At 1340, the computing system may provide for displaying a general question for more information to be obtained via a user input component of the user interface. Following 1340, the computing system may proceed to 1340.

[0173] 14A, 14B, and 14C show a flowchart diagram of an exemplary method 1400 for performing in accordance with an exemplary embodiment of the present disclosure. While FIGS. 14A-14C show steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the particularly shown order or arrangement. Various steps of method 1400 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0174] At 1402, the method 1400 can include obtaining, via a language model, a natural language input via a conversational campaign assistant interface. For example, the computing system can obtain, via a language model, a natural language input via a conversational campaign assistant interface. As described herein, the natural language input can include a URL, a description, a question, or other natural language input.

[0175] At (1404), method 1400 may include generating, by the custom language model, an output including the predicted user intent. For example, the computing system may generate, by the custom language model, an output including the predicted user intent. As described herein, the custom language model may be a model trained using distillation learning techniques. For example, the custom language model may be trained based on a previously trained large-scale language model.

[0176] In some examples, a custom language model can be trained to be an expert in a particular topic, which enables the custom language model to better determine predicted user intent based on both the current context of the session and the data on which the custom model was trained.

[0177] At 1406, method 1400 can include determining one or more actions to perform. For example, the computing system can determine one or more actions to perform. By way of example, the actions can include populating input fields, updating a user interface to open a new window, adjusting settings within the user interface, providing recommendations, visual indications of updates to input fields, or other actions.

[0178] As shown in FIG. 14B, in step (1406), the method 1400 may include steps (1418), (1420), and (1422).

[0179] For example, at 1418, step 1406 can include parsing the output generated by the custom language model. For example, the computing system can parse the output generated by the custom language model. For example, the output generated by the custom language model can include a user intent.

[0180] For example, in 1420, step 1406 can include determining an action associated with the output. For example, the computing system can determine the action associated with the output. By way of example, the user intent can include a specific action to be performed, a question, a request for recommendations, or other action.

[0181] For example, a user may ask for recommendations on bidding strategy settings to optimize cost or return on content spend (e.g., return on advertising spend). The settings may be tailored to the context of the user session. For example, recommendations may be based on previous input received in a conversation, information collected from websites associated with the user session, or other contextual information. In some examples, a user may request information on how to understand analytics data, request that data be exported in a particular format, or any other request. For example, a user may indicate the purpose for which the data needs to be exported. Requesting that data be exported for manual analysis in a spreadsheet may generate a different type of file than data exported for analysis via some other application. An application may be associated with a user device, a third-party service, or the like.

[0182] A content campaign can include multiple parameters. These parameters can include, for example, a desired audience (e.g., target), payment amount and performance measures (e.g., bid) for the content to be displayed, or content (e.g., creative) that can be displayed. Content campaigns associated with an organization can be large and managed by multiple individuals. There can be thousands of individual campaigns within an organization, which can include thousands of keywords, landing pages, business use cases, business units, etc. Business use cases can include customer acquisition, competitor targeting, new product launches, increased brand exposure, etc. Business units can include products and services, countries, languages, or geographic regions. The management of these campaigns can be disjoint, with no coordination between campaigns, which can result in overlapping content processing and display.

[0183] The content assistant component can be trained to provide recommendations for improving the allocation of computing resources. For example, the content assistant component can be trained to adjust, or provide recommendations for adjusting, various parameters associated with a content campaign to reduce redundant data transmission. Additionally, performance improvements can be made. For example, the content assistant component can obtain information about performance metrics, determine that changes in the metrics are attributable to parameter changes or external causes, and provide recommendations or auto-adjust parameters across an organization's content campaigns.

[0184] For example, the content assistant component can monitor one or more performance metrics. In response to determining a statistically significant change in the performance metrics, the content assistant component can provide a notification for display. The content assistant component can determine an explanation for the change in the performance metrics. The content assistant component can automatically update a user interface to display the updated information. For example, the updated user interface can include providing recommendations for updating parameters associated with the content campaign, providing a message indicating the reason for the change in the performance metrics, or automatically adjusting the parameters and providing a notification summarizing the changes made. The explanation for the change in the performance metrics can be determined based on obtaining multiple signals. The signals can include, for example, changes in query volume, changes in content elements (e.g., keywords, descriptions, creative assets), or changes in the display of the organization's content items compared to content items offered for display associated with other organizations.

[0185] Adjusting parameters can include adding or removing content elements (e.g., descriptions, keywords, images, audio, video), adjusting bidding strategies, or other adjustments. As described herein, adjustments can be made automatically and updated continuously or periodically. The content assistant component can continuously review content campaign performance metrics and provide recommendations and updates in real time. In some implementations, changes can include reducing the number of keywords used, determining the intent of one or more campaigns to consolidate campaigns, determining that a campaign is heterogeneous and needs to be split into multiple campaigns, or recommending a different campaign type.

[0186] For example, at 1422, step 1406 can include generating an action data structure including executable instructions that cause a processor to perform an operation associated with completing the action. For example, the computing system can generate an action data structure including executable instructions that cause a processor to perform an operation associated with completing the action. The action can include performing an operation within the content creation structured interface or can include performing an operation by an unrelated application.

[0187] The action data structure may include converting data generated by the custom language model into a format that can be input into the content creation structured interface. This may include determining what data to populate which input fields, what data to provide for display to the user, what data (e.g., metadata) should be hidden from the user, etc. This allows the output generated by the custom language model to be utilized by both the content creation structured interface and the content assistant component of the conversational campaign interface.

[0188] At 1408, method 1400 can include determining a natural language response. For example, the computing system can determine the natural language response. By way of example, the natural language response can include an indication of fields updated in the content creation structured interface, a number of recommended content elements to add to the content creation structured interface, a generated summary, etc.

[0189] As shown in FIG. 14C, in step (1408), the method 1400 may include steps (1424) and (1426).

[0190] For example, at 1424, step 1408 may include parsing the output generated by the custom language model. For example, the computing system may parse the output generated by the custom language model. The output generated by the custom language model may include data extracted from a website associated with a URL provided by a user. The extracted data may be generated in a format that is not easily accessible or understandable when displayed in raw format. However, the computing system may parse the generated output to obtain relevant portions of the output for use in generating response data.

[0191] For example, at 1426, step 1408 can include generating a response data structure including a natural language response to the captured natural language input. For example, the computing system can generate a response data structure including a natural language response to the captured natural language input. For example, the computing system can take the extracted data from step 1428 and convert the data into natural language output that can be understood by a user.

[0192] At 1410, the method 1400 can include sending an action data structure to the action component that includes executable instructions that cause the action component to automatically perform operations associated with completing the action. For example, a computing system can send an action data structure to the action component that includes executable instructions that cause the action component to automatically perform operations associated with completing the action.

[0193] An action data structure can contain data to be populated into one or more fields of a content creation structured interface.

[0194] The action component can include a content campaign performance model. The content campaign performance model can be configured to ingest data about content campaigns. The content campaign performance model can be a generic model that provides information about content campaign performance for multiple users or campaign types. For example, the content campaign performance model can provide general advice and strategies for improving content performance.

[0195] The action component can include a content campaign analytics model. The content campaign analytics model can be configured to provide insights into the performance of a content campaign. For example, the data can include conversion data, view data, click data, cost data, or any other data indicative of the performance of a content campaign. For example, the natural language input can be a question about the number of views of a particular content item or about which content items performed better than other content. The content campaign analytics model can provide recommendations customized to a particular content campaign or sub-campaign being managed via the user interface.

[0196] The action component can include a bid strategy model. The bid strategy model can provide personalized recommendations for spend allocation based on previous performance. The previous performance can be associated with a particular return (effectiveness) user and based on historical data. Additionally, or alternatively, the bid strategy model can obtain additional user input to learn more about the user's associated business, customer base, or other baseline knowledge used in generating recommendations.

[0197] The action component may include a generative model. The generative model may take an action data structure as an input prompt and generate output including creative assets. For example, the natural language input may include a uniform resource locator (URL). The action component may include a generative model that generates a summary of information parsed from a page (e.g., a website) associated with the URL. The generative model may be a machine learning model. The generative model may be trained using knowledge distillation techniques. Additionally or alternatively, the creative assets may include images, sounds, videos, or other forms of content.

[0198] At 1412, method 1400 can include transmitting a response data structure to the conversational campaign assistant interface, the response data structure including the natural language responses provided for display to the user via the conversational campaign assistant interface. For example, the computing system can transmit a response data structure to the conversational campaign assistant interface, the response data structure including the natural language responses provided for display to the user via the conversational campaign assistant interface.

[0199] At 1414, method 1400 may include, following transmitting the action data structure and the response data structure, obtaining user input indicating validation of the action data structure or the response data structure. For example, a computing system may, following transmitting the action data structure and the response data structure, obtain user input indicating validation of the action data structure or the response data structure. In some implementations, validation may be performed automatically. Additionally or alternatively, validation may be performed manually. Exemplary approaches to validation are discussed, for example, in FIGS. 2, 3, and 11.

[0200] At (1416), method 1400 may include updating the custom language model based on the user input. For example, the computing system may update the custom language model based on the user input. As described herein, updating the custom language model may include automatically adjusting the model (e.g., weights, parameters) to change the output produced by the model. The model may be automatically updated in real time. Additionally or alternatively, the model may be updated (e.g., trained) offline.

[0201] 15 through 20 illustrate exemplary user interfaces including a content assistant component, according to exemplary embodiments of the present disclosure. A computing system can be provided to display the content assistant component along with a content creation structured interface via a user interface. The content assistant component and the structured interface can include multiple interactive selectable elements. These elements can include clickable buttons, input fields, or other similar elements. A user can provide input via the content assistant component. The computing system can process the input to determine one or more actions (e.g., forms to fill or buttons to select) to complete in the content creation structured interface and to determine a natural language response to provide for display via the content assistant component. In some examples, the one or more actions can additionally or alternatively include additional actions outside of the content creation interface. Each of the following figures describes features and functionality of the exemplary user interfaces described herein.

[0202] FIG. 15 shows an example user interface 1500 including a content assistant component 1505 and a content creation structured interface 1510. The content assistant component 1505 can obtain user input 1515 including a URL of a website associated with a user (or the user's business). In response to obtaining the user input 1515 including the website URL, the content assistant component can generate a natural language response 1520. The natural language response 1520 can include a summary of the content of the website associated with the URL, generated by the content assistant component. For example, the content assistant component can parse the website associated with the URL to generate the natural language response 1520. In response to a user providing a URL, a computing system can generate a summary of the content of the website. The content assistant component 1505 can provide the summary to the user for display and request that the user provide input indicating that the summary is accurate. For example, the user can select an accept button, enter a message or speak to text, or provide other input. The system can obtain the input and update the user interface 1500 accordingly.

[0203] The content assistant component can be displayed in a variety of formats, including a set number of free-form text entry fields (e.g., as shown in FIG. 8), one or more progressive disclosure fields (e.g., as shown in FIG. 9), and / or a conversational interface (e.g., as shown in FIG. 10).

[0204] FIG. 16 shows an example user interface 1600 including a content assistant component 1605 and a content creation structured interface 1610. The content assistant component 1605 can obtain user input and provide for displaying a generated response as described herein. The content creation structured interface 1610 can include one or more interactive components. For example, the components can include freeform input or selectable elements. For example, user input indicating a selection of one of the fields can cause the selected field to expand. In some examples, the fields can evolve based on direct user input or input obtained via the content assistant component 1605 of the user interface 1600. For example, the content assistant component can obtain user input including a description of a product or service offered by a company associated with the user. The content assistant component 1605 can process the input and generate an action or a natural language response. For example, the natural language response can be provided for display via the content assistant component 1605, and the user interface 1600 can be automatically updated to populate with one or more recommended content elements.

[0205] As shown in FIG. 17 , the user interface 1700 can be automatically updated to include populated interactive user interface elements. For example, the user interface 1700 can include a content creation structured interface 1710. The content creation structured interface can include multiple keyword suggestions. The keyword suggestions can be pre-populated based on the content assistant component 1705 processing the captured input. The content assistant component 1705 can input the captured user input into a language model to determine the intent associated with the input. This can include determining the subject matter of the input, extracting keywords, parsing the input to determine and extract the most relevant portions, parsing a website provided by the user to determine the meaning of the additional input in the context of the website's content, or other related determinations. The content assistant component 1705 can automatically update to provide updated prompts via the user interface. For example, the updated prompt can be an answer to a question, an indication of which field in the user interface 1700 was populated, a request for additional information, a targeted question, a general question, or any other message or inquiry.

[0206] FIG. 18 illustrates an exemplary user interface 1800. The user interface 1800 may include a content assistant component 1805 and a content creation structured interface 1810. The content assistant component 1805 may automatically update to provide additional messages or selectable elements for display. For example, as shown in FIG. 18, the selectable input may include a “Review Images” button, and when this button can be selected or clicked, the user interface 1800 may update to display one or more images that can be selected to be added to the content item. The content creation structured interface 1810 may include multiple sections. For example, the sections may include a middle section having a content score and preview of the content item. The content score may be generated based on the number of completed input fields in the content creation structured interface or the quality of the input in the completed input fields. The completed input fields may include multiple headlines that may be generated by the content assistant component.

[0207] 19 illustrates an example user interface 1900. The user interface 1900 can include a content assistant component 1905 and a content creation structured interface 1910. The content creation structured interface 1910 can include multiple sections. For example, the sections can include a middle section having a content score and preview of the content item. The content score can be generated based on the number of completed input fields in the content creation structured interface or the quality of the input in the completed input fields. The completed input fields can include multiple descriptions that can be generated by the content assistant component. The input fields can include multiple images selected to be utilized to generate the content item.

[0208] The content assistant component 1905 can provide a plurality of selectable content elements. For example, the selectable elements can include a plurality of headlines, descriptions, images, or other content elements. The computing system can obtain user input via the user interface 1900. For example, the user input can indicate a selection of one or more content elements to be added to the content creation structured interface 1910. In response to receiving the user input via the user interface 1900, the computing system can automatically update the user interface to incorporate the selected content elements from the content assistant component 1905 into the content creation structured interface 1910.

[0209] FIG. 20 shows an exemplary updated user interface 2000. The user interface 2000 can include a content assistant component 2005 and a content creation structured interface 2010. The content assistant component 2005 can be updated to include an indication of a selected content element. For example, when a content element is selected, a visual indicator such as a check mark can replace the additional element. Additionally, or alternatively, the content creation structured interface 2010 can be updated to include multiple selected content elements. Thus, interactions with the content assistant component 2005 can directly affect the content creation structured interface. This can include a content strength score, a preview, or adjustments to multiple input fields.

[0210] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as operations performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0211] While the subject matter of the present disclosure has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided by way of explanation and not by way of limitation of the present disclosure. Those skilled in the art, once they arrive at the foregoing understanding, will be able to readily create modifications, variations, and equivalents to such embodiments. Accordingly, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with other embodiments to yield still other embodiments. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents.

[0212] The steps shown and / or described are merely exemplary and may be omitted, combined, and / or performed in an order other than that shown and / or described. The numbering of the steps as shown is for ease of reference only and does not imply that any particular order is necessary or preferred.

[0213] The functions and / or steps described herein may be embodied in computer-usable data and / or computer-executable instructions, executed by one or more computers and / or other devices to perform one or more functions described herein. Generally, such data and / or instructions include routines, programs, objects, components, data structures, etc., that, when executed by one or more processors of a computer and / or other data processing device, perform particular tasks and / or implement particular data types. The computer-executable instructions may be stored on a computer-readable medium, such as a hard disk, optical disk, removable storage medium, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. It will be understood that the functionality of such instructions may be combined and / or distributed as desired. Additionally, functionality may be embodied in whole or in part in firmware and / or hardware equivalents, e.g., integrated circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. Certain data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated as being within the scope of the computer-executable instructions and / or computer-usable data described herein.

[0214] Although not required, those skilled in the art will appreciate that various aspects described herein may be embodied as methods, systems, apparatus, and / or one or more computer-readable mediums storing computer-executable instructions. Accordingly, aspects may take the form of entirely hardware embodiments, entirely software embodiments, entirely firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0215] As described herein, the various methods and acts may be operable across one or more computing devices and / or networks. Functions may be distributed in any manner or may be located on a single computing device (e.g., a server, a client computer, a user device, etc.).

[0216] Aspects of the present disclosure have been described with respect to exemplary embodiments thereof. Numerous other embodiments, modifications, and / or variations within the scope and spirit of the appended claims will occur to those skilled in the art from a consideration of this disclosure. For example, one skilled in the art will recognize that the steps shown and / or described can be performed in a sequence other than that recited, and / or that one or more of the illustrated steps may be optional and / or combined. The features of any and all of the following claims can be combined and / or rearranged in any possible manner.

[0217] While the subject matter of the present disclosure has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided by way of explanation and not by way of limitation of the present disclosure. Those skilled in the art, upon arriving at the foregoing understanding, may readily create modifications, variations, and / or equivalents to such embodiments. Accordingly, the disclosure of the subject matter does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated and / or described as part of one embodiment may be used with other embodiments to yield still other embodiments. Accordingly, the present disclosure is intended to cover such modifications, variations, and / or equivalents.

Claims

1. A computing system, comprising: one or more processors; and one or more non-transitory computer-readable storage media having executable instructions stored thereon to cause the one or more processors to perform operations, the operations comprising: training a first machine learning model using knowledge distillation, wherein the first machine learning model is a student model and a large-scale language model tuned to generate content is a parent model; evaluating performance of the trained first machine learning model by comparing a quality score associated with output data of the first machine learning model to a threshold quality score; determining that the first machine learning model has a quality score that exceeds the threshold quality score; responsive to determining that the first machine learning model has the quality score above the threshold quality score, implementing the first machine learning model in a content creation flow; Including, implementing the first machine learning model in a content creation flow; obtaining input data including natural language input; generating an output including the one or more suggested content item components; and transmitting instructions that, when executed, cause a client device to provide the one or more suggested content item components for display via a user interface.

2. generating an initial user interface including a content assistant component; obtaining data indicative of user input; processing data indicative of the user input with the first machine learning model; obtaining output data from the first machine learning model indicative of one or more content item components; transmitting data that causes the one or more content item components to be provided for display on a user interface; obtaining data indicative of a user selection of approval for the one or more content item components; generating one or more content items including a plurality of the content item components in response to obtaining the data indicating the user selection of the approval of the one or more content item components; The computing system of claim 1 , comprising:

3. The computing system of claim 2 , wherein the content assistant component includes one or more input fields.

4. The computing system of claim 1 , wherein the first machine learning model is trained using a knowledge distillation training method.

5. The computing system of claim 4 , wherein the first machine learning model is trained based at least in part on output from a pre-trained second machine learning model.

6. The computing system of claim 5 , wherein the pre-trained second machine learning model is a large-scale language model.

7. The computing system of claim 6 , wherein the pre-trained second machine learning model is tuned using one or more prompts.

8. The computing system of claim 1 , wherein the input data comprises at least one of a free-form input or landing page content.

9. The computing system of claim 8 , wherein the input data comprises natural language input.

10. The computing system of claim 2 , wherein the one or more content item components include at least one of a headline or a description.

11. 1. A computer-implemented method comprising: generating an initial user interface including a content assistant component; obtaining data indicative of input received from a user; processing the data indicative of the input received from the user by a machine learning model interfacing with the content assistant component, wherein the machine learning model is trained using knowledge distillation, the machine learning model being a student model and a large-scale language model tuned to generate content being a parent model; and evaluating a performance of the machine learning model based on a quality score of output data obtained from the machine learning model; obtaining output data from the machine learning model that interfaces with the content assistant component, the output data being indicative of one or more content item components; transmitting data including instructions that, when executed, cause the one or more content item components to be provided for display via an updated user interface; obtaining data indicative of a user selection of approval for the one or more content item components; generating one or more content items in response to obtaining the data indicating the user selection of the approval of the one or more content item components; A computer-implemented method comprising:

12. 12. The computer-implemented method of claim 11, comprising training the machine learning model using a knowledge distillation training method.

13. The machine learning model, inputting the labeled data into a first machine learning model; obtaining output data from the first machine learning model; comparing the output data from the first machine learning model with output data from a second machine learning model; adjusting the first machine learning model based on comparing the output data of the first machine learning model and the second machine learning model; 12. The computer-implemented method of claim 11, comprising training by:

14. 14. The computer-implemented method of claim 13, wherein the labeled data comprises data output by a second machine learning model, the second machine learning model being a pre-trained model.

15. 15. The computer-implemented method of claim 14, wherein the output by the second machine learning model comprises annotated data.

16. 14. The computer-implemented method of claim 13, wherein the labeled data includes at least one of: (i) business and product descriptions, (ii) proxy data from websites associated with content creators, (iii) human-curated data, or (iv) free-form input.

17. Training the machine learning model includes: inputting unlabeled data into the machine learning model and a pre-trained second machine learning model; obtaining output data from the machine learning model and the pre-trained second machine learning model; comparing the output data from the machine learning model with the output data from the pre-trained second machine learning model; adjusting the machine learning model based on comparing the output data of the machine learning model with the output data of the pre-trained second machine learning model; 12. The computer-implemented method of claim 11, comprising:

18. The computer-implemented method of claim 11 , wherein the initial user interface comprises a graphical user interface.

19. One or more non-transitory computer-readable storage media having stored thereon instructions, the instructions being executable by one or more processors to perform operations, the operations comprising: training a first machine learning model using knowledge distillation, wherein the first machine learning model is a student model and a large-scale language model tuned to generate content is a parent model; evaluating performance of the trained first machine learning model by comparing a quality score associated with output data of the first machine learning model to a threshold quality score; determining that the first machine learning model has a quality score that exceeds the threshold quality score; responsive to determining that the first machine learning model has the quality score above the threshold quality score, implementing the first machine learning model in a content creation flow; Including, implementing the first machine learning model in a content creation flow; obtaining input data including natural language input; generating an output including the one or more suggested content item components; and transmitting instructions that, when executed, cause a client device to provide the one or more suggested content item components for display via a user interface.

20. generating an initial user interface including a content assistant component; obtaining data indicative of input received from a user; processing the data indicative of the input received from the user with the first machine learning model; obtaining output data from the first machine learning model indicative of one or more content item components; transmitting data that causes the one or more content item components to be provided for display on a user interface; obtaining data indicative of a user selection of approval for the one or more content item components; generating one or more content items in response to obtaining the data indicating the user selection of the approval of the one or more content item components; 20. One or more non-transitory computer-readable storage media according to claim 19, comprising:

Images