Conversation interface for creating and editing content using large language models

Abstract

An example embodiment of the present disclosure provides an example method including obtaining natural language input by a custom language model through a dialog activity assistant interface. The method includes generating, by the custom language model, an output including the predicted user intent. The method includes determining an action to be performed and determining a natural language response. The method includes transmitting an action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform an operation associated with completing the action. The method includes transmitting the response data structure to the dialogue activity assistant interface, the response data structure including providing the natural language response for display to a user through the dialogue activity assistant interface. The method includes obtaining a user input indicating verification of the action data structure or the response data structure, and updating the custom language model based on the user input.

Description

[0001] priority

[0002] This application claims the benefit of priority to U.S. non-provisional patent application No. 18 / 322,543, filed on May 23, 2023, which claims the benefit of priority to non-provisional patent application No. 17 / 968,472, filed on October 18, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to systems and methods for providing a conversational interface for content creation and editing using large language models. More specifically, the present disclosure relates to training and implementing one or more language models to facilitate a conversational interface for content creation and editing. Background Art

[0004] The computing device can perform many tasks and provide an interactive interface for content generation. The interactive interface for content generation may include a plurality of interactive components including an input field. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] In one example aspect, the present disclosure provides an example system, which includes one or more processors and one or more memory devices storing instructions, the instructions being executable to cause one or more processors to perform operations. In the example system, the operation includes obtaining natural language input by a custom language model through a dialogue activity assistant interface. In the example system, the operation includes generating an output including a predicted user intent by a custom language model. In the example system, the operation includes determining one or more actions to be performed by: parsing the output generated by the custom language model; determining an action associated with the output; or generating an action data structure including executable instructions, the instructions causing the processor to perform an operation associated with completing the action. In the example system, the operation includes determining a natural language response by: parsing the output generated by the custom language model; generating a response data structure including a natural language response to the acquired natural language input; or transmitting an action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform an operation associated with completing the action. In the example system, the operation includes transmitting a response data structure to the dialogue activity assistant interface, the response data structure including providing a natural language response for display to a user through the dialogue activity assistant interface. In an example system, the operations include, after transmitting the action data structure and the response data structure, obtaining user input indicating validation of the action data structure or the response data structure. In an example system, the operations include updating a custom language model based on the user input.

[0007] In an example aspect, the present disclosure provides a computer-implemented method, the method comprising obtaining natural language input by a custom language model through a dialogue activity assistant interface. The example method comprises generating an output including a predicted user intent by the custom language model. The example method comprises determining one or more actions to be performed by: parsing the output generated by the custom language model; determining an action associated with the output; or generating an action data structure including executable instructions, the executable instructions causing a processor to perform an operation associated with completing the action. The example method comprises determining a natural language response by: parsing the output generated by the custom language model; generating a response data structure including a natural language response to the obtained natural language input; or transmitting an action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform an operation associated with completing the action. The example method comprises transmitting a response data structure to the dialogue activity assistant interface, the response data structure including a natural language response provided for display to a user through the dialogue activity assistant interface. The example method comprises, after transmitting the action data structure and the response data structure, obtaining a user input indicating verification of the action data structure or the response data structure. The example method comprises updating the custom language model based on the user input.

[0008] In an example non-transitory computer-readable medium, an operation includes generating an initial user interface including a content assistant component. In an example non-transitory computer-readable medium, an operation includes obtaining natural language input by a custom language model through a conversation activity assistant interface. In an example non-transitory computer-readable medium, an operation includes generating output including a predicted user intent by a custom language model. In an example non-transitory computer-readable medium, an operation includes determining one or more actions to be performed by: parsing the output generated by the custom language model; determining an action associated with the output; or generating an action data structure including executable instructions that cause a processor to perform an operation associated with completing an action. In an example non-transitory computer-readable medium, an operation includes determining a natural language response by: parsing the output generated by the custom language model; generating a response data structure including a natural language response to the obtained natural language input; or transmitting an action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform an operation associated with completing an action. In an example non-transitory computer-readable medium, an operation includes transmitting a response data structure to a conversation activity assistant interface, the response data structure including providing a natural language response for display to a user through the conversation activity assistant interface. In an example non-transitory computer readable medium, the operations include, after transmitting the action data structure and the response data structure, obtaining user input indicating validation of the action data structure or the response data structure. In an example non-transitory computer readable medium, the operations include updating a custom language model based on the user input.

[0009] In one example aspect, the present disclosure provides an example system comprising one or more processors and one or more memory devices storing instructions executable to cause the one or more processors to perform operations. In the example system, the operation includes providing input data to a first machine learning model. In the example system, the operation includes obtaining output data indicating one or more suggested content item components. In the example system, the operation includes evaluating the first machine learning model based at least in part on the output data by obtaining data indicating a quality score associated with the output data of the first machine learning model. In the example system, the operation includes evaluating the first machine learning model based at least in part on the output data by comparing the quality score associated with the output data of the first machine learning model to a threshold quality score. In the example system, the operation includes determining that the first machine learning model has a quality score above a threshold quality score. In the example system, the operation includes implementing the first machine learning model in a content creation process in response to determining that the first machine learning model has a quality score above the threshold quality score.

[0010] In some embodiments of the example system, the operations include generating an initial user interface that includes a content assistant component. In some embodiments of the example system, the operations include obtaining data indicating user input. In some embodiments of the example system, the operations include processing data indicating user input by a first machine learning model. In some embodiments of the example system, the operations include obtaining output data indicating one or more content item components from the first machine learning model. In some embodiments of the example system, the operations include transmitting data that causes one or more content item components to be provided for display on a user interface. In some embodiments of the example system, the operations include obtaining data indicating an approved user selection of one or more content item components. In some embodiments of the example system, the operations include generating one or more content items including a plurality of content item components in response to obtaining data indicating an approved user selection of one or more content item components.

[0011] In some embodiments of the example system, the content assistant component includes one or more input fields.

[0012] In some embodiments of the example system, the first machine learning model has been trained using a knowledge distillation training method.

[0013] In some embodiments of the example system, the first machine learning model has been trained based at least in part on output from a pre-trained second machine learning model.

[0014] In some embodiments of the example system, the pre-trained second machine learning model is a large language model.

[0015] In some embodiments of the example system, one or more hints are used to tune the pre-trained second machine learning model.

[0016] In some embodiments of the example system, the input data includes at least one of free-form input or landing page content.

[0017] In some embodiments of the example system, the input data includes natural language input.

[0018] In some embodiments of the example system, the one or more content item components include at least one of a title or a description.

[0019] In an example aspect, the present disclosure provides a computer-implemented method comprising generating an initial user interface including a content assistant component. The example method comprises obtaining data indicating input received from a user. The example method comprises processing data indicating input received from a user by a machine learning model that interacts with the content assistant component. The example method comprises obtaining output data indicating one or more content item components from the machine learning model that interacts with the content assistant component. The example method comprises transmitting data that provides one or more content item components for display through an updated user interface. The example method comprises obtaining data indicating an approved user selection of one or more content item components. The example method comprises generating one or more content items in response to obtaining data indicating an approved user selection of one or more content item components.

[0020] In some embodiments of the example methods, the machine learning model is trained using a knowledge distillation training method.

[0021] In some embodiments of the example method, the method may include training the machine learning model by inputting the labeled data into the first machine learning model. The example method may include training the machine learning model by obtaining output data from the first machine learning model. The example method may include training the machine learning model by comparing the output data from the first machine learning model with the output data from the second machine learning model. The example method may include training the machine learning model by adjusting the first machine learning model based on comparing the output data of the first and second machine learning models.

[0022] In some embodiments of the example methods, the labeled data includes data output by a second machine learning model, the second machine learning model being a pre-trained model.

[0023] In some embodiments of the example methods, the output of the second machine learning model includes annotated data.

[0024] In some embodiments of the example methods, the tagged data includes at least one of: (i) business and product descriptions, (ii) proxy data from a website associated with a content creator, (iii) manually annotated data, or (iv) free-form input.

[0025] In some embodiments of the example methods, the method may include training the machine learning model by inputting unlabeled data into the machine learning model and a pre-trained second machine learning model. In some embodiments of the example methods, the method may include training the machine learning model by obtaining output data from the machine learning model and the pre-trained second machine learning model. In some embodiments of the example methods, the method may include training the machine learning model by comparing output data from the machine learning model with output data from the pre-trained second machine learning model. In some embodiments of the example methods, the method may include training the machine learning model by adjusting the machine learning model based on comparing the output data of the machine learning model with the output of the pre-trained second machine learning model. In some embodiments of the example methods, the initial user interface includes a graphical user interface.

[0026] In an example aspect, the present disclosure provides an example non-transitory computer-readable medium embodied in a computer-readable storage device and storing instructions that, when executed by a processor, cause the processor to perform operations. In the example non-transitory computer-readable medium, the operation includes providing input data to a first machine learning model. In the example non-transitory computer-readable medium, the operation includes obtaining output data indicating one or more suggested content item components. In the example non-transitory computer-readable medium, the operation includes evaluating the first machine learning model based at least in part on the output data by obtaining data indicating a quality score associated with the output data of the first machine learning model. In the example non-transitory computer-readable medium, the operation includes evaluating the first machine learning model based at least in part on the output data by comparing the quality score associated with the output data of the first machine learning model to a threshold quality score. In the example non-transitory computer-readable medium, the operation includes determining that the first machine learning model has a quality score above a threshold quality score. In the example non-transitory computer-readable medium, the operation includes implementing the first machine learning model in a content creation flow in response to determining that the first machine learning model has a quality score above the threshold quality score.

[0027] In an example non-transitory computer-readable medium, operations include generating an initial user interface including a content assistant component. In an example non-transitory computer-readable medium, operations include obtaining data indicating input received from a user. In an example non-transitory computer-readable medium, operations include processing, by a first machine learning model, data indicating input received from a user. In an example non-transitory computer-readable medium, operations include obtaining output data indicating one or more content item components from the first machine learning model. In an example non-transitory computer-readable medium, operations include transmitting data causing one or more content item components to be provided for display on a user interface. In an example non-transitory computer-readable medium, operations include obtaining data indicating a user selection of approval of the one or more content item components. In an example non-transitory computer-readable medium, operations include generating one or more content items in response to obtaining data indicating a user selection of approval of the one or more content item components. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0029] Figure 1 Depicted is a block diagram of an example system that provides a conversational interface for content creation and editing using a large language model according to an example embodiment of the present disclosure.

[0030] Figure 2 Depicted is a block diagram associated with a machine learning model according to an example embodiment of the present disclosure.

[0031] Figure 3 Depicted is a flowchart of an example method for training a machine learning model according to an example embodiment of the present disclosure.

[0032] Figure 4 A flowchart depicting an example process flow according to an example embodiment of the present disclosure is depicted.

[0033] Figure 5 A flowchart depicting an example process flow according to an example embodiment of the present disclosure is depicted.

[0034] FIG. 6A to FIG. 6C A flow chart of an example method according to an example embodiment of the present disclosure is depicted.

[0035] Figure 7 A flow chart of an example method according to an example embodiment of the present disclosure is depicted.

[0036] Figures 8 to 10 An example user interface is depicted according to an example embodiment of the present disclosure.

[0037] Fig.11Depicted is a block diagram associated with a machine learning model according to an example embodiment of the present disclosure.

[0038] Fig.12 A flowchart depicting an example process flow according to an example embodiment of the present disclosure is depicted.

[0039] Fig.13 A flowchart depicting an example process flow according to an example embodiment of the present disclosure is depicted.

[0040] FIG. 14A to FIG. 14C A flow chart of an example method according to an example embodiment of the present disclosure is depicted.

[0041] Figures 15 to 20 An example user interface is depicted according to an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In general, the present disclosure relates to systems and methods for generating and adjusting large language models that obtain natural language user input and generate content item components as output. Computing systems and methods may include providing an interface for generating and editing content items for display in response to obtaining and processing data indicating free-form user input. For example, a computing system may start the display of a content assistant component. The system may obtain user input including free-form speech through a content assistant component, and generate advertisements based on analyzing speech, extracting information from a website related to the user, and generating titles and / or other content items (e.g., advertising components) that the user usually needs to manually enter into multiple fields associated with a structured interface. The structured interface may be associated with a content creation process. The content creation process may be used to generate content items (e.g., advertisements, advertising campaigns). Traditional structured interfaces require manual input into multiple (e.g., sometimes 40 or more) input fields associated with rigid requirements.

[0043] Implementations of the present disclosure may use complex machine learning models to provide more efficient and user-friendly generation of content items. In some implementations, analyzing user input may be performed by a machine learning model (e.g., a natural language processing model). In some implementations, systems and methods may include training and / or adjusting the machine learning model. For example, a computing system may obtain input to the model from multiple sources. Sources may include, for example, business and product descriptions, proxy data for websites associated with content creators, manually annotated data, free-form input, and the like. The output of the model may include content item components (e.g., titles, descriptions, graphics, videos, color schemes). A computing system may train a machine learning model to implement as part of a content item generation process.

[0044] In some implementations, the training process may include performing knowledge distillation using one or more teacher and student models. For example, the teacher model may be an existing large language model. In some examples, the student model may be a custom model developed to generate custom content items (e.g., titles, descriptions, advertisements, videos, images, sound clips). For example, a machine learning model may provide an output including content item components (e.g., titles, descriptions, advertisements, videos, images, sound clips). The output of the student model may be evaluated to determine a quality score associated with the output. For example, the quality score may be associated with an automatically generated quality score, user satisfaction with the content item components, manually implemented ratings, and the like.

[0045] In some implementations, a computing system may collect multiple data sets, employ large language models to generate and annotate data examples, and / or train high-quality servable models for production. The computing system may evaluate the machine learning model. For example, the evaluation may include offline and / or online evaluation. The computing system may also determine whether the machine learning model is faithful.

[0046] The present disclosure provides multiple technical effects and benefits. For example, the present disclosure can reduce processing by reducing the number of manual inputs provided by users. In addition, the computing system can reduce processing by reducing the number of interface screens that must be acquired, loaded, interacted with, and updated. For example, a user can provide data indicating input (e.g., audio input, text input, etc.). The computing system can intelligently process the data indicating input and extract relevant information from the input. In some implementations, the computing system can provide follow-up questions to guide the user to provide input related to the desired content item to be generated. Since the user input is less, and the data indicating the user input is preprocessed to fill in multiple fields that are traditionally manually filled in by the user through structured user input, the computing system can reduce user errors.

[0047] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0048] Figure 1 A block diagram of an example computing system 100 is depicted that generates, trains, and / or uses a complex language model to interact with a user interface to facilitate generating content items according to an example embodiment of the present disclosure. The computing system 100 includes a client computing system 102, a server computing system 104, and a training computing system 106 that are communicatively coupled via a network 108.

[0049] The client computing system 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0050] The client computing system 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the client computing system 102 to perform operations.

[0051] In some implementations, the client computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). Reference Figure 2 and Figure 3 Discuss example machine learning model 120.

[0052] In some implementations, one or more machine learning models 120 may be received from a server computing system 104 via a network 108, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a client computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel retrieval and combination of modular application spaces on multiple instances of user data obtained through a user interface associated with a user device).

[0053] More specifically, the machine learning model can obtain data indicating user input (e.g., user session data 124A). The user input data can be associated with the current user session and / or include historical user data. For example, the data associated with the current user session can be data acquired in real time through the user input component 122. Historical user data may include data associated with a user account, user characteristics, etc. Historical user data may include data associated with a user device (e.g., a device identifier). In addition or alternatively, historical user data may include data associated with a user identifier. In some embodiments, historical user data may include aggregated data associated with multiple user identifiers. In some embodiments, training data 166 may include session data associated with one or more input devices (e.g., one or more input sessions), such as session data indexed by a type of input interface or device (e.g., a mobile device with a touch screen, a mobile device with a keyboard, a large touch screen, a small touch screen, a large touch screen, voice input, or a combination thereof, etc.). In some embodiments, training data 166 may include session data that is not associated with a user identifier. Using the machine learning model, the computing system can generate, train, and maintain one or more student models 164 (e.g., a custom language model). The computing system can use student model 164 to facilitate a user interface to generate customized content items (e.g., titles, descriptions, advertisements) based on natural language input obtained from a user. The computing system can retrieve, suggest, and / or generate one or more content items in response to obtaining user input (e.g., natural language input).

[0054] Additionally or alternatively, one or more machine learning models 126 may be included in or otherwise stored and implemented by a server computing system 104 that communicates with the client computing system 102 according to a client-server relationship. For example, the machine learning model 126 may be implemented by the server computing system 104 as part of a web service (e.g., a content development service, a campaign management service, a content policy management service). Thus, one or more machine learning models 120 may be stored and implemented at the client computing system 102, and / or one or more models 126 may be stored and implemented at the server computing system 104.

[0055] The client computing system 102 may also include one or more user input components 122 that may receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0056] The client computing system may include a user data database 124. The user data database 124 may include user session data 124A, user context data 124B, and / or user account data 124C. The user session data 124A may include data indicating a current user session obtained by the user input component 122. For example, the user session data 124A may include the current search term and / or other user input received within a threshold time of the current session. For example, a user may submit a first search and submit a second search five minutes later. The proximity in time between the first search and the second search may be user context data 124B. In response to the data indicating the user input to the structured user interface, the computing system may use the user context data 124B to determine the relevant content items to be provided and the predicted performance improvement (e.g., the predicted performance indicator) when processing the user query. The client computing system 102 may use the data indicating the user input to send a request for one or more suggested content item components (e.g., suggested titles, generated user interface elements, suggested tags) to the server computing system 104. The computing system may cause retrieval of one or more suggested content items, generation of one or more suggested content items, and / or presentation of one or more suggested content items to a user via a user interface of a device (e.g., a user device). User context data 124B may include context of a previous session and / or historical session data. User context data may include location, time, previous activity, previously used content items, devices used, type of construction workflow used, etc. User account data 124C may include data associated with a user account (e.g., login name, device identifier, user identifier).

[0057] The server computing system 104 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 104 to perform operations.

[0058] In some implementations, the server computing system 104 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 104 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0059] As described above, the server computing system 104 may store or otherwise include one or more machine learning models 126. For example, the machine learning model 126 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). Reference Figures 2 to 3 Discuss example machine learning model 126.

[0060] Client computing system 102 and / or server computing system 104 may train machine learning models 120 and / or 126 through interaction with training computing system 106 communicatively coupled via network 108. Training computing system 106 may be separate from server computing system 104 or may be part of server computing system 104.

[0061] The training computing system 106 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 154 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 106 to perform operations. In some implementations, the training computing system 106 includes or is otherwise implemented by one or more server computing devices (e.g., the server computing system 104).

[0062] The training computing system 106 may include a model trainer 160 that trains machine learning models 120 and / or 126 stored at the client computing system 102 and / or the server computing system 104 using various training or learning techniques (e.g., such as knowledge distillation). For example, the model trainer 160 may include one or more teacher models 162, one or more student models 164, and / or training data 166. For example, the teacher model 162 may include one or more pre-trained large language models. Large language models may include deep neural networks that require a lot of resources and time to train. Large language models may be equipped to determine meaning from natural language input.

[0063] In some implementations, the training data may include labeled data. For example, the labeled data may be input to the teacher model and / or the student model. In some examples, the training data 166 may be data obtained from the output of the teacher model 162. The computing system may obtain the output from the teacher model 162 and label the data. This labeled data may be used to train the student model 164. Figures 2 to 3 Training the student model 164 using the training data 166 is further described.

[0064] In some implementations, various training or learning techniques may include, for example, back propagation of errors. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update parameters over multiple training iterations.

[0065] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0066] Specifically, the model trainer 160 may train the machine learning model 120 and / or 126 based on the training data set 166. The training data 166 may include, for example, past performance indicators (e.g., predicted performance growth). In some implementations, one or more student models 164 may be the machine learning model 120 and / or 126.

[0067] In some implementations, if the user has consented, the training examples may be provided by the client computing system 102. Thus, in such implementations, the machine learning model 120 provided to the client computing system 102 may be trained by the training computing system 106 based on user-specific data received from the client computing system 102. In some cases, this process may be referred to as personalizing the model.

[0068] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 may be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer executable instructions stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0069] The network 108 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 108 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0070] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0071] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. The semantic intent output may include at least one word or phrase determined from the text or natural language data. As another example, the machine learning model may process the text or natural language data to generate an upgraded text or natural language output (e.g., a text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0072] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech conversion output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0073] In some implementations, the input of the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0074] In some implementations, the input to the machine learning model of the present disclosure may be statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. The machine learning model may process statistical data to generate an output. As an example, the machine learning model may process statistical data to generate an identification output. As another example, the machine learning model may process statistical data to generate a prediction output. As another example, the machine learning model may process statistical data to generate a classification output. As another example, the machine learning model may process statistical data to generate a segmentation output. As another example, the machine learning model may process statistical data to generate a visualization output. As another example, the machine learning model may process statistical data to generate a diagnostic output.

[0075] In some cases, the machine learning model can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating an embedding for the input data (e.g., input audio or visual data).

[0076] In some cases, the input includes audio data representing a spoken utterance, and the task is a speech recognition task. The output may include a text output mapped to the spoken utterance. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes a microprocessor execution task, such as branch prediction or memory address translation.

[0077] In some implementations, the machine learning model can be deployed on the device. For example, one or more components of a predictive machine learning model or pipeline can be deployed on the device to avoid uploading potentially sensitive information related to input type, device type, or input content (e.g., related to disability, contact information, address, etc.) to the server. For example, a server computing system can send a form with a learned context vector that describes one or more input fields associated with a component (e.g., a part of an application associated with the execution of a processing task). An onboard client model associated with the client computing system 102 can input local client characteristics (e.g., obtained through a user input component 122) and a context vector to generate a combined modular application. This on-device processing can increase the user's data privacy. In some embodiments, this can also reduce the amount of data transmitted outside the device, thereby reducing bandwidth usage.

[0078] Figure 2 An example system 200 is depicted for generating predicted performance improvements associated with suggested content items and providing an updated user interface including the suggested content items and the corresponding predicted performance improvements, according to an example embodiment of the present disclosure. The example system 200 includes a computing system 202. The computing system 202 can be any type of system of one or more computing devices (e.g., client computing system 102, server computing system 104, etc.). The computing device can be, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a node of a distributed computing device, a virtual instance hosted on a shared server, or any other type of computing device. In some embodiments, the computing system 202 includes multiple computing devices interconnected by a network or distributed in an interoperable manner. For example, the computing system 202 may include a server for serving content over a network (e.g., network 108). For example, computing system 202 may include a web server for hosting web content, for collecting data about web content (e.g., for receiving, monitoring, generating, or otherwise processing data about web content, such as usage, downloading, and / or interaction with web content).

[0079] The computing system 202 may include a processor 212 and a memory 214. The one or more processors 212 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 214 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 214 may store data 216 and instructions 218 that are executed by the processor 212 to cause the computing system 202 to perform operations.

[0080] In some implementations, the computing system 202 may store or otherwise implement one or more machine learning models of a machine learning model framework. In some embodiments, the content assistant component 220 includes a dual-model machine learning model framework. In some embodiments, the content assistant component does not include any combination of a machine learning model framework and / or a custom language model 222 and / or a large language model 226. The machine learning model framework may include a machine learning custom language model 222 (e.g., with a learnable weight 224) and / or a machine learning large language model 226 (e.g., with a learnable weight 228). In some embodiments, the content assistant component 220 may implement a single model that implements a custom language model 222 for content suggestions, content ranking, content generation, and / or any combination (e.g., by combining one or more aspects of a custom language model 222 and / or a large language model 226, by training a single model to directly obtain the desired output, etc.). One or more machine learning models may be or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model).

[0081] like Figure 2As shown, an embodiment of the example system 200 may be configured to process data 230. In response to processing data 230, computing system 202 may provide output 250. Output 250 may include a generated suggested content item component 255. The suggested content item component 255 may be provided for display via a user interface 270 of a client device associated with a client computing system 260. Client computing system 260 may include a processor 262 and a memory 264. One or more processors 262 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. Memory 264 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. Memory 264 may store data 266 and instructions 268 executed by processor 262 to cause client computing system 260 to perform operations.

[0082] The output 250 may include a suggested content item component 255. The suggested content item component 255 may include, for example, suggested titles, descriptions, follow-up communications, videos, images, sound clips, etc. For example, the custom language model 222 may obtain user input and provide a follow-up communication to collect additional input to generate further suggested content item components 255.

[0083] In some embodiments, data 230 is obtained through an input interface of a computing system (e.g., computing system 202 and / or client computing system 260). For example, in some embodiments, content assistant component 220 can be configured to process data 230 as input to computing system 202. In some embodiments, data 230 can be implicit in the structure and / or configuration of content assistant component 220.

[0084] In some implementations, the content suggestion generator model is trained to receive a set of input data (e.g., data 230) describing user input, and in response, provide output data (e.g., suggested content item component 255) indicating one or more suggested content items to be presented via the user interface. For example, Figure 2 An evaluation example is shown.

[0085] For example, the evaluation component 240 can perform offline evaluation and / or online evaluation (e.g., real business evaluation). The evaluation can include offline evaluation and online evaluation. The offline evaluation can include the use of a manual evaluation platform. The manual evaluation platform can include templates and / or scoring criteria for standardized evaluation. The online evaluation can include pilot and / or real-time business experiments to capture online indicators. In some implementations, the evaluation component 240 can include automatic evaluation and / or manual evaluation. In some implementations, the evaluation component 240 can include labeling and / or marking data as training data. The system can determine the quality score of the model. The system can compare the quality score of the model with a threshold quality score. If the quality score of the model is higher than the threshold quality score, the system can determine that the model is ready for use. If the quality score of the model is lower than the threshold quality score, the system can determine that the model requires additional training and / or adjustment before use.

[0086] In some implementations, the input data may include one or more features associated with an instance or example. In some implementations, the one or more features associated with an instance or example may be organized into a feature vector. In some implementations, the output data may include one or more predictions. Predictions may also be referred to as inferences. Thus, given features associated with a particular instance, a machine learning model may output a prediction for such instance based on the features.

[0087] The machine learning model may be or include one or more of a variety of different types of machine learning models. Specifically, in some implementations, the machine learning model may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0088] In some implementations, data 230 includes aggregated data 232, proxy data 234, and / or user input data 238. In some embodiments, data 230 may include instances of substantially any kind or type of data that may describe a variety of phenomena. In general, an instance refers to a set of one or more data values ​​that are combined together to describe a particular subject or subject matter. For example, an instance may be a feature vector. An instance may be associated with image data (e.g., a feature vector of an image, a hashed image, etc.). An instance may be associated with a measurement or other data collection event (e.g., at a particular time, or of a particular subject, or using a particular device, or from a particular perspective, etc.). An instance may be associated with a network session, such as a set of interactions with a web server. In some embodiments, an instance may be associated with a user's interaction with web content (e.g., anonymous or identified).

[0089] In some embodiments, the custom language model 222 may be trained using knowledge distillation. For example, the custom language model 222 may be a student model, and the large language model 226 may be a teacher model. A knowledge distillation training process may be performed to train a small custom language model 222 based on a pre-trained large language model 226. For example, a large language model may be a large model generated and trained using a large amount of computing resources. In order to save resources, the custom language model 222 may be trained using knowledge distillation to generate results similar to the pre-trained large language model 226.

[0090] In some implementations, prompts can be used to adjust and / or train the models of the content assistant component 220. For example, the computing system can obtain prompts indicating sample inputs and outputs. The sample inputs and outputs can be used to adjust the model to familiarize the types of inputs and outputs required for a particular implementation. For example, a particular implementation may include generating advertisements. Generating advertisements requires understanding the nuances of businesses, business goals, products, etc. In this way, by prompting one or more models (e.g., custom language models 222 and / or large language models 226), the computing system can adjust the model to provide higher quality outputs (e.g., outputs 250 that can be evaluated using evaluation component 240). In some cases, after the evaluation component 240 has performed an evaluation, the data can be aggregated and used as data 230 for further model training.

[0091] In some implementations, content assistant component 220 may be a machine learning model. In some implementations, training and tuning may include using data 230. Aggregate data 232 may include data obtained from one or more user sessions that have been aggregated by a system (e.g., computing system 202). Proxy data 234 may include, for example, output data obtained from content assistant component 220. In some implementations, proxy data 234 may be labeled and / or marked as representing sample inputs and / or outputs. User input data 238 may include user-generated inputs, including inputs and / or outputs associated with content item wishes and generated content items.

[0092] The trained model can be used to generate one or more suggested content items and / or update a structured user interface and / or build a workflow. In some embodiments, a simulated user model can be trained using training data to generate simulated inputs based on patterns learned from the training data. In some cases, the simulated user model can provide a simulated environment in which a reinforcement learning agent, a notification element subsystem can be trained.

[0093] Figure 3A flow chart of an example method 300 for training a student model 310 using knowledge distillation is depicted. Training the student model 310 can include utilizing a teacher model 315, training data 305, and / or a distillation component 320. For example, a computing system can define a large language model as the teacher model 315. The computing system can define a custom language model as the student model 310. In some implementations, the teacher model 315 can include a large language model with millions and / or billions of parameters. In some implementations, the student model 310 can include a custom language model with thousands of parameters.

[0094] The teacher model 315 can be fully trained. For example, the teacher model can be trained as part of the method 300 and / or trained before converging with one or more student models 310. Any training method can be used to train the teacher model 315. For example, the teacher model 315 can be trained until fully converged. For example, the loss function can be any loss function based on the problem statement.

[0095] The computing system can intelligently train the student model 310 using the teacher model 315. In some implementations, for example, the student model 310 and the teacher model 315 can include neural networks. For example, the student model 310 can be trained in coordination with the fully trained teacher model 315. In some implementations, the training data 305 is input into the teacher model 315. The output data 330 of the teacher model 315 can be obtained. The computing system can annotate the training data 305 by running the training data 305 through the teacher model 315. The computing system can obtain the annotated training data 325 as the output data 330 from the teacher model 315. The student model 310 can obtain the annotated training data 325 as input. The student model 310 can be trained using the annotated training data.

[0096] For example, the teacher model 315 may include a large language model that can be trained on the training data 305. The training data 305 may include, for example, a data set of multilingual cleaned public web documents, codes, and conversations from the Internet that includes more than 1.56 trillion words. In some implementations, the training data 305 may include a seed data set collected from humans. In some implementations, the computing system may obtain data to enhance the training data set. In some implementations, the training data 305 may include proxy data from a content provider (e.g., advertiser) website. For example, a content provider may have a website that includes pages such as "About Us," "Our Story," "Blog," "Services," etc. These pages may include business description content. One or more teacher and / or student models may obtain proxy data as input and generate output including one or more content items (e.g., title, description). In some implementations, the training data 305 may include manually annotated data. Manually annotated data may be data obtained through user input. For example, a user may provide a written business description. Manually annotated data may be used as seed training data. In some implementations, the computing system may use an existing large language model to generate data used as training data 305. For example, a large language model can generate descriptions and creative assets. In some implementations, the descriptions and creative assets can include free-form input and / or titles.

[0097] Additionally or alternatively, the teacher model 315 may include an encoder-decoder network. In some implementations, the encoder-decoder network may be a general, multimodal language model with multi-task learning capabilities. In some implementations, the encoder-decoder network may be pre-trained on text from billions of high-quality web documents and may be highly customized for specific subject areas (e.g., search results, advertising, specific businesses).

[0098] In some cases, the computing system can adjust the student model 310. For example, the computing system can input a subject-specific data set into the student model 310. By providing a subject-specific data set to the student model 310 during training, the computing system can demonstrate a significant improvement in subject-specific applications (e.g., the purpose of the student model). Adjusting and / or training the student model 310 to adapt to a specific subject can more carefully understand the input obtained from the user. A more detailed understanding of the input obtained from the user can produce better output. Due to processing repeated inputs and / or increased touch inputs obtained from users who need to manually enter data into multiple fields with strict requirements, better output can cause the computing system to complete less processing. In addition, providing better output can increase user experience and satisfaction.

[0099] Although the teacher model 315 is described as a single model for illustrative purposes, the teacher model 315 may include one or more models. Although the student model 310 is described as a single model for illustrative purposes, the student model 310 may include one or more models.

[0100] In some implementations, the teacher and student settings may include additional and / or alternative methods for knowledge distillation. For example, knowledge distillation may include using a distillation component 320 that includes a distillation loss function 335 and a student loss function 345. The distillation loss function 335 may be used for forward propagation of the teacher model 315 and the student model 310. Distillation may include backpropagation of the student model 310. For example, by converging the loss of the output of the student model 310 and the output of the teacher model 315.

[0101] Algorithms for knowledge distillation of the teacher model 315 and the student model 310 may include, for example, adversarial distillation, multi-teacher distillation, cross-modal distillation, graph-based distillation, attention-based distillation, data-free distillation, quantized distillation, lifelong distillation, and / or distillation based on neural architecture search. Training the student model 310 may include, for example, offline distillation, online distillation, and / or self-distillation. Knowledge-based distillation may include response-based knowledge, feature-based knowledge, and / or relationship-based knowledge. Knowledge distillation may be applied to various applications. Applications may include, for example, vision, natural language processing, and / or speech.

[0102] Vision applications may include, for example, image classification, facial recognition, image segmentation, action recognition, video interpretation, image retrieval, text-to-image synthesis, and / or video classification. NLP applications may include, for example, text generation, text recognition, neural machine translation, question answering, and / or document retrieval. Voice applications may include, for example, speech recognition, language identification, audio classification, speaker recognition, speech synthesis, speech enhancement, and / or sound event detection.

[0103] Knowledge distillation can provide various technical benefits and technical effects. For example, by using a pre-trained model to adjust a content-specific model, a computing system can save computing resources and processing power required to train the model so that it performs as well as a pre-trained model without a pre-trained model. In addition, in some implementations, the training data set can include one or more prompts for adjusting the teacher model 315 and / or the student model 310. By providing input as well as prompts, the computing system can adjust the model to produce better output than traditional model training methods.

[0104] Figure 4A flowchart of a process flow 400 for facilitating content generation through a responsive content building process is depicted at 420. For example, a computing system can include a content item table 405. At 410, the computing system can obtain user input indicating that a user selects (e.g., clicks) an option to generate a new content item (e.g., an advertisement, an advertising campaign). At 415, the computing system can obtain user input indicating that a user selects a content item group. In response to obtaining user input indicating a selection of a content item group, the computing system can initiate responsive content building at 420.

[0105] At 420, the computing system initiating responsive content construction can include providing a takeover promotion of the content assistant for display through the user interface. For example, at 425, the takeover promotion of the content assistant can be provided for display.

[0106] At 430, the computing system may provide a selectable interface element including a message for display. For example, the message may include a "use content assistant?" message. In some implementations, the message may be displayed using selectable user interface elements labeled "yes" and "no."

[0107] For example, the computing system may obtain data indicating that the user selected "No". In response, at 435, the computing system may cause the takeover promotion of the content assistant to disappear. At 440, the computing system may provide a selectable interface including a message for display. The message may include the following: "Does the user have a final URL? " In some implementations, selectable user interface elements marked as "Yes" and "No" may be used to display the message. The computing system may obtain data indicating that the user selected "Yes". In response, the computing system may pre-fill one or more input fields associated with the user interface at 445A for content generation. The computing system may obtain data indicating that the user selected "No". In response, the computing system may not pre-fill one or more input fields associated with the user interface at 445B for content generation.

[0108] For example, in response to the message display at 430, the computing system can obtain data indicating that the user selected "yes". In response, the computing system can update the user interface to display the content assistant 447. At 450, the computing system can provide the content assistant component for display through the user interface. The content assistant component can be displayed in a variety of formats. For example, a format can include a certain number of free-form text input fields (e.g., such as Figure 8 ), one or more progressive disclosure fields (e.g., Fig. 9) and / or a dialogue interface (e.g., as depicted) Fig.10 ). The computing system may execute content assistant process 455. Figure 5 More details on the content assistant process 455.

[0109] Go to Figure 5 , the content assistant process 455 includes providing an interface for user interaction (e.g., a content assistant component of a user interface). As described above, the interface can be an audio interface, a visual interface, etc. For the purpose of explanation, the content assistant process 455 will be described as a visual interface (e.g., a graphical user interface). At 505, the computing system can provide the following message for display: "tell me about your business? ". The message may include any message prompting the user to provide input data in a natural language and / or free form format. At 510, the computing system may obtain user input in response to the display of the message. For example, the displayed message may state "Could you please describe the product or service you want to advertise and how it helps your potential customers? ". For example, a user can provide information about his or her business. A user could provide the following as input: "We open 9-5 andoffer free refunds and ship to all 50 states. We are called PuppyThings.Wesell dog kibbles, snacks, toys, and anything for puppies. Great value, highquality products. We open 9-5 and offer free refunds and ship to all 50states."

[0110] At 515, the computing system may extract information from the user input. For example, the extracted information may include the business name and / or descriptive terms and phrases. Fig.10In the depicted example, the system can determine that the business name is PuppyThings. The system can extract other related terms such as puppy, dog food, treats, toys, high quality, business hours 9 to 5, free refunds, and shipping to all 50 states.

[0111] In some implementations, the computing system may determine that additional information is needed. In response, at 520, the computing system may provide for display a message that additional information is needed from the user. For example, the message may include follow-up questions, clarifying questions, information questions that were not included in the initial user input, etc. Fig.10 As depicted, the system may provide a follow-up question stating "Do you have a website that people should go to after they click the ad?"

[0112] At 522, the computing system may obtain user input in response to the message to obtain more information. Fig.10 As depicted, a user may provide the following input: "www.puppythings.com". The computing system may extract information from the user input at 515, as previously described. This process may be repeated as many times as necessary to collect relevant information.

[0113] At 525, the computing system may generate suggested content. For example, the suggested content may include the filling of one or more input fields. In some implementations, the content may include the entire generated advertisement and / or campaign. For example, the generated advertisement may include a visual display of the proposed advertisement. For example, the computing system may provide a summary of the advertising campaign strategy for display, including one or more generated advertisements, display requirements, display time, expected benefits, etc.

[0114] Steering Fig.10For example, the computing system may generate the following message: "We created the ad based on your information: 10 keywords, 5 headlines, 2 descriptions. Please let me know if you want to modify it." For example, the computing system may generate the following suggested headlines: Anything Your Puppy Needs, Dog Food, Treats, and Toys for Puppies, High Quality Products for Puppies, Business Hours 9-5, Free Refunds, and Shipping to All States. The computing system may generate the following suggested descriptions: Puppy Supplies, Great Value, High Quality, Free Shipping. Shop and / or welcome your new puppy today at puppythings.com and get everything they need. Dog food, treats, toys, and more. All available at puppythings.com. The computing system may determine the following target keywords: puppy supplies, dog food, dog treats, and more.

[0115] At 530, the computing system may obtain user input indicating a desire to modify the content of the suggestion. For example, the user may specify that they mean that they offer free refunds and ship to all 50 states. They may indicate that free shipping is not offered. The title and / or description of the suggestion may be modified based on the user input. For example, the computing system may obtain the user input in a natural language format. For example, the user may reply "Please modify the first headline to remove free shipping. Please modify the first headline to remove free shipping. Thank you."

[0116] At 535, the computing system can modify the content in response to the user input. In some implementations, the computing system can repeat steps 530 and 535 until the computing system obtains user input on the satisfaction of the suggested content.

[0117] At 540, the computing system may obtain user input indicating satisfaction. For example, the computing system may obtain data indicating that the user selected "continue", "accept", etc.

[0118] At 545, in response to obtaining user input indicating satisfaction, the computing system can end the content assistant process 455. In some implementations, the computing system will continue Figure 4 The process described in .

[0119] Transfer back Figure 4 , following the content assistant process 455, the computing system may obtain user input indicating that the "App Suggestions" option is selected at 460. In response, the computing system may fill in one or more input fields (e.g., associated with generating content items, advertisements, etc.). The user may check the filled input fields to determine whether there are any errors or problems. At 465, the computing system may provide a message for display through the user interface. The message may include the following: "Are there any errors or problems? In some implementations, selectable user interface elements marked as "yes" and "no" may be used to display the message. The computing system may obtain data indicating that the user selected "yes". In response, at 470, the computing system may initiate the display of an error message and / or a troubleshooting process. The computing system may obtain data indicating that the user selected "no". In response, at 475, the computing system may determine whether this is the first time the user has used the assistant.

[0120] In some implementations, the computing system may determine that this is not the first time the user has used the assistant. In response, at 480, the computing system may make the content assistant disappear and provide input fields for the content assistant input for display. In some implementations, the computing system may determine that this is the first time the user has used the assistant. In response, at 485, the computing system may provide a feature promotion sequence including an alert for display. The alert may include a notification highlighting one or more input fields populated by the content assistant and / or describing the location of the content item component and / or content item generated by the computing system through responsive content construction at 420.

[0121] FIG. 6A to FIG. 6C A flow chart is depicted for an example method 600 performed in accordance with an example embodiment of the present disclosure. FIG. 6A to FIG. 6C The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. The various steps of method 600 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0122] At (602), method 600 may include providing input data to a first machine learning model. For example, a computing system may provide input data to the first machine learning model. As described herein, the input data may include at least one of free-form input and / or landing page content. The input data may include natural language input.

[0123] In some implementations, the first machine learning model may be trained using a knowledge distillation training method. For example, the knowledge distillation training method may include training the first machine learning model (e.g., a student model) using a pre-trained large teacher model (e.g., a complex large language model). In some embodiments, the first machine learning model may be trained based at least in part on output from a pre-trained second machine learning model.

[0124] In some implementations, the pre-trained second machine learning model can be a large language model. The pre-trained second machine learning model can be adjusted using one or more prompts. For example, the system can adjust the model to generate suggested assets from free-form input through few-shot learning. For example, the input can include a prompt. The prompt can include a title and / or a description. For example, the title can include a new tycoon game, try our dog-friendly cafe, build your dog paradise, or the coolest dog app in the United States. The description can include serving delicious drinks in the friendliest dog cafe and Dog Cafe Tycoon: Feed hungry customers and grow your dog cafe empire.

[0125] At (604), method 600 may include obtaining output data indicating one or more suggested content item components. For example, the computing system may obtain output data indicating one or more suggested content item components. For example, the content item component may include at least one of a title or a description. For example, the output of the model may include a generated text output. For example, the text may include "Dog Cafe Tycoon is a Simulation app. Here is a brief introduction of the app: A new tycoon game to build your dog paradise. It serves drinks in the dog café and helps to grow your dog café empire. Keytopic keywords for this application are dog café. The target country is the US. (Dog Cafe Tycoon is a simulation app. Here is a brief introduction of the app: A new tycoon game to build your dog paradise. It serves drinks in the dog cafe and helps to grow your dog café empire. Keytopic keywords for this application are dog café. The target country is the US.)" Adjusting a large language model may improve the model's ability to generate more nuanced content item suggestions in the future. For example, generating an advertisement may include the need to understand the nuances associated with a business, the goals of a particular user, and the like. This type of language processing may be different from other language processing applications.

[0126] At (606), method 600 may include evaluating the first machine learning model based at least in part on the output data. For example, the computing system may evaluate the first machine learning model based at least in part on the output data. As described herein, the evaluation of the first machine learning model may be performed based at least in part on determining a quality score associated with the output data of the first machine learning model.

[0127] At (606A), method 600 may include obtaining data indicating a quality score associated with output data of the first machine learning model. For example, the computing system may obtain data indicating a quality score associated with output data of the first machine learning model. For example, the quality score may be automatically generated by the system and / or manually input by a user.

[0128] At (606B), method 600 may include comparing a quality score associated with output data of the first machine learning model to a threshold quality score. For example, the computing system may compare the quality score associated with output data of the first machine learning model to a threshold quality score. For example, the computing system may continuously evaluate the model. The computing system may determine that the quality of the output data is acceptable. For example, in a training implementation, output data may be obtained for input data that has been labeled and / or processed by a pre-trained model. The system may compare the actual output data to the expected output data.

[0129] At (608), method 600 may include determining that the first machine learning model has a quality score that is above a threshold quality score. For example, the computing system may determine that the first machine learning model has a quality score that is above a threshold quality score. As described herein, the quality score may be associated with the quality of the output data.

[0130] At (610), method 600 may include implementing the first machine learning model in a content creation process in response to determining that the first machine learning model has a quality score above a threshold quality score. For example, the computing system may implement the first machine learning model in a content creation process in response to determining that the first machine learning model has a quality score above a threshold quality score. The content creation process may be associated with a third party that provides a platform for content creators to generate custom content items (e.g., search results that link to websites, advertisements, generated content items for display). The content creation process may include a user interface for generating content items. The content creation process may include a content assistant component that provides a user interface for display.

[0131] At (612), method 600 may include generating an initial user interface including a content assistant component. For example, a computing system may generate an initial user interface including a content assistant component. As described herein, the initial user interface may include a content assistant component. The content assistant component may be configured to obtain user input and generate suggested content (e.g., title, description, advertisement, video, image, sound clip, etc.) in response. In some implementations, the content assistant component may include one or more input fields. For example, the content assistant component may include a certain number of predetermined input fields. In some implementations, the content assistant component may include a plurality of progressively displayed input fields. In some implementations, the content assistant component may be configured to provide a dialog interface.

[0132] In some implementations, the user input may include text snippets, documents, images, handwriting, audio, etc. In some implementations, the user may not have a robust login page associated with their website. In response, the computing system may initiate an input funnel to obtain the user input. Additionally or alternatively, the user input may include fine-grained authorization controls for content item policies. For example, the content item may be an advertisement, and the policy may be related to an activity. The system may obtain user input indicating that the user selects one or more fine-grained authorization controls, including through requirements for interaction with the system and adjustments for activities.

[0133] At (614), method 600 may include obtaining data indicating user input. For example, the computing system may obtain data indicating user input. The user input may include input obtained from the user by any means. The means may include touch input (e.g., via a touch sensor keyboard), voice input, etc.

[0134] At (616), method 600 may include processing, by the first machine learning model, data indicative of user input. For example, the computing system may process, by the first machine learning model, data indicative of user input. Processing the data indicative of user input may include converting the data from the natural language input into a feature vector or some other data structure for ingestion by the first machine learning model.

[0135] At (618), method 600 may include obtaining output data indicating one or more content item components from the first machine learning model. For example, the computing system may obtain output data indicating one or more content item components from the first machine learning model. For example, the one or more content item components may include a title, a description, a video, an image, a slogan, a sound clip, etc. The content items may be generated and / or retrieved by the content assistant component.

[0136] At (620), method 600 may include transmitting data that causes one or more content item components to be provided for display via a user interface. For example, the computing system may transmit data that causes one or more content item components to be provided for display via a user interface. For example, the content item components may be a title, a description, etc., which are populated in one or more input fields associated with a content creation process form.

[0137] At (622), method 600 may include obtaining data indicating a user selection of approval of one or more content item components. For example, the computing system may obtain data indicating a user selection of approval of one or more content item components. For example, the computing system may obtain data indicating a user selection of "Accept," "Confirm," "Create Ad," or some other notification indicating approval of the content item component.

[0138] At (624), method 600 may include generating one or more content items in response to obtaining data indicating a user selection of approval for one or more content item components. For example, the computing system may generate one or more content items in response to obtaining data indicating a user selection of approval for one or more content item components. For example, a content item may be an advertisement including multiple content item components. In some implementations, a content item may include multiple titles, descriptions, videos, images, slogans, etc. Based on the processed user input and / or other user data, the system may generate one or more content items using the one or more content item components.

[0139] Figure 7 A flow chart is depicted for an example method 700 performed in accordance with an example embodiment of the present disclosure. Figure 7 The steps performed in a particular order are depicted for purposes of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. The various steps of method 700 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0140] At (702), method 700 may include obtaining data indicating input received from a user. For example, the computing system may obtain data indicating input received from the user. The input from the user may include free-form input and / or landing page input. For example, the free-form input may include the user describing himself and / or an associated business using natural language. The user input may include a uniform resource locator (URL). In some cases, the URL may be associated with a website and / or a website landing page associated with the user. In some implementations, the system may obtain data from the landing page to generate one or more content item components and / or content items to suggest to the user.

[0141] At (704), method 700 may include processing, by the machine learning model, data indicative of input received from a user. For example, the computing system may process, by the machine learning model, data indicative of input received from a user.

[0142] In some implementations, method 700 may include training one or more machine learning models to implement in a content creation process. For example, a computing system may train one or more machine learning models to implement in a content creation process. For example, one or more machine learning models may be trained in any manner. For example, one or more machine learning models may be trained using a knowledge distillation training method. In some implementations, training one or more machine learning models may include inputting labeled data into a first machine learning model. In some implementations, the system may obtain output data from the first machine learning model to output data from an expected machine learning model. The system may adjust the first machine learning model based on comparing the output data of the first machine learning model and the expected machine learning model.

[0143] In some implementations, the labeled data may include data output by a second machine learning model, wherein the second machine learning model is a pre-trained model. The output obtained by the expected machine learning model may include annotated data. For example, when the same input data is processed by the expected machine learning model, the system may annotate the input data with the expected results based on the output. In some implementations, the annotated data may be a silver training set. For example, the labeled data may include at least one of business and product descriptions, proxy data from a website associated with a content creator, manually annotated data, or free-form input.

[0144] In some implementations, training one or more machine learning models may include inputting unlabeled data into a first machine learning model and a pre-trained second machine learning model. Training may include obtaining output data from the first machine learning model and the second machine learning model. Training may include adjusting the first machine learning model based on comparing the output data of the first machine learning model with the output of the second machine learning model. For example, the system may perform loss regression on the outputs of the first and second machine learning models. In some cases, the system may continuously update and reprocess the input until the difference between the outputs of the first and second machine learning models is within an acceptable threshold.

[0145] In some implementations, the method may include implementing a machine learning model in a content creation process. For example, a computing system may implement a machine learning model in a content creation process. The content creation process may be associated with a third party that provides a platform for content creators to generate custom content items (e.g., search results that link to a website, an advertisement, a generated content item constructed for display).

[0146] In some implementations, the method may include generating an initial user interface including a content assistant component. For example, a computing system may generate an initial user interface including a content assistant component. In some implementations, the initial user interface may be a graphical user interface.

[0147] At (706), method 700 may include obtaining output data indicating one or more content item components from the machine learning model. For example, the computing system may obtain output data indicating one or more content item components from the machine learning model. For example, the content item components may include one or more content suggestions. For example, the content may include a suggested title, description, image, video, slogan, etc. In some implementations, the content may include a generated advertisement.

[0148] At (708), method 700 may include transmitting data that causes one or more content item components to be provided for display via a user interface. For example, the computing system may transmit data that causes one or more content item components to be provided for display via a user interface. For example, the system may populate an existing user input field with content generated by the model. In some implementations, the system may generate an entire suggested content item (e.g., an advertisement).

[0149] At (710), method 700 may include obtaining data indicating a user selection of approval of one or more content item components. For example, the computing system may obtain data indicating a user selection of approval of one or more content item components. For example, the system may obtain data indicating a selection of a user interface element indicating that the user selected "Accept," "Confirm," "Create Ad," or some other notification indicating approval of the content item component.

[0150] At (712), method 700 may include generating one or more content items in response to obtaining data indicating a user selection of approval of one or more content item components. For example, the computing system may generate one or more content items in response to obtaining data indicating a user selection of approval of one or more content item components.

[0151] Figures 8 to 10 An example content assistant component of a user interface according to an example embodiment of the present disclosure is depicted. A computing system can provide a content assistant component for display via a user interface. The content assistant component can be displayed in a variety of formats. For example, a format can include a certain number of free-form text input fields (e.g., Figure 8 ), one or more progressive disclosure fields (e.g., Fig. 9 ) and / or a dialogue interface (e.g., as depicted) Fig.10 depicted).

[0152] Figure 8An example content assistant component 800 is depicted, which includes a number of free-form text input fields 805, 810, 815. For example, the first free-form text input field 805 may include prompting the user to "describe the product or service you want to advertise". The computing system may obtain data indicating user input to the free-form text input field 805. The free-form text input may be a natural language input by the user. The second free-form text input field 810 may include prompting the user to "explain what makes your product or service unique". The computing system may obtain data indicating user input to the free-form text input field 810. The third free-form text input field 815 may include prompting the user "what URL should this ad link to? In some implementations, the generated content may be an advertisement. For example, a user may provide a uniform resource locator (URL) that directs the user to a website associated with the user. For example, a user may be associated with a business called Puppy Things. The URL associated with the business may be "http: / / www.puppythings.com".

[0153] Fig. 9 An example content assistant component 900 including one or more progressive disclosure fields is depicted. For example, the content assistant component 900 can include a first progressive disclosure field 905. The first progressive disclosure field 905 can be presented when the content assistant component 900 is first launched. When a user provides input into the first progressive disclosure field 905, the system can provide a second progressive disclosure field 910 for display. After the system determines that the user has entered data into the second progressive disclosure field 910, the system can provide a third progressive disclosure field 915 for display.

[0154] In some implementations, progressive disclosure fields 905, 910, and / or 915 may be predetermined fields. Additionally or alternatively, progressive disclosure fields may be determined based on processing of information obtained as user input to previously disclosed progressive disclosure fields. For example, a first progressive disclosure field 905 may indicate "tell me about your business". If the user provides a short, 1 to 2 generic sentence answer, a second progressive disclosure field 910 may indicate a more targeted prompt than if the user provided a multi-paragraph answer in the first progressive disclosure field 905. When the user has completed providing input to one or more progressive disclosure fields, the system may obtain data indicating that the user selected a submission element 920. In response, the system may process the user input provided through the progressive disclosure fields and provide content suggestions through the content creation process.

[0155] Fig.10 An example content assistant component 1000 including a conversational user interface is depicted. In some implementations, the content assistant component 1000 can include a back-and-forth communication session between a user and a content assistant. For example, the content assistant can provide an initial prompt 1005 to the user. The user can provide a reply 1010. The computing system can facilitate the transmission of a variety of communications between the user and the content assistant. Figure 5 As described, a computing system can provide follow-up questions, process data indicative of user input associated with a user's response, and generate suggested content based on the processing of the user input (e.g., via a machine learning natural language processing model).

[0156] Fig.11An example system 1100 is depicted for generating predicted performance improvements associated with suggested content items and providing an updated user interface including the suggested content items and the corresponding predicted performance improvements, according to an example embodiment of the present disclosure. The example system 1100 includes a computing system 1102. The computing system 1102 can be any type of system of one or more computing devices (e.g., client computing system 102, server computing system 104, etc.). The computing device can be, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a node of a distributed computing device, a virtual instance hosted on a shared server, or any other type of computing device. In some embodiments, the computing system 1102 includes multiple computing devices interconnected by a network or distributed in an interoperable manner. For example, the computing system 1102 may include a server for serving content over a network (e.g., network 108). For example, computing system 1102 may include a web server for hosting web content, for collecting data about web content (e.g., for receiving, monitoring, generating, or otherwise processing data about web content, such as usage, downloading, and / or interaction with web content).

[0157] The computing system 1102 may include a processor 1112 and a memory 1114. The one or more processors 1112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or multiple processors operatively connected. The memory 1114 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1114 may store data 1116 and instructions 1118 that are executed by the processor 1112 to cause the computing system 1102 to perform operations.

[0158] In some implementations, the computing system 1102 may store or otherwise implement one or more machine learning models of a machine learning model framework. In some embodiments, the content assistant component 1120 includes a multi-model machine learning model framework. In some embodiments, the content assistant component 1120 does not include any combination of a machine learning model framework and / or a custom language model 1122, a language model 1126, or a model 1125. The machine learning model framework may include a machine-learned custom language model 1122 (e.g., with learnable weights 1124), a machine-learned language model 1126 (e.g., with learnable weights 1128), or a model 1125. Model 1125 may include a generative model 1129 or an action model 1127. Generative model 1129 may generate images, audio, text, audio-visual, or any other content. Action model 1127 may be able to interact with one or more other models or components to perform an action (e.g., make a recommendation, interact with another website). Model 1125 may also have associated learnable weights ( Fig.11 not depicted).

[0159] In some embodiments, the content assistant component 1120 may implement a single model that implements a custom language model 1122 for content suggestions, content ranking, content generation, general advertising content or recommendations, general analytical content or recommendations, and / or any combination (e.g., by combining one or more aspects of the custom language model 1122 and / or the language model 1126, by training a single model to directly obtain the desired output, etc.). One or more machine learning models may be or may otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). In some cases, the model 1125 may include a multimodal model that can understand images, text, or other inputs. In some cases, model 1125 may include multiple models, which may use a language model (e.g., a large language model) to better understand the intent of the acquired data (e.g., user input data 1138).

[0160] like Fig.11As shown, an embodiment of the example system 1100 may be configured to process data 1130. In response to processing data 1130, the computing system 1102 may provide output 1150. Output 1150 may include a generated suggested content item component 1155, a natural language response 1157, or a content campaign recommendation 1159. The suggested content item component 1155 may include multiple components or elements for generating a content item. For example, the suggested content item component may include text (e.g., a title or description), an image (e.g., obtained from a database or generated in real time), audio-visual, interactive, or other types of content. The natural language response 1157 may include text generated by a language model based on determining the intent of the acquired user input and providing an answer, a recommended natural language response, or other response as output 1150. The content campaign recommendation 1159 may include recommendations for analyzing historical content performance data (e.g., advertising performance, cost per click, return on advertising expenditure), recommendations for adjusting content campaign parameters based on the analyzed data (e.g., adjusting expenditures of different media channels, adjusting maximum or minimum bidding parameters), or any other recommendations associated with a content campaign.

[0161] The suggested content item component 1155, natural language response 1157, or content activity recommendation 1159 may be provided for display via a user interface 1170 of a client device associated with the client computing system 1160. The client computing system 1160 may include a processor 1162 and a memory 1164. The one or more processors 1162 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.), and may be one processor or a plurality of processors operatively connected. The memory 1164 may include one or more computer-readable storage media that may be non-transitory, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1164 may store data 1166 and instructions 1168 executed by the processor 1162 to cause the client computing system 1160 to perform operations.

[0162] Output 1150 may include suggested content item components 1155. Suggested content item components 1155 may include, for example, suggested titles, descriptions, follow-up communications, videos, images, audio, sound clips, etc. In some cases, suggested content item components 1155 may be creative assets that can be input into a free-form input or title. For example, custom language model 1122 may obtain user input and provide follow-up communications to collect additional user input to generate further suggested content item components 1155. For example, a custom model may be trained using distillation techniques and a previously trained language model (e.g., a previously trained large language model).

[0163] In some embodiments, data 1130 is obtained through an input interface of a computing system (e.g., computing system 1102 and / or client computing system 1160). For example, in some embodiments, content assistant component 1120 can be configured to process data 1130 as input to computing system 1102. In some embodiments, data 1130 can be implicit in the structure and / or configuration of content assistant component 1120.

[0164] In some implementations, the content suggestion generator model is trained to receive a set of input data (e.g., data 1130) describing user input and, in response, provide output data (e.g., suggested content item component 1155) indicating one or more suggested content items to be presented via a user interface. For example, Fig.11 An evaluation example is shown.

[0165] For example, the evaluation component 1140 can perform offline evaluation and / or online evaluation (e.g., real business evaluation). Offline evaluation can include the use of a manual evaluation platform. The manual evaluation platform can include templates and / or scoring criteria for standardized evaluation. Online evaluation can include pilot and / or real-time business experiments to capture online indicators. In some implementations, the evaluation component 1140 can include automatic evaluation and / or manual evaluation. In some implementations, the evaluation component 1140 can include labeling and / or marking data as training data. The system can determine the quality score of the model. The system can compare the quality score of the model with a threshold quality score. If the quality score of the model is higher than the threshold quality score, the system can determine that the model is ready for use. If the quality score of the model is lower than the threshold quality score, the system can determine that the model requires additional training and / or adjustment before use.

[0166] In some implementations, the input data may include one or more features associated with an instance or example. In some implementations, the one or more features associated with an instance or example may be organized into a feature vector. In some implementations, the output data may include one or more predictions. Predictions may also be referred to as inferences. Thus, given features associated with a particular instance, a machine learning model may output a prediction for such instance based on the features.

[0167] The machine learning model may be or include one or more of a variety of different types of machine learning models. Specifically, in some implementations, the machine learning model may perform classification, regression, clustering, anomaly detection, recommendation generation, and / or other tasks.

[0168] In some implementations, data 1130 includes aggregated data 1132, proxy data 1134, and / or user input data 1138. In some embodiments, data 1130 may include instances of substantially any kind or type of data that may describe a variety of phenomena. In general, an instance refers to a set of one or more data values ​​that are combined together to describe a particular subject or subject matter. For example, an instance may be a feature vector. An instance may be associated with image data (e.g., a feature vector of an image, a hashed image, etc.). An instance may be associated with a measurement or other data collection event (e.g., at a particular time, or of a particular subject, or using a particular device, or from a particular perspective, etc.). An instance may be associated with a network session, such as a set of interactions with a web server. In some embodiments, an instance may be associated with a user's interaction with web content (e.g., anonymous or identified).

[0169] In some embodiments, the custom language model 1122 can be trained using knowledge distillation. For example, the custom language model 1122 can be a student model, and the large language model 1126 can be a teacher model. A knowledge distillation training process can be performed to train the custom language model 1122 based on the pre-trained large language model 1126. For example, a large language model can be a large model generated and trained using a large amount of computing resources. For example, a large language model can be a language model composed of a neural network with many parameters (e.g., millions, billions, or trillions of weights). The language model can be trained on a large amount of unlabeled text through self-supervised learning or semi-supervised learning. In order to save resources, the custom language model 1122 can be trained using knowledge distillation to generate results similar to the pre-trained large language model 1126.

[0170] In some implementations, prompts can be used to adjust and / or train the model of the content assistant component 1120. For example, the computing system can obtain prompts indicating sample inputs and outputs. The sample inputs and outputs can be used to adjust the model to familiarize the types of inputs and outputs required for a particular implementation. For example, a particular implementation may include generating advertisements. The generation of advertisements requires understanding the nuances of the business, business goals, products, etc. In this way, by prompting one or more models (e.g., custom language models 1122 and / or large language models 1126), the computing system can adjust the model to provide higher quality output (e.g., output 1150 that can be evaluated using the evaluation component 1140). In some cases, after the evaluation component 1140 has performed an evaluation, the data can be aggregated and used as data 1130 for further model training.

[0171] In some implementations, the content assistant component 1120 can be a machine learning model. In some implementations, training and adjustment can include using data 1130. Aggregate data 1132 can include data obtained from one or more user sessions that have been aggregated by the system (e.g., computing system 202). Proxy data 1134 can include, for example, output data obtained from the content assistant component 1120. In some implementations, the proxy data 1134 can be labeled and / or marked as representing sample inputs and / or outputs. User input data 1138 can include user-generated inputs, including inputs and / or outputs associated with content item wishes and generated content items.

[0172] The trained model can be used to generate one or more suggested content items and / or update a structured user interface and / or build a workflow. In some embodiments, a simulated user model can be trained using training data to generate simulated inputs based on patterns learned from the training data. In some cases, the simulated user model can provide a simulated environment in which a reinforcement learning agent, a notification element subsystem can be trained.

[0173] Fig.12 An example content assistant process 1200 for determining when to provide a prompt to a user for display to obtain additional user input through a user interface is depicted. At 1202, an initial prompt can be provided for display through a user interface. At 1205, the system can determine whether user input has been obtained. If user input has been obtained, the system can proceed to 1210. At 1210, the content assistant can obtain one or more keyword recommendations. At 1215, the content assistant can generate suggested content. After 1215, the system can proceed to 1220. If user input has not been obtained at 1205, the computing system can proceed to 1220.

[0174] At 1220, the computing system may determine whether a prompt for more information should be provided for display through a user interface (e.g., through a content assistant component). For example, at 1225, the computing system may determine a content score and compare the content score to a threshold score value. For example, the score may be a numerical score or some other form of rating. The score may be determined based on the number of titles, descriptions, images, or other content elements selected or populated in the content activity interface. At 1230, the computing system may determine whether the content score is above a threshold. If the content score is not above a threshold, the computing system may proceed to step 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0175] If the content score is not above the threshold, the system can proceed to 1235. At 1235, the system can determine whether at least X minutes have passed since the last time a prompt was provided for display through the user interface. X can be any number of minutes (or can be a number of seconds) that can be automatically updated. Additionally or alternatively, X can be manually selected or entered. If X minutes have not passed, the computing system can proceed to 1255. At 1255, the content assistant component can periodically poll the client device to determine whether updated user input has been obtained.

[0176] If X minutes have passed, the computing system can proceed to 1240. At 1240, the computing system can determine whether question Y has been provided for display through the user interface. For example, the computing system can save a log of the current communication session. The computing system can determine the meaning of question Y and compare it with the log of the current communication session. If the computing system determines that question Y has been provided for display (e.g., asked to the user), the computing system can proceed to 1255. At 1255, the content assistant component can periodically poll the client device to determine whether updated user input has been obtained.

[0177] If the computing system determines that question Y has not yet been provided for display, the computing system can proceed to 1245. At 1245, the computing system can determine whether user input indicating typing has been obtained within Z seconds. Z can be any amount of time (e.g., seconds, minutes). For example, the computing system can automatically determine the Z time based on a determined average amount of time between received inputs. Additionally or alternatively, the Z time can be set (e.g., by manual input provided by the user). If the computing system determines that user input indicating typing has been obtained within Z seconds, the computing system can proceed to 1255. At 1255, the content assistant component can periodically poll the client device to determine whether updated user input has been obtained.

[0178] If the computing system determines that user input indicating typing has not been obtained within Z seconds, the computing system can continue at 1250. At 1250, the computing system can provide a prompt for display through the user interface. The prompt can include a message prompting the user to provide additional user input. The additional user input can be a free-form input, a selection of a selectable screen element, or any other user input.

[0179] As discussed herein, at 1255, the content assistant may periodically poll the client device to obtain user input. The computing system may proceed to 1260. At 1260, the content assistant component may determine that the user input has been obtained. In response, the content assistant may determine the intent of the user input. For example, the computing system may determine the semantic intent through natural language processing (e.g., through a large language model).

[0180] The computing system may determine that the content assistant is unable to provide a recommendation or perform an action related to the intent. In response, the computing system may proceed to 1265. At 1265, the content assistant may provide a message for display, the message including an indication that the determined intent could not be assisted. The computing system may proceed to step 1255. At 1255, the content assistant component may periodically poll the client device to determine whether updated user input has been obtained.

[0181] The computing system can determine that the content assistant can provide a recommendation or perform an action related to the intent. In response, the computing system can proceed to 1215 and the content assistant can generate suggested content as discussed herein.

[0182] Fig.13 An example content assistant flow loop 1300 is depicted for providing recommendations to update a content creation interface until a prediction score for generated content exceeds a threshold. At 1305, the computing system can evaluate the acquired user input to determine whether a prompt to acquire the user input should be provided for display. For example, the computing system can determine the intent of the acquired user input (e.g., a desire to generate a new content activity, inquiring about the performance of various settings for a content activity) and evaluate whether a response should be provided for display via a user device.

[0183] At 1310, the computing system may determine whether a prompt to obtain additional user input is provided for display. If no prompt to obtain additional user input is provided for display, the computing system may proceed to 1315. At 1315, the content assistant may periodically poll the client device to obtain updated user input. If a prompt to obtain additional user input is provided for display, the computing system may proceed to 1320. At 1320, the computing system may determine whether there are necessary model capabilities for the subject of the obtained user input. For example, the computing system may determine whether an answer or response can be generated in response to the user input. If the necessary model capabilities do not exist, the computing system may proceed to step 1340. At 1340, the computing system may provide general questions about obtaining more information through the user input component of the user interface for display.

[0184] If the necessary model capabilities exist for the topic, the computing system can proceed to 1325. At 1325, the computing system can determine whether one or more salient attributes are missing. If the computing system determines that there are missing salient attributes, the computing system can proceed to 1330. At 1330, the computing system can provide for display directed questions related to the missing attributes to be obtained through the user input component of the user interface. For example, the missing salient attributes can include a title, a business name, a location, or other relevant information related to the topic of the content item to be generated. The computing system can proceed to 1345 to obtain additional user input in response to providing questions for display through the user interface.

[0185] If the computing system determines that there are no missing significant attributes, the computing system can proceed to 1340. At 1340, the computing system can provide general questions for display regarding obtaining more information through a user input component of the user interface. After 1340, the computing system can proceed to 1340.

[0186] Fig.14A , Fig. 14B and Fig. 14C A flow chart of an example method 1400 is depicted for use in performing according to an example embodiment of the present disclosure. FIG. 14A to FIG. 14C The steps performed in a particular order are depicted for purposes of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. The various steps of method 1400 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0187] At (1402), method 1400 may include obtaining natural language input from a language model through a dialog activity assistant interface. For example, the computing system may obtain natural language input from a language model through a dialog activity assistant interface. As described herein, the natural language input may include a URL, a description, a question, or other natural language input.

[0188] At (1404), method 1400 may include generating, by a custom language model, an output including a predicted user intent. For example, the computing system may generate, by a custom language model, an output including a predicted user intent. As described herein, the custom language model may be a model trained using a distillation learning technique. For example, the custom language model may be trained based on a previously trained large language model.

[0189] In some cases, a custom language model can be trained to become an expert on a specific topic. This enables the custom language model to better determine the predicted user intent based on the current context of the conversation and the data the custom model was trained on.

[0190] At (1406), method 1400 may include determining one or more actions to perform. For example, the computing system may determine one or more actions to perform. For example, the action may include populating an input field, updating a user interface to open a new window, adjusting a setting within a user interface, providing a recommendation, a visual indication of an update to an input field, or other actions.

[0191] like Fig. 14B As shown, at step (1406), method 1400 may include steps (1418), (1420), and (1422).

[0192] For example, at (1418), step (1406) may include parsing the output generated by the custom language model. For example, the computing system may parse the output generated by the custom language model. For example, the output generated by the custom language model may include user intent.

[0193] For example, at (1420), step (1406) may include determining an action associated with the output. For example, the computing system may determine an action associated with the output. For example, the user intent may include a specific action to be performed, a question, a recommendation request, or another action.

[0194] For example, a user may request recommendations for bidding strategy settings to optimize spend or content spend return (e.g., return on ad spend). Settings may be customized based on the context of the user session. For example, recommendations may be based on prior input received in a conversation, information collected from a website associated with the user session, or other contextual information. In some cases, a user may request information about how to understand analytical data, a request for data to be exported in a particular format, or any other request. For example, a user may indicate the purpose for which the data needs to be exported. A request to export data in a spreadsheet for manual analysis may result in a different file type being generated than the type of data exported for analysis through some other application. Applications may be associated with a user device, a third-party service, and the like.

[0195] A content campaign may include multiple parameters. The parameters may include, for example, the desired audience (e.g., target audience), the amount of money to pay for the content displayed and performance metrics (e.g., bidding), or the content that can be displayed (e.g., creative). Content campaigns associated with an organization may be large in scale and managed by multiple people. There may be thousands of separate campaigns within an organization, which may include thousands of keywords, landing pages, business use cases, business units, etc. Business use cases may include customer acquisition, competitor positioning, new product launches, increasing brand awareness, etc. Business units may include products and services, countries, languages, or geographic regions. Due to the lack of coordination among these campaigns, the management of these campaigns may be disjointed and result in duplicate processing and display of content.

[0196] The content assistant component can be trained to provide recommendations for improved allocation of computing resources. For example, the content assistant component can be trained to adjust various parameters associated with content activities or provide recommendations for adjusting the parameters to reduce redundant data transmission. In addition, performance is improved. For example, the content assistant component can obtain information about performance indicators, determine whether changes in indicators are due to parameter changes or external reasons, and provide recommendations or automatically adjust parameters throughout the organization's content activities.

[0197] For example, the content assistant component can monitor one or more performance indicators. The content assistant component can provide notifications for display in response to determining statistically significant changes in performance indicators. The content assistant component can determine the explanation of performance indicator changes. The content assistant component can automatically update the user interface to display updated information. For example, the updated user interface can include providing parameter update recommendations associated with content activities, providing messages indicating the reasons for performance indicator changes, or automatically adjusting parameters and providing notifications summarizing the changes made. The explanation of performance indicator changes can be determined based on obtaining multiple signals. Signals can include, for example, changes in query volume, changes in content elements (e.g., keywords, descriptions, creative assets), or changes in the display of the content items of the organization compared to the content items provided for display associated with other organizations.

[0198] Adjustment of parameters may include adding or removing content elements (e.g., descriptions, keywords, images, audio, video), adjusting bidding strategies, or other adjustments. As discussed herein, adjustments may be made automatically and may be updated continuously or periodically. The content assistant component may continuously check content campaign performance indicators and provide recommendations and updates in real time. In some implementations, changes may include reducing the number of keywords used, determining the intent of one or more campaigns and merging the campaigns, determining that the campaigns are heterogeneous and should be divided into multiple campaigns, or recommending different campaign types.

[0199] For example, at (1422), step (1406) may include generating an action data structure that includes executable instructions that cause a processor to perform operations associated with completing the action. For example, the computing system may generate an action data structure that includes executable instructions that cause a processor to perform operations associated with completing the action. The action may include performing the operation within the content creation structured interface, or may include performing the operation using an unrelated application.

[0200] The action data structure may include converting the data generated by the custom language model into a format that can be input into the content creation structured interface. This may include determining what data to fill which input fields with, what data to provide for display to the user, which data should remain invisible to the user (e.g., metadata), etc. This may allow the output generated by the custom language model to be utilized by both the content creation structured interface and the content assistant component of the conversation activity interface.

[0201] At (1408), method 1400 may include determining a natural language response. For example, the computing system may determine a natural language response. For example, the natural language response may include an indication of updated fields in the content creation structured interface, a plurality of recommended content elements to be added to the content creation structured interface, a generated summary, etc.

[0202] like Fig. 14C As depicted, at step (1408), method 1400 may include steps (1424) and (1426).

[0203] For example, at (1424), step (1408) may include parsing the output generated by the custom language model. For example, the computing system may parse the output generated by the custom language model. The output generated by the custom language model may include data extracted from a website associated with the URL provided by the user. The extracted data may be generated in a form that would not be easily accessible or understandable if displayed in a raw format. However, the computing system may parse the generated output to obtain relevant portions of the output for generating the response data.

[0204] For example, at (1426), step (1408) may include generating a response data structure including a natural language response to the obtained natural language input. For example, the computing system may generate a response data structure including a natural language response to the obtained natural language input. For example, the computing system may obtain the data extracted from step (1428) and convert the data into a natural language output that can be understood by the user.

[0205] At (1410), method 1400 may include transmitting an action data structure including executable instructions to the action component, the executable instructions causing the action component to automatically perform operations associated with completing the action. For example, the computing system may transmit an action data structure including executable instructions to the action component, the executable instructions causing the action component to automatically perform operations associated with completing the action.

[0206] The action data structure may include data to be populated within one or more fields of the content creation structured interface.

[0207] The action component may include a content activity performance model. The content activity performance model may be configured to ingest data about content activity. The content activity performance model may be a general model that provides information about content activity performance for multiple users or activity types. For example, the content activity performance model may provide general recommendations and strategies for improving content performance.

[0208] The action component may include a content campaign analytics model. The content campaign analytics model may be configured to provide insight into content campaign performance. For example, the data may include conversion data, view data, click data, spend data, or any other data indicative of content campaign performance. For example, the natural language input may be a question about the number of views of a particular content item, or which content items are performing better than other content items. The content campaign analytics model may provide customized recommendations for a particular content campaign or sub-campaign managed through the user interface.

[0209] The action component may include a bidding strategy model. The bidding strategy model may provide personalized spending allocation recommendations based on previous performance. Previous performance may be associated with a specific returning user and based on historical data. Additionally or alternatively, the bidding strategy model may obtain additional user input to learn more about the user's associated business, customer base, or other baseline knowledge for generating recommendations.

[0210] The action component may include a generative model. The generative model may take an action data structure as an input prompt and generate an output including a creative asset. For example, a natural language input may include a uniform resource locator (URL). The action component may include a generative model that generates a summary of information parsed from a page (e.g., a website) associated with the URL. The generative model may be a machine learning model. The generative model may be trained using knowledge distillation techniques. Additionally or alternatively, the creative asset may include images, sounds, videos, or other forms of content.

[0211] At (1412), method 1400 may include transmitting a response data structure to the conversation activity assistant interface, the response data structure including a natural language response provided for display to the user through the conversation activity assistant interface. For example, the computing system may transmit a response data structure to the conversation activity assistant interface, the response data structure including a natural language response provided for display to the user through the conversation activity assistant interface.

[0212] At (1414), method 1400 may include, after transmitting the action data structure and the response data structure, obtaining user input indicating verification of the action data structure or the response data structure. For example, the computing system may, after transmitting the action data structure and the response data structure, obtain user input indicating verification of the action data structure or the response data structure. In some implementations, verification may be performed automatically. Additionally or alternatively, verification may be performed manually. For example, Figure 2 , Figure 3 and Fig.11 An example approach to validation is discussed in .

[0213] At (1416), method 1400 may include updating the custom language model based on the user input. For example, the computing system may update the custom language model based on the user input. As described herein, updating the custom language model may include automatically adjusting the model (e.g., weights, parameters) to change the output generated by the model. The model may be automatically updated in real time. Additionally or alternatively, the model may be updated offline (e.g., trained).

[0214] Figure 15 to Figure 2 1 depicts an example user interface including a content assistant component according to an example embodiment of the present disclosure. The computing system can provide a content assistant component and a content creation structured interface for display through a user interface. The content assistant component and the structured interface can include multiple interactive selectable elements. The elements can include clickable buttons, input fields, or other similar elements. The user can provide input through the content assistant component. The computing system can process the input to determine a natural language response to provide for display through the content assistant component, and simultaneously determine one or more actions to be completed in the content creation structured interface (e.g., a form to be filled in or a button to be selected). In some cases, one or more actions may additionally or alternatively include additional actions outside the content creation interface. Each of the following figures will discuss the features and functions of the example user interfaces described herein.

[0215] Fig.15Depicted is an example user interface 1500, which includes a content assistant component 1505 and a content creation structured interface 1510. The content assistant component 1505 can obtain user input 1515, which includes the URL of a website associated with a user (or a user's business). In response to obtaining the user input 1515 including the URL of the website, the content assistant component can generate a natural language response 1520. The natural language response 1520 can include a summary of the website content associated with the URL generated by the content assistant component. For example, the content assistant component can parse the website associated with the URL to generate a natural language response 1520. In response to the user providing the URL, the computing system can generate a summary of the website content. The content assistant component 1505 can provide a summary to the user for display, and request the user to provide an input indicating that the summary is accurate. For example, the user can select a button indicating acceptance, typing or speaking to send a text message, or provide other inputs. The system can obtain input and update the user interface 1500 accordingly.

[0216] The content assistant component can be displayed in a variety of formats. For example, a format may include a certain number of free-form text input fields (e.g., Figure 8 ), one or more progressive disclosure fields (e.g., Fig. 9 ) and / or a dialogue interface (e.g., as depicted) Fig.10 depicted).

[0217] Fig.16 Depicted is an example user interface 1600, which includes a content assistant component 1605 and a content creation structured interface 1610. The content assistant component 1605 can obtain user input and provide a response generated as discussed herein for display. The content creation structured interface 1610 may include one or more interactive components. For example, a component may include a free-form input or selectable element. For example, a user input indicating the selection of one of the fields may result in an expansion of the selected field. In some cases, a field may develop based on direct user input or input obtained by the content assistant component 1605 of the user interface 1600. For example, the content assistant component may obtain user input, including a description of a product or service provided by a company associated with the user. The content assistant component 1605 may process input and generate an action or natural language response. For example, a natural language response may be provided to be displayed by the content assistant component 1605, and the user interface 1600 may be automatically updated to be filled with one or more recommended content elements.

[0218] like Fig.17As depicted, user interface 1700 can be automatically updated to include populated interactive user interface elements. For example, user interface 1700 can include content creation structured interface 1710. Content creation structured interface can include multiple keyword suggestions. Keyword suggestions can be pre-filled based on the input obtained by content assistant component 1705 processing. Content assistant component 1705 can input the user input obtained into a language model to determine the intention associated with the input. This can include determining the subject of the input, extracting keywords, parsing the input to determine and extract the most relevant part, parsing the website provided by the user to determine the meaning of the additional input in the context of the website content or other related determinations. Content assistant component 1705 can be automatically updated to provide updated prompts through the user interface. For example, the updated prompt can be an answer to a question, an indication of which fields have been filled in the user interface 1700, a request for additional information, a targeted question, a general question, or any other message or inquiry.

[0219] Fig.18 An example user interface 1800 is depicted. The user interface 1800 may include a content assistant component 1805 and a content creation structured interface 1810. The content assistant component 1805 may be automatically updated to provide for displaying additional messages or selectable elements. For example, Fig.18 As depicted, the selectable input may include a "view image" button that may be selected or clicked to update the user interface 1800 to display one or more images that may be selected to be added to the content item. The content creation structured interface 1810 may include multiple sections. For example, the sections may include a middle section that includes a content score and a content item preview. The content score may be generated based on the number of completed input fields within the content creation structured interface or the quality of the input within the completed input field. The completed input field may include multiple titles that may be generated by the content assistant component.

[0220] Fig.19 An example user interface 1900 is depicted. The user interface 1900 may include a content assistant component 1905 and a content creation structured interface 1910. The content creation structured interface 1910 may include multiple sections. For example, the sections may include a middle section that includes a content score and a content item preview. The content score may be generated based on the number of completed input fields within the content creation structured interface or the quality of the input within the completed input field. The completed input field may include multiple descriptions that may be generated by the content assistant component. The input field may include multiple images selected to generate the content item.

[0221] The content assistant component 1905 may provide a plurality of selectable content elements. For example, the selectable elements may include a plurality of titles, descriptions, images, or other content elements. The computing system may obtain user input through the user interface 1900. For example, the user input may indicate a selection of one or more content elements to be added to the content creation structured interface 1910. In response to receiving the user input through the user interface 1900, the computing system may automatically update the user interface to merge the content elements selected from the content assistant component 1905 into the content creation structured interface 1910.

[0222] Fig. 20 Depicted is an example updated user interface 2000. The user interface 2000 can include a content assistant component 2005 and a content creation structured interface 2010. The content assistant component 2005 can be updated to include an indication of the selected content element. For example, once a content element is selected, a visual indicator such as a check mark can replace the additional element. Additionally or alternatively, the content creation structured interface 2010 can be updated to include multiple selected content elements. Therefore, interaction with the content assistant component 2005 can directly affect the content creation structured interface. This can include adjustments to a content strength score, a preview, or multiple input fields.

[0223] The techniques discussed herein refer to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and divisions of tasks and functions between and among components. For example, the processes discussed herein may be implemented using a single device or component or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0224] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, variations, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations, and equivalents.

[0225] The steps depicted or described are merely illustrative and may be omitted, combined, and / or performed in an order different from that depicted or described; the numbering of the depicted steps is merely for ease of reference and does not imply that any particular order is necessary or preferred.

[0226] The functions and / or steps described herein may be embodied in computer-usable data or computer-executable instructions executed by one or more computers or other devices to perform one or more functions described herein. Typically, such data and / or instructions include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific data types when executed by one or more processors in a computer and / or other data processing device. Computer-executable instructions may be stored on computer-readable media such as hard disks, optical disks, removable storage media, solid-state memory, read-only memory (ROM), random access memory (RAM), etc. As will be understood, the functionality of such instructions may be combined and / or distributed as needed. In addition, the functionality may be embodied in firmware and / or hardware equivalents such as integrated circuits, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. Specific data structures may be used to more effectively implement one or more aspects of the present disclosure, and such data structures are contemplated to be within the scope of computer-executable instructions and / or computer-usable data described herein.

[0227] Although not required, it will be appreciated by those of ordinary skill in the art that the various aspects described herein may be embodied as methods, systems, devices, and / or one or more computer-readable media storing computer-executable instructions. Thus, the various aspects may take the form of all-hardware embodiments, all-software embodiments, all-firmware embodiments, and / or embodiments combining software, hardware, and / or firmware aspects in any combination.

[0228] As described herein, various methods and actions may operate across one or more computing devices and / or networks. Functionality may be distributed in any manner, or may be located in a single computing device (eg, server, client computer, user device, etc.).

[0229] Various aspects of the present disclosure have been described with respect to illustrative embodiments thereof. Numerous other embodiments, modifications and / or variations within the scope and spirit of the appended claims may occur to one of ordinary skill in the art upon review of the present disclosure. For example, one of ordinary skill in the art may appreciate that the steps depicted and / or described may be performed in an order different from that described, and / or one or more of the steps shown may be optional and / or combined. Any and all features in the appended claims may be combined and / or rearranged in any possible manner.

[0230] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, variations and / or equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, variations and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown and / or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, variations and / or equivalents.

Claims

1. A computer-implemented method comprising: Obtain natural language input from a custom language model through the conversational activity assistant interface; generating, by the custom language model, an output including a predicted user intent; The action or actions to be performed are determined by: parsing the output generated by the custom language model; determining an action associated with the output; generating an action data structure comprising executable instructions that cause a processor to perform operations associated with completing the action; The natural language response is determined by: parsing the output generated by the custom language model; generating a response data structure comprising a natural language response to the acquired natural language input; transmitting the action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform operations associated with completing the action; transmitting the response data structure to the conversation activity assistant interface, the response data structure including the natural language response provided for display to a user via the conversation activity assistant interface; After transmitting the action data structure and the response data structure, obtaining user input indicating validation of the action data structure or the response data structure; as well as The custom language model is updated based on the user input. 2 . The computer-implemented method of claim 1 , wherein the action data structure comprises data to be populated within one or more fields of a content creation structured interface.

3. A computer-implemented method as described in any one of claims 1 or claim 2, wherein the natural language input includes a uniform resource locator (URL), and wherein the action component includes a generation model that generates a summary of information parsed from a page associated with the URL.

4. The computer-implemented method of any one of claims 1 to 3, wherein the action component comprises a content activity performance model.

5. The computer-implemented method of any one of claims 1 to 4, wherein the action component comprises a content activity analysis model.

6. A computer-implemented method as described in any one of claims 1 to 5, wherein the action component includes a bidding strategy model.

7. The computer-implemented method of any one of claims 1 to 6, wherein the action component comprises a generative model, wherein the generative model takes the action data structure as an input prompt and generates an output comprising a creative asset.

8. A computer-implemented method comprising: Obtain natural language input from the language model through the dialogue activity assistant interface; Generate output including predicted user intent from a custom language model; The action or actions to be performed are determined by: parsing the output generated by the custom language model; determining an action associated with the output; generating an action data structure comprising executable instructions that cause a processor to perform operations associated with completing the action; The natural language response is determined by: parsing the output generated by the custom language model; generating a response data structure comprising a natural language response to the acquired natural language input; transmitting the action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform operations associated with completing the action; transmitting the response data structure to the conversation activity assistant interface, the response data structure including the natural language response provided for display to a user via the conversation activity assistant interface; as well as After transmitting the action data structure and the response data structure, user input indicating validation of the action data structure or the response data structure is obtained.

9. The computer-implemented method of claim 8, comprising: The custom language model is updated based on the user input.

10. A computer-implemented method comprising: Obtain natural language input from the language model through the dialogue activity assistant interface; Generate output including predicted user intent from a custom language model; The action or actions to be performed are determined by: parsing the output generated by the custom language model; determining an action associated with the output; generating an action data structure comprising executable instructions that cause a processor to perform operations associated with completing the action; The natural language response is determined by: parsing the output generated by the custom language model; generating a response data structure comprising a natural language response to the acquired natural language input; transmitting the action data structure including executable instructions to an action component, the executable instructions causing the action component to automatically perform operations associated with completing the action; as well as The response data structure is transmitted to the conversation activity assistant interface, the response data structure including the natural language response provided for display to a user through the conversation activity assistant interface.

11. The computer-implemented method of claim 10, wherein the action component comprises a generative model, wherein the generative model takes the action data structure as an input prompt and generates an output comprising a creative asset.

12. The computer-implemented method of claim 11, wherein the generative model comprises a machine learning model.

13. The computer-implemented method of claim 12, wherein the generative model is trained using a knowledge distillation technique.

14. A computer-implemented method comprising: providing input data to a first machine learning model; obtaining output data indicating one or more suggested content item components; Evaluating the first machine learning model based at least in part on the output data by: obtaining data indicative of a quality score associated with the output data of the first machine learning model; comparing the quality score associated with the output data of the first machine learning model to a threshold quality score; determining that the first machine learning model has a quality score above the threshold quality score; as well as In response to determining that the first machine learning model has a quality score above the threshold quality score, implementing the first machine learning model in a content creation process.

15. The computer-implemented method of claim 14, comprising: generating an initial user interface including a content assistant component; Get data indicating user input; processing, by the first machine learning model, the data indicative of user input; obtaining output data indicative of one or more content item components from the first machine learning model; transmitting data causing the one or more content item components to be provided for display on a user interface; obtaining data indicative of a user selection of approval of the one or more content item components; as well as In response to obtaining the data indicative of the user selection of the approval of the one or more content item components, one or more content items are generated including a plurality of the content item components.

16. The computer-implemented method of claim 15, wherein the content assistant component includes one or more input fields.

17. A computer-implemented method as described in claim 14 or claim 15, wherein the first machine learning model has been trained using a knowledge distillation training method.

18. A computing system as described in any one of claims 14 to claim 17, wherein the first machine learning model has been trained at least in part based on the output of a pre-trained second machine learning model.

19. The computing system of claim 18, wherein the pre-trained second machine learning model is a large language model.

20. The computing system of claim 19, wherein the pre-trained second machine learning model is tuned using one or more hints.

21. The computing system of any one of claims 14 to 20, wherein the input data comprises at least one of free-form input or landing page content.

22. The computing system of claim 21, wherein the input data comprises natural language input.

23. The computing system of claim 14 or claim 15, wherein the one or more content item components include at least one of a title or a description.

24. A computer-implemented method comprising: generating an initial user interface including a content assistant component; obtaining data indicative of input received from a user; processing, by a machine learning model interacting with the content assistant component, the data indicative of the input received from the user; obtaining output data indicative of one or more content item components from the machine learning model interacting with the content assistant component; transmitting data causing the one or more content item components to be provided for display via an updated user interface; obtaining data indicative of a user selection of approval of the one or more content item components; as well as In response to obtaining the data indicative of the user selection of the approval of the one or more content item components, one or more content items are generated.

25. The computer-implemented method of claim 24, comprising training the machine learning model using a knowledge distillation training method.

26. The computer-implemented method of claim 24 or 25, comprising training the machine learning model by: inputting the labeled data into a first machine learning model; Obtaining output data from the first machine learning model; comparing the output data from the first machine learning model to the output data from a second machine learning model; and The first machine learning model is adjusted based on comparing the output data of the first machine learning model and the second machine learning model.

27. A computer-implemented method as described in claim 26, wherein the labeled data includes data output by a second machine learning model, wherein the second machine learning model is a pre-trained model.

28. The computer-implemented method of claim 27, wherein the output of the second machine learning model comprises annotated data.

29. A computer-implemented method as described in any one of claims 26 to 28, wherein the tagged data includes at least one of: (i) business and product descriptions, (ii) proxy data from a website associated with a content creator, (iii) manually annotated data, or (iv) free-form input.

30. The computer-implemented method of any one of claims 24 to 29, wherein training the machine learning model comprises: Inputting unlabeled data into the machine learning model and a pre-trained second machine learning model; Obtaining output data from the machine learning model and the pre-trained second machine learning model; comparing the output data from the machine learning model to the output data from the pre-trained second machine learning model; as well as The machine learning model is adjusted based on comparing the output data of the machine learning model with the output of the pre-trained second machine learning model.

31. The computer-implemented method of any one of claims 24 to 30, wherein the initial user interface comprises a graphical user interface.

32. A computer-implemented method comprising: providing input data to a first machine learning model; obtaining output data indicating one or more suggested content item components; Evaluating the first machine learning model based at least in part on the output data by: obtaining data indicative of a quality score associated with the output data of the first machine learning model; comparing the quality score associated with the output data of the first machine learning model to a threshold quality score; determining that the first machine learning model has a quality score above the threshold quality score; In response to determining that the first machine learning model has a quality score above the threshold quality score, implementing the first machine learning model in a content creation process.

33. The computer-implemented method of claim 32, comprising: generating an initial user interface including a content assistant component; obtaining data indicative of input received from a user; processing, by the first machine learning model, the data indicative of the input received from the user; obtaining output data indicative of one or more content item components from the first machine learning model; transmitting data causing the one or more content item components to be provided for display on a user interface; obtaining data indicative of a user selection of approval of the one or more content item components; as well as In response to obtaining the data indicative of the user selection of the approval of the one or more content item components, one or more content items are generated.

34. A computing system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions executable to cause the one or more processors to perform operations including the method of any one of claims 1 to 33.

35. One or more non-transitory computer-readable media storing instructions executable by one or more processors to perform operations comprising the method of any one of claims 1 to 33.

Images