Content generation using pre-existing media assets using generative machine learning models

Abstract

An example embodiment of the present disclosure provides an example method that includes generating a media asset based on a pre-existing media asset. The method includes extracting a signal including background image data, palette data, text data, or logo image data. The method includes generating background image assets, textual assets, and unique profile data based on the extracted signals. The method includes transmitting the media asset to a content item generation pipeline to generate a plurality of new candidate content items.

Description

[0001] priority

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 501,191, filed on May 10, 2023, which is incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to using generative machine learning models to automatically generate content items or media assets based on a user's profile or preferences and existing media assets. Background Art

[0004] A communication campaign can utilize a multimodal, multiplatform distribution system to distribute content items to different endpoints for different audiences. A content item can contain data or other information or a message. A content item can be or include a media asset. A user can create a communication campaign by providing a set of content items to a multimodal, multiplatform distribution system for distribution. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computing system for generating content items. The computing system may include one or more processors and one or more non-transitory computer-readable media. The computer-readable medium may collectively store a machine-learned generation model, a machine-learned selection model, and instructions. The machine-learned generation model may be configured to generate multiple content items. The machine-learned selection model may be configured to select a selected content item from multiple content items. The instructions, when executed by one or more processors, cause the computing system to perform operations. The operations may include receiving data indicating a request for multiple media assets, the multiple media assets including multiple media modalities to be generated based on pre-existing media assets. The operations may include generating a plurality of media asset components by: (i) extracting one or more signals from pre-existing media assets, the one or more signals including at least one of background image data, color palette data, text data, or logo image data; (ii) using a first generative machine learning model to generate a plurality of background image assets by: generating a mask image including one or more bounding boxes based on at least one of the background image data, the text data, or the logo image data; inpainting the mask image by generating pixels using the machine learning model to fill the one or more bounding boxes of the mask image; generating a plurality of images, wherein each image includes a different aspect ratio; (iii) using a second generative machine learning model to generate a plurality of text assets by: obtaining text data associated with the one or more bounding boxes; inputting the text data into the machine learning model; generating a plurality of text assets as output by the machine learning model; and (iv) using a third generative machine learning model to generate unique profile data based on at least one of the color palette data or the logo image data. The operations may include automatically transmitting a plurality of media assets including one or more image assets, text assets, and unique profile data to a content item generation pipeline for generating a plurality of candidate content items.

[0007] In some cases, generating a mask image including one or more bounding boxes includes identifying one or more text objects; and for each of the one or more text objects: determining that a first text object in the one or more text objects is an overlay text object. In some cases, generating a mask image including one or more bounding boxes includes: generating a first bounding box for the first text object based on determining that the first object is an overlay text object.

[0008] In some cases, generating the plurality of media asset components includes extracting a logo image. In some cases, generating the plurality of media asset components includes selecting a dominant color from an image palette; upscaling the logo image by: enlarging the logo image; and sharpening the logo image. In some cases, generating the plurality of media asset components includes generatively expanding the logo image to generate one or more images, wherein each image includes a set of different aspect ratios.

[0009] In some cases, generatively expanding the logo image includes generating pixels to blend with existing pixels of the logo image to generate an image that is larger than the original logo image.

[0010] In some cases, generating the plurality of background image assets includes adjusting at least one of brightness, saturation, or contrast of the respective background image assets.

[0011] In some cases, the unique profile data includes at least one of logo data, color palette data, font data, or image style data.

[0012] In some cases, the operations include inputting the plurality of output assets into a content creation pipeline. In some cases, the operations include generating, by the content creation pipeline, a plurality of content items, wherein each content item of the plurality of content items includes a unique combination of a content asset and an aspect ratio.

[0013] In some cases, generating a plurality of text assets comprises extracting a signal comprising text data. In some cases, generating a plurality of text assets comprises compiling the text data. In some cases, generating a plurality of text assets comprises inputting the compiled text data into a second generative machine learning model. In some cases, generating a plurality of text assets comprises obtaining at least one of a short title or a long title as an output from a second generative machine learning model. In some cases, generating a plurality of text assets comprises inputting at least one of a short title or a long title into a second generative machine learning model. In some cases, generating a plurality of text assets comprises obtaining a description as an output from a machine learning model.

[0014] In some cases, the one or more bounding boxes include a location and a size of the bounding box.

[0015] In some cases, a content item generation pipeline includes a prompt generation component and a content item generation component.

[0016] In some cases, the operations include generating, by a prompt generation component, input prompt data based on the background image, the text asset, and the unique profile data. In some cases, the operations include providing the input prompt data to a content item generation component. In some cases, the operations include generating, by the content item generation component, one or more candidate content items based on the input prompt data.

[0017] In some cases, the content item generation component includes a generative machine learning model.

[0018] In some cases, the operations include training a generative machine learning model by: generating a training data set based on comparing generated content item components to existing media assets; and automatically adjusting one or more parameters of the generative machine learning model based on comparing the generated content item components to the existing media assets to reduce the differences between the generated content items and the existing media assets.

[0019] In one example aspect, the present disclosure provides an example computer-implemented method. The example computer-implemented method includes receiving data indicating a request for a plurality of media assets, the plurality of media assets including a plurality of media modalities to be generated based on pre-existing media assets. The example method includes generating a plurality of media asset components by: (i) extracting one or more signals, the one or more signals including at least one of background image data, color palette data, text data, or logo image data; (ii) using a first generative machine learning model to generate a plurality of background image assets based on at least one of the background image data, text data, logo image data, or color palette data; (iii) using a second generative machine learning model to generate a plurality of text assets based on at least one of the text data or the background image data; (iv) using a third generative machine learning model to generate unique profile data based on at least one of the palette data, text data, or logo image data. The example computer-implemented method includes automatically transmitting a plurality of media assets including one or more image assets, text assets, and unique profile data to a content item generation pipeline for generating a plurality of candidate content items.

[0020] In some cases, the method includes receiving data indicative of a pre-existing media asset being uploaded.

[0021] In some cases, the method includes parsing a web resource associated with the unique profile data to obtain pre-existing media assets.

[0022] In some cases, the pre-existing media assets include flattened image data.

[0023] In some cases, the request is associated with a customer account, and wherein the customer account is associated with an account profile that stores inputs to the machine-learned media asset generation pipeline.

[0024] In some cases, the profile is retrieved from a database, and wherein the profile was previously generated prior to the request.

[0025] In some cases, the present disclosure provides an example transient or non-transitory computer-readable medium, which is embodied in a computer-readable storage device and stores instructions that, when executed by a processor, cause the processor to perform operations. In the example transient or non-transitory computer-readable medium, the operation includes receiving data indicating a request for a plurality of media assets, the plurality of media assets including a plurality of media modalities to be generated based on pre-existing media assets. The operation includes generating a plurality of media asset components by: (i) extracting one or more signals, the one or more signals including at least one of background image data, palette data, text data, or logo image data; (ii) using a first generative machine learning model to generate a plurality of background image assets based on at least one of the background image data, text data, logo image data, or palette data; (iii) using a second generative machine learning model to generate a plurality of text assets based on at least one of the text data or the background image data; (iv) using a third generative machine learning model to generate unique profile data based on at least one of the palette data, text data, or logo image data. The operations include automatically transmitting a plurality of media assets including one or more image assets, text assets, and unique profile data to a content item generation pipeline for use in generating a plurality of candidate content items.

[0026] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0027] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0029] Figure 1 A block diagram of an example system is depicted according to an example embodiment of the present disclosure.

[0030] Figure 2 A block diagram of an example system is depicted according to an example embodiment of the present disclosure.

[0031] Figure 3 A block diagram of an example system is depicted according to an example embodiment of the present disclosure.

[0032] Figure 4 A flow chart depicting an example method for generating media assets according to an embodiment of the present disclosure.

[0033] Figure 5 A flow chart depicting an example data flow for generating a media asset according to an embodiment of the present disclosure.

[0034] Figure 6 A block diagram depicts an example method for generating media assets according to an embodiment of the present disclosure.

[0035] Figure 7 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0036] Figure 8 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0037] Fig. 9 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0038] Fig.10 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0039] Fig.11 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0040] Fig.12 An example graphical depiction depicting an asset feedback layer according to an example embodiment of the present disclosure.

[0041] Fig.13 An example graphical depiction depicting an asset feedback layer according to an example embodiment of the present disclosure.

[0042] Fig.14 An example graphical depiction depicting an asset feedback layer according to an example embodiment of the present disclosure.

[0043] Fig.15 An example graphical depiction depicting steps for generating a media asset from existing media assets according to an embodiment of the present disclosure.

[0044] Fig.16 is a flow chart illustrating an example method for training a model for machine learning according to an example implementation of aspects of the present disclosure;

[0045] Fig.17 is a block diagram of an example process flow for processing an input to generate an output using a machine learning model according to an example implementation of aspects of the present disclosure;

[0046] Fig.18is a block diagram of an example sequence processing model according to an example implementation of aspects of the present disclosure;

[0047] Fig.19 is a block diagram of an example technique for populating an example input sequence for processing by a sequence processing model according to an example implementation of aspects of the present disclosure;

[0048] Fig. 20 is a block diagram of an example model development platform according to an example implementation of aspects of the present disclosure;

[0049] Fig.21 is a block diagram of an example training workflow for training a model for machine learning according to an example implementation of aspects of the present disclosure;

[0050] Fig. 22 is a block diagram of an inference system for operating one or more machine-learned models to perform inference according to an example implementation of aspects of the present disclosure;

[0051] Fig.23 is a block diagram of an example networked computing system according to an example implementation of aspects of the present disclosure;

[0052] Fig.24 is a block diagram of an example computing device according to an example implementation of aspects of the present disclosure; and

[0053] Fig.25 is a block diagram of an example computing device according to an example implementation of aspects of the present disclosure.

[0054] Reference numerals repeated in various figures are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0055] The present disclosure generally relates to generating media asset components based on received existing media content items. For example, the systems and methods described herein can decompose existing media content items into components. These components can be generated using generative machine learning models, which can be trained and adjusted based on profiles or requirements. These systems and methods can additionally or alternatively include recombining media asset components so that the generated content items are visually similar (e.g., the vector difference of two images is within a threshold amount). The present disclosure can include utilizing one or more generative machine learning models, such as large language models that can perform optical character recognition, generate bounding boxes for repair, and generate components and assets.

[0056] Existing methods fail to take an initial media asset, such as a flat image, and deconstruct it into its various components. Existing media channels require dynamic asset components to achieve optimal results in different display formats. For example, the size or aspect ratio required for presentation via a mobile device interface may be different than the size or aspect ratio required for presentation on a desktop computer. Additionally or alternatively, within the same device, different applications may require various aspect ratios, thumbnail descriptions, or other distinguishable content features. Therefore, it is important to have separate media asset components that can be combined as needed to meet these dynamic requirements. In some cases, existing media content items may be flattened images that cannot be resized without distorting the image or otherwise degrading the image quality. Therefore, a solution is needed that can take these existing media content items and decompose them into separate media assets that can be recombined.

[0057] In some cases, the initial media content item may include a flattened image in which the pixels between the superimposed text are consistent with the underlying image. Therefore, the superimposed text or other media components cannot be simply removed. In addition, some underlying background images may include text that needs to be distinguished from the superimposed text (for example, the SPF indication on the sunscreen bottle is compared with the graphic that says "Now on Sale"). Therefore, the present disclosure provides a technical advantage over existing systems by distinguishing superimposed text from text integrated into the background image to determine whether to start the warm-up or utilization of other generative models in the workflow. For example, additional models may include models that can fill the background image based on the generated bounding box, generate text assets based on detected superimposed text, or otherwise utilize the extracted data. By determining whether the text is superimposed text or text integrated into the background image, the system can save resources by preventing the use of additional computing resources to fill the background image area that should not be removed. This can prevent redundant processing and waste of resources.

[0058] While there are methods for performing the various actions described herein, there are complexities in using the output from one model as the input to another model to obtain a high-quality media asset component. For example, adjusting the generation of a mask used to determine the portion of an image to be filled may involve iterations of training a model to distinguish text that should be repaired from text that should not be repaired (e.g., text on a bottle within an image versus text overlaid as part of an existing media asset). Additionally, ensuring that the generated media asset is similar to an existing asset to meet requirements may require adjusting and training a generative model to produce a media asset that meets image quality metrics such as clarity or other metrics. The present disclosure improves asset generation by taking an existing asset, extracting text, performing repairs, extending a base image, generating new text, and overlaying the text with the extended base image to generate a new asset.

[0059] The present system and method is used to improve the utilization, training, and adjustment of machine learning models to perform each step of the process of generating new assets. By deconstructing assets into subcomponents, assets are no longer limited to the initial two-dimensional, fixed aspect ratio. Instead, subcomponents can be used to generate assets of various sizes, as well as expand to additional asset types, such as text-only assets, image-only assets, video assets, etc.

[0060] The present disclosure may include post-processing checks that provide feedback data to a model generating an output asset to determine the quality of an image (e.g., based on distortion or blur) and automatically adjust parameters of the system to provide improved image quality (less distortion or blur metrics of previously generated images).

[0061] In some implementations, the present disclosure relates to automatically generating content items or media assets based on a client's profile or preferences. Example implementations are for a media asset generation pipeline using machine learning, based on a media asset profile, generating multiple media assets by instructing a machine learning asset generation model to generate media assets that align with the client's media asset preferences. In addition, example implementations of the present disclosure generally relate to using a machine learning model to generate content based on information extracted from a website. For example, a user can enter a URL into the system, and the system can generate content for the user based on data extracted using the URL. Example techniques include automatically generating multiple content items for a communication campaign, and selecting a content item from the multiple content items based on a prediction of how the selected content item will perform in the communication campaign.

[0062] For example, a customer may be a user associated with a user account. A user may interact with the campaign generation system to create a new communication campaign. A user may interact with the campaign generation system using a user account. The campaign generation system may associate the user account with a set of campaign preferences, which may include a set of media asset preferences. If the user account is associated with other communication campaigns, the media asset preferences may include preferences obtained based on those other campaigns (e.g., preferences directly received via input from the user account, preferences implicitly learned from user account operations, etc.).

[0063] The new communication campaign may convey data that directs the audience to a data resource associated with the campaign. The data may be or include a resource locator (e.g., a URI, URL, deep link, app link, etc.). The data resource may be a web resource (e.g., a web page, a web application, etc.), a local resource (e.g., a native application running on a client device), etc.

[0064] A user may provide a resource locator of a data resource to the campaign generation system. The user may provide this information early in the campaign generation process, such as when initiating the generation of a new campaign. The campaign generation system may process the resource locator to identify the data resource and obtain data from and about the data resource. For example, for a campaign directed to a web page, the campaign generation system may use the provided URL to load the target web page, parse or crawl the page to extract pre-existing media assets (e.g., images, text, video, color palette, typography, etc.) and understand the theme, style, entity branding, etc. associated with the target web page. Other related resources may be parsed. A site map may be used to parse resources on the document tree where the data resource is located. The system may parse resources linked to the data resource, such as based on a relevance metric of the linked resources.

[0065] The activity generation system can predict resource locators to pre-fetch content, thereby improving latency. For example, for a user account with a known association with a known resource locator, the activity generation system can use the known resource locator to begin resolving data resources even before the user confirms the resource locator.

[0066] The activity generation system can pre-fetch content to improve wait time by starting to resolve data resources as soon as a resource locator is entered into an input field, even if the user has not yet completed other input fields on the same interface screen. In this way, for example, by the time the user advances to the next input screen, the resolution operation is already in progress or has been completed, thereby reducing the user's wait time.

[0067] Data parsed from data resources can be used to update media asset preferences. The campaign generation system can use media asset preferences to form or update an account profile that describes the account's communication personality (e.g., brand personality), account assets, performance data from any past campaigns, and learned characteristics of relevant audiences and learned trends or characteristics of the communicator group as a whole. This account profile can be dynamically maintained as campaigns are distributed and updated, as campaign communications are received and used by recipient endpoints, etc. As users interact with the machine learning media asset generation pipeline, the account profile can be dynamically updated to save current progress, current preferences, selections, inputs, signals, etc.

[0068] The campaign generation system may collect additional input signals from the user. The input signals may refine the predicted or pre-populated features of the account profile or media asset preferences. For example, based on the data parsed from the data resources, the campaign generation system may predict preliminary goals, overall themes and styles of the communication campaign, and other data resources related to the communication campaign. The user may refine, update, approve or reject these predictions by providing additional input signals.

[0069] Additional input signals may include product / service name, product / service description (e.g., free-form, multi-line input where the user specifies details about their product / service; suggestions / pre-population may be made), brand characteristics (e.g., adjectives describing the brand; suggestions may be made for a click interface), social media selections (e.g., permissions to assets from social media platforms associated with the user's account), etc. The machine-learned model may provide pre-population for one or more input fields of the signal based on the account profile or parsed data resources. A threshold may be used to pre-populate when a certain confidence level (e.g., corresponding to the quality of the pre-population) is exceeded.

[0070] Additional input signals may persist in association with a user account. Additional input signals may persist in association with assets generated based on the additional input signals. Additional input signals may include metadata indicating whether a particular signal was manually modified by a user. If the user creates another related activity in the future, this persisted signal data may be redisplayed to the user. For example, if a user creates an activity about the same or similar data resource, the signal data may be redisplayed without first re-parsing the data resource. This may improve latency and reduce processing requirements. While parsing the data resource, the signal data may be used as input to a machine-learned asset generation pipeline. A subset of the signal data may be used as input to a machine-learned asset generation pipeline, such as only signals that have been manually confirmed / modified. In this way, for example, a machine-learned asset generation pipeline can learn from user input / corrections and avoid making the same mistakes in future activities.

[0071] The campaign generation system can process the data parsed from the data resources, the account profile data, and the additional input signals to obtain media assets for use in the communication campaign. The campaign generation system can implement a machine learning media asset generation pipeline to retrieve or modify pre-existing media assets, generate new media assets, or retrieve new media assets from a database under the guidance of the account profile data and the additional input signals. For example, the machine learning media asset generation pipeline can generate images (e.g., background images), titles, descriptions, videos, logos, color palettes, site links, and visual styles and themes. The machine learning media asset generation pipeline can retrieve or modify pre-existing images, titles, descriptions, videos, logos, color palettes, site links, and visual styles and themes. The machine learning media asset generation pipeline can query a relevant database (e.g., a stock media asset database) to obtain new images, titles, descriptions, videos, logos, color palettes, site links, and visual styles and themes.

[0072] The machine learning media asset generation pipeline can retrieve or modify pre-existing media assets or pre-existing media content items. The machine learning media asset generation pipeline can parse the data resource to extract any content of the data resource. The content from the data resource can be modified or optimized. For example, the image or video can be resized, the text overlay on the image or video can be removed and filled (for example, using a machine learning repair model), and the image or video can be edited (for example, exposure, colorization, clarity, etc.). Text media assets can be rephrased, edited for clarity, etc. Logos can be recognized, rescaled, optimized for overlay (for example, removing backgrounds, generating alpha channels, etc.), recolored, etc. Other pre-existing assets can be obtained from the media library associated with the account. The media library can include assets used in past campaigns, assets uploaded or generated but not yet used, etc.

[0073] The machine learning media asset generation pipeline can generate media assets using one or more machine learning models. The machine learning media asset generation pipeline can use a machine learning natural language understanding model to parse text on a data resource to understand the content of the data resource and understand the context in which the content is presented (e.g., the style or theme of the data resource). For example, the machine learning natural language understanding model can utilize additional data, such as image data or other context data, to determine whether the text on the data resource is superimposed text or text embedded in the image. Therefore, the model can determine whether further steps related to the location of the content of the text should be performed. For example, the model can determine whether to generate a bounding box for generating a mask for an image to be filled or repaired. Additionally or alternatively, the model can determine whether the detected text should be ignored or should be used in the text generation process (e.g., generating text assets such as titles, descriptions, etc.). The machine learning media asset generation pipeline can obtain a set of asset generation instructions, which can be based on or include any one or more of the following: representations of content and its context, account profiles, media asset preferences, additional new input signals, etc.

[0074] The activity generation system may reference the allow list to determine whether a user account is approved to use the machine learning media asset generation pipeline. For example, activities related to products in sensitive vertical industries may bypass automatic asset generation and request manual control from the user. For example, a user may be required to provide manual input and control to generate assets using the machine learning media asset generation pipeline. A user may be completely locked out of the machine learning media asset generation pipeline.

[0075] The machine learning media asset generation pipeline can use a machine learning image generation model to process asset generation instructions to generate images based on and aligned with the asset generation instructions. Various image generation architectures can be used, including convolutional neural networks, transformers, generative adversarial networks, diffusion models, etc. The image generation model can process images from data resources as example inputs to prompt the model to generate similar images, text descriptions and other signals or instructions of desired images, learned soft prompts, etc. For example, product images from data resources can be provided to the image generation model to prompt the model to include the product in the generated image, expand the picture around the product in a new environment (e.g., generative fill), etc. This is an example of a technique for contextualizing or re-contextualizing product images while improving the faithful reproduction of product attributes. Other example techniques for image asset generation include processing assets from data resources to extract attributes (theme, color, emotion), generating prompts based on asset generation instructions and extracted attributes using a machine learning language model, and inputting prompts or asset generation instructions and prompts into the image generation model.

[0076] A machine learning media asset generation pipeline can process asset generation instructions using a text generation model to generate text based on and aligned with the asset generation instructions. Various text generation architectures can be used, including convolutional neural networks, transformers, generative adversarial networks, diffusion models, and others. Example architectures include encoder-only, encoder-decoder, or decoder-only transformer-based models trained on large text corpora. Text generation models can process images from data sources as example inputs to prompt for relevant descriptions, text prompts describing desired output text and other signals or instructions, learned soft prompts, and others.

[0077] In some examples, the text generation model can process resource locators, text from data resources, free-form text provided by a user or generated by an asset generation pipeline (e.g., using a prompt generator), existing text assets associated with a user account, tone and brand indicators (e.g., adjectives or other descriptors associated with a brand, such as can be obtained from additional signal inputs), etc. The text generation model can be configured to classify text assets (e.g., as calls to action, promotional phrases, descriptions, etc.). The quality of the generated assets can be evaluated (e.g., by the generation model itself, by a quality control model, etc.). This can be used to rank / select text assets later. For example, quality measures can include an assessment of relevance or rootedness about a data resource (e.g., assessing whether "contactless delivery" is a phrase that accurately describes the content of a data resource). For existing text assets, the campaign generation system can process the existing text assets together with any of the above inputs to rewrite the assets (e.g., change the tone, etc.).

[0078] The machine learning media asset generation pipeline can use a video generation model to process the asset generation instructions to generate a video based on and aligned with the asset generation instructions. Various video generation architectures can be used, including convolutional neural networks, transformers, generative adversarial networks, diffusion models, continuous or discrete time cascade diffusion models, etc.

[0079] The machine learning media asset generation pipeline can use an audio generation model to process the asset generation instructions to generate audio based on and aligned with the asset generation instructions. Various audio generation architectures can be used, including convolutional neural networks (e.g., processing spectrograms), transformers (e.g., processing sequences of audio data or embeddings thereof), generative adversarial networks, diffusion models, continuous or discrete time cascade diffusion models, etc.

[0080] The machine learning media asset generation pipeline can use a machine learning prompt generator model to generate prompts for input to other generative models in the pipeline. The machine learning prompt generator model can be trained end-to-end with one or more of the other generative models to improve performance. The prompt generator model can include a language generation model (e.g., a "large language model"). The machine learning media asset generation pipeline can prompt the generative model with a variety of different prompts to obtain a variety of different outputs. For example, the output layer of the prompt generation model can be sampled (e.g., random sampling, top-K sampling, etc.) to obtain a mixed prompt output. The mixture can be input to the corresponding generative model to generate various outputs related to the instruction. The prompt generator can receive a prompt provided by the user and rewrite the prompt based on an expression symbol or image (e.g., "progress" → "a person climbing a mountain"). For example, the prompt can be rewritten by inputting the original prompt and the instruction to the language generation model (e.g., the prompt model "suggests an image associated with 'progress'"). The prompt generator model can receive prompts provided by the user (e.g., obtained in a feedback loop, as described below) or system-rewritten and expand the prompts to be more expressive (e.g., “A person is climbing a mountain” → “A person is climbing a mountain. Photographic, detailed, HDR, high resolution, 4K”).

[0081] Generated assets can be associated with metadata. For example, an image asset created or enhanced using a machine learning media asset generation pipeline can have stored metadata containing information about which tools / pipelines (and which versions) were used to create or enhance the asset. This can flow to assets derived from assets created / enhanced by a machine learning media asset generation pipeline. "Enhancements" can include optimized / optimized features. In this way, the activity generation system can perform analysis on how the enhancements perform (and possibly test against non-enhanced versions). In addition, the activity generation system can facilitate recalling ("pulling down") generated / enhanced assets (or derivative assets) as needed. This can be limited to completely new generated content or to generated content that covers a major portion (20%+) of the image. For images generated using prompts, the prompts can be saved. Any user-entered prompts can be saved as well as any prompts generated by the prompt generator.

[0082] The machine learning media asset generation pipeline can query relevant databases to obtain assets. For example, a database of stock photos or videos can be queried to obtain content similar to assets retrieved from or generated based on the data resource.

[0083] The machine learning media asset generation pipeline can obtain assets (e.g., generate, modify, query a database) based on learned attribute insights. For example, the learned attribute insight model can map a subject (e.g., product, topic, etc.) to additional content or keywords or features (e.g., attributes) associated with better effects. For example, based on learned relationships, an image asset of "dog toy" might be mapped to a depiction of the outdoors and sunlight to achieve better effects. Such insights can be used for asset generation / modification of any type of asset. Such insights can be used to expand or narrow search queries for related assets from an asset database. Such insights can also be displayed to users during the generation workflow for additional information. Such insights can also be provided in prompts (e.g., passed directly to a generative model, passed to a prompt generator, etc.) to improve asset generation.

[0084] The machine learning media asset generation pipeline can optimize the obtained media assets. The optimization can include cropping, repairing, picture expansion, upscaling, recoloring, sharpening, or other editing or modification. The optimization can be implemented by one or more machine learning optimization models (e.g., image editing models, video editing models, audio editing models, etc.). The optimization can be recorded in metadata. The optimization steps can be rolled back by reloading the saved state of the asset from the metadata.

[0085] The machine learning media asset generation pipeline can rank the obtained media assets. For example, the machine learning ranking model can rank the obtained media assets based on the likelihood of the performance of the media asset in the communication activity (for example, the predicted likelihood of a user interacting with the corresponding content item to execute a hyperlink embedded in the content item). The machine learning ranking model can rank the obtained media items based on the relevance to the data resource. The machine learning asset generation pipeline can generate an embedded representation of the data resource and compare the embedded representations of the obtained media items to determine the relevance. The ranking can be based on the source of the image (for example, system generated, crawled from a data resource, user uploaded, etc.). The ranking can be based on image recognition results (for example, the image is recognized as a product described on the data resource). The ranking can be based on alignment with additional signals input by the user.

[0086] Ranking can also be performed based on best practices. A machine learning ranking model can be trained to identify best practices for media assets. Heuristic-based best practices can also be checked. Best practice scores can be provided. Scores can be based on estimated performance lift (e.g., for a specific audience). For example, it can be determined that placing a product at the center of a media asset tends to see a significant increase in website traffic.

[0087] Based on the ranking, the machine learning media asset generation pipeline can select generated assets from multiple assets (e.g., new assets and / or modified assets) in the content database to present to the user. For example, the top ranked assets can be selected for presentation. The top K groups of assets can be selected. A sampling of assets can be selected from different ranking positions (e.g., to be more robust to ranking errors).

[0088] The obtained assets can be presented in different ways based on the ranking. For example, reaching a threshold ranking, some assets can be pre-populated or pre-selected so that the user only needs to confirm the pre-selection to proceed. Below the threshold ranking, the assets can be provided as suggestions for the user to manually select. Similarly, the obtained assets can be presented in different ways based on the asset type. For example, text assets can be pre-populated as described above. In some cases, image assets may not be pre-populated.

[0089] The activity generation system may solicit user feedback about the obtained media assets. The activity generation system may provide a user interface to present the obtained media assets, the user interface having interactive input elements provided for editing the obtained media assets. The activity generation system may provide a user interface to present input fields for providing natural language instructions about changes to be made to the obtained media assets (e.g., "make the flowers look more colorful") or further instructions for generating new media assets based on the presented candidates (e.g., "generate more assets like this asset"). For example, in order to generate more assets similar to the presented assets, the activity generation system may input existing assets into the corresponding generative model as part of the prompts for generating similar assets. When generating more assets similar to previously generated assets, the prompts used to generate previously generated assets can be reused. A revision option includes multiple options of inputting existing assets (generated or otherwise) plus prompts into the model and then outputting revised assets based on the prompts.

[0090] User feedback can be fed back into the machine learning asset generation pipeline to regenerate or re-modify the media asset based on the feedback signal. This can be performed iteratively until the user approves the media asset.

[0091] User feedback may be obtained using a conversational input interface. For example, a voice or text natural language input and output interface may be provided to receive user input in natural language and implement the requested changes. The system may also generate output in natural language to describe the updates that have been performed.

[0092] The activity generation system can output the media assets to the content item generation system. The content item generation system can generate content items using the media assets. For example, the content item generation system can combine text assets (e.g., titles, slogans, descriptions) with image assets (e.g., product images, background images, etc.) to create content items for distribution. The content item generation system can generate content items based on the potential for utilization of the content items. For example, utilization of the content items can include interacting with the content items to execute hyperlinks embedded in the content items. For example, the hyperlinks can use resource locators to direct endpoint devices to data resources.

[0093] User feedback and selections can provide training data for improving one or more components of the machine-learned asset generation pipeline. For example, losses, rewards, or penalties can be based on user feedback and selections. The activity generation system can train one or more components of the machine-learned asset generation pipeline to reduce losses, increase rewards, or reduce penalties. Training techniques can involve supervised training (e.g., supervision provided by user input), unsupervised training (e.g., learning patterns of account behavior to optimize outputs based on those patterns), reinforcement learning (e.g., the asset generation pipeline as an intelligent agent seeking rewards), etc.

[0094] Other model alignment techniques may be used, such as soft hints. For example, one or more soft hints may be learned for input to any one or more generative models. Soft hints may be associated with specific user accounts or activities. In this way, for example, asset generation pipelines may be customized to improve performance for individual user accounts, optionally without having to retrain the entire pipeline to do so.

[0095] Ranking can be used earlier in a machine learning asset generation pipeline to classify the use of available processing bandwidth. For example, before generating an asset, instructions for generating an asset can be ranked (e.g., by processing the asset generation instructions and any other inputs using a machine learning ranking model), and the media asset generation pipeline can generate a top-ranked or top-K set of instructions. This pre-generation ranker can be trained based on the final output of the machine learning media asset generation pipeline. In this way, for example, by ranking instructions before generation, fewer low-ranked media assets will be generated first. In this way, processing for generating media assets can also be assigned to higher priority (e.g., higher ranked) generation tasks.

[0096] Generated assets can be processed through policy checks. For example, a policy check system can evaluate the generated output for any sensitive material (e.g., material that violates platform policies). Generated assets that violate policies can be screened out and not presented to users.

[0097] The policy checking system can be applied to the inputs to the activity generation system (e.g., inputs provided by users, data parsed from data resources, etc.). The policy checking system can screen for personally identifiable information (PII), obscene content, sensitive topics, or other policy-based screening rules. The policy checking system can screen any input provided by users or parsed from data resources and cross it out from further processing in any other model component.

[0098] According to some embodiments, the system can perform a product studio function. The product studio function can be a product that can generate product packaging photos for customers. The product studio function can upgrade images, remove backgrounds, and place products in scenes (e.g., converting packaging images into more lifelike images). For example, the product studio function can place clothing products on different models. In addition, the product studio function can perform automatic product variations. Automatic product variations can: use aggregated data to determine which types of scenes are attractive to the customer's target audience, or, alternatively, allow the customer to specify the scene; automatically identify product photos from sources (e.g., URLs); remove backgrounds (if any); generate multiple variations for product images (placing the product in a scene or background; alternatively, if it is clothing, it is placed on a model); and / or the scenes can all be tested in terms of traffic to see which performs best. For example, for a lotion advertising campaign, if the system determines that placing the product next to a palm tree, in a forest, etc. will bring more user clicks or conversions, the system can place the product in this way. The product studio function can also upgrade images.

[0099] The Product Studio feature can generate scenes by accepting image and text inputs using ML models. In addition, the system can generate some text inputs using formulas to get the best results:<product type> <platform>Surrounded by <surroundings>In front is <background>” Or, for example, “A skin care product jar is placed on a light-colored stone table, surrounded by almonds and with green plants in front.” The system can automatically generate this text. Additionally, the original product image can be layered back onto the ML output to ensure the integrity of the product. In some cases, the ML model can sometimes do more than just modify the background. The system can apply edge smoothing to ensure the product blends with the scene. The system can move the product below the center so that the ML model can better place the product on the surface.

[0100] Examples of the present disclosure provide a variety of technical effects, benefits and / or improvements in computing and artificial intelligence technologies, which involve the use of machine learning algorithms to generate new data, such as images, audio, text, video or other types of media. The technology described herein improves the use of generative models by improving the quality of the generated content. The quality of the generated content is tailored to the entity using data extracted from the web resources of the entity (e.g., company, user). For example, by using more content-related data, the system improves the performance of the generative model. In addition, the system utilizes better training techniques by developing more efficient and more effective training techniques for the entity (e.g., based on data extracted from the web resources of the entity) to reduce the time and resources required for training the model. In addition, the system can combine user feedback and provide feedback to the generative model via reinforcement learning or active learning, which can help the model learn from user preferences and improve over time. In addition, the present disclosure can reduce processing by reducing the number of manual inputs provided by the user and by reducing the number of interface screens that must be obtained, loaded, interacted with and updated. For example, a user may only need to enter the URL of a website, and the system can automatically extract content from the website and automatically generate content items for the user.

[0101] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0102] Figure 1 An example system for implementing a machine-learned media asset generation pipeline 100 is depicted. The machine-learned media asset generation pipeline 100 may include a machine-learned text generator 101. The machine-learned media asset generation pipeline 100 may include a machine-learned image generator 102. The machine-learned media asset generation pipeline 100 may include a machine-learned audio generator 103. The machine-learned media asset generation pipeline 100 may include a machine-learned video generator 104. The machine-learned media asset generation pipeline 100 may include one or more optimizers 105 to apply one or more optimization algorithms to the output of any one or more of the machine-learned generator models 101-104. The machine-learned media asset generation pipeline 100 may include one or more rankers 106 to rank the output of any one or more of the machine-learned generator models 101-104.

[0103] The machine-learned media asset generation pipeline 100 may ingest data from data resources 110 and data from account profiles 120. Account profiles 120 may include media asset preferences. Account profiles 120 may include media libraries 122. Account profiles 120 may include social media accounts 124. Account profiles 120 may include past signals / controls 126 input to the machine-learned media asset generation pipeline 100. The machine-learned media asset generation pipeline 100 may process data retrieved from data resources 110 and account profiles 120 according to new signals / controls 130. New signals / controls 130 may include user inputs that customize media asset generation.

[0104] The machine learning media asset generation pipeline 100 may include an asset feedback layer 140. The asset feedback layer 140 may facilitate input of user feedback on the generated assets and initiate the generation of updated or different assets. Fig.12 , Fig.13 and Fig.14 ), the machine learning media asset generation pipeline 100 can output media assets 150. Media assets 150 can include any type of media asset output. The media asset output can include, for example, text assets, image assets, audio assets, video assets, and / or unique profile data (e.g., brand profile data, color palette, logo).

[0105] Figure 2 A flow chart depicting an example machine learning media asset generation pipeline 200 according to an example embodiment of the present disclosure. In some cases, the system may receive a website and / or asset library at 202. At 204, the system may determine product and brand understanding based on the information received and / or obtained at 202. At 206, the system may identify existing assets based on the information received and / or obtained at 202. At 208, the system may customize the product and / or brand based on the determination at 204. At 210, the system may modify (e.g., update) the existing assets identified at 206. At 212, the system may determine a logo and color based on the information obtained at 208 and / or 210. At 214, the system may determine insights about the company and / or product based on the information obtained at 208 and / or 210. At 214, the system may also perform a gap analysis to predict missing information or automatically generate missing information based on the information obtained at 208 and / or 210.

[0106] Additionally, at 216, the system may generate a new asset based on the information obtained at 214. At 218, the system may modify the new asset generated at 216 by adding (e.g., modifying) text, images, videos, and / or site links. The text, images, videos, and / or site links selected at 218 may be determined or generated based on the information obtained at 212 and 214. At 220, the system may receive user input to customize the new asset generated at 216 and modified at 218. At 222, the system may provide (e.g., present) the customized asset 220 using an AI-driven format.

[0107] The machine-learned media asset generation pipeline 200 may include an overall model. The overall model may be a machine-learned generation model configured to generate multiple content items. Additionally or alternatively, the overall model may be a machine-learned selection model configured to select a selected content item from multiple content items. In some implementations, the overall model is trained to receive a set of input data 204 describing a web resource, and, due to receiving the input data 204, provides output data 206 that automatically generates new media assets and content items. For example, the system may receive user input associated with a web resource from a user device of a user. The system may extract multiple assets (e.g., images, words, videos, or audio files) from the web resource. In addition, the system may process multiple assets to generate multiple content items using an overall model (e.g., a machine-learned generation model). In addition, the system may determine a selected content item from multiple content items using an overall model (e.g., a machine-learned selection model). Subsequently, the system may cause the selected content item to be presented on a graphical user interface displayed on a user device.

[0108] In another embodiment, the system may receive data indicating a request for a plurality of media assets, the plurality of media assets comprising a plurality of media modalities. Additionally, the system may obtain a media asset profile for a client account associated with the request. The media asset profile may include data indicating the media asset preferences of the client account, and the media asset profile may be generated by processing pre-existing media assets associated with the client account. The system may use a machine learning media asset generation pipeline 200 to generate the plurality of media assets based on the media asset profile by instructing an overall model (e.g., a machine learning asset generation model) to generate media assets aligned with the media asset preferences. Subsequently, the system may, based on receiving data indicating that one or more media assets in the plurality of media assets have been selected, send the one or more media assets in the plurality of media assets to a content item generation system for generating a content item using the one or more media assets in the plurality of media assets.

[0109] According to some embodiments, the system can work with a client to automatically curate and create a variety of high-quality, engaging media assets for the client's business. Any business, regardless of size, can start advertising with the system in seconds, even without any assets. The system can lower the barrier for all businesses to reach their customers in a personalized and engaging way, and democratize ad creative development for everyone.

[0110] The system can combine the best machine learning models (including generative AI) and deep insights to help automatically populate entire asset groups for most new campaigns in real time. With a single click, customers can immediately begin using asset group sets to deliver results tailored to customer-specific goals and then be able to modify content items and / or media assets based on the recommendations received from the system.

[0111] For example, a customer can enter as much or as little information as necessary to generate content items, and as the customer generates these content items, the customer can, in some implementations, see the system's assumptions, have the opportunity to refine, and accept the media assets (e.g., content items) that the customer wants. Customers can publish the recommended media assets directly, or simply use them as a starting point for customizing or building their own media assets.

[0112] The system may include a user interface framework for collecting input for intelligent asset creation, collection, and assembly. The system may provide feedback to clients (e.g., customers) on these assets and the system's assumptions. The system may enable refinement of media assets based on user input, all during the media asset building process or guided flow.

[0113] Figure 3 A block diagram 300 depicts an example system according to an example embodiment of the present disclosure. The system may receive a URL 302 from a user. For example, the system may receive user input associated with the URL from a user device of the user. The system may extract multiple assets 304 from the data resource 110 associated with the URL 302. The multiple assets 304 may include brand understanding, product and service large language models (LLMs), images, site maps, logo understanding, social accounts, business LLMs, asset libraries, performance data, past activity data. In addition, the system, i.e., the machine learning media asset generation pipeline 100, may process the multiple assets 304 to generate multiple content items 308. The overall model 306 may perform ranking and insight determination, text and / or image generation artificial intelligence, asset automatic generation, inventory locking, product generation, and video creation. The multiple content 308 may include images, titles, descriptions, videos, logos, colors, site links, personalities, and visual styles. The system may determine selected media assets from multiple media assets using the machine learning content item generation pipeline 310 to generate content items 312. The system may then cause the new content item to be presented on a graphical user interface displayed on the user device.

[0114] Figure 4 A flowchart depicting an example method 400 for generating media assets by deconstructing existing content items according to some embodiments of the present disclosure. Method 400 may be performed by processing logic, which may include hardware (e.g., a processing device, a circuit system, dedicated logic, programmable logic, microcode, hardware of a device, an integrated circuit, etc.), software (e.g., instructions running or executed on a processing device), or a combination thereof. In some embodiments, method 400 is performed by a server computing system (e.g., server computing system 60) or a client computing system (e.g., computing device 50). Although shown in a particular order or sequence, the order of the processes may be modified unless otherwise specified. Therefore, the embodiments shown should be understood only as examples, and the processes shown may be performed in a different order, and some processes may be performed in parallel. In addition, one or more processors may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are also possible.

[0115] At operation 402, processing logic may receive data indicating a request for a plurality of media assets, the plurality of media assets including a plurality of media modalities to be generated based on pre-existing media assets. For example, in some implementations, processing logic may receive data indicating pre-existing media assets being uploaded. For example, a user may provide pre-existing media assets to the system via an online portal or other interface. In some implementations, the pre-existing media assets may be obtained by parsing a web resource associated with unique profile data to obtain the pre-existing media assets. For example, a user may have an account associated with unique profile data. A URL may be provided to obtain a web resource, which may be parsed to obtain the pre-existing media assets.

[0116] Pre-existing assets may include flattened image data. For example, a pre-existing asset may be a jpg or other image file containing a single layer of pixels. This can be distinguished from a multi-layered image file, which may include multiple layers that may have overlapping or different pixel values ​​based on the layer. Layered image files can be more easily broken down into components, whereas in a flattened image, multiple layers (such as a background image and overlaid text) are combined or merged into a single layer. As a result, pixels from the top layers may overwrite pixels from lower layers, making portions of the original background image irrecoverable.

[0117] In some implementations, the request is associated with a customer account, and wherein the customer account is associated with an account profile storing input to the machine-learned media asset generation pipeline. The profile may be retrieved from a database, and the profile may be previously generated prior to the request. For example, a user identifier may be associated with the profile or unique profile data.

[0118] At operation 404, processing logic may generate a plurality of media components by performing operations 406, 408, 410, and 412. For example, go to Figure 6 , the processing logic can obtain pre-existing media assets 605. The operations performed can generate multiple media assets. The media assets can include text assets, image assets 615, and unique profile assets, such as logo assets 620, color assets 625, font assets 630, and image styles 635.

[0119] The media assets may be obtained through a machine learning content item generation pipeline 640 to generate assets for various media channels, as will be discussed further with respect to operation 414 .

[0120] Generating the plurality of media asset components may include extracting the logo image. Generating the plurality of media asset components may include selecting a dominant color from an image color palette. Generating the plurality of media asset components may include upgrading the logo image by enlarging the logo image and / or sharpening the logo image.

[0121] Generating the plurality of media asset components may include generatively extending the logo image to generate one or more images. Each image may include a set of different aspect ratios. Generatively extending the logo image may include generating pixels to blend with existing pixels of the logo image to generate an image that is larger than the original logo image.

[0122] For example, go to Figure 8 , depicting a graphical depiction 800 of extracting and generating unique profile (e.g., brand profile) data such as dominant colors and logos. For example, pre-existing assets 805 can be used as input into a machine learning asset generation pipeline. The pipeline can extract or otherwise determine dominant colors 810. Additionally or alternatively, the pipeline can extract a logo 815 from an image. The extracted logo 815 may be low resolution or unsuitable for generating future assets. Therefore, the pipeline can upscale 820 the logo to create a larger version while maintaining a clear visual resolution. Additionally or alternatively, the pipeline can uncrop 825 the logo or otherwise generatively expand the logo to create a larger visual buffer around the logo.

[0123] At operation 406 , processing logic may extract one or more signals from the pre-existing media asset, the one or more signals comprising at least one of background image data, color palette data, text data, or logo image data.

[0124] At operation 408, processing logic may generate a plurality of background image assets using the first generative machine learning model. In some cases, generating the plurality of background images may include generating a mask image including one or more bounding boxes based on at least one of the background image data, the text data, or the logo image data. The one or more bounding boxes may include a location and a size of the bounding box.

[0125] In some cases, generating multiple background images may include inpainting the mask image by generating pixels using a machine-learned model to fill one or more bounding boxes of the mask image. In some cases, generating multiple background images may include generating multiple images, where each image includes a different aspect ratio.

[0126] Generating the plurality of background image assets may include adjusting at least one of brightness, saturation, or contrast of the respective background image assets.

[0127] Generating a mask image including one or more bounding boxes based on at least one of the background image data, the text data, or the logo image data may include identifying one or more text objects. For each of the one or more text objects, generating the mask image may include determining that a first text object of the one or more text objects is an overlay text object. For each of the one or more text objects, generating the mask image may include generating a first bounding box for the first text object based on determining that the first object is an overlay text object.

[0128] Figure 7 An example graphical depiction 700 depicting the generation of multiple background image media assets based on pre-existing media assets 705 (e.g., existing content items) is shown. For example, pre-existing media assets 705 may be input into a model that may generate multiple bounding boxes (e.g., bounding box 715 and bounding box 720) to create mask 710. The system may determine which bounding boxes should be filled by determining which bounding boxes are associated with overlay text (such as bounding box 715) and which bounding boxes are associated with underlying image text (such as bounding box 720).

[0129] For example, a model may be trained to distinguish between bounding boxes that should be inpainted and bounding boxes that should be ignored and not included in the mask. Output image 725 may be an image generated by inpainting bounding box 715.

[0130] In some implementations, the background image generation model may enhance the image by adjusting the saturation, sharpness, brightness, or other characteristics of the image to generate the enhanced image 730 .

[0131] In some implementations, the background image generation model can screen-expand or otherwise generatively expand the image to provide background images with different aspect ratios. For example, background image 735 may be more suitable for a full-screen mobile application advertisement, background image 740 may be more suitable for a search or email advertisement, and background image 745 may be more suitable for a desktop-rendered advertisement.

[0132] At operation 410, processing logic may generate a plurality of text assets using a second generative machine-learned model. Generating a plurality of text assets may include obtaining text data associated with one or more bounding boxes. Generating a plurality of text assets may include inputting the text data into the machine-learned model. Generating a plurality of text assets may include generating a plurality of text assets as output by the machine-learned model.

[0133] Generating a plurality of text assets may include extracting a signal including text data. Generating a plurality of text assets may include compiling the text data. Generating a plurality of text assets may include inputting the compiled text data into a second generative machine learning model. Generating a plurality of text assets may include obtaining at least one of a short title or a long title as an output from a second generative machine learning model. Generating a plurality of text assets may include inputting at least one of a short title or a long title into a second generative machine learning model. Generating a plurality of text assets may include obtaining a description as an output from a machine learning model.

[0134] Go to Fig. 9 , an example graphical depiction 900 of extracting and generating text data including a title and a description from a pre-existing media content item 905 can be seen. Processing logic can extract text from the pre-existing media content item 905 to generate a text data signal 910. The text data signal 910 can be collated to generate collated text 915. The collated text 915 can be used as an input in a second generative machine learning model to generate additional titles 920 and / or descriptions 925. The titles 920 and / or descriptions 925 can be used by the content item generation pipeline to generate one or more media content items for various media channels.

[0135] At operation 412, processing logic may generate unique profile data based on at least one of the color palette data or the logo image data using a third generative machine learning model. The unique profile data may include at least one of the logo data, the color palette data, the font data, or the image style data. The unique profile data may include brand data associated with a profile or other identifier. For example, the unique profile data may be associated with a "look and feel" of a typical content item associated with a profile (e.g., a business).

[0136] At operation 414, processing logic may automatically transmit a plurality of media assets including one or more image assets, text assets, and unique profile data to a content item generation pipeline for use in generating a plurality of candidate content items. The content item generation pipeline includes a prompt generation component and a content item generation component.

[0137] In some cases, processing logic may generate input prompt data based on the background image, text assets, and unique profile data by a prompt generation component. Processing logic may provide the input prompt data to a content item generation component. Processing logic may generate one or more candidate content items based on the input prompt data by the content item generation component. The content item generation component includes a model of generative machine learning.

[0138] In some implementations, the processing logic may train a generative machine learning model. For example, training the generative machine learning model may include generating a training data set based on comparing the generated content item components to existing media assets. Training the generative machine learning model may include automatically adjusting one or more parameters of the generative machine learning model based on comparing the generated content item components to existing media assets to reduce differences between the generated content item and the existing media assets.

[0139] In some implementations, processing logic can input the plurality of output assets into a content creation pipeline. Processing logic can generate a plurality of content items from the content creation pipeline, wherein each content item in the plurality of content items includes a unique combination of a content asset and an aspect ratio.

[0140] Return to Figure 6 , a machine-learned content item generation pipeline 640 can generate content items to be displayed via various media channels that require different formats. For example, media channels may include a video channel 645, an email channel 650, a display channel 655, a search channel 660, a discovery channel 665, and / or a map channel 670. Different channels may require different formats, for example, some channels may include images, text, and audio, while other channels may only include text. An example discovery channel asset 675 is depicted, which shows a combination of image, text, logo, and embedded (e.g., an optional component that links to a URL) assets.

[0141] Figure 5 A flow chart depicting an example data flow 500 for generating media assets according to an embodiment of the present disclosure. For example, a media asset generation pipeline 510 can obtain an existing media asset 505 (e.g., a pre-existing content item) as input. The media asset generation pipeline 510 can include an image processing component 515, a mask generation component 520, and / or a model of generative machine learning 525. The image processing component 515 can process the existing media asset 505 to extract background image data, text data, palette data, or other relevant data. The mask generation component 520 can determine which portions of the background image should be generatively filled (e.g., to remove superimposed text). The model of generative machine learning 525 can generatively fill based on the generated mask, as well as generate text or other assets based on any extracted signals associated with the existing media asset 505.

[0142] The media asset generation pipeline 510 can output media assets 530. The media assets 530 can include text assets 535, image assets 540, and / or unique profile data 545. As described herein, the text assets 535 can include words, sentences, paragraphs, or other phrases. The image assets 540 can include background images, logo images, or other images. The unique profile data 545 can include brand information or other information associated with the user profile.

[0143] The media assets 530 may be ingested by a content item generation pipeline 550 to generate candidate content items 560. The content item generation pipeline 550 may include a content item generation model 555, which may utilize the media assets 530 and / or other signals as input to generate candidate content items 560. The generated candidate content items 560 may be content items generated in a new manner such that they appear visually different from the original pre-existing media assets.

[0144] The systems described herein can perform a variety of methods. For example, the example methods can be performed by one or more computing systems (e.g., Figures 1 to 25 The various steps of the method may be omitted, rearranged, combined and / or adjusted in various ways without departing from the scope of the present disclosure.

[0145] The computing system may receive user input from a user device of a user associated with a web resource that is associated with an account of the user.

[0146] The computing system may extract a plurality of assets from the web resource, wherein each asset of the plurality of assets is an image, a word, a video, or an audio file.

[0147] The computing system may process the plurality of assets using a generative model of machine learning to generate a plurality of content items.

[0148] The computing system may determine the selected content item from the plurality of content items using a machine-learned selection model.

[0149] The computing system may cause the selected content item to be presented on a graphical user interface displayed on the user device.

[0150] In some cases, the operation may also include receiving a user interaction on a graphical user interface that modifies the selected content item. Additionally, the operation may include processing the user interaction and the selected content item using a machine-learned generative model to generate a modified content item. Additionally, the operation may include causing the modified content item to be presented on a graphical user interface displayed on a user device. Additionally, one or more parameters of the machine-learned generative model may be updated based on the user interaction.

[0151] In some cases, the operation may also include receiving a user interaction on a graphical user interface. The user interaction may be associated with rejecting the selected content item. In addition, the operation may include processing multiple content items and user interactions using a machine learning selection model to generate a new content item. In addition, the operation may include causing the new content item to be presented on a graphical user interface displayed on a user device.

[0152] In some cases, the operations may also include receiving a user interaction on a graphical user interface that accepts the selected content item. Additionally, the operations may include determining an advertising campaign based on the selected content item using a machine learning model. Furthermore, the operations may include causing the advertising campaign to be presented on a graphical user interface displayed on a user device.

[0153] In some cases, the web resource may be a website, and the user input is a uniform resource locator (URL) of the website.

[0154] In some cases, the plurality of content items may include a first content item, and the first content item may be generated by modifying an image asset in the plurality of assets. In addition, the plurality of content items may include a second content item, and the second content item is a generative image generated by a machine learning generative model using the image asset.

[0155] In some cases, the operation may also include calculating a conversion score for each of the plurality of content items using a machine learning selection model, the conversion score indicating the likelihood of a user interacting with the corresponding content item. For example, the selected content item may be the content item with the highest conversion score among the plurality of content items.

[0156] In some implementations, a computing system may receive data indicative of a request for a plurality of media assets including a plurality of media modalities.

[0157] The computing system may obtain a media asset profile for a client account associated with the request, wherein the media asset profile includes data indicative of media asset preferences of the client account, and wherein the media asset profile is generated by processing pre-existing media assets associated with the client account.

[0158] The computing system may use a machine learning media asset generation pipeline to generate the plurality of media assets by instructing a machine learning asset generation model to generate media assets aligned with the media asset preferences based on the media asset profile.

[0159] The computing system may, based on receiving data indicating selection of one or more media assets in the plurality of media assets, send the one or more media assets in the plurality of media assets to a content item generation system for generating a content item using the one or more media assets in the plurality of media assets.

[0160] In some cases, the plurality of media modalities includes two or more modalities selected from text, images, or audio.

[0161] In some cases, the operations may also include generating data for the media asset profile by parsing web resources associated with the client account.

[0162] In some cases, the operations may also include parsing the web resource to extract pre-existing media assets from the web resource.

[0163] In some cases, the operation may also include parsing the web resource to extract visual style data associated with the customer account. For example, the visual style may include color information, layout information, or typography information.

[0164] In some cases, the operation may also include parsing the web resource to extract text style data associated with the customer account. The text style data may include the tone or tone changes of the copy on the web resource.

[0165] In some cases, the operation may also include parsing the web resource to extract landing page data associated with the customer account. The landing page data may include a URL to a web page associated with the plurality of media assets.

[0166] In some cases, the media asset profile is retrieved from a database and the media asset profile was previously generated prior to the request.

[0167] In some cases, the operations may also include generating at least one of the plurality of media assets by editing a pre-existing image asset using at least one of the following editing operations: cropping, rotating, padding, recoloring, defocusing, deblurring, denoising, adjusting lighting. The editing operations may optionally be implemented using a machine-learned image editing tool. Additionally, the pre-existing image asset may be edited based on historical effect data associated with the image asset. Furthermore, the pre-existing image asset may be edited based on a set of content item guidelines for generating a content item using the pre-existing image asset.

[0168] In some cases, operations may also include inputting data from the media asset profile and a request for a generated asset consistent with the data from the media asset profile into a machine-learned media asset generation model.

[0169] In some cases, the operations may also include determining one or more generated assets using a machine-learned effect estimation model, wherein the machine-learned effect estimation model is configured to identify asset characteristics associated with historical effect data. Additionally, the operations may also include generating an enhanced input using the machine-learned effect estimation model for input to the machine-learned media asset generation model to result in asset characteristics associated with the historical effect data. Furthermore, the operations may include ranking the generated assets from the machine-learned media asset generation model using the machine-learned effect estimation model.

[0170] In some cases, the operation may also include presenting one or more generated media assets for review on a user interface accessible to a customer account. In addition, the operation may include receiving input providing corrections to one or more generated media assets via the user interface. In addition, the operation may include regenerating one or more generated media assets based on the received input using a media asset generation pipeline using machine learning. In addition, the user interface may include one or more selectable input elements that are associated with one or more generated media assets and indicate corresponding corrective actions to be performed for one or more generated media assets. The selectable input element may be configured to provide the received input after selection. The user interface may include a natural language input element for receiving correction input in a natural language format, wherein the natural language input element is configured to provide the received input.

[0171] In some cases, a media asset profile may be based on one or more of the following features that are associated with a customer account: a machine-learned model, an image, a site map, a logo, a social media account, an asset library, performance data, a collection of past media assets, a collection of past generated media assets.

[0172] In some cases, the plurality of media assets may include two or more of the following categories: image, title, description, video, logo, color, site link, call to action, audio.

[0173] Fig.10 An example graphical depiction depicting the extraction of existing media assets from sources 1005 associated with an account and utilizing media assets generated from the extracted assets and other sources as inputs to a machine-learned media asset generation pipeline 1030. For example, sources 1005 may include display network channels, uploads, social media, or referrals. Sources 1005 may be accessed to obtain image ads 1010. Image ads 1010 may be pre-existing content items, such as flattened image ads.

[0174] The machine-learned media asset generation pipeline 1030 can utilize various inputs to generate multiple media assets. The various inputs may include image ads 1010, URLs 1015, business profiles 1020, and / or mobile inputs 1025. These inputs may be used to generate background image assets, text assets, and / or unique profile data as described herein.

[0175] Fig.11 An example of extracting media assets from pre-existing content items is depicted. For example, the content items may include content item 1105A, content item 1105B, content item 1105C. The system may generate multiple candidate background images, such as background image 1110A, background image 1110B, and background image 1110C. The system may determine a dominant color and / or color palette, such as color 1115A, color 1115B, or color 1115C. The system may generate text assets, such as text asset 1120A, text asset 1120B, and text asset 1120C. Background images 1110A, 1110B, and / or 1110C may be generated using methods described herein, including mask generation, filling / painting, and generatively extending (e.g., uncropping).

[0176] Fig.12 An example user interface for interacting with the asset feedback layer 140 is shown. The asset feedback layer 140 can display the acquired media assets along with a source indicator (e.g., "from your URL," etc.). In some implementations, the asset feedback layer 140 can display a loading indicator (e.g., a solid bar instead of text that has not yet loaded) when receiving generated assets from the machine-learned media asset generation pipeline 100. The asset feedback layer 140 can pre-populate the fields of the asset with the generated asset.

[0177] The asset feedback layer 140 may display a loading indicator (e.g., a solid area in place of an image that has not yet been generated) while receiving generated assets from the machine-learned media asset generation pipeline 100. In some cases, a different loading status message is shown (e.g., "#Generating image using AI", "#Finding best matching stock images").

[0178] Fig.13 An example user interface is shown for interacting with asset feedback layer 140. Asset feedback layer 140 may provide menu options associated with an acquired asset for performing actions associated with the asset.

[0179] Fig.14 An example user interface for interacting with the asset feedback layer 140 is shown. The asset feedback layer 140 may provide a menu option for removing an asset. The asset feedback layer 140 may provide an interface for providing feedback associated with asset removal. The feedback may be used to train one or more components of the machine-learned media asset generation pipeline 100.

[0180] Additionally or alternatively, the asset feedback layer 140 may provide an interface for generating additional assets. The asset feedback layer 140 may provide suggested asset generation prompts that may be configured to be selectable to initiate processing of the suggested prompts. The asset feedback layer 140 may provide example generated assets. The asset feedback layer 140 may provide an interface for browsing output associated with a given prompt. The asset feedback layer 140 may provide an interface for inputting natural language prompts.

[0181] Additionally or alternatively, the asset feedback layer 140 may provide an interface for viewing other extracted and suggested media assets (such as colors, text assets, site links, etc.).

[0182] Fig.15 An example data flow for generating various content items for various distribution mechanisms (e.g., media channels) is shown. The content items may be configured such that data resources (e.g., such as Figure 1 1 ). For example, the machine learning content item generation pipeline 310 can generate content items for multiple media channels such as video, email, display, search, discovery, or maps. Fig.15 Two example formats are depicted in .

[0183] Fig.16 Flowchart depicting a method 1600 for training one or more machine-learned models according to aspects of the present disclosure. For example, example machine-learned models may include a machine-learned media asset generation pipeline, a machine-learned content item generation pipeline, a machine-learned text generator, a machine-learned image generator, a machine-learned audio generator, and a machine-learned video generator.

[0184] One or more portions of the example method 1600 may be implemented by a computing system including one or more computing devices (such as, for example, the computing systems described with reference to other figures). Each respective portion of the example method 1600 may be performed by any one (or any combination) of the one or more computing devices. In addition, one or more portions of the example method 1600 may be implemented on hardware components of the devices described herein, for example, to train one or more systems or models. Fig.16 Elements performed in a specific order are depicted for purposes of illustration and discussion. One of ordinary skill in the art will appreciate, using the disclosure provided herein, that the elements of any method discussed herein may be adjusted, rearranged, expanded, omitted, combined or modified in various ways without departing from the scope of the present disclosure. Fig.16 Elements / terms described with reference to other systems and figures are described for illustrative purposes and are not intended to be limiting. One or more portions of the example method 1600 may additionally or alternatively be performed by other systems.

[0185] At 1602, example method 1600 may include obtaining training instances. A set of training data may include multiple training instances divided between multiple data sets (e.g., a training data set, a validation data set, or a test data set). The training instances may be labeled or unlabeled. Although referred to as "training" instances in example method 1600, it should be understood that runtime reasoning may form training instances when an evaluation of the performance of the model on the runtime instance is used to train (e.g., train / learn online) the model. Example data types for training instances and various tasks associated therewith are described throughout this disclosure.

[0186] At 1604, the example method 1600 may include processing the training instance using one or more machine-learned models to generate an output. The output may be obtained directly from the one or more machine-learned models, or may be a downstream result of a chain of processing operations that includes the output of the one or more machine-learned models.

[0187] At 1606, example method 1600 may include receiving an evaluation signal associated with an output. The evaluation signal may be obtained using a loss function. Various losses may be determined, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, contrastive loss, or various other loss functions. The evaluation signal may be calculated using known baseline true labels (e.g., supervised learning), predicted or estimated labels (e.g., semi-supervised learning or self-supervised learning), or without labels (e.g., unsupervised learning). The evaluation signal may be a reward (e.g., for reinforcement learning). The reward may be calculated using a reward model of a machine learning configured to generate a reward based on the received output. The reward may be calculated using feedback data describing human feedback to the output.

[0188] At 1608, example method 1600 may include updating the machine-learned model using the evaluation signal. For example, in some embodiments, various training or learning techniques (such as, for example, back propagation) may be used to learn the values ​​of the parameters of the machine-learned model. For example, the evaluation signal may be back-propagated from the output (or another source of the evaluation signal) through the machine-learned model to update one or more parameters of the model (e.g., based on the gradient of the evaluation signal relative to the parameter value). For example, a system including one or more machine-learned models may be trained in an end-to-end manner. Gradient descent techniques may be used to iteratively update parameters in multiple training iterations. In some implementations, performing error back propagation may include performing truncated back propagation through time. Example method 1600 may include implementing a variety of generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0189] In some implementations, example method 1600 can be implemented to train a machine learning model from an initialized state to a fully trained state (e.g., when the model exhibits a desired performance profile, such as based on accuracy, precision, recall, etc.).

[0190] In some implementations, the example method 1600 may be implemented for a specific stage of the training process. For example, in some implementations, the example method 1600 may be implemented for pre-training a machine learning model. Pre-training may include, for example, large-scale training on potentially noisy data to achieve a wide range of performance level bases across a variety of tasks / data types. In some implementations, the example method 1600 may be implemented for fine-tuning a machine learning model. Fine-tuning may include, for example, smaller-scale training on higher quality (e.g., labeled, selected, etc.) data. Fine-tuning may affect all or part of the parameters of the machine learning model. For example, various parts of the machine learning model may be "frozen" for certain training stages. For example, parameters associated with the embedding space may be "frozen" during fine-tuning (e.g., to retain information learned from a wider domain than that present in the fine-tuning data set). Example fine-tuning methods include reinforcement learning. Reinforcement learning may be based on user feedback on model performance during use.

[0191] Fig.17 701 is a block diagram of an example processing flow for using a machine learning model 1701 701 to process an input 1702 to generate an output 1703.

[0192] The machine learning model 1701 may be or include one or more machine learning models or model components. For example, the machine learning model 1701 701 may include a machine learning media asset generation model 1701A and / or a machine learning content item generation model 1701B. The example machine learning model may include a neural network (e.g., a deep neural network). The example machine learning model may include a nonlinear model or a linear model. The example machine learning model may use other architectures to replace or supplement the neural network. The example machine learning model may include a decision tree-based model, a support vector machine, a hidden Markov model, a Bayesian network, a linear regression model, a k-means clustering model, and the like.

[0193] Example neural networks may include feedforward neural networks, recurrent neural networks (RNNs) (including recurrent neural networks based on long short-term memory (LSTM)), convolutional neural networks (CNNs), diffusion models, generative adversarial networks, or other forms of neural networks. Example neural networks may be deep neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models.

[0194] The machine-learned model 1701 may include a single or multiple instances of the same model configured to operate on data from the input 1702. The machine-learned model 1701 may include a collection of different models that can interact collaboratively to process data from the input 1702. For example, the machine-learned model 1701 701 can adopt an expert mixture structure. For example, see Zhou et al. Mixture-of-Experts with Expert Choice Routing, ARXIV:2202.09368v2 (October 14, 2022).

[0195] Input 1702 may generally include or otherwise represent various types of data. Input 1702 may include one type or many different types of data. For example, input may include existing media assets 1702A (e.g., existing content items) and / or data resources 1702B. Output 1703 may be the same type of data or a different type of data compared to input 1702. Output 1703 may include one type or many different types of data. For example, output 1703 may include media assets 1704 and / or content items 1705. Media assets 1704 may include, for example, text assets 1704A, image assets 1704B, and / or unique profile data 1704C.

[0196] Example data types for input 1702 or output 1703 include natural language text data, software code data (e.g., source code, object code, machine code, or any other form of computer-readable instructions or programming language), machine code data (e.g., binary code, assembly code, or other form of machine-readable instructions directly executable by a central processing unit of a computer), assembly code data (e.g., a low-level programming language that uses symbolic representations of machine code instructions to program a processing unit), genetic data or other chemical or biochemical data, image data, audio data, audio-visual data, tactile data, biometric data, medical data, financial data, statistical data, geographic data, astronomical data, historical data, sensor data in general (e.g., digital or analog values, such as voltage or other absolute or relative level measurements from real or artificial inputs (such as from audio sensors, light sensors, displacement sensors, etc.)), etc. The data may be raw or processed and may be in any format or mode.

[0197] In the multimodal input 1702 or output 1703, example combinations of data types include image data and audio data, image data and natural language data, natural language data and software code data, image data and biometric data, sensor data and medical data, etc. It should be understood that any combination of data types in the input 1702 or output 1703 may exist.

[0198] Example input 1702 may include one or more data types, such as the example data types noted above. Example output 1703 may include one or more data types, such as the example data types noted above. The data type of input 1702 may be the same or different from the data type of output 1703. It should be understood that the example data types noted above are provided for illustrative purposes only. The data types contemplated within the scope of the present disclosure are not limited to those examples noted above.

[0199] Fig.18 1701 is a block diagram of an example implementation of an example machine learning model configured to process a sequence of information. For example, an example implementation of a machine learning model 1701 may include a machine learning sequence processing model 4. An example system may pass an input 1702 to the sequence processing model 4. The sequence processing model 4 may include one or more machine learning components. The sequence processing model 4 may process data from the input 1702 to obtain an input sequence 5. The input sequence 5 may include one or more input elements 5-1, 5-2, ..., 5-M, etc. obtained from the input 1702. The sequence processing model 4 may process the input sequence 5 using a prediction layer 6 to generate an output sequence 7. The output sequence 7 may include one or more output elements 7-1, 7-2, ..., 7-N, etc. generated based on the input sequence 5. The system may generate an output 1703 based on the output sequence 7.

[0200] The sequence processing model 4 may include one or more machine learning model components configured to ingest, generate, or otherwise reason about sequences of information. For example, some example sequence processing models in the text domain are referred to as "large language models" or LLMs. See, for example, the PaLM 2 Technical Report, GOOGLE, https: / / ai.google / static / documents / palm2 techreport.pdf (nd). Other example sequence processing models can operate in other domains, such as, for example, image domains, e.g., see Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ARXIV:2010.11929v2 (June 3, 2021); audio domains, e.g., see Agostinelli et al., MusicLM: Generating Music From Text, ARXIV:2301.11325v1 (January 26, 2023); biochemical domains, e.g., see Jumper et al., Highly accurate protein structure prediction with AlphaFold, 596 Nature 583 (August 26, 2021). Sequence processing model 4 can process one or more types of data simultaneously. The sequence processing model 4 may include a relatively larger model (eg, more parameters, computationally intensive, etc.), a relatively smaller model (eg, fewer parameters, computationally lightweight, etc.), or both.

[0201] In general, the sequence processing model 4 can obtain the input sequence 5 using the data from the input 1702. For example, the input sequence 5 can include a representation of the data from the input 1702 in a format understood by the sequence processing model 4. One or more machine-learned components of the sequence processing model 4 can ingest the data from the input 1702, parse the data into segments that are compatible with the processing architecture of the sequence processing model 4 (e.g., via "lemmaization"), and project the segments into an input space associated with the prediction layer 6 (e.g., via "embedding").

[0202] The sequence processing model 4 can ingest data from the input 1702 and parse the data into a sequence of elements to obtain an input sequence 5. For example, a portion of the input data from the input 1702 can be decomposed into fragments that together represent the content of the portion of the input data. The fragments can provide elements of the sequence.

[0203] In some cases, elements 5-1, 5-2, ..., 5-M may represent building blocks for capturing or expressing meaningful information in a particular data domain. For example, an element may describe an "atomic unit" across one or more domains. For example, for a text input source, an element may correspond to a group of one or more word or sub-word components (such as a set of one or more characters).

[0204] For example, elements 5-1, 5-2, ..., 5-M may represent tokens obtained using a token analyzer. For example, a token analyzer may process a given portion of an input source and output a series of tokens representing that portion of the input source (e.g., corresponding to input elements 5-1, 5-2, ..., 5-M). Various methods may be used to tokenize. For example, a text input source may be tokenized using a byte pair encoding (BPE) technique. See, for example, Kudo et al., SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (System Demonstrations), pp. 66–71 (October 31–November 4, 2018), https: / / aclanthology.org / D18-2012.pdf. Image-based input sources can be tokenized by extracting and serializing patches from the image.

[0205] In general, any data type can be serialized and processed into an input sequence 5. It should be understood that Fig.18 The elements 5-1, 5-2, ..., 5-M depicted in FIG. 5 may be word-grams or may be embedded representations thereof.

[0206] The prediction layer 6 can predict one or more output elements 7-1, 7-2, ..., 7-N based on the input elements. The prediction layer 6 can include one or more machine learning model architectures, such as one or more learning parameter layers, which manipulate and transform the input to extract high-order meaning from the input elements 5-1, 5-2, ..., 5-M and the relationships between these input elements. In this way, for example, the example prediction layer 6 can predict new output elements based on the context provided by the input sequence 5.

[0207] The prediction layer 6 can evaluate associations between parts of the input sequence 5 and specific output elements. These associations can inform predictions about the likelihood that a specific output follows from the input context. For example, consider the text snippet "The carpenter's toolbox is small and heavy. It is filled with ___." The example prediction layer 6 can recognize that "it" mentions "toolbox" again by determining the relationship between the corresponding embeddings. The example prediction layer 6 can also relate "it" to properties of the toolbox, such as "small" and "heavy". Based on these associations, for example, the prediction layer 6 can assign a higher probability to the word "nails" than to the word "sawdust".

[0208] A transformer is an example architecture that can be used in the prediction layer 4. For example, see Vaswani et al., Attention Is All You Need, ARXIV:1706.03762v7 (August 2, 2023). A transformer is an example of a model architecture for machine learning that uses an attention mechanism to compute associations between items within a context window. The context window may include a sequence that contains an input sequence 5 and possibly one or more output elements 7-1, 7-2, ..., 7-N. A transformer block may include one or more attention layers and one or more post-attention layers (e.g., feed-forward layers such as multilayer perceptrons).

[0209] The prediction layer 6 may include other machine learning model architectures that complement or replace the transformer-based architecture. For example, recurrent neural networks (RNN) and long short-term memory (LSTM) models, as well as convolutional neural networks (CNNs), may also be used. In general, the prediction layer 6 may utilize various artificial neural networks that can understand or generate information sequences.

[0210] The output sequence 7 may include or otherwise represent the same or different data types as the input sequence 5. For example, the input sequence 5 may represent text data, and the output sequence 7 may represent text data. The input sequence 5 may represent image, audio, or audio-visual data, and the output sequence 7 may represent text data (e.g., describing the image, audio, or audio-visual data). It should be understood that the prediction layer 6 and any other gap model components of the sequence processing model 4 can be configured to receive multiple data types in the input sequence 5 and output multiple data types in the output sequence 7.

[0211] The output sequence 7 may have various relationships with the input sequence 5. The output sequence 7 may be a continuation of the input sequence 5. The output sequence 7 may be complementary to the input sequence 5. The output sequence 7 may translate, transform, enhance, or otherwise modify the input sequence 5. The output sequence 7 may answer, evaluate, confirm, or otherwise respond to the input sequence 5. The output sequence 7 may implement instructions provided via the input sequence 5 (or describe instructions for implementing the instructions).

[0212] The output sequence 7 can be generated autoregressively. For example, for some applications, the output of one or more prediction layers 6 can be passed through one or more output layers (e.g., a softmax layer) to obtain a probability distribution of an output vocabulary (e.g., a text or symbol vocabulary) conditioned on a set of input elements in a context window. In this way, for example, the output sequence 7 can be generated autoregressively by sampling a possible next output element, adding the element to the context window, and regenerating the probability distribution based on the updated context window, and sampling a possible next output element, etc.

[0213] The output sequence 7 can also be generated non-autoregressively. For example, multiple output elements of the output sequence 7 can be predicted together without explicit order conditions with respect to each other. See, for example, Saharia et al., Non-Autoregressive Machine Translation with Latent Alignments, ARXIV:2004.07437v3 (November 16, 2020).

[0214] The output sequence 7 may include one or more parts or elements. In an example content generation configuration, the output sequence 7 may include multiple elements corresponding to multiple parts of the generated output sequence (e.g., text sentences, values ​​of discrete waveforms, computer code, etc.). In an example classification configuration, the output sequence 7 may include a single element associated with a classification output. For example, the output "vocabulary" may include a set of classes into which the input sequence will be classified. For example, the visual transformer block may pass the potential state information to a multilayer perceptron that outputs possible class values ​​associated with the input image.

[0215] Fig.19 is a block diagram of an example technique for populating an example input sequence 8. The input sequence 8 may include various functional elements that form part of the model infrastructure, such as an element 8-0 obtained from a task indicator 9, which signals any model that processes the input sequence 8 that a specific task is being performed (e.g., to help adapt the performance of the model to the specific task). The input sequence 8 may include various data elements from different data modalities. For example, the input modality 10-1 may include a data modality. The data-to-sequence model 11-1 may process data from the input modality 10-1 to project the data into a format compatible with the input sequence 8 (e.g., one or more vectors of dimensions determined according to the dimensions of the input sequence 8) to obtain elements 8-1, 8-2, 8-3. Another input modality 10-2 may include different data modalities. The data-to-sequence model 11-2 may project the data from the input modality 10-2 into a format compatible with the input sequence 8 to obtain elements 8-4, 8-5, 8-6. Another input modality 10-3 may include yet another different data modality. The data-to-sequence model 11-3 may project the data from the input modality 10-3 into a format compatible with the input sequence 8 to obtain elements 8-7, 8-8, 8-9.

[0216] Input sequence 8 may be the same as or different from input sequence 5. Input sequence 8 may be a multimodal input sequence that includes elements that represent data from different modalities using a common dimensional representation. For example, the embedding space may have P dimensions. Input sequence 8 may be configured to include multiple elements having P dimensions. In this way, for example, example implementations may facilitate information extraction and reasoning across different data modalities by projecting data into elements in the same embedding space to perform comparisons, combinations, or other calculations between them.

[0217] For example, elements 8-0, ..., 8-9 may indicate specific locations within a multidimensional embedding space. Some elements may be mapped to a discrete set of locations in the embedding space. For example, elements corresponding to discrete members in a predetermined word-gram vocabulary may be mapped to discrete locations associated with those word-grams in the embedding space. Other elements may be distributed continuously across the embedding space. For example, some data types may be decomposed into continuously defined parts (e.g., image patches), which may be described using continuously distributed locations within the embedding space.

[0218] In some implementations, the expressive power of the embedding space may not be limited to the meaning associated with any particular set of word-grams or other building blocks. For example, a continuous embedding space may encode a series of high-order information. Individual fragments of information (e.g., word-grams) may be mapped to specific points in the space: for example, the word-gram of the word "dog" may be projected to an embedding value that points to a specific location in the embedding space associated with canine-related information. Similarly, an image patch of an image of a dog on grass may also be projected into the embedding space. In some implementations, the projection of the image of the dog may be similar to the projection of the word "dog", while also being similar to the projection of the word "grass", but potentially different from both. In some implementations, the projection of the image patch may not be completely aligned with any single projection of a single word. In some implementations, the projection of the image patch may be aligned with a combination of the projections of the words "dog" and "grass". In this way, for example, a high-order embedding space may encode information that may be independent of the data modality of the expressed information.

[0219] The task indicator 9 may include a model or model component configured to identify the task being performed and inject an input value represented by an element 8-0 into the input sequence 8, which element 8-0 signals which task is being performed. For example, the input value may be provided as a data type associated with the input modality and projected along with the input modality (e.g., the input value may be a text task label embedded along with other text data in the input; the input value may be a pixel-based representation of the task embedded along with other image data in the input; etc.). The input value may be provided as a data type that is different from or at least independent of other inputs. For example, the input value represented by the element 8-0 may be learned within a continuous embedding space.

[0220] The input modalities 10 - 1 , 10 - 2 , and 10 - 3 may be associated with a variety of different data types (eg, as described above with respect to input 1702 and output 1703 ).

[0221] The data-to-sequence models 11-1, 11-2, and 11-3 may be the same or different from each other. The data-to-sequence models 11-1, 11-2, and 11-3 may be adapted to each respective input modality 10-1, 10-2, and 10-3. For example, a text data-to-sequence model may subdivide a portion of the input text and project these subdivisions into elements in the input sequence 8 (e.g., elements 8-1, 8-2, 8-3, etc.). An image data-to-sequence model may subdivide an input image and project these subdivisions into elements in the input sequence 8 (e.g., elements 8-4, 8-5, 8-6, etc.). An arbitrary data type data-to-sequence model may subdivide the input of the arbitrary data type and project these subdivisions into elements in the input sequence 8 (e.g., elements 8-7, 8-8, 8-9, etc.).

[0222] The data-to-sequence models 11-1, 11-2, and 11-3 may form part of the machine-learned sequence processing model 4. The data-to-sequence models 11-1, 11-2, and 11-3 may be trained jointly with the machine-learned sequence processing model 4 or independently of the machine-learned sequence processing model 4. The data-to-sequence models 11-1, 11-2, and 11-3 may be trained end-to-end with the machine-learned sequence processing model 4.

[0223] Fig. 20 1 is a block diagram of an example model development platform 12 that can facilitate the creation, adaptation, and refinement of example machine-learned models (e.g., machine-learned model 1701, sequence processing model 4, etc.). The model development platform 12 can provide a number of different tool kits that a developer system can use to develop new or adapted machine-learned models.

[0224] The model development platform 12 may provide one or more model libraries 13 containing building blocks for new models. The model library 13 may include one or more pre-trained base models 13-1, which may provide a backbone of processing capabilities across various tasks. The model library 13 may include one or more pre-trained expert models 13-2, which may focus on performance in a specific professional field. The model library 13 may include various model primitives 13-3, which may provide low-level architectures or components (optionally pre-trained), which may be assembled in various arrangements as needed.

[0225] The model development platform 12 may receive a selection of various model components 14. The model development platform 12 may pass the selected model components 14 to a workbench 15, which combines the selected model components 14 into a development model 16.

[0226] The workbench 15 may facilitate further refinement and adaptation of the development model 16 by utilizing a number of different tool suites integrated with the model development platform 12. For example, the workbench 15 may facilitate alignment of the development model 16 with desired performance profiles on various tasks using a model alignment tool suite 17.

[0227] The model alignment toolkit 17 can provide a variety of tools for causing the development model 16 to generate outputs that are aligned with desired behavioral characteristics. Alignment can include increasing the accuracy, precision, recall, etc. of the model output. Alignment can include enforcing the use of output styles, patterns, or other preferred characteristics of the model output. Alignment can be general or domain-specific. For example, the pre-trained base model 13-1 can start from an initial performance level across multiple domains. Alignment of the pre-trained base model 13-1 can include improving performance in a specific information or task domain (e.g., even at the expense of performance in another information or task domain).

[0228] The model alignment toolkit 17 can integrate one or more datasets 17-1 for aligning the development model 16. The curated dataset 17-1 can include labeled or unlabeled training data. The dataset 17-1 can be obtained from a public domain dataset. The dataset 17-1 can be obtained from a private dataset associated with one or more developer systems for alignment of custom machine learning models customized for private use cases.

[0229] The pre-training pipeline 17-2 may include a model training workflow for machine learning configured to update the development model 16 on a large-scale, potentially noisy dataset. For example, pre-training may utilize unsupervised learning techniques (e.g., denoising, etc.) to process a large number of training instances to update model parameters from an initialized state and achieve a desired baseline performance. The pre-training pipeline 17-2 may utilize an unlabeled dataset in the dataset 17-1 to perform pre-training. The workbench 15 may implement the pre-training pipeline 17-2 to pre-train the development model 16.

[0230] The fine-tuning pipeline 17-3 may include a model training workflow of machine learning configured to use higher quality data to refine model parameters of the development model 16. The fine-tuning pipeline 17-3 may update the development model 16 by performing supervised training using a labeled dataset in the dataset 17-1. The fine-tuning pipeline 17-3 may update the development model 16 by performing reinforcement learning using a reward signal from a user feedback signal. The workbench 15 may implement the fine-tuning pipeline 17-3 to fine-tune the development model 16.

[0231] The prompt library 17-4 may include a set of inputs configured to induce behavior aligned with a desired performance standard. The prompt library 17-4 may include a few-trial prompt (e.g., an input that provides an example of the desired model output so that the head is appended to the desired runtime query), a thought chain prompt (e.g., an input that provides step-by-step reasoning within an example to encourage the model to perform comprehensive reasoning), etc.

[0232] Example prompts may be retrieved from the available repository of prompt library 17-4. One or more developer systems may use workbench 15 to facilitate example prompts.

[0233] In some implementations, a pre-trained or fine-tuned model can achieve satisfactory performance without examples in the input. For example, a zero-trial prompt can include an input that lacks examples. The zero-trial prompt can be within the domain within the training dataset or outside the training domain.

[0234] The prompt library 17-4 may include one or more prompt engineering tools. The prompt engineering tool may provide a workflow for retrieving or learning optimized prompt values. The prompt engineering tool may facilitate directly learning prompt values ​​(e.g., input element values) based on one or more training iterations. The workbench 15 may implement the prompt engineering tool in the development model 16.

[0235] The prompt library 17-4 may include a pipeline for prompt generation. For example, the development model 16 itself or another machine-learned model may be used to generate inputs. In this way, for example, a first model may process information about a task and output inputs for a second model to process in order to perform the steps of the task. The second model may be the same as or different from the first model. The workbench 15 may implement the prompt generation pipeline in the development model 16.

[0236] The prompt library 17-4 may include a pipeline for context injection. For example, the performance of the development model 16 on a particular task may be improved if provided with additional context for performing the task. The prompt library 17-4 may include software components configured to identify the desired context, retrieve the context from an external source (e.g., a database, a sensor, etc.), and add the context to the input prompt. The workbench 15 may implement the context injection pipeline in the development model 16.

[0237] Although various training examples described herein with respect to the model development platform 12 involve "pre-training" and "fine tuning", it should be understood that the model alignment toolkit 17 can generally support a variety of training techniques suitable for training a variety of machine learning models. The example training techniques can correspond to the example training method 1600 described above.

[0238] The model development platform 12 may include a model plug-in tool kit 18. The model plug-in tool kit 18 may include a variety of tools that are configured to enhance the functionality of the machine learning model by integrating the machine learning model with other systems, devices, and software components. For example, the machine learning model can use tools to improve the quality of performance where appropriate. For example, deterministic tasks can be offloaded to a dedicated tool instead of performing tasks in a probabilistic manner when the risk of error increases. For example, the machine learning model does not autoregressively predict the solution of the system of equations, but can identify the tools called to obtain the solution and pass the system of equations to the appropriate tool. The tool can be a traditional system of equations solver that can operate deterministically to solve the system of equations. The output of the tool can be returned in response to the original query. In this way, the use of tools can allow some example models to focus on the advantages of machine learning models—for example, understanding the intent in unstructured requests for tasks—while enhancing the performance of the model by offloading certain tasks to more focused tools to mechanically apply deterministic algorithms to well-defined problems.

[0239] The model plug-in tool suite 18 may include a validation tool 18-1. The validation tool 18-1 may include a tool that may parse and validate the output of a machine-learned model. The validation tool 18-1 may include engineered heuristics that establish certain thresholds to be applied to the model output. For example, the validation tool 18-1 may ground the output of a machine-learned model against a structured data source (e.g., to mitigate "hallucinations").

[0240] The model plug-in toolkit 18 may include a toolkit 18-2 for implementing one or more tools, which may include scripts or other executable code that can be executed with the development model 16. The toolkit 18-2 may include one or more inputs configured to enable the machine learning model to implement the tool (e.g., a few-trial prompts that induce the model to output tool calls with the correct syntax, etc.). For example, the toolkit 18-2 may include fine-tuning training data for training the model to use the tool.

[0241] The model plug-in tool suite 18 may include an interface for calling an external application programming interface (API) 18-3. For example, in addition to or instead of directly implementing tool calls or tool code using the development model 16, the development model 16 may also be aligned with output instructions that initiate API calls to send or obtain data via an external system.

[0242] The model plug-in toolkit 18 may be integrated with the hint library 17-4 to build a catalog of available tools for use with the development model 16. For example, the model may receive in input a catalog of available tools, and the model may generate an output that selects a tool from the available tools and initiates a tool call for using the tool.

[0243] The model development platform 12 may include a computing optimization tool suite 19 for optimizing the computing performance of the development model 16. For example, a model compression 19-1 tool may allow the development model 16 to reduce size while maintaining a desired performance level. For example, the model compression 19-1 may include quantization workflows, weight pruning, and sparsification techniques, etc. The hardware acceleration 19-2 tool may facilitate the configuration of model storage and execution formats to operate optimally on different hardware resources. For example, the hardware acceleration 19-2 may include tools for optimally slicing the model for distributed processing on multiple processing units to increase bandwidth, reduce unified memory requirements, etc. The distillation 19-3 tool may be used to train a more lightweight model based on the knowledge encoded in the development model 16. For example, the development model 16 may be a high-performance large-scale machine learning model optimized using the model development platform 12. In order to obtain a lightweight model for running in a resource-constrained environment, the smaller model may be a "student model" that learns to imitate the development model 16 as a "teacher model". In this way, for example, the investment in learning the parameters and configuration of the development model 16 can be efficiently transferred to a smaller model for more efficient inference.

[0244] The workbench 15 may implement one, more than one, or no tool suite implemented in the model development platform 12. The workbench 15 may output an output model 20 based on the development model 16. The output model 20 may be a deployed version of the development model 16. The output model 20 may be a development or training checkpoint of the development model 16. The output model 20 may be a distilled, compressed, or otherwise optimized version of the development model 16.

[0245] Fig.21 1 is a block diagram of an example training process for training a development model 16 for machine learning. One or more portions of the example training process may be implemented by a computing system including one or more computing devices (such as, for example, the computing systems described with reference to other figures). Each respective portion of the example training process may be performed by any one (or any combination) of the one or more computing devices. In addition, one or more portions of the example training process may be implemented on hardware components of the devices described herein, for example, to train one or more systems or models. Fig.21 Elements performed in a specific order are depicted for purposes of illustration and discussion. One of ordinary skill in the art will appreciate, using the disclosure provided herein, that the elements of any method discussed herein may be adjusted, rearranged, expanded, omitted, combined or modified in various ways without departing from the scope of the present disclosure. Fig.21 Elements / terms described with reference to other systems and figures are described for illustrative purposes and are not intended to be limiting. Additionally or alternatively, one or more portions of the example training process may be performed by other systems.

[0246] Initially, the development model 16 may be maintained in an initial state as an initialized model 21. The development model 16 may be initialized using weight values. The initial weight values ​​may be random or based on an initialization pattern. The initial weight values ​​may be based on previous pre-training for the same or a different model.

[0247] The initialized model 21 may be pretrained in a pretraining stage 22. The pretraining stage 22 may be implemented using one or more pretraining pipelines 17-2 on data from the dataset 17-1. For example, if the initialized model 21 has been pretrained (e.g., the development model 16 includes, is, or is based on a pretrained base model or expert model), pretraining may be omitted.

[0248] The pre-trained model 23 may then be a new version of the development model 16, which may be maintained as the development model 16 or a new development model. If the development model 16 has been pre-trained, the pre-trained model 23 may be the initial state. The pre-trained model 23 may be fine-tuned in a fine-tuning phase 24. The fine-tuning phase 24 may be implemented using one or more fine-tuning pipelines 17-3 on data from the dataset 17-1. For example, if the pre-trained model has satisfactory performance, if the model has already been fine-tuned, or if other methods of adjustment are preferred, fine-tuning may be omitted.

[0249] The fine-tuned model 29 may then be a new version of the development model 16, which may be maintained as the development model 16 or a new development model. If the development model 16 has already been fine-tuned, the fine-tuned model 29 may be the initial state. The fine-tuned model 29 may undergo refinement 26 with user feedback. For example, refinement 26 with user feedback may include performing reinforcement learning, optionally based on human feedback from human users of the fine-tuned model 25. Since reinforcement learning may be a form of fine-tuning, it should be understood that the fine-tuning stage 24 may include a stage for performing refinement 26 with user feedback. Refinement 26 with user feedback may produce a refined model 27. The refined model 27 may be output to a downstream system 28 for deployment or further development.

[0250] In some implementations, computational optimization operations may be applied before, during, or after each stage. For example, the initialized model 21 may be computationally optimized 29-1 (e.g., using the computational optimization toolkit 19) before the pre-training stage 22. The pre-trained model 23 may be computationally optimized 29-2 (e.g., using the computational optimization toolkit 19) before the fine-tuning stage 24. The fine-tuned model 25 may be computationally optimized 29-3 (e.g., using the computational optimization toolkit 19) before refinement 26 using user feedback. The refined model 27 may be computationally optimized 29-4 (e.g., using the computational optimization toolkit 19) before being output to a downstream system 28. The computational optimizations 29-1, ..., 29-4 may all be the same, all different, or include at least some different optimization techniques.

[0251] Fig. 22 1 is a block diagram of an inference system for operating one or more machine-learned models 1701 to perform inference (e.g., for training, for deployment, etc.). A model host 31 may receive a machine-learned model 1701. The model host 31 may host one or more model instances 31-1, which may be one or more instances of one or more models. The model host 31 may host the model instance 31-1 using available computing resources 31-2 associated with the model host 31.

[0252] The model host 31 can perform inference on behalf of one or more clients 32. The client 32 can transmit an input request 33 to the model host 31. Using the input request 33, the model host 31 can obtain an input 1702 for input to the machine learning model 1701. The machine learning model 1 can process the input 1702 to generate an output 1703. Using the output 1703, the model host 31 can return an output payload 34 for responding to the input request 33 from the client 32. The output payload 34 may include or be based on the output 1703.

[0253] The model host 31 can utilize various other resources and tools to enhance the reasoning task. For example, the model host 31 can communicate with the tool interface 35 to facilitate the model instance 31-1 to use the tool. The tool interface 35 may include a local or remote API. The tool interface 35 may include an integrated script or other software function. The model host 31 may use the online learning interface 36 to promote the continuous improvement of the machine learning model 1701. For example, the online learning interface 36 can be used in a reinforcement learning loop to retrieve user feedback on the reasoning provided by the model host 31. The model host 31 can access the runtime data source 37 for enhancing the input 1702 with additional context information. For example, the runtime data source 37 may include a knowledge graph 37-1 that facilitates structured information retrieval for information associated with the input request 33 (e.g., a search engine service). The runtime data source 37 may include a public or private, external or local database 37-2 that can store information associated with the input request 33 for enhancing the input 1702. The runtime data source 37 may include account data 37 - 3 , which may be retrieved in association with a user account corresponding to the client 32 , for use in customizing the behavior of the model host 31 accordingly.

[0254] The model host 31 may be implemented by one or more computing devices or systems. The client 2 may be implemented by one or more computing devices or systems, which may include a computing device or system shared with the model host 31 .

[0255] For example, the model host 31 may be operated on a server system that provides machine learning services (e.g., over a local area network or wide area network) to client devices operating the client 32. The client device may be an end-user device used by an individual. The client device may be a server system that operates the client 32 to provide various functions as services to downstream end-user devices.

[0256] In some implementations, the model host 31 may operate on the same device or system as the client 32. The model host 31 may be a machine learning service that runs on a device to provide machine learning functionality to one or more applications operating on a client device, which may include an application that implements the client 32. The model host 31 and the client 32 may be part of the same application. For example, the model host 31 may be a subroutine or method implemented by a portion of the application, and the client 32 may be another subroutine or method that uses the model host 31 to perform reasoning functionality within the application. It should be understood that the model host 31 and the client 32 may have a variety of different configurations.

[0257] Model instance 31-1 may include one or more machine learning models that can be used to perform reasoning. Model instance 31-1 may include weights or other model components stored on / in a persistent storage device, temporarily cached, or loaded into a high-speed memory. Model instance 31-1 may include multiple instances of the same model (e.g., for executing more requests in parallel on the same model). Model instance 31-1 may include instances of different models. Model instance 31-1 may include cached intermediate states of active or inactive models, and cached intermediate states are used to accelerate reasoning of those models. For example, a reasoning session with a particular model may generate a significant amount of computational results that can be reused for future reasoning runs (e.g., using a KV cache for a transformer-based model). These computational results may be stored in association with the reasoning session so that the session can be executed more efficiently when restored.

[0258] Computing resources 31-2 may include one or more processors (central processing units, graphics processing units, tensor processing units, machine learning accelerators, etc.) connected to one or more memory devices. Computing resources 31-2 may include a dynamic pool of available resources shared with other processes. Computing resources 31-2 may include a memory device large enough to fit the entire model instance in a single memory instance. Computing resources 31-2 may also span multiple memory device model instances (e.g., using data parallelism or tensor parallelism, etc.). Doing so can increase parallelism or execute large models using multiple memory devices that individually may not be able to fit the entire model in memory.

[0259] The input request 33 may include data for the input 1702. The model host 31 may process the input request 33 to obtain the input 1702. The input 2 may be obtained directly from the input request 33, or may be retrieved using the input request 33. The input request 33 may be submitted to the model host 31 via an API.

[0260] The model host 31 can perform reasoning on multiple batches of input requests 33 in parallel. For example, the model instance 31-1 can be configured with an input structure having a batch dimension. Individual inputs 1702 can be distributed across batch dimensions (e.g., rows of an array). Individual inputs 1702 can include completely different contexts. Individual inputs 1702 can be multiple reasoning steps for the same task. Individual inputs 1702 can be interleaved in the input structure so that any given reasoning cycle can operate on different parts of the corresponding input 1702. In this way, for example, the model host 31 can perform reasoning on batches in parallel, so that the output 1703 can also contain a batch dimension and return the reasoning results of the batched inputs 1702 in parallel. In this way, for example, multiple batches of input requests 33 can be processed in parallel to achieve a higher throughput of the output payload 34.

[0261] The output payload 34 may include or be based on the output 1703 from the machine-learned model 1701. The model host 31 may process the output 1703 to obtain the output payload 34. This may include chaining multiple rounds of reasoning (e.g., iteratively, recursively, across the same model or different models) to obtain the final output of the task to be returned in the output payload 34. The output payload 34 may be transmitted to the client 32 via an API.

[0262] The online learning interface 36 can facilitate reinforcement learning of the machine-learned model 1701. The online learning interface 36 can facilitate reinforcement learning with human feedback (RLHF). The online learning interface 36 can facilitate federated learning of the machine-learned model 1701.

[0263] The model host 31 can execute the machine learning model 1701 to perform reasoning for various tasks using various types of data. For example, various different inputs 1702 and outputs 1703 can be used for various different tasks. In some implementations, the input 1702 can be or otherwise represent image data. The machine learning model 1 can process the image data to generate an output. As an example, the machine learning model 1701 can process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model 1701 can process the image data to generate an image segmentation output. As another example, the machine learning model 1701 can process the image data to generate an image classification output. As another example, the machine learning model 1701 can process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model 1701 can process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of image data, etc.). As another example, the machine-learned model 1701 can process image data to generate an upgraded image data output. As another example, the machine-learned model 1701 can process image data to generate a predicted output.

[0264] In some implementations, the task is a computer vision task. In some cases, the input 1702 includes pixel data of one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that the one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines the corresponding possibility of each category in a set of predetermined categories for each pixel in the one or more images. For example, the group of categories may be foreground and background. As another example, the group of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in the one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at the pixel between images in the network input for each pixel of one of the input images.

[0265] In some implementations, input 1702 may be or otherwise represent natural language data. The machine-learned model 1 may process natural language data to generate an output. As an example, the machine-learned model 1701 may process natural language data to generate a language encoding output. As another example, the machine-learned model 1701 may process natural language data to generate a potential text embedding output. As another example, the machine-learned model 1701 may process natural language data to generate a translation output. As another example, the machine-learned model 1701 may process natural language data to generate a classification output. As another example, the machine-learned model 1701 may process natural language data to generate a text segmentation output. As another example, the machine-learned model 1701 may process natural language data to generate a semantic intent output. As another example, the machine-learned model 1701 may process natural language data to generate an upgraded text or natural language output (e.g., a text or natural language data with a higher quality than the input text or natural language, etc.). As another example, the machine-learned model 1701 may process natural language data to generate a prediction output (e.g., one or more predicted next parts of natural language content).

[0266] In some implementations, input 1702 may be or otherwise represent speech data (e.g., data describing a spoken natural language, such as audio data, text data, etc.). The machine-learned model 1 may process speech data to generate an output. As an example, the machine-learned model 1701 may process speech data to generate a speech recognition output. As another example, the machine-learned model 1701 may process speech data to generate a speech translation output. As another example, the machine-learned model 1701 may process speech data to generate a potential embedding output. As another example, the machine-learned model 1701 may process speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of speech data, etc.). As another example, the machine-learned model 1701 may process speech data to generate an upgraded speech output (e.g., speech data with a higher quality than the input speech data, etc.). As another example, the machine-learned model 1701 may process speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine-learned model 1701 may process speech data to generate a predicted output.

[0267] In some implementations, input 1702 may be or otherwise represent potentially coded data (e.g., a latent space representation of an input, etc.). Machine-learned model 1 may process potentially coded data to generate an output. As an example, machine-learned model 1701 may process potentially coded data to generate a recognition output. As another example, machine-learned model 1701 may process potentially coded data to generate a reconstruction output. As another example, machine-learned model 1701 may process potentially coded data to generate a search output. As another example, machine-learned model 1701 may process potentially coded data to generate a re-clustering output. As another example, machine-learned model 1701 may process potentially coded data to generate a prediction output.

[0268] In some implementations, input 1702 may be or otherwise represent statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from some other data source. Machine-learned model 1 may process statistical data to generate an output. As an example, machine-learned model 1701 may process statistical data to generate a recognition output. As another example, machine-learned model 1701 may process statistical data to generate a prediction output. As another example, machine-learned model 1701 may process statistical data to generate a classification output. As another example, machine-learned model 1701 may process statistical data to generate a segmentation output. As another example, machine-learned model 1701 may process statistical data to generate a visualization output. As another example, machine-learned model 1701 may process statistical data to generate a diagnostic output.

[0269] In some implementations, input 1702 may be or otherwise represent sensor data. The machine-learned model 1 may process sensor data to generate an output. As an example, the machine-learned model 1701 may process sensor data to generate a recognition output. As another example, the machine-learned model 1701 may process sensor data to generate a prediction output. As another example, the machine-learned model 1701 may process sensor data to generate a classification output. As another example, the machine-learned model 1701 may process sensor data to generate a segmentation output. As another example, the machine-learned model 1701 may process sensor data to generate a visualization output. As another example, the machine-learned model 1701 may process sensor data to generate a diagnostic output. As another example, the machine-learned model 1701 may process sensor data to generate a detection output.

[0270] In some implementations, the machine learning model 1701 can be configured to perform tasks including encoding input data to achieve reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task can be an audio compression task. The input can include audio data, and the output can include compressed audio data. In another example, the input includes visual data (e.g., one or more images or videos), the output includes compressed visual data, and the task is a visual data compression task. In another example, the task can include generating embeddings for input data (e.g., input audio or visual data). In some cases, the input includes audio data representing spoken utterances, and the task is a speech recognition task. The output can include text output mapped to spoken utterances. In some cases, the task includes encrypting or decrypting the input data. In some cases, the task includes microprocessor execution tasks, such as branch prediction or memory address conversion.

[0271] In some implementations, the task is a generative task, and the machine-learned model 1701 can be configured to output content generated from the input 1702. For example, the input 1702 can be or otherwise represent data of one or more modalities encoding a context for generating additional content.

[0272] In some implementations, the task may be a text completion task. The machine-learned model 1 may be configured to process an input 1702 representing text data and generate an output 1703 representing additional text data that completes a text sequence including the input 1702. For example, the machine-learned model 1701 may be configured to generate an output 1703 to complete a sentence, paragraph, or portion of text following a portion of the text represented by the input 1702.

[0273] In some implementations, the task may be an instruction-following task. The machine-learned model 1 may be configured to process an input 1702 representing an instruction for performing a function and generate an output 1703 that advances a goal of satisfying the instruction function (e.g., at least one step of a multi-step process for performing the function). Output 3 may represent data of the same or different modality as input 1702. For example, input 1702 may represent text data (e.g., natural language instructions for a task to be performed), and the machine-learned model 1701 may process input 1702 to generate output 1703, which represents text data in response to the instruction (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 may represent image data (e.g., image-based instructions for a task to be performed, optionally with text instructions), and the machine-learned model 1701 may process input 1702 to generate output 1703, which represents text data in response to the instruction (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 1703 may be generated iteratively or recursively to sequentially process and complete steps toward completing the requested function. For example, an initial output may be executed by an external system or processed by a machine-learned model 1701 to complete the initial steps of executing the function. Multiple steps may be performed and a final output responsive to the initial instruction may be obtained.

[0274] In some implementations, the task may be a question-answering task. The machine-learned model 1 may be configured to process an input 1702 representing a question to be answered and generate an output 1703 that advances the goal of returning an answer to the question (e.g., at least one step of a multi-step process for performing the function). Output 3 may represent data of the same or different modality as input 1702. For example, input 1702 may represent text data (e.g., natural language instructions for a task to be performed), and the machine-learned model 1701 may process input 1702 to generate output 1703, which represents text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). Input 2 may represent image data (e.g., image-based instructions for a task to be performed, optionally with text instructions), and the machine-learned model 1701 may process input 1702 to generate output 1703, which represents text data in response to the question (e.g., a natural language response, a programming language response, a machine language response, etc.). One or more outputs 1703 may be generated iteratively or recursively to sequentially process and complete steps toward answering the question. For example, an initial output may be executed by an external system or processed by a machine-learned model 1701 to complete the initial steps of obtaining an answer to the question (e.g., querying a database, performing calculations, executing scripts, etc.). Multiple steps may be performed and a final output responsive to the question may be obtained.

[0275] In some implementations, the task may be an image generation task. The machine-learned model 1 may be configured to process an input 1702 representing a context about a desired portion of the image content. The context may include text data, image data, audio data, etc. The machine-learned model 1 may be configured to generate an output 1703 representing image data depicting an image associated with the context. For example, the machine-learned model 1701 may be configured to generate pixel data for an image. The value of a channel associated with a pixel in the pixel data may be selected based on the context (e.g., based on a probability determined based on the context).

[0276] In some implementations, the task may be an audio generation task. The machine learning model 1 may be configured to process an input 1702 representing a context about a desired portion of the audio content. The context may include text data, image data, audio data, etc. The machine learning model 1 may be configured to generate an output 1703 representing audio data associated with the context. For example, the machine learning model 1701 may be configured to generate waveform data in the form of an image (e.g., a spectrogram). The value of a channel associated with a pixel of the image may be selected based on the context. The machine learning model 1 may be configured to generate waveform data in the form of a sequence of discrete samples of a continuous waveform. The value of the sequence may be selected based on the context (e.g., based on a probability determined according to the context).

[0277] In some implementations, the task may be a data generation task. The machine learning model 1 may be configured to process an input 1702 representing a context about a desired portion of data (e.g., data from various data domains, such as sensor data, image data, multimodal data, statistics, etc.). For example, the desired data may be synthetic data used to train other machine learning models. The context may include any data type. The machine learning model 1 may be configured to generate an output 1703 representing data aligned with the desired data. For example, the machine learning model 1701 may be configured to generate data values ​​for populating a data set. The value of a data object may be selected based on the context (e.g., based on a probability determined based on the context).

[0278] Fig.23 4 is a block diagram of an example networked computing system that can perform aspects of an example implementation of the present disclosure. The system can include multiple computing devices and systems that are communicatively coupled via a network 49. An example computing device 50 is described to provide an example of a computing device that can perform any aspect of the present disclosure (e.g., implement a model host 31, a client 32, or both). An example server computing system 60 is described as an example of a server computing system that can perform any aspect of the present disclosure (e.g., implement a model host 31, a client 32, or both). The computing device 50 and the server computing system 60 can interact collaboratively (e.g., via a network 49) to perform any aspect of the present disclosure (e.g., implement a model host 31, a client 32, or both). The model development platform system 70 is an example system that can host or provide a model development platform 12 for developing models for machine learning. The third-party system 80 is an example system that any of the computing device 50, the server computing system 60, or the model development platform system 70 can interact with when performing various aspects of the present disclosure (e.g., using third-party tools, accessing third-party databases or other resources, etc.).

[0279] Network 49 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications on network 49 may be carried via any type of wired or wireless connection using a wide variety of communications protocols (e.g., TCP / |IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), or protection schemes (e.g., VPN, secure HTTP, SSL). Network 49 may also be implemented via a system bus. For example, Fig.23 One or more devices or systems may be co-located with, housed by, or otherwise integrated into one or more other devices or systems.

[0280] Computing device 50 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, a server computing device, a virtual machine operating on a host device, or any other type of computing device. Computing device 50 may be a client computing device. Computing device 50 may be an end-user computing device. Computing device 50 may be a computing device for a provided service, which provides services to an end user (who may use another computing device to interact with computing device 50).

[0281] The computing device 50 may include one or more processors 51 and a memory 52. ​​The processor 51 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 52 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 52 may store data 53 and instructions 54 that may be executed by the processor 51 to cause the computing device 50 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein.

[0282] The computing device 50 may also include one or more input components that receive user input. For example, the user input component may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a camera, a LIDAR, a physical keyboard or other buttons, or other devices by which a user can provide user input.

[0283] The computing device 50 may store or include one or more machine-learned models 55. The machine-learned models 55 may include one or more machine-learned models 1701, such as a sequence processing model 4. The machine-learned models 55 may include one or more model instances 31-1. The machine-learned models 55 may be received from a server computing system 60, a model development platform system 70, a third-party system 80 (e.g., an application distribution platform), or developed locally on the computing device 50. The machine-learned models 55 may be loaded into the memory 52 and used or otherwise implemented by the processor 51. The computing device 50 may implement multiple parallel instances of the machine-learned models 55.

[0284] The server computing system 60 may include one or more processors 61 and memory 62. The processor 61 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 62 may include one or more non-transitory computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 62 may store data 63 and instructions 64, which may be executed by the processor 61 to cause the server computing system 60 to perform operations. These operations may implement any one or more of the features described herein. These operations may implement the example methods and techniques described herein.

[0285] In some implementations, the server computing system 60 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 60 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0286] The server computing system 60 may store or otherwise include one or more machine-learned models 65. The machine-learned models 65 may be the same as or different from the machine-learned models 55. The machine-learned models 65 may include one or more machine-learned models 1701, such as a sequence processing model 4. The machine-learned models 65 may include one or more model instances 31-1. The machine-learned models 65 may be received from the computing device 50, the model development platform system 70, a third-party system 80, or developed locally on the server computing system 60. The machine-learned models 65 may be loaded into the memory 62 and used or otherwise implemented by the processor 61. The server computing system 60 may implement multiple parallel instances of the machine-learned models 65.

[0287] In an example configuration, the machine-learned model 65 may be included in the server computing system 60 or otherwise stored and implemented by the server computing system 60 to establish a client-server relationship with the computing device 50 for providing model inference. For example, the server computing system 60 may implement the model host 31 on behalf of the client 32 on the computing device 50. For example, the machine-learned model 65 may be implemented by the server computing system 60 as part of a web service (e.g., a remote machine-learned model hosting service, such as an online interface for performing machine-learned model operations on the server computing system 60 over a network). For example, the server computing system 60 may communicate with the computing device 50 over a local intranet or Internet connection. For example, the computing device 50 may be a workstation or endpoint in communication with the server computing system 60, wherein the implementation of the machine-learned model 65 is managed by the server computing system 60 to remotely perform inference (e.g., for runtime or training operations), wherein the output is returned (e.g., projected, streamed, etc.) to the computing device 50. The machine-learned model 65 may work cooperatively or interoperably with the machine-learned model 55 on the computing device 50 to perform various tasks.

[0288] The model development platform system 70 may include one or more processors 71 and a memory 72. The processor 71 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a processor or multiple processors operatively connected. The memory 72 may include one or more non-temporary computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 72 may store data 73 and instructions 74, which may be executed by the processor 71 to enable the model development platform system 70 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functions described herein with respect to the model development platform 12. This function and other functions may be implemented by a developer tool 75.

[0289] The third-party system 80 may include one or more processors 81 and a memory 82. The processor 81 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a processor or multiple processors operatively connected. The memory 82 may include one or more non-temporary computer-readable storage media, such as HBM, RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 82 may store data 83 and instructions 84, which may be executed by the processor 81 to enable the third-party system 80 to perform operations. These operations may implement any one or more features described herein. These operations may implement the example methods and techniques described herein. Example operations include the functions described herein with respect to tools and other external resources (e.g., third-party resources 85) called when training a machine learning model 1701, 4, 16, 20, 55, 65, etc. or performing reasoning using the machine learning model.

[0290] Fig.23 An example arrangement of a computing system that can be used to implement the present disclosure is shown. Other computing system configurations may also be used. For example, in some implementations, one or both of the computing system 50 or the server computing system 60 may implement all or part of the operations of the model development platform system 70. For example, the computing system 50 or the server computing system 60 may implement the developer tool 75 (or its extension) to develop, update / train or refine machine learning models 1, 4, 16, 20, 55, 65, etc. using one or more techniques described herein with respect to the model alignment toolkit 17. In this way, for example, the computing system 50 or the server computing system 60 may develop, update / train or refine machine learning models based on a local data set (e.g., for model personalization / customization, as allowed by user data preference selections).

[0291] Fig.24 is a block diagram of an example computing device 98 executed according to an example embodiment of the present disclosure. Computing device 98 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement model host 31. For example, computing device 98 may include multiple applications (e.g., applications 1 to N). Each application may contain its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. Fig.24 As shown, each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0292] Fig.25 is a block diagram of an example computing device 99 executed in accordance with an example embodiment of the present disclosure. Computing device 99 may be the same as or different from computing device 98. Computing device 99 may be a user computing device or a server computing device (e.g., computing device 50, server computing system 60, etc.). Computing device 98 may implement model host 31. For example, computing device 99 may include multiple applications (e.g., applications 1 to N). Each application may communicate with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may use an API (e.g., a public API across all applications) to communicate with the central intelligence layer (and the models stored therein).

[0293] The central intelligence layer can include multiple machine learning models. Fig.25 As shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model is managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 99 or is otherwise implemented by the operating system.

[0294] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 99. Fig.25 As shown, the central device data layer can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, or additional components). In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0295] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0296] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes, or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, it is intended that the present disclosure encompass such changes, modifications, and equivalents.

[0297] Aspects of the present disclosure have been described with respect to its illustrative embodiments. Any and all features in the appended claims may be combined or rearranged in any possible manner, including combinations of claims that are not explicitly listed together in combination, because the example claim dependencies listed herein should not be understood to limit the scope of the possible combinations of features disclosed herein. Therefore, the scope of the present disclosure is illustrative rather than limiting, and the present disclosure does not exclude such modifications, changes or additions to the subject matter that will be easily understood by those of ordinary skill in the art. In addition, a list of example elements connected by conjunctions such as "and", "or", "but" is used herein to describe terms. It should be understood that such conjunctions are provided only for the purpose of explanation. For example, clauses and other project sequences connected by specific conjunctions such as "or" may refer to "and / or", "at least one of", "any combination" of the example elements listed therein, etc. Terms such as "based on" should be understood as "based at least in part on".

[0298] The term "may" should be understood to refer to the possibility of a feature in various implementations, rather than to specify a capability that must be present in every implementation. For example, the phrase "X may perform Y" should be understood to indicate that in various implementations, X may be configured to perform Y, rather than to indicate that in every instance X must always be able to perform Y. It should be understood that in various implementations, X may not be able to perform Y and still be within the scope of the present disclosure.

[0299] The term "may" should be understood to refer to the possibility of a feature in various implementations, rather than to specify a capability that must be present in every implementation. For example, the phrase "X may perform Y" should be understood to indicate that in various implementations, X may be configured to perform Y, rather than to indicate that in every instance X must always be able to perform Y. It should be understood that in various implementations, X may not be able to perform Y and still be within the scope of the present disclosure.< / background> < / surroundings> < / platform>

Claims

1. A computer-implemented method comprising: receiving data indicative of a request for a plurality of media assets including a plurality of media modalities to be generated based on pre-existing media assets; Generate multiple media asset components by: (i) extracting one or more signals from the pre-existing media asset, the one or more signals comprising at least one of background image data, color palette data, text data, or logo image data; (ii) using the first generative machine learning model to generate multiple background image assets in the following manner: generating a mask image including one or more bounding boxes based on at least one of the background image data, the text data, or the logo image data; inpainting the mask image by generating pixels using a machine learning model to fill the one or more bounding boxes of the mask image; generating a plurality of images, wherein each image comprises a different aspect ratio; (iii) using a second generative machine learning model to generate multiple text assets in the following manner: obtaining the text data associated with the one or more bounding boxes; Inputting the text data into a machine learning model; Generate multiple text assets as output through the machine learning model; as well as (iv) generating unique profile data based on at least one of the color palette data or the logo image data using a third generative machine learning model; as well as The plurality of media assets, including one or more image assets, text assets, and unique profile data, are automatically transmitted to a content item generation pipeline for use in generating a plurality of candidate content items.

2. The computer-implemented method of any preceding claim, wherein generating the mask image comprising one or more bounding boxes comprises: identifying one or more text objects; For each text object of the one or more text objects: determining that a first text object of the one or more text objects is an overlay text object; and Based on determining that the first object is an overlay text object, a first bounding box is generated for the first text object.

3. The computer-implemented method of any preceding claim, wherein generating the plurality of media asset components comprises: Extract logo image; Select a primary color from the image palette; Upgrading said logo image by: enlarging the logo image; as well as sharpening the logo image; as well as The logo image is generatively expanded to generate one or more images, wherein each image includes a set of different aspect ratios.

4. The computer-implemented method of claim 3, wherein generatively expanding the logo image comprises: Pixels are generated to merge with existing pixels of the logo image to generate an image that is larger than the original logo image. 5 . The computer-implemented method of claim 1 , wherein generating the plurality of background image assets comprises adjusting at least one of brightness, saturation, or contrast of respective background image assets.

6. The computer-implemented method of any preceding claim, wherein the unique profile data comprises at least one of logo data, color palette data, font data, or image style data.

7. A computer-implemented method as claimed in any preceding claim, comprising: inputting the plurality of output assets into a content creation pipeline; as well as A plurality of content items are generated by the content creation pipeline, wherein each content item of the plurality of content items comprises a unique combination of content assets and an aspect ratio.

8. The computer-implemented method of any preceding claim, wherein generating the plurality of text assets comprises: extracting a signal including text data; compiling the text data; Inputting the compiled text data into the second generative machine learning model; obtaining at least one of a short title or a long title as an output from the second generative machine learning model; Inputting at least one of the short title or the long title into the second generative machine learning model; as well as A description is obtained as an output from the machine learning model.

9. The computer-implemented method of any preceding claim, wherein the one or more bounding boxes comprise a position and a size of the bounding boxes.

10. The computer-implemented method of any preceding claim, wherein the content item generation pipeline comprises a prompt generation component and a content item generation component.

11. The computer-implemented method of claim 10, comprising: generating, by the prompt generation component, input prompt data based on the background image, the text asset, and the unique profile data; providing the input prompt data to the content item generation component; as well as One or more candidate content items are generated by the content item generation component based on the input prompt data.

12. The computer-implemented method of claim 1, wherein the content item generation component comprises a generative machine learning model.

13. The computer-implemented method of claim 12, comprising: The generative machine learning model is trained in the following way: generating a training data set based on comparing the generated content item components to existing media assets; as well as Based on comparing the generated content item components to the existing media assets, one or more parameters of the generative machine learning model are automatically adjusted to reduce differences between the generated content item and the existing media assets.

14. A computing system comprising: one or more processors; as well as One or more transitory or non-transitory computer-readable media storing instructions executable to cause the one or more processors to perform operations comprising: receiving data indicative of a request for a plurality of media assets including a plurality of media modalities to be generated based on pre-existing media assets; Generate multiple media asset components by: (i) extracting one or more signals, the one or more signals comprising at least one of background image data, palette data, text data, or logo image data; (ii) generating a plurality of background image assets based on at least one of the background image data, the text data, the logo image data, or the color palette data using a first generative machine learning model; (iii) generating a plurality of text assets based on at least one of the text data or the background image data using a second generative machine learning model; (iv) generating unique profile data based on at least one of the color palette data, the text data, or the logo image data using a third generative machine learning model; and The plurality of media assets, including one or more image assets, text assets, and unique profile data, are automatically transmitted to a content item generation pipeline for use in generating a plurality of candidate content items.

15. The computing system of claim 14, the operations comprising: Data indicative of the pre-existing media asset being uploaded is received.

16. The computing system of claim 14 or 15, wherein the operations include: The web resource associated with the unique profile data is parsed to obtain the pre-existing media asset.

17. The computing system of any one of claims 14 to 16, wherein the pre-existing media asset comprises flattened image data.

18. The computing system of claim 17, wherein the request is associated with a customer account, and wherein the customer account is associated with an account profile that stores input to a machine-learned media asset generation pipeline.

19. The computing system of claim 18, wherein the profile is retrieved from a database, and wherein the profile was previously generated prior to the request.

20. One or more transient or non-transitory computer-readable media storing instructions executable by one or more processors to perform operations including the computer-implemented method described in any one of claims 1 to 19.

Images