Digital content creation

Abstract

The technology relates to an artificial intelligence (AI) driven tool that can enhance existing digital video components and simplify and automate the creation of new digital video components. The techniques include a digital video component creation tool that generates a digital video component using existing material, an out-of-picture tool that may add an out-of-picture generated from text to the video component, and a video component evaluation tool that may evaluate whether the video component conforms to an attribute associated with an optimal practice of a video creative.

Description

Background Art

[0001] The campaign management platform manages and delivers digital content to users' user computing devices, which form part of the audience of computing devices targeted to receive digital content. Various events throughout the year, such as local, national, and / or international events, provide content providers with unique opportunities to reach the audience of computing devices targeted to receive digital content. New campaigns can be created for any number of events. However, creating new campaigns and associated digital content can be a time-consuming and resource-intensive process. Furthermore, creating seasonal campaigns often requires specialized knowledge and skills to be effective.

[0002] Digital video components are often more complex and challenging to create than text or image components. In this regard, video components often require more time and expense to create and produce than text or image components. For example, video components must account for the many different devices and formats on which they will be displayed. Each device and format may present its own unique set of challenges. For example, a video component intended for television may have different requirements than a video component intended for social media. Additionally, a video ad created for a mobile device may have different requirements than a video ad created for a desktop computer, television, or other larger display.

[0003] Small and medium-sized businesses (SMBs) often struggle to create effective video components because they typically have limited budgets and resources, making it difficult to produce high-quality video components for different devices and formats. Furthermore, SMBs often lack the expertise and experience required to create effective video components, or lack the resources to hire such personnel. Consequently, video components produced by SMBs may not be as effective as those created by businesses with the appropriate resources and experience. Summary of the Invention

[0004] The technology generally relates to generating digital content based on prompts and other inputs to an artificial intelligence (AI) model. These prompts can be, for example, inputs for modifying digital content associated with an existing activity. These prompts include natural language inputs. Natural language prompts can include, for example, subject matter, information related to an upcoming event, the topic of the activity, etc. These inputs can include information associated with the style of the activity creator. The input associated with the activity creator's style can take the form of an embedding. A system implementing the methods described herein can generate digital content for a modified activity based on the received input received from the activity. The generated digital content can take the form of text, images, video, audio, etc.

[0005] The technology involves artificial intelligence (AI)-driven tools that can enhance existing digital video components and simplify and automate the creation of new digital video components. The technology includes a digital video component creation tool that generates digital video components from existing assets, a voice-over tool that can add voice-over generated from text to video components, and a video component evaluation tool that can evaluate whether a video component conforms to attributes associated with best practices for video creatives.

[0006] Other implementations of this and other aspects include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the acts or operations of the methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a block diagram of an example digital content generation system in communication with a campaign management platform according to aspects of the present disclosure.

[0008] Figure 2 is a block diagram of another example digital content generation system in communication with a campaign management platform according to aspects of the present disclosure.

[0009] Figure 3 is a block diagram of another example digital content generation system in communication with a campaign management platform according to aspects of the present disclosure.

[0010] Figure 4 is an example illustration of a video component generated from a collection of footage according to aspects of the present disclosure.

[0011] Figure 5 is an example illustration of a generated video component according to aspects of the present disclosure.

[0012] Figure 6 Depicted is a flow diagram of an example process for generating digital content according to aspects of the present disclosure.

[0013] Figure 7 Depicted is a flow diagram of another example process for generating digital content according to aspects of the present disclosure.

[0014] Figure 8 is a block diagram illustrating one or more models according to aspects of the present disclosure, such as for deployment in a data center housing one or more hardware accelerators, on which the deployed models are to be executed to generate digital content based on generated and stored prompts.

[0015] Figure 9 is a block diagram of an example computing environment for implementing a digital content generation system. DETAILED DESCRIPTION

[0016] Overview

[0017] Aspects of the present disclosure relate to a system for dynamically generating digital content generated by an artificial intelligence (AI) model based on modification input, existing digital content, and other inputs. The modification input is data that at least partially represents the changes and / or updates to be made to the digital content of an existing activity. The existing digital content is digital content associated with the activity creator, such as text, images, videos, etc. The existing digital content can be digital content used in an existing activity, digital content provided for output on the activity creator's website or mobile app, etc. Other inputs can include, for example, embeddings associated with themes identified from the modification input, style embeddings associated with existing digital content, etc. Once the digital content is generated, it can be delivered to a target computing device audience. The generated digital content can be a modified version of existing digital content and / or new digital content.

[0018] The generated digital content can be images, text, video, audio, and so on. The digital content is generated to match the campaign creator's style. Style is the way digital content is presented, for example, to create a mood or feeling or to convey a message to a target audience. The style is determined based on the campaign creator's existing digital content. For example, an embed is determined for existing digital content associated with the campaign creator. The embed can be a stylistic embed, for example, one that identifies and / or quantifies the style of content associated with the campaign creator.

[0019] The style embeddings are used to determine a style vector. The style vector corresponds to a weight or scaling factor provided as input to the AI ​​model during model execution. The style vector is determined based on the average distance between at least some of the style embeddings associated with the activity creator. According to some examples, the style vector may be determined based on the average distance between each of the style embeddings associated with the activity creator. The AI ​​model uses, in part, the style vector to generate digital content in the style of the activity creator.

[0020] Previously, generating digital content in a given style required generating new digital content from scratch, e.g., without using existing digital content. Furthermore, to ensure that the new digital content matched the content creator's style and the campaign's subject matter, a significant amount of time was required in brainstorming sessions to come up with ideas, and a significant amount of money was required in filming and post-processing to execute the ideas. By generating digital content in the campaign creator's style and matching a given subject matter, the system can accurately generate and deliver digital content to a target computing device audience. Specifically, the system can accurately generate digital content that matches the campaign creator's style, is relevant to the campaign's subject matter, is identified based on modified inputs provided to the AI ​​model, and includes features that the target audience will engage with. By generating digital content that matches the style and subject matter and additionally captures the target audience, the AI ​​model improves the efficiency and accuracy of generating and delivering digital content.

[0021] Furthermore, generative models are not deterministic. For example, even if the model is fed the same input, it may not always produce the same output. For example, if the input to the generative model includes hints indicating the style of the content creator, the output of the generative model may have a different style each time. To ensure consistency for a given content creator, a style vector is provided as input to the model. The style vector provides consistency in the style of the generated output, thereby solving the problem of controlling the style of the generated output.

[0022] According to some examples, digital content is generated to match the context of the campaign and / or the creator of the campaign. The context includes, for example, information associated with when the digital content is delivered to the audience, the format of the digital content (e.g., text, video, image, audio, etc.), whether the digital content is provided for output on a website or in a mobile application, etc.

[0023] In some examples, an activity creator may correspond to a content creator. References to activity creators and / or content creators are used interchangeably throughout this document. Therefore, the use of activity creators or content creators is not intended to be limiting. Rather, activity creators and content creators can be considered the same creator.

[0024] Additional aspects of the present disclosure relate to content generation tools that utilize AI (including machine learning and generative AI) to assist in creating and enhancing digital video components. These tools may include creation tools, including video component creation tools that utilize existing components to generate new video components. These tools may also include enhancement tools, such as a voice-over tool, which can change the content of a video component. In this regard, the voice-over tool can add a voice-over generated from input text to a video component. These tools may also include evaluation tools that determine the degree to which a video component complies with attributes associated with best practices for video creativity so that the video component can be edited to improve its effectiveness. The evaluation tools may include the video component evaluation tool described in detail herein.

[0025] These tools can be applications (such as web-based applications provided to client devices from a server or stand-alone applications executed on client devices), or frameworks that can be incorporated into other applications. In addition, although these tools are described separately herein, each tool can be combined into a single application or framework, or otherwise implemented in conjunction with other tools within a single application or framework. For example, an application can include some or all of the features of a video component creation tool, a voice-over tool, and a video component evaluation tool. In another example, an application or framework can include a voice-over tool. Further, these tools can be implemented in separate frameworks or applications, which can be packaged together. For example, an application can include a voice-over tool and a video component evaluation tool.

[0026] Although each of the tools described herein is categorized as a creation tool, enhancement tool, or evaluation tool, these categorizations are for purposes of explanation and reference only. Each tool is not limited to the functionality of the category to which it is assigned in this disclosure. For example, a video component creation tool can create a video component, enhance the created video component, and evaluate the created video component, thereby satisfying each of these three categories.

[0027] The systems and methods described above and herein allow event creators to efficiently and effectively modify digital content for existing events and / or generate digital content for events through intuitive natural language input and style embedding. The AI ​​model can transform the event creator's existing digital content to capture different events, themes, topics, and the like. The digital content generated by the AI ​​model adapts the event creator's existing digital content to various scenarios, such as sales events, product launches, market trends, and the like, while maintaining the event creator's style and / or context. The AI ​​model, combined with an event management platform, allows for the creation, management, and optimization of events and associated digital content to reach target audiences.

[0028] Example System

[0029] Figure 1 is a block diagram of an example digital content generation system 190 in communication with a campaign management platform 150, in accordance with aspects of the present disclosure. In some examples, the digital content generation system 190 and the campaign management platform 150 may be part of a larger system, while in other examples, the digital content generation system 190 and the campaign management platform 150 are implemented on separate devices in one or more physical locations.

[0030] The digital content generation system 190 and the campaign management platform 150 can communicate via a network. In some examples, the digital content generation system 190 does not communicate with the campaign management platform, but instead communicates directly with the user computing devices 180A, 180B, and 180C to receive input and generate output. The recognition engine 175 can receive some or all of the data forming the modification input 130 via the campaign management platform 150. In some examples, the digital content generation system 190 is configured to perform some or all of the operations or components described as being performed by the campaign management platform 150.

[0031] The campaign management platform 150 can be configured to manage content delivery to the user computing devices 180A, 180B, and 180C and provide a user interface for doing so. For example, the user interface can be configured as a web interface, an API, a standalone software application, etc., for organizing digital content according to different targeting parameters and delivering the digital content to different user computing devices.

[0032] Content delivery can be organized into one or more campaigns, each logically associated with a particular topic of digital content. Campaigns can be further broken down into groups, representing potential differences in the type of content to be delivered. Groups can be further broken down into line items, representing even more specificity in the digital content to be delivered, the time when the content is delivered, and / or the computing devices targeted for the content. The time when the content is delivered corresponds to the delivery schedule for the content. Digital content, the time period during which the digital content is to be delivered to different user computing devices, and / or the targeting parameters for selecting which user computing devices to deliver the content can be selected at the campaign, group, or line item level.

[0033] After the campaign management platform 150 receives the content, the delivery schedule for the content, and the targeting parameters for delivering the content to the computing device, the campaign management platform 150 is configured to deliver the content to the user computing device 180A-C. In some examples, a separate component (e.g., a separate engine running on the same device or a separate device from the platform) can cause the content to be delivered, such as by directly delivering the content or sending a request or command to another system configured to deliver the content. The delivery schedule can be as short as the time it takes to deliver the content to the user computing device 180A-C. In other examples, the delivery schedule can be of any length, such as a few hours, days, weeks, etc. Delivering the content can include sending the content over a network for display or output by the device, or causing content stored on the user computing device to be displayed or otherwise output.

[0034] The digital content generation system 190 includes a seed generation engine 195, a recognition engine 175, an image embedding acquirer 105, a generative model 110, an image embedding repository 115, and a digital content repository 140, which, in various examples, can be implemented on one or more computing devices in one or more physical locations. The seed generation engine 195 is configured to receive a modification input 130. The modification input 130 can be retrieved through an interface, such as an API or a standalone software application configured to retrieve descriptions and / or values ​​from a source such as a database or other repository. In some examples, the data is retrieved from a campaign management platform 150, such as Figure 1 shown.

[0035] Modification input 130 is data that at least partially characterizes changes to be made to the digital content of an existing campaign. For example, the modification input may be related to a topic, product, service, event, etc. (collectively referred to herein as an "event") that is the subject of the digital content to be generated. As an example, the modification may be related to an upcoming event, and the modification input 130 may include the name of the event, a description of the event, keywords related to the event, etc. The modification input 130 may include natural language, tags, titles, etc. The digital content generation system 190 may automatically retrieve components of the modification input 130 or the modification input 130 from various sources, including the campaign management platform 150 and user input.

[0036] In some examples, modification input 130 includes data related to the campaign under which the digital content is to be delivered. For example, modification input 130 may include group information, advertising items, delivery schedule information, targeting information, etc. associated with the campaign.

[0037] Modification input 130 may include data in different modalities such as images, videos, computer graphics, audio, text, etc. For example, modification input 130 may include a text description of an event, a name of an event, and images or videos of the event in some context.

[0038] The seed generation engine 195 is configured to generate relevant seed information related to events related to the product or service of the activity creator. For example, the seed generation engine 195 is configured to identify events that will be used to update existing activities and / or existing digital content based on the modification input 130. Based on the identified event, the seed generation engine 195 generates relevant seed information about the event. For example, the seed generation engine 195 can generate a natural language prompt describing the elements of each modification input in the modification input 130 based on the modification input 130. The prompt can include text information included as part of the input 130 and a text description of the input of other modalities provided, such as a text description of an image, video, or audio transcription. In some examples, the seed generation engine 195 is further configured to generate relevant seed information for the event based on the computing device audience that is directed to receive the digital content.

[0039] As another example, the digital content generation system 190 may receive modification input 130 to transform existing digital content for the release of a new cake menu for the winter season. The modification input 130 may be received as user input provided via the campaign management platform 150. In some examples, the modification input 130 may be a prompt automatically and / or semi-automatically generated by another generative model. The landing page or website associated with the existing digital content may be "example.com / cakes." The seed generation engine 195 may receive the modification input 130, for example, a request to transform the existing digital content for the release of a new cake menu for the winter season. Based on the modification input, the seed generation engine 195 may generate new information that provides additional details about the landing page (e.g., "example.com / cakes") and event information (e.g., the release of a new cake menu for the winter season). In some examples, the seed generation engine 195 may use one or more prompting techniques to create more relevant information. For example, the seed generation engine 195 may provide rephrasing and response prompts to improve the modification input 130.

[0040] The output of the seed generation engine 195 (e.g., seed information) is provided to the recognition engine 175. The recognition engine 175 identifies the theme to be used to obtain the relevant image embedding. For example, the modification input 130 may indicate that an existing campaign is to be modified for the start of a summer sale. As an example, the natural language input of the modification input 130 may be "summer sale start." The recognition engine 175 may determine that the theme is "summer" and / or "sale" based on the seed information from the seed generation engine 195.

[0041] According to some examples, recognition engine 175 can generate a natural language prompt corresponding to the theme based on the seed information. For example, recognition engine 175 can be a generative model that can receive text information (e.g., seed information) as input and provide text information (e.g., theme) as output. The output of recognition engine 175 can be a prompt for querying embedding acquirer 105.

[0042] In some examples, the recognition engine 175 creates embedding tags that are used to represent the type and / or kind of embedding required to transform existing digital content. In some examples, the recognition engine 175 creates similar data to generate text materials. By separating the recognition engine 175 from the seed generation engine 195, the embedding acquirer 105 and / or the generative model 110, the generative model 110 is allowed to generate digital content and / or other materials that are related to each other, such as text materials. As an example, when the modification input is "new cake menu release for winter", the recognition engine 175 can generate tags such as cake, winter and snow to generate an image. Similar tags can be generated for other digital content (such as text).

[0043] The theme can be provided to the embedding obtainer 105. The embedding obtainer 105 can access the embedding repository 115. The embedding repository 115 can store image embeddings that will be used to represent events (e.g., summer sales start) or themes (e.g., summer) of the modified activity. The embeddings within the embedding repository 115 can include, for example, image, video, text, audio, and the like embeddings. According to some examples, the image embedding obtainer 105 can query the embedding repository 115 to identify one or more relevant embeddings. The relevant image embedding can be, for example, an embedding related to an event and / or theme associated with the modification input 130. According to some examples, the relevant embedding can be an embedding within a threshold distance of the theme. For example, the theme can correspond to a given embedding. The relevant image embedding can be an embedding within a threshold distance of a given embedding associated with the theme.

[0044] In some examples, seed generation engine 195 can be configured to query embedding repository 115 to identify relevant embeddings based on the determined subject matter, rather than using embedding fetcher 105 as an intermediary. For example, embeddings with similar labels can be embeddings that are within a threshold distance of the labels generated by recognition engine 175. In some examples, embedding fetcher 105 uses clustering techniques to find the closest matches.

[0045] As another example, the embedding acquirer 105 can generate embeddings instead of retrieving embeddings from a pre-populated repository. For example, the embedding acquirer 105 can implement an AI model that is trained to receive seed information as one or more modalities and identify corresponding images described by or associated with the seed information. The embedding acquirer 105 can further process the image by being configured to generate an embedding model based on the input, or the embedding acquirer 105 can be trained to output image embeddings directly based on the seed information, for example, by performing end-to-end training on inputs labeled with corresponding image embeddings.

[0046] Making the seed generation 195, recognition engine 175, and embedding acquirer 105 separate modules reduces hallucinations and sensitive outputs by adding controller filters between each module. For example, hallucinations may be caused in part by the model perceiving (e.g., images, videos, etc.) or understanding (e.g., text, etc.) things that do not exist or perceiving things in a way that humans cannot predict, causing the generative model 110 to produce meaningless outputs. Sensitive outputs can be outputs that violate content policies, such as digital content and / or text. Dividing the digital content generation system 190 into separate modules (e.g., seed generation engine 195, recognition engine 175, embedding acquirer 105, etc.) allows filters to be implemented between each module. The filter can be configured to remove data and / or inputs that have been identified as causing hallucinations and / or sensitive outputs, including modifying the input. In some examples, the filter can remove and / or average outliers, specific keywords, incomplete data, etc. in the original input.

[0047] Generative model 110 is an AI model that is trained to receive embeddings, subject matter, and existing digital content. The AI ​​model is trained to generate digital content based on the received input. The generated digital content can be, for example, informational information, entertainment, advertising, etc.

[0048] Existing digital content can include, for example, existing campaigns, previously generated digital content, digital content associated with a campaign creator's landing page (e.g., a website) or mobile application, and the like. Existing digital content can be represented in a variety of formats, including digital, categorized, textual, or other computer-readable formats. Existing digital content stored in the digital content repository 140 can be used to identify a style associated with the campaign creator. The style can be, for example, modern, emotional, classical, streamlined, text-heavy, colorful, abstract, and the like.

[0049] The style can be determined based on embeddings associated with the campaign creator's existing digital content. For example, embeddings for the campaign creator's existing digital content can be determined. As an example, the recognition engine 175 creates embed tags. The embed tags indicate the type of embed desired. The type of embed desired may be associated with, for example, different styles, content types (e.g., video, audio, image, text, etc.), different themes, colors, etc. The embedding fetcher 105 can use these tags when querying the embedding repository 115. For example, the embedding fetcher 105 can use these tags to query the embedding repository to identify embeddings with similar tags.

[0050] An embedding can correspond to a numerical representation in a fixed-dimensional space, such as an ordered set of a fixed number of numeric values. The embedding at least partially encodes some semantic meaning of the text, image, video, etc. represented by the embedding. Embeddings are used to enable more direct comparisons with other data represented by the embedding, especially when the data is of different modalities.

[0051] For example, the embedding can be a vector or matrix of floating point values ​​or other types of numerical values. The embedding can use less data to represent the characteristics of the existing digital content, thereby enabling more efficient processing. For example, instead of having the generative model 110 receive existing digital content (which may be large in size, such as gigabytes (GB)) as input, the generative model 110 receives embeddings that are relatively small in size, such as kilobytes (KB).

[0052] Embeddings can be pre-generated using one or more AI models trained to generate embeddings. An embedding model is a model trained to generate embeddings that can be enhanced, for example, to classify an input image or data of another modality corresponding to the type of embedding generated by the embedding model. Embeddings can be used to search for images using a set of predetermined embeddings. For example, a text embedding corresponding to a text search term can be compared to image embeddings for a set of images to identify text / image embeddings that are within a threshold of similarity to each other. Embeddings can be stored and queried, for example, using a database configured to process vectors or other data structures for representing embeddings.

[0053] The system can be configured to receive existing digital content associated with a given campaign creator and generate an embed. For example, the embed can be a style embed. For example, the embed engine can receive existing digital content associated with the campaign creator. The embed engine can generate a style embed associated with the campaign creator's style. In some examples, the embed engine identifies the campaign creator's style.

[0054] For example, existing digital content can be clustered based on visual similarity. Visual similarity can be based on the presence of certain colors, shapes, objects, features, characteristics, text, etc., for example. A style embedding is then determined for the clustered digital content. By first clustering the images and then determining the style embedding, the computational efficiency of the system is improved compared to determining an embedding for each image because embeddings for similar images can be determined. The embeddings can be stored in a database, memory, storage system, or the like accessible to the system, such as an image embedding repository 115 and / or a digital content repository 140. For example, as part of processing training data and / or inference data to generate digital content, the generative model 110 can access a database to retrieve the embeddings. The generated digital content can be, for example, images, audio, video, text, etc.

[0055] Style embeddings associated with existing digital content are used to determine a style vector associated with the campaign creator. In some examples, the style vector can be referred to as a scale or weight. The style vector for a given campaign creator can be determined based on an average of at least some of the differences between embeddings of the campaign creator's existing digital content. In some examples, the style vector can be based on an average of all differences between embeddings. When generating digital content, the style vector is provided as input to the generative model 110. The style vector allows the concept from the modified input 130 to be retained while also preserving the style of the content creator. In some examples, the generative model 110 can avoid hallucinations caused by the style vector.

[0056] According to some examples, a model can be used to determine a style vector associated with the activity creator. For example, an AI model can be trained to predict a style vector. The AI ​​model can be configured to receive as input a text embedding, a style embedding, and inertia associated with the embedded clusters, and provide a prediction of the style vector as output. The text embedding can be based on natural language input provided as part of modification input 130, such as a prompt.

[0057] In some examples, the style vector can be a range of values. This range of values ​​can be provided as input to the generative model 110, causing the generative model 110 to generate multiple digital content, each having a slightly different style based on a different style vector value within the range. In such an example, the digital content generation system 190 can filter the digital content so that only some of the generated digital content is provided for output. In another example, the digital content generation system 190 can provide the generated digital content as output to the campaign creator, for example, via the campaign management platform 150.

[0058] According to some examples, rather than generating a style vector to be provided as input to the generative model 110, the generative model 110 may determine the style vector when generating the digital content. For example, the generative model 110 may determine the style vector based on received input (e.g., input received from the recognition engine 175, the embedding acquirer 105) and existing digital content stored in the digital content repository 140. The generative model 110 may generate the digital content based on the determined style vector.

[0059] By determining a style vector based on the activity creator, the output of the generative model 110 (e.g., the generated digital content) has a style that corresponds to the existing style of the activity creator. This allows the generative model 110 to output activity creator-specific digital content that reflects the style, look, feel, etc. associated with the activity creator. Conversely, without an activity creator-specific style vector, the output of the generative model 110 would be the same style for all activity creators, regardless of their known style differences. The generative model is not deterministic, so that the same input will not produce the same output. By providing a style vector as an input to the model, the style vector can promote the consistency of the digital content generated for the activity creator. Even if other inputs change, the style vector can provide control and / or consistency of the content style of the activity creator.

[0060] According to some examples, the style vector is stored in memory along with the existing digital content. For example, during execution, the generative model 110 receives as input the image embedding acquired by the image embedding acquirer 105, the subject matter identified by the recognition engine 175, and the existing digital content (including the style vector). The generative model 110 provides as output generated digital content having a style corresponding to the style vector and matching the subject matter identified by the recognition engine 175.

[0061] For example, and as also referred to Figure 8 and Figure 9As described, the generative model 110 may include one or more generative models, such as a language model, a base model, and / or a graphical model. The generative model 110 may be trained to generate digital content of different modalities, either as a standalone model or as a multimodal model. In examples where the generative model 110 is trained to generate digital content of one or more different modalities, the generative model 110 receives input or some indication as to whether to generate digital content in a combination of text, images, video, etc.

[0062] The generative model 110 can implement one or more encoders and decoders for generating trained representations of input data and decoding these representations to generate new digital content. These representations can be discrete representations or continuous representations of the input data, for example, represented as vectors. The encoder can include a transformer with a self-attention mechanism to encode the input data, which can be received by the model 110 as a series of tokens, frames, or other data units. The encoding layer of the model 110 can be fed into addition and normalization layers and then further processed by a nonlinear model (such as a neural network).

[0063] The decoder of the generative model 110 can receive and process a representation of input data to obtain an output corresponding to some digital content in response to the input data. To generate an image or video from text, the model 110 can encode the prompt using one or more trained text encoders. The model 110 can implement a diffusion-based model or other model technology for taking a text representation as input and generating a corresponding image or other digital content item in response to the input. A diffusion model is a type of generative model that converts noise into samples from a learned data distribution. In general, any AI model technology for generating digital content from a text prompt can be used to implement the generative model 110. As described herein with reference to Figure 4 Details for training an example model such as generative model 110 are described.

[0064] Generative model 110 can generate digital content items 170A, 170B, and 170C. Digital content items 170A-170C can be generated by processing the subject matter identified by recognition engine 175, the image embedding obtained by image embedding obtainer 105, and existing digital content. Model 110 processes these inputs to generate digital content item 170A. Similarly, other inputs with different subject matter and / or events can be used to generate other content. These other prompts, when processed by generative model 110, cause the generative model to generate digital content items 170B and 170C for targeting user computing devices 180B and 180C, respectively.

[0065] According to some examples, the generative model 110 can generate digital content items as new content items and / or modifications to previous content items. For example, when generating new content items, the generative model 110 can generate new content items that match the modification input 130 and the style of the existing digital content. The generative model 110 can provide one or more new digital content items as output, these new digital content items representing different aspects of the product or service that match the modification input and the style of the campaign creator. In some examples, the generative model 110 can replace the background of the existing digital content item so that the background matches the modification input and style of the campaign creator.

[0066] Although Figure 1 Only three user computing devices 180A-C are shown, but in general, thousands or more computing devices may be targeted for delivery of digital content items 170A-C. Content delivery may be performed automatically, for example, in response to a request for content from a computing device. The campaign management platform 150 may determine which devices to target using different targeting parameter values, for example, based on previous interactions with the device, spontaneous information provided from the device to the campaign management platform 150, etc. In some examples, the device may not regularly interact with user input, but rather be deployed somewhere to output or display content at the deployed location (e.g., a subway transit station, a billboard, etc.).

[0067] The digital content items 170A-170C may differ in style, format, or the way they describe or depict a product or service. These differences are due to the way the model 110 is trained to generate digital content in response to different modification inputs. According to some examples, these differences may be due to the style vector provided as input to the model 110.

[0068] Figure 2 is a block diagram of an example digital content generation system 290 in communication with a campaign management platform according to aspects of the present disclosure. Similar to digital content generation system 190, digital content generation system 290 and campaign management platform 150 can be part of a larger system and / or can communicate over a network.

[0069] The digital content generation system 290 includes the seed generation engine 195, the context recognition engine 275, the generative model 110, and the digital content repository 140, which, in various examples, may be implemented on one or more computing devices in one or more physical locations.

[0070] The seed generation engine 195 is configured to receive modification input 130, such as Figure 1As described. Modification input 130 is data that at least partially characterizes the changes to be made to an existing campaign. Seed generation engine 195 is configured to generate relevant seed information related to an event related to the campaign creator's product or service. In some examples, seed generation engine 195 is further configured to generate relevant seed information for the event based on the audience of computing devices targeted to receive digital content. The output of seed generation engine 195 is provided to generative model 210. The output of seed generation engine 195 can be a natural language prompt describing the elements of each modification input in modification input 130.

[0071] According to some examples, upon receiving the modification input 130, the digital content generation system 290 identifies existing digital content stored in the digital content repository 140. The existing digital content stored in the digital content repository 140 can be used to identify a context associated with the activity. The context can be, for example, information associated with when the digital content is delivered to the audience, the format of the digital content (e.g., text, video, image, etc.), whether the digital content is provided for output on a website or in a mobile application, etc. The context recognition engine 275 receives and / or retrieves the existing digital content from the digital content repository and identifies the context of the activity.

[0072] Generative model 210 is an AI model trained to receive context determined by context recognition engine 275, seed information generated by seed generation engine 195, and existing digital content stored in digital content repository 140, and generate digital content. Generative model 210 can generate digital content items 270A, 270B, and 270C. Digital content items 270A-C can be in text format. The text output by the generative model can include a title for an activity, a description of the activity theme, keywords to be included in the activity, and the like.

[0073] Digital content items 270A-270C can be generated by processing the context determined by the context recognition engine 275, the seed generated by the seed generation engine 195, and the existing digital content stored in the digital content repository 140. The model 210 processes these inputs to generate the digital content item 270A. Similarly, other modified inputs that result in the generation of different seeds can be used to generate other content. These other prompts, when processed by the generative model 210, cause the generative model to generate digital content items 270B and 270C for targeting the user computing devices 180B and 180C, respectively.

[0074] According to some examples, generative model 210 can generate digital content items as new content items and / or modifications to previous content items. For example, when generating new content items, generative model 210 can generate new content items that match modification input 130 and the style of existing digital content. Generative model 210 can provide one or more new digital content items as output, each representing different aspects of a product or service that match the modification input and the style of the campaign creator. For example, the generated digital content provided as output by generative model 210 can include text output. The text output can include a modified description, keywords, digital content generation prompts, a title, etc., which can be used as part of the campaign. The text output can be modified based on modification input 130. For example, if modification input 130 includes an indication of a summer sale, the description of the product or service associated with the campaign can be modified to be relevant to summer sales. As an example, if the original description is "all-season shoes," the modified description can be "summer shoes." Similarly, the title associated with the campaign creator's product and / or service, keywords used as part of the campaign, etc. can be updated to reflect the modification input 130.

[0075] Figure 3 is a block diagram of an example digital content generation system 390 in communication with a campaign management platform 150, according to aspects of the present disclosure. Similar to the digital content generation systems 190, 290, the digital content generation system 390 and the campaign management platform 150 can be part of a larger system and / or can communicate over a network.

[0076] The digital content generation system 390 includes a digital content repository 140, a generative model 310, and an asset analysis and component generation system (AACGS) 301, which can be implemented on one or more computing devices. The digital content repository 140 can store existing digital content associated with the campaign creator. In some examples, the digital content repository 140 can store generated digital content generated by the AACGS 301.

[0077] Generative model 310 is trained to receive digital content stored in digital content repository 140 and generate features associated with the digital content, which can be provided as input to AACGS 301. These features can be, for example, embeddings, content descriptors, or the like associated with the digital content.

[0078] Embeddings can at least partially encode some semantic meaning of the text, image, video, etc. (e.g., digital content) represented by the embedding. In some examples, the embedding can be a representation of the style, personality, and / or characteristics of the existing digital content. Content descriptors can be, for example, natural language descriptions of text, images, videos, etc.

[0079] The output of generative model 310 may be stored in digital content repository 140 and / or another repository in communication with digital content generation system 390. According to some examples, rather than receiving digital content from digital content repository 140 (which may be large in size, e.g., gigabytes), AACGS 301 receives the output of generative model 310, e.g., embeddings and / or content descriptions, which may be relatively small in size, e.g., kilobytes.

[0080] Although the generative model 310 Figure 3 301, but in some examples, generative model 310 can be an engine within AACGS 301. In such examples, AACGS 301 can be configured to generate features associated with digital content and use the generated features when generating the digital content (e.g., a video component).

[0081] During execution, the generative model 310 receives as input existing digital content associated with a content creator, entity, and / or product. The generative model 310 provides as output generated features associated with the existing digital content. The generated features can be, for example, embeddings associated with the existing digital content and / or natural language descriptions of the existing digital content.

[0082] For example, and as also referred to Figure 8 and Figure 9 As described, the generative model 310 may include one or more generative models, such as a language model, a base model, and / or a graphical model. The generative model 310 may implement one or more encoders and decoders for generating trained representations of input data and decoding these representations to generate features associated with existing digital content. These representations may be discrete or continuous representations of the input data, for example, represented as vectors. The encoder may include a transformer with a self-attention mechanism for encoding input data, which may be received by the model 310 as a series of tokens, frames, or other data units. The encoding layer of the model 310 may be fed into addition and normalization layers, and then further processed by a nonlinear model (such as a neural network).

[0083] The decoder of the generative model 310 can receive and process a representation of the input data to obtain an output corresponding to certain features of the input data. To generate features from text, the model 310 can encode the prompt using one or more trained text encoders. The model 310 can implement a diffusion-based model or other model technology for taking a text representation as input and generating a corresponding embedding or natural language description in response to the input. A diffusion model is a type of generative model that converts noise into samples from a learned data distribution. In general, any AI model technology for generating digital content from text prompts can be used to implement the generative model 310.

[0084] AACGS 301 can be configured to receive reasoning data 330 and / or training data 320 to identify a subset of material to be included in a video component, determine the characteristics and traits of an entity, and generate a video component. For example, AACGS 301 can receive reasoning data 330 and / or training data 320 as part of a call to an application programming interface (API) that exposes AACGS to one or more computing devices. Reasoning data 330 and / or training data 320 can also be provided to AACGS 301 via a storage medium (such as a remote storage device connected to one or more computing devices via a network). Reasoning data 330 and / or training data 320 can be further provided as input via a user interface on a client computing device coupled to AACGS 301. Reasoning data 330 can include material associated with the entity for which the video component creation tool is generating a new video component. For example, reasoning data 330 can be digital content stored in a digital content repository 140.

[0085] AACGS 301 can include one or more engines (also referred to herein as modules and / or models) that are configured to identify a subset of materials to be included in a video component, determine the characteristics and traits of an entity, and / or generate a video component. In this regard, AACGS 301 includes a representation engine 303, a feature engine 305, a trait engine 307, and a creation engine 309. The representation engine 303 can be trained to identify a subset of materials to be included in a video component. The feature engine 305 can be trained to determine the characteristics associated with an entity. The trait engine 307 can be trained to determine the traits of an entity. The creation engine 309 can be trained to generate a video component based on the material set of an entity.

[0086] Engines 303-309 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination thereof. Figure 3Shown that AACGS 301 has four engines, comprise expression engine 303, characteristic engine 305, idiosyncratic engine 307 and creation engine 309, but AACGS 301 can have the engine of any number.In addition, the function of engine described herein can be combined in one or more engines.For example, the function of characteristic engine 305 and idiosyncratic engine 307 (comprising respectively determining the characteristic associated with entity and determining the idiosyncratic of entity) can be processed by the single engine that is trained to carry out these two functions.Although engine 303-309 as shown in the figure is all positioned at single AACGS 301, these engines also can be realized in multiple AACGS.

[0087] In addition, engines 303-309 can work in tandem and / or in collaboration. For example, feature engine 305 and trait engine 307 can provide output to presentation engine 303 for selecting material to be included in a video component. The presentation engine can then provide the selected material to creation engine 309 to generate the video component.

[0088] The training data 320 can correspond to artificial intelligence (AI) or machine learning (ML) tasks for identifying a subset of material to be included in a video component, determining characteristics and traits of an entity, generating a video component, and other such tasks performed by the engines 303-309. The training data 320 can be split into a training set, a validation set, and / or a test set. An example of a training / validation / test split can be an 80 / 10 / 10 split, but any other split is possible. The training data for the representation engine 303 can include examples of material that has been previously selected and not selected for inclusion in a video component. The training data for the characteristic engine 305 can include material with and without entity characteristics. The training data for the trait engine 307 can include material with and without features associated with specific entity traits. The training data for generating a video component can include previously created video components and, in some cases, data defining where, when, and / or how to merge material into a previously created video component.

[0089] The training data 320 can take any form suitable for use in training the engine, according to one of a variety of different learning techniques. The learning techniques used to train the engine can include supervised learning, unsupervised learning, and semi-supervised learning techniques. For example, the training data 320 can include a plurality of training examples that can be received as input by the engine. When processing labeled training examples, the training examples can be labeled with the expected output of the engine. For example, and with reference to the training data used to determine which material to include in the generated video component, material that was previously selected for inclusion in the video component can be marked as selected, while material that was not selected for inclusion in the video component can be marked as unselected.

[0090] The labels and engine outputs can be evaluated by a loss function to determine an error, which can be back-propagated through the engine to update the engine's weights. For example, if the machine learning task is a classification task corresponding to determining characteristics of an entity, the training examples can be images labeled with one or more classifications that categorize the characteristics depicted in the provided material. As another example, supervised learning techniques can be applied to calculate the error between the output and the ground truth labels of the training examples processed by the engine. Various loss or error functions suitable for the type of task the engine is trained for can be utilized, such as cross entropy loss for classification tasks or mean squared error for regression tasks. The gradient of the error with respect to the different weights of the candidate engine on the candidate hardware can be calculated, for example, using a back-propagation algorithm, and the engine's weights can be updated. The engine can be trained until a stopping criterion is met, such as the number of iterations of training, a maximum time period, convergence, or when a minimum accuracy threshold is met.

[0091] Based on the inference data 330 and / or the training data 320, the AACGS 301 can be configured to output one or more results related to identifying a subset of material to be included in a video component, determining characteristics and traits of an entity, and / or generating a video component, which are generated as output data 325. As an example, the output data 325 can be any type of score, classification, or regression output based on the input data output by the engines 303-309. Accordingly, the AI ​​or machine learning task can be a scoring, classification, and / or regression task for predicting an output based on an input.

[0092] These AI or machine learning tasks can correspond to various different applications of processing images, videos, text, voice, or other types of data to identify a subset of materials to be included in a video component, determine the characteristics and traits of an entity, and / or generate a video component. Output data 325 may include instructions associated with these tasks. For example, the creation engine 309 can be configured to provide the output data 325 as a set of computer-readable instructions, such as one or more computer programs, which can be executed by a computing device to generate a video component with selected materials and features determined by the creation engine 309. The computer program can be written in any type of programming language and according to any programming paradigm (e.g., declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative). The computer program can be written to perform one or more different functions and run in a computing environment (e.g., on a physical device, on a virtual machine, or across multiple devices). The computer program can also implement the functions described herein, such as those performed by a system, engine, module, or model. AACGS 301 can further be configured to forward the output data to one or more other devices that are configured to convert the output data into an executable program written in a computer programming language. AACGS 301 can also be configured to send the output data to a storage device for storage and later retrieval. Additionally or alternatively, a material creation tool can be configured to receive the output of AACGS 301 for further processing and / or implementation.

[0093] AACGS 301 is an AI model that is trained to receive existing digital content stored in the digital content repository 140 and / or output from the generative model 310 and generate digital content (including a video component). For example, during execution, AACGS 301 receives as input a prompt for generating digital content, existing digital content, and / or features associated with the existing digital content generated by the generative model 310. AACGS 301 provides as output the generated digital content having qualities and / or characteristics corresponding to an entity or product identified by AACGS 301.

[0094] The AACGS 301 can generate digital content items 370A, 370B, and 370C, e.g., output data 325. The digital content items 370A-C can be in a video format, e.g., a video component. The digital content items 370A-370C can be generated by processing existing digital content stored in the digital content repository 140 via engines 303-309. For example, the existing digital content can be processed by the representation engine 303 to identify a subset of materials to be included in the digital content item, processed by the characteristics engine 305 to determine characteristics associated with the content creator, processed by the trait engine 307 to determine the traits of the content creator, and / or processed by the creation engine 309 to generate a digital content item based on the content creator's collection of materials.

[0095] Digital content items 370A-370C may be generated by processing existing digital components through engines 303-309 of AACGS 301. When processed by AACGS 301, AACGS 301 generates digital content items 370A-C for targeting user computing devices 180A-C, respectively.

[0096] According to some examples, the traits of the content creator may correspond to the style of the content creator. For example, the traits or style of the content creator may be determined based on embeddings associated with the activity creator's existing digital content. In such an example, the AACGS 301 may include Figure 1 The recognition engine 175, embedding acquirer 105 and embedding repository 115 of the digital content generation system 190 of the activity creator are used. For example, embeddings of existing digital content for the activity creator are determined. The embedding acquirer 105 can use the embedding tags created by the embedding acquirer to query the embedding repository 115 to identify embeddings with similar tags. The embeddings associated with the existing digital content are used to determine the trait (e.g., style) vector associated with the activity creator. The trait vector for a given activity creator can be determined based on the average of at least some of the differences between some of the embeddings of the activity creator's existing digital content. In some examples, the trait vector can be based on the average of all differences between the embeddings. When generating digital content, the trait vector is provided as input to the AACGS 301 and / or the creation engine 309.

[0097] A digital component creation tool (e.g., AACGS 301) can generate new digital components, such as new video components, using existing digital content stored in the digital content repository 140. Existing digital content can include assets such as text, logos, images, videos, and other such multimedia. Figure 4As shown, the pizza business's assets 404 include text 403, logo 405, and image 407. It should be understood that assets 404 are non-limiting examples and may include other types of assets, including audio assets. Further, any number of assets may be included. In this regard, although Figure 4 Three text assets, two logos, and four images are shown, but the assets may include fewer or more text, logos, images, audio, etc. assets.

[0098] In operation, AACGS can analyze the material to identify a subset of the material that can be included in the new video component. Figure 4 As shown, the digital content generation system 390 may identify assets 404 to identify a subset of assets to include in the new video component 440. The assets may be identified from existing digital content stored in the digital content repository 140.

[0099] The asset subset may be those assets determined to meet certain criteria, such as being representative of a business, company, individual, non-profit organization, or other such entity for which the video component is being generated. For example, the video component creation tool may determine, from asset 404 corresponding to a pizza business entity, an asset subset including descriptions 442, 443, logos 445, 446, and images 447, 448 that may be included in the generated new video component.

[0100] The digital content generation system 390 can combine part or all of the material subsets into new digital content. The new digital content can be, for example, images, text, videos, etc. For example, Figure 4 As shown, a subset of materials including description 443, logo 446, and image 447 is integrated into a frame 434 of digital content (e.g., video component 440). In this regard, video component 440 includes a horizontally oriented video that includes a frame 434 having a logo 446 superimposed on an image 447. Although video component 440 is horizontally oriented, a video component creation tool can generate video components of any format or orientation. For example, a video component creation tool can generate video components of vertical orientation or different aspect ratios. In some cases, a video component creation tool can create the same video component of different orientations and / or aspect ratios to be deployed on different devices and media, such as televisions, mobile devices, web, etc.

[0101] Although not shown, digital content generation system 390 may generate a voice-over of description 443 that plays along with video component 440. Alternatively or in addition, AACGS may superimpose description 443 itself over an image (such as one of images 407), a logo (such as logos 445 or 446), or another image or video (such as a stock video or image available to the video component creation tool).

[0102] In some cases, the digital content generation system 390 can analyze the footage of an entity to determine characteristics of the entity, such as the entity's colors and / or fonts. In addition, the video component creation tool can analyze the footage to determine the entity's "character," such as sincere, authentic, professional, inspiring, sophisticated, rugged, etc. The entity's characteristics can correspond to the entity's style. These characteristics and traits can be incorporated into the generated video component for the entity.

[0103] For example, the digital content generation system 390 may analyze the footage 404 and determine that the pizza business has "authentic qualities" and fonts such as those used in the logo 445. The video component creation tool may generate one or more frames in the video based on the determined qualities and characteristics. For example, Figure 5 Frame 432 of a video component 440 created by a video component creation tool for a pizza business is shown. Frame 432 includes text 443 superimposed on an image 448. Text 443 has been formatted in the font of logo 445, which is recognizable as a font associated with a pizza business. The video component creation tool included text 443 because it represents the "authentic qualities" of the pizza business.

[0104] Although the two frames 434, 432 of the video component 440 illustrate a subset of assets selected from the assets 404, a video component may include any number of assets. The order of assets within a video component and the length of time each asset is displayed or otherwise played within the video component may also be determined by the video component creation tool and / or based on input from a user.

[0105] The assets may be stored in one or more databases, or may be uploaded when the video component creation tool generates a new video component. For example, the video component creation tool may retrieve assets from web resources such as web pages, app store landing pages, etc., associated with the entity for which the new video component is being generated. Additionally or alternatively, a user may upload assets to the video component creation tool or a location accessible to the video component creation tool, such as a database.

[0106] As described herein, material can include text, logos, images, videos, and the like, as well as other such multimedia. Although material is described as a piece of multimedia, such as text or a logo, a piece of multimedia can include more than one material. For example, a piece of multimedia (such as a logo) can include material that contains both the logo itself and the logo text. In another example, video material can include any number of image materials, logo materials within image materials, audio materials within video materials, and text within video materials. In some cases, material contained within other materials can be identified, stored, and / or otherwise retrieved by artificial intelligence models trained to recognize specific material types, visual and text recognition algorithms, and the like.

[0107] The digital content generation system 390 can identify a subset of material to be included in a video component, determine the characteristics and qualities of the entity, and generate the video component using machine learning models. These machine learning models can be implemented in a system such as AACGS 301.

[0108] Example Method

[0109] Figure 6 A flow chart depicts an example process 600 for generating digital content according to aspects of the present disclosure. The example process may be implemented in a system of one or more processors in one or more locations, such as Figure 1 The following operations do not have to be performed in the exact order described below. Instead, the various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0110] In block 610, the system receives existing digital content associated with a campaign creator. The system may include, for example, a digital content generation system such as a reference Figure 1 The digital content generation system 190 described herein can be obtained from, for example, reference Figure 1 The described campaign management platform 150 receives one or more existing digital content associated with a campaign creator.

[0111] In block 620, the system receives a natural language prompt for a generative model that is trained to generate content based on the natural language prompt. The natural language prompt includes changes to be made to existing digital content and group information associated with the activity.

[0112] In block 630, the system determines style embeddings associated with the existing digital content. To determine the style embeddings, the existing digital content is clustered based on visual similarities within the existing digital content. Visual similarities can include, for example, color schemes, subject matter, events, styles, etc. In some examples, visual similarities can be based on themes within the existing digital content, such as humans, animals, nature, etc. Once clustered, a style embedding can be determined for the cluster. By determining the style embedding for the cluster, computing resources, such as processing power and network overhead, can be saved compared to style embeddings for all existing digital content.

[0113] In block 640, the system determines a style vector for the activity creator based on the style embeddings. Determining the style vector includes determining an average of the differences between the style embeddings. The average of the differences between each of the style embeddings corresponds to the style vector.

[0114] In block 650, the system processes the natural language prompt and style vector using a generative model to generate digital content. The generated digital content is based on existing digital content. For example, when processing the natural language prompt and style vector using a generative model, the generative model is configured to modify the existing digital content based on the natural language prompt and style vector. In some examples, the generative model is configured to generate new digital content based on the natural language prompt and style vector.

[0115] Based on some examples, the system can generate seed information based on natural language prompts. The system can include a seed generation engine such as Figure 1 The seed generation engine 195 in the system can be used to identify the subject matter. For example, the system can include an identification engine such as Figure 1 The recognition engine 175 in the image processing unit can identify the subject matter based on the seed information. The subject matter can be used to identify embeddings within a subject matter threshold distance. For example, the system can include an image embedding acquirer such as Figure 1 The image embedding acquirer 105 in

[0045] may include embeddings related to the subject matter. The embeddings are processed by a generative model in combination with natural language cues and style vectors to generate digital content.

[0116] Figure 7 A flow chart depicts an example process 700 for generating digital content according to aspects of the present disclosure. The example process may be implemented in a system of one or more processors in one or more locations, such as Figure 3 The following operations do not have to be performed in the exact order described below. Instead, the various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0117] In block 710, the system receives existing digital content associated with an entity, creator, or product. The existing digital content includes a collection of assets. The entity may be, for example, a content creator. In some examples, the system receives existing digital content associated with the entity, content creator, and / or product. The system may identify a collection of assets based on the received digital content. In some examples, the digital content associated with the entity, content creator, and / or product corresponds to a collection of assets. Assets include text, logos, images, audio, and / or video.

[0118] In block 720 , the system receives a prompt for a generative model to generate a video component according to the prompt.

[0119] In block 730, the system determines characteristics associated with the entity. The determined characteristics may include, for example, colors, fonts, styles, etc. associated with the entity, content creator, and / or product. The determined characteristics are determined by a characteristic engine. The characteristic engine is trained to determine characteristics, such as colors, fonts, etc., associated with the entity, content creator, and / or product.

[0120] In block 740, the system processes the prompt and the determined characteristics through a generative model to generate digital content, such as a video component, using a subset of the asset collection. The asset subset can be determined by a presentation engine. The presentation engine is trained to recognize the asset subset. The digital content (e.g., a video component) is generated by a creation engine. The creation engine is trained to generate digital content, such as a video component, based on the asset collection of an entity, content creator, and / or product.

[0121] In some examples, the system determines traits associated with an entity, content creator, and / or product. The traits may be determined by a trait engine. The trait engine is trained to determine the traits of an entity, content creator, and / or product. In some examples, the traits of an entity, content creator, and / or product correspond to the style of the entity, content creator, and / or product. The style of the entity, content creator, and / or product may be determined using embeddings associated with existing digital content of the entity, content creator, and / or product, such as reference Figure 1 described.

[0122] In some examples, the system determines features associated with existing digital content. These features include at least one of an embedding associated with the existing digital content or a natural language description of the existing digital content. These features, along with the prompt and the determined characteristics, can be processed by a generative model when generating a video component.

[0123] Implementations of the present technology may each include, but are not limited to, the following. These features may be individual or combined with one or more other features described herein. In some examples, combinations of the following features are included:

[0124] (1) A method for delivering digital content, the method comprising: receiving, by one or more processors, existing digital content associated with an activity creator; receiving, by the one or more processors, a natural language prompt for a generative model, the generative model being trained to generate content based on the natural language prompt; determining, by the one or more processors, a style embedding associated with the existing digital content; determining, by the one or more processors, a style vector for the activity creator based on the style embedding; and processing, by the one or more processors, the natural language prompt and the style vector through the generative model to generate digital content, wherein the generated digital content is based on the existing digital content.

[0125] (2) The method of (1), wherein, when determining the style vector, the method further comprises determining, by the one or more processors, an average of the differences between the style embeddings based on the style embeddings, wherein the average of the differences between the style embeddings corresponds to the style vector.

[0126] (3) The method according to (1) or (2), wherein, when determining the style embedding, the method further includes: clustering the existing digital content based on the visual similarity by the one or more processors; and determining the style embedding of the clustered existing digital content by the one or more processors.

[0127] (4) The method according to any one of (1) to (3), further comprising: generating seed information based on the natural language prompt by the one or more processors; determining a subject matter based on the seed information by the one or more processors; and

[0128] Embeddings within a threshold distance of the subject matter are identified, by the one or more processors, based on the subject matter.

[0129] (5) The method of (4), wherein the embedding is processed by the generative model to generate the digital content.

[0130] (6) A method according to any one of (1) to (5), wherein, when the natural language prompt and the style vector are processed by the generative model to generate the digital content, the generative model is configured to modify the existing digital content based on the natural language prompt and the style vector; or to generate new digital content based on the natural language prompt and the style vector.

[0131] (7) The method of any one of (1) to (6), wherein the natural language prompt includes changes to be made to the existing digital content and group information associated with the activity.

[0132] (8) A system comprising one or more processors configured to execute the method as described in any one of (1) to (7).

[0133] (9) One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, are operable to cause the processors to perform the operations of any one of the methods of (1) to (7).

[0134] (10) A method for generating a video component, the method comprising: receiving, by one or more processors, existing digital content associated with an entity or product, wherein the existing digital content includes a collection of assets; receiving, by the one or more processors, prompts for a generative model to generate the video component based on the prompts; determining, by the one or more processors, characteristics associated with the entity or product; and processing, by the one or more processors, the prompts and the determined characteristics through the generative model to generate the video component using a subset of the collection of assets.

[0135] (11) The method according to (10), wherein the material collection includes one or more texts, logos, images, audio or video.

[0136] (12) The method of (10) or (11), wherein the determined characteristics include one or more of a color or a font associated with the entity or product.

[0137] (13) The method of any one of (10) to (12), wherein the subset of materials is determined by a presentation engine.

[0138] (14) The method of any one of (10) to (13), wherein the generative model comprises a property engine configured to determine the property associated with the entity or product.

[0139] (15) The method according to any one of (10) to (14), further comprising: determining a characteristic associated with the entity or product, wherein: the characteristic associated with the entity or product is based on an embedding associated with the existing digital content, and generating the video component includes using the determined characteristic.

[0140] (16) The method of (15), wherein the generative model comprises a trait engine configured to determine the trait of the entity or product.

[0141] (17) The method of any one of (10) to (16), further comprising determining, by the one or more processors, characteristics associated with the existing digital content.

[0142] (18) The method of (17), wherein the processing further comprises processing the determined features through the generative model to generate the video component.

[0143] (19) The method of (17), wherein the feature comprises at least one of an embedding associated with the existing digital content or a natural language description of the existing digital content.

[0144] (20) A system comprising one or more processors configured to perform the method as described in any one of (10) to (19).

[0145] (21) One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, are operable to cause the processors to perform the operations of any one of the methods of (10) to (19).

[0146] Sample computing environment

[0147] Figure 8 8 is a block diagram illustrating one or more models 810, such as for deployment in a data center 820 housing one or more hardware accelerators 830, on which the deployed models are to be executed to generate digital content based on generated and stored prompts, in accordance with aspects of the present disclosure. The models 810 may include, for example, the generative models 110, 210, 310 and / or the AACGS 301. The hardware accelerator 830 may be any type of processor, such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC), such as a tensor processing unit (TPU).

[0148] In some implementations, the technology disclosed herein uses artificial intelligence to generate digital content from prompts generated and stored according to various aspects of the present disclosure. Artificial intelligence (AI) is a branch of computer science that focuses on creating models that can perform tasks with little to no human intervention. Artificial intelligence systems can utilize, for example, machine learning, natural language processing, and computer vision. Machine learning and its subsets (such as deep learning) focus on developing models that can infer outputs based on data. Outputs can include, for example, predictions and / or classifications. Natural language processing focuses on analyzing and generating human language. Computer vision focuses on analyzing and interpreting images and videos. Artificial intelligence systems can include generative models that generate new content, such as images, videos, text, audio, and / or other content, in response to input prompts and / or based on other information.

[0149] Examples of machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models can utilize attention mechanisms, such as self-attention. For example, some machine learning models can include multi-head self-attention models (e.g., transformer models).

[0150] The model can be trained using various training or learning techniques. Training can implement supervised learning, unsupervised learning, reinforcement learning, etc. Training can use techniques such as, for example, back propagation of errors. For example, a loss function can be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters over multiple training iterations. A variety of generalization techniques (e.g., weight decay, dropout) can be used to improve the generalization ability of the model being trained.

[0151] Models can be pre-trained before domain-specific alignment. For example, a model can be pre-trained on a general training data corpus and fine-tuned on a more targeted training data corpus. Models can be aligned using prompts designed to elicit domain-specific outputs. Prompts can be designed to include learned prompt values ​​(e.g., soft prompts). The trained model can be validated using input data other than the training data before its use, and can be further updated or refined during its use based on additional feedback / input.

[0152] The architecture of a model can refer to characteristics that define the model, such as the characteristics of each layer of the model, how each layer processes input, or how the layers interact. For example, the model can be a convolutional neural network that includes a convolutional layer that receives input data, followed by a pooling layer, followed by a fully connected layer that generates a result. The architecture of the model can also define the type of operations performed within each layer. For example, the architecture of a convolutional neural network can define the use of a rectified linear unit (ReLU) activation function in the fully connected layers of the network. Other example architectures can include generative models, such as language models, base models, and / or graphical models. One or more model architectures can be generated that can output results associated with generating digital content based on prompts generated or stored by the digital content generation system 190, 290, 390.

[0153] As another example, with respect to reinforcement learning, situations encountered by an agent (e.g., a model, computing device, system, robot, etc.) are mapped to actions taken by the agent in those situations to maximize the reward or value of its actions. The agent can interact with the environment through its actions. At any given time or point where the agent can take an action, the environment can be represented as a state. The state can include any information or features about the environment that the agent can know. The value of a state is a measure of the total amount of reward that the agent can obtain from the current state and future states that can be reached from the current state. A value function can be defined or estimated to calculate, predict, or estimate the value of a state. Techniques for training machine learning models via reinforcement learning can focus on estimating or learning value functions to accurately predict values ​​under different environmental states.

[0154] The model or policy can be modified or updated until a stopping criterion is met, such as the number of training iterations, a maximum time period, convergence of estimated rewards or values ​​between actions, or when a minimum threshold is met. The model can be a composite of multiple models, or a composite of components of a processing or training pipeline. In some examples, the models or components are trained individually, while in other examples, the models or components are trained end-to-end.

[0155] Figure 95 is a block diagram of an example computing environment 500 for implementing the digital content generation systems 190, 290, 390. The digital content generation systems 190, 290, 390 can be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 515. The user computing device 512 and the server computing device 515 can be communicatively coupled to one or more storage devices 530 via a network 560. The storage devices 530 can be a combination of volatile and non-volatile memory and can be located in the same or a different physical location as the computing devices 512, 515. For example, the storage devices 530 can include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, an optical storage device, a memory card, ROM, RAM, a DVD, a CD-ROM, writable memory, and read-only memory.

[0156] Aspects of the present disclosure may be implemented in a computing system that includes back-end components (e.g., data servers), middleware components (e.g., application servers), or front-end components (e.g., user computing devices 512 having user interfaces, web browsers, or apps), or any combination thereof. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Examples of communications networks include local area networks (LANs) and wide area networks (WANs), such as the Internet. Data center 820 may also communicate with user computing devices 512 and server computing devices 515.

[0157] The computing system may include a client (e.g., user computing device 512) and a server (e.g., server computing device 515). The client and server may be remote from each other and interact via a communication network. The relationship of client and server arises due to computer programs running on the respective computers and having a client-server relationship to each other. For example, a server may transmit data (e.g., an HTML page) to a client device (e.g., for the purpose of displaying data to a user interacting with the client device and receiving user input from the user). Data generated at the client device, such as results of user interactions, may be received at the server from the client device.

[0158] The server computing device 515 may include one or more processors 513 and memory 514. The memory 514 may store information accessible to the processor 513, including instructions 521 that can be executed by the processor 513. The memory 514 may also include data 523 that can be retrieved, manipulated, or stored by the processor 513. The memory 514 may be a non-transitory computer-readable medium capable of storing information accessible to the processor 513, such as volatile memory and non-volatile memory. The processor 513 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application-specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0159] Instructions 521 may include one or more instructions that, when executed by processor 513, cause one or more processors to perform the actions defined by the instructions. Instructions 521 may be stored in an object code format for direct processing by processor 513, or in other formats, including interpretable scripts or independent source code modules that can be interpreted or pre-compiled as needed. Instructions 521 may include instructions for implementing digital content generation system 190 consistent with various aspects of the present disclosure. Digital content generation system 190 may be executed using processor 513 and / or using other processors located remotely from server computing device 515.

[0160] Processor 513 can retrieve, store or modify data 523 according to instructions 521. Data 523 can be stored in a computer register, as a table with multiple different fields and records, or as a JSON, YAML, proto or XML document in a relational or non-relational database. Data 523 can also be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII or Unicode. In addition, data 523 can include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations), or information used by functions to calculate related data.

[0161] User computing device 512 may also be configured similarly to server computing device 515, having one or more processors 516, memory 517, instructions 518, and data 519. For example, user computing device 512 may be a mobile device, a laptop computer, a desktop computer, a gaming console, etc. User computing device 512 may also include user output 526 and user input 524. User input 524 may include any suitable mechanism or technology for receiving input from a user, including acoustic input; visual input; tactile input, including touch motions or gestures, kinetic motions or gestures, or orientation motions or gestures; auditory input; voice input; etc. Example devices for user input 524 may include a keyboard, a mouse or other pointing device, a mechanical actuator, a soft actuator, a touch screen, a microphone, and sensors.

[0162] The server computing device 515 may be configured to transmit data to the user computing device 512, and the user computing device 512 may be configured to display at least a portion of the received data on a display implemented as part of the user output 526. The user output 526 may also be used to display an interface between the user computing device 512 and the server computing device 515. The user output 526 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces, or other tactile feedback that provides non-visual and non-auditory information to a platform user of the user computing device 512.

[0163] although Figure 9 The processors 513, 516 and memories 514, 517 are shown to be located within the computing devices 515, 512, but the components described in this specification (including the processors 513, 516 and memories 514, 517) may include multiple processors and memories that can be run in different physical locations rather than within the same computing device. For example, some of the instructions 521, 518 and data 523, 519 may be stored on a removable SD card, while other instructions may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in a location that is physically away from the processors 513, 516 but still accessible to these processors. Similarly, the processors 513, 516 may include a collection of processors that can perform concurrent and / or sequential operations. The computing devices 515, 512 may each include one or more internal clocks that provide timing information, which may be used for time measurement of operations and programs run by the computing devices 515, 512.

[0164] The server computing device 515 can be configured to receive requests for processing data from the user computing device 512. For example, the environment 500 can be part of a computing platform that is configured to provide various services to users through various user interfaces and / or APIs that expose platform services. One or more of the services can be a machine learning framework or a set of tools for training or executing generative or other machine learning models based on specified tasks and training data.

[0165] Devices 512 and 515 are capable of communicating directly and indirectly via network 560. Devices 515 and 512 may be configured with listening sockets that can accept initiated connections for sending and receiving information. Network 560 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, and a private network using one or more company-specific communication protocols. Network 560 may support a variety of short-range and long-range connections. Short-range and long-range connections may be made over different bandwidths, such as 2.402 GHz to 2.480 GHz (typically associated with the Bluetooth® standard), 2.4 GHz and 5 GHz (typically associated with the Wi-Fi® communication protocol), or using various communication standards, such as the LTE® standard for wireless broadband communication. Additionally or alternatively, network 560 may also support wired connections between devices 512 and 515, including over various types of Ethernet connections.

[0166] Although Figure 9 5. Although a single server computing device 515, user computing device 512, and data center 820 are shown in FIG, it should be understood that various aspects of the present disclosure can be implemented based on a variety of different configurations and numbers of computing devices, including sequential processing paradigms or parallel processing paradigms, or implemented on a distributed network of multiple devices. In some implementations, various aspects of the present disclosure can be performed on a single device or any combination thereof.

[0167] Aspects of the present disclosure may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, and / or in computer hardware (such as the structures disclosed herein, their structural equivalents, or combinations thereof). Aspects of the present disclosure may further be implemented as one or more computer programs, such as one or more engines or modules of computer program instructions, encoded on one or more tangible, non-transitory computer storage media, for execution by one or more data processing devices or to control the operation of one or more data processing devices.

[0168] A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination thereof. Computer program instructions may be encoded on an artificially generated propagated signal (such as a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer program may, but need not, correspond to a file in a file system. A computer program may be stored as part of a file that holds other programs or data (e.g., one or more scripts), in a single file, or in multiple coordinated files (e.g., files that store one or more engines, modules, subroutines, or code portions).

[0169] The term "configured" is used herein in conjunction with both system and computer program components. With respect to a system of one or more computers configured to perform specific operations or actions, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that causes the system to perform those operations or actions. With respect to one or more computer programs configured to perform those operations or actions, it is meant that the one or more programs include instructions that, when executed by one or more data processing devices, cause the devices to perform those operations or actions.

[0170] The term "data processing equipment" refers to data processing hardware and covers a variety of devices, apparatuses, and machines for processing data, including programmable processors, computers, or combinations thereof. A data processing equipment may include specialized logic circuitry, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), such as a tensor processing unit (TPU). A data processing equipment may include code that creates an execution environment for a computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination thereof.

[0171] The data processing device may include a dedicated hardware accelerator unit for implementing the machine learning model to handle the general and computationally intensive parts of machine learning training or production (such as inference or workloads). The machine learning model may be implemented and deployed using one or more machine learning frameworks (such as static or dynamic computational graph frameworks).

[0172] The term "computer program" refers to a program, software, software application, app, module, software module, script, or code. A computer program may be written in any form of programming language, including compiled, interpreted, declarative, or procedural languages, or a combination thereof. A computer program may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may correspond to a file in a file system and may be stored in a portion of a file that holds other programs or data (such as one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (such as files that store one or more modules, subroutines, or code portions). A computer program may be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0173] The term "database" refers to any collection of data. The data can be unstructured or structured in any manner. The data can be stored on one or more storage devices in one or more locations. For example, an index database may include multiple data collections, each of which can be organized and accessed differently.

[0174] The term "engine" may refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. An engine may be implemented as one or more software modules or components, or may be installed on one or more computers in one or more locations. A particular engine may have one or more processors or computing devices dedicated to it, or multiple engines may be installed and run on the same processor or computing device. In some examples, an engine may be implemented as a specially configured circuit, while in other examples, an engine may be implemented as a combination of software and hardware.

[0175] The processes and logic flows described herein can be performed by one or more computers that execute one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows can also be performed by a dedicated logic circuit system, or by a combination of a dedicated logic circuit system and one or more computers. Although operations are depicted in the drawings and described in the claims in a specific order, this should not be understood as requiring such operations to be performed in the specific order shown or in a continuous order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of the various system modules and components in the embodiments described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can be integrated together in one or more software-based or hardware-based devices or computer-readable media.

[0176] A computer or dedicated logic circuit system that executes one or more computer programs may include a central processing unit (including a general-purpose or dedicated microprocessor) for running or executing instructions and one or more memory devices for storing instructions and data. The central processing unit can receive instructions and data from one or more memory devices (such as read-only memory, random access memory, or a combination thereof) and can run or execute these instructions. The computer or dedicated logic circuit system may also include or be operably coupled to one or more storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, for receiving data from or transferring data to one or more storage devices. The computer or dedicated logic circuit system may be embedded in another device, such as a mobile phone, desktop computer, personal digital assistant (PDA), mobile audio or video player, game console, tablet computer, virtual reality (VR) or augmented reality (AR) device, global positioning system (GPS), or portable storage device, such as a universal serial bus (USB) flash drive, as an example. Examples of computers or dedicated logic circuit systems may include user computing device 512, server computing device 515, or hardware accelerator 830.

[0177] Computer-readable media suitable for storing one or more computer programs may include any form of volatile or non-volatile memory, medium, or storage device. Examples include semiconductor memory devices (e.g., EPROM, EEPROM, or flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks, DVD-ROM disks, or combinations thereof.

[0178] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the above-described features may be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be considered as illustrative rather than restrictive of the subject matter defined by the claims. In addition, the provisions of the examples described herein and the phrases "such as," "including," etc. should not be interpreted as limiting the subject matter of the claims to specific examples; rather, these examples are intended to illustrate only one of many possible examples. Further, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A method for generating a video component, the method comprising: Receiving, by one or more processors, existing digital content associated with an entity or product, wherein the existing digital content includes a collection of assets; receiving, by the one or more processors, a prompt for a generative model to generate the video component according to the prompt; determining, by the one or more processors, characteristics associated with the entity or product; and The prompts and the determined characteristics are processed by the one or more processors through the generative model to generate the video component using a subset of the collection of assets. 2 . The method according to claim 1 , wherein the collection of materials comprises one or more texts, logos, images, audio or video. 3 . The method of claim 1 , wherein the determined characteristics include one or more of a color or a font associated with the entity or product. The method of claim 1 , wherein the subset of assets is determined by a presentation engine. 5 . The method of claim 1 , wherein the generative model comprises a property engine configured to determine the property associated with the entity or product.

6. The method according to claim 1, further comprising: Identify the qualities associated with the entity or product, in: The trait associated with the entity or product is based on an embed associated with the existing digital content, and Generating the video component includes using the determined traits. The method of claim 6 , wherein the generative model comprises a trait engine configured to determine the trait of the entity or product.

8. The method of claim 1, further comprising determining, by the one or more processors, characteristics associated with the existing digital content.

9. The method of claim 8, wherein the processing further comprises processing the determined features through the generative model to generate the video component.

10. The method of claim 8, wherein the feature comprises at least one of an embedding associated with the existing digital content or a natural language description of the existing digital content.

11. A system for generating a video component, the system comprising: One or more processors configured to: Receiving existing digital content associated with an entity or product, wherein the existing digital content includes a collection of assets; receiving a prompt for a generative model to generate the video component according to the prompt; determining characteristics associated with the entity or product; and The prompts and the determined characteristics are processed by the generative model to generate the video component using a subset of the collection of assets.

12. The system of claim 11, wherein the collection of assets comprises one or more of text, logo, image, audio, or video.

13. The system of claim 11, wherein the determined characteristics include one or more of a color or a font associated with the entity or product. The system of claim 11 , wherein the subset of assets is determined by a presentation engine.

15. The system of claim 11, wherein the generative model comprises a property engine configured to determine the property associated with the entity or product.

16. The system of claim 11, wherein the one or more processors are further configured to: Identify the qualities associated with the entity or product, in: The trait associated with the entity or product is based on an embed associated with the existing digital content, and Generating the video component includes using the determined traits.

17. The system of claim 16, wherein the generative model comprises a trait engine configured to determine the trait of the entity or product.

18. The system of claim 11, wherein the one or more processors are further configured to determine a characteristic associated with the existing digital content, wherein the characteristic comprises at least one of an embedding associated with the existing digital content or a natural language description of the existing digital content.

19. The system of claim 18, wherein the processing further comprises processing the determined features through the generative model to generate the video component.

20. One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: Receiving existing digital content associated with an entity or product, wherein the existing digital content includes a collection of assets; receiving a prompt for a generative model to generate a video component based on the prompt; determining characteristics associated with the entity or product; and The prompts and the determined characteristics are processed by the generative model to generate the video component using a subset of the collection of assets.

Images