Scalable architecture for automatic generation of content distribution image

Abstract

To provide methods for automatic generation of content distribution images that include receiving user input corresponding to a content-distribution operation.SOLUTION: A method includes: parsing a user input to identify keywords; identifying image data corresponding to the keywords; executing image-processing operation on the image data; and executing a generative adversarial network on the processed image data, the execution including executing a first neural network on the processed-image data to generate first images that correspond to the keywords, the first images being generated based on a likelihood that each image of the first images would not be detected as having been generated by the first neural network. A user interface can display the first images with second images that include images that were previously part of content-distribution operations or images that were designated by an entity as being available for content-distribution operations.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is the benefit of U.S. Provisional Application No. 62 / 900,400, filed September 13, 2019, and entitled "SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES." The patent claims benefit of U.S. Non-Provisional Application No. 17 / 017,486, filed September 10, 2020, entitled "SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES." No. 60 / 019,597, filed on Dec. 1, 2003, the entire contents of which are incorporated herein by reference for all purposes.

[0002] Field The present invention relates generally to the automated generation of content for content distribution operations, and more particularly to a scalable architecture that utilizes neural networks for content distribution operations. [Background technology]

[0003] background Generating content for a content distribution operation can be resource-intensive and time-consuming. Often, the goal of a content distribution operation is to deliver specific content to a specific set of devices within a specific, relevant time interval. A company may employ a team of photographers, designers, artists, etc. to capture images for the operation, modify the images based on one or more criteria, convert the modified images into a format suitable for transmission or display, and send the formatted images to a specific set of devices. A company often spends significant resources over weeks designing a content distribution. Moreover, by the time the distribution operation begins, so much time may have passed that the generated content is no longer relevant to the specific set of devices that will receive the content. Therefore, a system and method are needed to reduce the resource requirements of content distribution operations. Summary of the Invention [Means for solving the problem]

[0004] overview A method for automated generation of images for a content delivery operation is disclosed, the method comprising: receiving user input corresponding to the content delivery operation; parsing the user input to identify one or more keywords associated with the content delivery operation; receiving image data from one or more databases corresponding to the one or more keywords associated with the content delivery operation; and performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of specific locations within the image that correspond to the one or more keywords. The method further comprises running a generative adversarial network on the processed image data to generate one or more images for the content delivery operation, the running the generative adversarial network comprising running a first neural network on the processed image data, the first neural network generating a first set of images that correspond to the one or more keywords, the first set of images being generated based at least in part on a likelihood that each image in the first set of images would not have been detected as having been generated by the first neural network. Performing the content delivery operation further includes displaying the first set of images via a first user interface along with a second set of images, the second set of images including images that were previously part of one or more content delivery operations or images designated as available for the content delivery operation by an entity associated with the content delivery operation.

[0005] Another aspect of the present disclosure includes a system comprising one or more processors and a non-transitory computer-readable medium containing instructions that, when executed by the one or more processors, cause the one or more processors to perform the above-described method.

[0006] Another aspect of the present disclosure includes a non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described method.

[0007] Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the present disclosure.

[0008] The present disclosure is described in connection with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of a scalable architecture for generating content delivery images according to at least one aspect of the present disclosure. [Figure 2] FIG. 1 is a block diagram illustrating a process flow for generating a content delivery image according to at least one aspect of the present disclosure. [Figure 3] FIG. 1 illustrates an example of a graphical user interface associated with generating a content delivery image in accordance with at least one aspect of the present disclosure. [Figure 4] FIG. 1 illustrates a flowchart for generating a content distribution operation in accordance with at least one aspect of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0010] Detailed Description In the accompanying drawings, similar components and / or features may have the same reference level. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes between the similar components. When only a first reference label is used herein, the description is applicable to any one of the similar components having the same first reference label, regardless of the second reference label.

[0011] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an enabling description for implementing the preferred exemplary embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0012] A scalable architecture for generating content and content delivery actions using artificial intelligence may be defined. A generative adversarial network may be trained at runtime to generate specific images for a content delivery action. For example, user input may be parsed to determine one or more keywords that correspond to the intended context of the content delivery action. The one or more keywords may, for example, be associated with one or more sub-sub-sub-sub-images of the content. It may represent a subject (such as a person), a characteristic of the subject (such as a person's race, gender, height, weight, etc.), one or more objects represented in the content, or a setting (such as a location, etc.).

[0013] In some examples, a generative adversarial network may not be trained to generate images for every possible keyword or keyword combination. In such cases, one or more keywords may be used to generate image data (e.g., a subset of keywords) that may be used to train the generator neural network of the generative adversarial network at runtime. For example, the adversarial network may generate images corresponding to any keyword or combination of keywords on demand.

[0014] Generate a discriminator neural network for the generative adversarial network. The generator neural network may be trained after the generator neural network. Training the classifier neural network may involve generating images using the generator neural network and passing those images, along with other images, such as images from the image data, to the classifier neural network. The classifier neural network may generate a prediction for each image indicating whether or not the image was generated by the generator neural network. The predictions can be fed back to the generator neural network. The generator neural network uses the predictions from the classifier neural network to improve image generation, producing images that are more likely to be labeled as real by the classifier neural network. Thus, the goal of the generator neural network is to trick the classifier neural network into predicting a false negative (e.g., predicting that an image was not generated by the generator neural network, when in fact the image was generated by the generator neural network).

[0015] After training the generator neural network and the discriminator neural network, the generator neural network may be run using one or more keywords to generate images for a content distribution campaign. In some examples, the discriminator neural network may continue to generate predictions that can be used by the generator neural network to further refine image generation. In such cases, the predictions may be further used to filter images generated by the generator neural network, for example, by removing images that the discriminator correctly predicts to have been generated by the generator neural network (e.g., images that do not look like real images). The generated images (or remaining filtered images) may be transmitted as part of a content distribution campaign to one or more client devices (e.g., devices associated with the entity that requested the generated images), a content distribution server (e.g., a web host), or one or more user devices (e.g., devices associated with users who are targeted by one or more client devices).

[0016] In some examples, user input defining a content delivery campaign may include natural language text, text commands, or pre-selected options. The input may describe the content to be delivered and / or the mechanism of delivery (e.g., a particular device to receive the content, a communications protocol to use in transmitting the content, or the format of the content). The user input may be processed to determine one or more keywords. In some examples, the natural language text or text commands may be processed using a rule system. The rule system may include a hierarchy of rules that, when applied to text, assign words or phrases to categories. Application of each rule may generate a class identifier that characterizes the word or phrase as corresponding to a particular category.

[0017] In some examples, categories may be hierarchical, such that if a word is characterized as corresponding to more than one category upon application of the rules, a more specific or higher priority category may be assigned to the word. For example, a first rule may identify phrases that refer to a first category, "mode of transportation," a second rule may identify phrases that refer to a second category, "car," and a third rule may identify phrases that refer to a third category, "sedan." Input text referring to a sedan triggers the application of all three rules, but the third rule is applied because it is the most specific. As a result, the keyword "sedan" is selected as the keyword for the phrase.

[0018] In some examples, a machine learning model may be used to identify keywords in the input. The machine learning model may be trained using training data derived from a set of previous content distribution operations, an open source database, a public database, and / or manufactured data (e.g., data generated to train the machine learning model). A set of features (e.g., words or phrases) may be extracted from the training data. The set of features may then be passed as input to the machine learning model. The machine learning model may be trained using unsupervised learning, semi-supervised learning, or supervised learning. Once trained, the machine learning model may receive the input text and output one or more keywords. In some examples, the machine learning model may also output a confidence for each keyword indicating the probability that each keyword of the one or more keywords corresponds to the input text.

[0019] For example, a set of previous content delivery operations may be used to identify a set of input texts for training a machine learning model. The machine learning model is trained using supervised learning, where each input text is associated with a label (e.g., a keyword obtained from analyzing the input text). The machine learning model may use the set of input texts to determine which words and / or phrases correspond to a particular category. In supervised learning, the machine learning model may compare the output generated by the model (e.g., predicted keywords) with the labels associated with the input texts to adjust the internal processes of the machine learning model to improve the accuracy of subsequent predictions. In unsupervised learning, the machine learning model may extract features from each input text in the set of input texts. The machine learning model may then identify relationships between related input texts (e.g., associated with the same keyword or keywords) in the set of input texts based on the extracted features, for example, using cluster analysis. Once trained, the machine learning model may be run using inputs for a particular content delivery operation to classify the words and phrases of the input.

[0020] One or more keywords may be extracted from the categorized input. The one or more keywords may correspond to words or phrases in the input that may represent a context for the content delivery operation. Alternatively, the one or more keywords may correspond to categories assigned to the words or phrases. In some examples, a confidence value assigned to each input text indicates the likelihood that the keyword was correctly selected for the input text. A threshold may be used to prevent one or more keywords from representing an inappropriate context for the content delivery operation by discarding keywords associated with confidence values ​​below the threshold.

[0021] One or more keywords may be used to obtain image data for input into the generative adversarial network. One or more keywords may be used to generate a query. The query may be executed to obtain image data from one or more databases and / or external networks. The image data may correspond to one or more keywords and may be associated with the context of the content delivery operation. In some examples, one or more queries are executed in an iterative process to obtain image data from one or more databases and / or external networks. For example, if a first query to a first database does not return image data corresponding to the one or more keywords, a second query may be generated and executed on another database. The process may continue with each successive query executed on a different database or external network. If none of the databases or networks returns image data corresponding to the one or more keywords, a final query may be generated and executed to retrieve image data from an open source database, a public database, or the Internet.

[0022] In some examples, the image data may be processed before being input into the generative adversarial network. For example, one or more first keywords may correspond to cars and trucks, and one or more second keywords may indicate that the background of the cars or trucks is an indoor environment. The retrieved image data may include images of cars and trucks in various indoor and outdoor environments. If the image data corresponding to cars and trucks in the indoor environment exceeds a threshold (e.g., a certain amount of images and / or a certain volume of data), the image data corresponding to cars or trucks in the outdoor environment may not be used.

[0023] If the image data corresponding to cars and trucks in the indoor environment does not exceed the threshold, the image data corresponding to cars or trucks in the outdoor environment may be processed to enable use of the data according to the one or more first keywords and one or more second keywords. For example, the image data may be processed by generating an indication (e.g., a label) for portions of each image that correspond to the one or more first keywords and / or one or more second keywords (or portions of the image data that do not correspond to the one or more first keywords or one or more second keywords). In some examples, portions of the image that correspond to one of the one or more first keywords and one or more second keywords may be extracted from portions of the image that do not correspond to one of the one or more first keywords and one or more second keywords. By generating indications for relevant portions of the image (or irrelevant portions), a generative adversarial network can be appropriately trained.

[0024] Image processing may include one or more operations that can be applied to modify image data to better train a generative adversarial network. The one or more operations may be performed in any particular order. Image processing may include using object detection to label one or more objects in the image, particularly portions of the one or more objects that correspond to one or more keywords. In some examples, object detection / labeling may be performed by a neural network such as a convolutional neural network (CNN), a regional CNN, a fast CNN, or a you-only-look-once (YOLO) algorithm. Once the objects in the image have been labeled, the image may be further processed, for example, by removing portions of the image that do not correspond to one or more keywords and reducing the size of the image.

[0025] The modified image data may be passed as input to a generator network of the generative adversarial network. The generator network may be a neural network (e.g., a convolutional or deconvolutional neural network, etc.). The generator network derives a first set of images from the input image data using a transformation function through a set of layers. The first set of images may correspond to one or more keywords (e.g., the context of a content distribution campaign) and may be in a format suitable for distribution as part of a content distribution campaign. The first set of images may be passed to a classifier network for evaluation.

[0026] The classifier network is configured to generate a first set of images, each of which is generated by a generator network. For each image, the classifier network may generate a prediction indicating whether the image is a simulated or real image. The classifier network may send the prediction back to the generator network to facilitate further training (e.g., backpropagation learning) of the generator network. In some examples, each image generated by the generator network may be accepted or rejected by a user before or after being passed to the classifier network. For example, each image may be labeled as accepted or rejected. In some examples, the images and corresponding labels may be passed to the generator network as a second layer of backpropagation learning. In such cases, the labels can be used to further refine image generation.

[0027] The prediction assigned to each image may be used to determine which images to include in the content distribution campaign and which images can be discarded (or used for further training of the generative adversarial network). For example, the content distribution campaign may include a second set of images that is a subset of the first set of images. The second set of images may include images from the first set of images that have been assigned a prediction from the discriminator network indicating that they were not generated by the generator network. The second set of images may then be transmitted to one or more client devices in accordance with the content distribution campaign. Alternatively or additionally, in some examples, the content distribution operation may include physical representations of images in the second set of images. For example, the images may be printed and sent by mail to one or more recipients, or may be distributed in another manner.

[0028] 1 is a block diagram of a system 100 for generating content distribution images according to at least one aspect of the present disclosure. System 100 includes one or more computing devices 104-1, 104-n that can be used to automatically generate content for content distribution operations. Each computing device 104-1 through 104-n can include the same or different hardware components as shown in 104-1. For example, computing device 104-1 can include hardware components for performing a particular aspect of content generation (e.g., text analysis, image processing, content generation, content evaluation, or a combination thereof).

[0029] Computing device 104-1 includes one or more processors (e.g., CPU 108) coupled to memory 112 via a bus. A user interface 116 for interacting with a user may be presented on a display 120. A user may manipulate input / output devices to define new content delivery actions within user interface 116. The input may include one or more attributes and text that characterize the new content delivery. For example, the text may be natural language text, an alphanumeric string, an alphanumeric command, or the like.

[0030] The content delivery behavior definition may be passed to a machine learning core 124. The machine learning core 124 may be one or more artificial intelligence modules that run together to generate an image that conforms to the content delivery behavior definition. The definition may be passed to a parser 128 within the machine learning core 124. The parser 128 may parse the text input to determine one or more keywords of the definition. The one or more keywords may correspond to the context of the definition, such as the subject, object, etc. of the content to be delivered. The parser 124 converts the text into a set of tokens (e.g., individual alphanumeric characters, individual words, and / or phrases based on the configuration of the parser 124). The set of tokens may then be individually labeled and combined (e.g., using a sliding window of a pre-set character length). This labeling may be based on matching the tokens to pre-defined keywords, a hierarchy of rules, or using machine learning. Machine learning implementations may be associated with context. It may output only those tokens (e.g., nouns, verbs, adjectives with nouns, etc.) and ignore or discard certain tokens (e.g., words with little contextual value), such as tokens associated with articles.

[0031] The machine learning core 124 may obtain image data using the extracted one or more keywords. In some examples, the machine learning core 124 may retrieve image data corresponding to the one or more keywords from the local image data 136. In other examples, the machine learning core 124 may generate and execute a query to an external data store, such as the computing device 104-n or the image data 152. If the data cannot be sourced within the network, the machine learning core 124 may generate and execute a query to retrieve image data from the Internet (e.g., using a search engine, a web crawler, etc.). The image data may include one or more images depicting a subject / object corresponding to one or more of the one or more keywords (i.e., a context or portion thereof). For example, the one or more keywords may include "race track" and "sedan," and the image data may include an image of a race track, an image of a sedan, and / or an image including both a sedan and a race track.

[0032] The machine learning core 124 may pass the image data to the image processor 128 to process the image into a format more suitable for one or more neural networks, such as a generative adversarial network. Image processing may include one or more operations to modify the image and / or label the content of the image. Image processing may include application of one or more filters, signal or frequency analysis, edge detection, orientation correction such as affine or Euclidean transformations, or a combination thereof, etc.

[0033] For example, edge detection modifies an image to emphasize the edges of objects (or subjects) in the image. Edge detection can begin with filtering techniques in which one or more filters can be applied to the image. The filters may modify the image by blurring, sharpening, transforming (e.g., but not limited to, one or more affine or Euclidean transformations), and / or otherwise. The filters may reduce image noise, for example, by removing image artifacts and / or other portions of the image that do not correspond to one or more keywords.

[0034] In some examples, portions of the image may be processed more than other portions of the image. For example, one portion of the image may appear blurred while another portion of the image is sharpened. Different filters may be applied to different portions of the image, and in addition, different sets of filters may be applied to different portions of the image. Different filters may be applied to different portions of the image. For example, a first portion of the image may be filtered to sharpen the first portion, and a second portion of the image may be filtered with an affine transformation filter and noise reduction. Any number of different filters may be applied to the image and / or each patch.

[0035] Once the filter is applied, changes in pixel intensity gradients across adjacent pixels can be identified. A large change in intensity between adjacent pixels may indicate the presence of an edge. For example, a first pixel with a high intensity value next to a pixel with a low intensity value may provide an indication that the first pixel is part of an edge. In some examples, pixels that are not part of an edge may be suppressed (e.g., set to predetermined red / green / blue values, such as black where red=0, blue=0, and green=0, or any predetermined red / green / blue value). Edge detection operators such as a Roberts Cross operator, a Prewitt operator, a Sobel operator, and / or others may be used as part of identifying pixel intensity gradients.

[0036] Using non-maximum suppression processing, pixels that do not strongly correspond to edges are suppressed. may be suppressed. The non-maximum suppression process uses pixel intensity gradients to assign an edge strength value to each pixel identified as being part of an edge. For each pixel identified as being part of an edge, the edge strength value of that pixel may be compared to the edge strength values ​​of its eight surrounding pixels. If the pixel has an edge strength value higher than the edge strength value of a surrounding pixel (e.g., a local maximum), the surrounding pixel is suppressed. Non-maximum suppression may be repeated for each pixel throughout the image.

[0037] A double thresholding process may then be performed to remove noise and / or spurious edge pixels introduced through the application of previous image processing techniques applied herein. Two thresholds of pixel intensity may be defined, one high and one low. The thresholds may be used to assign an intensity property to each pixel as strong or weak. Pixels with intensity values ​​higher than the high threshold may be assigned a strong intensity property, and pixels with intensity values ​​between the high and low thresholds may be assigned a weak intensity property. Pixels with intensity values ​​below the low threshold may be suppressed (e.g., in the same manner as above).

[0038] A hysteresis process may then be performed to remove pixels with weak intensity properties (i.e., weak due to noise, color shading, etc.). For example, a local statistical analysis (e.g., connected component analysis, etc.) may be performed for each pixel with a weak intensity property. Pixels with weak intensity properties that are not surrounded by pixels with strong intensity properties may be suppressed. The remaining pixels (e.g., unsuppressed pixels) after the hysteresis process include only pixels that are part of an edge. While the above five processes are described in a particular order, each process may be performed (e.g., iterated) any number of times and / or in any order without departing from the spirit or scope of this disclosure. In some examples, only a subset of the five processes need be performed on an image. For example, an image process may perform a pixel intensity gradient identification process without first performing a filtering process. In some examples, an image may be received in a partially processed state (e.g., with one or more of the above processes already performed). In such cases, one or more additional processes may be performed to complete the image processing.

[0039] In some examples, signal processing may be performed on an image (e.g., similar to a radio frequency signal). The image may be transformed into the frequency domain (e.g., using a Fourier transform, etc.) to represent the frequencies at which particular pixel characteristics (e.g., pixel intensity, RGB values, and / or other) exist in the image. In the frequency domain, one or more filters (such as, but not limited to, Butterworth filters, bandpass, and / or other) may be applied to the image (e.g., during preprocessing, edge detection, or after) to suppress or modify particular frequencies. Suppressing particular frequencies may reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels of particular colors or color gradients, normalize color gradients, and / or other. A high-pass filter may reveal edges in an image (e.g., sharp contrasts in color and / or intensity between adjacent pixels), while a low-pass filter may blend (e.g., blur) edges. Image padding may be performed prior to signal processing to improve signal processing techniques. In some examples, different portions and / or patches of an image may be processed differently, such that some are processed with a high-pass filter and other portions are processed with a low-pass filter. In some examples, thresholds (e.g., cutoff frequencies of high-pass or low-pass filters) may be modified for different portions of an image (e.g., based on image processing of one or more previous images, machine learning, and / or other).

[0040] Signal processing also involves coherence, which identifies relationships between pixels (e.g., edge detection, segmentation, etc.). Other properties of the image may also be determined, such as pixel coherence (used in classification, pattern analysis, etc.). Relationships between pixels may be used to further refine edge detection and / or identify structural properties of what is shown in the image. For example, coherence may be used to distinguish related portions of an image (e.g., parts of the same object that correspond to keywords) from unrelated portions of the image.

[0041] In some examples, an image segmentation operation may be performed instead of, during, or after edge detection. Image segmentation assigns a coherence value (e.g., as described above) to each pixel in the image. Image segmentation may use coherence values ​​identified during edge detection or one or more other operations, such as a Sobel model, graph partitioning, etc. The coherence values ​​represent labels for the pixels that can be used to group pixels according to a common label (e.g., objects depicted in the image). The coherence values ​​may be used to identify the position, orientation, and shape of objects in the image, which can be used via a lookup table to identify the object. If the object cannot be identified using the coherence and image characteristics, the processed image (awaiting further image processing) may be passed to a classifier.

[0042] The classifier may be a predictive machine learning model, such as, but not limited to, a neural network. The classifier may be trained to label objects / subjects corresponding to one or more keywords. The classifier can be trained using local image data 136 or from one or more training datasets received from the training server 148. The neural network may be trained using supervised or unsupervised learning. For example, in supervised learning, a set of labeled images may be input to the neural network. The neural network may define a feature set (e.g., a set of image characteristics that indicate the presence of a label). The feature set may be used by the classifier when unlabeled data is input to the neural network. In unsupervised learning, images may be passed as input to the classifier along with their labels. The classifier may learn the feature set through analysis of the images. If the accuracy of the classifier falls below a threshold, the classifier may be retrained using supervised or unsupervised learning with the training dataset and / or any additional labeled or unlabeled images.

[0043] In some examples, it may be determined whether the generative adversarial network has been trained to generate images corresponding to the image data before passing the image data as input. If the generative adversarial network has not been trained, the machine learning core 124 may train the generative adversarial network on-demand based on particular processed images and / or labeled image data that may be passed as input. For example, if the image data corresponds to a race truck and a sedan and the generative adversarial network has not been trained using images of the race truck and / or the sedan, the generative adversarial network may not be able to generate images that look like real images (e.g., do not look computer-generated).

[0044] The machine learning core 124 may attempt to locate and build a training dataset within the local image data 136. If none is found, the machine learning core 124 may generate and execute a query to the training server 148. The training server 148 may use the image data 152, or image data from a previous content distribution operation 156, to obtain images for training the generative adversarial network. For example, if a company previously ran a content distribution operation associated with race trucks and sedans, the machine learning core 124 may use the training data and / or generated training data from that content distribution campaign. The network may then access the images to train the generative adversarial network.

[0045] In some examples, multiple generative adversarial networks may be provided, each trained to generate images corresponding to one or more specific keywords. For example, image data (e.g., input data) may be classified by a classifier to determine the type of image to be generated (e.g., a specific subject, object, setting, etc.). A particular generative adversarial network from the multiple generative adversarial networks may be selected based on the particular generative adversarial network being trained to generate images corresponding to the category identified by the classifier. The multiple generative adversarial networks may be organized into a hierarchy based on the specificity of the keywords for which each generative adversarial network is trained.

[0046] After training, the generative adversarial network uses feedforward and backward A generator neural network 140 operating on backward propagation learning patterns and a discriminator neural network 144. The generator network 140 uses one or more layers of transformation / weight functions to identify features in training images that correspond to specific characteristics (e.g., background, foreground, keywords, objects, subjects, etc.). In supervised learning, the features may be compared to the image labels to determine the accuracy of the generator network 140. Training may continue until a threshold accuracy is reached. If the accuracy falls below the threshold, the generative network 140 may be retrained. Once trained, the generator network 140 may be used to generate a series of images that correspond to one or more keywords in the input. The generator network 140 attempts to generate images that appear indistinguishable from (or close to) real images.

[0047] Once the generator neural network 140 has been trained, the discriminator neural network 144 may be trained. The discriminator network 144 attempts to predict whether an image was generated by the generator neural network 140 (e.g., whether it is a fake image). During training, the images generated by the generator neural network 140 and a training set of images received from one or more other sources may be input to the discriminator network 144. The training data may be labeled (in the case of supervised learning) or unlabeled (in the case of unsupervised learning). The discriminator network 144 may use the images passed as input to improve the accuracy of subsequent predictions. Once trained, the discriminator network 144 may begin making predictions on images generated by the generator network 140. These predictions may be fed back to the generator neural network 140 to improve the images generated. This process may continue until a predetermined error rate is detected. For example, the error rate may correspond to the percentage of images generated by the generator neural network 140 that are predicted to be real by the discriminator neural network 144 (e.g., predicted by the discriminator neural network 144 as not being generated by the generator neural network 140). Because the two networks operate in both feedforward and backward propagation, the model always achieves more realistic images than those generated by the generator network 140.

[0048] In some examples, the classifier network 144 may output other characteristics of the image in addition to the prediction. For example, the classifier network 144 may assign a confidence value indicating the confidence level of the prediction. The classifier network 144 may also assign a precision value indicating how well the generated image corresponds to one or more keywords. For example, for the keywords "race track" and "sedan," if a generated image shows a race track without a sedan, the image may be given a lower precision value than a generated image that shows both a race track and a sedan. The images may be ranked according to precision (and / or prediction). . . 160-n.

[0049] Once generator 140 generates an image that fools classifier neural network 144 (e.g., that appears realistic enough that classifier neural network 144 predicts it to be real), the image may be further processed (e.g., using one or more of the image processing operations described above) and / or the image may be distributed over a network to one or more client devices 160-1, 160-2, 160-3, ... 160-n.

[0050] FIG. 2 is a block diagram illustrating a process flow for generating a content delivery image according to at least one aspect of the present disclosure. The process flow may begin with a user interface 204 that allows a user to define a content delivery operation. The user interface includes multiple fields for the content delivery operation, including a title field 208 and a text field 212. The text field 212 allows for receiving user input, such as an alphanumeric string describing the content delivery operation. Once received, an upload button 216 may be selected to send the alphanumeric string entered in the title 208 and text 212 to one or more machine learning cores for content generation. In some examples, at least one machine learning core may run locally on the same device as the user interface 204. In other examples, the user interface 204 may be presented on a different device than the machine learning core(s). The user interface 204 may include one or more additional fields that allow a user to indicate certain properties of the content delivery operation. Such properties may include, but are not limited to, the receiving device (or user), one or more communication protocols for transmitting the generated content, or a threshold accuracy for each generated image.

[0051] The text (and any other properties) may be passed to a text processing pipeline 228, which may parse the text to determine one or more keywords associated with the text field 212. The text may be parsed via a keyword lookup table, a machine learning model, or any other operation as described above. In block 232, the text processing pipeline 228 may use the one or more keywords identified from the text field 212 to identify image data corresponding to the one or more keywords. For example, block 232 identifies images or portions of images corresponding to each of the one or more keywords. In some examples, a predefined image volume may be defined for the image data. In such a case, block 232 may first identify a first set of images corresponding to each of the one or more keywords. If the first set of images is not at or above the predefined threshold, images corresponding to all but one of the one or more keywords may be added to the first set of images. If the first set of images is still not at or above the predefined threshold, images corresponding to all but one of the one or more keywords may be added to the first set of images, and so on, until the predefined image volume for the image data is achieved.

[0052] At block 236, one or more image processing techniques and / or object identification may be performed on the image data. For example, image segmentation or edge detection may be used to modify the image data to distinguish portions of the image that correspond to at least one of the keywords from portions of the image that do not correspond to one or more keywords. Each portion of the image that corresponds to at least one keyword may be labeled with the at least one keyword. In some examples, further image processing may be performed to remove (or otherwise mark) portions of the image that do not correspond to at least one keyword. You may do so.

[0053] The labeled and processed image data may be passed to block 240, which may use the labeled and processed image data to generate training data sets for the generative adversarial networks 252-256. For example, block 240 may determine whether the generative adversarial networks 252-256 have been trained to generate images corresponding to one or more keywords. If not, a training data set corresponding to one or more keywords may be generated to dynamically train the generative adversarial networks 252-256. For example, the training data set may be obtained from a training server 148, which may have access to image data from previous content distribution 156 and / or one or more remote networks (e.g., external networks, the Internet, etc.).

[0054] In some examples, images from the labeled and processed image data may be output to the user interface 204. For example, a representation of some or all of the labeled and processed image data may be sent to the user interface 204. The user may then review the images from the labeled and processed image data. In some examples, the user may discard one or more images from the labeled and processed image data (e.g., images that do not pass the user's review). In some examples, the user may supplement the labeled and processed image data, for example, by uploading 224 additional images to the labeled and processed image data.

[0055] The labeled and processed image data may be passed as input to a generative adversarial network, including a generator network 252 and a classifier network 256. The generator network 252 may use the labeled and processed image data to generate a set of new artificial images that can be evaluated by the classifier network 256. The classifier network 256 may generate predictions indicating whether the generated images are real or artificial. Images predicted to be real may be output to block 260, where the images are packaged for transmission to one or more remote clients or client devices. Images predicted to be artificial can be fed back to the generator network 252 as backpropagation in the generative adversarial network to improve the accuracy of future image generation and subsequent labeling.

[0056] 3 illustrates an example of a graphical user interface associated with generating a content delivery image according to at least one aspect of the present disclosure. The graphical user interface 304 includes one or more frames, each of which presents different information to a user. For example, the top frame may receive user input that defines and / or modifies aspects of the content delivery operation. After the text defining the content delivery operation is parsed into one or more keywords that describe the context of the content delivery operation, the user may be presented with the user interface 304 for modifying aspects of the content delivery operation definition. For example, the user may toggle the operator 308 to display the text defining the content delivery operation.

[0057] The user can toggle operator 312 to review some or all of the reference images that are fed to the machine learning core that generates the image. The reference images include images provided by the user and / or images from the image data identified by the machine learning core. Detected Keywords / Tokens 316 may include a representation of each keyword and / or associated token identified from the provided text. The user may select or deselect one or more keywords to generate new image data, or to deselect (or deselect) (already used) the keywords. The image data (if generated) may be modified. The keyword / token confidence filter 320 may be used to indicate to the user which keywords are likely to correspond highly to the supplied text and which keywords may have a low correspondence confidence. In some examples, the keyword / token confidence filter 320 may include one or more threshold filters that automatically remove keywords below a certain confidence value.

[0058] Images generated by the generative adversarial network may be presented in real time (e.g., as the images are generated) via a user interface 304. The user interface 304 may represent each generated image 324 as a full image or as a thumbnail image that can be presented as a full image upon receiving input to select an image. In some examples, the images presented via the user interface 304 may be all images generated by the generative adversarial network. In other examples, the images presented via the user interface 304 may include only images predicted by the discriminator network as not being generated by the generator network. Each image 324 may include an accept button 328 and a reject button 332. An input may be received to accept (234) the image for delivery to one or more clients or client devices or to discard the image by rejecting (324). In some examples, the selection of accept 328 or reject 332 may be assigned to the image 324 as a label. The images and assigned labels can be backpropagated to the generative adversarial network to add additional training layers that may further improve image generation for content delivery operations.

[0059] 4 shows a flowchart for generating a content delivery operation according to at least one aspect of the present disclosure. At block 404, user input for generating content for a content delivery (e.g., marketing) operation associated with an entity (e.g., a company selling a product / service or a company providing content delivery services) may be received. The user input may include one or more alphanumeric strings describing the content delivery operation. The one or more alphanumeric strings may include natural language text, text commands in a particular schema such as key / value pairs, or the like. The user input may also include one or more properties of the content delivery operation, such as the type of image to generate, the format of the image, the client or client device to receive the image, the communication protocol to use when transmitting the image, or a combination thereof.

[0060] In block 408, the user input may be parsed to identify one or more keywords associated with the context of the content delivery action. Parsing the user input may include a keyword analysis (e.g., identifying specific words and matching those words to a keyword lookup table), a natural language machine learning model (as described above), or the like. Each keyword of the one or more keywords represents a portion of content to be shown in the generated image. For example, a keyword may correspond to an object, a context, an action, a subject, a background, a foreground, a setting (e.g., a location such as a beach or a school), or a combination thereof.

[0061] At block 412, image data corresponding to at least one of the one or more keywords may be received from one or more databases (or a network, such as an external network or the Internet). In some examples, the image data may include structured or unstructured data that characterizes or represents images associated with the at least one keyword. In other examples, the image data includes structured or unstructured data as well as one or more images, each corresponding to the at least one keyword. In still other examples, the image data may include only one or more images.

[0062] At block 416, one or more image processing operations may be performed on the image data to derive processed image data. The one or more image processing operations may include operations that may modify or transform the image (e.g., affine transformation, edge detection, image segmentation, etc.). The one or more image processing operations may also include object detection. Object detection may use a trained neural network to identify one or more objects, subjects, settings, actions, contexts, or combinations thereof, within an image. For example, in an image of a sedan racing on a race track, the trained neural network may label portions of the image that correspond to the object (e.g., the sedan), portions of the image that correspond to the setting (e.g., the race track), and may label the context (e.g., day / night, weather, etc.). The one or more image processing operations may also include removing portions of the image that do not correspond to at least one keyword.

[0063] The one or more image processing operations may also include marking portions of the image data to be left unmodified by the generative adversarial network, which generates images similar to the particular image. For example, an object may be marked to be left unmodified so that, when input to the generative adversarial network, the generative adversarial network may generate different images containing the same object. The generative adversarial network may generate images of the object in different contexts (e.g., different settings, locations, backgrounds, foregrounds, or weather, etc.), with different other objects, etc.

[0064] The one or more image processing operations may also include marking certain image portions of the image data to be modified by the generative adversarial network. For example, an object may be marked not to be modified so that, when input to the generative adversarial network, the generative adversarial network may generate a different image in which the only difference is the object. If the object is a sedan shown on a racetrack, marking the sedan in this manner may enable the generative adversarial network to generate an image in which the only difference is a modification of the sedan. For example, the generative adversarial network may generate an image in which the sedan is a different color or a different make or model. The sedan may be replaced with another object, such as a truck, a bicycle, or the like. If the object corresponds to a human, human characteristics may be modified, such as, but not limited to, age, race, gender, height, weight, appearance (e.g., eye color, hair color, or clothing), or a combination thereof.

[0065] At block 420, a generative adversarial network may be run on the processed image data. This running includes:

[0066] At block 424, a first neural network may be run on the image data to generate a first set of new images, each corresponding to at least one keyword of the one or more keywords. The first neural network may be run using the one or more keywords (from the processed image data) as input or using one or more images. The first set of images may be synthetic images that are entirely computer-generated but are generated to appear realistic (e.g., not computer-generated). The first neural network may be a generator neural network as part of a generative adversarial network. The generator neural network may be trained with a second neural network such that the first set of images may be generated based on the likelihood that each image in the first set of images would not be detected by the second neural network as having been generated by the first neural network.

[0067] For example, the second neural network may be trained on the ancestor images of the training set of images (e.g. , images previously generated by the first neural network). The prediction may correspond to the likelihood that the ancestor image was generated by the first neural network. The training set of images may include ancestor images and images received from one or more other sources (e.g., previous content distribution operations, the internet, images from the same entity, etc.). The one or more other sources may include one or more images received from image data, local or remote databases, the internet, etc. The images received from the one or more other sources may be real images (e.g., images that are not generated by a computer, images generated by a computer such as another generative adversarial network, or a combination thereof).

[0068] The second neural network may be a discriminator neural network representing another part of the generative adversarial network. The goal of the generative adversarial network may be to generate images that look like real images and trick the second neural network into predicting that the generated images are "real" images rather than images generated by the generator neural network. The goal of the second neural network is to accurately predict whether an image was generated by the first neural network and return that prediction to the generator neural network (e.g., via backpropagation training). In some examples, the second neural network may assign a range of values ​​rather than a Boolean value (e.g., real or generated by the generator neural network). For example, the range of values ​​may indicate a probability or confidence that the image was generated by the generator neural network. In some examples, the first neural network and the second neural network may be the same neural network, while in other examples, the first neural network and the second neural network may be different neural networks.

[0069] The first neural network may be updated with predictions from the second neural network. The predictions assigned to images generated by the first neural network may provide the first neural network with an indication of how well the image fooled the second neural network into predicting that the image was or was not generated by the first neural network. This information may be used to modify one or more layers of the first neural network to enable the first neural network to generate images that the second neural network is more likely to predict as real (e.g., more likely to appear realistic rather than computer-generated).

[0070] Once trained, the first neural network may be run to generate images, each corresponding to at least one of the one or more keywords. The generated images may be used to continually train the first neural network to continually improve subsequently generated images (e.g., so that the images are less likely to appear computer-generated).

[0071] In block 428, the user interface may display the first set of images with the second set of images (e.g., side by side, in the same window, etc.). The second set of images may include images that were previously part of one or more content distribution operations or images designated by an entity (e.g., a business, a user within the business, etc.) as being associated with one or more keywords. In some examples, the first set of images may be transmitted to a client or client device (e.g., based on user input in block 404). The user interface may allow a user to review each image in the first set of images and compare those images with images in the second set of images. In some examples, the user may review each image in the first set of images and compare those images with images in the second set of images. The user may accept or reject individual images of the subject. In such cases, the user's choice of acceptance or rejection may be assigned to the image as a label. The images and corresponding labels may be passed to the first neural network for further training of the first neural network. In some examples, images labeled as real may be removed before further training. This may enable a second layer of backpropagation learning in the generative adversarial network, where the second neural network and the user can each further update or train the first neural network separately.

[0072] Each block in FIG. 4 may be performed in any particular order or with any particular frequency, such as sequentially, out of order, once, or multiple times (sequentially or out of order), without departing from the spirit or scope of the present disclosure.

[0073] In the above description, specific details are provided to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments can be practiced without these specific details. For example, circuits may be shown in block diagrams in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0074] The implementation of the techniques, blocks, steps, and means described above may be done in various ways. For example, these techniques, blocks, steps, and means may be implemented in hardware, software, or a combination thereof. In a hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ICs). ASIC, digital signal processor (DSP), digital signal processing device ( DSPD, programmable logic device (PLD) ), Field Programmable Gate Array (FP GA), processor, controller, microcontroller, microprocessor, other electronic unit designed to perform the functions described above, and / or combinations thereof.

[0075] It should also be noted that the embodiments may be described as a process that is depicted as a flowchart, flow diagram, swim diagram, data flow diagram, structure diagram, or block diagram. While the description may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be re-arranged. A process is terminated when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the function corresponds to a return of the function to the calling function or the main function.

[0076] Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium such as a storage medium. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment may send or receive information, data, arguments, parameters, and / or memory contents to another code segment or a hardware circuit. Information, arguments, parameters, data, etc. can be sent, forwarded, or transmitted by any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0077] For firmware and / or software implementations, methods may be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions may be used to implement the methods described herein. For example, software code may be stored in a memory. The memory may be implemented within the processor or external to the processor. As used herein, the term "memory" refers to any type of long-term, short-term, volatile, non-volatile, or other storage medium and is not limited to any particular type of memory or any particular number of memories or to any particular type of medium on which memory is stored.

[0078] Additionally, as disclosed herein, the term "storage medium" may refer to one or more memories for storing data, including read-only memory (ROM), random-access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The term "machine-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and / or various other media capable of storing, containing, or carrying instruction(s) and / or data.

[0079] While the principles of the present disclosure have been described above in connection with specific apparatus and methods, it is to be clearly understood that this description is made only by way of example and not as a limitation on the scope of the disclosure.

Claims

1. 1. A method comprising: receiving a user input corresponding to a content distribution action; Parsing the user input to identify one or more keywords associated with the content delivery action; receiving image data from one or more databases corresponding to the one or more keywords associated with the content distribution operation; performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of specific locations within the image that correspond to the one or more keywords, the method further comprising: executing a generative adversarial network on the processed image data to generate one or more images for the content distribution operation, wherein executing the generative adversarial network includes: and running a first neural network on the processed image data, the first neural network generating a first set of images corresponding to the one or more keywords, the first set of images being generated based at least in part on a likelihood that each image in the first set of images would not be detected as having been generated by the first neural network, and running the generative adversarial network further comprises:

10. A method comprising: displaying the first set of images via a first user interface together with a second set of images, the second set of images including images that were previously part of one or more content delivery operations or images designated as available for content delivery operations by an entity associated with the content delivery operations.

2. The method of claim 1 , wherein the one or more image processing operations includes image segmentation.

3. The method of claim 1 , wherein the generative adversarial network is trained at runtime based on the one or more keywords.

4. determining, based on the one or more keywords, that the generative adversarial network has not been trained to generate new images corresponding to at least one of the one or more keywords; sending a request for a training dataset to one or more databases, the training dataset including a plurality of images, a portion of each image of the plurality of images corresponding to at least one keyword of the one or more keywords; and The method of claim 1 , comprising training the generative adversarial network with the training dataset.

5. The method of claim 1 , wherein the image data includes one or more images from a previous content distribution operation.

6. The method of claim 1 , wherein the one or more image processing operations include labeling a portion of each image of the image data that corresponds to one of the one or more keywords.

7. Running a generative adversarial network also receiving an input assigning an accept or reject label to each image in the first set of images; training the first neural network based at least in part on the labels; and removing from the first set of images each image that is assigned a rejected label.

8. 1. A system comprising: one or more processors; a non-transitory computer-readable medium containing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: receiving a user input corresponding to a content distribution action; Parsing the user input to identify one or more keywords associated with the content delivery action; receiving image data from one or more databases corresponding to the one or more keywords associated with the content distribution operation; and performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of specific locations within the image that correspond to the one or more keywords, the operations further comprising: executing a generative adversarial network on the processed image data to generate one or more images for the content distribution operation, wherein executing the generative adversarial network includes: and running a first neural network on the processed image data, the first neural network generating a first set of images corresponding to the one or more keywords, the first set of images being generated based at least in part on a likelihood that each image in the first set of images would not be detected as having been generated by the first neural network, and running the generative adversarial network further comprises: displaying the first set of images together with a second set of images via a first user interface, the second set of images including images that were previously part of one or more content delivery operations or images designated as available for content delivery operations by an entity associated with the content delivery operations.

9. The system of claim 8 , wherein the one or more image processing operations includes image segmentation.

10. The system of claim 8 , wherein the generative adversarial network is trained at runtime based on the one or more keywords.

11. determining, based on the one or more keywords, that the generative adversarial network has not been trained to generate new images corresponding to at least one of the one or more keywords; sending a request for a training dataset to one or more databases, the training dataset including a plurality of images, a portion of each image of the plurality of images corresponding to at least one keyword of the one or more keywords; and The system of claim 8 , comprising training the generative adversarial network with the training dataset.

12. The system of claim 8 , wherein the image data includes one or more images from a previous content distribution operation.

13. The system of claim 8 , wherein the one or more image processing operations include labeling a portion of each image of the image data that corresponds to one of the one or more keywords.

14. Running a generative adversarial network also receiving an input assigning an accept or reject label to each image in the first set of images; training the first neural network based at least in part on the labels; and removing each image from the first set of images that is assigned a rejected label.

15. A non-transitory computer-readable medium containing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: receiving a user input corresponding to a content distribution action; Parsing the user input to identify one or more keywords associated with the content delivery action; receiving image data from one or more databases corresponding to the one or more keywords associated with the content distribution operation; and performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of specific locations within the image that correspond to the one or more keywords, the operations further comprising: executing a generative adversarial network on the processed image data to generate one or more images for the content distribution operation, wherein executing the generative adversarial network includes: and running a first neural network on the processed image data, the first neural network generating a first set of images corresponding to the one or more keywords, the first set of images being generated based at least in part on a likelihood that each image in the first set of images would not be detected as having been generated by the first neural network, and running the generative adversarial network further comprises: a first user interface displaying the first set of images together with a second set of images, the second set of images including images that were previously part of one or more content delivery operations or images designated as available for content delivery operations by an entity associated with the content delivery operations;

16. The non-transitory computer-readable medium of claim 15 , wherein the one or more image processing operations include image segmentation.

17. 16. The non-transitory computer-readable medium of claim 15, wherein the generative adversarial network is trained at runtime based on the one or more keywords.

18. determining, based on the one or more keywords, that the generative adversarial network has not been trained to generate new images corresponding to at least one of the one or more keywords; sending a request for a training dataset to one or more databases, the training dataset including a plurality of images, a portion of each image of the plurality of images corresponding to at least one keyword of the one or more keywords; and training the generative adversarial network using the training dataset.

16. The non-transitory computer-readable medium of claim 15.

19. 16. The non-transitory computer-readable medium of claim 15, wherein the one or more image processing operations include labeling a portion of each image of the image data that corresponds to one of the one or more keywords.

20. Running a generative adversarial network also receiving an input assigning an accept or reject label to each image in the first set of images; training the first neural network based at least in part on the labels; and removing each image from the first set of images that is assigned a rejected label.

Images