Extensible architecture for automatically generating content distribution images

Abstract

Methods and systems for automatically generating content distribution images are disclosed, including receiving user input corresponding to a content distribution operation. The user input can be parsed to identify a keyword. Image data corresponding to the keyword can be identified. Image processing operations can be performed on the image data. A generative adversarial network is performed on the processed image data, including performing a first neural network on the processed image data to generate first images corresponding to the keyword, the first images being generated based on a likelihood that each image in the first images is not detected as having been generated by the first neural network. A user interface can display the first images and a second image, the second image including an image that was previously part of the content distribution operation or an image specified by an entity as being available for the content distribution operation.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. non-provisional application 17 / 017,486, filed September 10, 2020, entitled “SCALABLE ARCHITECTURE FORAUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” which claims the benefit of U.S. provisional application 62 / 900,40,40, filed September 13, 2019, entitled “SCALABLE ARCHITECTURE FORAUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” the entire contents of which are incorporated herein by reference for all purposes. Technical Field

[0003] This invention generally relates to automatically generated content for content-distribution operations, and more particularly to a scalable architecture for content-distribution operations utilizing neural networks. Background Technology

[0004] Generating content for content distribution operations can be resource-intensive and time-consuming. Typically, the goal of a content distribution operation is to distribute specific content to a specific set of devices within a defined, relevant time interval. Companies may hire teams of photographers, designers, and artists to acquire images for the operation, modify these images based on one or more criteria, convert the modified images to a format suitable for transmission or display, and then transmit the formatted images to the specific set of devices. Companies often spend weeks and significant resource expenditure designing a single content distribution. Furthermore, by the time a distribution operation begins, enough time may have passed that the generated content may no longer be relevant to the specific set of devices receiving it. Therefore, systems and methods are needed to reduce the resource requirements of content distribution operations. Summary of the Invention

[0005] A method for automatically generating images for content distribution operations is disclosed. The method includes receiving user input corresponding to a content distribution operation; parsing the user input to identify one or more keywords associated with the content distribution operation; receiving image data corresponding to the one or more keywords associated with the content distribution operation from one or more databases; performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of a particular location within an image corresponding to the one or more keywords; performing a generative adversarial network on the processed image data to generate one or more images for the content distribution operation, wherein performing the generative adversarial network includes: performing a first neural network on the processed image data, the first neural network generating a first set of images corresponding to the one or more keywords, wherein the first set of images are generated based at least in part on a likelihood that each image in the first set of images is not detected as having been generated by the first neural network; displaying, via a first user interface, the first set of images and a second set of images, the second set of images including images that were previously part of one or more content distribution operations or images that are specified by an entity associated with the content distribution operation as being available for the content distribution operation.

[0006] Another aspect of the disclosure includes a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the above- described method.

[0007] Other aspects of the disclosure include a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described method.

[0008] Further areas of applicability of the present disclosure will become apparent from the detailed description provided next, according to which further objectives and advantages of the disclosure will be apparent. It should be understood that the detailed description and specific examples, while indicating various embodiments, are given by way of illustration only, and BRIEF DESCRIPTION OF DRAWINGS

[0009] The disclosure is described in conjunction with the appended drawings:

[0010] Figure 1 is a block diagram of a scalable architecture for generating content distribution images in accordance with at least one aspect of the present disclosure.

[0011] Figure 2 is a block diagram illustrating a process flow for generating content distribution images in accordance with at least one aspect of the present disclosure.

[0012] Figure 3An example of a graphical user interface associated with generating content distribution images is illustrated in accordance with at least one aspect of the disclosure.

[0013] Figure 4 A flow diagram for generating content distribution operations is illustrated in accordance with at least one aspect of the disclosure.

[0014] In the drawings, like reference numerals can be used to denote similar components throughout the several views. Additionally, various components of the same type can be distinguished from each other by a second label added to the reference numeral. For example, if a component is referred to as 100, then a similar, but not identical, component can be identified as 100a. Differences between the components of like reference numerals and second labels will be apparent to persons having ordinary skill in the art. DETAILED DESCRIPTION

[0015] The subsequent description is provided for (one or more) preferred exemplary embodiments only and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the subsequent description of (one or more) preferred exemplary embodiments will provide an enabling description for a skilled artisan to implement the preferred exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of elements without departing from the spirit and scope as expressed in the appended claims.

[0016] An extensible architecture can be defined to generate content and content distribution operations using artificial intelligence. A generative adversarial network can be trained at runtime to generate specific images for content distribution operations. For example, user input can be parsed to determine one or more keywords that correspond to an intended context for a content distribution operation. The one or more keywords can represent, for example, one or more subjects of the content (e.g., such as a person), characteristics of the subject (such as race, gender, height, weight, etc. of a person), one or more objects to be represented in the content, settings (e.g., such as a location), etc.

[0017] In some cases, a generative adversarial network can not be trained to generate images for every possible keyword or combination of keywords. In these cases, the one or more keywords can be used to generate image data (e.g., from a database of previous content distribution operations and / or tagged images) that can be used to train a generator neural network of the generative adversarial network at runtime. The generative adversarial network can then generate images corresponding to any keyword or combination of keywords on demand.

[0018] The discriminator neural network of the generative adversarial network can be trained after the generator neural network. Training the discriminator neural network can include using the generator neural network to generate images and passing them to the discriminator neural network along with other images, such as images from the image data. The discriminator neural network can generate a prediction for each image indicating whether the image was generated by the generator neural network. The predictions can be passed back to the generator neural network. The generator neural network uses the predictions of the discriminator neural network to improve image generation to generate images that are more likely to be labeled as real images by the discriminator neural network. Thus, the goal of the generator neural network is to fool the discriminator neural network into predicting false negatives (e.g., predicting that an image was not generated by the generator neural network, even though the image was actually generated by the generator neural network).

[0019] After the generator neural network and the discriminator neural network are trained, the generator neural network can be executed using one or more keywords to generate images for a content distribution campaign. In some cases, the discriminator neural network can continue to generate predictions that can be used by the generator neural network to further refine image generation. In these cases, the predictions can also be used to filter images generated by the generator neural network, for example, by removing those images that the discriminator correctly predicted as being generated by the generator neural network (e.g., the images did not appear as real images). The generated images (or the remaining filtered images) can be transmitted to one or more client devices (e.g., devices associated with entities that requested the images to be generated), content distribution servers (e.g., such as web hosts, etc.), or one or more user devices (e.g., devices associated with users that the one or more client devices are targeted towards) as part of a content distribution campaign.

[0020] In some cases, user input defining a content distribution campaign can include natural language text, textual commands, or preselected options. The input can describe the content to be distributed and / or the mechanism of distribution (e.g., specific devices to receive the content, communication protocols to be used to transmit the content, format of the content, etc.). The user input can be processed to determine one or more keywords. In some cases, the natural language text or textual commands can be processed using a rules system. The rules system can include a hierarchy of rules that, when applied to the text, assign words or phrases to categories. Application of each rule can generate a category identifier that characterizes the word or phrase as corresponding to a particular category.

[0021] In some cases, the categories can be hierarchical, such that if the application of a rule characterizes a word as corresponding to two or more categories, a more specific category or a higher priority category can be assigned to the word. For example, a first rule can identify phrases that refer to a first category, "modes of transportation," a second rule can identify phrases that refer to a second category, "cars," and a third rule can identify phrases that refer to a third category, "sedans." While input text that refers to a sedan would trigger the application of all three rules, the most specific third rule would be applied. Thus, the keyword "sedan" would be selected as the keyword for the phrase.

[0022] In some cases, a machine learning model can be used to identify keywords for input. The machine learning model can be trained using training data derived from a set of previous content distribution operations, open source databases, public databases, and / or from manufacturing data (e.g., data generated for the purpose of training the machine learning model). A set of features (e.g., words or phrases) can be extracted from the training data. The set of features can then be passed as input to the machine learning model. The machine learning model can be trained using unsupervised learning, semi-supervised learning, or supervised learning. Once trained, the machine learning model can receive input text and output one or more keywords. In some cases, the machine learning model can also output a confidence for each of the one or more keywords, which indicates a probability that the keyword corresponds to the input text.

[0023] For example, the set of previous content distribution operations can be used to identify a set of input texts for training a machine learning model. The machine learning model is trained using supervised learning, with each input text to be associated with a label (e.g., a keyword that would be produced by analyzing the input text). The machine learning model can use the set of input texts to determine which words and / or phrases correspond to particular categories. During supervised learning, the machine learning model can compare the output generated by the model (e.g., a predicted keyword) to the label associated with the input text to adjust the internal processes of the machine learning model and improve the accuracy of subsequent predictions. During unsupervised learning, the machine learning model can extract features from each of the set of input texts. The machine learning model can then use clustering analysis, e.g., to identify relationships between related input texts (e.g., associated with the same one or more keywords) in the set of input texts based on the extracted features. Once trained, the machine learning model can be executed using input for a particular content distribution operation to classify the input words and phrases.

[0024] One or more keywords can be extracted from the classified input. The one or more keywords can correspond to words or phrases within the input that can represent a context of the content distribution operation. Alternatively, the one or more keywords can correspond to the class assigned to the words or phrases. In some cases, a confidence value assigned to each input text indicates a likelihood that the input text correctly selected the keyword. A threshold can be used to discard keywords associated with a confidence value below the threshold to prevent the one or more keywords from representing an incorrect context of the content distribution operation.

[0025] The one or more keywords can be used to obtain image data for input into the generative adversarial network. A query can be generated using the one or more keywords. The query can be executed to obtain image data from one or more databases and / or external networks. The image data can correspond to the one or more keywords and be associated with a context of the content distribution operation. In some cases, one or more queries can be executed to obtain image data from one or more databases and / or external networks in an iterative process. For example, if a first query to a first database does not return image data corresponding to the one or more keywords, a second query can be generated and executed to another database. The process can continue with each successive query being executed on a different database or external network. If no database or network returns image data corresponding to the one or more keywords, a final query can be generated and executed to obtain image data from an open source database, a public database, or the internet.

[0026] In some cases, the image data can be processed prior to being input into the generative adversarial network. For example, one or more first keywords can correspond to cars and trucks, and one or more second keywords can indicate that the context of the cars or trucks is an indoor environment. Retrieved image data can include images of cars and trucks in various indoor and outdoor environments. If the image data corresponding to cars and trucks in an indoor environment exceeds a threshold (e.g., a particular number of images and / or a particular amount of data), image data corresponding to cars or trucks in an outdoor environment can not be used.

[0027] If the image data corresponding to cars and trucks in an indoor environment does not exceed the threshold, image data corresponding to cars or trucks in an outdoor environment can be processed to enable the use of the data while complying with one or more first keywords and one or more second keywords. For example, the image data can be processed by generating an indication (e.g., such as a label, etc.) regarding the portion of each image corresponding to one or more first keywords and / or one or more second keywords (or the portion of the image data that does not correspond to one or more first keywords or one or more second keywords). In some cases, the portion of the image corresponding to a keyword of one or more first keywords and one or more second keywords can be extracted from the portion of the image that does not correspond to a keyword of one or more first keywords and one or more second keywords. By generating an indication regarding the relevant portion of the image (or the irrelevant portion), the generative adversarial network can be properly trained.

[0028] The image processing can include one or more operations that can be applied to modify the image data to better train the generative adversarial network. The one or more operations can be performed in any particular order. The image processing can include using object detection to label a portion of one or more objects of the image (particularly one or more objects corresponding to one or more keywords). In some cases, the object detection / labeling can be performed by a neural network, such as a convolutional neural network (CNN), a region-CNN, a fast-CNN, a you-only-look-once (YOLO), etc. Once the objects of the image are labeled, the image can be further processed by, for example, reducing the size of the image by removing portions of the image that do not correspond to one or more keywords.

[0029] The modified image data can be passed as input into a generator network of the generative adversarial network. The generator network can be a neural network (e.g., a convolutional neural network, a deconvolutional neural network, etc.). The generator network derives, via a set of layers, a first set of images from the input image data using a transformation function. The first set of images can correspond to one or more keywords (e.g., a context of a content distribution campaign) and be in a format suitable for distribution as part of a content distribution campaign. The first set of images can be passed to a discriminator network for evaluation.

[0030] The discriminator network can generate a prediction for each image in the first set of images indicating whether the image was generated by the generator network or generated from a real image. The discriminator network can transmit the predictions back to the generator network to facilitate further training (e.g., backpropagation learning) of the generator network. In some cases, each image generated by the generator network can be accepted or rejected by a user before or after being passed to the discriminator network. For example, each image can be labeled as accepted or rejected. In some cases, the images and corresponding labels can be passed to the generator network as a second layer of backpropagation learning. In these cases, the labels can be used to further refine the image generation.

[0031] The predictions assigned to each image can be used to determine which images are to be included in a content distribution campaign and which images can be discarded (or used to generate further training of the generative adversarial network). For example, the content distribution campaign can include a second set of images that are a subset of the first set of images. The second set of images can include images from the first set of images that have been assigned a prediction from the discriminator network indicating that the image was not generated by the generator network. The second set of images can then be transmitted to one or more client devices in accordance with the content distribution campaign. Alternatively or additionally, in some cases, the content distribution operation can include a physical representation of the images in the second set of images. For example, the images can be printed and distributed via mail or otherwise to one or more recipients.

[0032] Figure 1 FIG. 1 is a block diagram of a system 100 for generating content distribution images in accordance with at least one aspect of the present disclosure. The system 100 includes one or more computing devices 104-1, 104-n that can be used to automatically generate content for a content distribution operation. Each computing device 104-1 through 104-n can include the same hardware components, such as those hardware components depicted within 104-1 or different hardware components. For example, the computing device 104-1 can include hardware components that perform particular aspects of content generation (e.g., text analysis, image processing, content generation, content evaluation, or a combination thereof).

[0033] The computing device 104-1 includes one or more processors (e.g., CPU 108) coupled to a memory 112 via a bus. A user interface 116 can be rendered on a display 120 for interaction with a user. The user can operate an input / output device to define a new content distribution operation within the user interface 116. The input can include one or more attributes and text characterizing the new content distribution. For example, the text can be natural language text, alphanumeric strings, alphanumeric commands, or the like.

[0034] The content distribution operation definition can be passed to a machine learning core 124. The machine learning core 124 can be one or more artificial intelligence modules that each execute together to generate an image that conforms to the content distribution operation definition. The definition can be passed to a parser 126 within the machine learning core 124. The parser 126 can break down the textual input to determine one or more keywords of the definition. The one or more keywords can correspond to a context of the definition, such as a subject, object, etc. of the content to be distributed. The parser 126 converts the text into a set of tokens (e.g., every alphanumeric character, every word, and / or phrase based on the parser 126’s settings). The set of tokens can then be tagged individually and in combination (e.g., using a sliding window of preset character length). The tagging can be based on matching the tokens to predefined keywords, a hierarchy of rules, or using machine learning. The machine learning implementation can output only those tokens that can be associated with the context (e.g., nouns, verbs, adjectives with nouns, etc.) while ignoring or discarding certain tokens, such as tokens associated with an article, etc. (e.g., words that have little contextual value).

[0035] The machine learning core 124 can obtain image data using the one or more extracted keywords. In some cases, the machine learning core 124 can retrieve image data corresponding to the one or more keywords from local image data 136. In other cases, the machine learning core 124 can generate and execute a query to an external data store, such as the computing device 104-n or image data 152. If the data cannot be obtained within the network, the machine learning core 124 can generate and execute a query to retrieve image data from the internet (e.g., using a search engine, web crawler, etc.). The image data can include one or more images that depict a subject / object corresponding to one or more of the one or more keywords (i.e., the context or a portion thereof). For example, the one or more keywords can include “race track” and “race car,” and the image data can include an image of a race track, an image of a race car, and / or an image that includes both a race car and a race track.

[0036] The machine learning core 124 can pass the image data to an image processor 128 to process the image into a format more suitable for one or more neural networks, such as a generative adversarial network. The image processing can include one or more operations that modify the image and / or tag the image content. The image processing can include applying one or more filters, signal or frequency analysis, edge detection, orientation correction, such as affine or Euclidean transformations, combinations thereof, etc.

[0037] For example, edge detection modifies an image to emphasize edges of objects (or subjects) within the image. Edge detection can be initiated by filtering techniques, where one or more filters can be applied to the image. Filters can modify the image by blurring, sharpening, transforming (such as, but not limited to, one or more affine transformations, Euclidean transformations, etc.), etc. Filters can reduce image noise by, for example, removing image artifacts and / or other portions of the image that do not correspond to one or more keywords.

[0038] In some cases, an image can have portions that can be processed more than other portions of the image. For example, a portion of an image can appear blurry, while another portion of the image can be clear. Different filters can be applied to different portions of the image, in addition, different groups of filters can be applied to different portions of the image. Different filters can be applied to different portions of the image. For example, a first portion of an image can be filtered to sharpen the first portion, and a second portion of the image can be filtered with an affine transform filter and noise reduction. Any number of different filters can be applied to the image and / or each patch.

[0039] Once filters have been applied, changes in pixel intensity gradients across adjacent pixels can be identified. Large changes in intensity between adjacent pixels can indicate the presence of an edge. For example, a first pixel having a high intensity value that is immediately adjacent to a pixel having a low intensity value can provide an indication that the first pixel is part of an edge. In some cases, pixels that do not belong to a portion of an edge can be suppressed (e.g., set to a predetermined red / green / blue value, such as black, where red = 0, blue = 0, and green = 0, or any predetermined red / green / blue value). Edge detection operators such as the Roberts cross operator, the Prewitt operator, the Sobel operator, etc. can be used as part of the identification of pixel intensity gradients.

[0040] A non-maximum suppression process can be used to suppress pixels that do not strongly correspond to an edge. The non-maximum suppression process assigns an edge intensity value to each pixel identified as part of an edge using pixel intensity gradients. For each pixel identified as part of an edge, the edge intensity value of the pixel can be compared to the edge intensity values of the eight surrounding pixels of the pixel. If the edge intensity value of the pixel is higher than the edge intensity values of the surrounding pixels (e.g., a local maximum), then the surrounding pixels are suppressed. The non-maximum suppression can be repeated for each pixel in the entire image.

[0041] A double thresholding process can then be performed to remove noise and / or false edge pixels carried through the application of the previous image processing techniques applied herein. Two pixel intensity thresholds can be defined, one high and one low. The thresholds can be used to assign an intensity characteristic to each pixel as strong or weak. Pixels comprising intensity values above the high threshold can be assigned a strong intensity characteristic, where pixels comprising intensity values between the high threshold and the low threshold can be assigned a weak intensity characteristic. Pixels comprising intensity values below the low threshold can be suppressed (e.g., in the same manner as described above).

[0042] A hysteresis process can then be performed to remove pixels having a weak intensity characteristic (i.e., weak due to noise, color variation, etc.). For example, a local statistical analysis (e.g., connected component analysis, etc.) can be performed on each pixel having a weak intensity characteristic. Pixels having a weak intensity characteristic that are not surrounded by pixels comprising a strong intensity characteristic can be suppressed. The remaining pixels (e.g., the unsuppressed pixels) after the hysteresis process only comprise those pixels that are part of an edge. While the above five processing procedures are described in a particular order, each procedure can be performed any number of times (e.g., repeated) and / or in any order without departing from the spirit or scope of the present disclosure. In some cases, only a subset of the five procedures need be performed on an image. For example, the image processing can perform the identification of pixel intensity gradients procedure without first performing the filtering procedure. In some cases, an image can be received that has been partially processed (e.g., one or more of the above procedures have already been performed). In these cases, one or more additional procedures can be performed to complete the image processing.

[0043] In some cases, signal processing can be performed on the image (e.g., similar to radio frequency signals). The image can be transformed into a frequency domain (e.g., using a Fourier transform, etc.) to represent frequencies at which certain pixel characteristics (e.g., pixel intensity, RGB values, etc.) are present in the image. In the frequency domain, one or more filters (such as, but not limited to, a Butterworth filter, a bandpass, etc.) can be applied to the image (e.g., during or after pre-processing, edge detection, etc.) to suppress or alter certain frequencies. Suppressing certain frequencies can reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels of certain colors or color gradients, normalize color gradients, etc. A high-pass filter can reveal edges in the image (e.g., a sharp contrast in color and / or intensity between adjacent pixels), while a low-pass filter can blend edges (e.g., blur). Image padding can be performed prior to signal processing to improve signal processing techniques. In some cases, different portions of the image and / or patches can be processed differently, with some being processed with a high-pass filter and others being processed with a low-pass filter. In some cases, thresholds (e.g., cutoff frequencies of high-pass or low-pass filters) can be modified for different portions of the image (e.g., based on processing one or more previous images of the image, machine learning, etc.).

[0044] Signal processing can also determine other characteristics of the image that relate to the pixels of the image, such as coherence (e.g., for edge detection, segmentation, pattern analysis, etc.). The relationships between the pixels can be used to further refine edge detection and / or identify structural characteristics of the depicted content within the image. For example, coherence can be used to identify relevant portions of the image (e.g., portions of the same object corresponding to a keyword) from irrelevant portions of the image.

[0045] In some cases, an image segmentation operation can be performed instead of, during, or after edge detection. Image segmentation assigns a coherence value (e.g., as described above) to each pixel of the image. Image segmentation can use coherence values identified during edge detection or one or more other operations, such as a Sobel model, graph partitioning, etc. The coherence values represent pixel labels that can be used to group pixels according to a common label (e.g., an object depicted in the image). The coherence values can be used to identify the location, orientation, and shape of objects within the image, which can be used to identify the object via a lookup table. If the object cannot be identified using coherence and image characteristics, the processed image can be passed to a classifier (awaiting further image processing).

[0046] The classifier can be a predictive machine learning model such as, but not limited to, a neural network, etc. The classifier can be trained to label objects / subjects corresponding to one or more keywords. The classifier can be trained using local image data 136 or from one or more training data sets received from training server 148. The neural network can be trained using supervised or unsupervised learning. For example, in supervised learning, a set of labeled images can be input into the neural network. The neural network can define a set of features (e.g., a set of image characteristics indicative of the presence of a label). The classifier can use the set of features when unlabeled data is input into the neural network. In unsupervised learning, images can be passed as input to the classifier with labels. The classifier can learn a set of features through analysis of the images. If the accuracy of the classifier is below a threshold, the classifier can be retrained using supervised or unsupervised learning using the training data sets and / or any additional labeled or unlabeled images.

[0047] In some cases, prior to passing image data as input, it can be determined whether the generative adversarial network is trained to generate images corresponding to the image data. If the generative adversarial network has not been trained, the machine learning core 124 can train the generative adversarial network on demand based on the particular processed image and / or labeled image data that can be passed as input. For example, if the image data corresponds to a race track and a sedan, and the generative adversarial network has not been trained using images of race tracks and / or sedans, the generative adversarial network can not be able to generate images that look like real images (e.g., do not look like they were computer generated).

[0048] The machine learning core 124 can attempt to locate and build a training data set in the local image data 136. If not found, the machine learning core 124 can generate and execute a query to the training server 148. The training server 148 can use image data 152 or image data from previous content distribution operations 156 to obtain images to train the generative adversarial network. For example, if the company previously performed a content distribution operation associated with a race track and a sedan, the machine learning core 124 can access the training data and / or generated images from that content distribution campaign to train the generative adversarial network.

[0049] In some cases, multiple generative adversarial networks can be provided, where each generative adversarial network is trained to generate images corresponding to a particular one or more keywords. For example, image data (e.g., input data) can be classified by a classifier to determine a type of image to generate (e.g., a particular subject, object, setting, etc.). A particular generative adversarial network of the multiple generative adversarial networks can be selected based on the particular generative adversarial network having been trained to generate images corresponding to a class identified by the classifier. The multiple generative adversarial networks can be organized in a hierarchy based on the specificity of the keywords on which each generative adversarial network is trained.

[0050] The generative adversarial network includes a generator neural network 140 and a discriminator neural network 144, which operate in a feedforward and backpropagation learning mode after training. The generator network 140 uses one or more layers of transformation / weight functions to identify features of training images corresponding to particular characteristics (e.g., background, foreground, keywords, objects, subjects, etc.). During supervised learning, the features can be compared to labels of the images to determine accuracy of the generator network 140. Training can continue until a threshold accuracy has been reached. If the accuracy is below the threshold, the generator network 140 can be retrained. Once trained, the generator network 140 can be used to generate a series of images corresponding to an input one or more keywords. The generator network 140 attempts to generate images that appear indistinguishable from (or close to) real images.

[0051] Once the generator neural network 140 is trained, the discriminator neural network 144 can be trained. The discriminator network 144 attempts to predict whether an image was generated by the generator neural network 140 (e.g., a fake image). During training, a set of training images generated by the generator neural network 140 and images received from one or more other sources can be input to the discriminator network 144. The training data can be labeled (for supervised learning) or unlabeled (for unsupervised learning). The discriminator network 144 can use the images passed as input to improve accuracy of subsequent predictions. Once trained, the discriminator network 144 can begin to predict images generated by the generator network 140. The predictions can be passed back to the generator neural network 140 to improve the generated images. This process can continue until a predetermined error rate is detected. For example, the error rate can correspond to a percentage of images generated by the generator neural network 140 but predicted as real by the discriminator neural network 144 (e.g., the discriminator neural network 144 predicts that the image was not generated by the generator neural network 140). Since the two networks operate simultaneously in a feedforward and backpropagation, the model continually obtains more realistic images generated by the generator network 140.

[0052] In some cases, in addition to the prediction, the discriminator network 144 can output other characteristics of the image. For example, the discriminator network 144 can assign a confidence value indicative of a confidence level of the prediction. The discriminator network 144 can also assign an accuracy value indicative of how close the generated image is to corresponding to one or more keywords. For example, for the two keywords “race track” and “sedan,” if the generated image depicts a race track but not a sedan, then it can be assigned a lower accuracy value than a generated image that simultaneously depicts a race track and a sedan. The images can be ranked according to accuracy (and / or prediction) to determine which generated images should be transmitted to the client devices 160-1, 160-2, 160-3,... 160-n.

[0053] Once the generator 140 generates an image that fools the discriminator neural network 144 (e.g., looks realistic enough that the discriminator neural network 144 predicts the image as real), the image can be further processed (e.g., using one or more of the previously described image processing operations) and / or the image can be distributed over a network to one or more client devices 160-1, 160-2, 160-3,... 160-n.

[0054] Figure 2 is a block diagram illustrating a process flow for generating a content distribution image in accordance with at least one aspect of the present disclosure. The process flow can be initiated with a user interface 204 that enables a user to define a content distribution operation. The user interface includes a plurality of fields for the content distribution operation, including a title field 208 and a text field 212. The text field 212 enables receiving user input, such as an alphanumeric string that describes the content distribution operation. Once received, an upload button 216 can be selected to transmit the alphanumeric string entered in the title 208 and text 212 to one or more machine learning cores for content generation. In some cases, at least one machine learning core can be executed locally on the same device as the user interface 204. In other cases, the user interface 204 can be presented on a different device than the machine learning cores. The user interface 204 can include one or more additional fields that allow the user to indicate particular characteristics of the content distribution operation, including but not limited to a receiving device (or user), one or more communication protocols to transmit the content once generated, a threshold accuracy for each generated image, and the like.

[0055] The text (and any other characteristics) can be passed to a text processing pipeline 228, which can parse the text to determine one or more keywords associated with the text field 212. The text can be parsed via a keyword lookup table, a machine learning model, or any other operation as described above. At block 232, the one or more keywords identified from the text field 212 at the text processing pipeline 228 can be used to identify image data corresponding to the one or more keywords. For example, block 232 identifies an image or a portion of an image corresponding to each of the one or more keywords. In some cases, a predefined number of images for the image data can be defined. In these cases, block 232 can first identify a first set of images corresponding to each of the one or more keywords. If the first set of images is not equal to or greater than a predefined threshold, then images corresponding to all but one of the one or more keywords can be added to the first set of images. If the first set of images is still not equal to or greater than the predefined threshold, then images corresponding to all but one of the one or more keywords can be added to the first set of images, and so on until the predefined number of images for the image data is obtained.

[0056] At block 236, one or more image processing techniques and / or object recognition can be performed on the image data. For example, image segmentation or edge detection can be used to modify the image data to distinguish image portions corresponding to the at least one keyword from image portions not corresponding to the one or more keywords. The portions of each image corresponding to the at least one keyword can be labeled with the at least one keyword. In some cases, further image processing can be performed to remove (or otherwise label) image portions not corresponding to the at least one keyword.

[0057] The labeled and processed image data can be passed to block 240, which can use the labeled and processed image data to generate a training data set for the generative adversarial network 252-256. For example, block 240 can determine whether the generative adversarial network 252-256 is trained to generate images corresponding to the one or more keywords. If not, a training data set corresponding to the one or more keywords can be generated to dynamically train the generative adversarial network 252-256. For example, the training data set can be obtained from the training server 148, which can have access to image data from previous content distribution 156 and / or one or more remote networks (e.g., an external network, the Internet, etc.).

[0058] In some cases, images from the tagged and processed image data can be output to the user interface 204. For example, a representation of some or all of the tagged and processed image data can be transmitted to the user interface 204. The user can then review images from the tagged and processed image data. In some cases, the user can discard one or more images from the tagged and processed image data (e.g., those that do not pass user review). In some cases, the user can supplement the tagged and processed image data by, for example, uploading additional images 224 to the tagged and processed image data.

[0059] The tagged and processed image data can be passed as input to a generative adversarial network including a generator network 252 and a discriminator network 256. The generator network 252 can use the tagged and processed image data to generate a set of new artificial images that can be evaluated by the discriminator network 256. The discriminator network 256 can generate a prediction indicating whether the generated images are real or artificial. Images predicted to be real can be output to block 260, where the images are packaged for transmission to one or more remote clients or client devices. Images predicted to be artificial can be passed back to the generator network 252 as backpropagation in the generative adversarial network to improve the accuracy of future image generation and subsequent tagging.

[0060] Figure 3 An example of a graphical user interface associated with generating content distribution images is illustrated in accordance with at least one aspect of the present disclosure. The graphical user interface 304 includes one or more frames, each presenting different information to the user. For example, the upper frame can receive user input defining and / or modifying aspects of the content distribution operation. After the text defining the content distribution operation is parsed into one or more keywords representing the context of the content distribution operation, the user interface 304 can be presented to the user to modify aspects of the content distribution operation definition. For example, the user can toggle the operator 308 to display the text defining the content distribution operation.

[0061] The user can toggle the operator 312 to review some or all of the reference images provided to the machine learning core that generates the images. The reference images include the user-provided images and / or images from image data identified by the machine learning core. The detected keywords / tokens 316 can include a representation of each keyword and / or related token identified from the provided text. The user can select or deselect one or more keywords to generate new image data or modify the image data (if already generated). The keyword / token confidence filter 320 can be used to indicate to the user which keywords can highly correspond to the provided text and which keywords can have a low confidence of correspondence. In some cases, the keyword / token confidence filter 320 can include one or more threshold filters that automatically remove keywords below a particular confidence value.

[0062] The images generated by the generative adversarial network can be presented in real-time (e.g., as the images are generated) via the user interface 304. The user interface 304 can represent each generated image 324 as a full image or a thumbnail that can be presented as a full image when an input is received to select the image. In some cases, the images presented via the user interface 304 can be all of the images generated by the generative adversarial network. In other cases, the images presented via the user interface 304 can include only those images predicted by the discriminator network to not be generated by the generator network. Each image 324 can include an accept button 328 and a reject button 332. An input can be received to accept 328 the image for distribution to one or more clients or client devices or reject 332 the image to discard the image. In some cases, the selection of accept 328 or reject 332 can be assigned as a label for the image 324. The images and assigned labels can be backpropagated to the generative adversarial network to add additional training layers, which can further improve the image generation for the content distribution operation.

[0063] Figure 4 A flow diagram for generating a content distribution operation is illustrated in accordance with at least one aspect of the present disclosure. At block 404, a user input can be received to generate content for a content distribution (e.g., marketing) operation associated with an entity (e.g., a company selling a product / service or a company providing a content distribution service). The user input can include one or more alphanumeric strings that describe the content distribution operation. The one or more alphanumeric strings can include natural language text, textual commands in a particular pattern (such as key / value pairs), and the like. The user input can also include one or more characteristics of the content distribution operation, such as a type of image to generate, a format of the image, a client or client device to receive the image, a communication protocol to transmit the image, a combination thereof, and the like.

[0064] At block 408, the user input can be parsed to identify one or more keywords associated with a context of a content distribution operation. Parsing the user input can include a keyword analysis (e.g., identifying particular words and matching them to a keyword lookup table), a natural language machine learning model (as described above), and the like. Each of the one or more keywords indicates a portion of content to be depicted in a generated image. For example, a keyword can correspond to an object, a context, an action, a subject, a setting (e.g., a location, such as a beach or a school), a combination thereof, and the like.

[0065] At block 412, image data corresponding to at least one of the one or more keywords can be received from one or more databases (or a network such as an external network or the Internet). In some cases, the image data can include structured or unstructured data characterizing or representing an image associated with the at least one keyword. In other cases, the image data includes one or more images each corresponding to the at least one keyword in addition to the structured or unstructured data. In yet other cases, the image data can include only the one or more images.

[0066] At block 416, one or more image processing operations can be performed on the image data to derive processed image data. The one or more image processing operations include operations that can modify or transform an image (e.g., affine transformations, edge detection, image segmentation, and the like). The one or more image processing operations can also include object detection. Object detection can use a trained neural network to identify one or more objects, subjects, settings, actions, contexts, combinations thereof, and the like in an image. For example, for an image of a sedan racing on a track, a trained neural network can label portions of the image corresponding to an object (e.g., the sedan), image portions corresponding to a setting (e.g., the track), and can label a context (e.g., such as day / night, weather, and the like). The one or more image processing operations can also include removing portions of an image that do not correspond to the at least one keyword.

[0067] The one or more image processing operations can also include labeling a portion of a particular image of the image data that is to remain unmodified by a generative adversarial network generating an image similar to the particular image. For example, an object can be labeled to remain unmodified such that when input into a generative adversarial network, the generative adversarial network can generate a different image including the same object. The generative adversarial network can generate images of the object in different contexts (e.g., different settings, locations, backgrounds, foregrounds, weather, and the like) with different other objects, and the like.

[0068] One or more image processing operations can also include labeling a portion of a particular image of the image data to be modified by the generative adversarial network. For example, an object can be labeled as not to be modified such that when input into the generative adversarial network, the generative adversarial network can generate different images that are unique in the modification of the object. If the object is a sedan depicted on a race track, labeling the sedan in this manner can enable the generative adversarial network to generate images that are unique in the modification of the sedan. For example, the generative adversarial network can generate images in which the sedan is a different color or is a different make or model. The sedan can be replaced with a different object, such as a truck, a bicycle, etc. If the object corresponds to a human, the human characteristics of the human can be modified, such as but not limited to, age, race, gender, height, weight, appearance (e.g., eye color, hair color, clothing, etc.), combinations thereof, etc.

[0069] At block 420, a generative adversarial network can be executed on the processed image data, including:

[0070] At block 424, a first neural network can be executed on the image data to generate a first set of new images, each new image corresponding to at least one of the one or more keywords. The first neural network can be executed using the one or more keywords as input (which is from the processed image data) or using one or more images. The first set of images can be completely computer-generated synthetic images, but generated to appear as if the images are real (e.g., not computer-generated). The first neural network can be a generative neural network that is part of the generative adversarial network. The generator neural network can be trained using a second neural network such that the first set of images can be generated based on a likelihood that each image in the first set of images is not detected by the second neural network as having been generated by the first neural network.

[0071] For example, the second neural network can generate a prediction for each ancestral image (e.g., an image previously generated by the first neural network) of a set of training images. The prediction can correspond to a likelihood of whether the ancestral image was generated by the first neural network. The set of training images can include the ancestral images and images received from one or more other sources (e.g., previous content distribution operations, the internet, images from the same entity, etc.). The one or more other sources can include image data, one or more images received from a local or remote database, the internet, etc. The images received from the one or more other sources can be real images (e.g., images that are not computer-generated, images that are computer-generated (such as by another generative adversarial network), or combinations thereof.

[0072] The second neural network can be a discriminator neural network representing another portion of a generative adversarial network. The goal of the generative adversarial network can be to generate images that look like real images, which induces the second neural network to predict the generated images as "real" images rather than images generated by the generator neural network. The goal of the second neural network is to correctly predict whether an image was generated by the first neural network and pass that prediction back to the generator neural network (e.g., backpropagation learning). In some cases, the second neural network can assign a range of values rather than a Boolean value (e.g., real or generated by the generator neural network). For example, the range of values can indicate a probability or a confidence that an image was generated by the generator neural network. In some cases, the first neural network and the second neural network can be the same neural network, while in other cases, the first neural network and the second neural network can be different neural networks.

[0073] The predictions of the second neural network can be used to update the first neural network. The predictions assigned to images generated by the first neural network can provide the first neural network with an indication of how well the images fooled the second neural network into predicting the images as being or not being generated by the first neural network. This information can be used to modify one or more layers of the first neural network so that the first neural network can produce images that the second neural network is more likely to predict as real images (e.g., more likely to look like real images rather than computer-generated images).

[0074] Once trained, the first neural network can be executed to generate images, each image corresponding to at least one of the one or more keywords. The generated images can be used to continually train the first neural network to continually improve the subsequently generated images (e.g., so that the images are less likely to look like they were computer-generated).

[0075] At block 428, the user interface can display the first set of images and the second set of images (e.g., side-by-side, within the same window, etc.). The second set of images can include images that were previously part of one or more content distribution operations or images that were designated by an entity (e.g., a company, a user within a company, etc.) as being associated with one or more keywords. In some cases, the first set of images can be transmitted to the client or client device (e.g., based on the user input of block 404). The user interface can enable the user to review each image in the first set of images and compare those images to the images in the second set of images. In some cases, the user can accept or reject individual images in the first set of images. In these cases, the user’s acceptance or rejection selection can be assigned as a label to the image. The images and corresponding labels can be passed to the first neural network to further train the first neural network. In some cases, images labeled as authentic can be removed prior to further training. This can enable backpropagation learning in a second layer of the generative adversarial network, where the second neural network and the user can each individually further update or train the first neural network.

[0076] Each block of the flowchart of FIG. 4 can be performed in any particular order or at any particular frequency (such as, for example, in sequence, out of sequence, one or more times (in sequence or out of sequence)). Figure 4

[0077] In the above description, specific details are given to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For example, circuits can be shown in block diagrams in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0078] Implementations of the techniques, blocks, steps and means described above can be implemented in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above and / or a combination thereof.

[0079] ​Furthermore, it should be noted that embodiments can be described as processes depicted as flowcharts, flow diagrams, swim lane diagrams, data flow diagrams, structure diagrams, or block diagrams. Although the processes depicted can be described as sequential processes, many of the operations can be performed in parallel, or concurrently, or in an order other than that depicted. Furthermore, the order of the operations depicted in the processes can be rearranged. A process is terminated when its operations are completed, but could be terminated before, upon, or after completion of the operations, without departing from the scope of the processes. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0080] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting language, and / or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0081] For firmware and / or software implementations, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example software codes can be stored in memory. Memory can be implemented within the processor or external to the processor. As used herein the term "memory" refers to any type of long term, short term, volatile, nonvolatile, or other memory and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0082] Furthermore, as disclosed herein, the term "storage media" can represent one or more memories for storing data including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine readable mediums for storing information. The term "machine- readable medium" includes, without being limited to, portable or fixed storage devices, optical storage devices, and / or various other storage mediums capable of storing that which includes or carries (one or more) instructions and / or data.

[0083] While the principles of the present disclosure have been described above in connection with specific apparatuses and methods, it is to be clearly understood that this description is made only by way of example and not as limitation on the scope of the disclosure.

Claims

1. A method comprising: Receive user input corresponding to content distribution operations; Parse the user input to identify one or more keywords associated with the content distribution operation; Receive image data corresponding to the one or more keywords associated with the content distribution operation from one or more databases; One or more image processing operations are performed on the image data to export processed image data. The image processing operations include an object detection operation, which is used to mark portions of one or more objects in the image data that correspond to keywords in the one or more keywords. The marked portions of the image data provide an indication of a specific location in the image that corresponds to the one or more keywords. The generative adversarial network is trained at runtime using a training dataset to generate a trained generative adversarial network, the trained generative adversarial network including a trained generator neural network and a trained discriminator neural network, the training dataset including multiple images, wherein a portion of each of the multiple images corresponds to at least one of the one or more keywords; Executing the trained generative adversarial network on the processed image data to generate a first set of images for the content distribution operation, wherein executing the trained generative adversarial network includes: The trained generator neural network is applied to the processed image data to generate a set of generated images corresponding to the one or more keywords, wherein the set of generated images is generated at least in part based on the probability that each image in the set of generated images will not be detected by the trained discriminator neural network as having been generated by the generator neural network; and The trained discriminator neural network is executed on the set of generated images to generate a set of predictions for the set of generated images, the set of predictions indicating which images in the set of generated images are predicted by the trained discriminator neural network as real or as already generated by the generator neural network; The first set of images used for the content distribution operation corresponds to the images within the generated set of images that are predicted as real by the trained discriminator neural network; and A first set of images and a second set of images are displayed via a first user interface. The second set of images includes images that were previously part of one or more content distribution operations or images that were designated by an entity associated with the content distribution operation as usable for the content distribution operation.

2. The method of claim 1, wherein the one or more image processing operations include image segmentation.

3. The method of claim 1, further comprising: Based on the one or more keywords, it is determined that the generative adversarial network has not been trained to generate a new image corresponding to at least one of the one or more keywords; Transmit a request for the training dataset to one or more databases.

4. The method of claim 1, wherein the image data includes one or more images from a previous content distribution operation.

5. The method of claim 1, wherein executing the generative adversarial network further comprises: Receive input that assigns an accept or reject flag to each image in the first group of images; The generator neural network is trained at least in part based on the aforementioned labels; as well as Remove each image assigned a rejection label from the first set of images.

6. A system comprising: One or more processors; as well as A non-transitory computer-readable medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including: Receive user input corresponding to content distribution operations; Parse the user input to identify one or more keywords associated with the content distribution operation; Receive image data corresponding to the one or more keywords associated with the content distribution operation from one or more databases; One or more image processing operations are performed on the image data to export processed image data. The image processing operations include an object detection operation, which is used to mark portions of one or more objects in the image data that correspond to keywords in the one or more keywords. The marked portions of the image data provide an indication of a specific location in the image that corresponds to the one or more keywords. The generative adversarial network is trained at runtime using a training dataset to generate a trained generative adversarial network, the trained generative adversarial network including a trained generator neural network and a trained discriminator neural network, the training dataset including multiple images, wherein a portion of each of the multiple images corresponds to at least one of the one or more keywords; Executing the trained generative adversarial network on the processed image data to generate a first set of images for the content distribution operation, wherein executing the trained generative adversarial network includes: The trained generator neural network is applied to the processed image data to generate a set of generated images corresponding to the one or more keywords, wherein the set of generated images is generated at least in part based on the probability that each image in the set of generated images will not be detected by the trained discriminator neural network as having been generated by the generator neural network; and The trained discriminator neural network is executed on the set of generated images to generate a set of predictions for the set of generated images, the set of predictions indicating which images in the set of generated images are predicted by the trained discriminator neural network as real or as already generated by the generator neural network; The first set of images used for the content distribution operation corresponds to the images within the generated set of images that are predicted as real by the trained discriminator neural network; and A first set of images and a second set of images are displayed via a first user interface. The second set of images includes images that were previously part of one or more content distribution operations or images that were designated by an entity associated with the content distribution operation as usable for the content distribution operation.

7. The system of claim 6, wherein the one or more image processing operations include image segmentation.

8. The system of claim 6, further comprising: Based on the one or more keywords, it is determined that the generative adversarial network has not been trained to generate a new image corresponding to at least one of the one or more keywords; Transmit a request for the training dataset to one or more databases.

9. The system of claim 6, wherein the image data includes one or more images from a previous content distribution operation.

10. The system of claim 6, wherein executing the generative adversarial network further comprises: Receive input that assigns an accept or reject flag to each image in the first group of images; The generator neural network is trained at least in part based on the aforementioned labels; as well as Remove each image assigned a rejection label from the first set of images.

11. A computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method as described in any one of claims 1-5.

Images