Scalable architecture for automatic generation of content delivery images

Abstract

Disclosed are methods and systems for automated generation of content distribution images, including receiving user input corresponding to a content distribution operation. The user input may be parsed to identify keywords. Image data corresponding to the keywords may be identified. An image processing operation may be performed on the image data. A generative adversarial network may be performed on the processed image data, the execution including running a first neural network on the processed image data to generate first images corresponding to the keywords, the first images being generated based on a likelihood that each of the first images would not have been detected as generated by the first neural network. A user interface may display the first images along with second images, including images that were previously part of the content distribution operation or images designated by an entity as available for the content distribution operation.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 62 / 900,400, filed on September 13, 2019, entitled "SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES", and claims priority based on U.S. Non - Provisional Application No. 17 / 017,486, filed on September 10, 2020, entitled "SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES", the entire contents of which are incorporated herein by reference for all purposes.

[0002] Field The present invention generally relates to the automatic generation of content for content distribution operations, and more specifically, to a scalable architecture that utilizes neural networks for content distribution operations.

Background Art

[0003] Background Generating content for content delivery operations is resource-intensive and can be time-consuming. Often, the goal of a content delivery operation is to deliver specific content to a specific set of devices within a specific, relevant time interval. A company might employ a team of photographers, designers, and artists to acquire images for the operation, modify those images based on one or more criteria, convert the modified images into a format suitable for transmission or display, and then send the formatted images to the specific set of devices. Companies often spend several weeks and considerable resources designing a single content delivery. Furthermore, by the time the delivery operation begins, so much time may have passed that the generated content is no longer relevant to the specific set of devices receiving that content. Therefore, systems and methods are needed to reduce the resource requirements of content delivery operations. [Overview of the Initiative] [Means for solving the problem]

[0004] overview A method for automatically generating images for content delivery operations is disclosed. The method comprises receiving user input corresponding to a content delivery operation, parsing the user input to identify one or more keywords associated with the content delivery operation, receiving image data from one or more databases corresponding to the one or more keywords associated with the content delivery operation, and performing one or more image processing operations on the image data to derive processed image data, wherein the image processing operations provide a representation of a specific location in the image corresponding to the one or more keywords, and the method further comprises running a generative adversarial network on the processed image data to generate one or more images for the content delivery operation, wherein running a generative adversarial network is performed on the processed image data The generative adversarial network further comprises running a first neural network on the data, the first neural network generating a first set of images corresponding to one or more keywords, the first set of images being generated at least in part on the possibility that each image in the first set of images would not be detected as having been generated by the first neural network, and displaying the first set of images together with a second set of images via a first user interface, the second set of images including images that were previously part of one or more content delivery operations, or images designated as available for content delivery operations by entities associated with the content delivery operations.

[0005] Another aspect of the present disclosure includes a system comprising one or more processors and a non-temporary computer-readable medium containing instructions, wherein, when executed by the one or more processors, the instructions cause the one or more processors to perform the method described above.

[0006] Other aspects of this disclosure include a non-temporary computer-readable medium containing instructions, which, when executed by one or more processors, cause the one or more processors to perform the method described above.

[0007] Further areas of application of this disclosure will become apparent from the detailed description provided below. While the detailed description and specific examples illustrate various embodiments, it should be understood that they are for illustrative purposes only and are not necessarily intended to limit the scope of this disclosure.

[0008] This disclosure will be described in conjunction with the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram of a scalable architecture for generating content delivery images relating to at least one aspect of the present disclosure. [Figure 2] A block diagram showing a processing flow for generating content delivery images relating to at least one aspect of this disclosure. [Figure 3] This figure shows an example of a graphical user interface associated with the generation of content delivery images relating to at least one aspect of this disclosure. [Figure 4] This figure shows a flowchart for generating content delivery operations relating to at least one aspect of this disclosure. [Modes for carrying out the invention]

[0010] Detailed explanation In the attached drawings, similar components and / or features may have the same reference level. Furthermore, different components of the same type may be distinguished by dashes and second labels following the reference label. Where only the first reference label is used herein, the description applies to any one of the similar components having the same first reference label, notwithstanding the second reference label.

[0011] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of preferred exemplary embodiments will provide a practical explanation for realizing the preferred exemplary embodiments. It will be understood that various modifications can be made to the function and configuration of the elements without departing from the spirit and scope described in the appended claims.

[0012] A scalable architecture can be defined for generating content and content delivery actions using artificial intelligence. A generative adversarial network may be trained at runtime to generate specific images for content delivery actions. For example, user input can be parsed to determine one or more keywords that correspond to the intended context of a content delivery action. One or more keywords may represent, for example, one or more subjects of the content (e.g., people), characteristics of the subjects (e.g., race, gender, height, weight), one or more objects represented in the content, or settings (e.g., location).

[0013] In some cases, a generative adversarial network may not be trained to generate images for all possible keywords or keyword combinations. In such cases, image data (for example, from a database of previous content delivery operations and / or labeled images) may be generated at runtime using one or more keywords, which can then be used to train the generator neural network of the generative adversarial network. The generative adversarial network can then generate images on demand for any keyword or keyword combination.

[0014] The discriminator neural network of a generative adversarial network may be trained after the generator neural network. Training the discriminator neural network may involve generating images using the generator neural network and passing those images to the discriminator neural network along with other images, such as images from image data. The discriminator neural network may generate a prediction for each image indicating whether or not it was generated by the generator neural network. This prediction can be fed back to the generator neural network. The generator neural network uses the predictions from the discriminator neural network to improve image generation so that it produces images that are more likely to be labeled as real images by the discriminator neural network. Thus, the goal of the generator neural network is to trick the discriminator neural network into making false negatives (for example, predicting that an image was not generated by the generator neural network when it actually was).

[0015] After training the generator neural network and the discriminator neural network, the generator neural network may be run with one or more keywords to generate images for a content delivery campaign. In some examples, the discriminator neural network may continue to generate predictions that can be used by the generator neural network to further improve image generation. In such cases, these predictions may be used to filter the images generated by the generator neural network, for example, by removing images that the discriminator correctly predicts were generated by the generator neural network (e.g., images that do not look like real images). The generated images (or the remaining filtered images) can be sent as part of a content delivery campaign to one or more client devices (e.g., devices associated with the entity that requested the generated images), a content delivery server (e.g., a web host), or one or more user devices (e.g., devices associated with users that are the target of one or more client devices).

[0016] In some examples, user input defining a content delivery campaign may include natural language text, text commands, or pre-selected options. This input may describe the content to be delivered and / or the delivery mechanism (e.g., the specific device receiving the content, the communication protocol used to transmit the content, or the format of the content). The user input may be processed to determine one or more keywords. In some examples, natural language text or text commands can be processed using a rule system. The rule system may include a hierarchy of rules that, when applied to text, assign words or phrases to categories. Applying each rule may generate a class identifier that characterizes the word or phrase as corresponding to a specific category.

[0017] In some examples, if applying a rule characterizes a word as corresponding to two or more categories, the categories may be hierarchical so that a more specific or higher-priority category can be assigned to that word. For example, a first rule may identify a phrase referring to the first category, “mode of travel,” a second rule may identify a phrase referring to the second category, “car,” and a third rule may identify a phrase referring to the third category, “sedan.” Input text referring to a sedan triggers the application of all three rules, but the third rule is applied because it is the most specific. As a result, the keyword “sedan” is selected as the keyword for the phrase.

[0018] In some examples, a machine learning model may be used to identify keywords in the input. The machine learning model may be trained using training data derived from a set of previous content delivery operations, open-source databases, public databases, and / or manufactured data (e.g., data generated to train the machine learning model). A set of features (e.g., words or phrases) can be extracted from the training data. This set of features can then be passed as input to the machine learning model. The machine learning model may be trained using unsupervised, semi-supervised, or supervised learning. Once trained, the machine learning model may receive input text and output one or more keywords. In some examples, the machine learning model may also output a confidence score for each of the one or more keywords, indicating the probability that each keyword corresponds to the input text.

[0019] For example, a set of previous content delivery operations may be used to identify a set of input texts for training a machine learning model. The machine learning model is trained using supervised learning, where each input text is associated with a label (e.g., a keyword derived from an analysis of the input texts). The machine learning model may use the set of input texts to determine which words and / or phrases correspond to a particular category. In supervised learning, the machine learning model can adjust its internal processes to improve the accuracy of subsequent predictions by comparing the output generated by the model (e.g., predicted keywords) with the labels associated with the input texts. In unsupervised learning, the machine learning model may extract features from each input text in the set of input texts. The machine learning model may then use, for example, cluster analysis to identify relationships between related input texts in the set of input texts (e.g., those associated with the same one or more keywords) based on the extracted features. Once trained, the machine learning model may be run with inputs for a specific content delivery operation to classify the words and phrases in the inputs.

[0020] One or more keywords may be extracted from the classified input. One or more keywords may correspond to words or phrases in the input that represent the context of a content delivery operation. Alternatively, one or more keywords may correspond to categories assigned to words or phrases. In some examples, the confidence value assigned to each input text indicates the likelihood that the keyword was correctly selected for the input text. A threshold may be used to prevent one or more keywords from representing inappropriate context for a content delivery operation by discarding keywords associated with confidence values ​​below that threshold.

[0021] Using one or more keywords, image data to be input into the adversarial generation network can be obtained. A query may be generated using one or more keywords. The query may be executed to obtain image data from one or more databases and / or external networks. The image data may correspond to one or more keywords and may be associated with the context of the content delivery operation. In some examples, one or more queries may be executed in an iterative process to obtain image data from one or more databases and / or external networks. For example, if a first query to a first database does not return image data corresponding to one or more keywords, a second query may be generated and executed on another database. The process may continue with each successive query executed on a different database or external network. If neither the database nor the network returns image data corresponding to one or more keywords, a final query may be generated and executed to obtain image data from an open-source database, a public database, or the Internet.

[0022] In some examples, the image data may be processed before being input into the adversarial generation network. For example, one or more first keywords may correspond to cars and trucks, and one or more second keywords may indicate that the background of the car or truck is an indoor environment. The retrieved image data may include images of cars and trucks in various indoor and outdoor environments. If the image data corresponding to cars and trucks in an indoor environment exceeds a threshold (e.g., a specific amount of images and / or a specific volume of data), the image data corresponding to cars or trucks in an outdoor environment may not be used.

[0023] If the image data corresponding to cars and trucks in the indoor environment does not exceed the threshold, the image data corresponding to cars or trucks in the outdoor environment may be processed to enable the use of data while following one or more first keywords and one or more second keywords. For example, the image data may be processed by generating a display (such as a label) for each part of the image corresponding to one or more first keywords and / or one or more second keywords (or for the part of the image data not corresponding to one or more first keywords or one or more second keywords). In some examples, the part of the image corresponding to one of the keywords of one or more first keywords and one or more second keywords may be extracted from the part of the image not corresponding to one of the keywords of one or more first keywords and one or more second keywords. By generating a display for the relevant part (or irrelevant part) of the image, the adversarial generation network can be properly trained.

[0024] Image processing may include one or more operations that can be applied to modify the image data to better train the adversarial generation network. The one or more operations may be executed in any particular order. Image processing may include using object detection to label parts of one or more objects in the image, particularly parts of one or more objects corresponding to one or more keywords. In some examples, object detection / labeling may be performed by a neural network such as a convolutional neural network (CNN), region CNN, fast CNN, or you-only-look-once (YOLO). Once the objects in the image are labeled, the image may be further processed, for example, by deleting parts of the image not corresponding to one or more keywords to reduce the size of the image.

[0025] The modified image data may be passed as input to the generator network of a generative adversarial network. The generator network may be a neural network (e.g., a convolutional or deconvolutional neural network). The generator network uses a transformation function through a set of layers to derive a first set of images from the input image data. The first set of images may correspond to one or more keywords (e.g., the context of a content delivery campaign) and may be in a format suitable for delivery as part of a content delivery campaign. The first set of images may be passed to a discriminator network for evaluation.

[0026] The discriminator network may generate a prediction for each image in a first set of images indicating whether it was generated by the generator network or is a real image. The discriminator network can send this prediction back to the generator network to facilitate further training of the generator network (e.g., backpropagation learning). In some examples, each image generated by the generator network may be accepted or rejected by the user before or after being passed to the discriminator network. For example, each image may be labeled as accepted or rejected. In some examples, the images and their corresponding labels may be passed to the generator network as a second layer of backpropagation learning. In such cases, the labels can be used to further improve image generation.

[0027] The predictions assigned to each image may be used to determine which images should be included in the content delivery campaign and which images can be discarded (or used for further training of the adversarial generative network). For example, a content delivery campaign may include a second set of images, which is a subset of a first set of images. The second set of images may include images from the first set of images that have been assigned predictions from a discriminator network indicating that they were not generated by the generator network. The second set of images may then be sent to one or more client devices in accordance with the content delivery campaign. Alternatively or in addition to this, in some examples, the content delivery operation may include physical representations of the images in the second set of images. For example, the images may be printed and mailed to one or more recipients, or delivered in another way.

[0028] Figure 1 is a block diagram of a system 100 for generating content delivery images relating to at least one aspect of the present disclosure. The system 100 includes one or more computing devices 104-1, 104-n that can be used to automatically generate content for content delivery operations. Each computing device 104-1 to 104-n may include the same or different hardware components as shown in 104-1. For example, computing device 104-1 may include hardware components for performing a specific aspect of content generation (e.g., text analysis, image processing, content generation, content evaluation, or a combination thereof).

[0029] The computing device 104-1 includes one or more processors (e.g., CPU 108) coupled to memory 112 via a bus. A user interface 116 for interacting with the user may be presented on the display 120. The user may operate input / output devices to define new content delivery behavior within the user interface 116. The input may include one or more attributes and text that characterize the new content delivery. For example, the text may be natural language text, an alphanumeric string, or an alphanumeric command.

[0030] The definition of a content delivery action may be passed to a machine learning core 124. The machine learning core 124 may be one or more artificial intelligence modules, each running together to generate images that conform to the definition of a content delivery action. The definition may also be passed to a parser 128 within the machine learning core 124. The parser 128 may decompose the text input to determine one or more keywords of the definition. One or more keywords may correspond to the context of the definition, such as the subject or object of the content being delivered. The parser 124 converts the text into a set of tokens (e.g., each alphanumeric character, each word, and / or phrase based on the settings of the parser 124). The set of tokens may then be individually labeled and combined (e.g., using a sliding window of a predefined character length). This labeling may be based on matching the tokens to predefined keywords, a hierarchy of rules, or by using machine learning. An example machine learning implementation might output only those tokens that can be associated with a context (e.g., nouns, verbs, adjectives with nouns, etc.), while ignoring or discarding certain tokens, such as those associated with articles (e.g., words with little contextual value).

[0031] The machine learning core 124 may acquire image data using one or more extracted keywords. In some examples, the machine learning core 124 may retrieve image data corresponding to one or more keywords from local image data 136. In other examples, the machine learning core 124 may generate and execute queries to external data stores, such as computing devices 104-n or image data 152. If data cannot be supplied within the network, the machine learning core 124 may generate and execute queries to retrieve image data from the internet (e.g., using a search engine, web crawler, etc.). The image data may include one or more images that show a subject / object corresponding to one or more of the keywords (i.e., context or part thereof). For example, one or more keywords may include "racetrack" and "sedan," and the image data may include an image of a racetrack, an image of a sedan, and / or an image that includes both a sedan and a racetrack.

[0032] The machine learning core 124 may pass the image data to the image processor 128 to process the image into a format more suitable for one or more neural networks, such as a generative adversarial network. Image processing may include one or more actions such as modifying the image and / or labeling the content of the image. Image processing may include applying one or more filters, signal or frequency analysis, edge detection, orientation correction such as affine or Euclidean transforms, or a combination thereof.

[0033] For example, edge detection modifies an image to highlight the edges of objects (or subjects) within the image. Edge detection can be initiated using filtering techniques that apply one or more filters to the image. Filters may modify the image by blurring, sharpening, transforming (e.g., one or more affine or Euclidean transforms, but not limited to these), and / or other means. Filters may reduce image noise, for example, by removing image artifacts and / or other parts of the image that do not correspond to one or more keywords.

[0034] In some examples, parts of an image may be processed more than other parts of the image. For example, part of the image may appear blurred while another part is sharpened. Different filters may be applied to different parts of the image, and in addition, different sets of filters may be applied to different parts of the image. Different filters may be applied to different parts of the image. For example, a first part of the image may be filtered to sharpen it, and a second part of the image may be filtered using an affine transform filter and noise reduction. Any number of different filters may be applied to the image and / or each patch.

[0035] Once the filter is applied, changes in the pixel intensity gradient across adjacent pixels can be identified. Large changes in intensity between adjacent pixels may indicate the presence of an edge. For example, a first pixel with a high intensity adjacent to a pixel with a low intensity may indicate that the first pixel is part of an edge. In some examples, pixels that are not part of an edge may be suppressed (for example, they may be set to predetermined red / green / blue values ​​such as black where red=0, blue=0, and green=0, or to any predetermined red / green / blue value). Edge detection operators such as the Roberts cross operator, Prewitt operator, Sobel operator, and / or others may be used as part of the pixel intensity gradient identification.

[0036] Non-maximum suppression may be used to suppress pixels that do not strongly correspond to edges. Non-maximum suppression assigns an edge intensity value to each pixel identified as part of an edge using a pixel intensity gradient. For each pixel identified as part of an edge, its edge intensity value may be compared to the edge intensity values ​​of its eight surrounding pixels. If the pixel has a higher edge intensity value than the surrounding pixels (e.g., maximal pixels), the surrounding pixels are suppressed. Non-maximum suppression may be repeated for every pixel in the entire image.

[0037] Next, dual thresholding may be performed to remove noise and / or spurious edge pixels transmitted through the application of previous image processing techniques as applied herein. Two thresholds for pixel intensity may be defined, one high and the other low. Using these thresholds, each pixel may be assigned an intensity property as strong or weak. Pixels with intensity values ​​higher than the high threshold may be assigned a strong intensity property, and pixels with intensity values ​​between the high and low thresholds may be assigned a weak intensity property. Pixels with intensity values ​​below the low threshold may be suppressed (for example, in the same manner as described above).

[0038] Next, hysteresis processing may be performed to remove pixels with weak intensity properties (i.e., weak due to noise, color unevenness, etc.). For example, a local statistical analysis (e.g., connected component analysis) may be performed on each pixel with weak intensity properties. Pixels with weak intensity properties that are not surrounded by pixels with strong intensity properties may be suppressed. The remaining pixels after hysteresis processing (e.g., unsuppressed pixels) include only pixels that are part of an edge. Although the above five processing steps have been described in a specific order, each process may be performed any number of times (e.g., iterated) and / or in any order without deviating from the spirit or scope of this disclosure. In some examples, it may be necessary to perform only a subset of the five processes on the image. For example, the image processing may perform the pixel intensity gradient processing identification without first performing the filtering process. In some examples, the image may be received in a partially processed state (e.g., with one or more of the above processes already performed). In such cases, one or more additional processes may be performed to complete the image processing.

[0039] In some examples, signal processing may be performed on images (similar to radio frequency signals, for example). Images may be transformed into the frequency domain (for example, using a Fourier transform) to represent the frequencies at which certain pixel characteristics (e.g., pixel intensity, RGB values, and / or others) exist in the image. In the frequency domain, one or more filters (such as, but not limited to, Butterworth filters, bandpass filters, and / or others) may be applied to the image (e.g., during preprocessing, edge detection, or after) to suppress or modify certain frequencies. Suppressing certain frequencies can reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels of a particular color or color gradient, normalize color gradients, and / or others. High-pass filters may highlight edges in an image (e.g., sharp color and / or intensity contrast between adjacent pixels), while low-pass filters may blend edges (e.g., blur them). Image padding can be performed before signal processing to improve signal processing techniques. In some examples, different parts and / or patches of an image may be processed differently, such that some parts are processed with a high-pass filter and others with a low-pass filter. In some examples, thresholds (e.g., cutoff frequencies for high-pass or low-pass filters) may be modified for different parts of the image (e.g., based on image processing, machine learning, and / or other methods for one or more previous images).

[0040] Signal processing may also determine other image properties, such as coherence (used in edge detection, segmentation, pattern analysis, etc.), which identifies relationships between pixels. Pixel relationships can be used to further improve edge detection and / or identify structural properties of what is shown in the image. For example, coherence can be used to distinguish related parts of an image (e.g., parts of the same object corresponding to a keyword) from unrelated parts of the image.

[0041] In some examples, instead of edge detection, image segmentation may be performed during or after edge detection. Image segmentation assigns a coherence value (e.g., as described above) to each pixel of the image. Image segmentation may use coherence values ​​identified during edge detection or during one or more other operations such as Sobel modeling or graph segmentation. The coherence value represents a pixel label that can be used to group pixels according to a common label (e.g., an object shown in the image). The coherence value may also be used to identify the location, orientation, and shape of objects in the image, and these locations, orientations, and shapes can be used to identify objects via a lookup table. If objects cannot be identified using coherence and image properties, the processed image (awaiting further image processing) may be passed to the classifier.

[0042] The classifier may be a predictive machine learning model, such as a neural network. The classifier may be trained to label objects / subjects corresponding to one or more keywords. The classifier can be trained using local image data 136 or from one or more training datasets received from the training server 148. The neural network may be trained using supervised or unsupervised learning. For example, in supervised learning, a set of labeled images may be input to the neural network. The neural network may define a feature set (for example, a set of image properties that indicate the presence of labels). The feature set may be used by the classifier when unlabeled data is input to the neural network. In unsupervised learning, images may be passed to the classifier as input along with labels. The classifier may learn the feature set through image analysis. If the accuracy of the classifier falls below a threshold, the classifier may be retrained using supervised or unsupervised learning with the training dataset and / or any additional labeled or unlabeled images.

[0043] In some examples, it may be determined whether the generative adversarial network is trained to generate images corresponding to image data before the image data is passed as input. If the generative adversarial network is not trained, the machine learning core 124 may train the generative adversarial network on demand based on specific processed and / or labeled image data that may be passed as input. For example, if the image data corresponds to a racetrack and a sedan, and the generative adversarial network is not trained with images of the racetrack and / or sedan, the generative adversarial network may not be able to generate images that look like real images (e.g., that do not look computer-generated).

[0044] The machine learning core 124 may attempt to locate and construct a training dataset within the local image data 136. If nothing is found, the machine learning core 124 may generate and execute a query to the training server 148. The training server 148 may use the image data 152, or image data from a previous content delivery operation 156, to obtain images for training the generative adversarial network. For example, if a company has previously performed content delivery operations associated with racetracks and sedans, the machine learning core 124 may access training data and / or generated images from that content delivery campaign to train the generative adversarial network.

[0045] In some examples, multiple generative adversarial networks may be provided, each trained to generate images corresponding to one or more specific keywords. For example, image data (e.g., input data) may be classified by a classifier to determine the type of image to be generated (e.g., a specific subject, object, setting, etc.). A particular generative adversarial network may be selected from among the multiple generative adversarial networks based on a specific generative adversarial network trained to generate images corresponding to the categories identified by the classifier. The multiple generative adversarial networks may be organized hierarchically based on the specificity of the keywords in which each generative adversarial network is trained.

[0046] The adversarial generative network includes a generator neural network 140 and a discriminator neural network 144 that operate in feedforward and backward propagation learning patterns after training. The generator network 140 uses one or more layers of transform / weight functions to identify features of a training image that correspond to a specific property (e.g., background, foreground, keyword, object, subject, etc.). In supervised learning, the accuracy of the generator network 140 may be determined by comparing the features to the labels of the image. Training may continue until a threshold accuracy is reached. If the accuracy falls below the threshold, the generator network 140 may be retrained. Once trained, the generator network 140 may be used to generate a series of images that correspond to one or more keywords of the input. The generator network 140 attempts to generate images that appear indistinguishable from (or very close to) real images.

[0047] Once the generator neural network 140 has been trained, the discriminator neural network 144 may be trained. The discriminator network 144 attempts to predict whether an image was generated by the generator neural network 140 (for example, whether it is a fake image). During training, a training set of images generated by the generator neural network 140 and images received from one or more other sources may be input to the discriminator network 144. The training data may be labeled (in the case of supervised learning) or unlabeled (in the case of unsupervised learning). The discriminator network 144 can use the images passed as input to improve the accuracy of subsequent predictions. Once training is complete, the discriminator network 144 may begin predicting images generated by the generator network 140. These predictions can be fed back to the generator neural network 140 to improve the generated images. This process may continue until a predetermined error rate is detected. For example, the error rate may correspond to the percentage of images that are generated by the generator neural network 140 but predicted to be real by the discriminator neural network 144 (e.g., images that the discriminator neural network 144 predicted not to have been generated by the generator neural network 140). Since the two networks operate in both feedforward and backward propagation, the model always achieves more realistic images than those generated by the generator network 140.

[0048] In some examples, the discriminator network 144 may output other characteristics of the image in addition to the prediction. For example, the discriminator network 144 may assign a confidence value indicating the confidence level of the prediction. The discriminator network 144 may also assign a precision value indicating how well the generated image corresponds to one or more keywords. For example, for the keywords "racetrack" and "sedan," if the generated image shows a racetrack without a sedan, that image may be given a lower precision value than a generated image that shows both a racetrack and a sedan. Images may be ranked according to precision (and / or prediction) to determine which generated images should be sent to client devices 160-1, 160-2, 160-3, ..., 160-n.

[0049] If the generator 140 generates an image that deceives the discriminator neural network 144 (for example, an image that looks realistic enough for the discriminator neural network 144 to predict that it is real), the image may be further processed (for example, using one or more of the image processing operations described above), and / or the image may be delivered over the network to one or more client devices 160-1, 160-2, 160-3, ... 160-n.

[0050] Figure 2 is a block diagram illustrating a processing flow for generating a content delivery image relating to at least one aspect of the present disclosure. The processing flow may be initiated using a user interface 204 that allows a user to define a content delivery operation. The user interface includes several fields for the content delivery operation, including a title field 208 and a text field 212. The text field 212 allows for the reception of user input, such as an alphanumeric string describing the content delivery operation. Once received, the user may select an upload button 216 to send the alphanumeric string entered in the title 208 and text 212 to one or more machine learning cores for content generation. In some examples, at least one machine learning core may run locally on the same device as the user interface 204. In other examples, the user interface 204 may be presented on a different device from the machine learning core. The user interface 204 may include one or more additional fields that allow a user to indicate specific properties of the content delivery operation. Such properties include, but are not limited to, the receiving device (or user), one or more communication protocols for sending the generated content, or a threshold precision for each generated image.

[0051] The text (and any other properties) may be passed to the text processing pipeline 228, which may parse the text to determine one or more keywords associated with the text field 212. The text may be parsed via a keyword lookup table, a machine learning model, or any other operation as described above. In block 232, the one or more keywords identified from the text field 212 in the text processing pipeline 228 may be used to identify image data corresponding to one or more keywords. For example, block 232 may identify an image or part of an image corresponding to each of the one or more keywords. In some examples, a predefined amount of images may be defined for the image data. In such cases, block 232 may first identify a first set of images corresponding to each of the one or more keywords. If the first set of images is not equal to or greater than a predefined threshold, images corresponding to all but one of the one or more keywords may be added to the first set of images. If the first set of images is still not equal to or greater than a predefined threshold, images corresponding to all but one of the one or more keywords may be added to the first set of images, and so on, until a predefined amount of images is obtained for the image data.

[0052] In block 236, one or more image processing techniques and / or object identification may be performed on the image data. For example, the image data may be modified using image segmentation or edge detection to distinguish portions of the image corresponding to at least one of the keywords from portions of the image that do not correspond to one or more keywords. Each portion of the image corresponding to at least one keyword may be labeled with that at least one keyword. In some examples, further image processing may be performed to remove (or otherwise mark) portions of the image that do not correspond to at least one keyword.

[0053] Labelled and processed image data may be passed to block 240, which may use the labeled and processed image data to generate training datasets for generative adversarial networks 252-256. For example, block 240 may determine whether the generative adversarial networks 252-256 are trained to generate images corresponding to one or more keywords. If not, training datasets corresponding to one or more keywords may be generated to dynamically train the generative adversarial networks 252-256. For example, the training datasets may be obtained from a training server 148 that has access to image data from previous content delivery 156 and / or one or more remote networks (e.g., an external network, the internet, etc.).

[0054] In some examples, images from the labeled and processed image data may be output to the user interface 204. For example, some or all representations of the labeled and processed image data may be sent to the user interface 204. The user may then review the images from the labeled and processed image data. In some examples, the user may discard one or more images from the labeled and processed image data (for example, images that do not pass the user's review). In some examples, the user may supplement the labeled and processed image data by, for example, uploading additional images 224 to the labeled and processed image data.

[0055] Labelled and processed image data may be passed as input to a generative adversarial network, including a generator network 252 and a discriminator network 256. The generator network 252 may use the labeled and processed image data to generate a set of new artificial images that can be evaluated by the discriminator network 256. The discriminator network 256 may generate predictions indicating whether the generated images are real or artificial. Images predicted to be real may be output to block 260, where the images are packaged for transmission to one or more remote clients or client devices. Images predicted to be artificial can be fed back to the generator network 252 as backpropagation in the generative adversarial network to improve the accuracy of future image generation and subsequent labeling.

[0056] Figure 3 shows an example of a graphical user interface associated with the generation of a content delivery image relating to at least one aspect of the present disclosure. The graphical user interface 304 comprises one or more frames, each presenting different information to the user. For example, the upper frame may receive user input defining and / or modifying aspects of a content delivery operation. After the text defining the content delivery operation is parsed into one or more keywords representing the context of the content delivery operation, the user interface 304 may be presented to the user for modifying aspects of the definition of the content delivery operation. For example, the user can switch operator 308 to display text defining the content delivery operation.

[0057] The user can switch operator 312 to review some or all of the reference images supplied to the machine learning core that generates the images. The reference images include images supplied by the user and / or images from image data identified by the machine learning core. Detected keywords / tokens 316 may include representations of each keyword and / or associated token identified from the supplied text. The user may select or deselect one or more keywords to generate new image data, or (if already generated) modify the image data. A keyword / token confidence filter 320 may be used to show the user which keywords are likely to correspond strongly to the supplied text and which keywords may have a low correspondence confidence level. In some examples, the keyword / token confidence filter 320 may include one or more threshold filters that automatically remove keywords below a certain confidence level.

[0058] Images generated by the generative adversarial network may be presented in real time (for example, as soon as an image is generated) via the user interface 304. The user interface 304 may represent each generated image 324 as a complete image, or as a thumbnail image that can be presented as a complete image when an input to select an image is received. In some examples, the images presented via the user interface 304 may be all images generated by the generative adversarial network. In other examples, the images presented via the user interface 304 may include only images that the discriminator network predicts were not generated by the generator network. Each image 324 may include an accept button 328 and a reject button 332. Inputs may be received to accept (234) or reject (324) an image for delivery to one or more clients or client devices. In some examples, the accept 328 or reject 332 selection may be assigned to the image 324 as a label. Images and assigned labels can be backpropagated to the generative adversarial network to add additional training layers that can further improve image generation for content delivery operations.

[0059] Figure 4 shows a flowchart for generating a content delivery operation relating to at least one aspect of this disclosure. In block 404, user input may be received to generate content for a content delivery operation (e.g., marketing) associated with an entity (e.g., a company that sells products / services, or a company that provides content delivery services). The user input may include one or more alphanumeric strings that describe the content delivery operation. The one or more alphanumeric strings may include natural language text, or text commands in a specific schema such as key / value pairs. The user input may also include one or more properties of the content delivery operation, such as the type of image to generate, the format of the image, the client or client device that will receive the image, the communication protocol used when transmitting the image, or a combination thereof.

[0060] In block 408, user input may be parsed to identify one or more keywords associated with the context of the content delivery operation. Parsing user input may involve a keyword analysis (e.g., identifying specific words and matching them to a keyword lookup table) or a natural language machine learning model (as described above). Each of the one or more keywords represents a portion of the content shown in the generated image. For example, keywords may correspond to objects, context, actions, subjects, backgrounds, foregrounds, settings (e.g., locations such as a beach or a school), or combinations thereof.

[0061] In block 412, image data corresponding to at least one of one or more keywords may be received from one or more databases (or external networks or networks such as the Internet). In some examples, the image data may include structured or unstructured data that characterizes or represents images associated with at least one keyword. In other examples, the image data may include structured or unstructured data, plus one or more images, each corresponding to at least one keyword. In yet another example, the image data may include only one or more images.

[0062] In block 416, one or more image processing operations may be performed on the image data to derive the processed image data. One or more image processing operations may include operations that can modify or transform the image (e.g., affine transformation, edge detection, image segmentation, etc.). One or more image processing operations may also include object detection. Object detection may use a trained neural network to identify one or more objects, subjects, settings, actions, contexts, or combinations thereof in the image. For example, in the case of an image of a sedan racing on a racetrack, the trained neural network may label parts of the image corresponding to objects (e.g., the sedan), parts of the image corresponding to settings (e.g., the racetrack), and the context (e.g., day / night, weather, etc.). One or more image processing operations may also include removing parts of the image that do not correspond to at least one keyword.

[0063] One or more image processing operations may also include marking specific parts of image data to be left untouched by a generative adversarial network (JAD) that generates images similar to the specific image. For example, an object may be marked to be left untouched so that, when input to the JAD, the JAD can generate different images containing the same object. The JAD may generate images of the object in different contexts (e.g., different settings, locations, backgrounds, foregrounds, or weather) with different other objects, etc.

[0064] One or more image processing operations may also include marking specific parts of image data to be modified by a generative adversarial network. For example, an object may be marked not to be modified so that, when input to the generative adversarial network, the network can generate different images where the only difference is the object. If the object is a sedan shown on a racetrack, marking the sedan in this way allows the generative adversarial network to generate an image where the only difference is a modification of the sedan. For example, the generative adversarial network may generate an image where the sedan is a different color or a different make or model. The sedan could be replaced with another object, such as a track or a bicycle. If the object corresponds to a human being, human features such as age, race, sex, height, weight, appearance (e.g., eye color, hair color, or clothing), or combinations thereof, could be modified.

[0065] In block 420, a generative adversarial network can be run on the processed image data. This run includes the following:

[0066] In block 424, a first neural network may be run on image data to generate a first set of new images, each corresponding to at least one of a set of keywords. The first neural network may be run on one or more keywords as input (from the processed image data) or on one or more images. The first set of images may be synthetic images that are entirely computer-generated but appear to be real (e.g., not computer-generated). The first neural network may be a generator neural network as part of a generative adversarial network. The generator neural network may be trained with a second neural network so that the first set of images can be generated based on the probability that each image in the first set of images will not be detected by the second neural network as having been generated by the first neural network.

[0067] For example, the second neural network may generate a prediction for each ancestor image in the training set of images (e.g., an image previously generated by the first neural network). The prediction may correspond to the possibility that the ancestor image was generated by the first neural network or not. The training set of images may include ancestor images and images received from one or more other sources (e.g., previous content delivery operations, the internet, images from the same entity). One or more other sources may include one or more images received from image data, local or remote databases, the internet, etc. Images received from one or more other sources may be real images (e.g., images not generated by a computer, images generated by a computer such as another generative adversarial network, or a combination thereof).

[0068] The second neural network may be a discriminator neural network representing the other part of the generative adversarial network. The goal of the generative adversarial network may be to generate an image that looks like a real image, and use that image to trick the second neural network into predicting that the generated image is a "real" image and not one generated by the generator neural network. The goal of the second neural network is to accurately predict whether the image was generated by the first neural network and to feed that prediction back to the generator neural network (e.g., backpropagation). In some examples, the second neural network may be assigned a range of values ​​rather than a Boolean value (e.g., whether it is real or generated by the generator neural network). For example, the range of values ​​may represent the probability or confidence that the image was generated by the generator neural network. In some examples, the first and second neural networks may be the same neural network, and in other examples, the first and second neural networks may be different neural networks.

[0069] The first neural network may be updated using predictions from the second neural network. Predictions assigned to images generated by the first neural network may provide the first neural network with an indication of how well the image has deceived the second neural network into predicting that the image was generated by the first neural network or not. Using this information, one or more layers of the first neural network can be modified so that the first neural network generates images that the second neural network is more likely to predict as real images (for example, images that are more likely to look like real images rather than computer-generated images).

[0070] Once trained, the first neural network may be run to generate images for at least one of a set of keywords, each corresponding to a specific keyword. The first neural network can be continuously trained using the generated images to continuously improve subsequent generated images (for example, so that they are less likely to appear computer-generated).

[0071] In block 428, the user interface may display a first set of images together with a second set of images (for example, side by side or in the same window). The second set of images may include images that were previously part of one or more content delivery operations, or images that are associated with one or more keywords and have been designated by an entity (for example, a company, a user within a company, etc.). In some examples, the first set of images may be sent to a client or client device (for example, based on user input in block 404). The user interface may allow the user to review each image in the first set of images and compare them to images in the second set of images. In some examples, the user may accept or reject individual images in the first set of images. In such cases, the user's acceptance or rejection choice may be assigned to the images as labels. The images and corresponding labels may be passed to the first neural network for further training of the first neural network. In some examples, images labeled as real may be removed before further training. This enables a second layer of backpropagation learning in generative adversarial networks, allowing the second neural network and the user to independently further update or train the first neural network.

[0072] Each block in Figure 4 may be performed in any particular order, or at any particular frequency, such as sequentially, in any order, once or multiple times (sequentially or in any order), without departing from the spirit or scope of this disclosure.

[0073] In the above description, specific details are provided to ensure a full understanding of the embodiments. However, it is understood that embodiments can be carried out without these specific details. For example, circuits may be shown in block diagrams so as not to obscure the embodiments with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0074] The implementation examples of the techniques, blocks, steps, and means described above may be carried out in a variety of ways. For example, these techniques, blocks, steps, and means may be implemented in hardware, software, or a combination thereof. In the case of hardware implementation examples, the processing unit may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0075] Furthermore, embodiments may be described as processes shown as flowcharts, flow diagrams, swim diagrams, data flow diagrams, structure diagrams, or block diagrams. While the description may describe operations as sequential processes, many operations can be executed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operation is complete, but it may have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, the termination of the function corresponds to returning the function to the calling function or the main function.

[0076] Furthermore, embodiments may be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented by software, firmware, middleware, scripting languages, and / or microcode, program code or code segments for performing the required tasks may be stored on a machine-readable medium such as a storage medium. Code segments or machine-executable instructions may represent procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, scripts, classes, or any combination of instructions, data structures, and / or program statements. Code segments can be combined with other code segments or hardware circuits by sending and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc., can be sent, transferred, or transmitted by any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0077] For firmware and / or software implementation examples, the methods may be implemented in modules (e.g., procedures and functions) that perform the functions described herein. Any machine-readable medium that tangibly embodies instructions may be used to implement the methods described herein. For example, software code may be stored in memory. Memory may be implemented within or outside the processor. As used herein, the term “memory” refers to any type of long-term, short-term, volatile, non-volatile, or other storage medium, and is not limited to any particular type of memory or any particular number of memories or any particular type of medium in which memory is stored.

[0078] Furthermore, as disclosed herein, the term “storage medium” may mean one or more memories for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage medium, optical storage medium, flash memory device, and / or other machine-readable media for storing information. The term “machine-readable medium” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and / or other various media capable of storing, containing, or carrying instructions (or multiple instructions) and / or data.

[0079] While the principles of this disclosure have been described above in relation to specific apparatuses and methods, it should be clearly understood that such descriptions are illustrative and not intended as limitations on the scope of this disclosure.

Claims

1. A method performed by a computer, Receiving user text input that corresponds to content delivery operations, Parsing the user's text input to identify one or more keywords associated with the content delivery operation, Querying one or more databases for one or more images corresponding to one or more keywords, The method further comprises performing one or more image processing operations on the one or more images to derive processed image data including one or more image segments of the one or more images, wherein each of the one or more image segments corresponds to one of the one or more keywords, and the method further The process includes running a generative adversarial network on the processed image data to generate one or more images for the content delivery operation, and running the generative adversarial network is The process further includes running a first neural network on the processed image data, the first neural network generating a first set of images corresponding to one or more keywords, the first set of images being generated at least in part on the possibility that each image in the first set of images would not be detected as having been generated by the first neural network, and the process of running the generative adversarial network further includes, A method performed by a computer, comprising displaying a first set of images together with a second set of images via a first user interface, wherein the second set of images includes images that were previously part of one or more content delivery operations, or images that have been designated as available for a content delivery operation by an entity associated with the content delivery operation.

2. The computer-based method according to claim 1, wherein the generative adversarial network is trained at runtime based on one or more keywords.

3. Based on the one or more keywords, it is determined that the generative adversarial network is not trained to generate a new image corresponding to at least one of the one or more keywords. The process further comprises sending a request for a training dataset to one or more databases, wherein the training dataset includes multiple images, and a portion of each of the multiple images corresponds to at least one of the one or more keywords, The computer-based method according to claim 1, comprising training the generative adversarial network using the training dataset.

4. The method performed by a computer according to claim 1, wherein the one or more images include one or more images from a previous content delivery operation.

5. The method performed by a computer according to claim 1, wherein the one or more image processing operations include labeling each portion of the one or more images corresponding to one of the one or more keywords.

6. Executing a generative adversarial network further involves, Receiving an input to assign an acceptance or rejection label to each image in the first set of the aforementioned images, Training the first neural network based at least partially on the aforementioned labels, A method performed by a computer according to claim 1, comprising removing each image to which a rejection label has been assigned from the first set of images.

7. It is a system, One or more processors, A non-temporary computer-readable medium containing instructions, wherein when an instruction is executed by one or more processors, it causes the one or more processors to perform an operation, and the operation is Receiving user text input that corresponds to content delivery operations, Parsing the user's text input to identify one or more keywords associated with the content delivery operation, Querying one or more databases for one or more images corresponding to one or more keywords, The process includes performing one or more image processing operations on the one or more images to derive processed image data that includes one or more image segments of the one or more images, wherein each of the one or more image segments corresponds to one of the one or more keywords, and the operation further includes: The process includes running a generative adversarial network on the processed image data to generate one or more images for the content delivery operation, and running the generative adversarial network is The process further includes running a first neural network on the processed image data, the first neural network generating a first set of images corresponding to one or more keywords, the first set of images being generated at least in part on the possibility that each image in the first set of images would not be detected as having been generated by the first neural network, and the process of running the generative adversarial network further includes, A system comprising displaying a first set of images together with a second set of images via a first user interface, wherein the second set of images includes images that were previously part of one or more content delivery operations, or images that have been designated as available for a content delivery operation by an entity associated with the content delivery operation.

8. The system according to claim 7, wherein the generative adversarial network is trained at runtime based on one or more keywords.

9. Based on the one or more keywords, it is determined that the generative adversarial network is not trained to generate a new image corresponding to at least one of the one or more keywords. The process further comprises sending a request for a training dataset to one or more databases, wherein the training dataset includes multiple images, and a portion of each of the multiple images corresponds to at least one of the one or more keywords, The system according to claim 7, further comprising training the generative adversarial network using the training dataset.

10. The system according to claim 7, wherein the one or more images include one or more images from a previous content delivery operation.

11. The system according to claim 7, wherein the one or more image processing operations include labeling each portion of the one or more images corresponding to one of the one or more keywords.

12. Executing a generative adversarial network further involves, Receiving an input to assign an acceptance or rejection label to each image in the first set of the aforementioned images, Training the first neural network based at least partially on the aforementioned labels, The system according to claim 7, further comprising removing each image to which a rejection label has been assigned from the first set of images.

13. A program that causes one or more processors to perform the method described in any one of claims 1 to 6.

Images