Scalable architecture for automatically generating content distribution images

Abstract

The invention relates to a scalable architecture for automatically generating content distribution images. A method and system for automatically generating a content distribution image includes receiving a user input corresponding to a content distribution operation. The user input may be parsed to identify a keyword. Image data corresponding to the keyword may be identified. An image processing operation may be performed on the image data. A generative adversarial network is performed on the processed image data, including performing a first neural network on the processed image data to generate a first image corresponding to the keyword, the first image generated based on a likelihood that each of the first images is not detected to have been generated by the first neural network. The user interface may display a first image and a second image including an image previously being part of a content distribution operation or an image designated by an entity as available for the content distribution operation.

Description

[0001] This application is a divisional application of PCT International Application No. PCT / CN2020 / 115593, filed September 11, 2020, entitled “SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” which claims priority to U.S. Non-Provisional Application No. 17 / 017,486, filed September 10, 2020, entitled “SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” which claims the benefit of U.S. Provisional Application No. 62 / 900,40,40, filed September 13, 2019, entitled “SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” the entire contents of which are incorporated herein by reference for all purposes.

[0002] Cross Reference to Related Applications

[0003] This application claims priority to U.S. Non-Provisional Application No. 17 / 017,486, filed September 10, 2020, entitled “SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” which claims the benefit of U.S. Provisional Application No. 62 / 900,40,40, filed September 13, 2019, entitled “SCALABLE ARCHITECTURE FOR AUTOMATIC GENERATION OF CONTENT DISTRIBUTION IMAGES,” the entire contents of which are incorporated herein by reference for all purposes. TECHNICAL FIELD

[0004] The present invention relates generally to automatic generation of content for content-distribution operations, and more particularly to a scalable architecture for content-distribution operations utilizing neural networks. BACKGROUND

[0005] Generating content for content-distribution operations can be resource intensive and time consuming. Typically, the goal of a content-distribution operation is to distribute specific content to a specific set of devices within a particular relevant time interval. A company can employ teams of photographers, designers, artists, etc. to obtain images for the operation, modify the images based on one or more criteria, convert the modified images into a format suitable for transmission or display, and transmit the formatted images to the specific set of devices. Companies often spend weeks of time and considerable resources to design a single content-distribution. Moreover, by the time the distribution operation begins, enough time can have passed such that the generated content can no longer be relevant to the specific set of devices that receive them. Thus, there is a need for systems and methods to reduce the resource requirements of content-distribution operations. SUMMARY

[0006] A method for automatically generating images for content distribution operations is disclosed. The method includes receiving user input corresponding to a content distribution operation; parsing the user input to identify one or more keywords associated with the content distribution operation; receiving image data corresponding to the one or more keywords associated with the content distribution operation from one or more databases; performing one or more image processing operations on the image data to derive processed image data, the image processing operations providing an indication of a particular location within an image corresponding to the one or more keywords; performing a generative adversarial network on the processed image data to generate one or more images for the content distribution operation, wherein performing the generative adversarial network includes: performing a first neural network on the processed image data, the first neural network generating a first set of images corresponding to the one or more keywords, wherein the first set of images are generated based at least in part on a likelihood that each image in the first set of images is not detected as having been generated by the first neural network; displaying, via a first user interface, the first set of images and a second set of images, the second set of images including images that were previously part of one or more content distribution operations or images that are specified by an entity associated with the content distribution operation as being available for the content distribution operation.

[0007] Another aspect of the disclosure includes a system comprising one or more processors and a non-transitory computer-readable medium comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the above- described method.

[0008] Other aspects of the disclosure include a non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described method.

[0009] Further areas of applicability of the present disclosure will become apparent from the detailed description provided next, according to which further objectives and advantages of the disclosure will be apparent. It should be understood that the detailed description and specific examples, while indicating various embodiments, are given by way of illustration only, and BRIEF DESCRIPTION OF DRAWINGS

[0010] The present disclosure is described in connection with the appended drawings:

[0011] Figure 1 is a block diagram of a scalable architecture for generating content distribution images in accordance with at least one aspect of the present disclosure.

[0012] Figure 2 is a block diagram illustrating a process flow for generating content distribution images in accordance with at least one aspect of the present disclosure.

[0013] Figure 3An example of a graphical user interface associated with generating content distribution images according to at least one aspect of this disclosure is illustrated.

[0014] Figure 4 A flowchart illustrating a content distribution operation according to at least one aspect of this disclosure is shown.

[0015] In the accompanying drawings, similar components and / or features may have the same reference numerals. Furthermore, various components of the same type can be distinguished by adding a dash after the reference numerals and a second reference numeral to differentiate similar components. If only the first reference numeral is used in the specification, then the description applies to any similar component having the same first reference numeral, regardless of the second reference numeral. Detailed Implementation

[0016] The following description provides only one or more preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Instead, the subsequent description of one or more preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing the preferred exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the appended claims.

[0017] A scalable architecture can be defined to use artificial intelligence to generate content and handle content distribution operations. Generative adversarial networks can be trained at runtime to generate specific images for content distribution operations. For example, user input can be parsed to determine one or more keywords corresponding to the expected context of the content distribution operation. These keywords can represent, for example, one or more subjects of the content (e.g., people), characteristics of the subjects (e.g., race, gender, height, weight, etc.), one or more objects to be represented in the content, settings (e.g., location), etc.

[0018] In some cases, a generative adversarial network (GAN) may not be trained to generate images for every possible keyword or combination of keywords. In these cases, one or more keywords can be used to generate image data (e.g., from a database of previous content distribution operations and / or labeled images), which can be used to train the generator neural network of the GAN at runtime. The GAN can then generate images corresponding to any keyword or combination of keywords on demand.

[0019] The discriminator neural network in a generative adversarial network (GAN) can be trained after the generator neural network. Training the discriminator neural network may involve using the generator neural network to generate images and feeding them along with other images, such as images from image data, to the discriminator neural network. The discriminator neural network can generate a prediction for each image indicating whether the image was generated by the generator neural network. The prediction can be passed back to the generator neural network. The generator neural network uses the discriminator neural network's prediction to improve image generation, generating images that are more likely to be labeled as real images by the discriminator neural network. Therefore, the goal of the generator neural network is to deceive the discriminator neural network into making false negative predictions (e.g., predicting that the image was not generated by the generator neural network, even if the image was actually generated by the generator neural network).

[0020] After the generator neural network and discriminator neural network are trained, one or more keywords can be used to execute the generator neural network to generate images for content distribution activities. In some cases, the discriminator neural network can continue to generate predictions that can be used by the generator neural network to further refine the image generation. In these cases, the predictions can also be used to filter the images generated by the generator neural network, for example by removing those images that the discriminator correctly predicted to be generated by the generator neural network (e.g., images that do not appear as real images). The generated images (or the remaining filtered images) can be transmitted to one or more client devices (e.g., devices associated with the entity that requested the image generation), a content distribution server (e.g., a web host), or one or more user devices as part of a content distribution activity (e.g., devices associated with the user targeted by one or more client devices).

[0021] In some cases, user input defining content distribution activities may include natural language text, text commands, or pre-selected options. The input may describe the content to be distributed and / or the mechanism of distribution (e.g., the specific device to receive the content, the communication protocol to be used to transmit the content, the content format, etc.). User input can be processed to determine one or more keywords. In some cases, a rule system may be used to process natural language text or text commands. A rule system may include a hierarchy of rules that, when applied to text, assign words or phrases to categories. The application of each rule can generate category identifiers that characterize words or phrases corresponding to a specific category.

[0022] In some cases, categories can be hierarchical, allowing a more specific or higher-priority category to be assigned to a word if the application of rules represents a word as corresponding to two or more categories. For example, a first rule might identify phrases referring to the first category "transportation mode," a second rule to the second category "car," and a third rule to the third category "sedan." While input text referring to a sedan will trigger the application of all three rules, the most specific third rule will be applied. Therefore, the keyword "sedan" will be selected as the keyword for that phrase.

[0023] In some cases, machine learning models can be used to identify keywords in input. Machine learning models can be trained using training data derived from a set of previous content distribution operations, open-source databases, public databases, and / or manufactured data (e.g., data generated for the purpose of training the machine learning model). A set of features (e.g., words or phrases) can be extracted from the training data. This set of features can then be passed as input to the machine learning model. Unsupervised learning, semi-supervised learning, or supervised learning can be used to train the machine learning model. Once trained, the machine learning model can receive input text and output one or more keywords. In some cases, the machine learning model can also output the confidence score for each of the one or more keywords, indicating the probability that the keyword corresponds to the input text.

[0024] For example, this set of previous content distribution operations can be used to identify a set of input texts for training a machine learning model. The machine learning model is trained using supervised learning, where each input text is associated with a tag (e.g., keywords generated by analyzing the input text). The machine learning model can use this set of input texts to determine which words and / or phrases correspond to a specific category. During supervised learning, the machine learning model can compare the outputs generated by the model (e.g., predicted keywords) with the tags associated with the input texts to adjust the internal processes of the machine learning model and improve the accuracy of subsequent predictions. During unsupervised learning, the machine learning model can extract features from each input text in this set of input texts. The machine learning model can then use clustering analysis, for example, based on the extracted features, to identify relationships between related input texts in this set of input texts (e.g., those associated with the same one or more keywords). Once trained, the machine learning model can be executed using inputs to a specific content distribution operation to classify the input words and phrases.

[0025] One or more keywords can be extracted from the categorized input. These keywords may correspond to words or phrases within the input that can represent the context of a content distribution operation. Alternatively, the keywords may correspond to categories assigned to words or phrases. In some cases, the confidence value assigned to each input text indicates the likelihood of correctly selecting keywords for the input text. A threshold can be used to discard keywords associated with confidence values ​​below the threshold to prevent one or more keywords from representing incorrect context for a content distribution operation.

[0026] One or more keywords can be used to obtain image data for input into a generative adversarial network (GAN). One or more keywords can be used to generate queries. Queries can be executed to obtain image data from one or more databases and / or external networks. Image data can correspond to one or more keywords and be associated with the context of content distribution operations. In some cases, one or more queries can be executed to obtain image data from one or more databases and / or external networks during iteration. For example, if a first query to a first database does not return image data corresponding to one or more keywords, a second query can be generated and executed for another database. The process can continue, with each successive query executed on different databases or external networks. If no database or network returns image data corresponding to one or more keywords, a final query can be generated and executed to obtain image data from an open-source database, a public database, or the internet.

[0027] In some cases, image data can be processed before being fed into a generative adversarial network (GAN). For example, one or more first keywords may correspond to cars and trucks, and one or more second keywords may indicate that the background of the car or truck is an indoor environment. The retrieved image data can include images of cars and trucks in various indoor and outdoor environments. If the image data corresponding to cars and trucks in indoor environments exceeds a threshold (e.g., a specific number of images and / or a specific amount of data), then image data corresponding to cars or trucks in outdoor environments may not be used.

[0028] If the image data corresponding to cars and trucks in an indoor environment does not exceed a threshold, then the image data corresponding to cars or trucks in an outdoor environment can be processed to make the data usable while simultaneously conforming to one or more first keywords and one or more second keywords. For example, the image data can be processed by generating indications (e.g., tags) about portions of each image corresponding to one or more first keywords and / or one or more second keywords (or portions of image data not corresponding to one or more first keywords or one or more second keywords). In some cases, portions of images corresponding to keywords corresponding to one or more first keywords and one or more second keywords can be extracted from portions of images not corresponding to keywords corresponding to one or more first keywords and one or more second keywords. By generating indications about relevant (or irrelevant) portions of the images, the generative adversarial network can be correctly trained.

[0029] Image processing can include one or more operations that can be applied to modify image data to better train a generative adversarial network. These operations can be performed in any particular order. Image processing can include using object detection to label portions of one or more objects in an image (specifically, portions of objects corresponding to one or more keywords). In some cases, object detection / labeling can be performed by neural networks such as convolutional neural networks (CNNs), region-based CNNs, fast CNNs, you-only-look-once (YOLO), etc. Once the objects in the image are labeled, the image can be further processed, for example, by reducing the image size by removing portions that do not correspond to one or more keywords.

[0030] The modified image data can be passed as input to the generator network of a generative adversarial network (GAN). The generator network can be a neural network (e.g., a convolutional neural network, a deconvolutional neural network, etc.). The generator network derives a first set of images from the input image data using a transformation function through a set of layers. The first set of images can correspond to one or more keywords (e.g., the context of a content distribution campaign) and be in a format suitable for distribution as part of the content distribution campaign. The first set of images can then be passed to a discriminator network for evaluation.

[0031] The discriminator network can generate a prediction for each image in the first set of images, indicating whether the image was generated by the generator network or by a real image. The discriminator network can then feed these predictions back to the generator network for further training (e.g., backpropagation). In some cases, each image generated by the generator network may be accepted or rejected by the user before or after being passed to the discriminator network. For example, each image can be labeled as accepted or rejected. In some cases, the images and their corresponding labels can be passed to the generator network as a second layer of backpropagation. In these cases, the labels can be used to further refine the image generation.

[0032] The predictions assigned to each image can be used to determine which images will be included in the content distribution campaign and which images can be discarded (or used for further training of the generative adversarial network). For example, the content distribution campaign may include a second set of images as a subset of the first set of images. The second set of images may include images from the first set of images that have been assigned predictions from the discriminator network indicating that the images were not generated by the generator network. The second set of images can then be transmitted to one or more client devices according to the content distribution campaign. Alternatively or additionally, in some cases, the content distribution operation may include physical representations of the images in the second set of images. For example, the images may be printed and distributed to one or more recipients via mail or other means.

[0033] Figure 1 This is a block diagram of a system 100 for generating content distribution images according to at least one aspect of this disclosure. System 100 includes one or more computing devices 104-1, 104-n, which can be used to automatically generate content for content distribution operations. Each computing device 104-1 to 104-n may include the same hardware components, such as those depicted within 104-1, or different hardware components. For example, computing device 104-1 may include hardware components that perform specific aspects of content generation (e.g., text analysis, image processing, content generation, content evaluation, and combinations thereof).

[0034] The computing device 104-1 includes one or more processors (e.g., CPU 108) coupled to the memory 112 via a bus. A user interface 116 may be displayed on the display 120 for user interaction. The user can operate input / output devices to define new content distribution operations within the user interface 116. Input may include one or more attributes and text characterizing the new content distribution. For example, the text may be natural language text, alphanumeric strings, alphanumeric commands, etc.

[0035] The content distribution operation definition can be passed to a machine learning core 124. The machine learning core 124 can be one or more AI modules, each executing together to generate an image that conforms to the content distribution operation definition. This definition can be passed to a parser 126 within the machine learning core 124. The parser 126 can break down the text input to determine one or more keywords defined. These keywords can correspond to the defined context, such as the subject, object, etc., of the content to be distributed. The parser 126 transforms the text into a set of tokens (e.g., each alphanumeric character, each word, and / or phrase based on the settings of the parser 126). This set of tokens can then be tagged individually and in combination (e.g., using a sliding window with a preset character length). Tagging can be based on matching tokens with predefined keywords, a rule hierarchy, or using machine learning. The machine learning implementation can output only those tokens that can be associated with the context (e.g., nouns, verbs, adjectives with nouns, etc.), while ignoring or discarding specific tokens, such as those associated with an article (e.g., words with little contextual value).

[0036] Machine learning core 124 can obtain image data using one or more extracted keywords. In some cases, machine learning core 124 can retrieve image data corresponding to one or more keywords from local image data 136. In other cases, machine learning core 124 can generate and execute queries to external data repositories such as computing device 104-n or image data 152. If data cannot be obtained within the network, machine learning core 124 can generate and execute queries to retrieve image data from the Internet (e.g., using a search engine, web crawler, etc.). Image data may include one or more images depicting a subject / object corresponding to one or more of the one or more keywords (i.e., context or a portion thereof). For example, one or more keywords may include "track" and "car," and image data may include images of a track, images of a car, and / or images including both a car and a track.

[0037] The machine learning core 124 can pass image data to the image processor 128 to process the image into a format more suitable for one or more neural networks, such as generative adversarial networks. Image processing can include one or more operations that modify the image and / or label the image content. Image processing can include applying one or more filters, signal or frequency analysis, edge detection, orientation correction, such as affine or Euclidean transformations, combinations thereof, etc.

[0038] For example, edge detection modifies an image to emphasize the edges of objects (or subjects) within the image. Edge detection can be initiated using filtering techniques, where one or more filters can be applied to the image. Filters can modify the image by blurring, sharpening, or transforming it (such as, but not limited to, one or more affine transforms, Euclidean transforms, etc.). Filters can reduce image noise by, for example, removing image artifacts and / or other parts of the image that do not correspond to one or more keywords.

[0039] In some cases, an image may have parts that require more processing than other parts. For example, one part of an image might appear blurry, while another part might be sharp. Different filters can be applied to different parts of an image, and furthermore, different groups of filters can be applied to different parts of an image. For instance, the first part of an image can be filtered to sharpen it, while the second part can be filtered using affine transformation filters and noise reduction. Any number of different filters can be applied to the image and / or each patch.

[0040] Once a filter is applied, variations in pixel intensity gradients across adjacent pixels can be identified. Large changes in intensity between adjacent pixels can indicate the presence of an edge. For example, a first pixel with a high intensity value immediately adjacent to a pixel with a low intensity value can provide an indication that the first pixel is part of an edge. In some cases, pixels that are not part of an edge can be suppressed (e.g., set to predetermined red / green / blue values, such as black, where red=0, blue=0, and green=0, or any predetermined red / green / blue value). Edge detection operators such as the Roberts crossover operator, the Prewitt operator, and the Sobel operator can be used as part of the identifier of the pixel intensity gradient.

[0041] Non-maximum suppression can be used to suppress pixels that do not strongly correspond to edges. The non-maximum suppression process assigns an edge intensity value to each pixel identified as part of an edge using its pixel intensity gradient. For each pixel identified as part of an edge, the pixel's edge intensity value is compared to the edge intensity values ​​of its eight surrounding pixels. If the pixel's edge intensity value is higher than the edge intensity values ​​of its surrounding pixels (e.g., a local maximum), then the surrounding pixels are suppressed. Non-maximum suppression can be repeated for every pixel in the entire image.

[0042] A dual-thresholding process can then be performed to remove noise and / or false edge pixels carried by the prior image processing techniques applied herein. Two pixel intensity thresholds, one high and one low, can be defined. The thresholds can be used to assign an intensity characteristic to each pixel as strong or weak. Pixels including intensity values ​​above the high threshold can be assigned a strong intensity characteristic, while pixels including intensity values ​​between the high and low thresholds can be assigned a weak intensity characteristic. Pixels including intensity values ​​below the low threshold can be suppressed (e.g., in the same manner as described above).

[0043] A hysteresis process can then be performed to remove pixels with weak intensity characteristics (i.e., weak due to noise, color variations, etc.). For example, local statistical analysis (e.g., connected component analysis, etc.) can be performed on each pixel with weak intensity characteristics. Pixels with weak intensity characteristics that are not surrounded by pixels including those with strong intensity characteristics can be suppressed. The remaining pixels after the hysteresis process (e.g., unsuppressed pixels) include only those pixels that are edges. Although the above five processes are described in a specific order, each process can be performed any number of times (e.g., repeated) and / or in any order without departing from the spirit or scope of this disclosure. In some cases, only a subset of the five processes needs to be performed on the image. For example, image processing can perform pixel intensity gradient identification without first performing the filtering process. In some cases, an image that has been partially processed (e.g., one or more of the above processes have already been performed) may be received. In these cases, one or more additional processes can be performed to complete the image processing.

[0044] In some cases, signal processing (e.g., similar to radio frequency signals) can be performed on an image. An image can be transformed to the frequency domain (e.g., using a Fourier transform, etc.) to represent the frequencies at which specific pixel characteristics (e.g., pixel intensity, RGB values, etc.) exist in the image. In the frequency domain, one or more filters (such as, but not limited to, Butterworth filters, bandpass filters, etc.) can be applied to the image (e.g., during or after preprocessing, edge detection) to suppress or alter specific frequencies. Suppressing specific frequencies can reduce noise, eliminate image artifacts, suppress non-edge pixels, eliminate pixels with specific colors or color gradients, normalize color gradients, etc. High-pass filters can reveal edges in an image (e.g., sharp contrast in color and / or intensity between adjacent pixels), while low-pass filters can blend edges (e.g., blur). Image padding can be performed before signal processing to improve signal processing techniques. In some cases, different parts and / or patches of an image can be processed differently, some with high-pass filters and others with low-pass filters. In some cases, thresholds (e.g., the cutoff frequency of a high-pass or low-pass filter) can be modified for different parts of an image (e.g., based on image processing of one or more previous images, machine learning, etc.).

[0045] Signal processing can also determine other properties of the relationships between recognizable pixels in an image, such as coherence (e.g., for edge detection, segmentation, pattern analysis, etc.). Relationships between pixels can be used to further refine edge detection and / or identify the structural characteristics of the content depicted within an image. For example, coherence can be used to identify relevant parts of an image from unrelated parts (e.g., parts of the same object corresponding to keywords).

[0046] In some cases, image segmentation can be performed instead of, during, or after edge detection. Image segmentation assigns a coherence value to each pixel of an image (e.g., as described above). Image segmentation can use the coherence values ​​identified during edge detection or one or more other operations, such as the Sobel model, graph partitioning, etc. Coherence values ​​represent pixel labels that can be used to group pixels based on common markers (e.g., objects depicted in the image). Coherence values ​​can be used to identify the location, orientation, and shape of objects within the image, which can be used to identify objects via a lookup table. If objects cannot be identified using coherence and image features, the processed image (awaiting further image processing) can be passed to a classifier.

[0047] The classifier can be a predictive machine learning model, such as, but not limited to, neural networks. The classifier can be trained to label objects / subjects corresponding to one or more keywords. The classifier can be trained using local image data 136 or based on one or more training datasets received from the training server 148. The neural network can be trained using supervised or unsupervised learning. For example, in supervised learning, a set of labeled images can be input into the neural network. The neural network can define a feature set (e.g., a set of image features indicating the presence of labels). When unlabeled data is input into the neural network, the classifier can use the feature set. In unsupervised learning, images can be passed as input to a labeled classifier. The classifier can learn the feature set through analysis of the images. If the classifier's accuracy falls below a threshold, the classifier can be retrained using the training dataset and / or any additional labeled or unlabeled images, using supervised or unsupervised learning.

[0048] In some cases, it can be determined whether the Generative Adversarial Network (GAN) has been trained to generate images corresponding to the image data before the image data is passed as input. If the GAN has not yet been trained, Machine Learning Core 124 can train the GAN on demand based on specific processed and / or labeled image data that can be passed as input. For example, if the image data corresponds to a racetrack and a car, and the GAN has not been trained using images of the racetrack and / or the car, then the GAN may not be able to generate images that look like real images (e.g., they do not look computer-generated).

[0049] Machine learning core 124 can attempt to locate and construct a training dataset in local image data 136. If not found, machine learning core 124 can generate and execute a query to training server 148. Training server 148 can obtain images using image data 152 or image data from a previous content distribution operation 156 to train the generative adversarial network. For example, if the company previously performed a content distribution operation associated with a racetrack and cars, machine learning core 124 can access training data and / or generated images from that content distribution activity to train the generative adversarial network.

[0050] In some cases, multiple generative adversarial networks (GANs) can be provided, each trained to generate images corresponding to one or more specific keywords. For example, image data (e.g., input data) can be classified by a classifier to determine the type of image to be generated (e.g., a specific subject, object, setting, etc.). A particular GAN can be selected from among multiple GANs based on the fact that a particular GAN has been trained to generate images corresponding to the categories identified by the classifier. Multiple GANs can be organized hierarchically based on the specificity of the keywords on which each GAN is trained.

[0051] The Generative Adversarial Network (GAN) comprises a generator neural network 140 and a discriminator neural network 144, which operate in feedforward and backpropagation learning modes after training. The generator network 140 uses one or more transformation / weighting functions to identify features of training images corresponding to specific characteristics (e.g., background, foreground, keywords, objects, subjects, etc.). During supervised learning, the features are compared with image labels to determine the accuracy of the generator network 140. Training can continue until a threshold accuracy has been reached. If the accuracy falls below the threshold, the generator network 140 can be retrained. Once trained, the generator network 140 can be used to generate a series of images corresponding to one or more input keywords. The generator network 140 attempts to generate images that appear indistinguishable from (or nearly indistinguishable from) real images.

[0052] Once the generator neural network 140 is trained, the discriminator neural network 144 can be trained. The discriminator network 144 attempts to predict whether an image was generated by the generator neural network 140 (e.g., a fake image). During training, a set of training images generated by the generator neural network 140 and images received from one or more other sources can be input into the discriminator network 144. The training data can be labeled (for supervised learning) or unlabeled (for unsupervised learning). The discriminator network 144 can use the images passed as input to improve the accuracy of subsequent predictions. Once trained, the discriminator network 144 can begin predicting images generated by the generator network 140. Predictions can be fed back to the generator neural network 140 to improve the generated images. This process can continue until a predetermined error rate is detected. For example, the error rate could correspond to the percentage of images generated by the generator neural network 140 but predicted as real by the discriminator neural network 144 (e.g., the discriminator neural network 144 predicts an image as not generated by the generator neural network 140). Because both networks operate simultaneously in both feedforward and backpropagation, the model continuously obtains more realistic images generated by the generator network 140.

[0053] In some cases, in addition to prediction, the discriminator network 144 can also output other features of the image. For example, the discriminator network 144 can assign confidence values ​​that indicate the confidence level of the prediction. The discriminator network 144 can also assign accuracy values ​​that indicate how close the generated image is to one or more keywords. For example, for the keywords "track" and "car," if the generated image depicts a track but not a car, it can be assigned a lower accuracy value than a generated image that depicts both a track and a car. Images can be ranked based on accuracy (and / or prediction) to determine which generated images should be transmitted to client devices 160-1, 160-2, 160-3, ... 160-n.

[0054] Once generator 140 generates an image that deceives discriminator neural network 144 (e.g., looks realistic enough that discriminator neural network 144 predicts the image as real), the image can be further processed (e.g., using one or more of the previously described image processing operations) and / or the image can be distributed over the network to one or more client devices 160-1, 160-2, 160-3, ... 160-n.

[0055] Figure 2 This is a block diagram illustrating a process flow for generating content distribution images according to at least one aspect of the present disclosure. The process flow can be initiated using a user interface 204 that allows a user to define content distribution operations. The user interface includes multiple fields for the content distribution operations, including a title field 208 and a text field 212. The text field 212 enables the reception of user input, such as an alphanumeric string describing the content distribution operation. Once received, an upload button 216 can be selected to transmit the alphanumeric string entered in the title 208 and text 212 to one or more machine learning cores for content generation. In some cases, at least one machine learning core can be executed locally on the same device as the user interface 204. In other cases, the user interface 204 can be presented on a different device than the machine learning core. The user interface 204 may include one or more additional fields allowing the user to indicate specific characteristics of the content distribution operation, including but not limited to the receiving device (or user), one or more communication protocols for transmitting content once generated, a threshold accuracy for each generated image, etc.

[0056] Text (and any other features) can be passed to text processing pipeline 228, which can parse the text to determine one or more keywords associated with text field 212. The text can be parsed via a keyword lookup table, a machine learning model, or any other operation as described above. At box 232, the one or more keywords identified from text field 212 at text processing pipeline 228 can be used to identify image data corresponding to the one or more keywords. For example, box 232 identifies an image or a portion of an image corresponding to each of the one or more keywords. In some cases, a predefined number of images can be defined for the image data. In these cases, box 232 can first identify a first set of images corresponding to each of the one or more keywords. If the first set of images is not equal to or greater than a predefined threshold, then images corresponding to all keywords except one of the one or more keywords can be added to the first set of images. If the first set of images is still not equal to or greater than the predefined threshold, then images corresponding to all keywords except one of the one or more keywords can be added to the first set of images, and so on, until a predefined number of images are obtained for the image data.

[0057] At box 236, one or more image processing techniques and / or object recognition can be performed on the image data. For example, image segmentation or edge detection can be used to modify the image data to distinguish image portions corresponding to at least one keyword from image portions not corresponding to one or more keywords. The portions of each image corresponding to at least one keyword can be labeled with at least one keyword. In some cases, further image processing can be performed to remove image portions not corresponding to at least one keyword (or otherwise label these portions).

[0058] Labeled and processed image data can be passed to box 240, which can use the labeled and processed image data to generate a training dataset for generative adversarial networks 252-256. For example, box 240 can determine whether generative adversarial networks 252-256 are trained to generate images corresponding to one or more keywords. If not, then a training dataset corresponding to one or more keywords can be generated to dynamically train generative adversarial networks 252-256. For example, the training dataset can be obtained from training server 148, which has access to image data from a previous content distribution 156 and / or one or more remote networks (e.g., external networks, the Internet, etc.).

[0059] In some cases, images from the labeled and processed image data can be output to the user interface 204. For example, some or all of the representations in the labeled and processed image data can be transmitted to the user interface 204. The user can then review the images from the labeled and processed image data. In some cases, the user can discard one or more images from the labeled and processed image data (e.g., those that did not pass the user's review). In some cases, the user can supplement the labeled and processed image data by, for example, uploading additional images 224 to the labeled and processed image data.

[0060] The labeled and processed image data can be passed as input to a generative adversarial network (GAN) comprising a generator network 252 and a discriminator network 256. The generator network 252 can use the labeled and processed image data to generate a new set of artificial images, which can be evaluated by the discriminator network 256. The discriminator network 256 can generate predictions indicating whether the generated images are real or artificial. Images predicted as real can be output to box 260, where the images are packaged for transmission to one or more remote clients or client devices. Images predicted as artificial can be passed back to the generator network 252 as backpropagation in the GAN to improve the accuracy of future image generation and subsequent labeling.

[0061] Figure 3 An example of a graphical user interface associated with generating a content distribution image according to at least one aspect of this disclosure is illustrated. The graphical user interface 304 includes one or more frames, each presenting different information to the user. For example, the upper frame may receive user input defining and / or modifying aspects of a content distribution operation. After the text defining the content distribution operation is parsed into one or more keywords representing the context of the content distribution operation, the user interface 304 may be presented to the user to modify aspects defined by the content distribution operation. For example, the user may toggle operator 308 to display the text defining the content distribution operation.

[0062] Users can toggle operator 312 to review some or all of the reference images provided to the machine learning core that generates the images. Reference images include user-provided images and / or images from image data identified by the machine learning core. Detected keywords / tokens 316 can include a representation of each keyword and / or associated token identified from the provided text. Users can select or deselect one or more keywords to generate new image data or modify image data (if already generated). Keyword / token confidence filter 320 can be used to indicate to the user which keywords are likely to highly correspond to the provided text and which keywords may have low correspondence confidence. In some cases, keyword / token confidence filter 320 may include one or more threshold filters that automatically remove keywords below a specific confidence value.

[0063] Images generated by the generative adversarial network (GAN) can be presented in real time via user interface 304 (e.g., when an image is generated). User interface 304 can represent each generated image 324 as a full image or a thumbnail; when input to select an image is received, the thumbnail can be presented as the full image. In some cases, the images presented via user interface 304 can be all images generated by the GAN. In other cases, the images presented via user interface 304 can include only those images predicted by the discriminator network as not generated by the generator network. Each image 324 may include an accept button 328 and a reject button 332. Input can be received to accept image 328 for distribution to one or more clients or client devices, or to reject image 332 and discard it. In some cases, the selection to accept 328 or reject 332 can be assigned as a label for image 324. The images and assigned labels can be backpropagated to the GAN to add additional training layers, which can further improve image generation for content distribution operations.

[0064] Figure 4 A flowchart illustrating a content distribution operation according to at least one aspect of this disclosure is shown. At block 404, user input may be received to generate content for a content distribution (e.g., marketing) operation associated with an entity (e.g., a company selling products / services or a company providing content distribution services). The user input may include one or more alphanumeric strings describing the content distribution operation. The one or more alphanumeric strings may include natural language text, text commands in a specific pattern (such as key / value pairs), etc. The user input may also include one or more characteristics of the content distribution operation, such as the type of image to be generated, the format of the image, the client or client device receiving the image, the communication protocol used to transmit the image, combinations thereof, etc.

[0065] At box 408, user input can be parsed to identify one or more keywords associated with the context of the content distribution operation. Parsing user input can include keyword analysis (e.g., identifying specific words and matching them against a keyword lookup table), a natural language machine learning model (as described above), etc. Each of the one or more keywords indicates a portion of the content to be depicted in the generated image. For example, keywords can correspond to objects, context, actions, subjects, backgrounds, foregrounds, settings (e.g., locations such as beaches or schools), combinations thereof, etc.

[0066] At box 412, image data corresponding to at least one of the one or more keywords can be received from one or more databases (or networks such as external networks or the Internet). In some cases, the image data may include structured or unstructured data representing or indicating images associated with at least one keyword. In other cases, in addition to structured or unstructured data, the image data may include one or more images, each corresponding to at least one keyword. In still other cases, the image data may consist of only one or more images.

[0067] At box 416, one or more image processing operations can be performed on the image data to derive processed image data. The one or more image processing operations include operations that can modify or transform the image (e.g., affine transformation, edge detection, image segmentation, etc.). The one or more image processing operations may also include object detection. Object detection can use a trained neural network to identify one or more objects, subjects, settings, actions, contexts, combinations thereof, etc., in the image. For example, for an image of cars racing on a track, a trained neural network can label the parts of the image corresponding to objects (e.g., cars), the parts corresponding to settings (e.g., the track), and the context (e.g., such as day / night, weather, etc.). The one or more image processing operations may also include removing parts of the image that do not correspond to at least one keyword.

[0068] One or more image processing operations may also include a portion of the image data that is labeled as specific to the image data, and this portion will remain unmodified by the generative adversarial network (GAN) that generates images similar to the specific image. For example, an object can be labeled as unmodified so that when input into the GAN, the GAN can generate different images including the same object. The GAN can generate images of the object in different contexts (e.g., different settings, locations, backgrounds, foregrounds, weather, etc.) with different other objects, etc.

[0069] One or more image processing operations may also include a specific portion of the image data to be modified by the generative adversarial network (GAN). For example, an object may be marked as unmodified so that when input into the GAN, it can generate a different image that is uniquely distinguishable by the object. If the object is a car depicted on a racetrack, marking the car in this way allows the GAN to generate an image that is uniquely distinguishable by modifications to the car. For example, the GAN could generate images of the car as a different color or a different brand or model. The car can be replaced with different objects, such as a truck, a bicycle, etc. If the object corresponds to a human, then human characteristics can be modified, such as, but not limited to, age, race, gender, height, weight, appearance (e.g., eye color, hair color, clothing, etc.), and combinations thereof.

[0070] At box 420, a generative adversarial network can be executed on the processed image data, including:

[0071] At box 424, a first neural network can be executed on the image data to generate a first set of new images, each new image corresponding to at least one of one or more keywords. The first neural network can be executed using one or more keywords as input (from the processed image data) or using one or more images. The first set of images can be entirely computer-generated synthetic images, but generated to appear as if the images were real (e.g., not computer-generated). The first neural network can be a generative neural network as part of a generative adversarial network. A second neural network can be used to train the generator neural network, enabling the generation of the first set of images based on the probability that each image in the first set is not detected by the second neural network as having been generated by the first neural network.

[0072] For example, a second neural network can generate predictions for each ancestor image in a set of training images (e.g., images previously generated by a first neural network). The predictions may correspond to the probability that an ancestor image was generated by the first neural network. The set of training images may include ancestor images and images received from one or more other sources (e.g., previous content distribution operations, the Internet, images from the same entity, etc.). The one or more other sources may include image data, one or more images received from local or remote databases, the Internet, etc. Images received from one or more other sources may be real images (e.g., images not generated by a computer, images generated by a computer (such as another generative adversarial network), or combinations thereof).

[0073] The second neural network can be a discriminator neural network representing another part of the generative adversarial network (GAN). The goal of the GAN can be to generate images that appear to be real, inducing the second neural network to predict the generated images as "real" rather than those generated by the generator neural network. The goal of the second neural network is to correctly predict whether an image was generated by the first neural network and pass that prediction back to the generator neural network (e.g., backpropagation learning). In some cases, the second neural network can assign a series of values ​​instead of Boolean values ​​(e.g., real or generated by the generator neural network). For example, this series of values ​​could indicate the probability or confidence that the image was generated by the generator neural network. In some cases, the first and second neural networks can be the same neural network, while in others, they can be different neural networks.

[0074] The predictions from the second neural network can be used to update the first neural network. The predictions assigned to images generated by the first neural network can provide the first neural network with an indication of how well the image can fool the second neural network into predicting whether or not it was generated by the first neural network. This information can be used to modify one or more layers of the first neural network so that it can produce images that the second neural network is more likely to predict as real images (e.g., images that are more likely to look like realistic images rather than computer-generated images).

[0075] Once trained, the first neural network can be executed to generate images, each corresponding to at least one of one or more keywords. The generated images can be used to continuously train the first neural network to continuously improve subsequently generated images (e.g., to make the images less likely to appear to be computer-generated).

[0076] At box 428, the user interface can display a first set of images and a second set of images (e.g., side-by-side, within the same window, etc.). The second set of images may include images previously part of one or more content distribution operations or images designated by entities (e.g., companies, users within companies, etc.) as associated with one or more keywords. In some cases, the first set of images may be transmitted to a client or client device (e.g., based on user input at box 404). The user interface allows the user to review each image in the first set and compare these images with images in the second set. In some cases, the user can accept or reject individual images in the first set. In these cases, the user's acceptance or rejection choice can be assigned as a label to the image. Images and corresponding labels can be passed to a first neural network for further training. In some cases, images labeled as real can be removed before further training. This enables backpropagation learning of the second layer in a generative adversarial network, where the second neural network and the user can each independently update or train the first neural network.

[0077] Without departing from the spirit or scope of this disclosure Figure 4 Each box can be executed in any particular order or at any particular frequency (such as, for example, in order, out of order, once or multiple times (in order or out of order)).

[0078] Specific details are set forth in the foregoing description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagrams to avoid obscuring the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0079] The implementation of the above-described techniques, blocks, steps, and means can be carried out in various ways. For example, these techniques, blocks, steps, and means can be implemented in hardware, software, or a combination thereof. In hardware implementations, the processing unit can be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the above-described functions, and / or combinations thereof.

[0080] Furthermore, it should be noted that the embodiments can be described as processes depicted as flowcharts, flow diagrams, swim diagrams, data flow diagrams, structure diagrams, or block diagrams. While the depiction may portray operations as sequential processes, many operations can be executed in parallel or concurrently. Additionally, the order of operations can be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process can correspond to a method, function, program, subroutine, subroutine, etc. When a process corresponds to a function, its termination corresponds to the function returning to the calling function or the main function.

[0081] Furthermore, embodiments can be implemented using hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented using software, firmware, middleware, scripting languages, and / or microcode, program code or code segments used to perform necessary tasks can be stored in a machine-readable medium such as a storage medium. Code segments or machine-executable instructions can represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, scripts, classes, or any combination of instructions, data structures, and / or program statements. Code segments can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0082] For firmware and / or software implementations, the methods can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly implements the instructions can be used to implement the methods described herein. For example, software code can be stored in memory. Memory can be implemented within or outside the processor. As used herein, the term "memory" means any type of long-term, short-term, volatile, non-volatile, or other storage medium, and is not limited to any particular type of memory or the number of memories, or the type of medium on which memory is stored.

[0083] Furthermore, as disclosed herein, the term "storage medium" can refer to one or more memories used for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media used for storing information. The term "machine-readable medium" includes, but is not limited to, portable or fixed storage devices, optical storage devices, and / or various other storage media capable of storing instructions and / or data containing or carrying one or more instructions.

[0084] Although the principles of this disclosure have been described above in conjunction with specific apparatus and methods, it should be clearly understood that the description is by way of example only and is not intended to limit the scope of this disclosure.

Claims

1. A computer-implemented method, comprising: Receive a content distribution operation definition corresponding to a content distribution operation, wherein the content distribution operation is used to automatically generate at least one composite image based on the content distribution operation definition; By utilizing a text processing pipeline trained to handle content distribution operation definitions, one or more keywords associated with the context of the content distribution operation are identified from the content distribution operation definitions, wherein the one or more keywords are determined based on parsed input; One or more images are retrieved by querying one or more databases using one or more keywords identified from the content distribution operation definition; Perform one or more image processing operations on the one or more images to create a processed image dataset, and update the processed image dataset based on user review, wherein the image processing operations include at least image segmentation or visual transformation of the one or more images; One or more synthetic images corresponding to the one or more keywords are generated by executing a generative network for content distribution operations, the generative network being configured to generate the one or more synthetic images using the processed image dataset; as well as The one or more composite images are displayed together with a set of images via a user interface, and user feedback is received for the one or more composite images used in a content distribution operation, wherein the set of images includes images from one or more previous content distribution operations or images specified by an entity associated with the content distribution operation.

2. The computer-implemented method of claim 1, wherein the generative model is trained as part of a generative adversarial network.

3. The computer-implemented method according to claim 1 further includes: The generative model is trained at runtime based on one or more of the aforementioned keywords.

4. The computer-implemented method according to claim 1 further includes: Based on the one or more keywords, it is determined that the generative model has not been trained to generate a new image corresponding to at least one of the one or more keywords; A request for a training dataset is transmitted to one or more databases, the training dataset comprising multiple images, wherein a portion of each of the multiple images corresponds to at least one of the one or more keywords; as well as The generative model is trained using the training dataset.

5. The computer-implemented method of claim 1, wherein the one or more images include one or more images from a previous content distribution operation.

6. The computer-implemented method of claim 1, wherein the one or more image processing operations include marking portions of each of the one or more images corresponding to keywords in the one or more keywords.

7. The computer-implemented method according to claim 1, wherein executing the generative model further comprises: Receive input that assigns an accept or reject flag to each of the set of images; The generative model is trained at least in part based on the labeled data; as well as Remove each image from the set of images that has been assigned a rejection tag.

8. The computer-implemented method of claim 1, wherein the machine learning model is configured to output a confidence value for each of the one or more keywords, the confidence value indicating the probability that each keyword corresponds to a content distribution operation defined.

9. The computer-implemented method of claim 1, wherein the one or more image processing operations include removing portions of each of the one or more images that do not correspond to the one or more keywords by image segmentation.

10. The computer-implemented method according to claim 1, wherein the text processing pipeline further comprises: The content distribution operation definition is parsed into one or more sets of tokens; Identify one or more keywords from the set or more sets of tokens; as well as The context of the content distribution operation definition is determined by at least utilizing machine learning-based techniques or by matching one or more keywords with a hierarchical structure of predefined keywords or rules.

11. The computer-implemented method of claim 1, wherein the content distribution operation definition includes at least user text input associated with the content distribution operation of the entity.

12. The computer-implemented method of claim 1, wherein the discriminator network evaluates the effectiveness of the one or more synthetic images generated by the generator network, wherein the evaluation from the discriminator network is associated with a confidence value and an accuracy value.

13. The computer-implemented method according to claim 1, wherein configuring the generation network further includes: The processed image dataset is classified using a classifier to obtain one or more types of the one or more synthetic images to be generated; The generative network is selected from a plurality of generative networks trained to generate one or more synthetic images of the recognized category; as well as The generative network is trained based on the processed image dataset classified by the classifier.

14. A system comprising: One or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform the method according to any one of claims 1-13.

15. A computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-13.

Images