Evaluating the visual quality of digital content

Abstract

To provide a method for evaluating visual quality of digital content.SOLUTION: Systems, devices, methods, and computer readable medium for evaluating visual quality of digital content are disclosed. Methods can include a step of identifying content assets including one or more images that are combined to create different digital components distributed to one or more client devices. The quality of each of the one or more images is evaluated using one or more machine learning models trained to evaluate one or more visual aspects that are deemed indicative of visual quality. Overall quality of the content assets is determined based, at least in part, on an output of the one or more machine learning models indicating the visual quality of each of the one or more images. A graphical user interface of a first computing device is updated to present a visual indication of the overall quality of the content assets.SELECTED DRAWING: Figure 2A

Description

[Technical Field]

[0001] TECHNICAL FIELD This specification relates generally to data processing, and more particularly to rating digital content. [Background technology]

[0002] Computing devices may generate digital components and distribute those digital components to various client devices. Digital components may be formed from content assets, such as images, text, and the like, that are collectively combined to form the digital component. In some cases, the quality of the resulting digital component may be adversely affected even if a single content asset used to form the digital component is of low quality—for example, the image is blurry, contains objectionable content (e.g., pornographic material), or has an orientation that does not fit the area in which the digital component is to be displayed. Summary of the Invention [Means for solving the problem]

[0003] Generally, one innovative aspect of the subject matter described herein may be embodied in a method including: identifying, by one or more processors, content assets including one or more images to be combined to generate distinct digital components to be delivered to one or more client devices; evaluating, by the one or more processors, the quality of each of the one or more images using one or more machine learning models trained to evaluate one or more visual aspects deemed indicative of visual quality; determining, by the one or more processors, an aggregate quality of the content asset based at least in part on the output of the one or more machine learning models indicative of the visual quality of each of the one or more images; and updating, by the one or more processors, a graphical user interface of a first computing device to present a visual indication of the aggregate quality of the content asset. Other embodiments of this aspect include corresponding methods, apparatuses, and computer programs configured to perform the actions of the method encoded on a computer storage device. These and other embodiments may each optionally include one or more of the following features.

[0004] The method may include receiving, by one or more processors, a modification of one of the one or more images; evaluating, by the one or more processors, a quality of the modified image; updating, by the one or more processors, an overall quality of the content asset based on the quality of the modified image; and updating, by the one or more processors, a graphical user interface of the first computing device to present an updated visual indication of the overall quality of the content asset.

[0005] The method may include comparing, by one or more processors, the overall quality with preset quality heuristics; determining, by the one or more processors, that the overall quality does not comply with the preset quality heuristics; generating, by the one or more processors, one or more recommendations for improving the overall quality in response to determining that the overall quality does not comply with the preset quality heuristics; and updating, by the one or more processors, a graphical user interface of the first computing device to present the one or more recommendations.

[0006] The one or more recommendations may include a first recommendation for modifying a visual feature of the image.

[0007] The step of assessing the quality of each of the one or more images may include deploying, by one or more processors, a plurality of machine learning models on the image to generate a score for each of a plurality of quality features; assigning a weight to each score by the one or more processors to generate a weighted score; combining, by the one or more processors, the weighted scores to generate a combined score for the image; and comparing, by the one or more processors, the combined score to one or more thresholds to generate a quality of the image.

[0008] Determining the overall quality of the content asset may include determining a possible total score for each image; calculating a ratio of the combined score to the possible total score for each image, where the ratio for an image is a portion of one or more ratios for one or more images; and calculating an average of the one or more ratios, where the average of the one or more ratios indicates the overall quality of the content asset.

[0009] The method may include determining a quality of a digital component including at least one image from the content asset and at least one other content asset; comparing, by one or more processors, the quality of the digital component to a threshold; determining, by the one or more processors, that the quality of the digital component is below the threshold; and limiting, by the one or more processors, delivery of the digital component to one or more client devices in response to determining that the quality of the digital component is below the threshold.

[0010] The subject matter described herein may provide various advantages. For example, quality information provided by the techniques discussed throughout this specification can be used to limit or prevent the generation, storage, and / or transmission of low-quality (e.g., below a threshold level of quality) content assets and / or digital components, thereby reducing computing resources, memory components, and / or network bandwidth consumed by the generation, storage, and / or transmission of these content assets and / or digital components. Quality information about content assets can be stored in an index, making this quality information available during the automated (or manual) generation of a digital component using two or more of the content assets so that the overall quality of the digital component can be evaluated based on the overall quality of the content assets used to generate the digital component. This can prevent low-quality digital components from being stored and occupying limited memory space. According to the described techniques, the image quality of individual images can be determined and stored in a database. Because multiple digital components may be generated by combining different permutations of images, this modular approach allows for more efficient determination of the image quality of a digital component. If the first digital component includes a first set of images and the second digital component includes a second set of images, there may be overlap between the first set of images and the second set of images, i.e., the first set of images and the second set of images may include portions of the same images. If the image quality of some of the second set of images has already been determined as part of determining the image quality of the first set of images, the image quality scores for these images may be retrieved from the database, thereby avoiding the need to repeat evaluation of the same images and thereby making more efficient use of available computational resources.The stored quality information can also be used to identify combinations of content assets that result in digital components below a specified quality level, so that computing resources are not wasted on generating, storing, and / or transmitting digital components that include those combinations of content assets. The techniques discussed herein can also identify situations in which one or more content assets included in a digital component are unreadable, occluded, or otherwise interfere with a user's visual perception of the information presented by the content assets, preventing the waste of computing resources on transmitting these digital components that fail to convey information to the user. Furthermore, some content publishers may evaluate digital components before publishing them. In this case, the digital component may be transmitted by the content provider to a content distribution system, transmitted from the content distribution system to the content publisher, and ultimately prevented from being published. By evaluating image quality in the content distribution system, the described techniques therefore reduce the processor and bandwidth requirements associated with transmitting low-quality digital components that are unlikely to be displayed. The techniques discussed herein can also detect when content assets included in a digital component lead to unintended user interaction with elements of the digital component that trigger a network call for additional information. Preventing the distribution of these types of digital components can prevent wasted computing resources and / or network bandwidth that would be used to initiate unintended network calls and / or the delivery of information to users in response to those unintended network calls. Thus, the techniques discussed herein can prevent the unintended transmission of information over a network, thereby making the network more efficient and reducing bandwidth needs.Given that digital components can be rapidly distributed to millions of users, resulting in unintended network calls, the efficiencies provided using the techniques discussed throughout this specification can be readily increased. The described techniques enable the evaluation, determination, and / or classification of digital images and digital image quality using machine learning models. This enables more efficient evaluation of image quality on a large scale, as well as more efficient generation, evaluation, and transmission of digital components.

[0011] Machine learning models, such as neural network models, can be used to assess the quality of content assets of a digital component. The machine learning models described herein can be trained on a large number of content assets, and can then generate accurate assessments of the quality of new content assets when deployed to those assets. Furthermore, the machine learning models can continuously learn based on new data being generated, and the relationships between the inputs and outputs of the various layers within each neural network model can be interrelated as desired and modified at any time. Such continuous updates and flexibility can keep the training of the machine learning models current, which in turn increases the accuracy of scores generated using the machine learning models.

[0012] In some implementations, a hardware accelerator may be used to deploy the machine learning model. Such a hardware accelerator may include several computational units (which may also be referred to as compute tiles) to which computation of a machine learning model (e.g., a neural network) deployed to calculate a score indicative of the quality of one or more content assets in a digital component may be distributed. Such distribution of computation to the compute tiles enables the neural network to be processed using a fewer number of instructions than would be required if the neural network were processed by a central processing unit. Such a reduction in the number of instructions increases the speed at which probabilities for multiple classes are calculated, thereby reducing latency in the process of determining a score indicative of the quality of one or more content assets in a digital component.

[0013] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a computing environment in which the quality of one or more content assets forming a digital component can be used to regulate the distribution of content. [Figure 2A] FIG. 1 illustrates a graphical user interface of a computing device that is rendered or updated by a content delivery system. [Figure 2B] FIG. 1 illustrates a graphical user interface of a computing device that is rendered or updated by a content delivery system. [Figure 3]FIG. 1 illustrates a process implemented by a content distribution system to facilitate displaying an indication of the quality of a set of content assets on a computing device. [Figure 4] FIG. 1 illustrates an example of a machine learning model trained and deployed by a content delivery system. [Figure 5] FIG. 1 illustrates the simultaneous deployment of multiple machine learning models to determine the quality of new images within a digital component. [Figure 6] FIG. 1 illustrates a process performed by a content delivery system to train and deploy machine learning models to determine the quality of content assets. [Figure 7] FIG. 1 is a block diagram of an exemplary computer system that may be used to perform the operations described above. DETAILED DESCRIPTION OF THE INVENTION

[0015] Like reference symbols in the various drawings indicate like elements.

[0016] 1 illustrates a computing environment 102 in which the quality of one or more content assets 104 forming a digital component 106 may be used to regulate the distribution of content. In some examples, the content assets 104 may be one or more of images, text, audio, animation, video, interactive content, any other multimedia, and / or user interface elements such as interactive buttons, and the digital component 106 may be a combination of two or more of the content assets 104.

[0017] The computing environment 102 includes a content distribution system 107 (e.g., including one or more servers or one or more other data processing devices) that receives content assets 104 from content providers 108 or content servers 110, possibly those assets 104 that can be used to form digital components 106, assembles (e.g., combines) at least some of those content assets 104 to form different digital components 106, and transmits the digital components 106 to respective client devices 114 in response to external calls 112. For simplicity, the following discussion refers to content assets 104 received from content providers 108, but the discussion is equally applicable to content assets 104 received from the content servers 110 or elsewhere. In some cases, the content distribution system 107 can use quality information (e.g., image quality information) about the content assets 104 and / or digital components 106 to adjust or regulate which digital components 106 are delivered and / or how the digital components 106 are delivered to client devices 114. For example, the content distribution system 107 may prevent (or limit) the distribution of digital components 106 that have a quality below a specified threshold quality level.

[0018] As described above, the digital component 106 is described as being generated from content assets 104 by the content delivery system 107. For example, the digital component 106 may be programmatically generated by the content delivery system 107 (or another computing system). For example, a content provider 108 may upload a set of content assets (e.g., images, text, and / or user interface elements such as interactive buttons) 104 and allow the content delivery system 107 to generate multiple different digital components 106 that use the set of content assets 104 in various combinations.

[0019] When a set of content assets 104 (e.g., eight different content assets 104) is uploaded to the content distribution system 107, the content distribution system 107 can evaluate (a) the quality of each individual content asset in the set, (b) the quality of the entire set of content assets 104, and / or (c) the quality of each digital component 106 formed using corresponding assets 104 (e.g., three content assets selected from the set of eight content assets for that digital component 106). These evaluations are described below.

[0020] The content delivery system 107 can evaluate the quality (e.g., image quality, also referred to as visual quality) of each individual content asset 104 in an uploaded set of content assets 104 and store quality data for each of the content assets 104 in the content asset database 116. In some implementations, the content delivery system 107 can evaluate the quality of the entire set of content assets 104 provided by the content provider 108. The content delivery system 107 can present the quality—e.g., the intrinsic quality (e.g., visual quality) of each individual content asset 104 and / or the overall quality of the set of content assets 104—to the content provider 108. In some examples, the evaluation of the quality of the set of images can be performed as follows: If quality heuristics (e.g., preset rules indicating criteria for generating and distributing digital components 106) indicate that a relatively high-quality digital component 106 can be produced when the set of content assets 104 includes at least a preset number (e.g., five) of high-quality images (e.g., images of at least a specified level of quality), the quality of the set of content assets 104 may be lowered when the set includes fewer than the preset number (e.g., fewer than five) of high-quality images. Similarly, the quality heuristics may indicate that a higher-quality digital component 106 can be produced when the set of content assets 104 includes at least two images having a first aspect ratio (e.g., corresponding to a square format) and at least two images having a second aspect ratio (e.g., corresponding to a landscape format). In this example, the quality level of the set of content assets 104 may be lowered when fewer than two images having the first aspect ratio or the second aspect ratio are present.

[0021] The quality level of each digital component 106 generated using content assets 104 may also be evaluated by the content delivery system 107. In some implementations, the evaluation of a digital component may be based on the overall quality of the content assets 104 used to generate the digital component 106 and / or the characteristics (e.g., visual or functional characteristics) of the assembled digital component 106. A digital component 106 that includes a blurred image that is also classified as sensational (e.g., a high number of skin-colored pixels) may have a lower quality score than another digital component 106 that has a less blurred image that is not classified as sensational (e.g., deemed not blurry by the model). Similarly, a digital component in which one content asset obscures or occludes some or all of the text contained in another content asset 104 may have a lower quality score than a different digital component 106 that arranges the same content assets 104 so that the text is not obscured or occluded.

[0022] In situations where the digital component 106 is generated by a content provider 108, the content delivery system 107 may provide the content provider 108 (e.g., the entity generating and / or uploading the digital component 106) with an interactive user interface that can present information regarding the quality of the digital component 106 either during the generation of the digital component 106 or after the digital component 106 is completed. For example, assume that the content provider 108 combines an image content asset 104 with a text content asset 104. In this example, the content delivery system 107 may present a rendering of the digital component 106 to the content provider 108 (e.g., provide the data that results in that representation) and further present a quality indication 104 that informs the content provider of the quality level of the digital component 106 as assessed by the content delivery system 107. Using the interactive user interface, the content provider 108 is enabled to interactively modify the digital component 106 (e.g., by rearranging the positions of the content assets 104 within the digital component 106 and / or changing the combination of content assets 104 included in the digital component 106) and obtain updated quality information from the content distribution system 107.

[0023] As discussed in detail below, the content delivery system 107 (a) trains and deploys machine learning models that generate indications 104 of quality of one or more content assets 104 and / or digital components 106 formed from one or more content assets 104, and (b) facilitates / controls the display of such indications 104 to content providers 108 and / or controls the distribution of the digital components 106 based on the indications 104 of quality.

[0024] The content assets 104 can be one or more of text, audio, images, animation, video, interactive content, or any other multimedia. In some implementations, the content assets 104 are uploaded from content providers 108 to a content distribution system 107. The content assets can then be assembled (e.g., combined together) to form a digital component 106. In one example, the digital component 106 can be a digital advertising document (which may also be referred to as an advertisement). In some implementations, the content assets 104 are used as received (e.g., without modifying them) when assembling them to generate the digital component 106. In other implementations, at least some of the content assets 104 may be modified before they are assembled to form the digital component 106. For example, the content delivery system 107 may deploy software modules that enable modification of visual characteristics of images in the content assets 104—such as blur, orientation (e.g., landscape, portrait), resolution, color, size of various objects and / or text, any other visual characteristics, and / or any combination thereof—on a graphical user interface of the content provider 108. Each software module can be part of a computer program that may include multiple independently developed modules that can be combined or linked by link modules. Each software module can include one or more software routines that are software code that perform a corresponding procedure or function.

[0025] The content delivery system 107 identifies content assets 104 uploaded by content providers 108, which may include one or more images that can be used to generate different digital components 106 that are delivered to client devices 114. The content delivery system 107 evaluates the quality of the uploaded content asset, including the one or more images, using one or more machine learning models trained to evaluate one or more visual properties deemed indicative of visual quality (e.g., blurriness, inclusion of objectionable content, orientation, or any other visual property). As described above, an indication of visual quality can be stored in the content asset database 116 in conjunction with (e.g., in association with) the content asset from which the indication of visual quality was generated. Associating the indication of visual quality with the content asset 104 facilitates use of the indication of visual quality when the content asset 104 is used to generate digital components 106 and / or to present an indication of quality to the content provider 108.

[0026] When a digital component 106 is generated using one or more content assets 104 identified in the content asset database 116, the content delivery system 107 may determine an overall quality of the content assets 104 used in the digital component 106 based at least in part on the output of one or more machine learning models that indicate the visual quality of the digital component including one or more images (e.g., the visual quality of a combination of images). A visual indication of the overall quality of the content assets 104 is presented within the indication 104.

[0027] As described above, when a set of content assets 104 (e.g., eight different content assets 104) is uploaded to the content distribution system 107, the content distribution system 107 can evaluate (a) the quality of each individual content asset in the set, (b) the quality of the entire set of content assets 104, and / or (c) the quality of each digital component 106 formed using corresponding assets 104 (e.g., three content assets selected from the set of eight content assets for that digital component 106). The content distribution system 107 can perform such evaluation by evaluating each content asset 104 to determine its quality, then aggregating such quality values ​​for all content assets in the uploaded set of content assets 104 to determine an overall quality for the set, and aggregating such quality values ​​for all content assets 104 in the digital component 106 to determine an overall quality for the digital component. The evaluation of each content asset 104 is performed as follows:

[0028] To evaluate each content asset 104, the content distribution system 107 can train and deploy multiple machine learning models, such as a blurriness model, an objectionable content model, and an orientation model. During the training phase, the content asset 104 can have one or more of multiple labels (e.g., a blurriness value, an objectionable content value, an orientation value). Each label can characterize a corresponding quality feature (e.g., blurriness, inclusion of objectionable content, orientation) of the content asset (e.g., an image). Each model can be trained to classify the content asset (e.g., an image) according to a respective one of the multiple labels. Once the models (e.g., the blurriness model, the objectionable content model, and the orientation model) are trained, the content distribution system 107 can deploy the models to the content assets 104 of the digital component 106 to generate scores for each of the labels (e.g., blurriness, objectionable content, and orientation). The content delivery system 107 may assign a weight to each score to generate a weighted score, and then combine the weighted scores to generate a combined score for the content asset 104. The content delivery system 107 may compare the combined score to one or more thresholds to generate a prognosis of the quality of the content asset. In some examples, the prognosis may be "very good" quality, "fair" quality, or "poor" quality.

[0029] The content delivery system 107 may assess the quality of a set of content assets 104 as follows: The quality of an uploaded set of content assets 104 may be a combination (e.g., sum)—or in some implementations, an average, weighted average, median, etc.—of the individual qualities of the various content assets 104 that form the set. In some cases, the overall quality of a set of content assets 104 may be calculated as the ratio of the combined score of all content assets 104 in the set to the possible total score for all of those content assets 104.

[0030] The content delivery system 107 can assess the quality of a digital component 106 as follows: The quality of a digital component 106 can be a combination of the individual qualities of the various content assets 104 that form the digital component 106. For example, the overall quality (e.g., average quality, weighted average, median quality, or another suitable aggregation) of the content assets 104 of a digital component 106 can be represented as a combination of four “very good” images and one “fair” image. In another example, the quality of a digital component 106 can be represented as a combination of four “very good” content assets 104 and three “fair” content assets 104. In other implementations, the overall quality 203 of a digital component 106 can be expressed as the sum of the combined scores of all content assets 104 used to form the digital component 106. In some cases, the overall quality of a digital component 106 can be calculated as the ratio of the combined score of all content assets to the total possible score for all content assets 104.

[0031] As described above, the content delivery system 107 can train one or more machine learning models and deploy the trained models to generate indications 120 for each new content asset 104. In some implementations, the content delivery system 107 may receive trained machine learning models from one or more other servers and deploy the received machine learning models. For example, the content delivery system 107 may receive a blur model from a first server coupled to the content delivery system 107 via a communications network, an objectionable content model from a second server coupled to the content delivery system 107 via a communications network, an orientation model from a third server coupled to the content delivery system 107 via a communications network, and so on. In some implementations consistent with this example, the first server, the second server, and the third server can be three separate servers that can be located either in the same physical location or in different physical locations. In other implementations, the first server, the second server, and the third server can be a single server located in a single physical location.

[0032] While the above examples discuss deploying machine learning models (e.g., blur models, objectionable content models, orientation models, etc.) as received, in other examples, received machine learning models can be customized—to fit the particular types of content assets 104 used by the content provider 108 to form the digital component 106—before such machine learning models are deployed. For example, such customization can be effective when a trained machine learning model was trained by other servers coupled to the content delivery system 107 based on some content assets 104 (e.g., images), but the content delivery system 107 may have access to additional related content assets 104 (e.g., images) that can more effectively train the machine learning model to make more accurate inferences. In some implementations, received machine learning models (e.g., blur models, objectionable content models, orientation models, etc.) can first be customized to fit system requirements imposed by the content delivery system 107 before such machine learning models are deployed. The content delivery system 107 may impose system requirements based on one or more of architectural information of the content delivery system 107, system requirements (e.g., architectural information) of the content providers 108, and / or system requirements (e.g., architectural information) of one or more client devices. In some implementations, the content delivery system 107 may customize machine learning models separately for each content provider 108, which may be beneficial when users (e.g., advertisers) of each computing device have different preferences regarding the types of images that are combined to form the respective sets of digital components 106.

[0033] The machine learning model can be a neural network model, a nearest neighbor model, a naive Bayes model, a decision tree model, a regression model, a support vector machine (SVM), any other machine learning model, and / or any combination thereof. The machine learning model can be customized. For example, if the machine learning model is a neural network model, the machine learning model can be implemented by changing one or more of the size of the neural network model (i.e., the number of nodes), the width of the neural network model (i.e., the number of nodes in a particular layer), the depth of the neural network model (i.e., the number of layers), the capacity of the neural network model (i.e., the type or structure of functions that can be learned by the network configuration, also referred to as expressive ability), or the architecture of the neural network model (i.e., the particular arrangement of layers and nodes).

[0034] Although the quality indication of each content asset 104 is described as being of "very good" quality, "fair" quality, or "poor" quality, in other examples, any other category name (i.e., category names other than very good, fair, or poor, such as excellent, fair, or poor, respectively) may be used. Furthermore, while three categories of quality are described, in some implementations the number of categories may be any other number greater than or equal to two. Of course, other types of quality indications may be used, such as a numerical scale (e.g., 1-10) or any other suitable quality indication that allows for comparison of quality among multiple different content assets 104.

[0035] The visual quality assessment may be stored in the content asset database 116 in conjunction with (e.g., in association with) the content asset 104 from which the visual quality indication was generated. Coordinating the visual quality indication of a content asset 104 allows the content distribution system 107 to quickly retrieve the quality of the content asset when needed (e.g., during any evaluation of the quality of that content asset 104, during evaluation of any set of content assets 104 that include that content asset 104, and during evaluation of any digital component 106 that uses that content asset 104). Such quick retrieval avoids the content distribution system 107 having to recalculate the quality for that content asset 104 every time such quality is needed (e.g., during any evaluation of the quality of that content asset 104, during evaluation of any set of content assets 104 that include that content asset 104, and during evaluation of any digital component 106 that uses that content asset 104). Such avoidance of having to reassess the quality of a content asset 104 can reduce latency during quality assessment. This in turn improves the speed at which quality is presented (eg, displayed) to the content provider 108 (eg, on a graphical user interface).

[0036] The digital component 106 may be modified prior to distribution, which may be desirable if the digital component 106 has undesirable / low overall quality (e.g., quality that does not meet quality heuristics, which may be pre-set rules that may indicate criteria for creating and distributing the digital component 106). In some examples, the content distribution system 107 may allow the content provider 108 to (a) update one or more content assets 104 used to form the digital component 106 and / or (b) replace one or more content assets 104 used to form the digital component 106 with one or more other content assets 104. In some examples, the content distribution system 107 may (a) automatically update one or more content assets 104 used to form the digital component 106 and / or (b) automatically replace one or more content assets 104 used to form the digital component 106 with one or more other content assets 104.

[0037] In one example, the quality heuristics may include the number of “very good” content assets (e.g., images) 104 that should be present in an uploaded set of content assets 104 (from which some content assets 104 are selected to generate the digital component 106), the orientation of various content assets (e.g., images) 104 in the uploaded set, content-based guidelines for the content assets 104 in the set, etc. The digital component 106 may be generated (a) automatically by the content distribution system 107 (e.g., by automatically selecting content assets 104—from the uploaded set of content assets 104—that satisfy the quality heuristics), (b) by the content provider 108 selecting one or more content assets 104 from the uploaded set and combining the selected content assets 104 to form the digital component, or (c) by the content provider 108 performing such selection and combination of content assets 104 to form the digital component. In implementations in which a content provider 108 generates a digital component 106 by combining content assets 104 specific to the digital component 106, the content delivery system 107 may enable the content provider 108 to make iterative modifications to the digital component 106 using the user interface 118. For example, the content delivery system 107 may generate recommendations for modifications to such content assets 104 and / or digital component 106 to enhance the quality of the content assets 104 and / or digital component 106.

[0038] In some examples, when the quality of the content asset 104 is “fair,” the content delivery system 107 can generate recommendations to increase the quality to “very good,” and when the quality of the content asset 104 is “poor,” the content delivery system 107 can generate a first set of recommendations to increase the quality to “fair” and / or a second set of recommendations to increase the quality to “very good.” Some recommendations to increase the quality can be to suggest changes to various visual characteristics of the identified image—e.g., brightness, contrast, color, color intensity, hue, saturation, size, noise, sharpness, luminance, undesirable elements, objectionable elements, any other characteristic, and / or any combination thereof.

[0039] In implementations in which the content distribution system 107 collectively evaluates an uploaded set of content assets 104, the content distribution system 107 can generate recommendations for modifications to content assets 104 in the set. In some implementations, the content distribution system 107 can generate such recommendations when, for example, the set of content assets 104 cannot support the generation of a digital component 106 that complies with quality heuristics. For example, a recommendation may be generated when preset rules regarding the visual quality of the digital component 106 indicate that the digital component 106 should contain three square images, but the set of content assets contains only two square images. In another example, a recommendation may be generated when preset rules regarding the visual quality of the digital component 106 indicate that the digital component 106 should contain three square images of “very good” quality, but the set of content assets contains only two square images of “very good” quality.

[0040] In implementations in which the content delivery system 107 evaluates the digital component 106, the content delivery system 107 can generate recommendations for modifications to the content assets 104 that form the digital component 106. In some implementations, the content delivery system 107 can generate such recommendations, for example, when the digital component 104 does not satisfy quality heuristics (i.e., rules). For example, a recommendation may be generated when preset rules regarding the visual quality of the digital component 106 indicate that the digital component 106 should contain three square images, but the digital component consists of only one square image. In another example, a recommendation may be generated when preset rules regarding the visual quality of the digital component 106 indicate that the digital component 106 should contain two square images of “very good” quality, but the set of content assets contains only one square image of “very good” quality. In some recommendations, the content delivery system 107 may recommend rearranging the location of one or more content assets 104 within the digital component 106 and / or changing the combination of content assets 104 included in the digital component 106 to improve the overall quality of the digital component 106.

[0041] When a recommendation is generated for modifying one or more content assets (in any implementation—i.e., in an implementation in which each individual content asset 104 is rated, an implementation in which an uploaded set of content assets 104 is rated, and / or an implementation in which a digital component is rated), the content delivery system 107 may render the recommendation on the graphical user interface 118. In response to such a recommendation, the content delivery system 107 may update the graphical user interface 118 to display options that the content provider 108 can select to approve or reject the automatic improvements made to one or more content assets 104. The content delivery system 107 may repeatedly update the quality of the uploaded set of content assets 104 and / or the digital component 106 (i.e., update the quality assessment each time the content provider 108 makes any changes) and display an updated indication of the quality on the user interface 118.

[0042] In some implementations, the content delivery system 107 may reassess the completed digital component 106 for other indications of quality, such as overlapping elements, trick-to-click, poor cropping, large buttons, etc. Such secondary assessment of the digital component 106 may be used to generate recommendations to enhance aggregate quality. For example, if a button covers a sensational or blurry portion of an image, the overall quality of the digital component 106 may be better than the combined quality of its constituent parts. On the other hand, a button that is too large and covers most of the image may degrade quality even if the assets are individually of high quality.

[0043] In implementations in which the content delivery system 107 generates the digital component 106 by combining content assets 104, the content delivery system 107 may make improvements to one or more content assets 104 and / or the digital component 106 automatically (i.e., without any interaction with the content provider 108). For example, the content delivery system 107 may adjust the position of a button asset overlaid on an image asset in a manner that enhances the overall quality of the digital component 106 (e.g., by covering up provocative portions of the image). The content delivery system 107 may update the quality of the digital component 106 as such improvements are made (i.e., update its assessment of the quality whenever the content provider makes any changes).

[0044] In some implementations, the content distribution system 107 can limit the distribution of the digital component 106 based on the final quality of the digital component 106, as follows: There may be pre-defined quality rules / heuristics for the digital component 106, which may indicate criteria for generating and distributing the digital component 106. The content distribution system 107 can compare the digital component 106 (which may have been updated as described above) to the pre-defined quality rules / heuristics to determine whether the digital component 106 is suitable for distribution. For example, if the quality heuristics indicate criteria—e.g., one or more of: (a) the digital component 106 should not contain “low” quality images; (b) the digital component 106 should not contain “sensational” images; (c) the digital component 106 should not have “landscape” images; (d) the digital component 106 should have a single image in a “square” format—then the content distribution system 107 may allow distribution of digital components 106 that meet the criteria but prevent distribution of other digital components 106 that do not meet the criteria. The content distribution system 107 may additionally or alternatively restrict digital components 106 for distribution based on meeting such criteria. Thus, the content distribution system 107 may automatically restrict distribution of digital components 106.

[0045] While automatic control of delivery is described above, in some implementations, the content delivery system 107 may provide the content provider 108 with the opportunity to manually control the delivery of the digital component 106 based on the quality determined by the content delivery system 107. For example, the content delivery system 107 may provide an indication 120 of the quality of the digital component 106 along with associated quality heuristics on the content provider's 108 interactive user interface 118. The interactive user interface 118 may further provide the content provider 108 with options to proceed with delivery of the digital component 106, prevent delivery of the digital component 106, and / or modify the digital component 106 (e.g., to improve the quality of the digital component 106 to satisfy the quality heuristics). In such an implementation, the content provider 108 may be able to control distribution—for example, the content provider may allow distribution of the digital component 106 even when the digital component 106 does not meet quality heuristics, or may prevent distribution of the digital component 106 even when the digital component 106 meets quality heuristics.

[0046] The content providers 108 may interact with the content delivery system 107 via a computing device, such as a laptop computer, a desktop computer, a tablet computer, a phablet computer, a telephone, a kiosk computer, any other computing device, and / or any combination thereof. The computing device may be configured to be managed, for example, by a domain host (e.g., an advertiser), by entering authentication data, such as, for example, a username and password, biometric data, security data obtained from an external device (e.g., security data embedded in a security key fob, a one-time password obtained from an email or software application connected to the computing device, etc.), any other authentication data, and / or any combination thereof.

[0047] Users interact with the content delivery system 107 using client devices 114, such as laptop computers, desktop computers, tablet computers, phablet computers, phones, kiosk computers, any other computing devices, and / or any combination thereof. The client devices 114 may be configured for use by any end user. The content asset database 116 can be a memory device, such as a computer-readable storage device, a computer-readable storage board, a random or serial access memory array or device, any other storage device or devices, and / or any combination thereof. Either the content server 110 or the content delivery system 107 can be a system having at least one programmable processor and a machine-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform various operations described herein.

[0048] Each of the content servers 110 can be a system having at least one programmable processor and a machine-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform various operations, such as collecting content assets 104 and transmitting those content assets 104 to content providers 108 when needed for the digital component 106. In some implementations, the content server 110 can be a laptop computer, a desktop computer, a tablet computer, a phablet computer, a phone, a kiosk computer, any other computing device, and / or any combination thereof.

[0049] The arrows between the various components shown in FIG. 1 may indicate communication networks between those components, which may include either wired or wireless connections via a local area network, a wide area network, the Internet, an intranet, a Bluetooth network, an infrared network, any other network or networks, and / or any combination thereof.

[0050] 2A and 2B illustrate graphical user interfaces that may be presented to a content provider 108. FIG. 2A illustrates a graphical user interface 200 that displays (a) a set 202 of content assets 104 (e.g., images) uploaded by a content provider 108, from which particular content assets 104 are selected to form a digital component 106, (b) an overall quality 203 of all content assets 104 in the set 202, and (c) buttons 204 related to various recommendations for improving the quality of (i) at least one content asset 104 forming the set 202 or (ii) the entire set 202 so that the digital component 106 complies with quality heuristics (e.g., pre-set rules that may indicate quality thresholds for the digital component 106 or the content assets 104 within the digital component 106 so that the digital component 106 may be delivered to various client computing devices 114). In some examples, the digital component 106 can be a digital advertising document, such as an advertisement formed by a combination of two or more content assets 104. The content assets 104 in this example set 202 include images 206, 208, 210, 212, and 214, and text 216. Although the content assets 104 in set 202 are described as images or text, in some implementations the content assets 104 may also include audio, animation, video, interactive content, other multimedia, and / or user interface elements such as interactive buttons.

[0051] Any of the images 206-218 can be digital images in any format, such as Joint Photographic Experts Group (JPEG), JPEG File Interchange Format (JFIF), Exchangeable image file format (Exif), Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), bitmap (BMP), Portable Network Graphics (PNG), portable pixmap (PPM), portable graymap (PGM), portable bitmap (PBM), portable anymap (PNM, which can be a combination of PPM, PGM, and PBM), raw image format, and / or any other format.

[0052] The overall quality 203 of the set 202 may be generated as follows: First, the content distribution system 107 may evaluate the quality of each content asset 104 forming the set (i.e., the quality of each of the images 206, 208, 210, 212, and 214 and the text 216). To do so, the content distribution system 107 may train and deploy multiple machine learning models, such as a blur model, an objectionable content model, and an orientation model. During the training phase, each content asset 104 (e.g., one of the images 206-214) may have one or more of multiple labels (e.g., a blur value, an objectionable content value, an orientation value). Each label may characterize a corresponding quality feature of the image (e.g., blur, inclusion of objectionable content, orientation). Each model may be trained to classify the image according to a respective one of the multiple labels. Once the models (e.g., blur model, objectionable content model, and orientation model) are trained, the content delivery system 107 can deploy those models to each image in the images 206-214 to generate a score for each of the labels (e.g., blur, objectionable content, and orientation). The content delivery system 107 can assign a weight to each score to generate a weighted score and then combine the weighted scores to generate a combined score for the image. The content delivery system 107 can compare the combined score to one or more thresholds to generate a predicted quality for the content asset (e.g., image). In some examples, the predicted quality can be “very good” quality, “fair” quality, or “poor” quality.

[0053] Although the quality indication of each content asset 104 is described as being of "very good" quality, "fair" quality, or "poor" quality, in other examples, any other category name (i.e., category names other than very good, fair, or poor, such as excellent, fair, or poor, respectively) may be used. Furthermore, while three categories of quality are described, in some implementations the number of categories may be any other number greater than or equal to two. Of course, other types of quality indications may be used, such as a numerical scale (e.g., 1-10) or any other suitable quality indication that allows for comparison of quality among multiple different content assets 104.

[0054] The overall quality 203 of the set 202 may, in some implementations, be represented as a combination of four "very good" images and three "fair" images. In some cases, the overall quality 203 of the set 202 may be represented as the sum of the combined scores of all images 206-214. In some cases, the overall quality 203 of the set 202 may be calculated as the ratio of the combined score of all images 206-214 to the total possible score for all images 206-214, as shown in FIG. 2A. In the example shown, the total possible score for all images 206-214 is 10.

[0055] The content delivery system 107 can generate recommendations to improve the quality of the set 202. In some examples, the content delivery system 107 may generate such recommendations only when the overall quality 203 of the set 202 is below a threshold (e.g., below 5 / 10). Such a threshold may indicate a minimum quality value that conforms to quality heuristics for a digital component 106 to be formed using one or more content assets 104 in the set 202. In some examples, if the quality heuristics indicate a criterion that the digital component 106 must include images with “very good” quality in “landscape format,” the content delivery system 107 may evaluate the images 206-214 to determine whether such criterion is met. If such criterion is not met, the content delivery system 107 may recommend that (a) the quality of a “landscape” image among the images 206-210 may be improved, or (b) that such an image be replaced by another “landscape” image with “very good” image quality. The graphical user interface 200 provides the content provider 108 with the option of clicking a button 204 corresponding to such a recommendation to make modifications to the individual image on the graphical user interface of FIG. 2B.

[0056] FIG. 2B illustrates a graphical user interface 232 that enables a content provider 108 to make modifications to individual images 206 or other assets within the set 202 of content assets 104 discussed above in connection with FIG. 2A to improve the quality of the images 206 and thus the collective quality 203 of the set 202. The graphical user interface 232 indicates the quality of the image as “very good,” “fair,” or “poor,” as described above. In the illustrated example, the quality of the image 206 is determined to be “fair” by deploying a machine learning model, as described above in connection with FIG. 2A. The graphical user interface 232 can provide the content provider 108 with options 236 for formatting the image 206. The formatting options 236 may include options for automatically modifying the image 206 as recommended by the content delivery system 107 to improve the quality of the image 206. In some implementations, the formatting options 236 can enable the content provider 108 to manually make changes to the image 206.

[0057] The graphical user interface 232 further includes an option 238 that allows the content provider 108 to approve the modifications to the image 206, another option 240 that allows the content provider 108 to reject the modifications to the image 206, and another option 242 that allows the content provider 108 to remove the image from the digital component 108. The content provider 108 may want to remove an image from the digital component 108 if the image has such poor quality that it cannot be significantly improved.

[0058] As the image 206 is modified or improved, the content delivery system 107 may iteratively update the quality indication 234 to correspond to the modified or improved image 206 (i.e., deploy a trained machine learning model to update the quality assessment each time the image 206 is modified). The quality of each of the images 206-214 (including the updated quality for the image 206) may be used to generate an overall quality 203 for the set 202.

[0059] 3 illustrates a process implemented by a content delivery system 107 that facilitates an indication of the quality of each individual content asset 104, of a set 202 of content assets, and / or of a digital component 106 to a content provider 108. The content delivery system 107 can identify 302 the content assets 104 (i.e., content assets that can be combined to generate different digital components 106 that can be delivered to one or more client devices 114)—including images—that are present in the set 202. Identifying the content assets 104 can include retrieving the particular content assets 104 from the content server 110 and / or the content asset database 116. If the quality of the content asset 104 was previously determined, such quality is then associated with the content asset 104 in the content asset database 116. Such association allows for easy retrieval of the quality for the content asset 104 from the content asset database 116 when needed, which can eliminate the need to recalculate the quality of the content asset 104.

[0060] At 304, the content delivery system 107 can evaluate the quality of each image forming the set 202 (e.g., each of the images 206-214 in the set 202) using one or more machine learning models trained to evaluate visual properties deemed indicative of visual quality (e.g., blurriness, objectionability, orientation, etc.). For example, the content delivery system 107 can train and deploy the one or more machine learning models. In some implementations, the content delivery system 107 can separately train a blur model, an objectionable content model, an orientation model, and / or any other machine learning model trained to detect some image quality feature. In some implementations, the content delivery system 107 can generate and train the machine learning models. In other implementations, the content delivery system 107 can obtain machine learning models from one or more other servers and then further train those models to, for example, suit the architectural details of the content provider 108 and various client devices 114. The content delivery system 107 can deploy the trained models to generate respective quality scores. For example, a blur model may generate a blur score that quantifies the blur in an image, a objectionable content model may generate an objectionable content score that quantifies the objectivity of the content in the image, a third model may generate an orientation score that quantifies the conformance of the width and height of the image with the standard width and height of a typical digital component 106 (e.g., a digital component having a preset size of width and height), and so on.

[0061] The content delivery system 107 can determine the quality of an image based on the output of one or more machine learning models, which indicate the visual quality of each image. In some implementations, the content delivery system 107 can assign a weight to each score generated by each machine learning model to generate a weighted score. For example, the content delivery system 107 can assign a first weight to the blur score, a second weight to the objectionable content score, and a third weight to the orientation score. The content delivery system 107 can then combine the weighted scores to generate a combined score for the image. The content delivery system 107 can compare the combined score to one or more thresholds to generate an estimate of the quality of the image. For example, if the score is below a lower threshold, the content delivery system 107 may infer that the image has "poor" quality, if the score is between a lower threshold and an upper threshold (including both), the content delivery system 107 may infer that the image has "fair" quality, and if the score is greater than the upper threshold, the content delivery system 107 may infer that the image has "very good" quality. Such inferences are also referred to herein as quality indications 120.

[0062] The content delivery system 107 may determine 306 an overall quality 203 of the content assets 104 (e.g., images 206-214) that form the set 202. The overall quality 203 of the set 202 may, in some examples, be indicated as a combination of four "very good" images and three "average" images. In some cases, the overall quality 203 of the set 202 may be calculated as the sum of the combined scores of all images 206-214 that form the digital component 106. In some cases, the overall quality 203 of the digital component 106 may be calculated as the ratio of the combined score of all images 206-214 to the total possible score for all images 206-214 (as shown in FIG. 2A where the total possible score for all images 206-214 is 10).

[0063] The content delivery system 107 can update 308 the graphical user interface 200 of the content provider 108 to present a visual indication of the overall quality 203 of the set 202. Updating the graphical user interface 203 can include sending a visual indication of the overall quality 203 of the images that form the set 202 to the content provider 108.

[0064] In some implementations, the content delivery system 107 can also generate recommendations for modifications to a content asset 104 to improve the quality of the content asset 104 and, therefore, the quality of the set 202 that includes the content asset 104. In some examples, when the quality of the content asset 104 indicates that the quality is “fair,” the content delivery system 107 can generate recommendations to improve the quality to “very good,” and when the quality of the content asset 104 is determined to be “low,” the content delivery system 107 can generate a first set of recommendations to improve the quality to “fair” and / or a second set of recommendations to improve the quality to “very good.” Some recommendations for improving quality can be to suggest changes to various characteristics of the image—e.g., brightness, contrast, color, color intensity, hue, saturation, size, noise, sharpness, luminance, undesirable elements, objectionable elements, any other characteristics, and / or any combination thereof. In some further implementations, the content delivery system 107 can automatically improve the image with the suggested changes. The content distribution system 107 may update the graphical user interface of the content provider 108 to display options that the content provider 108 can select to approve or reject the automatic enhancements made to the image.

[0065] 4 illustrates an example of a machine learning model trained and deployed by the content delivery system 107. The illustrated machine learning model is a neural network model 402. The neural network model 402 may also be referred to simply as a neural network, an artificial neural network, or an artificial neural network model. While a neural network model is described, in other implementations, the machine learning model can be any other machine learning model, such as a nearest neighbor model, a naive Bayes model, a decision tree model, a regression model, a support vector machine (SVM), any other machine learning model, and / or any combination thereof (which may include any such model in combination with a neural network model). The type of neural network model 402, as shown, can be a feedforward neural network. In other implementations, the neural network model 402 can be of any other type, such as a radial basis function neural network, a multilayer perceptron, a convolutional neural network, a recurrent neural network, a modular neural network, a sequence to sequence model consisting of two recurrent neural networks, and / or any other type of neural network.

[0066] The neural network model 402 may have multiple layers, including an input layer 404, one or more internal layers (also called hidden layers) 406, and an output layer 408. Each of the layers 404, 406, and 408 may have one or more nodes, which may also be called neurons or artificial neurons. For example, in the illustrated implementation, layer 404 has three nodes, each of layers 406 has four nodes, and layer 408 has a single node. Each node (also called a neuron) is a computational unit. In some implementations, the number of hidden layers and the number of nodes in each corresponding layer may be changed based on the amount / number of images on which the model is trained. For example, when the amount / number of images is very large (e.g., greater than a threshold such as 1,000, 5,000, 10,000, 100,000, or any other implementation-specific threshold), a larger number of nodes and / or a larger number of layers may be used to facilitate inference during the training and deployment processes. In some implementations, the number of nodes and / or layers may be varied as the training phase progresses to empirically determine the ideal number of nodes and / or layers in the neural network 402.

[0067] The input layer 404 is configured to receive content assets (e.g., images) as inputs. During the training phase for the model 402 (i.e., the duration of time the model 402 is being trained), the input layer 404 receives the images on which the model 402 is trained. During the deployment phase (i.e., the duration of time the trained model 402 is being used to generate inference results), the input layer 404 receives new images (e.g., images 204) whose quality is to be inferred. Each node (also called a neuron) in the input layer 404 is a computational unit that combines data from images in some configurable way and has an output connection.

[0068] The internal layers 406 are also called hidden layers because they are not directly observable from the inputs and outputs of the system that implements the neural network model 402. Each node (also called a neuron) in each hidden layer 406 is a computational unit that has one or more weighted input connections, a transfer function that combines the inputs in some configurable way, and an output connection.

[0069] The output layer 408 outputs a score for the input image (e.g., any of images 206-214). For example, the output layer 408 for the blur model may output a blur score, the output layer 408 for the objectionable content model may output an objectionable content score, the output layer 408 for the orientation model may output an orientation score, and so on. During the training phase, each model 402 may be trained on a different (or, in other implementations, the same) set of images. However, during the deployment phase, each model receives the same new image (e.g., any of images 206-214) for which quality inferences are to be made. The output layer 408 in the illustrated example has a single node with one or more weighted input connections and a transfer function that combines the inputs in some configurable way to produce a score (e.g., a blur score if the model is a blur model, an objectionable content score if the model is an objectionable content model, an orientation score if the model is an orientation model, etc.).

[0070] In some implementations, a hardware accelerator may be used to deploy the machine learning model. Such a hardware accelerator may include several computational units (which may also be referred to as compute tiles) to which computation of a machine learning model (e.g., a neural network) deployed to calculate a score indicative of the quality of one or more content assets in a digital component may be distributed. Such distribution of computation to the compute tiles enables the neural network to be processed using a fewer number of instructions than would be required if the neural network were processed by a central processing unit. Such a reduction in the number of instructions increases the speed at which probabilities for multiple classes are calculated, thereby reducing latency in the process of determining a score indicative of the quality of one or more content assets in a digital component.

[0071] FIG. 5 illustrates the simultaneous deployment of multiple machine learning models 402-1, 402-2, ..., and 402-n to determine the quality of a new image (e.g., any of images 206-214), where n can be any fixed integer value. Each model 402-i (i is any integer value ranging from 1 to n) can receive a new image (e.g., any of images 206-214) as input and output a score S that quantifies the visual features that model 402-i is trained to determine. For example, model 402-1 can be a blur model that can receive image 204 as input and output a blur score S, model 402-2 can be an objectionable content model that can receive image 204 as input and output an objectionable content score S, ...model 402-n can be an orientation model that can receive image 204 as input and output an orientation score S. While the discussion here focuses on specific machine learning models, in some implementations, further models (not shown) may additionally or alternatively be trained and deployed to detect large buttons within the digital component 106, trick-to-click issues within the digital component 106, sensationalism of images within the digital component 106, poor cropping of images within the digital component 106, inappropriate overlapping text within the digital component 106, location of logos within the digital component 106, etc., and classify the digital component accordingly.

[0072] Each of the models 402-i (where i is any integer value ranging from 1 to n) may have different hidden layers 406 (e.g., different numbers of hidden layers 406 and / or different numbers of nodes within each hidden layer 406) because hidden layers 406 may be added or modified between different models 402-i to adapt each model to make corresponding inferences.

[0073] The content distribution system 107 may include a computation unit (e.g., one or more processors) 502 that can assign a weight Wi (where i is any fixed integer ranging from 1 to n) to the score Si of each model 402-i and add up the weighted scores WiSi for all models to generate a total score 504 (which may also be referred to as a combined score).

[0074] The total score 504 may indicate the quality of the input image (e.g., any of the images 206-218). The content delivery system 107 may compare the total score 504 to one or more thresholds to generate a prediction of the quality of the input image (e.g., any of the images 206-218). In some examples, the prediction may be “very good,” “fair,” or “poor.” While the quality indication 120 is described as being “very good,” “fair,” or “poor,” in other examples, any other category name (i.e., category name other than very good, fair, or poor, such as excellent, fair, or poor, respectively) may be used. Furthermore, while three categories of quality are described, in some implementations, the number of categories may be any other number greater than or equal to two.

[0075] The content delivery system 107 can determine an overall quality 203 of the content assets 104 (e.g., images 206-214) that form the set 202. The overall quality 203 of the set 202 can, in some examples, be shown as a combination of four "very good" images and three "average" images. In some cases, the overall quality 203 of the set 202 can be calculated as the sum of the combined scores of all the images 206-214 that form the set 202. In some cases, the overall quality 203 of the digital component 106 can be calculated as the ratio of the combined score of all the images 206-214 to the total possible score for all the images 206-214 (as shown in FIG. 2A where the total possible score for all the images 206-218 is 10).

[0076] The content delivery system 107 can update the graphical user interface of the content provider 108 to present a visual indication 120 of the overall quality 203 of the content assets 104 that form the digital component 106. The updating of the graphical user interface can include sending a visual indication 203 of the overall quality 203 of the images that form the set 202 to the content provider 108.

[0077] Although a particular neural network model is described herein, in other implementations, any other neural network model or any suitable machine learning model that does not compromise latency and processing power may be used. For example, any neural network model 402-i (where i is any fixed integer ranging from 1 to n) may be a feedforward neural network (as shown), or any other neural network, such as a radial basis function neural network, a multilayer perceptron, a convolutional neural network, a recurrent neural network, a modular neural network, a sequence-to-sequence model consisting of two recurrent neural networks, and / or any other type of neural network.

[0078] 6 illustrates a process performed by the content distribution system 107 to train and deploy machine learning models 402-i (i is any fixed integer ranging from 1 to n). The content distribution system 107 may train 602 multiple machine learning models (e.g., a blur model, an objectionable content model, and an orientation model) on multiple images retrieved from the content server 110 and / or the content asset database 116. During such training, each of the images is assigned a score (e.g., a blur score, an objectionable content score, an orientation score) that indicates a quality feature of the image (e.g., blur, amount of objectionable content, orientation compatibility with the digital component 106). Each machine learning model (e.g., a blur model, an objectionable content model, or an orientation model) is trained to score the image with respect to its respective quality feature (e.g., blur, amount of objectionable content, orientation compatibility with the digital component 106, respectively).

[0079] The content delivery system 107 may receive 604 a request from a content provider 108 to evaluate the quality of a new image (e.g., any of images 206-214) included in the set 202 of content assets 104 uploaded by the content provider 108. In response to such a request, the content delivery system 107 may deploy 606 machine learning models (e.g., a blur model, an objectionable content model, and an orientation model) on the new image to generate scores S1, S2, ..., Sn for each quality feature (e.g., a score S1 for blur, a score S2 for amount of objectionable content, ..., and a score Sn for orientation suitability).

[0080] The content delivery system 107 may include a computation unit (e.g., one or more processors) 502 that can assign a weight W (where i is any fixed integer ranging from 1 to n) to the score S of each model 402-i (e.g., each of the blur model, the objectionable content model, and the orientation model). The content delivery system 107 may combine 608 the weighted scores W, S for each model to generate a combined score 504 for the image (e.g., any of the images 206-214). The total score 504 may indicate the quality of the input image (e.g., any of the images 206-214).

[0081] The content delivery system 107 can compare 610 the total score 504 to one or more thresholds to generate a predicted result of the quality of the input image (e.g., any of the images 206-214). The predicted result is also referred to as a quality indication. In some examples, the predicted result (i.e., the quality indication) can be “very good,” “fair,” or “poor.” While the quality indications are described as being “very good,” “fair,” or “poor,” in other examples, any other category name (i.e., category names other than very good, fair, or poor, such as excellent, fair, or poor, respectively) can be used. Furthermore, while three categories of quality are described, in some implementations, the number of categories can be any other number greater than or equal to two.

[0082] The content delivery system 107 may determine 612 an overall quality 203 of the content assets 104 (e.g., all of the images 206-214) that form the digital component 106. The overall quality 203 of the digital component 106 may, in some examples, be indicated as a combination of four "very good" images and three "fair" images. In some cases, the overall quality 203 of the digital component 106 may be calculated as the sum of the combined scores of all of the images 206-214 that form the digital component 106. In some cases, the overall quality 203 of the digital component 106 may be calculated as the ratio of the combined score of all of the images 206-214 to the total possible score for all of the images 206-214 (as shown in FIG. 2A where the total possible score for all of the images 206-214 is 10).

[0083] The content distribution system 107 can transmit data to the content provider 108 indicating the overall quality 203 of the set 202. Such transmission can update the graphical user interface of the content provider 108 to display an estimate (i.e., an indication 203) of the quality of the set 202 of one or more new images.

[0084] 7 is a block diagram of an exemplary computer system 700 that may be used to perform the operations described above. System 700 includes a processor 710, a memory 720, a storage device 730, and an input / output device 740. Each of the components 710, 720, 730, and 740 may be interconnected using, for example, a system bus 750. Processor 710 may process instructions for execution within system 700. In one implementation, processor 710 is a single-threaded processor. In another implementation, processor 710 is a multi-threaded processor. Processor 710 may process instructions stored in memory 720 or storage device 730.

[0085] The memory 720 stores data within the system 700. In one implementation, the memory 720 is a computer-readable medium. In one implementation, the memory 720 is a volatile memory unit. In another implementation, the memory 720 is a non-volatile memory unit.

[0086] Storage device 730 can provide mass storage for system 700. In one implementation, storage device 730 is a computer-readable medium. In various different implementations, storage device 730 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0087] The input / output device(s) 740 provide input / output operations to the system 700. In one implementation, the input / output device(s) 740 may include one or more of a network interface device, e.g., an Ethernet card, a serial communication device, e.g., an RS-232 port, and / or a wireless interface device, e.g., an 802.11 card. In another implementation, the input / output device(s) may include a driver device configured to receive input data and send output data to other input / output devices, e.g., a keyboard, a printer, and a display device 760. However, other implementations, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0088] Although an exemplary processing system is shown in FIG. 7, the implementation and functional operations of the subject matter described herein can be implemented in other types of digital electronic circuitry, or computer software, firmware, or hardware, or in combinations of one or more of them, including the structures disclosed herein and their structural equivalents.

[0089] Embodiments and operations of the subject matter described herein may be implemented in digital electronic circuitry, or computer software, firmware, or hardware, or a combination of one or more of them, including the structures disclosed herein and their structural equivalents. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal—e.g., a machine-generated electrical, optical, or electromagnetic signal—that is generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus. The computer storage medium may be, or may be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Furthermore, the computer storage medium may not be a propagated signal, but may be a source or destination of computer program instructions encoded in an artificially generated propagated signal. Also, a computer storage medium may be, or be contained in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0090] The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0091] The term "data processing apparatus" encompasses all kinds of apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a plurality or combination thereof. An apparatus may include dedicated logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program in question, e.g., code constituting a processor's firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as a web services infrastructure, a distributed computing infrastructure, and a grid computing infrastructure.

[0092] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple associated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and connected together by a communications network.

[0093] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0094] Processors suitable for executing a computer program include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for performing actions in accordance with the instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to such mass storage devices, or both. However, a computer need not have such devices. Furthermore, a computer can be incorporated into another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0095] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying data to the user, and a keyboard and pointing device, e.g., a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0096] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., as a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Examples of communications networks include local area networks (“LANs”) and wide area networks (“WANs”), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0097] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of clients and servers arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to client devices (e.g., for the purpose of displaying the data to users interacting with the client devices and receiving user input from such users). Data generated at the client devices (e.g., results of online interactions) may be received at the server from the client devices.

[0098] While the specification contains many specific implementation details, these should not be considered limitations on the scope of any invention or what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in a combination, and may even be initially claimed as such, one or more features of a claimed combination may in some cases be deleted from the combination, and a claimed combination may be directed to a subcombination or a variation of a subcombination.

[0099] Similarly, while operations are shown in a particular order in the figures, this should not be understood as requiring such operations to be performed in the particular order shown, or in sequential order, or that all of the operations shown be performed to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system components in the above-described embodiments should not be understood as requiring such division in all embodiments, and it should be understood that the described program components and systems generally can be integrated together in a single software product or packaged in multiple software products.

[0100] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0101] 102 Computing Environment 104 Content Assets, Quality Indication 106 Digital Components 107 Content Distribution System 108 Content Providers 110 Content Server 112 External Calls 114 client devices 116 Content Asset Database 118 User Interface 120 Indication, Indication of Quality 200 Graphical User Interface 202 pairs 203 Total Quality 204 Button 206 images 208 images 210 images 212 images 214 images 216 Text 232 Graphical User Interface 234 Quality Indication 236 options 238 Options 240 options 242 options 402 Neural Network Model 402-1 Machine Learning Model 402-2 Machine Learning Model 402-i model 402-n machine learning model 404 Input Layer 406 Internal and Hidden Layers 408 Output Layer S1 Blur Score S2 Objectionable Content Score Si score Sn orientation score 502 computing units 504 Total score, combined score 700 Computer Systems 710 processor 720 memory 730 Storage Devices 740 Input / Output Devices 750 System Bus 760 display device

Claims

1. Identifying, by one or more processors, a plurality of content assets including one or more images to be combined to generate different digital components to be delivered to one or more client devices; assessing, by the one or more processors, a first quality of each of the one or more images using one or more machine learning models trained to assess one or more properties of the one or more images deemed to be indicative of visual quality; determining a second quality of a digital component including at least one image from the plurality of content assets based on the first quality of each of the one or more images; comparing, by the one or more processors, the second quality of the digital component to a threshold; determining, by the one or more processors, that the second quality of the digital component is less than the threshold; limiting, by the one or more processors, delivery of the digital component to the one or more client devices in response to determining that the second quality of the digital component is below the threshold; A method comprising:

2. determining an overall quality of the plurality of content assets based at least in part on an output of the one or more machine learning models indicative of a first quality of each of the one or more images; receiving, by the one or more processors, a modification of one of the one or more images; evaluating, by the one or more processors, an updated quality of the modified image; updating, by the one or more processors, the overall quality of the plurality of content assets based on the updated quality of the modified image; updating, by the one or more processors, a graphical user interface of a first computing device to present a visual indication of the updated overall quality of the plurality of content assets; The method of claim 1 further comprising:

3. comparing the updated overall quality by the one or more processors to preset quality heuristics; determining, by the one or more processors, that the updated overall quality does not conform to the preset quality heuristics; generating, by the one or more processors, one or more recommendations for improving the updated overall quality in response to determining that the updated overall quality does not conform to the preset quality heuristics; updating, by the one or more processors, the graphical user interface of the first computing device to present the one or more recommendations; The method of claim 2 further comprising:

4. The method of claim 3 , wherein the one or more recommendations include a first recommendation for modifying a visual feature of an image.

5. assessing the first quality of each of the one or more images deploying, by the one or more processors, the one or more machine learning models on the image to generate a quality feature score for each of a plurality of quality features; assigning a weight to each score by the one or more processors to generate a weighted score; combining, by the one or more processors, the weighted scores to generate a combined score for the image; and comparing, by the one or more processors, the combined score to one or more thresholds to generate the first quality of the image.

6. determining the overall quality of the plurality of content assets, determining a total possible score for each image; calculating, for each image, a ratio of the combined score to the possible total score, wherein the ratio for the image is a portion of one or more ratios for the one or more images; calculating an average of the one or more ratios, the average of the one or more ratios being indicative of the overall quality of the plurality of content assets; The method according to claim 5, which is based on claim 2 and includes:

7. a memory for storing computer executable instructions; one or more computers configured to execute the instructions, wherein execution of the instructions causes the one or more computers to perform the operations of any one of claims 1 to 6; A system including:

8. A computer-readable storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any one of claims 1 to 6.

Images