Artificial intelligence for efficient image editing

Abstract

Methods, systems, and apparatuses for enabling artificial intelligence to generate updated images based on one or more conditions are described, including a computer program encoded on a computer storage medium. In one aspect, the method includes receiving data indicating that a first image violates one or more conditions. In response to receiving the data indicating that the first image violates the one or more conditions, an image editing prompt is generated that instructs an image editing model to edit the first image to satisfy the one or more conditions. The image editing prompt and the first image are provided as inputs to the image editing model. A second image is received as an output of the image editing model. The second image is provided to one or more devices.

Description

[Background technology]

[0001] This specification relates to data processing, artificial intelligence, and generating images using artificial intelligence.

[0002] Advances in machine learning are enabling artificial intelligence to be implemented in more applications. For example, large-scale language models are being implemented to enable image editing, allowing for more efficient image editing using the information provided in association with the image. Summary of the Invention [Means for solving the problem]

[0003] In general, one innovative aspect of the subject matter described herein can be embodied in a method including the actions of: receiving data indicating that a first image violates one or more conditions; generating, in response to receiving the data indicating that the first image violates the one or more conditions, an image editing prompt that instructs an image editing model to edit the first image to satisfy the one or more conditions, the image editing prompt including generating descriptive data indicating the location of content within the image that violates the one or more conditions; generating the image editing prompt based on the descriptive data and the one or more conditions using a language model; providing the image editing prompt and the first image as input to the image editing model; receiving a second image as output of the image editing model; and providing the second image to one or more devices. Other implementations of this aspect include corresponding apparatuses, systems, and computer programs encoded on computer storage devices and configured to perform aspects of the method.

[0004] Each of these and other embodiments may optionally include one or more of the following features: In some aspects, generating the explanation data includes providing the image and the one or more conditions to a multimodal model trained to identify locations within the image that violate the input conditions.

[0005] In some aspects, generating the explanation data includes providing the image and one or more conditions to a multimodal model trained to predict whether the image violates the input conditions and to output data indicating the location of content within the image that may violate at least one of the input conditions.

[0006] In some aspects, generating the description data includes providing the first image to a first machine learning model trained to generate an image caption for the image; receiving an image caption for the image from the first machine learning model; providing the image caption and one or more conditions to a second machine learning model trained to output description data for the image based on the input image caption and the input conditions; and receiving the description data for the first image from the second machine learning model.

[0007] In some aspects, the description data includes a location indicator. The location indicator indicates a location of content within the first image determined to violate at least one of the one or more conditions. The location indicator may include a bounding box depicted in the first image around the content within the first image determined to violate at least one of the one or more conditions. The location indicator may include coordinates defining a bounding box around the content within the first image determined to violate at least one of the one or more conditions. Providing the image editing prompt and the first image as input to the image editing model may include providing the location indicator to the image editing model.

[0008] In some aspects, the explanation data includes an explanation indicating why the first image violates one or more conditions. In some aspects, the image editing prompt includes at least a portion of the explanation data.

[0009] In some aspects, using a language model to generate an image editing prompt based on the explanatory data and the one or more conditions includes generating a prompt for the language model using the explanatory data and providing the prompt to the language model. The prompt can include instructions instructing the language model to generate the image editing prompt based on the explanatory data and data defining each condition violated by the first image. The explanatory data can include a name of each condition violated by the first image.

[0010] In some embodiments, generating the image editing prompt includes obtaining a prompt template adapted to the image editing model and inputting at least a portion of the descriptive data into the prompt, the descriptive data including the name of the condition that the first image has been determined to violate. The image editing prompt output by the language model is adapted to the image editing model.

[0011] Particular embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages: The techniques described herein enable the use of artificial intelligence (AI) to generate updated images based on data related to one or more conditions. An AI system can evaluate images that fail to meet a condition(s) and use one or more machine learning models to determine why the image fails to meet the condition(s) and / or what portions of the image cause the image to fail to meet the condition(s). Without the described techniques, a user may struggle to determine why an image is rejected by a system that evaluates images based on conditions and attempt to modify the image in various ways to satisfy the condition(s). This can result in the user making unnecessary modifications, resulting in a lower-quality image. This can also result in the user uploading multiple versions of an image to the system in an attempt to find a version that meets the condition(s), placing unnecessary strain on the system and the network connecting the system to the user's device, thereby wasting computing resources and network bandwidth. Additionally, uploading multiple versions of an image can raise data privacy / confidentiality issues, as consistent uploads enable adversarial attacks against machine learning models that learn how the model works. Even worse, some systems may use many different criteria for different types of images, or for images of different types of items. Thus, an image may meet one or more criteria for one type of item (e.g., one type of product), but not another.

[0012] The technology described herein can solve these problems by using AI to determine why an image fails to meet a set of conditions and edit the image so that it meets the conditions. In this way, a user need only upload an image once to obtain a conforming image, thereby saving resources that would otherwise be wasted uploading and evaluating multiple images. For example, a user can submit a first image to the system, and the system can determine whether the first image violates the conditions. If so, the system can use a machine learning model (e.g., an image editing model) to update the image so that the updated image does not violate the conditions, thus allowing the user to efficiently submit images to the system without having to upload additional images that the user desires to comply with the conditions. In this way, conforming images are generated without the need to upload multiple images, thereby reducing the computational load placed on the system that evaluates the images (e.g., processing cycles to evaluate the images, data storage to store the images, etc.), reducing the amount of network bandwidth consumed and the associated load placed on network resources for transmitting multiple images, and reducing the computational load placed on the user's device when modifying and transmitting the image to the system that evaluates the image.

[0013] A chain of prompts to one or more machine learning models can be used to evaluate an image to determine why the image fails to meet a set of conditions and, based on the evaluation, generate an updated image. In this way, tasks are separated, and models trained for a specific task can be used to generate higher-quality outputs that take into account the outputs of previous models. This enables the system to generate higher-quality explanations of why the image fails to meet the conditions and to generate high-quality, specialized image editing prompts that precisely instruct the image editing model to correct the image. By separating the process into multiple separate AI tasks, the amount of information provided to the AI ​​model per task can be reduced, thereby preventing hallucinations and other AI model errors that typically occur when a significant amount of information is provided as input to an AI model.

[0014] Multiple machine learning models can be used to evaluate and edit images based on conditions. For example, a multimodal model with question-and-answer and image generation capabilities can evaluate an image and a set of conditions and output explanatory data indicating why and / or which portions of the image cause the image to fail to meet the conditions. A language model can then generate an image editing prompt based on the explanatory data, and an image editing model, such as a text-to-image model, can generate an updated version of the image based on the prompt and the image. Using multiple models in this manner allows for more accurate output, and therefore higher quality images, than using a single model to perform all of these different tasks. Furthermore, using multiple models rather than a single model to handle all tasks in the process allows the system to scale each model for deployment and produce higher quality images while efficiently managing resources. In particular, image editing models may be relatively larger and more complex than other models (e.g., models for generating image editing prompts for image editing models), resulting in improved image quality, higher throughput, reduced memory usage, and reduced latency.

[0015] Using a multimodal model to generate explanatory data and a language model to generate prompts for an image editing model enables the use of off-the-shelf, general-purpose, or other pre-trained image editing models without customizing or retraining the image editing model. Instead, the techniques described herein can include the use of specialized prompts that instruct a language model to generate image editing prompts specifically tailored to the image editing model. This therefore eliminates the need to adapt or retrain the image editing model to accept input generated by other models while ensuring that the image editing model produces high-quality edited images. Using a combination of models in this manner allows the overall AI system to initially generate edited images that comply with a large set of conditions, thereby reducing the number of images uploaded for evaluation and the number of evaluations performed on images, and providing various computational savings described above and elsewhere herein. Thus, the techniques described herein provide specific application to AI models and prompts to apply conditions to images and solve problems that arise when generating images that meet the conditions more generally in the automated image generation field.

[0016] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram of an exemplary environment in which images are evaluated based on criteria and edited using artificial intelligence. [Figure 2] FIG. 1 is a block diagram illustrating interactions between an artificial intelligence system, a multimodal model, a language model, and an image editing model. [Figure 3]1 is a flowchart of an exemplary process for generating an image based on one or more conditions. [Figure 4] FIG. 1 is a block diagram of an exemplary computer. DETAILED DESCRIPTION OF THE INVENTION

[0018] Like reference symbols and designations in the various drawings indicate like elements.

[0019] This specification describes techniques for enabling artificial intelligence to generate updated images based on one or more conditions. For example, the techniques can be used to evaluate images that do not meet one or more conditions and edit the images so that the edited images meet the condition(s). Artificial intelligence (AI) is a field of computer science that focuses on creating intelligent agents that can learn and act autonomously (e.g., without human intervention). Artificial intelligence can utilize machine learning, which focuses on developing algorithms that can learn from data; natural language processing, which focuses on understanding and generating human language; and / or computer vision, a field that focuses on understanding and interpreting images and videos.

[0020] The techniques described throughout this specification enable an AI model to edit an image that fails to satisfy one or more conditions so that the edited image satisfies the condition(s). The conditions may be, for example, policy conditions of an entity that makes the image available for viewing by others, such as an entity that distributes the image digital component to a user. Generally speaking, an AI system may receive data indicating that an image violates a condition, and the AI ​​system can use an image editing model to generate an updated version of the image that does not violate the condition, for example, by editing the image. The AI ​​system can use one or more machine learning models, such as, for example, a language model and / or a multimodal model (which may be a language model or other type of multimodal model), to evaluate the image to determine why the image fails to satisfy the condition(s) and generate an image editing prompt that instructs the image editing model (which may be a language model) to edit the image in a particular way so that the edited image satisfies the condition(s). The AI ​​system can provide the image editing prompt and the image to the image editing model, and the image editing model can edit the image and output an updated version of the image.

[0021] As described herein, using AI to evaluate images based on conditions and generate image editing prompts enables the creation of specialized image editing prompts that instruct an image editing model to make appropriate edits to the image to produce an image that satisfies the conditions without unnecessary edits, thereby producing a high-quality image that is as close as possible to the original image while satisfying the conditions. Furthermore, specialized prompts are provided to a language model to ensure that the image editing prompts provided to the image editing model are tailored to the image editing model so that the image editing model does not need to be adapted or retrained for use with the language model. Using AI to edit images in this manner reduces the waste of computing resources that would otherwise be used to generate and upload multiple versions of an image and evaluate the multiple versions until a version that meets the conditions is reached. This also increases the likelihood of creating a matching image compared to a user who edits an image without fully understanding the conditions that the image does not meet. All of this contributes to a system that can more quickly generate updated images that meet the conditions, allowing images to be created and provided in a real-time interactive environment, for example, in response to a user search query or a component request for a digital component to display on a user's device.

[0022] As used throughout this document, the phrase "digital component" refers to a discrete unit of digital content or information (e.g., a video clip, an audio clip, a multimedia clip, game content, an image, text, a bulleted list, artificial intelligence output, language model output, or other unit of content). A digital component can be stored electronically on a physical memory device as a single file or as a collection of files; a digital component can take the form of a video file, an audio file, a multimedia file, an image file, or a text file; and can include advertising information; thus, an advertisement is a type of digital component.

[0023] 1 is a block diagram of an exemplary environment 100 in which images are evaluated based on conditions and edited using artificial intelligence. The exemplary environment 100 includes a network 102, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects electronic document servers 104, user devices 106, digital component servers 108, and service appliances 110. The exemplary environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0024] The service apparatus 110 is configured to provide various services to client devices 106 and / or publishers of electronic documents 150. In some embodiments, the service apparatus 110 can provide search services by providing responses to search queries received from client devices 106. For example, the service apparatus 110 can include a search engine and / or an AI agent or other chat agent that allows a user to interact with the agent over the course of multiple conversational queries and responses. The service apparatus 110 can also deliver digital components to the client devices 106 for presentation along with the responses and / or along with the electronic documents 150. For example, other search service computer systems can send component requests 112 to the service apparatus 110, and these component requests 112 can include one or more queries. The service apparatus 110 and component requests 112 are described in further detail below.

[0025] A client device 106 is an electronic device capable of requesting and receiving online resources over network 102. Exemplary client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices capable of sending and receiving data over network 102. A client device 106 typically includes a user application, such as a web browser, to facilitate sending and receiving data over network 102, although native applications (other than a browser) executed by client device 106 may also facilitate sending and receiving data over network 102.

[0026] A gaming device is a device that allows a user to participate in a gaming application, for example, by allowing the user to control one or more characters, avatars, or other rendered content presented in the gaming application. A gaming device typically includes a computer processor, a memory device, and a controller interface (physical or visually rendered) that allows user control over content rendered by the gaming application. A gaming device can store and execute gaming applications locally, or execute gaming applications that are at least partially stored and / or provided by a cloud server (e.g., online gaming applications). Similarly, a gaming device can execute gaming applications and interface with a gaming server that "streams" the gaming application to the gaming device. A gaming device may be a tablet device, a mobile communication device, a computer, or other device that performs functions other than running gaming applications.

[0027] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input via voice and can respond with content using audible feedback and present other audible information. In some situations, the digital assistant device also includes a visual display or is in communication with a visual display (e.g., via a wireless or wired connection). If a visual display is present, feedback or other information can also be provided visually. In some situations, the digital assistant device can also control other devices, such as lights, locks, cameras, room temperature control devices, alarm systems, and other devices registered with the digital assistant device.

[0028] As shown, client device 106 is presenting electronic document 150. An electronic document is data that presents a set of content at client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search result pages, and feed sources. Native applications (e.g., "apps" and / or game applications), such as applications installed on mobile, tablet, or desktop computing devices, are also examples of electronic documents. Electronic documents may be provided to client device 106 by electronic document servers 104 ("Electronic Doc Servers").

[0029] For example, the e-document server 104 may include a server hosting a publisher's website. In this example, a client device 106 may initiate a request for a web page of a given publisher, and the e-document server 104 hosting the web page of the given publisher may respond to the request by sending machine-executable instructions that initiate the presentation of the given web page at the client device 106.

[0030] In another example, the electronic document server 104 may include an app server from which the client device 106 can download apps. In this example, the client device 106 may download files needed to install the app on the client device 106 and then execute the downloaded app locally (i.e., on the client device). Alternatively or additionally, the client device 106 may initiate a request to execute the app, which is sent to a cloud server. In response to receiving the request, the cloud server may execute the application and stream the application's user interface to the client device 106 so that the client device 106 does not have to execute the app itself. Rather, the client device 106 may present the user interface generated by the cloud server executing the app and transmit any user interactions with the user interface back to the cloud server for processing.

[0031] Electronic documents can include a variety of content. For example, electronic document 150 can include native content 152 that is internal to electronic document 150 itself and / or does not change over time. Electronic documents can also include dynamic content that can change over time or on a per-request basis. For example, the publisher of a given electronic document (e.g., electronic document 150) can maintain data sources used to populate portions of the electronic document. In this example, the given electronic document can include scripts, such as script 154, that cause client device 106 (or cloud server) to request content (e.g., digital components) from data sources when the given electronic document is processed (e.g., rendered or executed) by client device 106 (or cloud server). Client device 106 (or cloud server) integrates the content (e.g., digital components) retrieved from the data sources into the given electronic document to create a composite electronic document that includes the content retrieved from the data sources.

[0032] In some situations, a given electronic document (e.g., electronic document 150) may include a digital component script (e.g., script 154) that references the service device 110 or a particular service provided by the service device 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. Execution of the digital component script configures the client device 106 to generate a request for the digital component 112 (referred to as a “component request”), which is transmitted over the network 102 to the service device 110. For example, the digital component script may enable the client device 106 to generate a packetized data request that includes a header and payload data. The component request 112 may include event data specifying characteristics such as the name (or network location) of the server from which the digital component is being requested, the name (or network location) of the requesting device (e.g., client device 106), and / or information the service device 110 can use to select one or more digital components or other content to be provided in response to the request. The component request 112 is sent by the client device 106 to a server of the service apparatus 110 over the network 102 (eg, a telecommunications network).

[0033] The component request 112 may include event data specifying the requested electronic document and other event characteristics, such as characteristics of the location of the electronic document where the digital component can be presented. For example, event data specifying a reference (e.g., a URL) to the electronic document (e.g., a web page) where the digital component will be presented, available locations of the electronic document available for presenting the digital component, the size of the available locations, and / or media types eligible for presentation at the locations may be provided to the service device 110. Similarly, event data specifying keywords associated with the electronic document (“document keywords”) or entities referenced by the electronic document (e.g., people, places, or things) may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate identification of digital components eligible for presentation with the electronic document. The event data may also include a search query submitted by the client device 106 to obtain a search result page.

[0034] The component request 112 may also include event data related to other information, such as information provided by a user of a client device, geographic information indicating the state or region in which the component request was submitted, or other information that provides context about the environment in which the digital component is displayed (e.g., the time of the component request, the day of the week for the component request, the type of device on which the digital component is displayed, such as a mobile device or tablet device). The component request 112 may be transmitted, for example, over a packetized network, and the component request 112 itself may be formatted as packetized data having a header and payload data. The header may specify the destination of the packet, and the payload data may include any of the information described above.

[0035] Service device 110 selects a digital component (e.g., third-party content such as video files, audio files, images, text, game content, augmented reality content, and combinations thereof, all of which can be in the form of advertising or non-advertising content) to be presented with a given electronic document (e.g., at a location specified by script 154) in response to receiving component request 112 and / or using information included in component request 112. In some implementations, selecting a digital component includes selecting a digital component based on textual features.

[0036] In some implementations, the digital component is selected in less than one second to avoid errors that would be caused by delays in selecting the digital component. For example, delays in providing the digital component in response to the component request 112 could cause page load errors on the client device 106 or could cause portions of the electronic document to remain unpopulated even after other portions of the electronic document have been presented on the client device 106. The described techniques are adapted to generate the digital component in a short amount of time such that these errors and their impact on the user experience are reduced or eliminated.

[0037] Also, as the delay in providing the digital component to the client device 106 increases, the likelihood that the electronic document will not be presented at the client device 106 when the digital component is delivered to the client device 106 increases, thereby adversely affecting the user experience with the electronic document. Furthermore, delays in providing the digital component can result in failed delivery of the digital component, for example, if the electronic document is not presented at the client device 106 when the digital component is provided.

[0038] In some embodiments, the service appliance 110 is implemented in a distributed computing system, including, for example, a server and a set of interconnected computing devices 114 that identify and deliver digital components in response to requests 112. The set of computing devices 114 may store millions of available digital components (DCs). 1-x ), which work together to identify a set of digital components that are eligible to be presented in an electronic document. Millions of available digital components may be indexed, for example, in the digital component database 116. Each digital component index entry may reference a corresponding digital component and / or may include delivery parameters (DP1 through DP2) that contribute to (e.g., trigger, condition, or limit) the delivery / transmission of the corresponding digital component. x For example, the delivery parameters may contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., either exactly or at some pre-specified level of similarity) one of the delivery parameters of the digital component.

[0039] In some embodiments, the distribution parameters for a particular digital component may include distribution keywords that must be matched (e.g., by electronic documents, document keywords, or terms specified in the component request 112) for the digital component to be eligible for presentation. Additionally or alternatively, the distribution parameters may include embeddings that can use various different dimensions of data, such as website details and / or consumption details (e.g., page viewports, user scrolling speed, or other information about data consumption). The distribution parameters may also require that the component request 112 include information specifying a particular geographic region (e.g., country or state) and / or information specifying that the component request 112 originated from a particular type of client device (e.g., a mobile device or tablet device) for the digital component to be eligible for presentation. The distribution parameters may also specify an eligibility value (e.g., a ranking score or some other specified value) used to evaluate the eligibility of the digital component for distribution / transmission (e.g., among other available digital components).

[0040] The identification of eligible digital components can be segmented into multiple tasks 117a-117c, which are then allocated among computing devices in the set of multiple computing devices 114. For example, different computing devices in the set 114 can each analyze different portions of the digital component database 116 to identify various digital components having delivery parameters that match the information included in the component request 112. In some implementations, each given computing device in the set 114 can analyze a different data dimension (or set of dimensions) and pass (e.g., transmit) results (Res1-Res3) 118a-118c of the analysis to the service appliance 110. For example, the results 118a-118c provided by each of the computing devices in the set 114 can identify a subset of digital components that are eligible for delivery in response to the component request and / or a subset of digital components that have particular delivery parameters. Identifying the subset of digital components can include, for example, comparing event data to the delivery parameters and identifying the subset of digital components having delivery parameters that match at least some characteristics of the event data.

[0041] The service apparatus 110 aggregates the results 118a-118c received from the set of multiple computing devices 114 and uses information associated with the aggregated results to select one or more digital components to provide in response to the request 112. For example, the service apparatus 110 may select the set of acquired digital components (one or more digital components) based on the results of one or more content evaluation processes, as described below. The service apparatus 110 may then generate and transmit, via the network 102, reply data 120 (e.g., digital data representing a response) that enables the client device 106 to integrate the set of acquired digital components into a given electronic document, such that the set of acquired digital components (e.g., acquired third-party content) and the content of the electronic document are presented together on the display of the client device 106.

[0042] In some implementations, the client device 106 executes instructions included in the reply data 120 that configure the client device 106 to retrieve a set of acquired digital components from one or more digital component servers 108, enabling the client device 106 to retrieve the set of acquired digital components from one or more digital component servers 108. For example, the instructions in the reply data 120 may include a network location (e.g., a uniform resource locator (URL)) and a script that causes the client device 106 to send a server request (SR) 121 to the digital component server 108 to retrieve the given acquired digital component from the digital component server 108. In response to the request, the digital component server 108 identifies the given acquired digital component specified in the server request 121 (e.g., in a database that stores multiple digital components) and sends the client device 106 digital component data (DC data) 122 that presents the given acquired digital component within an electronic document at the client device 106.

[0043] When the client device 106 receives the digital component data 122, the client device renders the digital component (e.g., third-party content) and presents the digital component at a location specified by or assigned to the script 154. For example, the script 154 may create a walled garden environment, such as a frame, that is presented within the native content 152 of the electronic document 150, e.g., next to the native content 152 of the electronic document 150. In some implementations, the digital component is overlaid on top of (or adjacent to) a portion of the native content 152 of the electronic document 150, and the service device 110 may specify the presentation location within the electronic document 150 in the reply 120. For example, when the native content 152 includes video content, the service device 110 may specify a location or object within a scene depicted in the video content over which the digital component is to be presented.

[0044] Service device 110 may also include an artificial intelligence system 160 configured to autonomously generate digital components either prior to request 112 (e.g., offline) and / or in response to request 112 (e.g., online or in real time). As described in more detail throughout this specification, artificial intelligence ("AI") system 160 may collect online content about a particular entity (e.g., a digital component provider or other entity) and summarize the collected online content using one or more language models 170, which may include large-scale language models.

[0045] A large-scale language model ("LLM") is a model trained to generate and understand human language. LLMs are trained on large datasets of text and code and can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another; summarize text such as website content, search results, news articles, or research papers; answer questions about text such as "What is the capital of Georgia?"; create chatbots that can have conversations with humans; and generate creative text such as poetry, stories, and code.

[0046] Language model 170 may be any suitable language model neural network that receives an input sequence of text tokens selected from a lexicon and autoregressively generates an output sequence of text tokens from the lexicon. For example, language model 170 may be a transformer-based language model neural network or a recurrent neural network-based language model.

[0047] In some circumstances, when the neural network used to implement language model 170 autoregressively generates an output sequence of tokens, language model 170 can be referred to as an autoregressive neural network. More specifically, the autoregressively generated output is created by generating each particular token in the output sequence conditional on the current input sequence, including every token that precedes a particular text token in the output sequence, i.e., the token already generated for every previous position in the output sequence that precedes the particular position of the particular token, and contextual input that provides context for the output sequence.

[0048] For example, the current input sequence when generating a token at any given position in the output sequence may include the input sequence and any preceding tokens that precede the given position in the output sequence. As a specific example, the current input sequence may include the input sequence followed by any preceding tokens that precede the given position in the output sequence. Optionally, the input sequence and the current output sequence may be separated by one or more predetermined tokens in the current input sequence.

[0049] More specifically, to generate a particular token at a particular position in the output sequence, the neural network of language model 170 may process the current input sequence to generate a score distribution, e.g., a probability distribution, that assigns a respective score, e.g., a respective probability, to each token in the lexicon of tokens. The neural network of language model 170 may then use the score distribution to select a token from the lexicon as the particular token. For example, the neural network of language model 170 may greedily select the highest-scoring token or may sample the token from the distribution using, for example, kernel sampling or other sampling techniques.

[0050] As a particular example, language model 170 may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each of which applies a self-attention operation, and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0051] The price tag 170 is a slightly less expensive one of the snowflakes These include J. Hoffmann, S. Borgeau d, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdLCasas, LAHendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv database, arXiv:2203.15556, 2022, JWRae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HFSong, J.A slanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van then Driessche, LAHendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Hig gins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskay, D. Budden, E. Sutherland, K. Simonyan. M. Paganini, L. Sifre, L. Martens, XLLi, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mes ch, J. Lespiau, M. Tsimpoukelli, N. Grigoev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BAHechtman, L. Weidinger, I. Gabriel, WSIsaac, E. Lockhart, S. Osindero, L.Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving: Scaling language models; gopher, CoRR, abs / 2122.11446, 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer arXiv Publication: arXiv:1910.10683, Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V.Le, Towards a Human-Like Open-Domain Chatbot, CoRR, abs / 2001.09977, 2020, Tom B Brown, and Benjamin Mann、Nick Ryder、Melanie Subbiah、Jared Kaplan、Prafulla Dhariwal、Arvind Neelakantan、Pranav Shyam、Girish Sastry、Amanda Askell, et al.、Language models are few-shot learners.The arXiv database is arXiv:2005.14165.

[0052] In general, however, a Transformer-based neural network includes a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a respective input hidden state for each input token in the given input sequence. The attention blocks then at least partially update each of the hidden states by applying self-attention to generate a respective output hidden state for each input token. The input hidden state of the first attention block is the embedding of the input token in the input sequence, and the input hidden state of each subsequent attention block is the output hidden state generated by the preceding attention block.

[0053] In this example, the output sub-network processes the output hidden state produced by the last attention block in the sequence of the last input token in the input sequence to produce a score distribution.

[0054] In general, because language models are autoregressive, the service device 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, for example, by using beam search decoding from the score distribution generated by the language model 170, by using a sample-and-rank decoding strategy, by using different random number seeds for a pseudo-random number generator used in sampling for different runs through the language model 170, or by using other decoding strategies that exploit the autoregressive nature of the language model.

[0055] In some implementations, language model 170 is pre-trained, i.e., trained on language modeling tasks that do not require providing evidence in response to user questions, and service device 110 (e.g., using AI system 160) causes language model 170 to generate output sequences according to a predetermined syntax via natural language prompts in input sequences.

[0056] For example, service device 110 (e.g., AI system 160), or a separate training system, pre-trains language model 170 (e.g., a neural network) on a language modeling task, such as a task requiring, given a current sequence of text tokens, to predict the next token that follows the current sequence in training data. As a specific example, language model 170 may be pre-trained with a maximum likelihood objective on a large dataset of text, such as text that is publicly available from the Internet or other text corpus.

[0057] The AI ​​system 160 can use the language model 170 to generate image editing prompts for editing images that fail to meet one or more conditions. The service appliance 110 can maintain conditions for digital components and / or other images sent to the client device 106. For example, conditions can be used to ensure that the digital components do not contain explicit content or language. The conditions can vary based on the type of item corresponding to the image and / or based on the type of image. For example, there may be a first set of conditions for a first type of product and a second set of conditions for a second type of product. Furthermore, the service appliance 110 can maintain conditions for each publisher of the electronic document 150. For example, a publisher that publishes web pages aimed at children may have stricter conditions for digital components presented on those web pages.

[0058] In some implementations, language model 170 can be adapted to generate image editing prompts using zero-shot learning or few-shot learning. In some examples, the system can provide manually created examples to language model 170 for few-shot learning. In few-shot learning, AI system 160 can provide language model 170 with a small number of training examples (e.g., 3-7, 10, or some other number). These training examples include an original image, a set of conditions, and an image editing prompt that includes instructions for editing the image to meet the conditions.

[0059] The service device 110 can evaluate the image digital component and / or other images to determine whether those images meet a set of conditions for each image. For images that do not meet the conditions, the service device 110 can use the AI ​​system 160 to evaluate why the image does not meet the conditions and / or to edit the image so that it meets the conditions.

[0060] AI system 160 can use language model 170 to generate an image editing prompt that instructs the image editing model to generate an updated image that satisfies the condition. For example, AI system 160 can generate prompt 172 that instructs language model 170 to generate an image editing prompt based on explanation data related to an explanation for why the image does not satisfy the condition. In some implementations, prompt 172 includes explanation data. The explanation data may include data indicating that the image violates a condition, a description of the violation (e.g., the name of the policy that is violated), the portion of the image that causes the image to violate the condition, the condition itself, and / or other data. The data indicating the portion of the image that causes the image to violate the condition may include a location indicator depicted within the image.

[0061] For example, the condition definition might state, "If [policy definition], then the image violates the [policy name] policy." In this example, the description data might state, "This particular image violates the [policy name] policy because [explanation of why the image violates the policy]." Prompt 172 might then state, "If [policy definition], then the image violates the [policy name] policy. This particular image violates the [policy name] policy because [explanation of why the image violates the policy]. My task is to edit the image so that it does not violate the [policy name] policy. I use the diffusion model by providing it with the original image and image editing prompts. Generate the image editing prompts to make the image conform to the policy. Ensure that the image editing prompts do not remove or edit anything important that the image is trying to communicate." The text in parentheses can be populated with appropriate data by the AI ​​system 160. For example, the AI ​​system 160 can populate [policy name] with the policy name of the policy that the original image data violates.

[0062] Here, prompt 172 is specifically adapted to instruct language model 170 for the subsequent task to generate an image editing prompt for the diffusion model by informing language model 170, "I will use the diffusion model by providing the diffusion model with the original image and the image editing prompt." In this way, language model 170 can generate an image editing prompt specifically adapted for the diffusion model that accepts the image editing prompt and the original image as input. This eliminates the need to adapt or retrain the image editing model.

[0063] In some implementations, AI system 160 can maintain a prompt template for each of one or more image editing models. Each prompt template can be in the form of an example prompt 172 with bracketed fields that AI system 160 can populate using policy information and explanation information. For example, when evaluating and editing an image that violates conditions, such as a policy condition, AI system 160 can receive explanation data from a multimodal model with question-and-answer and image editing capabilities and use the explanation data to populate the template.

[0064] If a multimodal language model is used as language model 170, prompt 172 may include an image. If language model 170 accepts only text input, prompt 172 may include a caption that describes the content of the image to language model 170.

[0065] In some examples, the explanatory data may include an image with a bounding box around the portion of the image that is deemed to be the reason the image violates the condition. In this example, the bounding box is a location indicator, and the language model 170 is a multimodal model that accepts text and image prompts 172 as input.

[0066] In some implementations, AI system 160 generates explanation data using language model 170. For example, AI system 160 can use language model 170 or another model to generate a caption for an image. The caption can describe the content of the image. For example, AI system 160 can generate prompt 172 that instructs language model 170 or another model (e.g., another multimodal model described with reference to FIG. 2) to generate a caption for an input image. AI system 160 can receive the caption, evaluate the caption and a set of conditions, and generate another prompt 172 that instructs language model 170 to output an explanation of why the image described by the caption violates the conditions or which parts of the image described by the caption violate the conditions. In some examples, prompt 172 includes a list of objects identified in the image based on bounding boxes. In this example, prompt 172 can include a caption and a set of conditions.

[0067] For example, prompt 172 may state: "Image violates [policy name] policy when [policy definition]. This image contains [image description]. The following objects and their bounding boxes are included in the more spatially aware context of the image, i.e., [Object 1:<x1、y1、x2、y2> , object 2:<x1、y1、x2、y2> "To provide you with a prediction of whether the image violates the [policy name] policy based on the context I provide you about the image. To provide a very detailed explanation of your decision, including the areas that violate the policy." In this example, x1 and x2 are coordinate axes in the image along one dimension (e.g., the x-axis or horizontal direction), and y1 and y2 are coordinates in the image along the other dimension (e.g., the y-axis or vertical direction). These coordinates tell language model 170 where objects 1 and 2 can be found in the image.

[0068] In some implementations, AI system 160 uses a multimodal model to generate explanation data based on an image and a set of conditions. In this example, AI system 160 provides the multimodal model with an image and a set of conditions and requests that the multimodal model output explanation data, such as an explanation for the image with a bounding box or other location indicator indicating why the image does not satisfy the conditions and / or the portion of the image that violates the conditions.

[0069] In either example, language model 170 can evaluate prompt 172 using the description data and generate output 174, including an image editing prompt, based on the input data. Language model 170 can generate the image editing prompt in a manner (e.g., structured) that directs the image editing model to generate an updated image based on the input image and prompt.

[0070] The AI ​​system 160 can use the image editing prompts to generate an updated image using an image editing model, as described in further detail with reference to FIG. 2. The updated image is an edited version of the initial image that does not violate the condition. For example, the updated image may not include or depict an area (e.g., location of content) in the image that violates the condition. In certain examples, the image editing model may replace the content in that portion of the image with content that satisfies the condition.

[0071] For example, an image editing prompt to obscure a specific area of ​​a person in an image or remove an item may state, "cover [body part] of the person and remove [item] from the image." Another example of an image editing prompt to edit a person in an image may state, "make the facial expression of the person(s) in the image neutral." Another example of an image editing prompt to remove an item from an image may state, "crop the image to remove an item from the image." Another example of an image editing prompt to remove an item from an image may state, "erase the adult beverage bottle from the image."

[0072] The AI ​​system 160 can then send (e.g., provide) the updated image to one or more devices (e.g., one or more client devices 106) as a reply 120. For example, the AI ​​system 160 can generate a digital component for provision in response to a request 112 from a user. The digital component can include the updated image. The digital component can include electronic documents, metadata related to the subject matter of the digital component (e.g., the item depicted by the image), and / or links to other data and / or files that enable the client device 106 to render the updated image.

[0073] Although a single language model 170 is shown in Figure 1, different language models may be trained specifically to process different prompts at different stages in the processing pipeline. For example, a language model may be trained to generate description data for an image, while another language model may be trained to generate image editing prompts based on the description data.

[0074] 2 is a block diagram 200 illustrating interactions between an AI system 160, a multimodal model 202, a language model 170, and an image editing model 204. The AI ​​system 160 can include an image assessment device 206, a prompt device 208, and a digital component device 210.

[0075] Language model 170 can be trained to perform various tasks, as described above. AI system 160 can use language model 170 to generate explanatory data and / or generate image editing prompts for image editing model 204. While one language model 170 is shown in FIG. 2, AI system 160 can interact with any number of language models 170 to generate image editing prompts to instruct image editing model 204 to generate updated images that satisfy one or more conditions, such as one or more policy conditions.

[0076] The multimodal model 202 can be implemented as a machine learning model trained to generate explanation data 212. For example, the training process can use a set of training images and ground truth explanation data corresponding to the training data. For example, the ground truth training data can include, for each image that violates a condition, a label indicating the condition that was violated and why the image violates the condition. The label can also indicate the part of the image that violates the condition. Based on this set of training images, the multimodal model 202 can be trained to generate explanation data 212.

[0077] The multimodal model 202 can be trained to generate text based on text and image input. For example, the multimodal model 202 can be trained to output an explanation explaining why an image violates one or more conditions based on input including, as explanatory data, an image and text illustrating one or more conditions. In some implementations, the multimodal model 202 can be trained to output, as explanatory data, an image with a location indicator (e.g., a bounding box) indicating the portion of the image that violates the condition. The bounding box can be outlined using the x and y coordinates of the image, similar to the example provided above. The multimodal model 202 can receive images and text as input, and the multimodal model 202 can generate text as output. During training, images and questions (e.g., text asking a question about the explanatory data) are used as inputs for the model, and the multimodal model 202 is trained to generate answers (e.g., text that answers the question about the explanatory data). In a supervised learning example, training samples can include images and questions including conditions, along with labels with answers including explanations explaining why the image does not satisfy the condition. In some implementations, the multimodal model 202 may be a neural network or other type of machine learning model trained to respond to questions, provide answers, and edit images.

[0078] The image editing model 204 may be a machine learning model, such as a text-to-image neural network, trained to generate images based on an input image and image editing prompts 215 that instruct the image editing model 204 on how to edit the image. In some implementations, the image editing model 204 is a language model trained to edit images. In some implementations, the image editing model 204 is a diffusion model.

[0079] During training, the image editing model 204 can take as input an original image caption and an image editing prompt for an image, and the image editing model 204 can be trained to generate a target text prompt by applying the image editing prompt to the image caption. Given the image and the target text prompt, the image editing model 204 can encode the target text prompt to generate an initial text embedding. The image editing model 204 then processes (e.g., optimizes) the initial text embedding to reconstruct the input image. The system then fine-tunes the image editing model 204 (e.g., a diffusion model of the image editing model 204) to improve overall accuracy by interpolating the target text prompt with the input image to generate the output of the image editing model (e.g., an edited image).

[0080] The AI ​​system 160 may also include or be configured to interact with a memory structure 216 to extract and / or store information and content. The memory structure 218 may include one or more databases or other data structures stored in one or more memories and / or data storage devices. In particular, the memory structure 218 may store the digital component database 116, digital components 220, images 222, and condition data 224.

[0081] As described above, the digital component database 116 may include distribution parameters for the digital components 220. The distribution parameters for the digital components 220 may include, for example, keywords and / or geographic locations for which the digital component 220 is eligible to be distributed to the client device 106. The digital component database 116 may also include, for each digital component 220, metadata for the digital component, a caption for each image 222 corresponding to the digital component, data related to the digital component provider providing the digital component, and / or other data related to the digital component. The digital components 220 may include candidate digital components that can be provided in response to component requests 112 and / or queries received by the service appliance 110. The images 222 may include one or more images for each digital component 220. The AI ​​system 160 may obtain images for the digital components 220 from the digital component provider or from other sources.

[0082] The conditions database 224 can store conditions for images. The conditions database 224 can store one or more sets of conditions for each type of image, each type of item depicted by the image, each publisher, and / or other entity.

[0083] The AI ​​system 160 can interact with the memory structure 218 and the models 170, 202, 204 to evaluate images and to generate updated images for images that do not satisfy one or more conditions for the image. In some examples, the AI ​​system 160 can receive an image (e.g., a first image 211) from a client device 106 (e.g., a user device of a user or a device of a digital component provider). The image evaluator 206 can retrieve a set of conditions 214 for the first image 211 from a condition database 224. For example, the image evaluator 206 can retrieve a set of conditions 214 for the image based on the type of item (e.g., type of product) that is the subject of the digital component that includes the first image 211. The image evaluator 206 can evaluate the first image 211 based on the conditions 214 and output data indicating whether the first image 211 satisfies the conditions 214. In other examples, a human can review the first image 211 and provide the AI ​​system 160 with data indicating whether the first image 211 satisfies the conditions 214.

[0084] If the first image 211 does not satisfy the condition 214, the AI ​​system 160 may generate an image editing prompt 215 that instructs the image editing model 204 to edit the first image 211 to create a second image 216 that satisfies the condition 214. The AI ​​system 160 may use the multimodal model 202 and / or the language model 170 to generate the image editing prompt 215.

[0085] In some implementations, AI system 160 sends first image 211 and condition 214 to multimodal model 202 along with a request or question that requests multimodal model 202 to output explanation data 212 that indicates why first image 211 (e.g., textually) violates condition 214 and / or the location of the portion of first image 211 that violates condition 214, using, for example, a bounding box or other type of visual indicator overlaid on a portion of first image 211, a textual description of the location, or other location indicator such as coordinates (e.g., pixel coordinates) of the portion of first image 211.

[0086] In this example, AI system 160 can generate prompt 172 based on explanation data 212. Prompt 172 can include instructions to instruct language model 170 to generate image editing prompt 215 that instructs image editing model 204 to generate second image 216 that satisfies condition 214. Prompt 172 can include first image 211, explanation data 212, and / or condition 214.

[0087] In some implementations, AI system 160 sends first image 211 to multimodal model 202 with a request that multimodal model 202 generate a caption for first image 211. The caption may describe the content of the image. In this example, AI system 160 may generate prompt 172 based on the caption and condition 214. For example, prompt 172 may instruct language model 170, on behalf of multimodal model 202, to output explanation data 212 based on the caption and condition 214. In this example, explanation data 212 may indicate which portion of first image 211 that references the caption violates condition 214 and / or why that portion of first image 211 violates condition 214. AI system 160 may then generate a second prompt 172 to language model 170 that instructs language model 170 to output image editing prompt 215 based on explanation 212, similar to the previous example.

[0088] AI system 160 may then provide image editing prompts 212 to image editing model 204. AI system 160 may also provide first image 211 and / or a version of first image 211 that includes location indicators that indicate portions of the image that violate conditions 214. Image editing model 204 may edit first image 211 based on image editing prompts 215 and output the edited version of first image 211 as second image 216.

[0089] In some implementations, the AI ​​system 160 can evaluate the second image 216 to ensure that the second image 216 satisfies the condition 214, for example, using the image evaluator 206, as described above. If the second image 216 does not satisfy the condition 214, the AI ​​system 160 can generate another edited version of the first image 211 using similar techniques. However, the AI ​​system 160 can modify the prompt(s) 172 to the language model 170 to ensure that the edited image satisfies the condition 214 or to increase the likelihood that the edited image will satisfy the condition 214. In some examples, if the edited image does not satisfy the condition 214, the system can provide the AI ​​system 160 with the original image caption, the original descriptive data, the image editing prompt, the edited image caption, and the edited image description, and the AI ​​system 160 can modify the prompt 172 based on the provided information to increase the likelihood that the edited image will satisfy the condition 214.

[0090] If the second image 216 meets the conditions, the AI ​​system 160 can send the second image 216 to the client device 106. In some implementations, the digital component device 210 can generate a digital component using the second image 216 and send the digital component to the client device 106. For example, the digital component device 210 can generate a digital component that depicts the second image 216 and includes electronic documents and / or links to data / files that enable the client device 106 to render the digital component. The AI ​​system 160 can provide the digital component to the service device 110, as described above, and the service device 110 can deliver the digital component to the client device 106 in response to the component request 112.

[0091] 3 is a flowchart of an example process 300 for generating personalized image advertisements. The operations of process 300 may be performed, for example, by AI system 160 of FIG. 1 or other data processing devices. The operations of process 300 may also be implemented as instructions stored on a computer-readable medium, which may be non-transitory. Execution of the instructions by one or more data processing devices causes the one or more data processing devices to perform the operations of process 300.

[0092] The system receives data indicating that a first image violates one or more conditions (302). For example, the system may evaluate the first image based on the one or more conditions or receive data from another system indicating that the first image violates one or more conditions. In some implementations, the system provides the image and the conditions along with a request to a language model or multimodal model to predict whether the image violates any of the conditions and, if so, output explanatory data explaining why the image violates the condition(s).

[0093] In response to receiving data indicating that the first image violates one or more conditions, the system generates an image editing prompt to edit the first image (304). As described above, the system can generate the image editing prompt using a chain of prompts for one or more machine learning models, such as a multimodal model and / or one or more language models. The image editing prompt can instruct the image editing models to edit the first image so that the first image satisfies the one or more conditions.

[0094] The system provides the image editing prompts and the first image as input to the image editing model (306). The image editing model can generate a second image by editing the first image based on the image editing prompts. The system receives the second image as an output of the image editing model.

[0095] The system provides the second image to one or more devices (310). For example, the system can provide the second image to the device that provided the first image to the system and / or other devices, such as as an image digital component provided in response to a component request. For example, the system may be part of the service appliance 110 that delivered the image digital component to the client device 106, as described above.

[0096] 4 is a block diagram of an exemplary computer system 400 that can be used to perform the operations described above. System 400 includes a processor 410, a memory 420, a storage device 430, and an input / output device 440. Components 410, 420, 430, and 440 can be interconnected using, for example, a system bus 450. Processor 410 is capable of processing instructions for execution within system 400. In one embodiment, processor 410 is a single-threaded processor. In another embodiment, processor 410 is a multi-threaded processor. Processor 410 can process instructions stored in memory 420 or storage device 430.

[0097] Memory 420 stores information within system 400. In one embodiment, memory 420 is a computer-readable medium. In one embodiment, memory 420 is a volatile memory unit. In other embodiments, memory 420 is a non-volatile memory unit.

[0098] Storage device 430 is capable of providing mass storage for system 400. In one implementation, storage device 430 is a computer-readable medium. In various different implementations, storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or some other mass storage device.

[0099] The input / output device(s) 440 provide input / output operations for the system 400. In one embodiment, the input / output device(s) 440 may include one or more of a network interface device, such as an Ethernet card, a serial communication device, such as an RS-232 port, and / or a wireless interface device, such as an 802.11 card. In other embodiments, the input / output device(s) may include driver devices configured to receive input data and send output data to other devices, such as keyboards, printers, displays, and other peripheral devices 460. However, other embodiments, such as mobile computing devices, mobile communication devices, set-top boxes, television client devices, etc., may also be used.

[0100] Although an example processing system is described in FIG. 4, implementations of the subject matter and functional operations described herein can be implemented in other kinds of digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and structural equivalents thereof, or one or more combinations thereof.

[0101] An electronic document (for simplicity we will simply call it a document) does not necessarily correspond to a file: a document may be stored as part of a file that holds other documents, in a single file dedicated to the document, or in multiple associated files.

[0102] In situations where the systems described herein collect and / or use personal information about a user, the user may be provided with an opportunity to enable / disable control programs or features that may collect and / or use personal information (e.g., information about the user's social network, social behavior or activities, the user's preferences, or the user's current location, etc.). Additionally, certain data may be processed in one or more ways such that personally identifiable information associated with the user is removed before it is stored or used. For example, the user's identity may be processed such that personally identifiable information about the user cannot be determined, or if location information is obtained (e.g., to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific location cannot be determined.

[0103] Embodiments of the subject matter and operations described herein can be implemented in digital electronic circuitry, or computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus. Alternatively, or additionally, the program instructions can be encoded in an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus. The computer storage medium can be, or can be included in, a computer-readable storage device, a computer-readable storage substrate, or a random or serial access memory array or device, or one or more combinations thereof. Furthermore, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. A computer storage medium may also be, or may be included in, one or more separate physical components or media (such as multiple CDs, disks, or other storage devices).

[0104] The operations described herein may be performed as operations performed by a data processing apparatus on data stored in one or more computer-readable storage devices or received from other sources.

[0105] The term "data processing apparatus" encompasses all types of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system-on-chip, or a plurality or combination of the above. An apparatus may include, for example, special-purpose logic circuitry such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, an apparatus may also include code that creates an execution environment for the computer program of interest, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0106] This document refers to a service device. As used herein, a service device is one or more data processing devices that perform operations to facilitate the delivery of content over a network. A service device is depicted as a single block in a block diagram. However, while a service device may be a single device or a single set of devices, this disclosure contemplates that a service device may also be a group of devices or multiple different systems that communicate to provide various content to client devices. For example, a service device may encompass one or more of a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a gaming service, or any other service.

[0107] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted, declarative or procedural, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple associated files (e.g., files storing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0108] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0109] Processors suitable for executing a computer program include, for example, both general-purpose and special-purpose processors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory, a random-access memory, or both. The essential elements of a computer are a processor that performs actions in accordance with the instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes, or is operatively coupled to receive and / or transfer data from, one or more mass storage devices for storing data, such as, for example, magnetic, magneto-optical, or optical disks. However, a computer need not have such devices. Furthermore, a computer can be incorporated into other devices, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as a universal serial bus (USB) flash drive). Suitable storage devices for storing computer program instructions and data include all types of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0110] To interact with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to interact with the user. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, a computer may interact with a user by sending and receiving documents to a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0111] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components, e.g., a data server, or includes middleware components, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

[0112] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server sends data (e.g., HTML pages) to client devices (e.g., for the purpose of displaying the data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., the result of a user interaction) may be received from the client device by the server.

[0113] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features unique to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, either separately or in any suitable subcombination. Furthermore, even if features may be described above as functioning in a particular combination and originally claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to subcombinations or variations of the subcombination.

[0114] Similarly, while operations are shown in the figures in a particular order, this should not be understood as requiring such operations to be performed in the particular order shown or in sequential order, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0115] Thus, specific embodiments of the present invention have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. Furthermore, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method comprising: receiving data indicating that the first image violates one or more conditions; generating an image editing prompt in response to receiving the data indicating that the first image violates the one or more conditions, the image editing prompt instructing an image editing model to edit the first image to satisfy the one or more conditions; generating descriptive data indicating the location of content within the image that violates the one or more conditions; generating the image editing prompt based on the description data and the one or more conditions using a language model; generating a providing the image editing prompt and the first image as inputs to an image editing model; receiving a second image as an output of the image editing model; providing the second image to one or more devices; A method comprising:

2. 10. The method of claim 1 , wherein generating the explanatory data comprises providing the image and the one or more conditions to a multimodal model trained to identify locations within the image that violate input conditions.

3. 2. The method of claim 1 , wherein generating the explanatory data comprises providing the image and the one or more conditions to a multimodal model trained to predict whether an image violates input conditions and to output data indicative of the location of content within the image that may violate at least one of the input conditions.

4. generating the description data, providing a first image to a first machine learning model trained to generate an image caption for the image; receiving an image caption for the image from the first machine learning model; providing the image caption and the one or more conditions to a second machine learning model trained to output image description data based on the input image caption and the input conditions; receiving the description data for the first image from the second machine learning model; 3. The method of claim 1 or 2, comprising:

5. 5. The method of claim 1, wherein the description data includes a location indicator, the location indicator indicating a location of content within the first image that has been determined to violate at least one of the one or more conditions.

6. 6. The method of claim 5, wherein the location indicator comprises a bounding box depicted in the first image around the content in the first image that is determined to violate at least one of the one or more conditions.

7. 6. The method of claim 5, wherein the location indicator comprises coordinates that define a bounding box around the content in the first image that is determined to violate at least one of the one or more conditions.

8. 8. The method of claim 5, wherein providing the image editing prompt and the first image as input to an image editing model comprises providing the position indicator to the image editing model.

9. The method of any one of claims 1 to 8, wherein the explanation data includes an explanation indicating why the first image violates the one or more conditions.

10. The method of any preceding claim, wherein the image editing prompt includes at least a portion of the descriptive data.

11. 11. The method of claim 1, wherein using the language model to generate the image editing prompt based on the description data and the one or more conditions comprises: using the description data to generate a prompt for the language model; and providing the prompt to the language model.

12. 12. The method of claim 11 , wherein the prompt includes instructions to direct the language model to generate the image editing prompt based on the descriptive data and data defining each condition violated by the first image, the descriptive data including a name of each condition violated by the first image.

13. generating the image editing prompt, obtaining a prompt template adapted to the image editing model; entering at least a portion of the description data into the prompt, the description data including the name of a condition that the first image has been determined to violate; Including, the image editing prompts output by the language model are adapted to the image editing model; 13. The method of claim 11 or 12.

14. 1. A system comprising: one or more processors; one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving data indicating that the first image violates one or more conditions; generating an image editing prompt in response to receiving the data indicating that the first image violates the one or more conditions, the image editing prompt instructing an image editing model to edit the first image to satisfy the one or more conditions; generating descriptive data indicating the location of content within the image that violates the one or more conditions; generating the image editing prompt based on the description data and the one or more conditions using a language model; generating a providing the image editing prompt and the first image as inputs to an image editing model; receiving a second image as an output of the image editing model; providing the second image to one or more devices; A system that causes an operation including

15. 15. The system of claim 14, wherein generating the explanation data comprises providing the image and the one or more conditions to a multimodal model trained to identify locations within the image that violate input conditions.

16. 15. The system of claim 14, wherein generating the explanation data comprises providing the image and the one or more conditions to a multimodal model trained to predict whether an image violates input conditions and to output data indicative of the location of content within the image that may violate at least one of the input conditions.

17. 1. A computer-readable storage medium having instructions that, when executed by one or more processors, cause the one or more processors to: receiving data indicating that the first image violates one or more conditions; generating an image editing prompt in response to receiving the data indicating that the first image violates the one or more conditions, the image editing prompt instructing an image editing model to edit the first image to satisfy the one or more conditions; generating descriptive data indicating the location of content within the image that violates the one or more conditions; generating the image editing prompt based on the description data and the one or more conditions using a language model; generating a providing the image editing prompt and the first image as inputs to an image editing model; receiving a second image as an output of the image editing model; providing the second image to one or more devices; 1. A computer-readable storage medium for causing a computer to perform operations including:

18. 20. The computer-readable storage medium of claim 17, wherein generating the explanation data comprises providing the image and the one or more conditions to a multimodal model trained to identify locations within the image that violate input conditions.

19. 1. A computer program product comprising instructions that, when executed by one or more computers, cause the one or more computers to: receiving data indicating that the first image violates one or more conditions; generating an image editing prompt in response to receiving the data indicating that the first image violates the one or more conditions, the image editing prompt instructing an image editing model to edit the first image to satisfy the one or more conditions; generating descriptive data indicating the location of content within the image that violates the one or more conditions; generating the image editing prompt based on the description data and the one or more conditions using a language model; generating a providing the image editing prompt and the first image as inputs to an image editing model; receiving a second image as an output of the image editing model; providing the second image to one or more devices; A computer program product for causing a computer to perform operations including:

20. 20. The computer program product of claim 19, wherein generating the explanation data comprises providing the image and the one or more conditions to a multimodal model trained to identify locations within the image that violate input conditions.

Images