Artificial intelligence for efficient image editing

Abstract

Methods, systems, and apparatuses for enabling artificial intelligence to generate updated images based on one or more conditions are described, including a computer program encoded on a computer storage medium. In one aspect, the method includes receiving data indicating that a first image violates one or more conditions. In response to receiving the data indicating that the first image violates the one or more conditions, an image editing prompt is generated that instructs an image editing model to edit the first image to satisfy the one or more conditions. The image editing prompt and the first image are provided as inputs to the image editing model. A second image is received as an output of the image editing model. The second image is provided to one or more devices.

Description

Background Art

[0001] This specification relates to data processing, artificial intelligence, and the generation of images using artificial intelligence.

[0002] Advances in machine learning have made it possible to implement artificial intelligence in more applications. For example, large language models are implemented to enable image editing. This enables more efficient image editing using information provided in association with the image.

Summary of the Invention

Means for Solving the Problems

[0003] Generally, one innovative aspect of the subject matter described in this specification is to receive data indicating that a first image violates one or more conditions, and in response to receiving the data indicating that the first image violates one or more conditions, generate an image editing prompt that instructs an image editing model to edit the first image to meet one or more conditions, including generating explanatory data indicating the position of the content within the image that violates one or more conditions, and using a language model to generate an image editing prompt based on the explanatory data and one or more conditions, and providing the image editing prompt and the first image as inputs to the image editing model, and receiving a second image as the output of the image editing model, and providing the second image to one or more devices. Other embodiments of this aspect include corresponding devices, systems, and computer programs configured to execute the aspects of the method, encoded on a computer storage device.

[0004] These embodiments and other embodiments may each optionally include one or more of the following features. In some embodiments, generating explanatory data includes providing an image and one or more conditions to a multimodal model trained to identify locations in the image that violate input conditions.

[0005] In some embodiments, generating explanatory data involves providing an image and one or more conditions to a multimodal model trained to predict whether an image violates input conditions and to output data indicating the location of content within the image that may violate at least one of the input conditions.

[0006] In some embodiments, generating descriptive data includes providing a first image to a first machine learning model trained to generate image captions for images; receiving image captions for images from the first machine learning model; providing image captions and one or more conditions to a second machine learning model trained to output descriptive data for images based on input image captions and input conditions; and receiving descriptive data for the first image from the second machine learning model.

[0007] In some embodiments, the descriptive data includes a location indicator. The location indicator indicates the location of content in a first image that is determined to violate at least one of one or more conditions. The location indicator may include a bounding box drawn in the first image around the content in the first image that is determined to violate at least one of one or more conditions. The location indicator may include coordinates defining the bounding box around the content in the first image that is determined to violate at least one of one or more conditions. Providing an image editing prompt and a first image as input to an image editing model may include providing the location indicator to the image editing model.

[0008] In some embodiments, the explanatory data includes an explanation indicating why the first image violates one or more conditions. In some embodiments, the image editing prompt includes at least a portion of the explanatory data.

[0009] In some embodiments, using a language model to generate image editing prompts based on descriptive data and one or more conditions includes generating prompts for the language model using the descriptive data and providing prompts to the language model. A prompt may include instructions that instruct the language model to generate image editing prompts based on the descriptive data and data defining each condition violated by the first image. The descriptive data may include the names of each condition violated by the first image.

[0010] In some embodiments, generating an image editing prompt involves obtaining a prompt template adapted to an image editing model and inputting at least a portion of descriptive data, including the name of the condition that the first image is determined to violate, into the prompt. The image editing prompt output by the language model is adapted to the image editing model.

[0011] Certain embodiments of the subject matter described herein can be implemented to achieve one or more of the following advantages. The techniques described herein enable the use of artificial intelligence (AI) to generate updated images based on data related to one or more conditions. The AI ​​system can evaluate images that do not meet the conditions and can use one or more machine learning models to determine why the image does not meet the conditions and / or which parts of the image prevent the image from meeting the conditions. Without the techniques described, users may struggle to determine why an image is rejected by a system that evaluates images based on conditions and to attempt to modify the image in various ways to meet the conditions. As a result, users may make unnecessary modifications, resulting in lower quality images. Also as a result, users may upload multiple versions of the image to the system in an attempt to find a version that meets the conditions, placing an unnecessary load on the system and the network connecting the system to the user's device, thereby wasting computing resources and network bandwidth. Furthermore, uploading multiple versions of an image can lead to data privacy / confidentiality issues, as consistent uploads can allow adversarial attacks against machine learning models to learn how the model works. Even worse, some systems may use many different criteria for different types of images or images of different types of items. Therefore, an image may meet the criteria(s) for one type of item (e.g., one type of product) but not for another type of item(s).

[0012] The technology described herein can solve these problems by using AI to determine why an image fails to meet a set of conditions and then editing the image to meet those conditions. In this way, the user only needs to upload an image once to obtain a suitable image, thereby saving resources that would otherwise be wasted by uploading and evaluating multiple images. For example, the user can submit a first image to the system, and the system can determine whether the first image violates the conditions. If it does, the system can use a machine learning model (e.g., an image editing model) to update the image so that the updated image does not violate the conditions, thus enabling the user to efficiently submit images to the system without having to upload additional images that they hope will comply with the conditions. In this way, a suitable image is generated without the need to upload multiple images, which reduces the computational load on the system that evaluates the images (e.g., processing cycles for evaluating the images, data storage for storing the images, etc.), the amount of network bandwidth consumed, and the associated load on network resources for submitting multiple images, and the computational load on the user's device when modifying the image and submitting it to the system that evaluates it.

[0013] A chain of prompts to one or more machine learning models can be used to evaluate an image to determine why it fails to meet a set of conditions, and, based on the evaluation, generate an updated image. In this way, tasks are isolated, and models trained for a particular task can be used to generate higher-quality outputs that take into account the outputs of previous models. This allows the system to generate higher-quality, specialized image editing prompts that produce a higher-quality explanation of why an image fails to meet the conditions and precisely instruct the image editing model to correct the image. By separating the process into multiple separate AI tasks, the amount of information provided to the AI ​​model for each task can be reduced, thereby preventing hallucinations and other AI model errors that typically occur when a substantial amount of information is provided as input to the AI ​​model.

[0014] Multiple machine learning models can be used to evaluate and edit images based on conditions. For example, a multimodal model with question-answering and image generation capabilities can evaluate an image and a set of conditions, and output explanatory data indicating why parts of the image do not satisfy the conditions and / or which parts of the image do not satisfy the conditions. A language model can then generate image editing prompts based on the explanatory data, and an image editing model, such as a text-to-image model, can generate an updated version of the image based on the prompts and the image. Using multiple models in this way allows for more accurate output and therefore higher quality images than using a single model to perform all of these different tasks. Furthermore, by using multiple models instead of a single model to handle all tasks in the process, the system can scale each model to deploy, generating higher quality images while efficiently managing resources. In particular, image editing models may be relatively larger and more complex than other models (e.g., models for generating image editing prompts for image editing models), and as a result, they can improve image quality, increase throughput, reduce memory usage, and reduce latency.

[0015] Using a multimodal model to generate descriptive data and a language model to generate prompts for an image editing model allows the use of off-the-shelf image editing models, general-purpose image editing models, or other pre-trained image editing models without customizing or retraining the image editing model. Alternatively, the techniques described herein may include the use of special prompts that instruct the language model to generate image editing prompts specifically tailored to the image editing model. This thus eliminates the need to adapt or retrain the image editing model to accept inputs generated by other models while ensuring that the image editing model produces high-quality edited images. Using a combination of models in this way allows the overall AI system to first generate edited images that conform to a large set of conditions, thereby reducing the number of images uploaded for evaluation and the number of evaluations performed on the images, providing various computational savings described above and elsewhere in this specification. Thus, the techniques described herein provide a concrete application of AI models and prompts to AI models to solve problems that arise when applying conditions to images and generating images that more generally satisfy the conditions in an automated image generation field.

[0016] Details of one or more embodiments of the subject matter described herein are illustrated by the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from this specification, the drawings, and the claims. [Brief explanation of the drawing]

[0017] [Figure 1] This is a block diagram of an exemplary environment where images are evaluated based on criteria and edited using artificial intelligence. [Figure 2] This block diagram illustrates the interaction between an artificial intelligence system, a multimodal model, a language model, and an image editing model. [Figure 3]This is a flowchart illustrating an exemplary process for generating images based on one or more conditions. [Figure 4] This is an illustrative block diagram of a computer. [Modes for carrying out the invention]

[0018] Similar reference symbols and designations in various drawings indicate the same elements.

[0019] This specification describes a technique for enabling artificial intelligence to generate updated images based on one or more conditions. For example, the technique can be used to evaluate an image that does not meet one or more conditions and to edit the image so that the edited image meets the conditions (may be more than one). Artificial intelligence (AI) is a branch of computer science that focuses on creating intelligent agents that can learn and act autonomously (e.g., without human intervention). Artificial intelligence can utilize machine learning, which focuses on developing algorithms that can learn from data; natural language processing, which focuses on understanding and generating human language; and / or computer vision, which focuses on understanding and interpreting images and videos.

[0020] The techniques described herein enable an AI model to edit an image that fails to meet one or more conditions so that the edited image meets those conditions. These conditions may be policy conditions of an entity that makes an image available for others to view, such as an entity that delivers digital components of an image to a user. Generally speaking, an AI system may receive data indicating that an image violates a condition, and the AI ​​system may use an image editing model to generate an updated version of the image that does not violate the condition, for example, by editing the image. The AI ​​system may use one or more machine learning models, such as a language model and / or a multimodal model (which may be a language model or another type of multimodal model), to evaluate the image, determine why the image fails to meet the condition, and generate an image editing prompt that instructs the image editing model (which may be a language model) to edit the image in a specific way so that the edited image meets the condition. The AI ​​system can provide the image editing model with the image editing prompt and the image, which can then edit the image and output an updated version of the image.

[0021] As described herein, by using AI to evaluate images based on conditions and generate image editing prompts, it becomes possible to instruct an image editing model to make appropriate edits to an image that produce a condition-satisfying image without making unnecessary edits, thereby enabling the creation of specialized image editing prompts that produce a high-quality image that is as close as possible to the original image while still satisfying the conditions. Furthermore, the specialized prompts are provided to the language model to ensure that the image editing prompts provided to the image editing model are adapted to the image editing model, so that the image editing model does not need to be adapted or retrained for use with the language model. Editing images in this way using AI reduces the waste of computing resources that would otherwise be used to generate, upload, and evaluate multiple versions of the image until a condition-satisfying version is reached. This also increases the likelihood of creating a suitable image compared to a user who edits an image without fully understanding the conditions that the image does not satisfy. All of this contributes to a system that can generate updated images that satisfy the conditions more quickly, so that images can be created and delivered in a real-time interactive environment, for example, in response to a user search query or a component request that asks a digital component to be displayed on a user's device.

[0022] As used throughout this book, the term "digital component" refers to an individual unit of digital content or digital information (e.g., a video clip, audio clip, multimedia clip, game content, image, text, bullet points, artificial intelligence output, language model output, or other unit of content). A digital component can be stored electronically in a physical memory device as a single file or as a collection of files, and since a digital component can take the form of a video file, audio file, multimedia file, image file, or text file, and can contain advertising information, advertising is a type of digital component.

[0023] FIG. 1 is a block diagram of an exemplary environment 100 in which images are evaluated based on conditions and edited using artificial intelligence. The exemplary environment 100 includes a network 102 such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 102 connects an electronic document server 104, a user device 106, a digital component server 108, and a service device 110. The exemplary environment 100 may include many different electronic document servers 104, user devices 106, and digital component servers 108.

[0024] The service device 110 is configured to provide various services to the client device 106 and / or the issuer of the electronic document 150. In some embodiments, the service device 110 can provide a search service by providing a response to a search query received from the client device 106. For example, the service device 110 can include a search engine and / or an AI agent, or other chat agents that enable a user to interact with the agent over a process of multiple conversational queries and responses. The service device 110 can also distribute digital components for presentation to the client device 106, along with the response and / or along with the electronic document 150. For example, other search service computer systems can send component requests 112 to the service device 110, and these component requests 112 can include one or more queries. The service device 110 and the component requests 112 are described in more detail below.

[0025] The client device 106 is an electronic device capable of requesting and receiving online resources via the network 102. Exemplary client devices 106 include personal computers, gaming devices, mobile communication devices, digital assistant devices, augmented reality devices, virtual reality devices, and other devices capable of transmitting and receiving data via the network 102. The client device 106 typically includes a user application such as a web browser to facilitate the transmission and reception of data via the network 102, but native applications (other than browsers) executed by the client device 106 may also facilitate the transmission and reception of data via the network 102.

[0026] A gaming device is a device that enables a user to participate in a gaming application, whereby, for example, the user can control one or more characters, avatars, or other rendered content presented in the gaming application. Gaming devices typically include a computer processor, a memory device, and a controller interface (physically or visually rendered) that enables user control over the content rendered by the gaming application. The gaming device can execute the gaming application stored locally or execute a gaming application (such as an online gaming application) stored and / or provided at least in part by a cloud server. Similarly, the gaming device can interface with a game server that executes the gaming application and "streams" the gaming application to the gaming device. The gaming device may be a tablet device, a mobile communication device, a computer, or other device that performs functions other than executing a gaming application.

[0027] Digital assistant devices include devices that include a microphone and a speaker. Digital assistant devices are generally capable of receiving input via voice, responding with content using audible feedback, and presenting other audible information. In some situations, digital assistant devices also include or communicate with a visual display (e.g., via a wireless or wired connection). Where a visual display is present, it can also provide feedback or other information visually. In some situations, digital assistant devices can also control other devices such as lighting, locks, cameras, room temperature control devices, alarm systems, and other devices registered with the digital assistant device.

[0028] As shown in the figure, client device 106 presents an electronic document 150. An electronic document is data that presents a set of content on client device 106. Examples of electronic documents include web pages, word processing documents, portable document format (PDF) documents, images, videos, search results pages, and feed sources. Native applications (e.g., “apps” and / or game applications) such as applications installed on mobile, tablet, or desktop computing devices are also examples of electronic documents. Electronic documents may be provided to client device 106 by an electronic document server 104 (“Electronic Doc Servers”).

[0029] For example, the electronic document server 104 may include a server that hosts the publisher's website. In this example, a client device 106 can initiate a request for a given publisher's webpage, and the electronic document server 104 hosting the given publisher's webpage can respond to the request by sending a machine-executable command to initiate the presentation of the given webpage on the client device 106.

[0030] In another example, the electronic document server 104 may include an app server from which a client device 106 can download an app. In this example, the client device 106 can download the files necessary to install the app on the client device 106 and then run the downloaded app locally (i.e., on the client device). Alternatively or additionally, the client device 106 can initiate a request to run the app, which is sent to the cloud server. In response to receiving the request, the cloud server can run the application and stream the application's user interface to the client device 106, so that the client device 106 does not need to run the app itself. Rather, the client device 106 can present the user interface generated by the cloud server running the app and return any user interactions with the user interface to the cloud server for processing.

[0031] Electronic documents can contain a variety of content. For example, electronic document 150 may contain native content 152 that is internal to electronic document 150 itself and / or does not change over time. Electronic documents can also contain dynamic content that may change over time or on a per-request basis. For example, the publisher of a given electronic document (e.g., electronic document 150) may maintain a data source used to input parts of the electronic document. In this example, the given electronic document may contain a script, such as script 154, which causes the client device 106 (or cloud server) to request content (e.g., digital components) from the data source when the given electronic document is processed (e.g., rendered or executed) by the client device 106 (or cloud server). The client device 106 (or cloud server) integrates the content (e.g., digital components) obtained from the data source into the given electronic document to create a composite electronic document that includes the content obtained from the data source.

[0032] In some situations, a given electronic document (e.g., electronic document 150) may include a digital component script (e.g., script 154) that references a service device 110 or a specific service provided by the service device 110. In these situations, the digital component script is executed by the client device 106 when the given electronic document is processed by the client device 106. When the digital component script is executed, the client device 106 is configured to generate a request for a digital component 112 (referred to as a “component request”), which is sent to the service device 110 via the network 102. For example, the digital component script may enable the client device 106 to generate a packetized data request that includes header and payload data. The component request 112 may include event data specifying characteristics such as the name (or network location) of the server on which the digital component is being requested, the name (or network location) of the requesting device (e.g., client device 106), and / or information that the service device 110 can use to select one or more digital components, or other content, to be provided in response to the request. The component request 112 is sent by the client device 106 to the server of the service device 110 via the network 102 (for example, a telecommunications network).

[0033] The component request 112 may include event data specifying the electronic document being requested and other event characteristics, such as the location of the electronic document where the digital component can be presented. For example, the service device 110 may be provided with event data specifying a reference (e.g., URL) to the electronic document (e.g., web page) where the digital component is presented, the available locations of the electronic document available for presenting the digital component, the size of the available locations, and / or the media types eligible for presentation at the locations. Similarly, event data specifying keywords associated with the electronic document ("document keywords") or entities (e.g., people, places, or things) referenced by the electronic document may also be included in the component request 112 (e.g., as payload data) and provided to the service device 110 to facilitate the identification of digital components eligible for presentation with the electronic document. The event data may also include search queries submitted from the client device 106 to retrieve the search results page.

[0034] The component request 112 may also include event data related to other information, such as information provided by the user of the client device, geographical information indicating the state or region where the component request was submitted, or other information that provides context about the environment in which the digital component will be displayed (e.g., the time of the component request, the day of the week of the component request, the type of device on which the digital component will be displayed, such as a mobile device or a tablet device). The component request 112 can be transmitted, for example, over a packetized network, and the component request 112 itself can be formatted as packetized data having a header and payload data. The header may specify the destination of the packet, and the payload data may include any of the information described above.

[0035] The service device 110 selects digital components (e.g., third-party content such as video files, audio files, images, text, game content, augmented reality content, and combinations thereof, all of which can take the form of advertising or non-advertising content), and the digital components are presented together with a given electronic document (e.g., at a location specified by script 154) in response to receiving a component request 112 and / or using the information contained in the component request 112. In some embodiments, selecting digital components includes selecting digital components based on textual features.

[0036] In some embodiments, digital components are selected in less than one second to avoid errors that might be caused by delays in the selection of digital components. For example, a delay in providing a digital component in response to a component request 112 could result in a page load error on the client device 106, or leave a portion of the electronic document unentered even after other portions of the electronic document have been presented on the client device 106. The techniques described are adapted to generate digital components in short timeframes so that these errors and the impact on user experience are reduced or eliminated.

[0037] Furthermore, as the delay in providing digital components to the client device 106 increases, there is a higher probability that the electronic document will not be presented to the client device 106 when the digital component is delivered, thereby negatively impacting the user experience with the electronic document. In addition, if the delay in providing digital components causes the electronic document to not be presented to the client device 106 when the digital component is provided, for example, the delivery of the digital component may fail.

[0038] In some embodiments, the service device 110 is implemented in a distributed computing system, which includes, for example, a server and a set of multiple computing devices 114 that are interconnected and identify and deliver digital components in response to requests 112. The set of multiple computing devices 114 has millions of available digital components (DC). 1-x They work together to identify a set of digital components from the corpus that are eligible to be presented in an electronic document. Millions of available digital components can be indexed, for example, in a digital component database 116. Each digital component index entry can reference the corresponding digital component and / or contribute to the delivery / transmission of the corresponding digital component (e.g., triggers, conditions, or restricts) delivery parameters (DP1~DP1). x ) may include. For example, delivery parameters may contribute to (e.g., trigger) the transmission of a digital component by requiring that a component request include at least one criterion that matches (e.g., exactly or at some pre-specified level of similarity) one of the delivery parameters of the digital component.

[0039] In some embodiments, the delivery parameters for a particular digital component may include delivery keywords that the digital component must match (e.g., an electronic document, document keywords, or terms specified in component request 112) in order to qualify for presentation. Additionally or alternatively, the delivery parameters may include embeddings that can use data of various different dimensions, such as website details and / or consumption details (e.g., page viewport, user scroll speed, or other information about data consumption). The delivery parameters may also require that component request 112 include information specifying a particular geographical region (e.g., a country or state), and / or information specifying that component request 112 was originated from a particular type of client device (e.g., a mobile device or a tablet device) in order for the digital component to qualify for presentation. The delivery parameters may also specify a qualification value (e.g., a ranking score, or some other specified value) used to assess the qualification for delivery / transmission of the digital component (e.g., among other available digital components).

[0040] The identification of eligible digital components can be segmented into multiple tasks 117a-117c, which are then assigned among computing devices within a set of computing devices 114. For example, different computing devices within set 114 can each analyze different parts of the digital component database 116 to identify various digital components that have delivery parameters matching the information contained in the component request 112. In some embodiments, each given computing device within set 114 can analyze different data dimensions (or sets of dimensions) and pass (e.g., transmit) the results of the analysis (Res1-Res3) 118a-118c to the service device 110. For example, the results 118a-118c provided by each computing device within set 114 may identify a subset of digital components that are eligible for delivery in response to a component request, and / or a subset of digital components that have specific delivery parameters. Identifying a subset of digital components may include, for example, comparing event data with delivery parameters and identifying a subset of digital components that have delivery parameters matching at least some features of the event data.

[0041] The service device 110 aggregates the results 118a to 118c received from a set of multiple computing devices 114 and uses the aggregated results and associated information to select one or more digital components to be provided in response to the request 112. For example, the service device 110 may select a set of acquired digital components (one or more digital components) based on the results of one or more content evaluation processes, as described below. The service device 110 may then generate and transmit reply data 120 (e.g., digital data representing the response) via the network 102, enabling the client device 106 to integrate the acquired set of digital components into a given electronic document, and as a result, the acquired set of digital components (e.g., acquired third-party content) and the content of the electronic document are presented together on the display of the client device 106.

[0042] In some embodiments, the client device 106 executes instructions contained in the reply data 120, thereby configuring the client device 106 to retrieve a set of digital components obtained from one or more digital component servers 108, enabling the client device 106 to retrieve a set of digital components obtained from one or more digital component servers 108. For example, the instructions in the reply data 120 may include a network location (e.g., a Uniform Resource Locator (URL)) and a script that causes the client device 106 to send a Server Request (SR) 121 to the digital component server 108 to retrieve a given set of digital components obtained from the digital component server 108. In response to the request, the digital component server 108 identifies the given set of digital components specified in the Server Request 121 (e.g., in a database storing multiple digital components) and sends digital component data (DC data) 122 to the client device 106, presenting the given set of digital components in an electronic document for the client device 106.

[0043] When client device 106 receives digital component data 122, the client device renders the digital component (e.g., third-party content) and presents it at a location specified by or assigned to script 154. For example, script 154 can create a garden environment enclosed by walls, such as a frame, which is presented within the native content 152 of electronic document 150, for example, next to the native content 152 of electronic document 150. In some embodiments, the digital component is overlaid on (or adjacent to) a portion of the native content 152 of electronic document 150, and service device 110 can specify the presentation location within electronic document 150 in the reply 120. For example, when the native content 152 includes video content, service device 110 can specify the location or object in the scene depicted in the video content on which the digital component is presented.

[0044] The service device 110 may also include an artificial intelligence system 160 configured to autonomously generate digital components either before the request 112 (e.g., offline) and / or in response to the request 112 (e.g., online or in real time). As will be described in more detail throughout this specification, the artificial intelligence ("AI") system 160 may collect online content about a particular entity (e.g., a digital component provider or other entity) and summarize the collected online content using one or more language models 170, which may include a large language model.

[0045] Large-scale language models ("LLMs") are models trained to generate and understand human language. LLMs are trained on large datasets of text and code and can be used for a variety of tasks. For example, LLMs can be trained to translate text from one language to another, summarize text such as website content, search results, news articles, or research papers, answer questions about text such as "What is the capital of Georgia?", create chatbots that can converse with humans, and generate creative text such as poetry, stories, and code.

[0046] Language model 170 may be any suitable language model neural network that receives an input sequence consisting of text tokens selected from a vocabulary and autoregressively generates an output sequence consisting of text tokens from the vocabulary. For example, language model 170 may be a transformer-based language model neural network or a recurrent neural network-based language model.

[0047] In some situations, when the neural network used to implement language model 170 autoregressively generates a sequence of token outputs, language model 170 can be called an autoregressive neural network. More specifically, the autoregressively generated output is created by generating each particular token in the output sequence, conditional on the current input sequence which includes all tokens preceding a particular text token in the output sequence—that is, tokens already generated for all previous positions in the output sequence preceding a particular position of a particular token—and a context input that provides context to the output sequence.

[0048] For example, the current input sequence when generating a token at any given position in the output sequence may include the input sequence and a token at any preceding position that precedes the given position in the output sequence. Specifically, the current input sequence may include an input sequence followed by a token at any preceding position that precedes the given position in the output sequence. Optionally, the input sequence and the current output sequence can be separated by one or more predetermined tokens in the current input sequence.

[0049] More specifically, in order to generate a specific token at a specific position in the output sequence, the neural network of language model 170 can process the current input sequence to generate a score distribution, such as a probability distribution, which assigns a score to each token in the token vocabulary, such as its respective probability. The neural network of language model 170 can then use the score distribution to select a token from the vocabulary as a specific token. For example, the neural network of language model 170 can greedily select the token with the highest score, or it can sample tokens from the distribution using, for example, nuclear sampling or other sampling techniques.

[0050] As a specific example, language model 170 may be an autoregressive transformer-based neural network that includes (i) multiple attention blocks, each applying a self-attentional action, and (ii) an output subnetwork that processes the output of the last attention block to generate a score distribution.

[0051] The price tag 170 is a slightly less expensive one of the snowflakes These include J. Hoffmann, S. Borgeau d, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, DdLCasas, LAHendricks, J. Welbl, A. Clark, et al., Training compute-optimal large language models, arXiv database, arXiv:2203.15556, 2022, JWRae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, HFSong, J.A slanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. van then Driessche, LAHendricks, M. Rauh, P. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mellor, I. Hig gins, A. Creswell, N. McAleese, A. Wu, E. Elsen, SM Jayakumar, E. Buchatskay, D. Budden, E. Sutherland, K. Simonyan. M. Paganini, L. Sifre, L. Martens, XLLi, A. Kuncoro, A. Nematzadeh, E. Gribovskaya, D. Donato, A. Lazaridou, A. Mes ch, J. Lespiau, M. Tsimpoukelli, N. Grigoev, D. Fritz, T. Sottiaux, M. Pajarskas, T. Pohlen, Z. Gong, D. Toyama, C. de Masson d'Autume, Y. Li, T. Terzi, V. Mikulik, I. Babuschkin, A. Clark, D. de Las Casas, A. Guy, C. Jones, J. Bradbury, M. Johnson, BAHechtman, L. Weidinger, I. Gabriel, WSIsaac, E. Lockhart, S. Osindero, L.Rimell, C. Dyer, O. Vinyals, K. Ayoub, J. Stanway, L. Bennett, D. Hassabis, K. Kavukcuoglu, and G. Irving: Scaling language models; gopher, CoRR, abs / 2122.11446, 2021, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer arXiv Publication: arXiv:1910.10683, Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V.Le, Towards a Human-Like Open-Domain Chatbot, CoRR, abs / 2001.09977, 2020, Tom B Brown, and Benjamin Mann、Nick Ryder、Melanie Subbiah、Jared Kaplan、Prafulla Dhariwal、Arvind Neelakantan、Pranav Shyam、Girish Sastry、Amanda Askell, et al.、Language models are few-shot learners.The arXiv database is arXiv:2005.14165.

[0052] However, generally, transformer-based neural networks include a sequence of attention blocks, and during processing of a given input sequence, each attention block in the sequence receives a hidden state for each input token in the given input sequence. The attention block then updates each of the hidden states at least partially by applying self-attention to generate an output hidden state for each input token. The input hidden state of the first attention block is the embedding of the input tokens in the input sequence, and the input hidden state of each subsequent attention block is the output hidden state generated by the preceding attention block.

[0053] In this example, the output subnetwork processes the output hidden state generated by the last attention block in the sequence of the last input token in the input sequence to generate the score distribution.

[0054] Generally, since language models are autoregressive, the service device 110 can use the same language model 170 to generate multiple different candidate output sequences in response to the same request, for example, by using beam search decoding from the score distribution generated by the language model 170, by using a sample-and-rank decoding strategy, by using different random number seeds for the pseudo-random number generator used for sampling for different runs via the language model 170, or by using other decoding strategies that take advantage of the autoregressive nature of the language model.

[0055] In some embodiments, the language model 170 is pre-trained, i.e., trained on a language modeling task that does not require it to respond to user questions and provide evidence, and the service device 110 (for example, using the AI ​​system 160) causes the language model 170 to generate an output sequence according to a predetermined syntax via natural language prompts in the input sequence.

[0056] For example, a service device 110 (e.g., an AI system 160), or a separate training system, pre-trains a language modeling model 170 (e.g., a neural network) on a language modeling task, such as a task that requires predicting the next token following the current sequence in the training data, given the current sequence of text tokens. As a specific example, the language model 170 can be pre-trained for the maximum likelihood objective on a large dataset of text, such as text publicly available from the internet or other text corpora.

[0057] The AI ​​system 160 can use the language model 170 to generate image editing prompts for editing images that do not meet one or more conditions. The service device 110 can maintain conditions for digital components and / or other images sent to the client device 106. For example, conditions can be used to ensure that digital components do not contain explicit content or language. Conditions may vary based on the type of item corresponding to the image and / or the type of image. For example, there may be a first set of conditions for a first type of product and a second set of conditions for a second type of product. Furthermore, the service device 110 can maintain conditions for each publisher of the electronic document 150. For example, a publisher of web pages for children may have enhanced conditions for the digital components presented on those web pages.

[0058] In some embodiments, the language model 170 can be adapted to generate image editing prompts using zero-shot learning or fusion-shot learning. In some examples, the system can provide the language model 170 with manually created examples for fusion-shot learning. For fusion-shot learning, the AI ​​system 160 can provide the language model 170 with a small number of training examples (e.g., 3-7, 10, or other numbers). These training examples include an original image, a set of conditions, and an image editing prompt containing instructions for editing the image to fit the conditions.

[0059] The service device 110 can evaluate the digital image components and / or other images to determine whether these images satisfy a set of conditions for each image. If an image does not satisfy the conditions, the service device 110 can use the AI ​​system 160 to evaluate why the image does not satisfy the conditions and / or to edit the image so that it conforms to the conditions.

[0060] The AI ​​system 160 can use the language model 170 to generate image editing prompts that instruct the image editing model to generate updated images that satisfy the conditions. For example, the AI ​​system 160 can generate a prompt 172 that instructs the language model 170 to generate an image editing prompt based on descriptive data related to an explanation of why the image does not satisfy the conditions. In some embodiments, the prompt 172 includes descriptive data. The descriptive data may include data indicating that the image violates the conditions, a description of the violation (e.g., the name of the policy being violated), the portion of the image that causes the image to violate the conditions, the conditions themselves, and / or other data. The data indicating the portion of the image that causes the image to violate the conditions may include position indicators depicted within the image.

[0061] For example, the condition definition might state, "[Policy Definition] then the image violates the [Policy Name] policy." In this example, the descriptive data could state, "This particular image violates the [Policy Name] policy because of [Explanation of why the image violates the policy]." Prompt 172 might then read, "[Policy Definition] then the image violates the [Policy Name] policy. This particular image violates the [Policy Name] policy because of [Explanation of why the image violates the policy]. My task is to edit the image so that it does not violate the [Policy Name] policy, and I will use the diffusion model by providing the diffusion model with the original image and the image editing prompt. Generate an image editing prompt to make the image conform to the policy. Make sure the image editing prompt does not remove or edit anything important that the image is trying to convey." The text in parentheses can be populated with appropriate data by the AI ​​system 160. For example, the AI ​​system 160 could populate [Policy Name] with the policy name of the policy that the original image data violates.

[0062] Here, prompt 172 is specifically adapted to instruct language model 170 to generate an image editing prompt for the diffusion model by notifying language model 170 of the subsequent task, "I will use the diffusion model by providing the diffusion model with the original image and the image editing prompt." In this way, language model 170 can generate an image editing prompt specifically adapted for the diffusion model, which accepts the image editing prompt and the original image as input. This eliminates the need to adapt or retrain the image editing model.

[0063] In some implementations, the AI ​​system 160 can maintain prompt templates for each of one or more image editing models. Each prompt template may take the form of an exemplary prompt 172 having parenthetical fields into which the AI ​​system 160 can input using policy information and descriptive information. For example, when evaluating and editing an image that violates conditions such as policy conditions, the AI ​​system 160 can receive descriptive data from a multimodal model having question-answering and image editing functions and use the descriptive data to input into the template.

[0064] If a multimodal language model is used as language model 170, prompt 172 may include an image. If language model 170 only accepts text input, prompt 172 may include a caption that describes the content of the image to language model 170.

[0065] In some examples, the descriptive data may include an image with a bounding box around the portion of the image that is considered to be the reason the image violates a condition. In this example, the bounding box is a position indicator, and the language model 170 is a multimodal model that accepts text and image prompts 172 as input.

[0066] In some embodiments, the AI ​​system 160 uses a language model 170 to generate descriptive data. For example, the AI ​​system 160 can use the language model 170 or other models to generate image captions. Captions can describe the content of an image. For example, the AI ​​system 160 can generate a prompt 172 instructing the language model 170 or other models (e.g., other multimodal models described with reference to Figure 2) to generate a caption for an input image. The AI ​​system 160 can generate other prompts 172 instructing the language model 170 to receive a caption, evaluate the caption and a set of conditions, and output an explanation of why the image described by the caption violates the conditions, or which part of the image described by the caption violates the conditions. In some examples, the prompt 172 includes a list of objects identified in the image based on bounding boxes. In this example, the prompt 172 may include a set of captions and states.

[0067] For example, prompt 172 reads: "If [Policy Definition], the image violates the [Policy Name] policy. This image contains [Image Description]. The following objects and their bounding boxes are in a more spatially recognized context for the image, i.e., [Object 1:<x1、y1、x2、y2> Object 2:<x1、y1、x2、y2> I will provide you with the following: Based on the context I have provided you with about the image, predict whether the image violates the [Policy Name] policy. Provide a very detailed description of your decision, including the areas that violate the policy. In this example, x1 and x2 are coordinate axes in the image along one dimension (e.g., the x-axis or horizontal), and y1 and y2 are coordinates in the image along the other dimension (e.g., the y-axis or vertical). These coordinates inform the language model 170 where objects 1 and 2 can be found in the image.

[0068] In some embodiments, the AI ​​system 160 uses a multimodal model to generate explanatory data based on a set of images and conditions. In this example, the AI ​​system 160 provides the multimodal model with a set of images and conditions and requests the multimodal model to output explanatory data, such as an explanation of why an image does not meet the conditions and / or an explanation of an image that has a bounding box or other position indicator showing the part of the image that violates the conditions.

[0069] In each example, the language model 170 can evaluate the prompt 172 using the explanatory data and generate an output 174, including an image editing prompt, based on the input data. The language model 170 can generate an image editing prompt (e.g., one with structure) in a way that instructs the image editing model to generate an updated image based on the input image and the prompt.

[0070] The AI ​​system 160 can use image editing prompts to generate an updated image using an image editing model, as will be explained in more detail with reference to Figure 2. The updated image is an edited version of the initial image that does not violate any conditions. For example, the updated image may not include or depict areas in the image that violate the conditions (e.g., the location of content). In a particular example, the image editing model may replace the content in that part of the image with content that satisfies the conditions.

[0071] For example, an image editing prompt to hide a specific area of ​​a person in an image or remove an item might read, "Cover the person's [body part] and remove the [item] from the image." Another example of an image editing prompt to edit a person in an image might read, "Neutralize the facial expression of the person(s) in the image." Another example of an image editing prompt to remove an item from an image might read, "Crop the image to remove the item." Another example of an image editing prompt to remove an item from an image might read, "Remove the adult beverage bottle from the image."

[0072] The AI ​​system 160 can then send (e.g., provide) the updated image to one or more devices (e.g., one or more client devices 106) as a reply 120. For example, the AI ​​system 160 can generate a digital component to provide in response to a request 112 from a user. The digital component may include the updated image. The digital component may include links to electronic documents, metadata, and / or other data and / or files related to the subject of the digital component (e.g., the item depicted by the image) that enable the client device 106 to render the updated image.

[0073] A single language model 170 is shown in Figure 1, but different language models can be specifically trained to handle different prompts at different stages of the processing pipeline. For example, one language model can be trained to generate descriptive data for an image, while another language model can be trained to generate image editing prompts based on that descriptive data.

[0074] Figure 2 is a block diagram 200 showing the interaction between the AI ​​system 160, the multimodal model 202, the language model 170, and the image editing model 204. The AI ​​system 160 may include an image evaluation device 206, a prompting device 208, and a digital component device 210.

[0075] As described above, the language model 170 can be trained to perform a variety of tasks. The AI ​​system 160 can use the language model 170 to generate descriptive data and / or generate image editing prompts for the image editing model 204. Although Figure 2 shows one language model 170, the AI ​​system 160 can interact with any number of language models 170 to generate image editing prompts to instruct the image editing model 204 to generate updated images that satisfy one or more conditions, such as one or more policy conditions.

[0076] The multimodal model 202 can be implemented as a machine learning model trained to generate explanatory data 212. For example, the training process can use a set of training images and corresponding ground truth explanatory data. For example, the ground truth training data may include, for each image that violates a condition, a label indicating the violated condition and why the image violates the condition. The label may also indicate the part of the image that violates the condition. Based on this set of training images, the multimodal model 202 can be trained to generate explanatory data 212.

[0077] The multimodal model 202 can be trained to generate text based on text and image inputs. For example, the multimodal model 202 can be trained to output an explanation explaining why an image violates one or more conditions, based on an input that includes an image and text indicating one or more conditions as explanatory data. In some embodiments, the multimodal model 202 can be trained to output an image with a position indicator (e.g., a bounding box) indicating the portion of the image that violates the condition as explanatory data. The bounding box can be outlined using the x and y coordinates of the image, as in the example provided above. The multimodal model 202 can accept images and text as inputs and can generate text as an output. During training, images and questions (e.g., text asking about the explanatory data) are used as inputs to the model and the multimodal model 202 is trained to generate answers (e.g., text answering the questions about the explanatory data). In a supervised learning example, a training sample may include an image and a question containing a condition, along with a label that has an answer containing an explanation explaining why the image does not satisfy the condition. In some embodiments, the multimodal model 202 may be a neural network or other type of machine learning model trained to respond to questions, provide answers, and edit images.

[0078] The image editing model 204 may be a machine learning model, such as a text-to-image neural network, trained to generate an image based on an input image and an image editing prompt 215 that instructs the image editing model 204 on how to edit the image. In some embodiments, the image editing model 204 is a language model trained to edit images. In some embodiments, the image editing model 204 is a diffusion model.

[0079] During training, the image editing model 204 can take the original image caption and image editing prompt as input, and can be trained to generate a target text prompt by applying the image editing prompt to the image caption. Given the image and target text prompt, the image editing model 204 can encode the target text prompt to generate an initial text embedding. The image editing model 204 then processes (e.g., optimizes) the initial text embedding to reconstruct the input image. The system then fine-tunes the image editing model 204 (e.g., a diffusion model of the image editing model 204) to improve overall accuracy by interpolating the target text prompt with the input image to generate the output of the image editing model (e.g., the edited image).

[0080] The AI ​​system 160 may also include a memory structure 216 or be configured to interact with a memory structure 218 for extracting and / or storing information and content. The memory structure 218 may include one or more databases or other data structures stored in one or more memories and / or data storage devices. In particular, the memory structure 218 may store a digital component database 116, digital components 220, images 222, and conditional data 224.

[0081] As described above, the digital component database 116 may include delivery parameters for the digital components 220. These delivery parameters may include, for example, keywords and / or geographical locations that qualify the digital component 220 for delivery to the client device 106. The digital component database 116 may also include, for each digital component 220, metadata for the digital component, captions for each image 222 corresponding to the digital component, data related to the digital component provider that provides the digital component, and / or other data related to the digital component. The digital component 220 may include candidate digital components that can be provided in response to component requests 112 and / or queries received by the service device 110. The image 222 may include one or more images for each digital component 220. The AI ​​system 160 can obtain images of the digital components 220 from the digital component provider or from other sources.

[0082] The condition database 224 can store conditions for images. The condition database 224 can store one or more sets of conditions for each image type, each type of item depicted by the image, each publisher, and / or other entities.

[0083] The AI ​​system 160 can interact with memory structure 218 and models 170, 202, and 204 to evaluate images and generate updated images for images that do not meet one or more conditions. In some examples, the AI ​​system 160 can receive an image (e.g., a first image 211) from a client device 106 (e.g., a user's device or a digital component provider's device). The image evaluation device 206 can retrieve a set of conditions for the first image 211 from a condition database 224. For example, the image evaluation device 206 can retrieve a set of image conditions 214 based on the type of item (e.g., product type) that is the subject of the digital component containing the first image 211. The image evaluation device 206 can evaluate the first image 211 based on the conditions 214 and output data indicating whether the first image 211 meets the conditions 214. In other examples, a human can review the first image 211 and provide the AI ​​system 160 with data indicating whether the first image 211 meets the conditions 214.

[0084] If the first image 211 does not satisfy condition 214, the AI ​​system 160 can generate an image editing prompt 215 instructing the image editing model 204 to edit the first image 211 to create a second image 216 that satisfies condition 214. The AI ​​system 160 can generate the image editing prompt 215 using the multimodal model 202 and / or the language model 170.

[0085] In some embodiments, the AI ​​system 160 sends a first image 211 and a condition 214 to a multimodal model 202 along with a request or question, the request or question asks the multimodal model 202 to output descriptive data 212 indicating why the first image 211 (e.g., in text) violates condition 214 and / or the location of the part of the first image 211 that violates condition 214, using, for example, a bounding box or other type of visual indicator overlaid on a portion of the first image 211, a text description of the location, or other location indicators such as the coordinates (e.g., pixel coordinates) of the portion of the first image 211.

[0086] In this example, the AI ​​system 160 can generate a prompt 172 based on the descriptive data 212. The prompt 172 may include an instruction to the language model 170 to generate an image editing prompt 215 that instructs the image editing model 204 to generate a second image 216 that satisfies condition 214. The prompt 172 may include the first image 211, the descriptive data 212, and / or condition 214.

[0087] In some embodiments, the AI ​​system 160 sends the first image 211 to the multimodal model 202 along with a request for the multimodal model 202 to generate a caption for the first image 211. The caption can describe the content of the image. In this example, the AI ​​system 160 can generate a prompt 172 based on the caption and condition 214. For example, the prompt 172 can instruct the language model 170 to output descriptive data 212 based on the caption and condition 214 on behalf of the multimodal model 202. In this example, the descriptive data 212 can indicate which part of the first image 211, referring to the caption, violates condition 214, and / or why that part of the first image 211 violates condition 214. The AI ​​system 160 can then generate a second prompt 172 to the language model 170 instructing the language model 170 to output an image editing prompt 215 based on the descriptive data 212, as in the previous example.

[0088] The AI ​​system 160 can then provide the image editing model 204 with an image editing prompt 212. The AI ​​system 160 can also provide the first image 211 and / or a version of the first image 211, including a position indicator showing the portion of the image that violates condition 214. The image editing model 204 can edit the first image 211 based on the image editing prompt 215 and output the edited version of the first image 211 as the second image 216.

[0089] In some implementations, the AI ​​system 160 can evaluate the second image 216 to ensure that the second image 216 satisfies condition 214, for example, using the image evaluation device 206 as described above. If the second image 216 does not satisfy condition 214, the AI ​​system 160 can generate other edited versions of the first image 211 using similar techniques. However, the AI ​​system 160 can modify prompt(s) 172 to the language model 170 to ensure that the edited image satisfies condition 214, or to increase the likelihood that the edited image satisfies condition 214. In some examples, if the edited image does not satisfy condition 214, the system can provide the AI ​​system 160 with the caption of the original image, the original descriptive data, the image editing prompt, the caption of the edited image, and the description of the edited image, and the AI ​​system 160 can modify prompt(s) 172 based on the provided information to increase the likelihood that the edited image satisfies condition 214.

[0090] If the second image 216 satisfies the conditions, the AI ​​system 160 can transmit the second image 216 to the client device 106. In some embodiments, the digital component device 210 can use the second image 216 to generate a digital component and transmit the digital component to the client device 106. For example, the digital component device 210 can depict the second image 216 and generate a digital component that includes a link to an electronic document and / or data / file that enables the client device 106 to render the digital component. As described above, the AI ​​system 160 can provide the digital component to the service device 110, and the service device 100 can deliver the digital component to the client device 106 in response to the component request 112.

[0091] Figure 3 is a flowchart of an exemplary process 300 for generating personalized image advertisements. The operation of process 300 can be performed, for example, by the AI ​​system 160 in Figure 1 or other data processing devices. The operation of process 300 can also be implemented as instructions stored on a computer-readable medium, which may be non-temporary. The execution of instructions by one or more data processing devices causes one or more data processing devices to perform the operation of process 300.

[0092] The system receives data indicating that the first image violates one or more conditions (302). For example, the system may evaluate the first image based on one or more conditions, or receive data from another system indicating that the first image violates one or more conditions. In some embodiments, the system provides a language model or a multimodal model with an image and conditions, along with a request to predict whether the image violates any of the conditions, and if so, output explanatory data explaining why the image violates the condition(s).

[0093] The system generates an image editing prompt to edit the first image in response to receiving data indicating that the first image violates one or more conditions (304). As described above, the system may generate image editing prompts using a chain of prompts for one or more machine learning models, such as a multimodal model and / or one or more language models. The image editing prompt may instruct the image editing model to edit the first image so that the first image satisfies one or more conditions.

[0094] The system provides an image editing prompt and a first image as input to the image editing model (306). The image editing model can generate a second image by editing the first image based on the image editing prompt. The system receives the second image as output to the image editing model.

[0095] The system provides a second image to one or more devices (310). For example, the system may provide the second image to the device that provided the first image to the system and / or other devices, such as as an image digital component provided in response to a component request. For example, the system may be part of a service device 110 that delivered the image digital component to the client device 106 as described above.

[0096] Figure 4 is a block diagram of an exemplary computer system 400 that can be used to perform the operations described above. The system 400 includes a processor 410, memory 420, a storage device 430, and an input / output device 440. Each component 410, 420, 430, and 440 can be interconnected using, for example, a system bus 450. The processor 410 is capable of processing instructions to be executed within the system 400. In one embodiment, the processor 410 is a single-threaded processor. In another embodiment, the processor 410 is a multi-threaded processor. The processor 410 can process instructions stored in memory 420 or the storage device 430.

[0097] Memory 420 stores information within the system 400. In one embodiment, memory 420 is a computer-readable medium. In one embodiment, memory 420 is a volatile memory unit. In other embodiments, memory 420 is a non-volatile memory unit.

[0098] The storage device 430 can provide mass storage to the system 400. In one embodiment, the storage device 430 is a computer-readable medium. In various different embodiments, the storage device 430 may include, for example, a hard disk device, an optical disk device, a storage device shared over a network by multiple computing devices (e.g., a cloud storage device), or several other mass storage devices.

[0099] The input / output device 440 provides input / output operation for the system 400. In one embodiment, the input / output device 440 may include one or more of the following: a network interface device such as an Ethernet card, a serial communication device such as an RS-232 port, and / or a wireless interface device such as an 802.11 card. In other embodiments, the input / output device may include a driver device configured to receive input data and transmit output data to other devices such as a keyboard, printer, display, and other peripheral devices 460. However, other embodiments such as mobile computing devices, mobile communication devices, and set-top box television client devices may also be used.

[0100] While an example of a processing system is illustrated in Figure 4, embodiments of the subject matter and functional operation described herein may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware, including the structures and their structural equivalents disclosed herein, or any combination thereof.

[0101] Electronic documents (simply called documents for brevity) do not necessarily correspond to files. A document can be part of a file that holds other documents, a single file dedicated to a document, or multiple associated files.

[0102] Where the systems described herein collect and / or use personal information about a user, the user may be given the opportunity to enable / disable control programs or functions that may collect and / or use personal information (e.g., information about the user's social networks, social behavior or activities, the user's preferences, or the user's current location). Furthermore, certain data may be processed in one or more ways before being stored or used so that personally identifiable information associated with the user is removed. For example, a user's identity may be processed in such a way that personally identifiable information cannot be determined, or if location information is obtained (e.g., at the city, zip code, or state level), the user's geographical location may be generalized so that the user's specific location cannot be determined.

[0103] The subject matter and embodiments of operation described herein can be implemented in digital electronic circuits, or in computer software, firmware, or hardware, including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a computer storage medium to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, program instructions may be encoded in artificially generated propagating signals, such as machine-generated electrical signals, optical signals, or electromagnetic signals, which are generated to encode information to be sent to a suitable receiving device for execution by the data processing device. The computer storage medium may be or may be a computer-readable storage device, a computer-readable memory board, or a random or serial access memory array or device, or one or more combinations thereof. Furthermore, although the computer storage medium is not a propagating signal, the computer storage medium may be the source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media may also be, or be comprised of, one or more separate physical components or media (such as multiple CDs, disks, or other storage devices).

[0104] The operations described herein may be performed as operations performed by a data processing device on data stored in one or more computer-readable storage devices or received from other sources.

[0105] The term “data processing device” encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, systems on a chip, or a combination of several of the above. A device may include, for example, special-purpose logic circuits such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, a device also includes code that creates the execution environment for a computer program of interest, such as processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or code that constitutes one or more of these. Devices and execution environments can realize a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[0106] This document refers to a service device. As used herein, a service device is one or more data processing devices that perform operations to facilitate the delivery of content over a network. In block diagrams, a service device is depicted as a single block. However, while a service device may be a single device or a single set of devices, this disclosure also intends that a service device may also be a group of devices or multiple different systems that communicate to deliver various content to client devices. For example, a service device may encompass one or more of the following services: a search system, a video streaming service, an audio streaming service, an email service, a navigation service, an advertising service, a game service, or any other services.

[0107] Computer programs (also known as programs, software, software applications, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, objects, or other units suitable for use in a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. A program can be stored in part of a file containing other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program of interest, or in multiple collaborative files (e.g., a file containing one or more modules, subprograms, or parts of code). Computer programs can be deployed to run on one computer, or on multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0108] The process and logic flows described herein can be executed by one or more programmable processors executing one or more computer programs, performing actions on input data, and generating outputs. Process and logic flows can also be executed by special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and devices can also be implemented as these special-purpose logic circuits.

[0109] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose processors, and one or more processors of any type of digital computer. Generally, a processor receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a processor that performs actions according to instructions, and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to them for receiving data from them, transferring data to them, or both. However, a computer is not required to have such devices. Furthermore, computers can also be incorporated into other devices such as mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or portable storage devices (such as Universal Serial Bus (USB) flash drives). Suitable storage devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices including EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory may be complemented by or incorporated into special-purpose logic circuits.

[0110] To interact with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can input to the computer. Other types of devices may also be used to interact with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. Furthermore, the computer may interact with the user by sending and receiving documents to and from devices used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0111] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component, such as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface or web browser on which a user can interact with the embodiments of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), internetworks (such as the Internet), and peer-to-peer networks (such as ad-hoc peer-to-peer networks).

[0112] A computing system can include a client and a server. The client and server are generally geographically distant from each other and typically interact through a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to the client device (for example, to display data to a user interacting with the client device and to receive user input from that user). Data generated on the client device (e.g., the results of user interaction) can be received by the server from the client device.

[0113] While this specification includes details of many specific embodiments, these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, individually or in any preferred secondary combination. Furthermore, features may be described above as functioning in a particular combination, and even if initially claimed as such, one or more features from the claimed combination may be removed from the combination, and the claimed combination may cover secondary combinations or variations of secondary combinations.

[0114] Similarly, while the operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific illustrated or sequential order, or that all illustrated operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and the described program components and systems should be understood as generally being able to be integrated into a single software product or packaged into multiple software products.

[0115] Specific embodiments of the present invention have been described in this manner. Other embodiments are within the scope of the following claims. In some cases, the desired results may still be obtained by performing the actions described in the claims in a different order. Furthermore, the processes shown in the accompanying figures do not necessarily require to be performed in a specific or sequential order shown in order to obtain the desired results. In certain embodiments, multitasking and parallel processing may be advantageous.

Claims

1. It is a method, Receiving data indicating that the first image violates one or more conditions, In response to receiving the data indicating that the first image violates one or more of the conditions, generating an image editing prompt instructing an image editing model to edit the first image to satisfy one or more of the conditions, To generate descriptive data indicating the location of content in the first image that violates one or more of the above conditions, Using a language model, generate the image editing prompt based on the descriptive data and one or more conditions, Including generating, The image editing prompt and the first image are provided as input to the image editing model. The image editing model receives a second image as its output, To provide the second image to a device, Methods that include...

2. The method according to claim 1, wherein generating the descriptive data includes providing the first image and the one or more conditions to a multimodal model trained to identify locations in an image that violate input conditions.

3. The method according to claim 1, wherein generating the descriptive data includes providing the first image and the one or more conditions to a multimodal model trained to predict whether an image violates an input condition and to output data indicating the location of content in the image that may violate at least one of the input conditions.

4. To generate the aforementioned explanatory data, The first image is provided to a first machine learning model trained to generate image captions for the image, The first machine learning model receives the image caption of the image, To provide the image caption and one or more conditions to a second machine learning model trained to output descriptive data for an image based on the input image caption and the input conditions, Receiving the explanatory data of the first image from the second machine learning model The method according to claim 2, including the method described in claim 2.

5. The method according to claim 1, wherein the descriptive data includes a location indicator, the location indicator indicating the location of content in the first image that is determined to violate at least one of the one or more conditions.

6. The method according to claim 5, wherein the position indicator includes a bounding box drawn in the first image around the content in the first image that is determined to violate at least one of the one or more conditions.

7. The method according to claim 5, wherein the position indicator includes coordinates for defining a bounding box around the content in the first image which is determined to violate at least one of the one or more conditions.

8. The method according to claim 5, wherein providing the image editing prompt and the first image as input to the image editing model includes providing the position indicator to the image editing model.

9. The method according to claim 1, wherein the explanatory data includes an explanation of why the first image violates one or more of the conditions.

10. The method according to claim 1, wherein the image editing prompt includes at least a portion of the descriptive data.

11. The method according to claim 1, wherein generating the image editing prompt using the language model based on the descriptive data and one or more conditions includes generating a prompt for the language model using the descriptive data and providing the prompt to the language model.

12. The method according to claim 11, wherein the prompt includes an instruction that instructs the language model to generate the image editing prompt based on the descriptive data and data defining each condition violated by the first image, the descriptive data including the name of each condition violated by the first image.

13. To generate the aforementioned image editing prompt, Obtaining a prompt template that is compatible with the aforementioned image editing model, Enter at least a portion of the descriptive data, including the name of the condition in which the first image was determined to be in violation, into the prompt. Includes, The image editing prompt output by the language model is adapted to the image editing model. The method according to claim 11.

14. It is a system, One or more processors, The system comprises one or more storage devices for storing instructions, and when an instruction is executed by the one or more processors, the one or more processors Receiving data indicating that the first image violates one or more conditions, In response to receiving the data indicating that the first image violates one or more of the conditions, generating an image editing prompt instructing an image editing model to edit the first image to satisfy one or more of the conditions, To generate descriptive data indicating the location of content in the first image that violates one or more of the above conditions, Using a language model, generate the image editing prompt based on the descriptive data and one or more conditions, Including generating, The image editing prompt and the first image are provided as input to the image editing model. The image editing model receives a second image as its output, To provide the second image to a device, A system that performs actions including those mentioned above.

15. The system according to claim 14, wherein generating the descriptive data includes providing the first image and one or more conditions to a multimodal model trained to identify locations in an image that violate input conditions.

16. The system according to claim 14, wherein generating the descriptive data includes providing the first image and the one or more conditions to a multimodal model trained to predict whether an image violates an input condition and to output data indicating the location of content in the image that may violate at least one of the input conditions.

17. A computer-readable storage medium having instructions, wherein when an instruction is executed by one or more processors, the one or more processors... Receiving data indicating that the first image violates one or more conditions, In response to receiving the data indicating that the first image violates one or more of the conditions, generating an image editing prompt instructing an image editing model to edit the first image to satisfy one or more of the conditions, To generate descriptive data indicating the location of content in the first image that violates one or more of the above conditions, Using a language model, generate the image editing prompt based on the descriptive data and one or more conditions, Including generating, The image editing prompt and the first image are provided as input to the image editing model. The image editing model receives a second image as its output, To provide the second image to a device, A computer-readable storage medium that enables the execution of operations including [specific actions].

18. The computer-readable storage medium according to claim 17, wherein generating the descriptive data includes providing the first image and one or more conditions to a multimodal model trained to identify locations in an image that violate input conditions.

19. A computer program including instructions, wherein when the instructions are executed by one or more computers, the one or more computers shall Receiving data indicating that the first image violates one or more conditions, In response to receiving the data indicating that the first image violates one or more of the conditions, generating an image editing prompt instructing an image editing model to edit the first image to satisfy one or more of the conditions, To generate descriptive data indicating the location of content in the first image that violates one or more of the above conditions, Using a language model, generate the image editing prompt based on the descriptive data and one or more conditions, Including generating, The image editing prompt and the first image are provided as input to the image editing model. The image editing model receives a second image as its output, To provide the second image to a device, A computer program that causes an action to be performed, including the actions mentioned above.

20. The computer program according to claim 19, wherein generating the descriptive data includes providing the first image and one or more conditions to a multimodal model trained to identify locations in an image that violate input conditions.

Images