Generative image filling using reference image

By using reference images and text prompts, combined with self-attention operations, editing the masked part of the input image, solving the problem of object insertion and text description feature fusion in the prior art, and achieving efficient image editing and generation effects.

CN119068079APending Publication Date: 2024-12-03ADOBE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410681611.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-11-21
Filing Date
2024-05-29
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to effectively edit the masked portion of a digital image, especially when maintaining the uniqueness and recognizability of the object while inserting it into the input image and generating an image based on features described in natural language texts, there are challenges. .

Method used

By using reference images and text prompts, combined with image generation models, especially self-attention operations, we edit the masked part of the input image, generate synthetic images, and achieve object insertion and style coordination, avoiding the additional encoder Dependence improves image generation and repair effects.

Benefits of technology

It is realized that the image generation model is more easily directed to edit the masked portion of the input image, maintaining the uniqueness and recognizability of the object, while improving the accuracy and efficiency of image generation, and improving the existing image generation and repair systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068079A_ABST
    Figure CN119068079A_ABST
Patent Text Reader

Abstract

Embodiments of the present disclosure relate to generative image filling using a reference image. Embodiments include systems and methods for generative image filling based on text and reference images. In one aspect, a system obtains an input image, a reference image, and a textual cue. The system then encodes the reference image to obtain an image embedding, and encodes the text cue to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 18 / 515,378, filed on November 12, 2023, which claims priority under 35 U.S.C. §119 to U.S. Provisional Patent Application No. 63 / 505,902, filed in the U.S. Patent and Trademark Office on June 2, 2023, the disclosures of which are incorporated herein by reference in their entirety. Background Art

[0003] The following content generally relates to machine learning, and more specifically, to machine learning for image editing. Digital image processing generally refers to the process of changing a digital image using a computer or other electronic device. A computer or other electronic device can change a digital image using an algorithm, a processing network, etc. In some cases, image processing software can be used for various image processing tasks, such as image editing, image generation, etc. Some image processing systems can implement machine learning techniques to, for example, perform tasks using predictive models (e.g., without explicitly programming the system for each task), perform tasks more accurately or in less time, perform tasks using dedicated hardware, and so on. Examples of machine learning architectures for generating image data include generative adversarial networks (GANs) and diffusion models. Summary of the invention

[0004] The present disclosure describes systems and methods for image editing. Embodiments of the present disclosure include an image editing device configured to fill a masked portion of an input image with a reference image. The image editing device may obtain an input image, a reference image, and a text prompt, and the image editing device may generate a composite image based on the input image, the reference image, and the text prompt. In particular, the image editing device may edit the masked portion of the input image based on the reference image and the text prompt. In some examples, the masked portion of the input image may be edited to include an object from a reference image or to depict a style or texture from a reference image.

[0005] Methods, apparatus, non-transitory computer-readable media, and systems for machine learning for image editing are described. One or more aspects of the method, apparatus, non-transitory computer-readable media, and systems include: obtaining an input image, a reference image, and a text prompt; encoding the reference image using an image encoder to obtain an image embedding; encoding the text prompt using a text encoder to obtain a text embedding; and generating a composite image based on the input image, the image embedding, and the text embedding using an image generation model.

[0006] Methods, apparatus, non-transitory computer-readable media, and systems for machine learning for image editing are described. One or more aspects of the methods, apparatus, non-transitory computer-readable media, and systems include: obtaining an input image and a reference image; inserting the reference image into the input image to obtain a combined input image; encoding the reference image using an image encoder to obtain an image embedding; and generating a composite image based on the combined input image and the image embedding using an image generation model.

[0007] An apparatus, system, and method for machine learning for image editing are described. One or more aspects of the apparatus, system, and method include: at least one memory; at least one processing device coupled to the at least one memory, wherein the processing device is configured to execute instructions stored in the at least one memory; an image encoder configured to encode a reference image to obtain an image embedding; a text encoder configured to encode a text prompt to obtain a text embedding; and an image generation model configured to generate a synthetic image based on an input image, the image embedding, and the text embedding. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An example of an image editing system according to aspects of the present disclosure is shown.

[0009] Figure 2 An example of an image editing apparatus according to aspects of the present disclosure is shown.

[0010] FIG. 3A to FIG. 3N An example of generative image filling using reference images according to aspects of the present disclosure is shown.

[0011] Figure 4 An example of an architecture for generative image filling using reference images in accordance with aspects of the present disclosure is shown.

[0012] Figure 5 An example of a guided latent diffusion model according to aspects of the present disclosure is shown.

[0013] Figure 6 An example of a method for generative image filling using a reference image according to aspects of the present disclosure is shown.

[0014] Figures 7 and 8 An example of a method for generating a composite image according to aspects of the present disclosure is shown.

[0015] Fig. 9 An example of a method for training a diffusion model according to aspects of the present disclosure is shown.

[0016] Fig.10 An example of a computing device for image editing in accordance with aspects of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The present disclosure describes systems and methods for image editing. Embodiments of the present disclosure include an image editing apparatus configured to fill in masked portions of an input image using a reference image.

[0018] Some image generation models (e.g., diffusion-based models) may support conditional image generation to allow a user to specify one or more conditions for generating an image. The conditions may correspond to image features in a latent space, and the latent space may guide the image generation model to generate an image having the image features. "Features" generally refer to representative characteristics of data and may be represented by vectors or tensors of, for example, numbers. In some examples, the conditions may be specified by the user using natural language text. The natural language text may be tokenized and mapped to one or more vectors in the latent space of the image generation model. However, in some examples, it may be challenging to find appropriate text to describe the image features to include in the generated image. In addition, if the text specifies that an object is to be inserted into an input image, it may be challenging to make the object conform to the style and texture of the input image while maintaining the uniqueness and recognizability of the object.

[0019] Embodiments of the present disclosure include an image editing device configured to edit an input image using a reference image. The image editing device may obtain an input image, a mask, a reference image, and a text prompt, and the image editing device may generate a composite image based on the input image, the mask, the reference image, and the text prompt. In particular, the image editing device may edit the masked portion of the input image based on the reference image and the text prompt. In some cases, the mask may be received from a user, or may be determined based on the text prompt, the input image, or the reference image. In some examples, the image editing device may insert a reference image into the input image to obtain a combined input image, and the image editing device may perform a self-attention operation on the combined input image to include the reference image in the masked portion of the input image while retaining the identity of the reference image.

[0020] Because the image editing device can take a reference image as input, the user can more easily specify desired image features to guide the image generation model to edit the input image. In addition, because the image editing device can use a self-attention operation to insert objects into the input image, the object can be inserted into the input image without having to train an additional image encoder to encode the object. The self-attention operation can blend the object into the masked portion of the input image while maintaining the uniqueness and recognizability of the object. Accordingly, an embodiment improves existing image generation and inpainting systems by enabling the insertion of image content into an image so that the image content includes features from both a text description and a reference image. Reference Figure 1-Figure 5 Details on the architecture of an example image editing device are provided. Figure 6-Figure 8 Provides sample methods for image editing. Fig. 9 An example training process is described.

[0021] Network Architecture

[0022] Figure 1-Figure 5 An example image editing system and examples of inputs and outputs of the system are shown. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: at least one memory; at least one processing device coupled to the at least one memory, wherein the processing device is configured to execute instructions stored in the at least one memory; an image encoder configured to encode a reference image to obtain an image embedding; a text encoder configured to encode a text prompt to obtain a text embedding; and an image generation model configured to generate a composite image based on an input image, a mask, an image embedding, and a text embedding.

[0023] In some aspects, the image generation model includes a diffusion model. In some aspects, the image generation model includes a self-attention layer configured to operate on a combination of an input image and a reference image. In some aspects, the image encoder and the text encoder are components of a multimodal encoder. In some aspects, the image generation model uses image embedding and text embedding for classifier-free guidance. Some examples of the device, system, and method also include a user interface configured to obtain a selection input from a user, wherein the mask is based on the selection input. Some examples of the device, system, and method also include a training component configured to train the image generation model.

[0024] Figure 1 An example of an image editing system 100 according to aspects of the present disclosure is shown. In one aspect, the image editing system 100 includes a user 105, a computing device 110, an image editing apparatus 115, a database 120, and a cloud 125. The image editing apparatus 115 is a reference Figure 2Examples of corresponding elements described, or including references Figure 2 Aspects of the corresponding elements described.

[0025] The user 105 may interact with the image editing software on the user device 110. The user device 110 may communicate with the image editing apparatus 115 via the cloud 125. In some examples, the user 105 may provide the input image 130, the mask 135, the reference image 140, and the text prompt 145 to the image editing apparatus 115 via the user device 110. In some examples, the user device 110 includes a user interface configured to obtain a selection input (e.g., a selection input corresponding to the mask 135) from the user 105. The image editing apparatus 115 may then generate a composite image 150 depicting the input image 130, the input image 130 having a masked portion corresponding to the mask 135 filled using the reference image 140 and based on the text prompt 145. In some examples, the image editing apparatus 115 may upload the composite image 150 to the database 120, or the image editing apparatus 115 may provide the composite image 150 to the user 105 (e.g., via the user device 110). Thus, the image editing apparatus 115 may be used to fill in masked portions of the input image using the reference image.

[0026] In some examples, the image editing device 115 may include a server. The server provides one or more functions to users (e.g., user 105) linked by one or more various networks. In some cases, the server includes a single microprocessor board, which includes a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses a microprocessor and a protocol to exchange data with other devices / users on one or more networks via a hypertext transfer protocol (HTTP) and a simple mail transfer protocol (SMTP), although other protocols such as a file transfer protocol (FTP) and a simple network management protocol (SNMP) may also be used. In some cases, the server is configured to send and receive files in a hypertext markup language (HTML) format (e.g., for displaying web pages). In various embodiments, the server includes a general-purpose computing device (e.g., user device 110), a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0027] Database 120 is an organized collection of data. For example, database 120 stores data in a specific format known as a schema. Database 120 can be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller can manage data storage and processing in the database. In some cases, user 105 interacts with the database controller. In other cases, the database controller can operate automatically without user interaction.

[0028] Cloud 125 is a computer network configured to provide on-demand availability of computer system resources (such as data storage and computing power). In some examples, cloud 125 provides resources without active management by user 105. The term "cloud" is sometimes used to describe a data center that is available to many users via the Internet. The functions of some large cloud networks are distributed in multiple locations from a central server. If a server has a direct or close connection with a user, the server is designated as an edge server. In some cases, cloud 125 is limited to a single organization. In other examples, cloud 125 is available to many organizations. In one example, cloud 125 includes a multi-layer communication network including multiple edge routers and core routers. In another example, cloud 125 is based on a local collection of switches in a single physical location.

[0029] User device 110 (eg, computing device) is a personal computer, laptop computer, mainframe computer, handheld computer, personal assistant, mobile device, or any other suitable processing device.

[0030] Figure 2 An example of an image editing device 200 according to aspects of the present disclosure is shown. The image editing device 200 is a reference Figure 1 Examples of corresponding elements described, or including references Figure 1 In one aspect, the image editing device 200 includes a processor unit 205, an I / O module 210, a training component 215, and a memory unit 220. In one aspect, the memory unit 220 includes an image encoder 225, a text encoder 230, and an image generation model 235. That is, the image encoder 225, the text encoder 230, and the image generation model 235 can be embedded in the memory unit 220, and the parameters of the image encoder 225, the text encoder 230, and the image generation model 235 can be stored in the memory unit 220, or the image encoder 225, the text encoder 230, and the image generation model 235 can communicate with the memory unit 220.

[0031] The processor unit 205 includes a processor. The processor unit 205 is an intelligent hardware device (e.g., a general processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor unit 205 is configured to operate the memory array using a memory controller. In other cases, the memory controller is integrated into the processor unit 205. In some cases, the processor unit 205 is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor unit 205 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0032] The memory unit 220 includes a memory containing instructions that can be executed by the processor. Examples of the memory unit 220 include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of the memory unit 220 include solid-state memory and a hard disk drive. In some examples, the memory is used to store computer-readable computer executable software including instructions, which, when executed, causes the processor to perform various functions described herein. In some cases, the memory unit 220 includes a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices). In some cases, a memory controller operates a memory cell. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the memory cells within the memory unit 220 store information in the form of logical states.

[0033] I / O module 210 (e.g., input / output interface) may include an I / O controller. An I / O controller may manage input signals and output signals for a device. An I / O controller may also manage peripherals that are not integrated into the device. In some cases, an I / O controller may represent a physical connection or port to an external peripheral. In some cases, an I / O controller may utilize a processor such as a processor. , or other known operating systems. In other cases, the I / O controller may represent a modem, keyboard, mouse, touch screen, or similar device, or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, the I / O controller may be implemented as part of the processor. In some cases, a user may interact with the device via the I / O controller or a hardware component controlled by the I / O controller.

[0034] In some examples, the I / O module 210 includes a user interface. The user interface can enable a user to interact with the device. In some embodiments, the user interface can include an audio device (such as an external speaker system), an external display device (such as a display screen) or an input device (for example, a remote control device that is directly or through an I / O controller module and docked with the user interface). In some cases, the user interface can be a graphical user interface (GUI). In some examples, the communication interface operates at the boundary between the communication entity and the channel, and can also record and process communication. Providing a communication interface in this article is to enable a processing system to be coupled to a transceiver (for example, a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for a communication device via an antenna.

[0035] In some examples, the image editing device 200 includes an artificial neural network (ANN) implemented by a computer. For example, the image encoder 225, the text encoder 230, and the image generation model 235 may all include an ANN. An artificial neural network is a hardware or software component that includes a plurality of connected nodes (i.e., artificial neurons) that roughly correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another node (like a physical synapse in the brain). When a node receives a signal, the node processes the signal and then transmits the processed signal to other connected nodes. In some cases, the signal between nodes includes real numbers, and the output of each node is calculated by a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how to process and transmit signals.

[0036] In some examples, the image editing device 200 includes a computer-implemented convolutional neural network (CNN). CNN is a type of neural network commonly used in computer vision or image classification systems. In some cases, CNN can process digital images with minimal preprocessing. CNN is characterized by the use of convolutional (or cross-correlation) hidden layers. These layers apply convolution operations to the input and then signal the results to the next layer. Each convolution node can process data for a limited input field (i.e., receptive field). For example, each node of the first layer of a CNN (after the input layer that usually corresponds to the input image) can evaluate a limited field corresponding to a block of pixels in the input image. During the forward pass of the CNN, the filter at each layer can be convolved on the total input amount to calculate the dot product between the filter and the input. During the training process, the filters can be modified so that they are activated when specific features within the input are detected.

[0037] In some examples, the image editing device 200 includes a transformer. A transformer or transformer network is a neural network model for natural language processing tasks. The transformer network uses an encoder and a decoder to transform one sequence into another sequence. The encoder and decoder include modules that can be stacked on top of each other multiple times. The module includes multi-head attention and a feedforward layer. The input and output (target sentence) are first embedded in an n-dimensional space. The position encoding of different words (i.e., a relative position is given to each word / part in the sequence because the sequence depends on the order of its elements) is added to the embedding representation (n-dimensional vector) of each word. In some examples, the transformer network includes an attention mechanism, in which the attention looks at the input sequence and decides which other parts of the sequence are important at each step. The attention mechanism involves queries, keys, and values ​​represented by Q, K, and V, respectively. Q corresponds to a matrix containing a query (a vector representation of a word in a sequence), K corresponds to all keys (a vector representation of all words in a sequence), and V corresponds to a value, which is also a vector representation of all words in a sequence. For encoders and decoders, multi-head attention modules, V consists of the same word sequence as Q. However, for attention modules that consider both encoder and decoder sequences, V is different from the sequence represented by Q. In some cases, the values ​​in V are multiplied by some attention weights a and summed.

[0038] In some examples, training component 215 is implemented as software stored in memory and executable by a processor of a separate computing device, firmware in a separate computing device, one or more hardware circuits of a separate computing device, or a combination thereof. In some examples, training component 215 is part of another device different from image editing device 200 and communicates with image editing device 200. In some examples, training component 215 can be configured to train image generation model 235.

[0039] In some examples, the image encoder 225 or the text encoder 225 includes a multimodal encoder, such as a contrastive language-image pre-training (CLIP) encoder. CLIP is a neural network-based model that is trained on a large-scale dataset of images and text (e.g., image captions). CLIP uses a technique called contrastive learning to learn the basic patterns and features of the data. This allows CLIP to understand the relationship between different objects and scenes in the image, and classify the objects and scenes based on the content in the objects and scenes. CLIP is multimodal because it can process and understand multiple types of data inputs, such as text and images. In some examples, CLIP can be fine-tuned for specific tasks (such as identifying specific objects in an image). CLIP's ability to generalize from one task to another and fine-tune for new tasks makes it a highly versatile model.

[0040] In some examples, the image generation model 235 includes a diffusion model. A diffusion model is a type of generative neural network that can be trained to generate new data (e.g., novel images) whose features are similar to those found in the training data. Diffusion models can be used for various image generation tasks, including image super-resolution, image generation using perceptual indicators, conditional generation (e.g., generation based on text guidance), image restoration, and image manipulation. Types of diffusion models include denoising diffusion probabilistic models (Denoising Diffusion Probabilistic Model, DDPM) and denoising diffusion implicit models (Denoising Diffusion Implicit Model, DDIM). In DDPM, the generative process includes reversing the random Markov diffusion process. On the other hand, DDIM uses a deterministic process so that the same input produces the same output. The diffusion model can also be characterized by whether the noise is added to the image itself or to the image features generated by the encoder (i.e., implicit diffusion).

[0041] In some examples, the image generation model 235 can be used for classifier-free image generation. Classifier-free image generation refers to the process of generating images without relying on explicit classifiers or labels. Traditional image generation methods often use predefined classifiers or labels to guide the generation process, such as conditional generative models that generate images based on specific category labels. However, in classifier-free image generation, the focus is on allowing the model to generate images freely without any specific constraints or guidance from the classifier. This approach is intended to encourage more creative and diverse image synthesis. Therefore, classifier-free image generation methods can allow more open exploration and creativity because the model is not limited to predefined labels or classifications.

[0042] According to some aspects, the image editing device 200 obtains an input image, a mask, a reference image, and a text prompt. According to some aspects, the image encoder 225 encodes the reference image to obtain an image embedding. According to some aspects, the text encoder 230 encodes the text prompt to obtain a text embedding. According to some aspects, the image generation model 235 generates a synthetic image based on the input image, the mask, the image embedding, and the text embedding. In some aspects, the image generation model 235 uses the image embedding and the text embedding for classifier-free guidance.

[0043] In some examples, the image editing apparatus 200 provides image embedding and text embedding as guidance for the image generation model 235. In some aspects, the synthesized image includes content corresponding to the reference image in an area corresponding to the mask. In some aspects, the synthesized image includes content corresponding to the input image in an area outside the mask.

[0044] In some aspects, the textual prompt describes an object in a reference image. In some aspects, the reference image comprises a portion of an input image. For example, in some cases, the reference image can be obtained by identifying a portion of an input image, rather than from a separate image. In some aspects, the reference image comprises a style from the input image. In some aspects, the composite image comprises an object described by the textual prompt, the object having a style from the reference image. In some aspects, the mask indicates a background area generated by an image generation model based on the reference image.

[0045] According to some aspects, the image editing apparatus 200 obtains an input image, a mask, and a reference image. In some examples, the image editing apparatus 200 inserts the reference image into the input image to obtain a combined input image. According to some aspects, the image encoder 225 encodes the reference image to obtain an image embedding. According to some aspects, the image generation model 235 generates a composite image based on the combined input image, the mask, and the image embedding.

[0046] In some examples, the image generation model 235 performs a self-attention operation on the combined input image, where the composite image is generated based on the self-attention operation. For example, the image generation model 235 includes a self-attention layer configured to operate on a combined input image (e.g., a combination of an input image and a reference image).

[0047] According to some aspects, the text encoder 230 obtains the text hint. In some examples, the text encoder 230 encodes the text hint to obtain a text embedding, wherein the synthetic image is generated based on the text embedding. In some aspects, the reference image is inserted into the input image in an area outside the mask.

[0048] Figure 3A-Figure 3N An example of generative image filling using reference images according to aspects of the present disclosure is shown. Figure 3A and Figure 3B An example of similar object insertion according to aspects of the present disclosure is shown. Given an image of an object, an image editing apparatus can fill holes in the input image with similar objects. Figure 3A In the example shown, the image generation model inpaints a hat onto an image of a man based on a reference image of a hat and a textual cue “hat.” In this example, the image generation model infers what the hat would look like if worn on the man’s head (i.e., a different view of the hat than the reference image).

[0049] Figure 3C An example of generating object variations according to aspects of the present disclosure is shown. The reference image can be the same cropped image within the hole of the input image, and the image editing device can generate more variations of the content within the hole. In this example, the user identifies the lighthouse and buildings on the back of the turtle by drawing a mask. According to some aspects, the system presents the variations to the user for editing or approval.

[0050] Figure 3D An example of object insertion with coordinated enhancement with a background according to aspects of the present disclosure is shown. The reference image can be any local crop within the input image, and the image editing device can generate an object that is better coordinated with the image background style. The reference image can be used for global context and coordinated enhancement. In this context, coordination refers to the plausibility of the restored content with its background, which may be affected by, for example, lighting details, color details, and semantic features.

[0051] Figure 3E-Figure 3I An example of object replacement using similar object texture or style according to aspects of the present disclosure is shown. Given another image with a specific style or texture, an image editing apparatus can generate an object with a similar style or texture. Figure 3E An example of object replacement using a target image as a reference image is shown. Figure 3F-Figure 3I An example of object replacement using a texture image as a reference image is shown.

[0052] Figure 3J-Figure 3L An example of scene completion or outpainting using similar content from another image in accordance with various aspects of the present disclosure is shown. Most of the missing areas in the image can be masked, and given a reference image, the image editing device can generate a scene including content similar to the reference image (e.g., to fill in the missing areas). Figure 3J An example of scene completion using a scene image as a reference image is shown. Figure 3K and Figure 3L An example of extrapolation using a reference image is shown.

[0053] Figure 3M An example of background replacement using similar content from another image according to aspects of the present disclosure is shown. Given a reference background image, an image editing apparatus can generate a background that follows the elements or style of the reference background image.

[0054] Figure 3N An example of using a stylized scene of a similar style from another image according to various aspects of the present disclosure is shown. Given a stylized image, an image editing device can generate a scene that follows a specific style of the stylized image and synthesize the style of the stylized image and the input image.

[0055] Figure 4 An example of an architecture 400 for generative image filling using a reference image according to aspects of the present disclosure is shown. The architecture 400 includes an image encoder 405, a text encoder 410, and a generative model 415. The image encoder 405, the text encoder 410, and the generative model 415 are reference Figure 2 Examples of corresponding elements described, or including references Figure 2 The image encoder 405 can generate an image embedding based on a reference image, and the text encoder 410 can generate a text embedding based on a text prompt. The generative model 415 can use the image embedding from the image encoder 405 and the text embedding from the text encoder 410 to generate a synthetic image based on the input image and the mask.

[0056] The generative model 415 can modify the region of the input image corresponding to the mask using a reference image (e.g., a reference image based on an image embedding) and using a textual hint (e.g., a textual hint based on a text embedding). Thus, the output of the generative model 415 can be an input image with holes filled in using the content or style of the reference image. During training of an image editing apparatus including the image encoder 405, the text encoder 410, and the generative model 415, the reference image can be the same as the unmasked input image. In some examples, for the non-classifier guidance in speculation, a large portion (e.g., 90%) of the reference images may be randomly discarded during training. During testing, any reference image can be randomly fed to the image editing apparatus to run the restoration algorithm.

[0057] An image editing apparatus (e.g., a model pipeline) implementing architecture 400 may support image filling using reference images. The image editing apparatus may use reference image embedding together with generative image inpainting to implement reference guided image filling, and support a wide variety of applications (e.g., guided inpainting, image inpainting, object insertion). For example, the image editing apparatus may use another image as a reference for filling holes in an input image. In some cases, an image generation model may use a reference image for text-to-image generation, or use a reference image to guide the generation of a style. The image editing apparatus may support the use of a reference image as a guide for filling holes in an input image, support flexible references from style to content, support the use of reference images together with text prompts, and support a variety of real-world use cases.

[0058] An image editing apparatus implementing architecture 400 may also support inserting an object into an input image while retaining the identity of the object from a reference image. In some examples, it may be appropriate to insert an object into an input image by referencing a reference image. However, because the reference image can be encoded into a fixed-size embedding (e.g., 1024 dimensions), it may be challenging to capture all of the spatial details of the object. In particular, it may be challenging to maintain the uniqueness and recognizability of the object inserted into the input image. In other words, even after the object has been inserted and may have been modified to fit the new context, the object may not be recognizable as the same object from the reference image.

[0059] In some examples, the image generation model can use additional image encoders and adapters to exploit the cross-attention of the image generator for identical object insertion. However, it may be appropriate to perform additional training on the additional image encoders and adapters, and the identity preservation quality of such an image generation model may still be poor for large geometric changes. An image editing device implementing architecture 400 can apply self-attention to perform object insertion while preserving the identity of objects in a reference image. In this way, additional training for object insertion can be avoided. The image editing device can exploit the image property that recurring objects may be widely present in natural images (e.g., a single training image).

[0060] During speculation, assuming that the image generation model (e.g., generative model 415) has a self-attention component, objects can be directly copied and pasted with a high level of identity preservation (e.g., if a reference image including the object is synthesized next to a hole or mask corresponding to a mask). Because an image editing device implementing architecture 400 can use self-attention to perform object insertion, the image editing device can avoid additional fine-tuning or training to encode the object before insertion. In some examples, adding a copy-and-paste extension for training can improve the performance of the image editing device. In addition, because architecture 400 may not include an additional encoder for encoding objects before insertion, architecture 400 can save computing time. In addition, architecture 400 can achieve reasonable identity preservation when using reference images for object insertion (e.g., synthesis) tasks.

[0061] Figure 5 An example of a guided latent diffusion model 500 according to aspects of the present disclosure is shown. Figure 5 The guided implicit diffusion model 500 described in reference Figure 2 An example of an image generation model described, or including references Figure 2 Describes various aspects of image generation models.

[0062] Diffusion models are a class of generative neural networks that can be trained to generate new data with features similar to those found in the training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for a variety of image generation tasks, including image super-resolution, image generation using perceptual metrics, conditional generation (e.g., text-guided generation), image inpainting, and image manipulation.

[0063] Types of diffusion models include DDPM and DDIM. In DDPM, the generative process involves reversing the stochastic Markov diffusion process. On the other hand, DDIM uses a deterministic process so that the same input produces the same output. Diffusion models can also be characterized by whether the noise is added to the image itself or to the image features generated by the encoder (i.e., implicit diffusion).

[0064] The diffusion model works by repeatedly adding noise to the data during the forward process and then learning to recover the data by denoising the data in the reverse process. For example, during training, the guided latent diffusion model 500 may take as input an original image 505 in a pixel space 510 and apply an image encoder 515 to convert the original image 505 into an original image feature 520 in a latent space 525. Then, the forward diffusion process 530 gradually adds noise to the original image feature 520 to obtain noise features 535 (also in the latent space 525) at various noise levels.

[0065] Next, the back diffusion process 540 (e.g., U-Net ANN) gradually removes noise from the noise features 535 at various noise levels to obtain denoised image features 545 in the latent space 525. In some examples, the denoised image features 545 are compared with the original image features 520 at each operating level in the various noise levels, and the parameters of the back diffusion process 540 of the diffusion model are updated based on the comparison. Finally, the image decoder 550 decodes the denoised image features 545 to obtain an output image 555 in the pixel space 510. In some cases, the output image 555 is created at each noise level in the various noise levels. The output image 555 can be compared with the original image 505 to train the back diffusion process 540.

[0066] In some cases, the image encoder 515 and the image decoder 550 are pre-trained before training the back diffusion process 540. In some examples, they are trained jointly, or the image encoder 515 and the image decoder 550 are fine-tuned jointly with the back diffusion process 540.

[0067] The back diffusion process 540 may also be guided based on the textual cue 560 or another guiding cue such as an image, layout, segmentation map, etc. The textual cue 560 may be encoded using a text encoder 565 (e.g., a multimodal encoder) to obtain guiding features 570 in a guiding space 575. The guiding features 570 may be combined with the noise features 535 at one or more layers of the back diffusion process 540 to ensure that the output image 555 includes the content described by the textual cue 560. For example, the guiding features 570 may be combined with the noise features 535 using a cross-attention block in the back diffusion process 540.

[0068] Image Editing

[0069] exist Figure 6-Figure 8 In the invention, a method, apparatus, non-transitory computer-readable medium, and system for machine learning for image editing are described. One or more aspects of the method include: obtaining an input image, a mask, a reference image, and a textual prompt; encoding the reference image using an image encoder to obtain an image embedding; encoding the textual prompt using a textual encoder to obtain a text embedding; and generating a synthetic image based on the input image, the mask, the image embedding, and the text embedding using an image generation model.

[0070] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include providing image embedding and text embedding as guidance for the image generation model. In some aspects, the synthesized image includes content corresponding to the reference image in an area corresponding to the mask. In some aspects, the synthesized image includes content corresponding to the input image in an area outside the mask.

[0071] In some aspects, the textual hint describes an object in a reference image. The reference image can be a separate image from the input image that does not include content from the input image, or can include a portion of the input image. In some aspects, the reference image includes a style from the input image.

[0072] In some aspects, the composite image includes an object described by a textual cue using a style from a reference image. In some aspects, the mask indicates a background region generated by an image generation model based on the reference image.

[0073] Another method, apparatus, non-transitory computer-readable medium, and system for machine learning for image editing are described. One or more aspects of the method include: obtaining an input image, a mask, and a reference image; inserting the reference image into the input image to obtain a combined input image; encoding the reference image using an image encoder to obtain an image embedding; and generating a composite image based on the combined input image, the mask, and the image embedding using an image generation model.

[0074] Some examples of the method, apparatus, non-transitory computer-readable medium, and system also include performing, by the image generation model, a self-attention operation on the combined input image, wherein the composite image is generated based on the self-attention operation.

[0075] Some examples of the method, apparatus, non-transitory computer-readable medium, and system further include obtaining a textual prompt. Some examples further include encoding the textual prompt using a textual encoder to obtain a text embedding, wherein the synthetic image is generated based on the text embedding. In some aspects, the reference image is inserted into the input image in an area outside of the mask.

[0076] Figure 6 An example of a method 600 for generative image filling using a reference image according to aspects of the present disclosure is shown. In some examples, these operations are performed by a system including a processor that executes a code set to control functional elements of the device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0077] In operation 605, the user may provide an input image, a mask, a reference image, or a text prompt to the image editing apparatus. The mask may be a masked area of ​​the input image, and the reference image and the text prompt may be used as a guide for editing the masked area of ​​the input image. In some cases, the operation of this step involves a reference image. Figure 1 A user device described herein, or may be used by a user via a reference Figure 1 The user equipment described is executed.

[0078] In operation 610, the image editing apparatus may encode the reference image to obtain an image embedding. The image embedding may be an embedding in a latent space of an image generation model and may be provided as a guide for the image generation model to edit the input image. In some cases, the operation of this step involves the reference image. Figure 2 The image encoder described in Figure 2 The image encoder described is implemented.

[0079] At operation 615, the image editing apparatus may encode the text hint to obtain a text embedding. The text embedding may be an embedding in a latent space of the image generation model and may be provided as a guide for the image generation model to edit the input image. In some examples, the text embedding may be an embedding in the same latent space as the image embedding. In some cases, the operation of this step involves referring to Figure 2 The text encoder described in this document can be referenced by Figure 2 The text encoder described here is implemented.

[0080] At operation 620, the image editing apparatus may generate a composite image based on the input image, the mask, the reference image, and the text prompt. For example, the image editing apparatus may edit the region of the input image corresponding to the mask based on the reference image and the text prompt. In some cases, the operation of this step involves the reference image. Figure 2 The image generation model described in Figure 2 The image generation model described is performed.

[0081] In operation 625, the image editing apparatus may provide a composite image to the user. The composite image may incorporate the reference image and the text prompt obtained in operation 605. For example, the composite image may include objects from the reference image, or may be stylized in a style similar to the reference image. In some cases, the operation of this step involves the reference image. Figure 1 and Figure 2 The image editing device described herein may be referred to as Figure 1 and Figure 2 The described image editing device is performed.

[0082] Figure 7 An example of a method 700 for generating a composite image according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a code set to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, the operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0083] In some examples, method 700 can support reference-guided image generation for generative hole filling. In some examples, method 700 can be implemented to adapt reference-guided image hole filling to many real-user applications including object insertion and scene completion. In some examples, method 700 can enable generating content within an image hole by following the style, texture, or content of a reference image.

[0084] At operation 705, the system obtains an input image, a reference image, and a text prompt. The system may also receive a mask. The mask may be specified by the user, or may be determined based on the text prompt, the input image, or the reference image. The mask may be a masked area of ​​the input image, and the reference image and text prompt may be used as a guide to edit the masked area of ​​the input image. In some cases, the operation of this step involves the reference image. Figure 1 and Figure 2 The image editing device described herein may be referred to as Figure 1 and Figure 2 The described image editing device is performed.

[0085] At operation 710, the system encodes the reference image using an image encoder to obtain an image embedding. The image embedding can be an embedding in the latent space of the image generation model and can be provided as a guide for the image generation model to edit the input image. In some cases, the operation of this step involves the reference image. Figure 2 The image encoder described in Figure 2 The image encoder described is implemented.

[0086] At operation 715, the system encodes the text prompt using a text encoder to obtain a text embedding. The text embedding can be an embedding in a latent space of the image generation model and can be provided as a guide for the image generation model to edit the input image. In some examples, the text embedding can be an embedding in the same latent space as the image embedding. In some cases, the operation of this step involves reference Figure 2 The text encoder described in this document can be referenced by Figure 2 The text encoder described here is implemented.

[0087] At operation 720, the system generates a composite image based on the input image, the mask, the image embedding, and the text embedding using the image generation model. In some examples, the composite image may include objects from the reference image, or may be stylized in a style similar to the reference image. In some cases, the operation of this step involves the reference image. Figure 2 The image generation model described in Figure 2 The image generation model described is performed.

[0088] Figure 8 An example of another method 800 for generating a composite image according to aspects of the present disclosure is shown. In some examples, the operations are performed by a system including a processor that executes a code set to control functional elements of the device. Additionally or alternatively, dedicated hardware is used to perform certain processes. Typically, the operations are performed according to the methods and processes described according to aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in combination with other operations.

[0089] At operation 805, the system obtains an input image and a reference image. The system may also receive a mask. The mask may be specified by a user, or may be determined based on the input image or the reference image. The mask may be a masked area of ​​the input image, and the reference image may be used as a guide to edit the masked area of ​​the input image. In some cases, the operation of this step involves using a reference image. Figure 1 and Figure 2 The image editing device described herein may be referred to as Figure 1 and Figure 2 The described image editing device is performed.

[0090] At operation 810, the system inserts the reference image into the input image to obtain a combined input image. Because the reference image can be inserted into the input image, the image generation model can use a self-attention operation to fill in the masked portion of the input image using the reference image (e.g., by associating the masked portion of the input image with the portion of the combined input image that includes the inserted reference image). In some cases, the operation of this step involves the reference image being inserted into the input image. Figure 1 and Figure 2 The image editing device described herein may be referred to as Figure 1 and Figure 2 The described image editing device is performed.

[0091] At operation 815, the system encodes the combined image using an image encoder to obtain an image embedding. The image embedding can be an embedding in the latent space of the image generation model and can be provided as a guide for the image generation model to edit the input image. In some cases, the operation of this step involves referring to Figure 2 The image encoder described in Figure 2 The image encoder described is implemented.

[0092] At operation 820, the system generates a composite image based on the combined input image, the mask, and the image embedding using the image generation model. The composite image may include an object from the reference image blended with features of the composite image while preserving the identity of the object. In some cases, the operation of this step involves the reference image. Figure 2 The image generation model described in Figure 2 The image generation model described is performed.

[0093] Image Editing

[0094] Fig. 9 An example of a method 900 for training a diffusion model according to aspects of the present disclosure is shown. The method 900 represents an example for training a back diffusion process. In some examples, these operations are performed by a system including a processor that executes a code set to control an apparatus (such as Figure 2 functional elements of the device described in .

[0095] Additionally or alternatively, some processes of method 900 may be performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described in accordance with various aspects of the present disclosure. In some cases, the operations described herein are composed of various sub-steps or are performed in conjunction with other operations.

[0096] At operation 905, the user initializes the untrained model. Initialization may include defining the architecture of the model and establishing initial values ​​for model parameters. In some cases, initialization may include defining hyperparameters such as the number of layers, the resolution and channels of each layer block, the location of skip connections, etc.

[0097] At operation 910, the system adds noise to the training image using a forward diffusion process in N stages. In some cases, the forward diffusion process is a fixed process that adds Gaussian noise to the image successively. In a latent diffusion model, Gaussian noise can be added successively to features in the latent space.

[0098] At operation 915, the system, at each stage n, starting from stage N, uses a back diffusion process to predict an image or image features at stage n-1. For example, the back diffusion process can predict the noise added by the forward diffusion process, and the predicted noise can be removed from the image to obtain a predicted image. In some cases, the original image is predicted at each stage of the training process.

[0099] At operation 920, the system compares the predicted image (or image features) at stage n-1 with the actual image (or image features) (such as the image at stage n-1 or the original input image). For example, given the observed data x, the diffusion model can be trained to minimize the negative log-likelihood of the training data - log p θ (x) The upper limit of the change.

[0100] At operation 925, the system updates the parameters of the model based on the comparison. For example, the parameters of the U-Net can be updated using gradient descent. The time-dependent parameters of the Gaussian transition can also be learned.

[0101] Fig.10 An example of a computing device 1000 for image editing according to aspects of the present disclosure is shown. In one aspect, computing device 1000 includes processor(s) 1005, memory subsystem 1010, communication interface 1015, I / O interface 1020, user interface component(s) 1025, and channel 1030.

[0102] In some embodiments, computing device 1000 is Figure 2 An example of an image processing apparatus 200 including Figure 2In some embodiments, the computing device 1000 includes one or more processors 1005, which can execute instructions stored in the memory subsystem 1010 to: obtain an input image, a mask, a reference image, and a text hint; encode the reference image using an image encoder to obtain an image embedding; encode the text hint using a text encoder to obtain a text embedding; and generate a synthetic image based on the input image, the mask, the image embedding, and the text embedding using an image generation model.

[0103] According to some aspects, the computing device 1000 includes one or more processors 1005. In some cases, the processor is an intelligent hardware device (e.g., a general processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or a combination thereof). In some cases, the processor is configured to operate the memory array using a memory controller. In other cases, the memory controller is integrated into the processor. In some cases, the processor is configured to execute computer-readable instructions stored in the memory to perform various functions. In some embodiments, the processor includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0104] According to some aspects, the memory subsystem 1010 includes one or more memory devices. Examples of memory devices include random access memory (RAM), read-only memory (ROM) or hard disk. Examples of memory devices include solid-state memory and hard disk drives. In some examples, the memory is used to store computer-readable computer executable software including instructions, which, when executed, causes the processor to perform various functions described herein. In some cases, the memory unit includes a basic input / output system (BIOS) that controls basic hardware or software operations (such as interaction with peripheral components or devices), etc. In some cases, a memory controller operates a storage unit. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, the storage unit in the memory stores information in the form of a logical state.

[0105] According to some aspects, the communication interface 1015 operates at the boundary between the communication entities (such as the computing device 1000, one or more user devices, the cloud, and one or more databases) and the channel 1030, and can record and process the communication. In some cases, the communication interface 1015 is provided to enable the processing system to be coupled to a transceiver (e.g., a transmitter and / or a receiver). In some examples, the transceiver is configured to transmit (or send) and receive signals for the communication device via an antenna.

[0106] According to some aspects, I / O interface 1020 is controlled by an I / O controller to manage input and output signals for computing device 1000. In some cases, I / O interface 1020 manages peripherals that are not integrated into computing device 1000. In some cases, I / O interface 1020 represents a physical connection or port to an external peripheral. In some cases, an I / O controller uses a communication interface such as a 1020 or other known operating systems. In some cases, the I / O controller represents a modem, keyboard, mouse, touch screen, or similar device, or interacts with a modem, keyboard, mouse, touch screen, or similar device. In some cases, the I / O controller is implemented as a component of the processor. In some cases, a user interacts with the device via the I / O interface 1020 or a hardware component controlled by the I / O controller.

[0107] According to some aspects, user interface component(s) 1025 enable a user to interact with computing device 1000. In some cases, user interface component(s) 1025 include an audio device (such as an external speaker system), an external display device (such as a display screen), an input device (e.g., a remote control device that interfaces with a user interface directly or through an I / O controller), or a combination thereof. In some cases, user interface component(s) 1025 include a GUI.

[0108] The description and drawings described herein represent example configurations and do not represent all implementations within the scope of various aspects. For example, operations and steps may be rearranged, combined, or otherwise modified. In addition, structures and devices may be represented in the form of block diagrams to represent the relationship between components and to avoid blurring the concepts described. Similar components or features may have the same name, but may have different reference numerals corresponding to different drawings.

[0109] For those skilled in the art, some modifications to the present disclosure may be readily apparent, and the principles defined herein may be applied to other changes without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope consistent with the principles and novel features disclosed herein.

[0110] The described methods may be implemented or performed by devices including general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general purpose processor may be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration). Therefore, the functions described herein may be implemented in hardware or software and may be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored in the form of instructions or code on a computer readable medium.

[0111] Computer-readable media include both non-transitory computer storage media and communication media (including any media that facilitate the transmission of code or data). Non-transitory storage media can be any available media that can be accessed by a computer. For example, non-transitory computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk (CD) or other optical disk storage device, magnetic disk storage device, or any other non-transitory medium for carrying or storing data or code.

[0112] In addition, the connecting component can actually be referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals) is used to transmit code or data from a website, server, or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology is included in the definition of medium. Combinations of media are also included within the scope of computer-readable media.

[0113] In the present disclosure and the following aspects, the word "or" indicates an inclusive list, so that, for example, a list of X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. In addition, the phrase "based on" is not used to represent a closed set of conditions. For example, a step described as "based on condition A" can be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted to mean "based at least in part on". In addition, the word "a" or "an" indicates "at least one".

Claims

1. A method comprising: Obtain an input image, a reference image, and a text prompt; encoding the reference image using an image encoder to obtain an image embedding; encoding the text prompt using a text encoder to obtain a text embedding; as well as A synthetic image is generated based on the input image, the image embedding, and the text embedding using an image generation model.

2. The method of claim 1 , wherein generating the composite image comprises: The image embedding and the text embedding are provided as guidance for the image generation model.

3. The method according to claim 1, wherein: The composite image includes content corresponding to the reference image in a region corresponding to the mask.

4. The method according to claim 1, wherein: The composite image includes content corresponding to the input image in areas outside the mask.

5. The method according to claim 1, wherein: The textual cue describes an object in the reference image.

6. The method according to claim 1, wherein: The reference image includes a portion of the input image.

7. The method according to claim 1, wherein: The reference image includes a style from the input image.

8. The method of claim 1, wherein: The composite image includes an object described by the textual cue, the object having a style from the reference image.

9. The method according to claim 1, further comprising: A mask is received from a user, wherein the mask indicates a region generated by the image generation model based on the reference image.

10. The method according to claim 1, further comprising: A mask is generated based on the input image, the reference image, or the textual hint, wherein the mask indicates a region generated by the image generation model.

11. A method comprising: Obtaining an input image and a reference image; inserting the reference image into the input image to obtain a combined input image; encoding the reference image using an image encoder to obtain an image embedding; as well as A synthetic image is generated based on the combined input image and the image embedding using an image generation model.

12. The method according to claim 11, further comprising: A self-attention operation is performed by the image generation model on the combined input image, wherein the composite image is generated based on the self-attention operation.

13. The method according to claim 11, further comprising: Get text alerts; as well as The text prompt is encoded using a text encoder to obtain a text embedding, wherein the synthetic image is generated based on the text embedding.

14. The method according to claim 11, wherein: The reference image is inserted into the input image in areas outside the mask.

15. An apparatus comprising: at least one memory; at least one processor coupled to the at least one memory, wherein the processor is configured to execute instructions stored in the at least one memory; an image encoder configured to encode a reference image to obtain an image embedding; a text encoder configured to encode the text prompt to obtain a text embedding; as well as An image generation model is configured to generate a synthetic image based on an input image, the image embedding, and the text embedding.

16. The system of claim 15, wherein: The image generation model includes a diffusion model.

17. The system of claim 15, wherein: The image generation model includes a self-attention layer configured to operate on a combination of the input image and the reference image.

18. The system of claim 15, wherein: The image encoder and the text encoder are components of a multimodal encoder.

19. The system of claim 15, wherein: The image generation model uses the image embedding and the text embedding for classifier-free guidance.

20. The system of claim 15, further comprising: A user interface is configured to obtain selection input from a user, wherein a mask is created based on the selection input.