Automatic photo editing via language requests

Abstract

Embodiments of this disclosure relate to automated photo editing via language requests. This disclosure relates to systems and methods for automatically processing images based on user requests. In some examples, the request is divided into retouching commands (e.g., global edits) and image patching commands (e.g., local edits). Retouching masks and image patching masks are generated to indicate the areas where editing will be applied. Photo request attention and multimodal modulation processes are applied to features representing the image, and the modified features are used to generate a modified image incorporating the user request.

Description

Technical Field

[0001] The following generally deals with image processing, and more specifically with automated image processing using machine learning. Background Technology

[0002] Digital image processing refers to using computers to edit digital images using algorithms or processing networks. For example, digital image processing can be performed on images taken by novice photographers to enhance the overall appeal of the image. In some cases, images can be processed to improve image sharpness. For example, a digital image may contain dark areas, unwanted tones, or objects.

[0003] However, even with advanced image editing software, editing these attributes can be difficult and time-consuming. Furthermore, conventional image editing systems cannot perform image editing based on natural language requests. Therefore, there is a need in the art for improved image processing systems and methods to automatically edit images based on user requests. Summary of the Invention

[0004] Embodiments of this disclosure relate to systems and methods for automatically processing images based on user requests. In some examples, the request is divided into retouching commands (e.g., global editing) and inpainting commands (e.g., local editing). Retouching masks and inpainting masks are generated to indicate the areas where editing will be applied. Photo request attention and multimodal modulation processes are applied to features representing the image, and the modified features are used to generate a modified image that incorporates the user request.

[0005] A method, apparatus, non-transitory computer-readable medium, and system for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: identifying an image and editing commands for the image; encoding the editing commands to obtain an image inpaint vector and a retouching vector; generating an image inpaint mask based on the image inpaint vector and generating a retouching mask based on the retouching vector; generating an image feature representation based on the image and the image inpaint mask; generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vector; and generating a modified image based on the modified image feature representation, wherein the modified image represents the application of the editing commands to the image.

[0006] A method, apparatus, non-transitory computer-readable medium, and system for training a neural network for image processing are described. One or more aspects of the method, apparatus, non-transitory computer-readable medium, and system include: receiving training data, which includes training images, editing commands, and ground truth images, the ground truth images representing the application of editing commands to the training images; encoding the editing commands to obtain image inpaint vectors and retouching vectors; generating an image inpaint mask based on the image inpaint vectors, and generating a retouching mask based on the retouching vectors; generating an image feature representation based on the training images and the image inpaint mask; generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vectors; generating a modified image based on the modified image feature representation, wherein the modified image represents the application of editing commands to the training images; computing a loss function based on the modified image and the ground truth image; and training the neural network based on the loss function.

[0007] An apparatus, system, and method for image processing are described. One or more aspects of the apparatus, system, and method include: a text encoder configured to encode an editing command for an image to obtain an image inpaint vector and a patch vector; a phrase conditional truth (PCG) network configured to generate an image inpaint mask based on the image inpaint vector and to generate a patch mask based on the patch vector; an image encoder configured to generate an image feature representation based on the image and the image inpaint mask; a multimodal modulation network configured to generate a modified image feature representation based on the image feature representation, the patch mask, and an attention matrix computed using the patch vector; and a decoder configured to generate a modified image based on the modified image feature representation, wherein the modified image represents the application of an editing command to the image. Attached Figure Description

[0008] Figure 1 An example of an image editing system according to aspects of this disclosure is shown;

[0009] Figure 2 An example of an image editing process according to aspects of this disclosure is shown;

[0010] Figure 3 An example of automatic image editing according to aspects of this disclosure is shown;

[0011] Figure 4 An example of an attention map for automatic image editing according to aspects of this disclosure is shown;

[0012] Figure 5 An example of an image editing apparatus according to aspects of this disclosure is shown;

[0013] Figure 6An example of a machine learning model for image editing according to aspects of this disclosure is shown;

[0014] Figure 7 Examples of a Photo Request Attention (PRA) network and a multimodal modulation network according to aspects of this disclosure are shown;

[0015] Figure 8 An example of a phrase conditional truth (PCG) network according to aspects of this disclosure is shown;

[0016] Figure 9 An example of image processing according to aspects of this disclosure is shown;

[0017] Figure 10 An example of a process for generating a revision vector based on editing commands, according to aspects of this disclosure, is shown;

[0018] Figure 11 An example of generating an image patching mask according to aspects of this disclosure is shown;

[0019] Figure 12 An example of generating image feature representations according to aspects of this disclosure is shown;

[0020] Figure 13 An example of a modified image feature representation according to aspects of this disclosure is shown;

[0021] Figure 14 Examples of training machine learning models according to aspects of this disclosure are shown; and

[0022] Figure 15 An example diagram of a training method according to aspects of this disclosure is shown. Detailed Implementation

[0023] This disclosure describes systems and methods for image processing, and particularly for editing images based on language commands. According to some embodiments, editing requests are divided into retouching commands (e.g., global edits) and image patching commands (e.g., local edits). Retouching masks and image patching masks are generated to indicate the areas where editing will be applied. In some embodiments, the image patching mask is applied before the image is encoded into a set of image features. A photo request attention process then adaptively embeds the retouching commands and visual features in the same space to determine how much editing should be applied at different locations. A multimodal modulation process transforms the retouching commands into modulation parameters for modifying image features. The modified features are then decoded to produce an image that incorporates the original editing requests.

[0024] Image editing can include various subtasks, including image patching and retouching. Image patching refers to modifying specific objects in an image, such as adding or removing objects. Retouching refers to editing the global features of an image. However, in some cases, it is necessary to apply retouching commands to different parts of an image in different ways. Retouching commands can include operations such as changing brightness, color, or contrast.

[0025] Conventional image editing systems cannot effectively apply unconstrained image editing commands, such as those that do not specify which regions of the image to apply the command to. Furthermore, these systems cannot handle requests that combine both image patching and retouching commands. For example, deep neural networks such as Generative Adversarial Networks (GANs) can be trained to perform a limited set of predefined operations. These networks can use reinforcement learning to learn the optimal parameters for each operation. However, such systems are limited in their use of linguistic requests and cannot edit images when the request is not explicitly mapped to a predefined operation. Moreover, existing networks cannot extract relevant editing requests from natural language requests. Therefore, the operations of these networks may be limited to the application of a limited set of operations that includes a detailed description of the target image. Consequently, existing methods have low effectiveness in practical applications.

[0026] Embodiments of this disclosure include an image editing system for detecting both image inpainting commands and retouching commands in natural language requests. In some embodiments, a bidirectional attention-based model is used to extract linguistic features from the user request. The linguistic features can be used to generate image masks corresponding to image inpainting and retouching elements. The image inpainting mask can be applied before encoding the image features. The retouching command is then applied based on the retouching mask using a photo request attention algorithm and a multimodal modulation network. As a result, the image editing system described in this disclosure can edit real-world images based on both ambiguous (i.e., they do not specify which parts of the image should be retouched) and complex (i.e., they include both image inpainting and retouching elements) commands.

[0027] Some embodiments of this disclosure can be implemented in the context of image editing. Figures 1 to 4 Examples of image editing systems are provided, along with instructions on how to apply such systems to edit images. Figures 5 to 8 An example system architecture is described. Figures 9 to 13 Examples of image processing methods are provided. Figure 14 and Figure 15 A method for training a neural network to perform image processing is described.

[0028] Image editing

[0029] Figure 1An example of an image editing system according to aspects of this disclosure is shown. The example includes a user 100, a user device 105, an image editing apparatus 110, a cloud 115, and a database 120. In some embodiments, the user 100 may provide an image and an image editing request to the image editing apparatus 110 via the cloud 115 to perform an image editing operation. In some embodiments, the user device 105 may include image editing software that incorporates elements of the image editing apparatus 110. In some cases, the image is stored and retrieved from the database 120.

[0030] In the example shown, user 100 provides an image including multiple people and requests to "remove the pink tones and make the colors more vibrant." Image editing device 110 can receive the image and the user request and automatically perform an operation to fulfill the request.

[0031] Image editing has a wide range of applications, such as removing pedestrians from wedding photos or retouching overexposed or underexposed outdoor images. In some cases, photo editing software (e.g., Adobe) can be used. These edits can be performed. However, because learning the editing process takes time, beginners may find it difficult or impossible to perform complex edits.

[0032] Conventional methods for automated image editing are limited to simple requests and a finite number of operations using predefined global operations. However, in practice, user requests can be both complex and ambiguous. For example, a request might be complex because it could indicate multiple commands, including image patching and retouching commands. Requests might be ambiguous because they might not provide detailed information about the operation to be performed or the area to be edited.

[0033] Therefore, existing methods may apply operations inappropriately. For example, consider an input image with a dark background, a bright foreground, and a blurred user request (i.e., to brighten the image). In this case, brightening the background more and the foreground less will increase image sharpness. Applying the same edits equally to all areas may produce undesirable results. Furthermore, conventional editing tools cannot handle requests that may not be represented by predefined operations (e.g., removing pink tones from an image). The limited number of predefined operations that conventional systems can employ does cover the range of real-world operations that a user might want to apply to the target image.

[0034] Embodiments of this disclosure include image editing tools that can automatically edit images using text-based requests without complex training. Embodiments can adaptively edit photographs using unconstrained language requests. In some cases, Photo Request Attention (PRA) adaptively edits the input image at different spatial locations. PRA computes the attention between the language request embedding and patches on a visual feature map. In some examples, PRA can learn to assign appropriate weights to each location using a ground truth target image that has been manually retouched. For example, when requesting to brighten an image, very bright areas in the image might receive lower attention because the real intention is likely just to brighten the darker areas.

[0035] Furthermore, the multimodal modulation network directly embeds the language request into the conditional signal for GANs to eliminate constraints dependent on a predefined set of operations. The multimodal modulation network bridges the speech and visual spaces for conditional image synthesis by generating two modulation parameters from the speech embedding for the revision request. The modulation parameters are then used to scale and shift visual features. Figure 1 This is how you edit.

[0036] To train the model, a pre-trained speaker model can be used to compute a Re-description Similarity Score (RSS) to generate a request given an input image and a synthetic image. Next, the similarity between the generated request and the real request is calculated, where a higher similarity indicates the performance of the enhancement. The results can be used to update the model's parameters. Implementing the model on the Real Image Editing Request (GIER) dataset and the Request Enhancement dataset demonstrates the effectiveness of an image editing tool that incorporates the techniques described in this paper.

[0037] User device 105 may be a personal computer, laptop computer, mainframe computer, handheld computer, personal assistant, mobile device, or any other suitable processing device. In some examples, user device 105 includes image editing software that can communicate with image editing device 110. In other examples, user device 105 includes components capable of directly performing the automatic image editing operations described herein.

[0038] According to some aspects, the image editing device 110 resides on a server connected to the user equipment 105 via a cloud 115. Specifically, the image editing device 110 identifies images and editing commands for the images, and automatically edits the images according to the editing commands. In various embodiments, the image editing device 110 includes a general-purpose computing device, a personal computer, a laptop computer, a mainframe computer, a supercomputer, or any other suitable processing device.

[0039] The server provides one or more functions to users linked through one or more networks. In some cases, the server includes a single microprocessor board containing a microprocessor responsible for controlling all aspects of the server. In some cases, the server uses the microprocessor and protocols to exchange data with other devices / users on one or more networks via Hypertext Transfer Protocol (HTTP) and Simple Mail Transfer Protocol (SMTP), although other protocols such as File Transfer Protocol (FTP) and Simple Network Management Protocol (SNMP) may also be used. In some cases, the server is configured to send and receive Hypertext Markup Language (HTML) formatted files (e.g., for displaying web pages). Image editing device 110 is a reference. Figure 5 Examples of the corresponding elements described, or including references Figure 5 The aspects of the corresponding element described.

[0040] Cloud 115 is a computer network configured to provide on-demand availability of computer system resources, such as data storage and computing power. In some examples, Cloud 115 provides resources without active management by user 100. The term cloud is sometimes used to describe a data center available to many users 100 via the Internet. Some large cloud networks have the capability to be distributed across multiple locations from a central server. If a server has a direct or close connection to user 100, that server is designated as an edge server. In some cases, Cloud 115 is limited to a single organization. In other examples, Cloud 115 can be used by many organizations. In one example, Cloud 115 includes a multi-layered communication network comprising multiple edge routers and a core router. In another example, Cloud 115 is based on a collection of local switches in a single physical location.

[0041] Database 120 is an organized collection of data. For example, database 120 may store images in a specified format called a schema. Database 120 may be constructed as a single database, a distributed database, multiple distributed databases, or an emergency backup database. In some cases, a database controller may manage the data storage and processing within database 120. In some cases, users interact with the database controller. In other cases, the database controller may operate automatically without user interaction.

[0042] Figure 2 An example of an image editing process according to aspects of this disclosure is shown. Figure 2The aspects of the process shown can be performed by an automated image editing tool that can edit images using text-based requests (e.g., receiving text-based requests from a user). In one example, the editing tool can save time spent on image editing tasks and can be used by photography learners.

[0043] Conventional image editing tools use predefined global operations and therefore cannot handle unconstrained language requests. Embodiments of this disclosure include image editing tools that can achieve spatially adaptive photo editing using unconstrained language requests. In some cases, a Photo Request Attention (PRA) is developed, which learns the degree of editing for each spatial location. The PRA makes the editing process spatially adaptive and interpretable. Furthermore, a multimodal modulation network is developed that embeds the language request into modulation parameters used to achieve editing by shifting and scaling visual feature maps. The multimodal modulation network is capable of handling unconstrained language requests. One or more embodiments include a metric called a re-description similarity score to evaluate the match between the edited result and the request.

[0044] At step 205, the user identifies the image and provides a voice editing command. For example, the user could provide a voice command such as "Remove the pink tint and make the colors more vibrant." In some examples, the voice editing command is provided using an audio device such as a microphone. In some cases, this step involves operations such as those described in the reference. Figure 1 The user described, or may be referred to as such Figure 1 The user described is responsible for execution.

[0045] At operation 210, the system transcribes the language editing request into text. For example, the system can utilize speech-to-text software such as Julius or Kaldi.

[0046] At operation 215, the system distinguishes between local and global editing. For example, the transcribed text may be encoded by a text encoder and divided into text indicating image repair commands and text identifying retouching commands. In some cases, this step involves referencing... Figure 1 and Figure 5 The image editing apparatus described, or may be referenced Figure 1 and Figure 5 The image editing device described herein shall be used to perform this action.

[0047] At operation 220, the system processes the image based on both local and global editing. For example, a retouching mask can be generated based on a retouching command, and an image inpainting mask can be generated based on an image inpainting command. The image inpainting mask can be applied to the image before encoding. Then, the photo request attention module can use the retouching mask to generate attention weights for different parts of the image, and the multimodal modulation component can convert the retouching command and attention weights into modulation parameters for modifying image features. Finally, a modified image can be generated based on the modified image features.

[0048] Conventional image editing methods set predefined operations and predict the editing parameters for these operations. For example, rule-based methods map user phrases to specific image editing operations and are therefore not based on machine learning. In some cases, reinforcement learning is applied to learn the optimal parameters for predefined operations. In some cases, language requests for image editing use predefined global operations. Alternatively, generative adversarial networks (GANs) can be used to learn predefined operators as convolutional kernels. Thus, conventional methods can only handle restricted language requests that can be represented by a predefined set of operations. However, by combining a generative decoder with a multimodal modulation network, embodiments of this disclosure can generate edited images with edits that exceed a predetermined set of operations.

[0049] At operation 225, the system provides the user with the modified image. The modified image may represent the original image with modifications based on a language editing request. In some examples, the modified image is provided from a server located in the cloud. In other examples, the modified image is generated by software located on the user's device and provided to the user.

[0050] Figure 3 An example of automatic image editing according to aspects of this disclosure is shown. The example includes an original image 300, an edit request 305, and a modified image 310. According to aspects of this disclosure, the modified image 310 can be automatically generated from the original image 300 based on the edit request 305.

[0051] Editing request 305 may include multiple subcommands, such as image patching and retouching. Image patching may include tasks such as adding or removing objects or filling gaps in an image. Image retouching includes global editing operations such as brightness, contrast, etc. Some requests may include multiple commands, such as multiple retouching commands (e.g., "Remove pink tones from the image to make the colors more vibrant"), or both image patching and retouching commands (e.g., "Brighten the image and remove the woman on the left"). In some examples, these language editing requests are voice commands from the user and can be transcribed into text using speech-to-text software.

[0052] Figure 4 An example of an attention map 420 for automatic image editing according to aspects of this disclosure is shown. The example includes an input image 400, a target image 405, an output image 410, an edit request 415, and an attention map 420. The output image 410 can be automatically generated based on the input image 400 to achieve the target image 405 according to the edit request 415. The output image 410 can be generated based on the attention map 420 indicating which parts of the image to apply the edit request 415 to. The output image 410 can be generated according to reference... Figures 9 to 13 The described method uses reference Figures 5 to 8 The system described is generated.

[0053] System Architecture

[0054] Reference device for image processing Figures 5 to 8 The device is described in one or more aspects, including: a text encoder configured to encode an editing command for an image to obtain an image inpaint vector and a retouching vector; a phrase conditional truth (PCG) network configured to generate an image inpaint mask based on the image inpaint vector and a retouching mask based on the retouching vector; an image encoder configured to generate an image feature representation based on the image and the image inpaint mask; a multimodal modulation network configured to generate a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vector; and a decoder configured to generate a modified image based on the modified image feature representation, wherein the modified image represents the application of the editing command to the image.

[0055] Some examples of the apparatus, system, and method also include a Photo Request Attention (PRA) network configured to compute an attention matrix based on image feature representations and retouching vectors. In some aspects, the PRA network includes an embedding network configured to embed the image feature representations and retouching vectors into a common embedding space.

[0056] Some examples of the apparatus, system, and method also include an audio input device configured to receive and transcribe editing commands from a user, wherein the encoding is based on the transcribed editing commands. In some aspects, the text encoder includes bidirectional long short-term memory (BiLSTM). In some aspects, the image encoder includes gated convolutional layers.

[0057] The description and accompanying drawings described herein represent exemplary configurations and do not represent all implementations within the scope of the claims. For example, operations and steps may be rearranged, combined, or otherwise modified. Furthermore, structures and devices may be represented in block diagram form to illustrate relationships between components and to avoid confusion with the described concepts. Similar components or features may have the same name but may have different reference numerals corresponding to different drawings.

[0058] Some modifications to this disclosure will be apparent to those skilled in the art, and the principles defined herein can be applied to other variations without departing from the scope of this disclosure. Therefore, this disclosure is not limited to the examples and designs described herein, but is given the broadest scope consistent with the principles and novel features disclosed herein.

[0059] Figure 5 An example of an image editing apparatus 500 according to aspects of this disclosure is shown. The image editing apparatus 500 is a reference. Figure 1 Examples of the corresponding elements described, or references may be included. Figure 1 The corresponding elements described. On one hand, the image editing device 500 includes a processor unit 505, a memory unit 510, an I / O module 515, a training component 520, and a machine learning model 525. In some examples, the I / O module 515 includes or is connected to an audio input device that collects audio information from the user (e.g., for processing verbal editing requests).

[0060] Processor unit 505 may be an intelligent hardware device (e.g., a general-purpose processing component, a digital signal processor (DSP), a central processing unit (CPU), a graphics processing unit (GPU), a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, processor unit 505 is configured to use a memory controller to operate a memory array. In other cases, the memory controller is integrated into the processor. In some cases, processor unit 505 is configured to execute computer-readable instructions stored in memory to perform various functions. In some embodiments, processor unit 505 includes dedicated components for modem processing, baseband processing, digital signal processing, or transmission processing.

[0061] Storage unit 510 may include random access memory (RAM), read-only memory (ROM), or a hard disk. Examples of storage unit 510 include solid-state memory and hard disk drives. In some examples, storage unit 510 is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform the various functions described herein. In some cases, storage unit 510 includes a basic input / output system (BIOS) that controls basic hardware or software operations, such as interaction with peripheral components or devices. In some cases, a memory controller operates the storage unit. For example, a memory controller may include a row decoder, a column decoder, or both. In some cases, storage units within the memory store information in the form of logical states.

[0062] I / O module 515 manages input and output signals for the device. For example, I / O module 515 can manage signals from an external audio device used to capture voice requests for image editing. I / O module 515 can also manage peripheral devices not integrated into the device. In some cases, I / O module 515 can represent a physical connection or port to an external peripheral device. In some cases, I / O module 515 can utilize the operating system, such as... Or other known operating systems. In other cases, I / O module 515 may represent or interact with a modem, keyboard, mouse, touchscreen, or similar device. In some cases, I / O module 515 may be implemented as part of a processor. In some cases, a user may interact with the device via I / O module 515 or via hardware components controlled by I / O module 515.

[0063] Training component 520 can be used to train machine learning model 525. According to some embodiments, training component 520 receives training data including training images, editing commands, and ground truth images representing the application of editing commands to the training images. In some examples, training component 520 computes a loss function based on the modified image and the ground truth image. In some examples, training component 520 trains a neural network based on the loss function. In some examples, training component 520 computes an unconditional adversarial loss using a discriminator network based on the modified image and the ground truth image. In some examples, training component 520 trains a neural network based on the unconditional adversarial loss. In some examples, the loss function includes L1 loss.

[0064] Machine learning model 525 can be used to apply editing requests to images. In some examples, machine learning model 525 can be implemented as an artificial neural network (ANN). An ANN is a hardware or software component that comprises multiple connected nodes (i.e., artificial neurons) that loosely correspond to neurons in the human brain. Each connection or edge transmits a signal from one node to another (like a physical synapse in the brain). When a node receives a signal, it processes it and then transmits the processed signal to other connected nodes. In some cases, the signals between nodes include real numbers, and the output of each node is computed as a function of the sum of its inputs. Each node and edge is associated with one or more node weights that determine how the signal is processed and transmitted.

[0065] During training, these weights are adjusted to increase the accuracy of the results (i.e., by minimizing a loss function that corresponds in some way to the difference between the current result and the target result). Edge weights increase or decrease the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which no signal is transmitted at all. In some examples, nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is called the input layer, and the last layer is called the output layer. In some cases, the signal traverses certain layers multiple times.

[0066] In some examples, machine learning models 525 include convolutional neural networks (CNNs). CNNs are a class of neural networks commonly used in computer vision or image classification systems. In some cases, CNNs can enable the processing of digital images with minimal preprocessing. CNNs can be characterized by using convolutional (or cross-correlation) hidden layers. These layers apply convolution operations to the input before signaling the results to the next layer. Each convolutional node can process data for a finite input domain (i.e., the receptive field). During the forward pass of a CNN, the filters of each layer can be convolved across the entire input volume to compute the dot product between the filter and the input. During training, the filters can be modified so that they activate when specific features in the input are detected.

[0067] In some examples, machine learning model 525 includes a recurrent neural network (RNN). An RNN is a class of ANN in which the connections between nodes form a directed graph along an ordered (i.e., temporal) sequence. This allows RNNs to model temporal dynamics, such as predicting which element in a sequence should follow. Therefore, RNNs are well-suited for tasks involving ordered sequences, such as text recognition (sorting words in a sentence). The term RNN can include finite spiking recurrent networks (characterized by nodes forming a directed acyclic graph) and infinite spiking recurrent networks (characterized by nodes forming a directed cyclic graph).

[0068] In some examples, machine learning model 525 includes a generative adversarial network (GAN). A GAN is a class of machine learning models in which two neural networks compete against each other. For example, the generator network can generate an image, and the discriminator network can try to determine whether the image is a real image or a generated image.

[0069] Given a training set, a GAN can learn to generate new data with the same statistics as the training set. For example, a GAN trained on images can generate new images that at least superficially appear realistic. GANs can be used for semi-supervised learning, fully supervised learning, and reinforcement learning.

[0070] According to some embodiments, GANs can be used to construct conditional generative adversarial networks to synthesize images based on conditional signals. Input conditional signals can include images, human poses, or semantic segmentation masks. Text-to-image synthesis (which learns a mapping from text descriptions to images) is used as embedded textual information as a condition for the GAN. GAN variants are developed based on the text-to-image framework (e.g., Semantic Image Synthesis GAN (SISGAN), Text Adaptive GAN (TAGAN), GAN for Image Manipulation (ManiGAN), etc.) to manipulate input images using textual descriptions. However, the textual descriptions used in these GAN variants are similar to summarizing the properties of the target image.

[0071] For example, TAGAN concatenates textual and image features as a condition for GANs lacking design for multimodal feature combinations. ManiGAN computes attention between words and image patches, which may not be suitable for less detailed descriptions of the target image. Similarly, the Generative Neural Visual Artist (GeNeVA) task and GeNeVA-GAN can be used for iterative image generation, where new objects are added individually based on linguistic requests. However, such methods focus on iterative editing and are not suitable for real-world scenarios. Embodiments of this disclosure include image editing tools that use PRA and multimodal modulation algorithms, model linguistic requests, and can be applied to real-world scenarios.

[0072] In one embodiment, the machine learning model 525 includes a text encoder 530, a phrase conditional truth (PCG) network 535, an image encoder 540, a photo request attention (PRA) network 545, a multimodal modulation network 550, and a decoder 555.

[0073] According to some embodiments, text encoder 530 encodes editing commands to obtain image inpainting vectors and revision vectors. In some examples, text encoder 530 generates word embeddings for each word in a vocabulary set for the editing command. In some examples, text encoder 530 generates probability vectors that include image inpainting probabilities and revision probabilities for each word in the vocabulary set. In some examples, text encoder 530 applies the image inpainting probabilities to the corresponding words in the vocabulary set to obtain image inpainting weighted word representations, where the image inpainting vectors include image inpainting weighted word representations. In some examples, text encoder 530 applies revision probabilities to the corresponding words in the vocabulary set to obtain revision weighted word representations, where the revision vectors include revision weighted word representations. In some aspects, the image inpainting vectors indicate local editing of the image. In some aspects, the revision vectors indicate global editing of the image.

[0074] In some respects, the text encoder 530 incorporates a bidirectional long short-term memory (Bi-LSTM). A Bi-LSTM is an RNN that includes feedback connections. In one example, an LSTM includes units, input gates, output gates, and forget gates. Units store values ​​for a certain period, and gates determine the flow of information in and out of the units. LSTM networks can be used for predictions based on a series of data, where there may be gaps of unknown size between relevant information in the series. LSTMs can help mitigate the vanishing (and exploding) gradient problems when training RNNs. The text encoder 530 is a reference... Figure 6 Examples of the corresponding elements described, or including their aspects.

[0075] According to some embodiments, the PCG network 535 generates an image inpainting mask based on an image inpainting vector (or image inpainting text) and a retouching mask based on a retouching vector (or retouching text). In some examples, the PCG network 535 encodes the image to generate image features. In some examples, the PCG network 535 identifies the image inpainting object based on an editing command. In some examples, the PCG network 535 generates attention weights in the image inpainting object based on the editing command. In some examples, the PCG network 535 generates the image inpainting mask based on image features and attention weights. In some examples, the PCG network 535 determines whether the retouching vector indicates a global edit. In some examples, the PCG network 535 generates the retouching mask based on this determination. The PCG network 535 is a reference. Figure 6 Examples of the corresponding elements described, or including their aspects.

[0076] According to some embodiments, image encoder 540 generates image feature representations based on an image and an image inpainting mask. In some examples, image encoder 540 encodes the image to generate image features. In some examples, image encoder 540 applies an image inpainting mask to the image features to obtain image-inpainted image features. In some examples, image encoder 540 performs convolution operations on the image-inpainted image features to obtain image feature representations. For example, image encoder 540 may include a CNN. In some aspects, image encoder 540 includes gated convolutional layers. Image encoder 540 is a reference... Figure 6 Examples of the corresponding elements described, or including their aspects.

[0077] According to some embodiments, the PRA network 545 computes the attention matrix based on image feature representations and retouching vectors. In some examples, the PRA network 545 expands the retouching vectors based on a retouching mask to obtain an expanded retouching vector. In some examples, the PRA network 545 weights the expanded retouching vectors based on the attention matrix to obtain a weighted retouching vector.

[0078] In some embodiments, the PRA network 545 includes an embedding network configured to embed image feature representations and retouching vectors into a common embedding space. The PRA network 545 can be an attention network. An attention mechanism is a method of assigning different importance levels to different elements of the input. Calculating attention can involve three basic steps. First, the similarity between the query vector obtained from the input and the key vector is calculated to generate attention weights. Similarity functions can include dot products, concatenation, detectors, etc. Next, the weights are normalized using a flexible maximum function. Finally, the attention weights are weighted together with their corresponding values. The PRA network 545 is a reference... Figure 6 and Figure 7 Examples of the corresponding elements described, or including their aspects.

[0079] According to some embodiments, the multimodal modulation network 550 generates a modified image feature representation based on an image feature representation, a retouching mask, and an attention matrix computed using the retouching vector. In some examples, the multimodal modulation network 550 generates scaling and shifting parameters based on a weighted retouching vector. In some examples, the multimodal modulation network 550 adds the product of the scaling parameters and the image feature representation to the shifting parameters to obtain the modified image feature representation. The multimodal modulation network 550 is a reference... Figure 6 and Figure 7 Examples of the corresponding elements described, or including their aspects.

[0080] According to some embodiments, decoder 555 generates a modified image based on a modified image feature representation, wherein the modified image representation represents the application of an editing command to the image. In some examples, decoder 555 performs a convolution operation on the modified image feature representation to obtain the modified image. In some cases, decoder 555 includes gated convolutional layers. Decoder 555 is a reference... Figure 6 Examples of the corresponding elements described, or including their aspects.

[0081] The described methods can be implemented or performed by devices including general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof. A general-purpose processor can be a microprocessor, a conventional processor, a controller, a microcontroller, or a state machine. A processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration). Therefore, the functions described herein can be implemented in hardware or software and can be performed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on a computer-readable medium in the form of instructions or code.

[0082] Computer-readable media include both non-transitory computer storage media and communication media, including any medium that facilitates the transfer of code or data. Non-transitory storage media can be any available medium that can be accessed by a computer. For example, non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), optical discs (CDs) or other optical disk storage, magnetic disk storage, or any other non-transitory medium used to carry or store data or code.

[0083] Furthermore, the connecting components may be appropriately referred to as computer-readable media. For example, if code or data is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (such as infrared, radio, or microwave signals), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology are all included in the definition of media. Combinations of media are also included within the scope of computer-readable media.

[0084] Figure 6 Examples of machine learning models for image editing according to aspects of this disclosure are shown. The examples include a text encoder 600, a PCG network 605, an image encoder 610, a PRA network 615, a multimodal modulation network 620, and a decoder 625.

[0085] An image editing device edits a given photograph x according to the modifications specified in the input language request t, to produce an output edited photograph. Therefore, there are two types of language requests: image inpainting (e.g., removing the woman on the left) and retouching (e.g., brightening an image). In some cases, the model locates the region to be edited, and then image inpainting and retouching operations are performed using different modules. Gated convolutions are applied to the model's layers for image inpainting operations. Furthermore, photo request attention predicts the degree of editing in the spatial dimensions for appropriate editing for retouching operations. Retouching is performed by modeling the language request as modulation parameters that modify the image feature map at the desired location in the image.

[0086] A neural network encoder (e.g., a BiLSTM encoder) extracts language features from an input language request t. in Let be the embedding of the i-th word, and T be the length of the input language request. The input language requests are classified into patch phrases and image patching phrases. The probability vector for the i-th word is described as calculated as follows:

[0087] p i =softmax(W p l i +b p (1)

[0088] in and These are learnable parameters. and represents the probability that the i-th word is included in the revision phrase and the image patching phrase, respectively.

[0089] Image Patch Phrase L inp The revised version of the phrase L ret The weighted word embedding can be calculated as follows:

[0090]

[0091] Where j∈{inp,ret}.

[0092] The PCG network 605 is used to locate the region to be edited for each phrase. Given an input phrase L, a binary classifier f is used. 有效 The existence of a valid phrase is determined. For example, the phrase "image inpainting" is not a valid phrase for the "brighten image" request. When the input phrase is valid, the real model is applied. Image inpainting is a local editing operation, while retouching can be global.

[0093] Another binary classifier f 全局It can be used to determine the input phrase L ret Whether to instruct global editing of valid modified phrases. Real Model J 真实 (For example, MattNet) can be used to perform localization. The PCG network 605 uses image features V and speech phrases L inp and L ret As input, and the following output corresponds to mask M inp and M ret :

[0094]

[0095] Where when j = inp, j ∈ {inp, ret} and f 全局 =0. The ground truth mask labeled in the training dataset can be used to pre-train a real model.

[0096] In some cases, the image encoder 610, including gated convolutional layers, can be used for image inpainting. The PRA network 615 and the multimodal modulation network 620 can be used to obtain the image inpainting phrase L. inp Revised phrase L ret and the corresponding mask M inp and M ret Then, revisions were made.

[0097] In some cases, when obtaining M for image inpainting inp Then, the objects in the original input image x are erased as follows:

[0098] x′=x⊙(1-M inp (4)

[0099] Where ⊙ represents the Hadamard product. In some examples, the image editing tool takes an image x′ of the image to be repaired that has been erased as input. Gated convolutions are used in the layers of the framework, which fill the image repair area with contextual features. The feature map V of the input image x′ in the i-th convolutional layer. i Calculated as follows:

[0100] Where σ is the sigmoid function, with an output gate value between 0 and 1, and φ is the LeakyReLU activation function. W g and W f These are different convolutional filters. Feature map V i It is sent to the retouching section and finally passes through decoder 625 to generate the output edited image.

[0101] The text encoder 600, PCG network 605, image encoder 610, and decoder 625 are references. Figure 5 Examples of the corresponding elements described, or aspects thereof. The PRA network 615 and the multimodal modulation network 620 are references. Figure 5 and Figure 7 Examples of the corresponding elements described, or including their aspects.

[0102] Figure 7 Examples of a PRA network 700 and a multimodal modulation network 705 according to aspects of this disclosure are shown. Conventional image editing systems can only automatically apply user editing requests when the request is both detailed and simple (e.g., a request to brighten all parts of an image). However, embodiments of this disclosure can apply multiple global editing requests differently to different parts of an image, even if the request is ambiguous.

[0103] This is accomplished using a PRA network 700 and a multimode modulation network 705. The PRA network 700 and the multimode modulation network 705 are references. Figure 5 and Figure 6 Examples of the corresponding elements described, or including their aspects.

[0104] According to embodiments of this disclosure, the image editing tool adaptively edits the photo based on the relevance of the input photo and the request. The PRA network 700 first considers visual features... Japanese revised version phrases Embedded within the same embedding space. Attention matrix. The calculation is as follows:

[0105] A = Sigmoid((W1L) ret ) T (W2V)) (6)

[0106] in and These are learnable parameters. The sigmoid function can be used to normalize the degree weights to [0, 1]. Attention calculates multimodal similarity, such that a larger value of A indicates a higher degree of editing.

[0107] Revision phrase embedding L ret Expanding in spatial dimensions to obtain spatially adaptive conditions for image synthesis. Next, in the region to be edited, through M... ret To locate the location, see below:

[0108] in It is an extended language embedding. The elements in the following are reweighted using attention matrix A to obtain

[0109]

[0110] in These are learnable parameters, and ⊙ represents the Adamama product. In reweighting... The speech embeddings are then fed into a multimodal modulation network as conditional signals for editing.

[0111] The multimodal modulation network 705 bridges the speech and visual spaces for conditional image synthesis by directly modeling language requests as modulation parameters used to modify visual feature maps.

[0112] Modulation parameters and In obtaining weighted speech embeddings The following was generated afterward.

[0113]

[0114]

[0115] in and It is a learnable convolutional filter. The resulting modulation parameters γ and β are obtained by analyzing the visual feature map. Image retouching is achieved through scaling and shifting. The edited visual feature map V′ is calculated as follows:

[0116] V′=γ⊙V+β (10)

[0117] The edited visual feature map V′ is then fed into a subsequent decoder to generate the result.

[0118] Therefore, the Photo Request Attention (PRA) process calculates attention weights for embedding language requests and patches on the visual feature map within the image editing tool. The calculated weight matrix indicates the degree of editing for each location. The use of the weight matrix is ​​suitable for cases where requests are both blurry and simple. For underexposed input images with blur requests such as brightening the photo or increasing brightness and contrast, the PRA learns to assign an appropriate degree of editing to the spatial dimensions by increasing the brightness of dark areas and decreasing the brightness of bright areas. Alternatively, for overexposed photos requiring reduced brightness, the PRA can precisely reduce the brightness of light rays and circular halos.

[0119] Figure 8 An example of a PCG network according to aspects of this disclosure is shown. Figure 8 The network depicted in the text is a reference. Figure 6The example described is a PCG network 605, which can be used to generate image patching masks and retouching masks. Examples shown include a text encoder 800 (e.g., Bi-LSTM), an attention network 805, a modular attention network 810 (e.g., MattNet), a convolutional network 815 (e.g., ResNet), and a classifier network 820 (e.g., an MLP network).

[0120] The text encoder 800 can generate word embeddings based on text. Word embeddings are learned representations of text where words with the same meaning have similar representations. BERT, GloVe, and Word2vec are examples of systems for obtaining word vector representations. BERT is a bidirectional encoder representation derived from Transformers. GloVe is an unsupervised algorithm used to train a network using aggregated global word co-occurrence statistics from a corpus. Similarly, the Word2vec model can include shallow neural networks trained to reconstruct the linguistic context of words. GloVe and Word2vec models can take large text corpora and produce vector spaces as outputs. In some cases, the vector space can have many dimensions. Each word in the corpus is assigned a vector in the space. The word vectors are positioned in the vector space such that similar words are located nearby. In some cases, the embedding space can include syntactic or contextual information in addition to semantic information for individual words.

[0121] Attention network 805 can be used to determine which words are relevant to specific aspects of an editing request (e.g., a retouching request or image inpainting request related to a particular image inpainting object). Therefore, in some cases, the image inpainting object is fed into attention network 805 to identify which words correspond to it. In some cases, attention network 805 may output multiple phrases (e.g., multiple retouching phrases, or a retouching phrase and an image inpainting phrase).

[0122] The modular attention network 810 can be used to locate image regions described by natural language expressions. For example, it can generate image inpainting masks corresponding to image inpainting commands, or it can generate retouching masks corresponding to retouching commands.

[0123] The convolutional network 815 can acquire images and output image features, which can be used to generate retouching masks or image patching masks. In some examples, the convolutional network 815 includes a residual network model.

[0124] The classifier network 820 can determine whether a word or phrase corresponds to an image patching command (i.e., a local command) or a retouching command (i.e., a global command). In some cases, a retouching command is used to edit multiple parts of an image, but the edit may not correspond to the entire image, or may not correspond to the entire image equivalently. In some cases, the classifier network 820 is a multilayer perceptron (MLP). An MLP is a feedforward neural network, which is typically composed of multiple perceptrons. Each component perceptron layer may include an input layer, one or more hidden layers, and an output layer. Each node may include a non-linear activation function. An MLP can be trained using backpropagation (i.e., calculating the gradient of the loss function with respect to the parameters).

[0125] reasoning

[0126] Methods for Image Processing Figures 9 to 13 The method is described in detail. One or more aspects of the method include identifying an image and an editing command for the image; encoding the editing command to obtain an image inpaint vector and a retouching vector; generating an image inpaint mask based on the image inpaint vector and a retouching mask based on the retouching vector; generating an image feature representation based on the image and the image inpaint mask; generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vector; and generating a modified image based on the modified image feature representation, wherein the modified image represents the application of the editing command to the image.

[0127] Examples of methods, apparatuses, non-transitory computer-readable media, and systems also include receiving editing commands from a user via voice commands. Some examples also include transcribed editing commands, wherein the encoding is based on the transcribed editing commands.

[0128] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include: generating word embeddings for each of a plurality of words in response to an editing command. Examples also include: generating probability vectors comprising image inpainting probabilities and revision probabilities for each of the plurality of words. Examples further include: applying the image inpainting probabilities to corresponding words in the plurality of words to obtain image inpainting weighted word representations, wherein the image inpainting vector includes image inpainting weighted word representations. Examples also include: applying revision probabilities to corresponding words in the plurality of words to obtain revision weighted word representations, wherein the revision vector includes revision weighted word representations.

[0129] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include: encoding an image to generate image features; identifying image inpainting objects based on editing commands; generating attention weights in image inpainting objects based on editing commands; and generating an image inpainting mask based on image features and attention weights.

[0130] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include determining whether a revision vector indicates a global edit. Some examples also include generating a revision mask based on this determination.

[0131] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include: encoding an image to generate image features; applying an image inpainting mask to the image features to obtain image-inpainted image features; and performing a convolution operation on the image-inpainted image features to obtain an image feature representation.

[0132] Examples of methods, devices, non-transitory computer-readable media, and systems include: computing an attention matrix based on image feature representations and retouching vectors; expanding retouching vectors based on a retouching mask to obtain expanded retouching vectors; and weighting the expanded retouching vectors based on the attention matrix to obtain weighted retouching vectors.

[0133] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include generating scaling and shift parameters based on a weighted revision vector. Some examples also include adding the product of the scaling parameter and an image feature representation to the shift parameter to obtain a modified image feature representation.

[0134] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include performing a convolution operation on a modified image feature representation to obtain a modified image. In some examples, the image patch vector indicates a local edit of the image. In some examples, the patch vector indicates a global edit of the image.

[0135] Figure 9 Examples of image processing according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0136] At operation 905, the system identifies the image and the image editing commands. According to aspects of this disclosure, editing commands may include vague terms and may include actions such as those described in reference... Figures 2 to 4 The intent behind both image retouching and repainting is described. In some cases, this step involves referencing... Figure 1 and Figure 5 The image editing apparatus described, or may be referenced Figure 1 and Figure 5 The image editing device described is used to perform this operation.

[0137] At operation 910, the system encodes the editing command to obtain image inpainting vectors and retouching vectors. For further details on the process used to generate image inpainting and retouching vectors, please refer to [link to relevant documentation]. Figure 6 Text encoder and Figure 10 The flowchart in the document is provided. In some cases, the operation of this step involves referring to... Figure 5 and Figure 6 The text encoder described, or can be found in the reference. Figure 5 and Figure 6 The text encoder described is used for execution.

[0138] At operation 915, the system generates an image inpainting mask based on the image inpainting vector and a patch mask based on the patch vector. For further details on the process used to generate the image inpainting mask, please refer to [link to relevant documentation]. Figure 6 PCG network and Figure 11 The flowchart in the document is provided. In some cases, the operation of this step involves referring to... Figure 5 and Figure 6 The PCG network described, or can be found in the reference Figure 5 and Figure 6 The PCG network described is used for execution.

[0139] At operation 920, the system generates an image feature representation based on the image and the image inpainting mask. Further details regarding the process used to generate the image feature representation can be found in [reference needed]. Figure 6 Image encoder and Figure 12 The flowchart in the document is provided. In some cases, the operation of this step involves referring to... Figure 5 and Figure 6 The image encoder described, or which may be referenced Figure 5 and Figure 6 The image encoder described is used to perform this.

[0140] At operation 925, the system generates a modified image feature representation based on the image feature representation, the retouching mask, and the attention matrix computed using the retouching vector. Further details regarding the process for generating the modified image feature representation can be found in [reference needed]. Figure 7Photo request attention network and multimodal modulation network and Figure 13 The flowchart is provided. In some cases, the operation of this step involves referring to... Figures 5 to 7 The described photo request attention network and multimodal modulation network, or can be derived from the reference... Figures 5 to 7 The described photo request attention network and multimodal modulation network are used to perform this.

[0141] In operation 930, the system generates a modified image based on a modified image feature representation, where the modified image represents the application of editing commands to the image. In some examples, the modified image is derived from a reference image. Figure 6 The described image decoder is generated. In some examples, the decoder includes gated convolutional layers. In some cases, the decoder includes a GAN. In some cases, the operation of this step involves a reference layer. Figure 5 and Figure 6 The decoder described, or may be referenced Figure 5 and Figure 6 The decoder described is used for execution.

[0142] Figure 10 Examples of processes for generating revision vectors based on editing commands according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0143] At operation 1005, the system generates word embeddings for each word in the vocabulary set for the editing command. In some cases, this step involves referencing... Figure 5 and Figure 6 The text encoder described, or which may be referenced Figure 5 and Figure 6 The text encoder described is used for execution.

[0144] At operation 1010, the system generates a probability vector, which includes the image patching probability and the retouching probability for each word in the word set. In some cases, this step involves referencing... Figure 5 and Figure 6 The text encoder described, or which may be referenced Figure 5 and Figure 6 The text encoder described is used for execution.

[0145] At operation 1015, the system applies the image instigation probability to the corresponding word in the word set to obtain a weighted word representation for image instigation, where the image instigation vector includes the weighted word representation for image instigation. In some cases, this step involves referencing... Figure 5 and Figure 6 The text encoder described, or which may be referenced Figure 5 and Figure 6 The text encoder described is used for execution.

[0146] At operation 1020, the system applies the revision probability to the corresponding word in the word set to obtain the revision-weighted word representation, where the revision vector includes the revision-weighted word representation. In some cases, this step involves referencing... Figure 5 and Figure 6 The text encoder described, or which may be referenced Figure 5 and Figure 6 The text encoder described is used for execution.

[0147] Figure 11 Examples of methods for generating image inpainting masks according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the means. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0148] At operation 1105, the system encodes the image to generate image features. In some cases, this step involves referencing... Figure 5 and Figure 6 The PCG network described, or can be found in the reference Figure 5 and Figure 6 The PCG network described is used for execution.

[0149] At operation 1110, the system identifies the image repair objects based on editing commands. In some cases, this step involves referencing... Figure 5 and Figure 6 The PCG network described, or may be referenced Figure 5 and Figure 6 The PCG network described is used for execution.

[0150] At operation 1115, the system generates attention weights in the image inpainting object based on editing commands. In some cases, this step involves referencing... Figure 5 and Figure 6 The PCG network described, or may be referenced Figure 5 and Figure 6The PCG network described is used for execution.

[0151] At operation 1120, the system generates an image inpainting mask based on image features and attention weights. In some cases, this step involves referencing... Figure 5 and Figure 6 The PCG network described, or may be referenced Figure 5 and Figure 6 The PCG network described is used for execution.

[0152] Figure 12 Examples of methods for generating image feature representations according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of a device. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0153] At operation 1205, the system encodes the image to generate image features. In some cases, this step involves referencing... Figure 5 and Figure 6 The image encoder described, or which may be referenced Figure 5 and Figure 6 The image encoder described is used to perform this.

[0154] At operation 1210, the system applies an image inpainting mask to the image features to obtain image inpainted features. In some cases, this step involves referencing... Figure 5 and Figure 6 The image encoder described, or which may be referenced Figure 5 and Figure 6 The image encoder described is used to perform this.

[0155] At operation 1215, the system performs a convolution operation on the image features to obtain an image feature representation. In some cases, this step involves a reference... Figure 5 and Figure 6 The image encoder described, or which may be referenced Figure 5 and Figure 6 The image encoder described is used to perform this.

[0156] Figure 13Examples of methods for modifying image feature representations according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the means. Additionally or alternatively, some processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0157] At operation 1305, the system computes the attention matrix based on the image feature representation and the retouched vector. In some cases, this step involves referencing... Figures 5 to 7 The described PRA network, or can be found in the reference Figures 5 to 7 The described PRA network is used for execution.

[0158] At operation 1310, the system expands the retouching vector based on the retouching mask to obtain the expanded retouching vector. In some cases, this step involves referencing... Figures 5 to 7 The described PRA network, or can be found in the reference Figures 5 to 7 The described PRA network is used for execution.

[0159] At operation 1315, the system weights the expanded revision vector based on the attention matrix to obtain a weighted revision vector. In some cases, this step involves referencing... Figures 5 to 7 The described PRA network, or can be found in the reference Figures 5 to 7 The described PRA network is used for execution.

[0160] At operation 1320, the system generates scaling and shift parameters based on the weighted revision vector. In some cases, this step involves a reference... Figures 5 to 7 The described multimodal modulation network, or can be derived from the reference... Figures 5 to 7 The described multimodal modulation network is used to perform this.

[0161] At operation 1325, the system adds the product of the scaling parameter and the image feature representation to the shift parameter to obtain the modified image feature representation. In some cases, this step involves referencing... Figures 5 to 7 The described multimodal modulation network, or can be derived from the reference... Figures 5 to 7 The described multimodal modulation network is used to perform this.

[0162] In operation 1330, the system generates a modified image based on the modified image feature representation, where the modified image represents the application of editing commands to the image. In some cases, this step involves referencing... Figure 5 and Figure 6 The decoder described, or may be referenced Figure 5 and Figure 6 The decoder described is used for execution.

[0163] Training and Assessment

[0164] Example methods for training neural networks for image processing (reference) Figure 14 and Figure 15 Description. One or more aspects of the method include receiving training data, which includes training images, editing commands, and ground truth images representing the application of editing commands to the training images; encoding the editing commands to obtain image inpaint vectors and patch vectors; generating an image inpaint mask based on the image inpaint vectors and generating a patch mask based on the patch vectors; generating an image feature representation based on the training images and the image inpaint mask; generating a modified image feature representation based on the image feature representation, the patch mask, and an attention matrix computed using the patch vectors; generating a modified image based on the modified image feature representation, wherein the modified image represents the application of editing commands to the training images; computed a loss function based on the modified image and the ground truth images; and training a neural network based on the loss function.

[0165] Examples of methods, apparatuses, non-transitory computer-readable media, and systems include computing an unconditional adversarial loss using a discriminator network based on a modified image and a ground truth image. Some examples also include training a neural network based on the unconditional adversarial loss. In some aspects, the loss function includes L1 loss.

[0166] Figure 14 Examples of methods for training machine learning models according to aspects of this disclosure are shown. In some examples, these operations are performed by a system including a processor that executes a set of code to control functional elements of the means. Additionally or alternatively, certain processes are performed using dedicated hardware. Typically, these operations are performed according to the methods and processes described according to aspects of this disclosure. In some cases, the operations described herein consist of various sub-steps or are performed in combination with other operations.

[0167] exist Figure 14 In the example, the neural network is trained based on a loss function. The term "loss function" refers to how a machine learning model trained in a supervised learning model is affected. Specifically, during each training iteration, the model's output is compared to known annotation information in the training data. The loss function provides a value indicating how close the predicted annotation data is to the actual annotation data. After calculating the loss function, the model's parameters are updated accordingly, and a new prediction set is prepared for the next iteration.

[0168] At operation 1405, the system receives training data, which includes training images, editing commands, and a ground truth image representing the application of the editing commands to the training images. In some cases, this step involves a reference... Figure 5 The training components described, or those that can be found in the reference... Figure 5 The described training components are used to perform this.

[0169] At operation 1410, the system uses the encoder of the image editing device to encode the editing command to obtain an image inpaint vector and a retouching vector. At operation 1415, the system generates an image inpaint mask based on the image inpaint vector and a retouching mask based on the retouching vector. In some cases, this step involves referencing... Figure 5 and Figure 6 The PCG network described, or may be referenced Figure 5 and Figure 6 The PCG network described is used for execution.

[0170] At operation 1420, the system generates an image feature representation based on the training image and the image inpainting mask. In some cases, this step involves a reference... Figure 5 and Figure 6 The image encoder described, or which may be referenced Figure 5 and Figure 6 The image encoder described is used to perform this.

[0171] At operation 1425, the system generates a modified image feature representation based on the image feature representation, the retouching mask, and the attention matrix computed using the retouching vector. In some cases, this step involves referencing... Figures 5 to 7 The described multimodal modulation network, or can be derived from the reference... Figures 5 to 7 The described multimodal modulation network is used to perform this.

[0172] At operation 1430, the system generates a modified image based on the modified image feature representation, where the modified image represents the application of editing commands to the training image. In some cases, this step involves referencing... Figure 5 and Figure 6 The decoder described, or may be referenced Figure 5 and Figure 6 The decoder described is used for execution.

[0173] At operation 1435, the system calculates the loss function based on the modified image and the ground truth image. In some cases, this step involves referencing... Figure 5 The training components described, or those that can be found in the reference... Figure 5 The described training components are used to perform this.

[0174] At operation 1440, the system trains the neural network based on the loss function. In some cases, this step involves a reference... Figure 5 The training components described, or those that can be found in the reference... Figure 5 The described training components are used to perform this.

[0175] In some cases, L1 loss and adversarial loss are used for the network. L1 loss is applied to the output image of an image editing tool. Between the target image y and the output image, the output image is made similar to the target image.

[0176]

[0177] Utilize the following unconditional methods to counteract losses:

[0178]

[0179] The final objective function for generator G and discriminator D is:

[0180]

[0181]

[0182] in and It is used to balance the weights of multiple objectives.

[0183] Common metrics used to evaluate photo editing include L1 distance and user studies. However, since each input request corresponds to only one target image, quantitatively measuring L1 distance may not reflect the quality of the edit, and the results may vary when matching requests. A baseline of the unedited image can be used. However, since the baseline reflects L1 distance, it may not be suitable as a metric. Similarly, metrics used in text-to-image or image manipulation (e.g., manipulation precision (MP)) are also unsuitable because verbal information describes the edit request, i.e., it does not summarize the properties of the target image.

[0184] Figure 15 An example diagram of a training method according to aspects of this disclosure is shown. The example shown includes an input image 1500, an editing request 1505 (i.e., t and The input image 1500 includes a target image 1510, a synthesized image 1515, a loss function 1520, and an evaluation metric 1525. In some examples, the loss function 1520 is calculated between the original input image 1500 and the target image 1510. The synthesized image 1515 can then be generated based on the input image 1500, and the synthesized image 1515 is compared with the target image 1510 using the loss function 1520.

[0185] In some cases, automatic difference detectors (or difference detectors) can also compare images to generate text representing the differences between them. An evaluation metric called the Re-description Similarity Score (RSS) can be used to facilitate training. The RSS can be calculated using a trained difference-speaking method. Given an input image 1500 denoted as x and an image denoted as x... Synthetic image 1515, trained speaker generates edit request Next, the generated request (i.e., the edit request) will be evaluated. The RSS is calculated based on the similarity between the input image 1500 and the ground truth request (i.e., edit request t1505). Therefore, edit requests can be automatically generated based on the differences between the input image 1500 and the target image 1510. 1505. This can be compared to edit request t1505. Edit Request Image 1505 can be generated based on the difference between the synthesized image 1515 and the input image 1500.

[0186] RSS includes transformation metrics (e.g., BLEU-4, CIDEr, METEOR, ROUGE-L) used to assess the similarity between statements. A high RSS value indicates that the generated request is similar to the ground truth request. The maximum RSS value is obtained by generating an edit request using an input image 1500 and a target image 1510. The maximum value of the RSS is obtained by comparing it with the edit request t1505, and this maximum value is used as the upper bound of the RSS.

[0187] In some cases, the Real Image Editing Request (GIER) dataset and the Request Augmentation dataset are used. For example, the GIER dataset consists of 30,000 samples, where each sample is a triplet including a source image, language, target image, and a list of applied actions. In some examples, the Request Augmentation dataset (MIT-Adobe 5K dataset) consists of 25,000 samples, where each sample is an input-target image pair. The Request Augmentation dataset is used for image retouching and does not include speech annotations. An annotation process is followed to collect language requests for each input-target image pair to form the Request Augmentation dataset (i.e., the MIT-Adobe 5K Aug dataset).

[0188] In one implementation, the image is resized to 256×256. For example, an encoder-bottleneck decoder structure consisting of two downsampling blocks, three bottleneck blocks, and two upsampling blocks is used in image editing tools. A multimodal modulation network is employed in the first bottleneck because features in the earlier layers maintain the semantic structure of the image.

[0189] The example embodiments of this disclosure can be compared with baseline methods such as Pix2pixAug, OMN, GeNeVA, TAGAN, and SISGAN. Pix2pixAug is a GAN-based model that follows a speech enhancement pix2pix model, which uses predefined operations for image retouching. OMN is an Operation Module Network composed of sub-modules of predefined global operations. The parameters of each operation are predicted by the modular network. OMN depends on the annotation of the operation. In some cases, the GeNeVA method is used for learning to progressively generate images based on text descriptions, and this method is used for single-step generation. TAGAN and SISGAN are methods for text-guided image manipulation. woPRA means using image editing tools without PRA, such that positions receive the same modulation parameters. Mani-GAN relies on the attention between the attributes of the target image and the words in the text description. The synthesized text features are used in the decoder to generate the output. An L1 loss is added to the baseline model trained on unpaired data.

[0190] In some cases, quantitative comparisons are performed using Re-Description Similarity Scores (RSS) against a baseline on the Real Image Editing Request (GIER) dataset and the Request Augmentation dataset (i.e., the 5K dataset). The RSS score is calculated for an input image x1500 and a ground truth target image x1510. For example, the language requests in the 5K dataset are simple and include retouching operations. User studies are conducted to evaluate the image editing methods and baselines. In some examples, for a given image generated by different methods, users are asked to rank the image by considering both the realistic image and the image matching the language request. Multiple images (e.g., 100 images) are generated for each method, and volunteers are invited to rank the generated images as testers.

[0191] The effectiveness of the PRA can be seen using the RSS value. Furthermore, the image editing tool without PRA still exhibits higher performance compared to conventional image editing systems, further demonstrating the effectiveness of the multimodal modulation network. The image editing tool of this disclosure outperforms a baseline using predefined operations by embedding language requests as modulation parameters. In some cases, the multimodal modulation network can handle requests that may not be represented by predefined operations.

[0192] Considering the value of RSS and user research, the embodiments of this disclosure outperform the baseline model. Additionally, the ranking of user research is similar to that of RSS, demonstrating that evaluation metrics can reflect user opinions. Example embodiments of this disclosure include quantitative comparisons with the baseline on the GIER dataset and the request-enhanced (i.e., 5K) dataset.

[0193] In some cases, when the input image includes abnormal lighting conditions (i.e., the image is overexposed or underexposed), the corresponding edited image has abnormal lighting due to increased brightness in the originally bright areas of the image, thus without using PRA. Using PRA produces natural lighting conditions that are clear and meet the user's requirements because the PRA learns to appropriately determine the degree of editing at each location. Spatial adaptive editing provides high brightness in dark areas and low brightness in the originally bright areas. Alternative methods that rely on predefined operations may not be suitable for requests not represented by predefined operations. In contrast, embodiments of this disclosure directly embed the request into the conditional signal of the GAN, which provides an unconstrained linguistic request.

[0194] Embodiments of this disclosure include image editing tools that outperform image editing methods on the GIER and 5K datasets. Language requests on the GIER dataset consist of image patching and retouching. Retouching results adaptively edited by the image editing tools are clear and match the request. However, baseline methods cannot handle images with unusual lighting conditions (i.e., overexposure or underexposure) and difficult requests (removing objects or changing the image's tone). In contrast, embodiments of this disclosure correctly locate and remove unwanted objects from the image. Similarly, embodiments of this disclosure outperform baseline models on the 5K dataset, which includes editing requests in the form of simple retouching operations. Therefore, results generated by these embodiments are clear and match the user's request.

[0195] Therefore, this disclosure describes a photo editing apparatus that takes an image and a speech request as input and outputs an edited image that matches the speech request. A multimodal modulation network directly embeds the request into modulation parameters as a conditional input signal, thereby achieving an unconstrained request. Furthermore, this model does not use predefined operations. The PRA assigns an appropriate degree of editing for each location to produce a reasonable and spatially adaptive result when the request is blurred.

[0196] In this disclosure and the following claims, the word "or" indicates a list of inclusion, so, for example, a list of X, Y, or Z means X or Y or Z or XY or XZ or YZ or XYZ. Furthermore, the phrase "based on" is not used to indicate a closed set of conditions. For example, a step described as "based on condition A" could be based on both condition A and condition B. In other words, the phrase "based on" should be interpreted as meaning "at least partially based on". Additionally, the words "a" or "an" mean "at least one".

Claims

1. A method of image processing, comprising: identifying an image and an edit command for the image; encoding the edit command to obtain an image repair vector and a polish vector, wherein the image repair vector indicates a local edit to the image and the polish vector indicates a global edit to the image; generating an image repair mask based on the image repair vector and a polish mask based on the polish vector; generating an image feature representation based on the image and the image repair mask; generating a modified image feature representation based on the image feature representation, the polish mask, and an attention matrix computed using the polish vector; and generating a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.

2. The method of claim 1, further comprising: receiving the edit command from a user via an audio input device; transcribing the edit command, wherein the encoding is based on the transcribed edit command.

3. The method of claim 1, further comprising: generating a word embedding for each word of a plurality of words of the edit command; generating a probability vector including an image repair probability and a polish probability for each word of the plurality of words; applying the image repair probability to a corresponding word of the plurality of words to obtain an image repair weighted word representation, wherein the image repair vector includes the image repair weighted word representation; and applying the polish probability to a corresponding word of the plurality of words to obtain a polish weighted word representation, wherein the polish vector includes the polish weighted word representation.

4. The method of claim 1, further comprising: encoding the image to generate an image feature; identifying an image repair object based on the edit command; generating attention weights in the image repair object based on the edit command; and generating the image repair mask based on the image feature and the attention weights.

5. The method of claim 1, further comprising: determining whether the polish vector indicates a global edit; and generating the polish mask based on the determination.

6. The method of claim 1, further comprising: encoding the image to generate an image feature; applying the image repair mask to the image feature to obtain an image repair image feature; and performing a convolution operation on the image repair image feature to obtain the image feature representation.

7. The method of claim 1, further comprising: computing the attention matrix based on the image feature representation and the polish vector; expanding the polish vector based on the polish mask to obtain an expanded polish vector; and weighting the expanded polish vector based on the attention matrix to obtain a weighted polish vector.

8. The method of claim 7, further comprising: generating a scaling parameter and a shifting parameter based on the weighted polish vector; and ​ ​ ​ ​ ​ ​ ​ ​ A product of the scaling parameter and the image feature representation is added to the shift parameter to obtain the modified image feature representation.

9. The method of claim 1, further comprising: performing a convolution operation on the modified image feature representation to obtain the modified image.

10. A method of training a neural network for image processing, comprising: receiving training data, the training data comprising a training image, an edit command, and a ground truth image, the ground truth image representing an application of the edit command to the training image; encoding the edit command to obtain an image inpainting vector and a retouching vector, wherein the image inpainting vector indicates a local edit to the image and the retouching vector indicates a global edit to the image; generating an image inpainting mask based on the image inpainting vector and a retouching mask based on the retouching vector; generating an image feature representation based on the training image and the image inpainting mask; generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vector; generating a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the training image; computing a loss function based on the modified image and the ground truth image; and training a neural network based on the loss function.

11. The method of claim 10, further comprising: computing an unconditional adversarial loss using a discriminator network based on the modified image and the ground truth image; and training the neural network based on the unconditional adversarial loss.

12. The method of claim 10, wherein: the loss function comprises an LI loss.

13. An apparatus for image processing, comprising: an encoder configured to encode an edit command for an image to obtain an image inpainting vector and a retouching vector, wherein the image inpainting vector indicates a local edit to the image and the retouching vector indicates a global edit to the image; a phrase-condition ground truth (PCG) network configured to generate an image inpainting mask based on the image inpainting vector and a retouching mask based on the retouching vector; a convolution network configured to generate an image feature representation based on the image and the image inpainting mask; a multi-modal modulation network configured to generate a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix computed using the retouching vector; and a decoder configured to generate a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.

14. The apparatus of claim 13, further comprising: a photo request attention (PRA) network configured to compute the attention matrix based on the image feature representation and the retouching vector.

15. The apparatus of claim 14, wherein: ​ ​ The PRA network includes an embedding network configured to embed the image feature representation and the finishing vector into a common embedding space.

16. The apparatus of claim 13, further comprising: an audio input device configured to receive the edit command from a user and transcribe the edit command, wherein the encoding is based on the transcribed edit command.

17. The apparatus of claim 13, wherein: the encoder includes a Bi-LSTM.

18. The apparatus of claim 13, wherein: the convolutional network includes a gated convolutional layer.

Images