Method and system for cue-based image editing

Abstract

Some embodiments relate to a method of performing hint-based image editing. The method includes: accessing an image; receiving a selected region of the image; receiving a prompt, wherein the prompt indicates an editing instruction; generating a latent image by transforming the image into visual noise; predicting a noise image corresponding to the latent image based on the latent image and the cue; subtracting at least a portion of the noise image from the latent image to generate an updated latent image; generating a noisy representation of the image; a masked latent image is generated based on the noisy representation, the updated latent image, and a selected region, wherein the masked latent image includes the updated latent image in a region corresponding to the selected region and the noisy representation in a region that does not correspond to the selected region.

Description

Technical Field

[0001] The described embodiments relate to systems, methods, and computer program products for performing prompt-based image editing. In particular, the described embodiments relate to systems, methods, and computer program products for automatically editing digital images based on prompts. Background Art

[0002] Digital image editing processes can be used to produce a wide variety of modifications to digital images. For example, the color properties of the image can be modified, image elements, such as foreground or background objects, can be removed or replaced, or image elements can be added.

[0003] Historically, digital image editing has been performed manually using digital image editing software to manipulate the image. However, this can be an extremely long and tedious task if quality results are required, especially when working on large areas. This is because this method may require pixel-level manipulation of the image to maintain a realistic and seamless result. Some automated methods have been developed, but these often produce unrealistic or undesirable results.

[0004] There is a need to address or ameliorate one or more shortcomings or disadvantages associated with previous systems and methods for performing image editing, or at least provide useful alternatives thereto.

[0005] Throughout this specification, the word "comprise" or variations such as "comprises" or "comprising", will be understood to imply the inclusion of stated elements, integers or steps, or groups of elements, integers or steps, but not the exclusion of any other elements, integers or steps, or groups of elements, integers or steps.

[0006] Any discussion of documents, acts, materials, devices, articles or the like contained in this specification is not to be taken as an admission that any or all of these form part of the prior art base or were common general knowledge in the field relevant to the present disclosure as they existed before the priority date of each of the appended claims. Summary of the invention

[0007] Some embodiments relate to a method of performing hint-based image editing, the method comprising:

[0008] Access images;

[0009] receiving a selected region of the image;

[0010] receiving a prompt, wherein the prompt indicates an editing instruction;

[0011] generating a latent image by transforming the image into visual noise;

[0012] predicting a noise image corresponding to the latent image based on the latent image and the hint;

[0013] subtracting at least a portion of the noise image from the latent image to generate an updated latent image;

[0014] generating a noisy representation of the image;

[0015] A masked latent image is generated based on the noisy representation, the updated latent image, and the selected region, wherein the masked latent image includes the updated latent image in a region corresponding to the selected region and the noisy representation in a region not corresponding to the selected region.

[0016] Some embodiments further comprise the step of repeating a denoising loop over a series of time steps, wherein the denoising loop comprises the steps of predicting a noisy image, generating an updated latent image, generating a noisy representation, and generating a masked latent image.

[0017] According to some embodiments, predicting the noisy image, given a current time step t, comprises predicting the visual noise that will be present in the latent image at time step t+1 during the noisy process.

[0018] In some embodiments, generating the noisy representation of the first image comprises adding noise to the image based on the current time step t.

[0019] In some embodiments, the time step t is decremented after each iteration.

[0020] In some embodiments, the denoising cycle is repeated 15 to 40 times.

[0021] In some embodiments, the denoising cycle is repeated 20 to 30 times.

[0022] According to some embodiments, the portion of the noise image to be subtracted from the latent image is determined based on a noise schedule.

[0023] In some embodiments, said step of predicting said noise image is performed by a diffusion model.

[0024] According to some embodiments, the diffusion model is trained using a dataset comprising a first image, an editing instruction, and a second image edited based on the editing instruction.

[0025] According to some embodiments, the diffusion model is trained by causing the diffusion model to add noise to the first image with increasing time steps t, and subsequently causing the diffusion model to remove noise by decreasing time steps t to generate a second edited image.

[0026] Some embodiments further comprise determining an encoding of the received hint, wherein said step of predicting the noisy image is performed using said encoding of said hint.

[0027] Some embodiments further include cropping the accessed image based on the selected area.

[0028] Some embodiments further comprise changing a resolution of the accessed image to a working resolution prior to generating the latent image, wherein the working resolution is a lower resolution than the resolution of the accessed image.

[0029] Some embodiments further include mapping pixel information corresponding to the accessed image to a latent image space prior to generating the latent image, such that the latent image is a representation of the image in the latent image space.

[0030] Some embodiments further include inserting the generated masked latent image into the accessed image to produce an output image.

[0031] Some embodiments further include performing a pixel blending process on the generated masked latent image to blend the generated masked latent image with the accessed image.

[0032] Some embodiments further include outputting the image by saving the output image to a memory location, displaying the output image to a user, or sending the output image to an external device.

[0033] Some embodiments further include generating a binary mask based on the selected area, and using the binary mask to generate the masked latent image.

[0034] In some embodiments, the masked latent image is calculated based on the following equation: masked_latent=mask*latent+(1-mask)*noisy_representation,

[0035] Wherein mask represents a mask generated based on the selected area, latent represents the updated latent image, and noisy_representation represents the noisy representation.

[0036] Some embodiments further include receiving an indication that the prompt involves a local edit.

[0037] In some embodiments, the first visual noise is Gaussian noise.

[0038] Some embodiments relate to a non-transitory computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform the method of some other embodiments.

[0039] Some embodiments relate to a computing device comprising:

[0040] The non-transitory computer-readable storage medium of some other embodiments; and

[0041] A processor configured to execute the instructions stored in the non-transitory computer-readable storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The various figures in the drawings depict only example embodiments of the present disclosure and are not to be considered limiting of its scope.

[0043] Figures 1A to 1D Example images showing results of image editing based on global cues according to some known methods are shown;

[0044] Figures 2A to 2C A first series of example images is shown, the first series of example images showing results of local cue-based image editing according to some known methods;

[0045] Figures 3A to 3C showing a second series of example images according to some known methods, the second series of example images showing results of local cue-based image editing;

[0046] Figure 4 is a block diagram of a system for performing hint-based image editing according to some embodiments;

[0047] Figure 5 According to some embodiments, the use Figure 4 A process flow chart of a method for performing hint-based image editing by a system;

[0048] Figures 6A to 6D is a set of example schemata showing the results of local cue-based image editing according to some known methods;

[0049] Figures 7A to 7K is performed according to some embodiments Figure 5 Example diagrams of steps and results of the method;

[0050] Figures 8A to 8D is performed according to some embodiments Figure 5 a first set of example images of results of the method;

[0051] Figures 9A to 9D is performed according to some embodiments Figure 5 a second set of example images of results of the method;

[0052] Figures 10A to 10D is an example image illustrating the fusion process according to some embodiments; and

[0053] Fig.11 According to some embodiments, the use Figure 4 A process flow chart of a method for training a machine learning model to perform prompt-based image editing in a system; DETAILED DESCRIPTION

[0054] The described embodiments relate to systems, methods, and computer program products for performing prompt-based image editing. In particular, the described embodiments relate to systems, methods, and computer program products for automatically editing digital images based on prompts.

[0055] Prompt-based image editing refers to automatically editing an image based on an input prompt. Manual techniques for editing images can be time-consuming and often require high skill to produce results that look convincing. Existing prompt-based image editing techniques can be used to perform some image editing processes, such as automatic image repair processes that can be used to insert or remove image elements. However, some prompt-based image editing techniques can produce unrealistic or undesirable results. This is especially true when the supplied prompt is only for a portion of the image to be edited (e.g., a single object within the image). For example, when a prompt-based editing technique is used to attempt to edit the color of a discrete image element, the result will typically change the color of the entire image or image elements outside the element to be edited.

[0056] Some of the described embodiments provide a hint-based image editing technique that enables editing of specific image elements in a more realistic manner than some previously known image editing methods. Specifically, some embodiments provide a hint-based image editing technique that better limits the editing of an image to specific image elements than some previously known image editing techniques. This is achieved by introducing a masking step during a denoising process performed by a model trained to perform hint-based image editing.

[0057] In the following description, the term "pixel information" as it relates to an image may include any information indicating, associated with, derived / derivable from, included in, and / or consisting of information indicating pixels included in the image. For example, the pixel information may be the red, green, and blue (RGB) values ​​of the sub-pixels that constitute a single pixel. In some embodiments, the pixel information may include a numerical representation and / or transformation of the RGB values ​​of the sub-pixels, such as a latent image mapping of the RGB values ​​of the sub-pixels mapped to a lower dimensional latent image space. In some embodiments, the pixel information may include any other type of information or information representation indicating the RGB values ​​of pixels and / or sub-pixels.

[0058] Figures 1A to 1D Examples of results from some previously known hint-based image editing techniques are shown.

[0059] Figure 1A An example original image 100 is shown including a background 105 and an object, the object being the Eiffel Tower 110. The background 105 shows a cloudy blue sky.

[0060] Figure 1B An example edited image 120 is shown. Image 120 is an edited version of image 100 and is generated using a hint-based image editing technique performed by an image editing application according to some known techniques. For example, image 120 may have been generated using a diffusion machine learning (ML) model, which is a neural network model trained or otherwise configured to denoise an image containing Gaussian noise by learning to reverse the diffusion process. Image 120 may have been generated using the techniques described in Brooks, T., Holynski, A., and Efros, AA, 2023; “Instructpix2pix: Learning to follow image editing instructions,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 18392-18402) (“Brooks et al.”), the contents of which are incorporated herein by reference in their entirety.

[0061] Image 120 is generated by feeding a suitably trained ML model with the original image 100 and a text prompt “Add fireworks to the sky.” The result of the editing is that image 120 now has an edited background 125 showing a dark night sky filled with fireworks, and an edited Eiffel Tower 130 that has changed color to match the lighting of the edited background 125.

[0062] Figure 1C An example original image 140 is shown including a background 145 and an object, which is a man 150. The background 145 shows a blue storefront, and the man 150 is sitting on a stand in front of the storefront. The man is wearing a white shirt and sleeves with a dark vest and pants.

[0063] Figure 1D An example edited image 160 is shown. Image 160 is an edited version of image 170 and is generated using a hint-based image editing technique performed by an image editing application according to some known techniques. For example, image 160 may have been generated using a diffusion machine learning (ML) model, which is a neural network model trained or otherwise configured to denoise an image containing Gaussian noise by learning to reverse the diffusion process. Image 120 may have been generated using the techniques described by Brooks et al.

[0064] Image 160 is generated by feeding a suitably trained ML model with the original image 140 and the text prompt "What would it look like if it snowed?" The result of the editing is that image 160 now has an edited background 165 showing snow falling in front of the storefront and accumulating on the shelves, and an edited man 160 who is now wearing a hat and scarf.

[0065] Figure 1B and 1D The edits shown in have been applied to the entire image and produce a relatively realistic edited image.

[0066] Figures 2A to 2C 3A to 3C illustrate examples of unrealistic and undesirable appearance of edited images that may occur when using prior art techniques for cue-based image editing, where the cues involve discrete image elements.

[0067] Figure 2AAn example original image 200 is shown that includes a background 205 and an object, which is a kayak 215. In the example shown, the kayak 215 is placed on a concrete platform 210. The kayak 215 has a red exterior, a gray interior, and some black details. The background 205 shows a body of water with green bushes or trees in the distance, and a portion of the sky visible above the horizon. The image 200 may be an image selected or uploaded by a user for editing using an image editing application.

[0068] Figure 2B User interface element 230 is shown. User interface element 230 may be displayed by a device executing an image editing application, such as an application used by a user to edit image 200. User interface element 230 includes a text entry field 235 for allowing a user to enter a text prompt. In some embodiments, text entry field 235 may appear when a particular editing tool is selected. In some embodiments, text entry field 235 may be a permanent component of the user interface of the image editing application. In the illustrated embodiment, the user has entered prompt 240. Prompt 240 displays "Make the boat green."

[0069] Figure 2C An example edited image 260 is shown. Image 260 is an edited version of image 200 and is generated using hint 240 using a hint-based image editing technique performed by an image editing application according to some known techniques. For example, image 260 may have been generated using a diffusion machine learning (ML) model, which is a neural network model trained or otherwise configured to denoise an image containing Gaussian noise by learning to reverse the diffusion process. Image 260 may have been generated using the techniques described by Brooks et al.

[0070] However, editing image 200 to generate image 260 has produced some undesirable results. Image 260 shows background 265 and an object, which is kayak 275. Kayak 275 is placed on platform 270. According to prompt 240, kayak 275 has become green compared to kayak 215. However, the remaining elements of image 260 have also been changed. Background 265 and platform 270 have become significantly greener compared to background 205 and platform 210. In other words, the edits made to kayak 215 / 275 were not contained to the image elements, but were applied to the entire image 200 / 260.

[0071] Such a result may occur when the hint 240 is a local hint that specifies a particular location (e.g., a particular object, thing, or image element) to which an edit is to be applied. This may be particularly evident where the hint involves a change to the color of the specified location, object, thing, or element. Image elements other than those referenced in the hint may be undesirably edited, which may include edits that change the color, brightness, contrast, saturation, and / or other visual characteristics of the image elements.

[0072] Figures 3A to 3C Another such example is shown.

[0073] Figure 3A An example original image 300 is shown including a background 305 and an object, which is an apple 315. In the example shown, the apple is red and is placed on a white lace tablecloth forming the background 305. Other food items such as sushki 310 are placed on the tablecloth. The image 300 may be an image selected or uploaded by a user for editing using an image editing application.

[0074] Figure 3B User interface element 330 is shown. User interface element 330 may be displayed by a device executing an image editing application, such as an application used by a user to edit image 300. The user interface element includes a text entry field 335 for allowing a user to enter a text prompt. In some embodiments, text entry field 335 may appear when a particular editing tool is selected. In some embodiments, text entry field 335 may be a permanent component of the user interface of the image editing application. In the illustrated embodiment, the user has entered prompt 340. Prompt 340 displays "Make the apple gold."

[0075] Figure 3C An example edited image 360 ​​is shown. Image 360 ​​is an edited version of image 300 and is generated using a hint-based image editing technique performed by an image editing application according to some known techniques using hint 340. For example, image 360 ​​may have been generated using a diffusion machine learning (ML) model, which is a neural network model trained or otherwise configured to denoise an image containing Gaussian noise by learning to reverse the diffusion process. Image 360 ​​may have been generated using the techniques described by Brooks et al.

[0076] However, editing image 300 to generate image 360 ​​has produced some undesirable results. Image 360 ​​shows background 365 and an object, which is an apple 375. Apple 375, along with other food products such as sushki 370, are placed on a tablecloth that forms background 365. According to prompt 340, parts of apple 375 have been edited to be a more golden color compared to apple 315. However, the remaining elements of image 360 ​​have also been changed. Compared to background 305 and sushki 310, background 365, sushki 370 and other food products have become significantly more yellow. In other words, the edits made to apple 315 / 375 are not included in the image elements, but have been applied to the entire image 300 / 360.

[0077] In some embodiments, the following reference Figures 4 to 11 The described systems and methods illustrate the possibility of avoiding the above reference Figures 2A to 3C Some disadvantages of the method are described.

[0078] Figure 4 The arrangement of system components is shown, including hardware and software of the system that can be used to perform the method disclosed herein. It will be easily understood by those skilled in the art that Figure 4 The system is only one example of many potential embodiments suitable for performing the method of the present invention.

[0079] Figure 4 4 is a block diagram illustrating an example system 400 that can be used to perform prompt-based editing, according to some described embodiments. The system 400 includes a user computing device 410 that can be controlled by a user who wishes to edit one or more images. In the illustrated embodiment, the system 400 further includes a server system 420. The user computing device 410 can communicate with the server system 420 via a network 430. However, in some embodiments, the user computing device 410 can be configured to independently perform the described methods without access to the network 430 or the server system 420.

[0080] For example, the user computing device 410 may be a computing device such as a personal computer, a laptop computer, a desktop computer, a tablet, or a smart phone. The user computing device 410 includes a processor 411, which is configured to read and execute program code. The processor 411 may include one or more data processors for executing instructions, and may include one or more of the following: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs).

[0081] The user computing device 410 further includes at least one memory 412. The memory 412 may include one or more memory storage locations, which may include volatile memory and non-volatile memory, and may be in the form of ROM, RAM, flash memory, or other memory types. The memory 412 may also include system memory, such as BIOS.

[0082] The memory 412 is arranged to be accessible by the processor 411 and stores data that can be read by and written to the processor 411. The memory 412 may also include program code 414 that can be executed by the processor 411 to cause the processor 411 to perform various functions. For example, the program code 414 may include an image editing application 415. The processor 421 executing the image editing application 415 may be caused to perform various aspects of the hint-based image editing method, as described below with reference to Figures 5 to 11 Described in further detail.

[0083] According to some embodiments, the image editing application 415 may be a web browser application (e.g., Chrome, Safari, Internet Explorer, Opera, or any other alternative web browser application) that may be configured to access a web page that provides image editing functionality via an appropriate uniform resource locator (URL).

[0084] Program code 414 may include Figure 4 Additional applications not shown, such as an operating system application, which may be a mobile operating system if the user computing device 410 is a mobile device, a desktop operating system if the user computing device 410 is a desktop device, or an alternative operating system.

[0085] The user computing device 410 may further include user input and output peripherals 416. For example, these user input and output peripherals may include one or more of a display screen, a touch screen display, a mouse, a keyboard, a speaker, a microphone, and a camera. The user I / O 416 may be used to receive data and instructions from a user and to communicate information to the user.

[0086] The user computing device 410 may further include a communication interface 417 to facilitate communication between the user computing device 410 and other remote or external devices. The communication module 417 may allow wired or wireless communication between the user computing device 410 and the external device, and may use Wi-Fi, USB, Bluetooth, or other communication protocols. According to some embodiments, for example, the communication module 417 may facilitate communication between the user computing device 410 and the server system 420 via the network 430.

[0087] The network 430 may include one or more local or wide area networks that facilitate communication between elements of the system 400. For example, according to some embodiments, the network 430 may be the Internet. However, the network 430 may include at least a portion of any one or more networks having one or more nodes that transmit, receive, forward, generate, buffer, store, route, switch, process one or more messages, packets, signals, some combination thereof, or a combination of these operations, etc. The network 430 may include, for example, one or more of the following: a wireless network, a wired network, the Internet, an intranet, a public network, a packet-switched network, a circuit-switched network, an ad hoc network, an infrastructure network, a public switched telephone network (PSTN), a cable network, a cellular network, a satellite network, a fiber optic network, or some combination thereof.

[0088] The server system 420 may include one or more computing devices and / or server devices (not shown), such as one or more servers, databases, and / or processing devices that communicate over a network with a computing and / or server device that hosts one or more applications, libraries, APIs, or other software elements. The components of the server system 420 may provide server-side functionality to one or more client applications, such as the image editing application 415. The server-side functionality may include operations such as user account management, login, and content creation functions such as image editing, saving, publishing, and sharing functions. According to some embodiments, the server system 420 may include a cloud-based server system. Although a single server system 420 is shown, the server system 420 may include multiple systems of servers, databases, and / or processing devices. According to some described embodiments, the server system 420 may host one or more components of a platform for performing image editing.

[0089] Server system 420 may include at least one processor 421 and memory 422. Processor 421 may include one or more data processors for executing instructions, and may include one or more of the following: a microprocessor, a microcontroller-based platform, a suitable integrated circuit, and one or more application-specific integrated circuits (ASICs). Memory 422 may include one or more memory storage locations, and may be in the form of ROM, RAM, flash memory, or other memory types.

[0090] Memory 422 is arranged to be accessible by processor 421 and contains data 423 that processor 421 is configured to read and write. Data 423 may store data such as user account data, image data, and data related to image editing tools, such as machine learning models trained to perform image editing functions.

[0091] In the illustrated embodiment, data 423 includes image data 430, prompt data 431, and user-selected area data 432. While this data is shown as residing in memory 422 of server system 420, in some embodiments, some or all of this data may alternatively or additionally reside in memory 412 of user computing device 410, or in the alternative in a local or remote memory location.

[0092] Image data 430 may store image data related to an image to be edited by image editing application 415. Image data 430 may be received from user computing device 410 executing image editing application 415 in response to a user selecting or uploading an image to be edited. Figure 2A and 3A , images 200 and / or 300 may be stored in image data 430. Image data 430 may additionally or alternatively store image data related to an image that is in the process of being edited or a final edited image.

[0093] In response to a user entering a prompt while executing the image editing application 415, prompt data 231 may be received from the user computing device 410 to perform an image editing function. Figure 2A In the example shown in , the user enters prompt 240 “Make the boat green.” Prompt 240 may be stored in prompt data 228 as a text string.

[0094] In response to a user selecting an area of ​​an image while executing the image editing application 415, user selected area data 432 may be received from the user computing device 410 to perform image editing functions in the area. According to some embodiments, a user wishing to edit an image may interact with the user I / O 416 of the user computing device 410 displaying the image to indicate which area of ​​the image they wish to edit. For example, a user may use a brush tool, a tracing tool, or other tool to digitally select, trace, circle, or "paint" over image elements via a user interface to produce a user selected area, which may be used as a mask to perform editing methods, as described in further detail below. The user selected area may be stored as user selected area data 432.

[0095] In some cases, the user-selected area may be stored as image data representing a mask defined by the user-selected area to user-selected area data 432. For example, a binary mask may be generated in which a first pixel value may represent an area of ​​the image that falls within the user-selected area, and a second pixel value different from the first pixel value may be used to represent an area that falls outside the user-selected area. In some embodiments, white pixels may represent the user-selected area, i.e., an area selected for editing, and black pixels may represent areas outside the user-selected area, where editing is not performed.

[0096] exist Figure 6C , 7B Examples of user-selected areas are shown in , 8B, 9B, and 10B and are described in further detail below with reference to those images.

[0097] Memory 422 further includes program code 424, which can be executed by processor 421 to cause processor 421 to perform a workflow. For example, program code 424 includes server application 433, which can be executed by processor 421 to cause server system 420 to perform server-side functions. According to some embodiments, for example, in the case where image editing application 415 is a web browser, server application 433 may include a web server, such as Apache, IIS, NGINX, GWS, or an alternative web server. In some embodiments, server application 433 may include an application server specifically configured to interact with image editing application 415. Server system 420 may have both a web server and an application server module.

[0098] Program code 424 may also include one or more code modules, such as one or more of an encoding module 434 , a cropping module 435 , a resolution adjustment module 436 , a visual noise module 437 , a denoising module 438 , and an image combination module 439 .

[0099] As described in further detail below with reference to step 540 of method 500, executing encoding module 434 may cause processor 421 to perform tokenization, encoding, and / or embedding processes on input from a user, which in some embodiments may be a prompt. According to some embodiments, processor 421 executing encoding module 434 may be caused to generate a prompt representation based on the user input. For example, this may be accomplished by determining a lower dimensional representation of the input that may be interpreted by a machine trained model for generating an image. A text encoder such as OpenAI's CLIP text encoder may be used to determine the prompt representation. For example, the prompt representation may be an encoding or embedding of the prompt stored in prompt data 431.

[0100] Executing the cropping module 435 may cause the processor 421 to perform a cropping process on the supplied image, as described in further detail below with reference to step 530 of the method 500. Specifically, the cropping module 435 may be configured to apply a cropping process to the image retrieved from the image data 430 based on the user-selected region retrieved from the user-selected region data 432. Some image editing methods may be performed on the cropped section of the original image, such as described below with reference to Figures 5 to 11 Cropping an image prior to editing may result in a higher quality edited image, as described in further detail below. In the event that the user-selected area retrieved from the user-selected area data 432 is significantly smaller than the size of the image to be edited retrieved from the image data 430, the image to be edited may be cropped to a smaller size, wherein the cropped image is larger than and includes the entire user-selected area. Figure 7C and 7D Examples of cropping an image based on a user-selected region are shown in and described in further detail below with reference to those images.

[0101] Executing the resolution adjustment module 436 may cause the processor 421 to adjust the resolution of the supplied image, as described in further detail below with reference to steps 535 and 580 of the method 500. This may be used to convert an original image, such as an image retrieved from the image data 430, to a working resolution, or to convert a working resolution image back to the original image resolution. For example, in some embodiments, the working resolution may be between 448×448 pixels and 640×640 pixels. In the event that the supplied image is not a square, the resolution adjustment module 436 may be configured to adjust the resolution so that the working resolution image has the same number of pixels as a 448×448 to 640×640 pixel square. According to some embodiments, the resolution adjustment module 436 may be configured to adjust the resolution of the supplied image so that the number of pixels in the working resolution image is a multiple of 32. The working resolution image or a lower compression representation of the image produced by the resolution adjustment module 436 may be referred to as a latent image of the image, a latent image representation of the image, or a latent image map of the image. The latent image space represents data in which the relationship between different data points is more easily understood and analyzed. The resolution adjustment module 436 may act as an encoder to compute a lower dimensional compressed representation of the served image, and as a decoder to convert the lower dimensional compressed representation back into the pixel space of the original served image.

[0102] Executing the visual noise module 437 may cause the processor 421 to add randomly generated signal noise to the pixel information of the supplied image or the supplied latent image of the image, as described in further detail below with reference to step 545 of method 500. The signal noise may be Gaussian noise. In some embodiments, the visual noise module 437 may be configured to add a certain level, degree, or amount of visual noise to the pixel information of the supplied image or the supplied latent image of the image based on a noise intensity parameter. Fig. 7E An example illustrating a process of adding noise to an image is described in further detail.

[0103] Executing the denoising module 438 may cause the processor 421 to perform an automated denoising process on the supplied image in order to perform the editing process. This is described in further detail below with reference to steps 550 to 575 of method 500. The denoising module 438 may include or access a machine learning (ML) model to perform denoising, and in some embodiments, the machine learning model may be a diffusion ML model. The diffusion ML model may include a neural network model that is trained or otherwise configured to denoise an image containing noise, such as Gaussian noise, by learning to reverse the diffusion process. Specifically, the diffusion ML model can be trained by adding noise to the image during a forward process until the image consists of pure noise. Noise can be added to the image over multiple time steps. In other words, the model can be trained by causing the model to corrupt the training data image by continuously adding noise, and then causing the model to recover the data by reversing the denoising process. This process is described below with reference to Fig.11 Once trained, the ML model can be used to generate image data by following a backward or inverse process in which the ML model starts with an image composed of noise and subtracts the noise from the image to generate the image data.

[0104] In some embodiments, the denoising process can be performed using a U-net. U-net is a fully convolutional neural network and consists of a contracting path and an expanding path. The contracting path is a typical convolutional network that consists of repeatedly applying convolutions, each followed by a rectified linear unit (ReLU) and a maximum pooling operation. During contraction, spatial information is reduced, while feature information is increased. The expanding path combines feature and spatial information with high-resolution features from the contracting path through a series of up-convolutions and concatenations. In other words, the image is converted to a vector, but then converted back from the vector to an image.

[0105] As described in further detail below with reference to steps 550 to 575 of method 500, the denoising process may include inferring, by the U-net, a first estimate of the image to be generated from the generated visual noise based on the initial visual noise and the representation of the prompt. The first estimate may then be subtracted from the initial visual noise to generate a subsequent visual noise, which is then fed back into the U-net along with the representation of the prompt to generate a second estimate of the image to be generated. For example, this process may be iteratively repeated until a termination criterion is met, such as a decrementing time step counter reaching zero or a predetermined amount of time having passed.

[0106] The ML model may be an ML model trained using the techniques described by Brooks et al. and as referenced below Fig.11 Described in further detail.

[0107] The ML model may guide the diffusion process using the hint representation generated by the encoding module 434. Specifically, the ML model may use the hint representation to manipulate an image representation containing visual noise in such a way that the noise is transformed into an image corresponding to the hint representation.

[0108] In some embodiments, the ML model may be a latent diffusion ML model, which is a diffusion ML model configured to perform a denoising process on a mapping of an image to a lower dimensional latent image space, as described above with reference to the resolution adjustment module 436. According to some embodiments, the ML model may be a stable diffusion ML model.

[0109] In some embodiments, for example, the ML model may be a pre-trained and commercially available, open-use, or open-source ML model, such as Open AI's DALL-E / DALL-E 2, Google's Imagen, and / or a stable proliferation of CompVis.

[0110] Executing the image combining module 439 may cause the processor 421 to combine two or more images together. In some embodiments, the processor 421 may be caused to combine the images together based on a mask, which may be generated based on the user-selected region data 432. According to some embodiments, executing the image blending module 439 may cause the processor 421 to perform a fusion process on the pixel information of the images to be combined so that the images blend together naturally and / or substantially unnoticeably. According to some embodiments, this may include performing a pixel adjustment process, as described below with reference to Figures 10A to 10D Described in further detail.

[0111] The encoding module 434, cropping module 435, resolution adjustment module 436, visual noise module 437, denoising module 438, and image combination module 439 may be software modules, such as add-ons or plug-ins, that operate in conjunction with the image editing application 415 to extend its functionality. In alternative embodiments, the modules 434, 435, 436, 437, 438, and / or 439 may be native to the image editing application 415. In yet further alternative embodiments, the modules 434, 435, 436, 437, 438, and / or 439 may be stand-alone applications (running on the user computing device 410, the server system 420, or an alternative server system (not shown)) that communicate with the image editing application 415, such as over the network 430.

[0112] Modules 434, 435, 436, 437, 438, and / or 439 have been described and shown as part of / installed on server system 420, and may be configured as an add-on or extension to server application 433, a separate stand-alone server application in communication with server application 433, or a native portion of server application 433. Inputs such as user interactions, user prompts, and / or images of elements to be edited may be provided and / or received at / through user computing device 410 and subsequently communicated to server system 420 so that prompt-based editing methods may be performed by components of server system 420.

[0113] In some alternative embodiments (not shown), the functionality provided by one or more of modules 434, 435, 436, 437, 438, and / or 439 may alternatively be provided by user computing device 410 based on locally or remotely stored image data 430, hint data 431, and user-selected region data 432. One or more of modules 434, 435, 436, 437, 438, and / or 439 may reside as an add-on or extension to image editing application 415, a separate stand-alone application that communicates with image editing application 415, or a native portion of image editing application 415.

[0114] In an alternative embodiment (not shown), all functions including receiving hints, user-selected areas, and images may be performed by server system 420. Alternatively, in some embodiments, an application programming interface (API) may be used to interface with server system 420 for performing the presently disclosed hint-based image editing techniques.

[0115] The server system 420 may also include a communication interface 427 to facilitate communication between the server system 420 and other remote or external devices. The communication module 427 may allow wired or wireless communication between the server system 420 and the external device, and may use Wi-Fi, USB, Bluetooth, or other communication protocols. According to some embodiments, for example, the communication module 427 may facilitate communication between the server system 420 and the user computing device 410.

[0116] The server system 420 may include additional functional components beyond those shown and described, such as one or more firewalls (and / or other network security components), load balancers (for managing access to the server applications 433), and / or other components.

[0117] Figure 5 is a process flow diagram of a method 500 for performing hint-based image editing according to some embodiments. In some embodiments, the method 500 may be performed at least in part by a processor 411 executing an image editing application 415. In some embodiments, the method 500 may be performed at least in part by a processor 421 executing a server application 433. Although certain steps of the method 500 have been described as being performed by specific elements of the system 400, in some embodiments, these steps may be performed by different elements. In addition, although the steps of the method 500 have been shown and described as occurring in a particular order, some of the steps may be performed in an alternative order without affecting the results of the method.

[0118] At step 505, processor 421 executing server application 433 accesses an image for editing. This image will be referred to as the "original image". In some embodiments, the original image may be an image selected by a user. In some embodiments, access may be from a memory location, from user I / O, or from an external device. For example, in some embodiments, processor 421 may access the original image from image data 430.

[0119] In some embodiments, the original image may be sent from the user computing device 410 to the server system 420. This may be in response to the user of the user computing device 410 capturing an image for editing using a camera forming part of the user I / O 416, or by the user selecting an image from a memory location. The memory location may be in memory 412 stored locally on the user computing device 410, or in data 423 stored remotely in memory 422 in the server system 420. Depending on where the image editing process is to be performed, a copy of the original image may be stored to a second memory location to allow efficient access to the image file by the processor 411 and / or processor 421. For example, a copy of the original image may be stored in image data 430 in memory 422 for access by the processor 421.

[0120] According to some embodiments, the original image may be displayed within a user interface of image editing application 415 , which may be displayed on a display screen (not shown) forming part of user I / O 416 .

[0121] As described above Figure 1A , 1C , 2A and / or 3A and as described in further detail below Fig. 6A , 8A Example images that may be received at step 505 are shown in FIGS. 9A and / or 10A.

[0122] At step 510, the processor 421 executing the server application 433 determines whether the user has selected to perform local editing of the original image. In other words, the processor 421 determines whether the user attempts to edit the original image received at step 505 as a whole, or attempts to edit a specific area, object, element or part of the original image. In certain embodiments, this can be determined by receiving a user input indicating whether editing is a local editing from the user I / O 416. The image editing application 415 can be made to present user interface elements to allow the user to provide such instructions. The user interface element can be a check box, a radio button, a drop-down box, a virtual button or other user interface elements that allow the user to select or type in data to indicate whether they wish to perform local editing of the original image received at step 505.

[0123] In some other embodiments, partial editing may be the default option and processor 421 may proceed to step 520 without making any determination at step 510 .

[0124] In some embodiments, step 510 may be performed after either or both of steps 520 and 525, and the data received from steps 520 and / or 525 may be used by processor 421 to determine whether local editing is required. This will be described below with reference to each of these steps.

[0125] If the processor 421 determines at step 510 that the user does not need or want local editing, the processor 421 executing the server application 433 may continue to perform non-masking editing techniques to perform any edits to the original image received at step 505. For example, the processor 421 may continue to perform any edits using the automated editing techniques described by Brooks et al.

[0126] If the processor 421 determines at step 510 that a local edit is needed, the processor 421 proceeds to step 520. At step 520, the processor 421 executing the server application 433 receives a user input corresponding to a prompt, the prompt relating to the edits that the user wishes to perform on the original image accessed at step 505. The prompt may indicate or include the edit instructions. In some embodiments, the prompt may be a text prompt, an audio recording, a selection from a list, or any other suitable type of prompt. When the prompt is a text prompt, a text input field such as a text box may be used to type the prompt. When the prompt is an audio recording, the audio recording may be in the form of .MP3 or .WAV or any other suitable audio file format.

[0127] In the case where step 510 is performed after step 520, processor 421 may use the content of the prompt received at step 520 to determine whether the user wants to perform a local edit or a global edit on the original image. For example, processor 421 may determine whether the prompt includes a reference to a specific area, object, element, or portion of the original image received at step 505.

[0128] At step 525, processor 421 executing server application 433 receives user input corresponding to a selection of an area of ​​the original image that they would like to edit, which is referred to as a selected area or user-selected area. According to some embodiments, a user who wishes to edit the original image may interact with user I / O 416 of computing device 410 displaying the original image to indicate which area of ​​the original image they wish to edit. For example, a user may use a brush tool, a tracing tool, or other tool to digitally select, trace, circle, or "draw" over an element to be edited via user I / O 416 to produce a user-selected area that may be used to perform an editing method, as described in further detail below with reference to steps 530, 565, and 585. The user-selected area may be stored in user-selected area data 432.

[0129] In some cases, the user-selected area may be used to generate image data representing a mask defined by the user-selected area. For example, a binary mask may be generated in which a first pixel value may represent an area of ​​the image that falls within the user-selected area, and a second pixel value different from the first pixel value may be used to represent an area that falls outside the user-selected area. In some embodiments, white pixels may represent the user-selected area, i.e., an area selected for editing, and black pixels may represent an area outside the user-selected area, where no editing is performed. The generated image data representing the mask may be stored to user-selected area data 432.

[0130] exist Figure 6C , 7B , 8B, 9B and 10B show some example images showing the user-selected area that may be received at step 525, as described in further detail below.

[0131] In the case where step 510 is performed after step 525, the processor 421 may use the user-selected area received at step 525 to determine whether the user wants to perform a local edit or a global edit on the original image. For example, the processor 421 may determine whether the user-selected area only includes a specific area, object, element, or portion of the original image received at step 505, or whether the user area substantially includes the entire area of ​​the original image received at step 505.

[0132] In some embodiments, steps 520 and 525 may be performed in reverse order, where the user-selected area is received prior to the prompting.

[0133] At optional step 530, processor 421 executing cropping module 435 may crop the image accessed at step 515 to generate a cropped image. The cropped image may be stored to image data 430 for further processing. Performing the cropping step may improve the quality of the resulting edited image. The cropped area may be rectangular. According to some embodiments, the border of the cropped area may be parallel to the border of the accessed image. The cropped area may include a user-selected area and at least one unselected area. For example, in the case where the cropped area is a rectangle, the unselected area may be one or more areas defined by the border of the user-selected area and the border of the rectangle.

[0134] Cropping the image can improve the quality and resolution of the edited image elements because the relative size of the image elements within the image can be increased. This is particularly true when the resolution of the image being processed is adjusted to the working resolution, as described below with reference to step 535. When processing a smaller image at a set working resolution, the pixel density of the image can be increased and a greater number of pixels can be allocated to the image elements being edited.

[0135] In some embodiments, the processor 421 executing the crop module 435 may determine the smallest cropped area that completely encompasses the area selected by the user. The smallest cropped area may be the smallest rectangle that completely encompasses the area selected by the user and has a border parallel to the border of the accessed image. In some embodiments, the processor 421 executing the crop module 435 may determine a buffer area around the area selected by the user and / or the smallest cropped area to create the cropped area. In some cases, the buffer area may be set to zero so that the cropped image is the smallest cropped area.

[0136] The buffer area may be an additional area around the minimum cropped area. The buffer area may be determined based on the total area of ​​the minimum cropped area and / or the area selected by the user. For example, the buffer area may be an additional area between 1% and 50% of the size of the minimum cropped area and / or the area selected by the user. The buffer area may be an additional area between 5% and 30% of the size of the minimum cropped area. The buffer area may be an additional area between 10% and 20% of the size of the minimum cropped area. The buffer area may be an additional area of ​​approximately 15% of the size of the minimum cropped area. Thus, the cropped image may include the minimum cropped area around the area selected by the user, as well as an additional buffer area.

[0137] Where the original image received at step 505 is cropped as described above, the same boundary coordinates may also be used to crop the image data defining the user-selected region generated at step 525 and stored as user-selected region data 432 .

[0138] At optional step 535, the processor 421 executing the resolution adjustment module 436 may adjust the resolution of the image being edited to the working resolution. This may be the original image received at step 505, or if a cropping step is performed, this may be the cropped image generated at step 530. According to some embodiments, the working resolution may be between 448×448 pixels and 640×640 pixels. In the case where the image being processed is not a square, the working resolution may have the same number of pixels as a 448×448 to 640×640 pixel square. According to some embodiments, the number of pixels in the image may be adjusted to a multiple of 32. In some embodiments, changing the working resolution of the image may include generating a latent image map of the pixel information of the image or a mapping of the pixel information to the latent image space. This may allow faster or more efficient processing of the image. According to some embodiments, an encoder may be used to perform compression of the image.

[0139] The resulting image from the last of steps 505, 530, or 535 to be performed may be considered the "image portion to be edited."

[0140] At step 540, the processor 421 executing the encoding module 434 is caused to determine the encoding or embedding of the prompt received at step 520. The encoding may be a representation of the prompt, such as a numerical representation. The encoding or embedding may be generated by first tokenizing the prompt and then using the token to generate the encoding. In some embodiments, when the prompt is a text prompt, the encoding module 434 may use a text encoder to determine a numerical value and / or a set / series of numerical values ​​indicating the meaning or content of the prompt from the prompt. The encoding module 434 may use any suitable text encoder / text send encoding process such as frequency document vectorization, one-hot encoding, index-based encoding, word embedding, or contrastive language-image pre-training (CLIP) to determine the encoding. In some embodiments, the encoding module may use the CLIP ViT-L / 14 text encoder to determine the encoding of the prompt.

[0141] In some embodiments, when the prompt is an audio file, the processor 421 executing the encoding module 434 can be caused to determine a text representation of the audio recording before performing the encoding step. The text representation of the audio recording can be determined using a speech-to-text ML model such as Google Speech-to-text, DeepSpeech, Kaldi, or Wav2Letter, or any other suitable speech-to-text ML model.

[0142] According to some embodiments, step 540 may be performed at any point after receiving the prompt at step 520 and before starting the denoising loop 595 at step 550 .

[0143] At step 545, the processor 421 executing the visual noise module 437 is caused to generate visual noise based on the image portion to be edited to generate a latent image. This may be referred to as the "original latent image". Visual noise may be generated by adding randomly generated visual noise to the image portion to be edited. In some embodiments, the noise may be Gaussian noise. In some embodiments, visual noise is generated by completely replacing the pixel information of the image portion to be edited with visual noise. In other words, the entire pixel information set or the latent image mapping of the pixel information may be deleted and replaced with pure visual noise so that the image portion to be generated is transformed into visual noise. In the case where the noise intensity parameter defines the amount of noise to be added, this may be accomplished by setting the noise intensity parameter to 1.0. In some alternative embodiments, a noisy image that is not pure noise may be generated. This may be accomplished by adding less noise to the image or using a noise intensity parameter less than 1.0. This may allow some visual features of the image to be retained so that the editing process retains more original features of the image.

[0144] Visual noise may refer to the variance of the properties / quality of the pixel information of a digital image or a latent image mapping of the pixel information, such as a random variance. The properties of the pixel information of a digital image or a latent image mapping of the pixel information may be brightness, color (e.g., color model), dimension, bit depth, hue, chroma (saturation) and / or value (brightness). Visual noise may include multiple pixels whose information has been randomly changed / varied. In some embodiments, visual noise may be Gaussian noise, which is a signal noise having a probability density function equal to that of a normal distribution (also called a Gaussian distribution). In some embodiments, Gaussian visual noise may be white Gaussian noise, in which the values ​​at any time pair are identically distributed and statistically independent (and therefore uncorrelated). According to some embodiments, noise may be added multiple times with a relatively low standard deviation.

[0145] Steps 550 to 575 of method 500 include a denoising loop 595, in which a back-diffusion process is used to transform the original latent image generated at step 545 into an edited image through a series of predictions generated by the ML model based on the prompt received at step 520. Each pass through the loop includes a time step in the denoising process, which is considered to be a backward step with respect to a reference time t. Denoising is a backward or inverse process compared to the forward denoising process performed during the training of the ML model, as described below with reference to Fig.11 Described in further detail.

[0146] The steps performed during the denoising loop 595 described below with reference to steps 550 to 575 enhance the image editing process by reducing the impact of the editing process outside of the image elements selected for user editing. Specifically, the steps are configured to restrict any editing to the user-selected area of ​​the original image without affecting the unselected elements of the image. This reduces undesirable effects, such as color leakage outside of the element being edited, such as Figure 2C and 3C As shown in .

[0147] Immediately before the denoising cycle begins can be considered to be at time step t+1, where the latent image at time step t+1 is the original latent image, i.e., the visual noise generated at step 545. Since the denoising cycle is a backward process, each iteration of the cycle ends with a time step moving backward so that t=t-1. In other words, increasing the time step results in an increase in noise, while decreasing the time step results in a decrease in noise. Therefore, the start of the denoising cycle occurs at time step t.

[0148] At step 550, the processor 421 executing the denoising module 438 is caused to predict a noisy image, i.e., an image representing the amount of noise present in the latent image at time step t+1. The noisy image may be generated based on at least the latent image and the hint. In some embodiments, the noisy image is generated relative to a final image generated based on the original image and the editing instructions provided by the hint. The denoising module 421 has learned to predict the noisy image during the training process by learning to associate each time step with the amount of noise present at that time step based on the total noise added to the image at that time during the forward denoising process, as described below with reference to Fig.11 described.

[0149] Processor 421 may perform this prediction step by providing the current latent image, the portion of the image to be edited, and the hint encoded at step 540 to an image editing model trained to perform the denoising step (e.g., the ML model described by Brooks et al.). In some embodiments, this may be as described below with reference to Fig.11 ML models trained as described.

[0150] At step 552, the processor 421 executing the denoising module 438 is caused to determine the amount of noise to be removed from the latent image at the current time step t based on the noise schedule. The noise schedule may define how much noise to remove at each time step, and may be based on the noise schedule used to train the ML model during the forward noisy process. In some embodiments, more noise may be removed during initial higher time steps, and less noise may be removed during later lower time steps.

[0151] At step 555, the processor 421 executing the denoising module 438 is caused to generate an updated latent image for the current time step t. This is generated based on the noise image predicted at step 550, the amount of noise determined at step 552, and the previous latent image at time step t+1. Specifically, the updated latent image at time step t is calculated by subtracting at least a portion of the noise image from the previous latent image. According to some embodiments, the updated latent image is generated by multiplying the noise image calculated at step 550 by the amount of noise determined at step 552 and subtracting the result from the previous latent image at time step t+1. The latent image at time step t+1 may be the original latent image during the first iteration of the denoising loop 595 or a latent image from a previous iteration of the denoising loop 595. Due to the subtraction of the noise, the updated latent image at time step t has less noise than the previous latent image at time step t+1.

[0152] At step 560, the processor 421 executing the denoising module 438 is caused to generate a noisy representation of the image portion to be edited based on the current time step (i.e., time step t). The noisy representation is calculated by adding random noise to the image portion to be edited. The intensity of the added noise is controlled by a variance parameter and depends on the current time step t. Specifically, a higher value of the time step t results in more noise being added to the noisy representation. The variance parameter is a parameter of the noise function, where a high variance means more noise and a smaller variance means less noise. This variance depends on the time step t, so that a higher t means a higher variance, which results in more noise being added to the image portion.

[0153] At step 565, the processor 421 executing the denoising module 438 is caused to calculate the masked latent image based on the updated latent image at time step t generated at step 555, the noisy representation at time t generated at step 560, and the user-selected region received at step 525. Specifically, a binary mask defining the user-selected region is retrieved from the user-selected region data 432.

[0154] The masked latent image at time step t is calculated as:

[0155] masked_latent(t)=mask*latent(t)+(1-mask)*noisy_representation(t)

[0156] in:

[0157] mask is a binary mask representing the user-selected region retrieved from the user-selected region data 432;

[0158] latent(t) is the updated latent image at time step t generated at step 555; and

[0159] noisy_representation(t) is the noisy representation at time step t generated at step 560 .

[0160] The masked latent image is used according to the above equation to restrict the original image to be partially edited to the area selected or masked by the user. Outside the masked area, the original image data from the image portion to be edited passes through unedited. Therefore, mask*latent(t) represents the area of ​​the image portion where editing will occur under the guidance of the input instruction prompt, and (1-mask)*noisy_representation(t) represents the complementary area that will remain similar to the original image. This prevents edits to the image (such as changes in color) from overflowing outside the image element to be edited. Fig.6D ,8C and 9C further describe in detail example images with and without this step being performed.

[0161] At step 570, the processor 421 executing the denoising module 438 is caused to determine whether further processing is required. According to some embodiments, the denoising module 438 may be configured to perform a predetermined number of denoising cycles or iterations through the denoising loop 595, and thus step 570 may include comparing the number of denoising cycles that have been performed to a predetermined number of denoising cycles to be performed. According to some embodiments, 15 to 40 denoising cycles may be performed. In some embodiments, 20 to 30 denoising cycles may be performed. For example, approximately 20, 25, or 30 denoising cycles may be performed.

[0162] If the processor 421 determines that further denoising steps are to be performed, the processor 421 proceeds to step 575. At step 575, the processor 421 executing the denoising module 438 starts a new denoising cycle by decrementing the time step. For example, this can be achieved by setting the time step t to t-1.

[0163] If processor 421 determines not to perform further denoising steps, processor 421 proceeds to step 580. The final masked latent image determined at step 565 may be considered the edited image portion.

[0164] At step 580, if the working resolution of the image being processed was adjusted at step 535, the processor 421 executing the resolution adjustment module 436 is caused to restore the resolution of the edited image portion to be equivalent to the resolution of the original image accessed at step 505. This may be performed using a decoder configured to decompress or reconstruct the image to restore the pixel representation of the image based on the latent image representation. If the working resolution was not adjusted at step 535, the processor 421 may skip step 580.

[0165] At step 585, where the image being processed was cropped at step 530, the processor 421 executing the image combination module 439 may insert at least a portion of the edited image portion into an area of ​​the original image received at step 505 corresponding to the cropped area or the user-selected area to create an output image. In some embodiments, only the area of ​​the edited image portion corresponding to the user-selected area may be inserted back into the original image. In other embodiments, the entire edited image portion may be inserted back into the original image.

[0166] In some embodiments, at step 585, the processor 421 executing the image combining module 439 may be caused to blend the pixel information of the edited image portion with the original image so that the edited image portion blends naturally and / or substantially unnoticeably with the original image. This may be necessary because some pixel information outside of the edited image element may have been inadvertently edited or distorted during the editing process. As described in further detail below, Figures 10A to 10D A process of fusing pixel information is shown which may be performed by the processor 421 at step 585. The result of the fusing process may be considered as a final edited image.

[0167] At step 590, the processor 421 executing the server application 433 is caused to output the final edited image. This may be accomplished by one or more of the following operations: saving the image to the memory 412, the memory 422, or an external memory location; sending the image to an external device for storage or display, and / or displaying the image via the user I / O 416 of the user computing device 410.

[0168] Fig. 6A and 6B is a diagrammatic representation of the stages of performing a previously known automatic image editing method on an image.

[0169] Fig. 6A An example original image 600 is shown. Image 600 includes a background 605, which includes a mountain 610, and a foreground element, which is a car 615. Image 600 may be an image selected or uploaded by a user for editing using an image editing application.

[0170] Figure 6B An example edited image 620 is shown. Image 620 is an edited version of image 600 and was generated using a hint-based image editing technique performed by an image editing application according to some known techniques using a supplied hint "darken the car." For example, image 620 may have been generated using a diffusion machine learning (ML) model, which is a neural network model trained or otherwise configured to denoise an image containing Gaussian noise by learning to reverse the diffusion process. Image 620 may have been generated using the techniques described by Brooks et al.

[0171] However, editing image 600 to generate image 620 produces some undesirable results. Image 620 shows a background 625 including a mountain 630, and a car 635. According to the provided hint, car 635 has been rendered as a darker color than car 615. However, the remaining elements of image 620 have also been changed. Fig. 6AThe background 625 including the mountain 630 has become significantly darker than the background 605 and the mountain 610. In other words, the edits made to the car 615 / 635 are not contained in the image element, but have been applied to the entire image 600 / 620.

[0172] Figure 6C and 6D is a graphical representation of the stages of a partial masking editing process performed on an image. Specifically, Figure 6C and 6D A method similar to method 500 is involved, but wherein at least steps 560 and 565 are omitted.

[0173] Figure 6C An example masked image 640 is shown. Image 640 includes Fig. 6A Background 605 including mountains 610 and car 615. Image 640 further includes a user-selected area 645 indicating an area of ​​the image that the user wishes to edit.

[0174] Fig.6D An example edited image 660 is shown. Image 660 is an edited version of image 600 and is generated using the partially masked editing process as described above with the supplied hint “make the car darker.” For example, image 620 may have been generated using a diffusion machine learning (ML) model.

[0175] In this case, image 660 shows background 605 including substantially unedited mountain 610. According to the provided hint, image 660 includes car 635 that has been rendered as a darker color than car 615. However, around car 635, there is a Figure 6C Region 665 of user-selected region 645 has also been altered and is darker than the corresponding region of image 600. Although the changes have been limited to user-selected region 645, the changes have not been limited to the image element identified for editing, namely the car.

[0176] Figures 7A to 7K Is executed as above reference Figure 5 A diagrammatic representation of some of the stages of the described hint-based image editing method 500. As shown in the images, performing method 500 may increase the extent to which editing of an image is restricted to particular image elements compared to the method shown in images 6A-6D.

[0177] Fig. 7A Shown with Fig. 6A. Image 600 includes: background 605, which includes mountains 610; and foreground element, which is car 615. Image 600 may be the image accessed for editing and stored to image data 430 at step 505 of method 500.

[0178] A prompt (not shown) may be received from a user at step 520 of method 500 and stored in prompt data 431. Figures 7A to 7K In the example shown in , the example prompt received is "make the car darker."

[0179] Figure 7B An example masked image 705 is shown. Image 705 contains Fig. 7A 605 and a car 615. Image 705 further includes a user-selected area 645 indicating an area of ​​the image that the user wishes to edit. User-selected area 645 may be received at step 525 of method 500 and stored in user-selected area data 432.

[0180] Figure 7C A cropped masked image 710 is shown. Portions of the background 605 and the mountain 610 are visible in the cropped masked image 710, but these portions have been partially cropped. The entirety of the car 615 and the user-selected area 645 have been retained in the image 710. The image 710 is a version of the image 705 that has been cropped tightly around the user-selected area 645. The image 710 may be generated by the cropping module 435 at step 530 of the method 500 and may be used to define the binary mask used at step 565.

[0181] Fig.7D A cropped image 715 is shown. Portions of the background 605 and the mountain 610 are visible in the cropped masked image 715, but these portions have been partially cropped. The entirety of the car 615 has been retained in the image 710. The image 715 is a version of the image 600 that has been cropped closely around the user-selected area 645. In other words, the image 715 is a version of the image 710 from which the user-selected area has been removed. The image 710 may be generated by the cropping module 435 at step 530 of the method 500 and may be used to generate an edited version of the car 615.

[0182] Fig. 7E Latent image 720 is shown. Image 720 includes visual noise 725, which may be Gaussian noise. Image 720 may be generated by visual noise module 437 at step 545 of method 500.

[0183] Figure 7FNoise image estimate 730 is shown. Noise image estimate 730 includes visual noise 735, which is a prediction of the amount of noise added to latent image 720 at time step t+1. Noise estimate 730 may be generated by denoising module 438 during step 550 of method 500.

[0184] Figure 7G An updated latent image 740 is shown. The updated latent image 740 shows the generated background 742 including the mountain 744 and the car 746. The updated latent image 740 still contains an amount of visual noise 748. The updated latent image 740 may be generated by the denoising module 438 during step 555 of the method 500 by subtracting the noise image estimate 730 from the latent image 720. By doing so, image elements 742, 744, and 746 begin to emerge or be generated from the noise, where the editing is performed according to the applied hint. Specifically, the car 746 is darker than the car 615. However, the background 742 and the mountain 744 are also darker than the background 605 and the mountain 610.

[0185] Figure 7H A noisy representation 750 is shown. The noisy representation 750 shows a background 752 including a mountain 754 and a car 756. The noisy representation 750 is a version of the cropped image 715 with noise 758 applied. In other words, the background 752, the mountain 754, and the car 756 are simply the background 605, the mountain 610, and the car 615 with the noise 758 added. The noise 758 may be calculated by the denoising module 438 during step 560 based on the current time step t and applied to the cropped image 715 to produce the noisy representation 750.

[0186] Fig.7I The masked latent image 760 is shown. The masked latent image 760 shows a background 752 and a mountain 754 that are substantially the same as the background and the mountain in the noisy representation 750 including the noise 758. The masked latent image 760 further shows a car 746 and an area 762 around the car 746 that are substantially the same as the car and the area around the car in the updated latent image 740. The masked latent image 760 may be generated by the denoising module 438 during step 565 of the method 500, during which the updated latent image 740 is combined with the noisy representation 750 based on the user-selected area 645. In other words, the area of ​​the updated latent image 740 that corresponds to the user-selected area 645 is combined with the area of ​​the noisy representation 750 that is outside of the user-selected area 645.

[0187] The masked latent image 760 may be reprocessed by iteratively following steps 550 to 565 until a predetermined stop parameter is reached. For example, the predetermined stop parameter may be the number of iterations. With each step, additional noise is removed, thereby constructing the edited image elements while retaining the unedited background features.

[0188] Figure 7J An example final masked latent image 770 is shown where the background 605 including the mountain 610 has been retained, while the car 675 has been edited to be darker than the car 615. The color "bleed" of the car 675 is not visible in the other image elements.

[0189] Figure 7K Final image 780 is shown. Final image 780 may be generated during step 585 of method 500. Image 780 is the result of inserting the final masked latent image 770 back into the original image 600. This results in a complete image showing the entire unedited background 605 including the mountains 610, as well as the edited car 675.

[0190] Figures 8A to 8D 9A to 9D show a partial mask editing method and the above reference Figure 5 Some example images of the results of both described hint-based image editing methods 500. As shown in the images, performing the method 500 may increase the extent to which editing of an image is restricted to specific image elements, as compared to partial masking methods.

[0191] Fig. 8A An example original image 200 is shown that includes a background 205 and an object, which is a kayak 215. In the example shown, the kayak is placed on a concrete platform 210. The kayak 215 has a red exterior, a gray interior, and some black details. The background 205 shows a body of water with green bushes or trees in the distance, and a portion of the sky visible above the horizon. Image 200 may be an image selected or uploaded by a user for editing using an image editing application. Image 200 and Figure 2A The same as image 200.

[0192] Figure 8B An example masked image 820 is shown. Image 820 includes background 205, platform 210, and kayak 215 of image 200. Image 820 further includes a user-selected area 825 indicating an area of ​​the image that the user wishes to edit.

[0193] Figure 8C An example edited image 840 is shown. Image 840 is an edited version of image 200 and is generated using the partially masked editing process as described above with the supplied cue “make the boat green.” For example, image 840 may have been generated using a diffusion machine learning (ML) model.

[0194] In this case, image 840 shows a substantially unedited background 205 and platform 210. In accordance with the provided prompt, image 840 includes kayak 845 that has been rendered a greener color than kayak 215. However, area 850 surrounding kayak 845 corresponding to user-selected area 825 of image 820 has also been altered and is greener than the corresponding area of ​​image 200. While the changes have been limited to user-selected area 825, the changes have not been limited to the image elements identified for editing, namely kayak 215 / 845.

[0195] Fig.8D Another example edited image 860 is shown. Image 860 is also an edited version of image 200, but was generated using method 500, as described above with reference to Figure 5 Image 860 illustrates that using method 500, background 205 and platform 210 have been retained, while kayak 865 has been edited to be greener than kayak 215. The color "bleed" of kayak 865 is not visible in the other image elements.

[0196] Fig. 9A An example original image 300 is shown that includes a background 305 and an object, which is an apple 315. In the example shown, the apple is red and is placed on a white lace tablecloth that forms the background 305. Other food items such as sushiki 310 are placed on the tablecloth. Image 300 may be an image selected or uploaded by a user for editing using an image editing application. Image 300 and Figure 3A The same as image 300.

[0197] Fig. 9B An example masked image 920 is shown. Image 920 includes background 305, food 310, and apple 315 of image 300. Image 920 further includes a user-selected area 925 indicating an area of ​​the image that the user wishes to edit.

[0198] Fig. 9C An example edited image 940 is shown. Image 940 is an edited version of image 300 and is generated using the supplied hint “make the apples gold” using the partial masking editing process as described above. For example, image 940 may be generated using a diffusion machine learning (ML) model.

[0199] In this case, image 940 shows a substantially unedited background 305 and food 310. According to the provided prompt, image 940 includes apple 945 that has been rendered a more yellow color than apple 315. However, area 950 surrounding apple 945 corresponding to user-selected area 925 of image 920 has also been altered and is more yellow than the corresponding area of ​​image 300. While the changes have been limited to user-selected area 925, the changes have not been limited to the image elements identified for editing, namely apple 315 / 945.

[0200] Fig.9D Another example edited image 960 is shown. Image 960 is also an edited version of image 300, but was generated using method 500, as described above with reference to Figure 5 Image 960 illustrates that using method 500, background 305 and food 310 have been retained, while apple 965 has been edited to be more yellow or gold than apple 315. The color "bleed" of apple 965 is not visible in the other image elements.

[0201] Fig.8D and 9D Several advantages of the described systems and methods are demonstrated.

[0202] Figures 10A to 10D is an example image depicting a fusion of pixel information of the edited portion of the image with the original image, as described above with reference to step 585 of method 500. Figures 10A to 10D As can be seen from the following description, the pixel information of the edited image portion can be changed so that the pixel information of the edited image portion is more similar to and / or substantially the same as the pixel information of the original image.

[0203] Fig. 10A 1 is an example original image 1000 showing a background 1002 and a foreground element 1004, which is a spaceship in the illustrated embodiment. The spaceship is displayed on a pink sky region.

[0204] Fig. 10B 10 is an example image 1010 that also shows background 1002 and foreground element 1004. Image 1010 also includes user-selected area 1006. User-selected area 1006 may be an area selected by a user in which image editing is to be performed.

[0205] Fig. 10C1 is an example edited image 1020 generated by a diffusion-based ML model through a diffusion process (e.g., by following the steps of method 500). Image 1020 is generated based on a prompt "make the spaceship pink". Image 1020 includes background 1002 and new foreground element 1022, which is an edited version of foreground element 1004. Specifically, the color of edited foreground element 1022 is more pink than the color of original foreground element 1004. However, image 1020 also includes non-matching region 1024. Non-matching region 1024 is a region surrounding foreground element 1022 that does not obviously match the rest of image 1020 in color. This may be attributed to the unintentional editing of pixels of non-matching region 1024 during the editing of foreground element 1022.

[0206] Fig. 10D 1030 is an example image in which a pixel blending process has been performed to remove the non-matching region 1024. Image 1040 includes background 1002 and new foreground element 1022. In addition, region 1034 around foreground element 1022 matches background 1002, so that example image 1030 does not include non-matching region 1024.

[0207] To perform the fusion process, the processor 421 executing the image combination module 439 can transform the non-matching area 1024 into a matching area 1034 by adjusting the value of the pixel information of the area corresponding to the user-selected area 1006. This can be achieved by first determining the difference between the average value of the pixel information of the edited user-selected area and the average value of the pixel information of the same area in the original image. The edited image can then be corrected by the determined difference. In some embodiments, the user-selected area can be feathered, blended, or faded outward from the boundary of the user-selected area so that the edited image portion is better blended into the original image.

[0208] Fig.11 1 is a process flow diagram of a method 1100 for training an image editing model to perform an image editing process according to some embodiments. The image editing model may be an ML model, which may be a diffusion model. In some embodiments, the method 1100 may be used to train an image editing ML model, such as a denoising module 438, to perform a denoising process. According to some embodiments, the method 1100 may be performed at least in part by a processor 421 executing a server application 425. Although certain steps of the method 1100 have been described as being performed by specific elements of the system 400, in some embodiments, these steps may be performed by different elements. In addition, although the steps of the method 1100 have been shown and described as occurring in a particular order, some of the steps may be performed in an alternative order without affecting the results of the method.

[0209] At step 1110, processor 421 executing server application 433 accesses initial training data to begin the training process. In some embodiments, the initial training data may be retrieved from a memory location such as memory 422 received by communication interface 427 from an external device or from a user interface.

[0210] In some embodiments, the initial training data may include artificially generated training data, which may be text-based training data. According to some embodiments, the training data includes a data set of edit triplets, each edit triplet including a title of a first image, an edit instruction for performing an edit on the first image, and a title of a second image that is an edited version of the first image that has been edited based on the edit instruction. A selection of example triplets is provided in the following table:

[0211]

[0212] According to some embodiments, 500 to 1000 triplets may be accessed.

[0213] At step 1120, processor 421 executing server application 433 uses the training data received at step 1110 to fine-tune the text prompt generation model to perform text prompt generation. For example, the text prompt generation model can be an ML model, which can be a large language model (LLM), such as a GPT-3 Davinci model. The text prompt generation model can be trained using a single epoch of the training data accessed at step 1110, and in some embodiments, default training parameters can be used.

[0214] At step 1130, processor 421 executing server application 433 uses the text hint generation model fine-tuned at step 1120 to generate a large collection of text-based training data in the form of triples as described above. Specifically, the text hint generation model is supplied with a dataset of first image titles and is used to generate corresponding image editing instructions and second image titles for images edited based on the editing instructions. According to some embodiments, the image titles used as input may be retrieved from an image title dataset, such as from the LAION-Aesthetics dataset. In some embodiments, 400,000 to 500,000 training data triples may be generated.

[0215] At step 1140, the processor 421 executing the server application 433 generates a set of image-based training data using the text-based training data generated at step 1130. This can be done using an image generation model, which can be an ML model, and in some embodiments can be a diffusion ML model. For each training data triple, at least one image pair is generated. The image pair is generated using the first image title and the second image title of each triple. In order to promote the similarity of the generated image pairs, a prompt-prompt method can be used. The prompt-prompt method promotes multiple generation similarities from the text-image diffusion model. This is done by borrowing cross-attention weights in a certain number of denoising steps. Specifically, the prompt-prompt has a parameter that can control the similarity between the two images, i.e., a fraction p of denoising steps with shared attention weights. According to some embodiments, the processor 421 can be caused to generate multiple image pairs for each text triple, wherein each image pair is generated with a different p. In some embodiments, the p value can be maintained between 0.1 and 0.9. According to some embodiments, 50 to 200 image pairs can be generated for each text triple.

[0216] At step 1150, the processor 421 executing the server application 433 is caused to filter the image generated at step 1150 to ensure the degree of similarity between the two images and the editing instructions. This filtering can be performed using one or more CLIP-based metrics, such as: an image-image CLIP metric to ensure that the image pair is not too different; an image-title CLOP metric to ensure that the edited image corresponds to the editing instructions; and / or a directional similarity of the two images in CLIP space to ensure that the front and back images match the front and back titles. According to some embodiments, an image-image CLIP threshold between 0.6 and 0.9, between 0.7 and 0.8, or about 0.75 can be used. In some embodiments, an image-title CLIP threshold between 0.1 and 0.3, between 0.15 and 0.25, or about 0.2 can be used. In some embodiments, a directional CLIP similarity threshold between 0.1 and 0.3, between 0.15 and 0.25, or about 0.2 can be used. Data sets that do not fall within one or more of the thresholds can be discarded. In some embodiments, the datasets may be sorted by the CLIP metric and only a selection of the highest-rated datasets may be retained. For example, the datasets may be sorted based on directional CLIP similarity and only the top 4 images may be retained.

[0217] At step 1160, the processor 421 executing the server application 433 is caused to train an image editing model. The image editing model may be an ML diffusion model, which may be a conditional diffusion model. The image editing model may be supplied with a pair of images and corresponding editing instructions retrieved from the training data. Starting with a first image, during a forward process, the image editing model is caused to add noise to the image based on a noise schedule, wherein additional noise is added at each time step t so that the total noise present in the image increases within the time step t. During a backward process, the image editing model is supplied with editing instructions, and then the image editing model is caused to incrementally remove noise in a manner that generates a second edited image, again using the noise schedule to determine how much noise to remove. The actual noise added during the forward process may be compared to the predicted noise to determine a loss value, and the model may be configured to minimize the loss value for each prediction. This produces a model that can predict the noise present in an image at any time step t given an image and editing instructions. The trained image editing model may be used as part of the denoising module 438 to perform step 550 of method 500, as described above.

[0218] It will be appreciated by those skilled in the art that many changes and / or modifications may be made to the above-described embodiments without departing from the broad general scope of the present disclosure. Therefore, the present embodiments should be considered in all aspects as illustrative and not restrictive.

Claims

1. A method for performing hint-based image editing, the method comprising: Access images; receiving a selected region of the image; receiving a prompt, wherein the prompt indicates an editing instruction; generating a latent image by transforming the image into visual noise; predicting a noise image corresponding to the latent image based on the latent image and the hint; subtracting at least a portion of the noise image from the latent image to generate an updated latent image; generating a noisy representation of the image; A masked latent image is generated based on the noisy representation, the updated latent image, and the selected region, wherein the masked latent image includes the updated latent image in a region corresponding to the selected region and the noisy representation in a region not corresponding to the selected region.

2. The method of claim 1, further comprising the step of repeating a denoising cycle over a series of time steps, wherein the denoising cycle comprises the following steps: Predicting a noisy image, generating an updated latent image, generating a noisy representation, and generating a masked latent image.

3. The method of claim 2, wherein predicting the noisy image comprises predicting the visual noise that will be present in the latent image at time step t+1 during the noise addition process given a current time step t. 4 . The method of claim 2 , wherein generating the noisy representation of the first image comprises adding noise to the image based on the current time step t. The method of claim 2 , wherein the time step t is decremented after each iteration. The method of claim 2 , wherein the denoising cycle is repeated 15 to 40 times. The method according to claim 6 , wherein the denoising cycle is repeated 20 to 30 times.

8. The method of claim 1, wherein the portion of the noise image to be subtracted from the latent image is determined based on a noise schedule.

9. The method of claim 1, wherein the step of predicting the noise image is performed by a diffusion model.

10. The method of claim 9, wherein the diffusion model is trained using a data set comprising a first image, an editing instruction, and a second image edited based on the editing instruction.

11. The method of claim 10, wherein the diffusion model is trained by causing the diffusion model to add noise to the first image as time steps t increase, and then causing the diffusion model to remove noise by decreasing time steps t to generate a second edited image.

12. The method of claim 1, further comprising determining an encoding of the received hint, wherein the step of predicting a noisy image is performed using the encoding of the hint.

13. The method of claim 1, further comprising cropping the accessed image based on the selected area.

14. The method of claim 1, further comprising changing a resolution of the accessed image to a working resolution before generating the latent image, wherein the working resolution is a lower resolution than the resolution of the accessed image.

15. The method of claim 1, further comprising mapping pixel information corresponding to the accessed image to a latent image space before generating the latent image, such that the latent image is a representation of the image in the latent image space.

16. The method of claim 1, further comprising inserting the generated masked latent image into the accessed image to produce an output image.

17. The method of claim 16, further comprising performing a pixel blending process on the generated masked latent image to blend the generated masked latent image with the accessed image.

18. The method of claim 17, further comprising outputting the image by saving the output image to a memory location, displaying the output image to a user, or sending the output image to an external device.

19. The method of claim 1, further comprising generating a binary mask based on the selected area, and using the binary mask to generate the masked latent image.

20. The method of claim 1, wherein the masked latent image is calculated based on the following equation: masked_latent = mask*latent + (1-mask)*noisy_representation, Wherein mask represents a mask generated based on the selected area, latent represents the updated latent image, and noisy_representation represents the noisy representation.

21. The method of claim 1, further comprising receiving an indication that the prompt involves a local edit.

22. The method of claim 1, wherein the first visual noise is Gaussian noise.

23. A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform the method according to any one of claims 1 to 22.

24. A computing device comprising: The non-transitory computer-readable storage medium of claim 23; as well as A processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.

Images