Recommended services for controllable diffusion model-based image galleries

Abstract

Aspects of this disclosure include methods and systems for utilizing a controllable diffusion model for dynamic image search in an image gallery recommendation service. An exemplary method may include displaying an image gallery having multiple gallery images and dynamic image frames. The dynamic image frames may include generated images and interactive widgets. The method may include receiving user input in the interactive widget and generating an updated generated image by inputting the user input into a controllable diffusion model in response to receiving the user input. The method may include replacing a generated image in a dynamic image frame with an updated generated image.

Description

Technical Field

[0001] Introduction

[0001] This disclosure relates to an image search and recommendation system, and more particularly to leveraging a controllable diffusion model for dynamic image search in an image gallery recommendation service.

Background Art

[0002]

[0002] An image gallery recommendation system (also referred to as a visual or image-based discovery system) plays an increasingly important role in many diverse domain applications including e-commerce, social media, and entertainment. The primary goal of an image gallery recommendation system is to predict and recommend one or more relevant images that a user is likely to find interesting or appealing. To achieve this, image gallery recommendation systems utilize a variety of techniques (such as collaborative filtering, content-based filtering, and deep learning) to provide personalized recommendations to users based on their personality, preferences, and behavior.

[0003]

[0003] Collaborative filtering involves analyzing the past behavior and preferences of similar users to make recommendations. By examining historical data of users with similar hobbies and preferences as a given user, image gallery recommendations can more accurately identify images that are likely to be interesting to the user. Collaborative filtering can be either item-based, user-based, or both, where the former focuses on the similarity between items (images), and the latter focuses on the similarity between users.

[0004]

[0004] Content-based filtering refers to the analysis of the content and features of the images themselves in order to make recommendations. An image gallery recommendation system may extract relevant information (such as color, texture, shape, and other visual attributes) from an image and use this extracted information to discover similar images (by using feature similarity, distance measures, etc.). By recommending images that are visually similar to those the user has already shown interest in, content-based filtering aims to capture the user's preferences based on image features.

[0005]

[0005] Conventional deep learning technologies such as convolutional neural networks (CNNs), variational autoencoders (VAEs), and transformer networks have revolutionized image recommendation systems. CNNs can learn subtle patterns and features (hierarchical representations) from images by processing them through multiple layers of interconnected neurons. By training on large datasets, these networks can capture complex relationships (local and global image features) and make accurate predictions about user preferences based on image content.

[0006]

[0006] A VAE is a generative model that can learn a compact representation (latent space) of input data. In the context of image recommendation, a VAE can learn a low-dimensional representation of images that captures the intrinsic structure and variation within a dataset. By leveraging this latent space, a VAE can generate new and diverse images that match user preferences, thereby enhancing the recommendation capabilities of image gallery recommendation services.

[0007]

[0007] Transformer networks were originally designed for natural language processing tasks, but have been found to excel in a range of other applications, including computer vision with image recommendation. Transformers model long-term dependencies and capture contextual information within the data. In image gallery recommendation systems, transformer networks can be used to learn complex contextual relationships between images and to generate more accurate recommendations based on this contextual information.

[0008]

[0008] The image gallery recommendation system may also rely on user behavior data (user interaction) to enhance user satisfaction, engagement, and the overall user experience. In terms of user interaction, the image gallery recommendation system may offer several ways for users to engage with the system. For example, in an implementation where a user interacts with the image gallery recommendation system through a user interface such as a mobile app or website, the user may be presented with an initial curated set of images. The user may then interact with the system by viewing the images (e.g., by scrolling through a set of recommended images), by liking / disliking the images, by saving the images, by sharing the images (e.g., via a combined social media platform), and / or otherwise by actively or passively interacting with one or more images in the gallery. These user interactions may be used as feedback to the system to better determine the user's preferences and to refine recommendations in the future.

[0009]

[0009] User interaction can play an important role in training and refining the image gallery recommendation system, but it is important to note that these interactions are somewhat limited (in particular, users do not have direct control over the intrinsic algorithms and model parameters of the image gallery recommendation system). The system can learn from aggregated user data to improve recommendations for individual users and the entire user base, but individual users typically only engage with the image gallery through a few predetermined paths (viewing still images, clicking like / dislike, saving images, commenting on images, etc.). Unfortunately, these techniques have inherent limitations in capturing the full dynamics of user preferences. [Overview of the Initiative] [Means for solving the problem]

[0010] overview

[0010] Embodiments of the present invention are directed toward a method for utilizing a controllable diffusion model for dynamic image search in an image gallery recommendation service. Non-limiting exemplary methods include displaying an image gallery having a plurality of gallery images and dynamic image frames. Dynamic image frames may include generated images and interactive widgets. The method may include receiving user input in the interactive widget and, in response to receiving user input, generating updated generated images by inputting the user input into a controllable diffusion model. The method may include replacing generated images in dynamic image frames with updated generated images.

[0011]

[0011] In some embodiments, the method includes receiving an image query in a field of an image gallery. In some embodiments, a generated image is produced by inputting the image query into a controllable diffusion model. In some embodiments, a plurality of gallery images and a generated image are selected according to the degree of matching with one or more features in the image query.

[0012]

[0012] In some embodiments, the method includes determining one or more constraints in the image query. In some embodiments, the generated image is generated by inputting one or more constraints into a controllable diffusion model.

[0013]

[0013] In some embodiments, one or more constraints in the image query include at least one of a pose skeleton and an object boundary. In some embodiments, determining one or more constraints includes extracting an object boundary when a feature in the image query includes one of a structural feature and a geological feature. In some embodiments, determining one or more constraints includes extracting a pose skeleton when a feature in the image query includes one of a human and an animal.

[0014]

[0014] In some embodiments, multiple gallery images are supplied from an image database.

[0015]

[0015] In some embodiments, the interactive widget includes a text field. In some embodiments, receiving user input in the interactive widget includes receiving a text string input into the text field.

[0016]

[0016] In some embodiments, the interactive widget includes one or more of a dropdown menu, a checkbox, a slider, a color picker, a canvas interface for drawing or sketching, and rating buttons.

[0017]

[0017] In some embodiments, the interactive widget includes a canvas for magic wand input. In some embodiments, receiving user input in the interactive widget includes receiving magic wand input by graphically selecting one of a particular feature and a particular region in the generated image. In some embodiments, the interactive widget further includes a text field. In some embodiments, receiving user input in the interactive widget further includes receiving a text string input having contextual information for magic wand input.

[0018]

[0018] Embodiments of the present invention are directed to a system for utilizing a controllable diffusion model for dynamic image search in an image gallery recommendation service. A non-limiting exemplary system includes a memory having computer-readable instructions and one or more processors for executing computer-readable instructions. The computer-readable instructions control one or more processors to perform various operations. These operations include receiving an image query from a client device communicably coupled to the system. This operation includes providing a plurality of gallery images and generated images to the client device according to the degree of matching with one or more features in the image query. These operations include receiving user input from the client device and generating updated generated images in response to the user input by inputting the user input into a controllable diffusion model. These operations include providing the updated generated images to the client device.

[0019]

[0019] Embodiments of the present invention are directed to a system for utilizing a controllable diffusion model for dynamic image search in an image gallery recommendation service. A non-limiting exemplary system includes a memory having computer-readable instructions and one or more processors for executing computer-readable instructions. The computer-readable instructions control one or more processors to perform various operations. These operations include receiving a plurality of gallery images and generated images from an image gallery recommendation service that is communicably coupled to the system. These operations include displaying an image gallery having a plurality of gallery images and a dynamic image frame having generated images and an interactive widget. These operations include receiving user input in the interactive widget, sending the user input to the image gallery recommendation service, and receiving updated generated images from the image gallery recommendation service. These operations include replacing generated images in the dynamic image frame with updated generated images.

[0020]

[0020] The above features and advantages, as well as other features and advantages of this disclosure, will be readily apparent from the following detailed description made in relation to the attached drawings.

[0021] Brief explanation of the drawing

[0021] Details of the exclusive rights described herein are specifically pointed out and explicitly claimed in the claims in the conclusions herein. The aforementioned and other features and advantages of embodiments of the present invention are evident from the following detailed description made in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0022] [Figure 1]

[0022] A block diagram is drawn for using a controllable diffusion model according to one or more embodiments. [Figure 2]

[0023] A block diagram is drawn to illustrate how to utilize a controllable diffusion model for dynamic image search according to one or more embodiments. [Figure 3]

[0024] Depicts an exemplary image gallery according to one or more embodiments. [Figure 4]

[0025] Depicts the exemplary image gallery of FIG. 3 after user interaction according to one or more embodiments. [Figure 5]

[0026] Depicts the exemplary image gallery of FIG. 4 after user interaction according to one or more embodiments. [Figure 6]

[0027] Depicts the exemplary image gallery of FIG. 5 after user interaction according to one or more embodiments. [Figure 7]

[0028] Depicts the exemplary image gallery of FIG. 6 after user interaction according to one or more embodiments. [Figure 8]

[0029] Depicts an exemplary image gallery according to one or more embodiments. [Figure 9]

[0030] Depicts the exemplary image gallery of FIG. 8 after user interaction according to one or more embodiments. [Figure 10]

[0031] Depicts a block diagram of a computer system according to one or more embodiments. [Figure 11]

[0032] Depicts a flowchart of a method of utilizing a controllable diffusion model for dynamic image search within an image gallery recommendation service according to one or more embodiments. **DETAILED DESCRIPTION OF THE INVENTION**

[0023]

[0033] The diagrams depicted herein are exemplary. Many variations to the diagrams or operations described herein may exist without departing from the spirit of the invention. For example, some acts may be performed in various orders, or some acts may be added, deleted, or modified.

[0024]

[0034] In the accompanying drawings of the embodiments described in the present invention and in the following detailed description, various elements shown in the accompanying drawings are provided with two- or three-digit reference numerals. With few exceptions, the leftmost digit of each reference numeral corresponds to the drawing in which the element is first shown.

[0025] Detailed explanation

[0035] Image gallery recommendation systems are used in a variety of fields, including e-commerce, social media, and entertainment, to provide personalized recommendations to users. These systems employ various technologies (such as collaborative filtering, content-based filtering, and deep learning) to recommend images to users. However, these technologies have inherent limitations in capturing the dynamics of user preferences. Typically, users only interact with these systems by viewing still images, clicking like / dislike on images, saving images, and / or sharing and commenting on them.

[0026]

[0036] This disclosure introduces the use of so-called controllable diffusion models for dynamic image search into an image gallery recommendation service. Diffusion models refer to a class of generative models that leverage the diffusion process to generate high-quality synthetic data. Diffusion refers to the gradual propagation or dispersion of information or noise across a data space (e.g., images), and the diffusion process in a diffusion model involves repeatedly transforming an initial noise vector into samples by applying a series of diffusion steps. Each diffusion step gradually reduces the noise level (noise vector) in a way that progressively refines the generated image, while adding controlled noise to the image. By carefully controlling the noise process, diffusion models can generate high-quality images that present compelling details. In the context of image recommendation, diffusion models can be used to generate realistic, abstract, synthetic, retextured, artistic, and other images from user prompts that are visually similar to true images in the training dataset. For example, a diffusion model could generate a novel watercolor painting of a yacht on a river from the prompt "paint a picture of a yacht and a river with watercolors."

[0027]

[0037] A “controllable” diffusion model refers to a type of diffusion model that can be dynamically guided and fine-tuned post-image generation through additional user interaction. In some embodiments, for example, a controllable diffusion model is accessed via a dynamic image frame containing an image (which is itself the output of the diffusion model) and an interactive widget that can be selected, edited, and / or otherwise manipulated by the user. In some embodiments, the interactive widget may receive input from the user, such as text input, sketches, or images. In some embodiments, the controllable diffusion model may generate a new image and / or modify a previously generated image by using user input received via the interactive widget in the dynamic image frame as guidance. Continuing from the previous example, the controllable diffusion model may modify a generated watercolor of a yacht on a river by replacing the yacht with a motorboat in response to the user entering the additional text “Change yacht to motorboat” into the interactive widget.

[0028]

[0038] In some embodiments, dynamic image frames and their respective images are placed among a collection of other images in the entire image gallery (referred to herein as gallery images) as part of an image gallery recommendation service. Gallery images may include search images (non-generated images) from an image database. In some embodiments, search images are images that match an image query received from a user and / or one or more characteristics of the user. In this way, the user can quickly navigate the collection of images (both generated and search images) to find the image in question.

[0029]

[0039] Advantageously, leveraging a controllable diffusion model for dynamic image exploration in an image gallery recommendation service according to one or more embodiments enables a more natural and dynamic image exploration and image gallery experience for the user. Unlike conventional diffusion models that can frustrate users due to their inherent limitations, the controllable diffusion model and dynamic image frames described herein allow the user to easily guide the output of the diffusion model within the image recommendation framework to achieve resulting images that more closely reflect the user's requirements. In short, the controllable diffusion model and dynamic image frames enable the user and the diffusion model to cooperate and iteratively interact in a straightforward process to incrementally fine-tune the generated images to the user's precise specifications. The result is an image gallery recommendation service that efficiently generates highly relevant images in a cooperative and engaging manner with the user.

[0030]

[0040] Figure 1 depicts a block diagram 100 for using a controllable diffusion model 102 according to one or more embodiments of the present invention. As shown in Figure 1, the image query 104 is received by topic recognition and constraint map 106. The image query 104 may be generated and / or otherwise supplied from an external system (e.g., a client device, see Figure 2). The image query 104 (also called a prompt) may include not only text but also other types of input modalities as described earlier. For example, the image query 104 may include the text string "Show a picture". In another example, the image query 104 may include a user-supplied sketch or line drawing of a person or animal. In yet another example, the image query 104 may include the text string "Give this a natural wood texture" combined with an image of a metal chair.

[0031]

[0041] In some embodiments, topic recognition and constraint map 106 receive and process the image query 104. In some embodiments, topic recognition and constraint map 106 includes a module configured to identify the subject and / or theme (collectively, “topic”) conveyed in the image query 104. Topic recognition helps the controllable spread model 102 better determine the content and / or context of the prompt, thereby ensuring a more appropriate and consistent output. For example, by recognizing that the topic in the prompt is a birthday celebration, the controllable spread model 102 can tailor its response to match the intended subject (e.g., showing a birthday cake, candles, presents, etc.), resulting in a more accurate and meaningful output.

[0032]

[0042] In the case of text prompts, topic recognition may involve analyzing the text to extract key information representing the subject and / or theme in question. In some embodiments, topic recognition and constraint map 106 include a natural language processing (NLP) module configured for NLP topic extraction (such as keyword extraction (named entity recognition and / or topic modeling)). These methods help identify important keywords, entities, or topics within the prompt.

[0033]

[0043] With respect to image prompts, topic recognition may involve analyzing visual content to understand objects, scenes, entities, and / or concepts depicted within the image. In some embodiments, topic recognition and constraint map 106 include a visual processing (VP) module configured for object detection, scene recognition, image capture, and / or other techniques for extracting relevant information from visual input.

[0034]

[0044] With respect to speech prompts containing voice / speech data, topic recognition may be involved in applying automatic speech recognition (ASR) techniques regardless of subsequent NLP methods for extracting relevant information from the prompt. In some embodiments, topic recognition and constraint map 106 include ASR modules configured for: converting voice input (e.g., spoken language) to text; preprocessing steps for cleaning and normalizing the text data (e.g., this may involve removing punctuation, converting text to lowercase, and handling any specific text challenges related to the speech conversion process); and topic modeling (such as Latent Dirichlet Allocation (LDA) and Non-negative Matrix Factorization (NMF)) for discovering intrinsic themes and / or topics within the converted text.

[0035]

[0045] Similar techniques may be used for topic recognition of other input modalities, and those provided herein are merely examples of the breadth of available techniques. Once a topic is recognized, it can be supplied as input to the controllable diffusion model 102.

[0036]

[0046] In some embodiments, the topic recognition and constraint map 106 includes a module configured to identify one or more constraints in the image query 104. These constraints provide additional information and / or limitations that must be observed when generating the output, and help ensure that the generated results will satisfy specific requirements or exhibit desired characteristics. These constraints may include, for example, two-dimensional (2D) and three-dimensional (3D) object boundaries and pose skeletons. Other constraint types are possible.

[0037]

[0047] Object boundaries (both 2D and 3D) can be used as constraints to induce the generation of images having a specific shape or contour. For example, if a user wants to generate an image of a car, the user can provide the car's rough contour or boundary as a constraint. In some embodiments, a controllable diffusion model 102 can extract this constraint and then utilize it to generate an image of a vehicle that matches the defined boundary.

[0038]

[0048] A pose skeleton represents the internal structure or arrangement of body parts within an image (e.g., a figure of a person). The pose skeleton can define the relative positions and orientations of various body parts (e.g., joints and limbs) of the figure within the image. By extracting the pose skeleton as a constraint, the controllable diffusion model 102 can generate an image that conforms to a defined pose.

[0039]

[0049] Other constraint types include, for example, style constraints, contextual constraints, spatial constraints, and color constraints. Style constraints can define a specific style, aesthetics, etc., that the user wants the generated output to present. Contextual constraints capture the desired context to influence the generation process. Contextual constraints may include factors such as time of day, weather conditions, and the presence of specific objects in a particular area of ​​the output image. Spatial constraints relate to the spatial relationships between objects in an image. For example, a spatial constraint might define the relative positions of two objects in a scene (e.g., the chair is placed to the right of the person). Color constraints define the colors used in the generated output. This may include extracting not only specific colors from prompts but also a color palette, dominant colors, and color distribution.

[0040]

[0050] Constraint identification may involve recognizing and extracting appropriate constraints from a prompt and may vary depending on the type of constraints used. For example, constraint identification may involve analyzing explicit annotations, understanding natural language descriptions, processing visual elements within a prompt, and so on. Once constraints are identified using topic recognition and constraint map 106, they can be incorporated into the generation process of the controllable diffusion model 102.

[0041]

[0051] In some embodiments, the controllable diffusion model 102 receives identified topics and arbitrary constraints from the topic recognition and constraint map 106. In some embodiments, the controllable diffusion model 102 is configured to generate a generated image 108 from the topics and constraints identified in the image query 104.

[0042]

[0052] In some embodiments, the controllable diffusion model 102 further receives user input 110. For example, the controllable diffusion model 102 may receive user input 110 via an interactive widget of dynamic image frames (see Figure 2). User input 110 may include, for example, text and / or other input modalities that represent and / or specify additional constraints on the generated image 108. For example, user input 110 may include the text "Add more trees," which, in combination with an image query 104 containing a scene with some trees near a lake, can cause the controllable diffusion model 102 to generate a generated image 108 with more trees near a lake. Implementations of user input 110 and the interactive midge are discussed in more detail with respect to Figure 2.

[0043]

[0053] In some embodiments, the user input 110 is passed directly to the controllable diffusion model 102. In some embodiments, the user input 110 is passed to the controllable diffusion model 102 via topic recognition and constraint map 106. For example, topic recognition and constraint map 106 may identify one or more constraints in the user input 110 and pass those constraints to the controllable diffusion model 102. In some embodiments, the user input 110 is passed directly to the controllable diffusion model 102 and also passed as additional constraints extracted by topic recognition and constraint map 106.

[0044]

[0054] Figure 2 depicts a block diagram 200 for utilizing a controllable diffusion model 102 for dynamic image search according to one or more embodiments of the present invention. As shown in Figure 2, the controllable diffusion model 102 (see Figure 1 for additional internal details) may be incorporated within or as part of an image gallery recommendation service 202. The implementation forms of the image gallery recommendation service 202 are not intended to be particularly limited, but may include, for example, a remote or local server (or a service running on or with a server), an application accessible via a browser and / or mobile device app (e.g., a web-based application), a content management system, etc. In some embodiments, the image gallery recommendation service 202 is an application and / or service that is integrated / embedded within another service or platform (e.g., within a social media platform, within a browser search page, etc.).

[0045]

[0055] In some embodiments, the image gallery recommendation service 202 is accessed by a client device 204. The client device 204 may include, but is not intended to be particularly limited, personal computers (desktops, laptops, e-readers, etc.), smartphones, tablets, smart home devices, wearable devices (smartwatches, fitness trackers, etc.), smart TVs, streaming devices, game consoles, headsets (virtual reality, augmented reality, etc.), and / or any other type of device used for consumer access to an information stream.

[0046]

[0056] In some embodiments, a client device 204 submits an image query 104 to an image gallery recommendation service 202 and receives one or more gallery images 206 in response. In some embodiments, the gallery images 206 are supplied from a gallery module 208. The gallery module 208 may be incorporated into or in cooperation with the image gallery recommendation service 202. In some embodiments, the gallery module 208 searches an image database 210 for one or more gallery images 206. In some embodiments, the image database 210 contains a collection of images, and the gallery module 208 is configured to select a subset of the collection of images in response to the image query 104. In some embodiments, images in the image database 210 are tagged for searching or otherwise associated with metadata. For example, an image of a horse may be tagged with "animal", "horse", etc. In this way, the image gallery recommendation service 202 may provide appropriate gallery images 206 to the image query 104. For example, the image gallery recommendation service 202 can search for various images of paintings from the image database 210 in response to the image inquiry 104, "Show me a painting."

[0047]

[0057] In some embodiments, the client device 204 includes a user interface 212 configured to display an image gallery 214. In some embodiments, the client device 204 and / or the image gallery recommendation service 202 configure the image gallery 214 to display gallery images 206 graphically within the user interface 212. In this way, the image gallery recommendation service 202 may provide the user with one or more recommended images in the context of image search.

[0048]

[0058] In some embodiments, a client device 204 submits an image query 104 to an image gallery recommendation service 202 and, in response, receives a generated image 108 along with one or more gallery images 206. In some embodiments, the client device 204 and / or the image gallery recommendation service 202 configure an image gallery 214 to graphically display the generated image 108 along with the gallery images 206. In some embodiments, the image gallery 214 includes a dynamic image frame 216 that displays the generated image 108. In some embodiments, the user interface 212 and / or the dynamic image frame 216 include an interactive widget 218. In some embodiments, the interactive widget 218 includes a user-interactive field and / or buttons in which the user can provide user input 110. In this way, the image gallery recommendation service 202 can provide a more dynamic image exploration experience, as will be described in more detail herein.

[0049]

[0059] The user-interactive fields and / or buttons of the interactive widget 218 are not intended to be particularly limited. In some embodiments, the interactive widget 218 includes a text input field (see Figure 3). The text input field allows the user to directly input user input 110 (e.g., text prompts or commands) into the dynamic image frame 216. The user may type in keywords, descriptions, and / or specific requests to guide the controllable diffusion model 102 into the image generation process. In some embodiments, the interactive widget 218 includes one or more drop-down menus. The drop-down menus may provide a list of predetermined options for the user to select to fine-tune the user's image. These options may include, for example, predetermined categories, styles, and / or other attributes that the user can select to fine-tune their prompts (image queries 104). For example, the drop-down menus may include optional options such as "decolorize," "new style," etc. In some embodiments, the interactive widget 218 includes one or more checkboxes. Checkboxes may be used to allow the user to select one or more optional selectors from a given list and to define several preferences, constraints, and / or features to be included in the generated image 108. For example, a checkbox may include an optional selector to hold a specific feature / element discovered by topic recognition and constraint map 106 (e.g., checking this optional selector to hold a person in the background of the image). In some embodiments, the interactive widget 218 includes one or more sliders. The sliders may allow the user to adjust values ​​within a range by dragging the slider handles. The sliders may be used to capture ongoing preferences or numerical constraints (e.g., to control the level, intensity, and / or size of any feature in the generated image 108). For example, the sliders may allow the user to dynamically scale (increase or decrease the size of) an object (e.g., the moon, a building, etc.) in the generated image 108. In some embodiments, the interactive widget 218 includes a color picker.A color picker allows the user to select a specific color by selecting from a color palette and / or by selecting a color code, and can help define the color preferences or constraints of the generated image 108. In some embodiments, the interactive widget 218 includes a drawing or sketching interface or canvas. The drawing or sketching interface may allow the user to generate or modify visual input within the dynamic image frame 216 by using a pen, mouse, touch input, and / or stylus. For example, the user may draw object outlines, sketch poses, and provide other visual cues within the interactive widget 218. In some embodiments, the interactive widget 218 includes one or more rating buttons to allow the user to express their preferences or opinions on a numerical or relative scale. The rating buttons allow the user to rate specific attributes of the generated output, such as quality, style, and appropriateness. For example, the interactive widget 218 may include “I like this” and “I don’t like this” rating buttons to further refine the generated image 108.

[0050]

[0060] In particular, the interactive widget 218 can be configured to receive a wide variety of input types. In some embodiments, the interactive widget 218 may receive one or more (or even a combination of) multimodal inputs, which may include, but are not limited to, the following data: text data (e.g., natural language text, documents, transcripts, etc.), image data (e.g., visual representations, drawings, sketches, photographs, etc.), video data (e.g., a sequence of images with or without sound), audio data (e.g., audio recordings, music, speech, and other forms of audio signals, etc.), sensor data (e.g., data collected from sensors such as temperature sensors, accelerometers, GPS devices, and environmental sensors), gestures (e.g., physical motion and gestures captured via devices such as motion sensors and depth cameras), metadata Data (e.g., descriptive and / or contextual information related to other modalities such as timestamps, user demographics, and location data), structured data (e.g., tabular data including numerical data, categorical variables, relational databases, etc., or otherwise structured data formats), emotional data (e.g., information related to emotional states, expressions, and feelings that can be expressed and inferred through text, voice, tone, facial expressions, etc.), biometric data (e.g., personal physical and physiological data such as fingerprints, iris scans, heart rate, and electroencephalogram patterns), and social data (e.g., data related to social interactions, social networks, and social graphs that capture interpersonal connections, relationships, and communication patterns, etc.).

[0051]

[0061] In some embodiments, the interactive widget 218 may receive so-called magic wand input, which refers to a user interface technique that allows the user to graphically select or otherwise indicate specific features and / or regions within a more or less desirable image. In this case, the magic wand style user input may be used to guide the fine-tuning or updating process of the controllable diffusion model 102 to highlight and / or remove certain features within the generated image 108. In some embodiments, the user may use a brush tool or other type of selection tool within the dynamic image frame 216 to mark regions or features of the image (e.g., the initial generated image 108) that the user wants to enhance, remove, or otherwise modify. Effectively, the marked regions or features act as guidance signals for the controllable diffusion model 102 (indicating areas to be highlighted or suppressed during the updating process). Leveraging the magic wand style technique in this manner allows the user to have more precise control over the overall output of the controllable diffusion model 102, enabling better customization or personalization of the generated image 108 based on the user's specific preferences.

[0052]

[0062] In some embodiments, the interactive widget 218 may receive a combination of multimodal inputs. For example, in some embodiments, the interactive widget 218 may receive not only text data but also magic wand style selections. For example, consider a generated painting of a yacht (or motorboat, etc.) on a river. In some embodiments, the user may use the interactive widget 218 to provide not only a magic wand selection of riverbank areas adjacent to the river but also text inputs such as "add trees" or "add buildings," etc. In some embodiments, the controllable diffusion model 102 may update the generated image 108 by using the inputs received via the interactive widget 218. Continuing this example, the painting of the yacht may be modified to add (or remove) trees and buildings adjacent to the river. In another example, the user may use the magic wand to surround features such as clouds in the painting with the text input "larger" (or by manipulating a size slider) to have the controllable diffusion model 102 modify (enlarge) the clouds. Other combinations of multimodal inputs are possible (e.g., gesture data combined with voice data, text data with biometric data, text data, magic wand input and gestures), and all such configurations are within the scope intended of this disclosure.

[0053]

[0063] A comprehensive list of all interactions and configurations of the interactive widget 218 is omitted for clarity. However, it should be understood that the provided example is a general illustration of dynamically generated image processing between the user and the interactive widget 218 and the image gallery provided by the controllable diffusion model 102. Other configurations (such as the type of user prompt, the selection of multimodal inputs, the range of inputs, the number of nested fine-tuning inputs, the type of interactive button, sliders, dialog boxes, etc.) are possible using the interactive widget 218, and all such configurations are within the intended scope of this disclosure.

[0054]

[0064] Figure 3 illustrates an exemplary image gallery 214 according to one or more embodiments of the present invention. The image gallery 214 may be presented to the user within a user interface (e.g., user interface 212 in Figure 2). As shown in Figure 3, the image gallery 214 may include an image query 104 (here, the string "painting"), one or more gallery images 206 (here, a collection of paintings), and dynamic image frames 216. For the sake of ease of discussion alone, the image gallery 214 is shown having a single dynamic image frame 216 and a specific number (here, 11) and arrangement of gallery images 206, and is therefore not intended to be particularly limited. The image gallery 214 may include any number of dynamic image frames 216 and gallery images 206 arranged as desired. In some embodiments, the gallery images 206 and generated images 108 are retrieved and / or generated in response to the image query 104 by using an image gallery recommendation service 202 (see Figure 2). In some embodiments, the gallery images 206 are retrieved from a database (e.g., an image database 210). In some embodiments, the generated image 108 is dynamically generated using a controllable diffusion model 102 (see Figures 1 and 2).

[0055]

[0065] In some embodiments, the dynamic image frame 216 includes a generated image 108 (here, a sketch of a yacht near a building) and an interactive widget 218 (here, a selectable button with pre-generated text "Create a painting for me"). The configurations of the dynamic image frame 216 and the interactive widget 218 are shown for illustrative purposes only and may include any number of additional embodiments or features described herein (e.g., sliders, canvas areas, checkboxes, pull-down menus, etc.).

[0056]

[0066] In some embodiments, the pre-generated text for the interactive widget 218 may be generated from the image query 104. In some embodiments, the image query 104 may be provided to an image gallery recommendation service 202 having topic recognition and constraint map 106 configured to identify the topic in question within the image query 104 (for example, by using NLP as described earlier). In some embodiments, the pre-generated text includes the identified topic. For example, for the image query 104 of "painting", the interactive widget 218 may include the pre-generated text "Please create a painting for me" (as shown).

[0057]

[0067] Figure 4 depicts the exemplary image gallery 214 of Figure 3 after a user clicks or otherwise selects an interactive widget 218 having the pre-generated text “Create a painting for me” according to one or more embodiments of the present invention. As shown in Figure 4, the generated image 108 in the dynamic image frame 216 has been transformed into a new dynamically generated painting. In some embodiments, the new dynamically generated painting includes a more fleshed-out image based on the previous painting sketch. It should be observed that “the painting still includes a yacht near a building, but additional details, textures and elements have been added to provide a more finished look by reassembling those of the actual painting.”

[0058]

[0068] In some embodiments, the pre-generated text of the interactive widget 218 (see Figure 3) can be overwritten by user input 110. For example, the user can enter the string "A style like the Sistine Chapel by Michelangelo" into the interactive widget 218 (e.g., via a text field).

[0059]

[0069] Figure 5 depicts the exemplary image gallery 214 of Figure 4 after the user has entered a user input 110 according to one or more embodiments of the present invention into the interactive widget 218. As shown in Figure 5, the generated image 108 in the dynamic image frame 216 has now changed again to a new version of the painting reworked in the style of Michelangelo. In some embodiments, the input 110 is provided to a controllable diffusion model 102 in combination with an image query 104 (see Figures 1 and 2). In some embodiments, the controllable diffusion model 102 generates an updated generated image 108 in response to receiving the input 110 via the interactive widget 218. As further shown in Figure 5, the user input 110 has changed (now to the string "Change to Van Gogh style").

[0060]

[0070] Figure 6 depicts the exemplary image gallery 214 of Figure 5 after the user has entered a new user input 110 according to one or more embodiments of the present invention into the interactive widget 218. As shown in Figure 6, the generated image 108 in the dynamic image frame 216 has now transformed again into a new version of the painting, now reworked in the style of Van Gogh (here by utilizing the style of the oil painting "Starry Night").

[0061]

[0071] As further shown in Figure 6, user input 110 has changed (now to the string "Make Bigger"). Furthermore, the dynamic image frame 216 now includes a magic wand input 602 and / or other types of graphic tools for selecting or otherwise indicating specific features and / or regions within the image, as previously described herein. In some embodiments, the magic wand input 602 and user input 110 (which may include the magic wand input 602) work together to fully define the user's intent. For example, the phrase "Make Bigger" is inherently ambiguous regarding which feature of the generated image 108 is being referred to. However, in combination with a graphic selection (via the magic wand input 602) of the moon feature in the upper right corner of the generated image 108, the intended feature is clear. The cooperation between multimodal inputs within the interactive widget 218 is not intended to be particularly limited, and other combinations are therefore possible. For example, interactive widget 218 may include a slider (not shown separately) that, when combined with a selection of the moon via magic wand input 602, can progressively and continuously increase or decrease the size of the moon.

[0062]

[0072] Figure 7 depicts the exemplary image gallery 214 of Figure 6 after the user has entered a new user input 110 and a magic wand input 602 according to one or more embodiments of the present invention into the interactive widget 218. As shown in Figure 7, the generated image 108 in the dynamic image frame 216 has been fine-tuned. Observe that the painting remains largely unchanged (still in the style of Van Gogh's "Starry Night"), but the moon in the upper right corner has been made larger.

[0063]

[0073] As further shown in Figure 7, user input 110 was replaced with new pre-generated text (here, "rather like this"). In some embodiments, the image gallery recommendation service 202 may infer, based on the number of sequential inputs and selections by the user, that "the finely tuned generated image 108 of the result in Figure 7 is of particular interest to each user." In this way, the image gallery recommendation service 202 may adjust the pre-generated text as a result of sequential user-system interaction. Although omitted for clarity, by selecting "rather like this," the interactive widget 218 may result in changing any of the dynamic image frames 216 and / or gallery images 206 to additional paintings created in the finely tuned style of generated image 108. For example, the new images may include various versions of generated image 108 with varying moon sizes.

[0064]

[0074] Figure 8 depicts an exemplary image gallery 214 according to one or more embodiments of the present invention. The image gallery 214 may be presented to the user in a user interface (e.g., user interface 212 in Figure 2) in a manner similar to that discussed with respect to Figures 3-7. However, in contrast to the image gallery 214 shown in Figure 3, the image query 104 is blank. This may occur, for example, when the user first accesses the image gallery recommendation service 202 (e.g., via the “Image” icon under the image query 104). This may also occur during interaction with the image gallery recommendation service 202.

[0065]

[0075] In any case, it should be observed that the image gallery 214 may still contain one or more gallery images 206 (here a collection of various images such as birds, cities, artwork, etc.) and one or more dynamic image frames 216. The one or more dynamic image frames 216 may include, for example, a sketch of a cat and a sketch of a room. In scenarios where the image gallery recommendation service 202 does not benefit from the image query 104, image recommendations may still be made by using available information related to the user. In some embodiments, the user may be identified via a user session identifier (ID), device ID, account ID, etc. Once identified, gallery images 206 are retrieved from a database (e.g., image database 210) based on known and / or inferred information related to the user.

[0066]

[0076] It should be noted that the term “identified user” as used herein does not necessarily mean the precise identification of an individual, but rather the relative identification of various otherwise useful user characteristics (i.e., non-personal information) of the types of images the user may be interested in. For example, in some embodiments, the image gallery recommendation service 202 may recommend one or more gallery images 206 and one or more generated images 108 based on the user’s available identifying and / or non-identifying information. User information may include, for example, the user’s browsing history, the user’s previously indicated preferences (i.e., conventional image selections and prompts), the user’s location / country (inferred, for example, via metrics tied to the client device 204 and / or network used to access the image gallery recommendation service 202), the user’s preferred language, and / or various other usage metrics (what types of images the user typically saves, likes, or shares, etc.).

[0067]

[0077] In some embodiments, the image gallery recommendation service 202 may score each image from a pool of available images in the initial population of the image gallery 214. In some embodiments, the image gallery recommendation service 202 may select any number of highest-scoring images from the gallery images 206. Images may be scored according to any predetermined criteria (e.g., by a coherence metric (e.g., distance measure) for one or more characteristics of the user). The use of distance measures (e.g., Euclidean distance, Tanimoto distance, Jaccard similarity coefficient, etc.) to qualify coherence between object features is known, and therefore any preferred process may be used. For example, the image gallery recommendation service 202 may score images of birds highly and provide lower scores to photographs of construction sites for a user known to have a preference for animals. Other predetermined criteria (e.g., scoring images overall or partially according to a commercial metric) are possible. For example, the image gallery recommendation service 202 may rate images related to advertising partners and / or images with relatively high impression profitability higher, and may give lower scores to pictures related to market competitors and / or images with relatively low impression profitability.

[0068]

[0078] Figure 9 depicts the exemplary image gallery 214 of Figure 8 after the user has selected the interactive widget 218 “Draw Me a Modern Room” according to one or more embodiments of the present invention. As shown in Figure 9, the generated image 108 in the upper right dynamic image frame 216 has changed to a new dynamically generated drawing of a modern room. In some embodiments, the new dynamically generated drawing of a modern room includes a more fleshed-out image based on the previous drawing sketch. It should be observed that “the new drawing of a modern room still includes bookshelves, a bed, a desk and a chair, but additional details, textures and elements have been added to provide a more complete look by reassembling those of an actual room.”

[0069]

[0079] Furthermore, observe that the generated image 108 in the lower left dynamic image frame 216 has also changed. This generated image 108 now shows an island scene with a yacht and a sunset (the cat sketch has been replaced). In some embodiments, the image gallery recommendation service 202 may infer that "the user was not interested in generating an image of an animal (another optional choice presented in Figure 8) because they decided to interact with an interactive widget 218 that has a modern room sketch instead."

[0070]

[0080] In some embodiments, the image gallery recommendation service 202 may change one or more generated images 108 within one or more dynamic image frames 216 to a new sketch with a new prompt (here, "Draw me a beach with palm trees"). In some embodiments, the new prompt / image may include the next best guess (e.g., next highest score) of an image of the user's likely interest. In some embodiments, the image gallery recommendation service 202 may update arbitrary guesses / scores based on user interaction. For example, the image gallery recommendation service 202 may reduce the score of an image containing an animal in the sense that the user ignored that arbitrary selector in a previous interaction. Note that in some situations, the new generated image 108 in the dynamic image frame 216 may not have the same dimensions (height, width) as the conventional image. In some embodiments, the image gallery recommendation service 202 may dynamically resize the dynamic image frame 216 to accommodate the size changes of each generated image 108 (as shown).

[0071]

[0081] Figure 10 shows an embodiment of a computer system 1000 that can perform various aspects of the embodiments described herein. In some embodiments, the computer system 1000 may be implemented and / or otherwise incorporated into any of the workflows, processes, systems, and services previously described herein (e.g., the image gallery recommendation service 202). In some embodiments, the computer system 1000 may be implemented on the client side. For example, the computer system 1000 may be configured to display and perform the functionality of the user interface 212. In some embodiments, the computer system 00 may be implemented on the server side. For example, the computer system 1000 may be configured to receive an image query 104 and / or user input 110, and to provide a generated image 108 and / or gallery image 206 accordingly.

[0072]

[0082] The computer system 1000 includes at least one processing device 1002 which typically includes one or more processors or processing units for performing a wide variety of functions (for example, completing any part of the dynamic image search workflow previously described with respect to Figures 1-9). The components of the computer system 1000 also include system memory 1004 and a bus 1006 which connects various system components, including the system memory 1004, to the processing device 1002. The system memory 1004 may include a wide variety of computer system-readable media. Such media can be any available media accessible by the processing device 1002 and include both volatile and non-volatile media, and removable and non-removable media. For example, the system memory 1004 may include non-volatile memory 1008 such as a hard drive, and may also include volatile memory 1010 such as random access memory (RAM) and / or cache memory. The computer system 1000 may further include other removable / non-removable, volatile / non-volatile computer system storage media.

[0073]

[0083] System memory 1004 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments described herein. For example, system memory 1004 stores various program modules that generally perform the functions and / or methodologies of the embodiments described herein. Modules or groups of modules 1012 and 1014 may be included to perform functions related to dynamic image search as previously described herein. The computer system 1000 is not limited in this respect, so other modules may be included depending on the desired functionality of the computer system 1000. As used herein, the term “module” means a processing circuit configuration which may include application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated, or grouped) and memory, combinational logic circuits and / or other suitable components that provide the functionality described herein.

[0074]

[0084] The processing device 1002 may also be configured to communicate with one or more external devices 1016, such as a keyboard, a pointing device, and / or any device (e.g., a network card, modem, etc.) that enables the processing device 1002 to communicate with one or more other computing devices (e.g., a client device 204 and / or an image gallery recommendation service 202). Communication with various devices may occur via input / output (I / O) interfaces 1018 and 1020.

[0075]

[0085] The processing device 1002 may also communicate with one or more networks 1022, such as a network adapter 1024, over a local area network (LAN), a general wide area network (WAN), a bus network, and / or a public network (e.g., the Internet). In some embodiments, the network adapter 1024 is or includes an optical network adapter for communication over an optical network. It should be understood that other hardware and / or software components, not shown, may be used in conjunction with the computer system 1000. Some examples, but not limited to, include microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, and data archive storage system data.

[0076]

[0086] Referring here to Figure 11, a flowchart 1100 for leveraging a controllable diffusion model for dynamic image search within an image gallery recommendation service is generally shown according to one embodiment. Flowchart 1100 is described with reference to Figures 1-10, but may include additional steps not depicted in Figure 11. Although depicted in a specific order, the blocks depicted in Figure 11 may be rearranged, subdivided, and / or combined in some embodiments.

[0077]

[0087] Block 1102 includes displaying an image gallery having multiple gallery images. The image gallery further includes a dynamic image frame having a generated image and an interactive widget.

[0078]

[0088] Block 1104 includes receiving user input in an interactive widget.

[0079]

[0089] Block 1106 includes generating an updated generated image by inputting the user input into a controllable diffusion model in response to receiving user input.

[0080]

[0090] Block 1108 includes replacing the generated image within a dynamic image frame with the updated generated image.

[0081]

[0091] In some embodiments, the method includes receiving an image query within a field of an image gallery. In some embodiments, a generated image is produced by inputting the image query into a controllable diffusion model. In some embodiments, a plurality of gallery images and a generated image are selected according to the degree of matching with one or more features in the image query.

[0082]

[0092] In some embodiments, the method includes determining one or more constraints in an image query. In some embodiments, the generated image is produced by inputting one or more constraints into a controllable diffusion model.

[0083]

[0093] In some embodiments, one or more constraints in the image query include at least one of a pose skeleton and an object boundary. In some embodiments, determining one or more constraints includes extracting an object boundary when a feature in the image query is one of a structural or geological feature. In some embodiments, determining one or more constraints includes extracting a pose skeleton when a feature in the image query is one of a human or an animal.

[0084]

[0094] In some embodiments, multiple gallery images are supplied from an image database. In some embodiments, one or more of the multiple gallery images are not generated images (i.e., supplied images rather than dynamically generated images from a diffusion model). In some embodiments, one or more of the multiple gallery images are previously generated images.

[0085]

[0095] In some embodiments, the interactive widget includes a text field. In some embodiments, receiving user input in the interactive widget includes receiving a text string input into the text field.

[0086]

[0096] In some embodiments, the interactive widget includes one or more of a dropdown menu, checkboxes, sliders, a color picker, a canvas interface for drawing or sketching, and rating buttons.

[0087]

[0097] In some embodiments, the interactive widget includes a canvas for magic wand input. In some embodiments, receiving user input in the interactive widget includes receiving magic wand input by graphically selecting one of a specific feature and a specific region within the generated image. In some embodiments, the interactive widget further includes a text field. In some embodiments, receiving user input in the interactive widget further includes receiving a text string input with contextual information for magic wand input.

[0088]

[0098] While this disclosure has been described with reference to various embodiments, it will be understood by those skilled in the art that some modifications are possible and that equivalents may substitute for its elements without departing from the scope of this disclosure. Various tasks and processes described herein may be incorporated into more comprehensive procedures or processes having additional steps or functionalities not described in detail herein. In addition, many modifications may be made to adapt specific circumstances or materials to the teachings of this disclosure without departing from their essential scope. Therefore, this disclosure is intended to include all embodiments that fall within the scope of this disclosure, but are not limited to the specific embodiments disclosed herein.

[0089]

[0099] Unless otherwise defined, the technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which this disclosure pertains.

[0090]

[0100] Various embodiments of the present invention are described herein with reference to the relevant drawings. The drawings depicted herein are illustrative. Many variations may exist to the diagrams and / or processes (or operations) described herein without departing from the spirit of the disclosure. For example, some actions may be performed in a different order, or some actions may be added, deleted, or modified. All these variations are considered part of the disclosure.

[0091]

[0101] The terminology used herein is for the sole purpose of describing specific embodiments and is not intended to be limiting. As used herein, unless the context explicitly indicates otherwise, the singular forms "a," "an," and "the" are intended to include the plural forms as well. It will also be understood that the terms "comprises" and / or "comprising," as used herein, specify the presence of the described features, integers, processes, actions, elements, and / or parts, but do not exclude the presence or addition of one or more other features, integers, processes, actions, elemental parts, and / or groups thereof. The term "or" means "and / or" unless the context otherwise explicitly indicates otherwise.

[0092]

[0102] The terms "received from," "receive from," "passed to," and "passed to" describe a communication path between two elements, and therefore, unless otherwise specified, do not mean a direct connection between elements without any intervening elements / connections. Each communication path can be a direct or indirect communication path.

[0093]

[0103] All means or process plus elements in the following claims are intended to include any structures, materials, or actions to function in combination with other claimed elements, such as those specifically claimed.

[0094]

[0104] For the sake of brevity, prior art relating to the creation and use of embodiments of the present invention may or may not be described in detail herein. In particular, various embodiments of computing systems and specific computer programs that implement the various technical features described herein are well known. Accordingly, for the sake of brevity, many prior implementation details are described only briefly herein or omitted without providing well known system and / or process details.

[0095]

[0105] The present invention may be a system, method, and / or computer program product at any possible level of technical detail of integration. The computer program product may include a computer-readable storage medium (or group of mediums) having computer-readable program instructions thereon for causing a processor to perform certain aspects of the present invention.

[0096]

[0106] Various embodiments are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products. It will be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0097]

[0107] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to generate a machine that generates means for performing functions / actions defined in flowcharts and / or block diagram blocks or groups of blocks, which are executed via the computer's processor or other programmable data processing device. These computer-readable program instructions may also be stored in computer-readable storage media that can instruct computers, programmable data processing devices, and / or other devices to function in a particular way, such that the computer-readable storage medium containing the instructions therein contains a product that includes instructions for performing modes of functions / actions defined in flowcharts and / or block diagram blocks or groups of blocks.

[0098]

[0108] Computer-readable program instructions can also be loaded onto a computer, another programmable device, or another device to generate a computer execution process in which a series of operations are performed on the computer, another programmable device, or another device, so that instructions executed on the computer, another programmable device, or another device perform functions / actions defined in a flowchart and / or block diagram block or group of blocks.

[0099]

[0109] The flowcharts and block diagrams in the accompanying drawings illustrate examples of the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part thereof of instructions containing one or more executable instructions for performing a defined logical function. In some alternative implementations, the functions described within a block may occur in a different order than those shown in the accompanying drawings. For example, two consecutively shown blocks may actually be executed almost simultaneously, or these blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks within the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system performing a defined function or action, or by a combination of dedicated hardware and computer instructions.

[0100]

[0110] The descriptions of the various embodiments described herein are presented for illustrative purposes only and are not intended to be exhaustive of or limitless to the disclosed forms. The embodiments have been selected and described to best illustrate the principles of this disclosure. Many modifications and variations will be obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been selected to best illustrate the principles of various embodiments, practical applications, or technical improvements beyond the art found in the market, or to enable those skilled in the art to understand the embodiments described herein.

Claims

1. Displaying an image gallery 214 containing multiple gallery images 206, wherein the image gallery 214 further includes a dynamic image frame 216 containing a generated image 108 and an interactive widget 218; The interactive widget 218 receives user input 110; In response to receiving the user input 110, the updated generated image 108 is generated by inputting the user input 110 to the controllable diffusion model 102; and A method comprising replacing the generated image 108 in the dynamic image frame 216 with the updated generated image 108.

2. The method according to claim 1, further comprising receiving an image inquiry within the field of the image gallery.

3. The method according to claim 2, wherein the generated image is generated by inputting the image query to the controllable diffusion model.

4. The method according to claim 2, wherein the plurality of gallery images and the generated image are selected according to the degree of consistency with one or more features in the image query.

5. The method according to claim 2, further comprising determining one or more constraints in the image query.

6. The method according to claim 5, wherein the generated image is generated by inputting the one or more constraints into the controllable diffusion model.

7. The method according to claim 5, wherein the one or more constraints in the image query include at least one of a pose skeleton and an object boundary.

8. The method according to claim 7, wherein determining one or more of the constraints includes extracting the object boundary when a feature in the image query includes one of structural and geological features.

9. The method according to claim 7, wherein determining one or more of the above constraints includes extracting the pose skeleton when the feature in the image query includes one of a person and an animal.

10. The method according to claim 1, wherein the plurality of gallery images are supplied from an image database.

11. The method according to claim 1, wherein the interactive widget includes a text field, and receiving the user input in the interactive widget includes receiving a text string input into the text field.

12. The method according to claim 1, wherein the interactive widget includes one or more of a dropdown menu, a checkbox, a slider, a color picker, a canvas interface for drawing or sketching, and rating buttons.

13. The method according to claim 1, wherein the interactive widget includes a canvas for magic wand input, and receiving the user input in the interactive widget includes receiving the magic wand input by graphically selecting one of a particular feature and a particular region in the generated image.

14. The method according to claim 13, wherein the interactive widget further includes a text field, and receiving the user input in the interactive widget further includes receiving a text string input having contextual information for the magic wand input.

15. A system 202 having memory 1104, computer-readable instructions, and one or more processors 1002 for executing the computer-readable instructions, wherein the computer-readable instructions are as follows: Receiving an image inquiry 104 from a client device 204 that is communicably connected to the system 202; To provide the client device 204 with multiple gallery images 206 and generated images 108 according to the degree of consistency with one or more features in the image query 104; Receiving user input 110 from the client device 204; In response to the user input 110, the updated generated image 108 is generated by inputting the user input 110 to the controllable diffusion model 102; and The updated generated image 108 is provided to the client device 204. A system 202 that controls the one or more processors 1002 to perform operations including those mentioned above.

16. The system according to claim 15, wherein the generated image is produced by inputting the image query into a controllable diffusion model.

17. The system according to claim 15, further comprising determining one or more constraints in the image query.

18. The system according to claim 17, wherein the generated image is generated by inputting the one or more constraints into the controllable diffusion model.

19. A system 204 having memory 1104, computer-readable instructions, and one or more processors 1002 for executing the computer-readable instructions, wherein the computer-readable instructions are as follows: Multiple gallery images 206 and generated images 108 are received from an image gallery recommendation service 202 which is connected to the system 204 in a communicable manner; Displaying an image gallery 214 containing the aforementioned multiple gallery images 206, and a dynamic image frame 216 having the generated image 108 and an interactive widget 218; The interactive widget 218 receives user input 110; Sending the user input 110 to the image gallery recommendation service 202; Receiving the updated generated image 108 from the image gallery recommendation service 202; and Replacing the generated image 108 in the dynamic image frame 216 with the updated generated image 108. A system 204 controls the one or more processors 1002 to perform operations including those mentioned above.

20. The system according to claim 19, wherein the interactive widget includes a text field, and receiving the user input in the interactive widget includes receiving a text string input into the text field.

Images