Optimizing cues for AI image generation for image-based internet search

Abstract

The image-based search may provide enhanced search results as compared to the text-based search. Techniques for generating images usable by a search engine include generating optimized image model cues from text-based inputs using a language model. The optimized image model cues include a more textualized description of an item described by the text-based input. The optimized image model cues are provided to a language model that generates a photo-level realistic image of the item. The search engine uses the photo-level realistic image to identify and return search results.

Description

Technical Field

[0001] The present application relates to the field of search. Background Art

[0002] Image-based Internet search allows users to search for information using images instead of text keywords. By uploading an image or linking to an image, the search engine analyzes the visual content and provides relevant results, such as similar images, product details, or information about the depicted objects and scenes.

[0003] Generative AI (artificial intelligence) is a class of machine learning models that are able to create new data samples that are similar to a given set of training data. This can include tasks such as generating text, images, or music, and is often associated with various deep learning techniques. Summary of the invention

[0004] At a high level, the present technology may involve AI techniques for optimizing hints used to generate images for image-based searches. The present technology may allow a user to enter a text-based input that is expanded by a language model to generate optimized image model hints suitable for rendering an image by an image model. The image may be used by a search engine to identify search results that are more relevant to the initial text-based input than many traditional text-based search methods provide.

[0005] To this end, a text-based input may be received at a search engine. The text-based input is provided to a language model, such as a text-based generative AI model. The language model may expand the text-based input to generate an optimized image model hint. The optimized image model hint may include additional text that is a more literal text description of the object described by the text-based input.

[0006] The optimized image model hint may be provided to the image model. The image model may generate a photorealistic image based on text within the optimized image model hint. The photorealistic image may be provided to a search engine, which outputs search results for the image. The search results may be provided to a user computing device.

[0007] This summary is intended to introduce in simplified form a selection of concepts further described in the detailed description section of this disclosure. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter. Some of the additional purposes, advantages, and novel features of the present technology will be set forth in the following description, and some will become apparent to those skilled in the art after careful reading of this disclosure or through practical study of the present technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present technology is described in detail below with reference to the accompanying drawings, in which:

[0009] Figure 1 An example operating environment is shown in which aspects of the present technology can be employed according to aspects described herein;

[0010] Figure 2 An example of generating optimized image model hints from a language model according to aspects described herein is shown;

[0011] Figure 3 An example of generating a photo-realistic image by an image model using optimized image model hints according to aspects described herein is shown;

[0012] Figure 4 An example of modified optimized image model hints modified by a language model according to aspects described herein is shown;

[0013] Figure 5 An example of using a segmentation engine to generate a segmented image according to aspects described herein is shown;

[0014] Figure 6 An example of a modified image obtained by modifying an image model using a modified optimized image model prompt according to aspects described herein is shown;

[0015] Figure 7 shows an example search using photo-realistic images according to aspects described herein;

[0016] Figure 8 shows example search results using text-based searching according to aspects described herein;

[0017] Fig. 9 shows example search results using image-based searching according to aspects described herein;

[0018] Figures 10 to 12 A block diagram is shown with various example methods for identifying items as search results according to aspects described herein;

[0019] Fig.13 A block diagram is shown with an example method for generating a segmented image to provide to an image model to modify an initial photo-realistic image according to aspects described herein;

[0020] Fig.14 A block diagram is shown with an example method for performing an updated image-based search using a modified image according to aspects described herein; and

[0021] Fig.15An example computing device suitable for implementing aspects of the described techniques according to aspects described herein is shown. DETAILED DESCRIPTION

[0022] Text-based Internet search can refer to the process of finding information on a network (e.g., the Internet) using a search engine by entering a text query. In this method, a user enters a series of words or phrases into a search system, and the search engine returns a list of items, including web pages, documents, images, or other types of files that are considered relevant to the query.

[0023] Traditionally, search engines process text-based queries to understand their intent. This can involve parsing the query, correcting spelling errors, and sometimes expanding the query using synonyms or related terms (a process known as query expansion). Search engines maintain an extended index of web pages and other online content. The processed query is used to search the index for matching results.

[0024] Search engines can use ranking algorithms to sort results based on various factors, such as relevance, page quality, and the number of inbound links. The user is presented with a sorted list of results, typically with a title, a summary of the content, and the URL (Uniform Resource Locator) of the page. The user can then click on these links to visit the web page and access the information they are searching for.

[0025] Text-based internet search has evolved significantly since its inception. Early search engines used primarily keyword matching and had a less sophisticated understanding of the context or semantics of a query. Modern search engines use complex algorithms that incorporate machine learning, natural language processing, and other advanced techniques to provide more accurate and contextually relevant results.

[0026] Text-based internet search, while useful, has limitations. A common problem is "vocabulary mismatch," where the terms used in the search query may not match the terms used in the relevant documents. This can lead to incomplete or less relevant search results. Various techniques, such as query expansion, have been developed to mitigate this problem, but they come with their own set of challenges, such as including irrelevant results.

[0027] Image-based search (sometimes referred to as "reverse image search") is a type of search that uses an image as a query. In this method, a user uploads an image to a search engine, which then analyzes the image and returns a list of items, which may include similar images or related information. Search engines may use a variety of techniques, such as feature extraction, color histograms, and machine learning algorithms, to identify patterns, shapes, and other characteristics within an image.

[0028] Traditionally, search engines process images to extract key features such as color distribution, texture, and shape. The extracted features are used to search a database of indexed images for matches. Similar to text-based searches, search engines rank the results based on a similarity metric and return the ranked results to the user computing device.

[0029] Image-based search is particularly useful when users cannot adequately describe what they are looking for in text. For example, it is often easier to identify a landmark or piece of art using an image than using a text description. Image-based search can be beneficial for users who are looking for a product but don't know the exact name or brand. They can just upload a picture of the item to find similar products.

[0030] Additionally, image-based search provides a richer contextual framework for queries than text-based search. A single image can encompass multiple elements that would otherwise require extensive text descriptions, such as furniture, colors, textures, lighting, and more. This visual complexity allows for more precise and nuanced search results.

[0031] Furthermore, images can resolve ambiguities that often exist in text queries; for example, an image of an apple immediately clarifies whether the user is referring to the fruit or the tech company. The multifaceted information contained in an image (which can include multiple objects or concepts related in a particular way) can provide a more complete understanding of the user's intent. Furthermore, images capture non-verbal elements (such as emotion, style, and atmosphere) that are difficult to convey through text or at least would require large and lengthy text queries to attempt to capture. Finally, the universality of images transcends language barriers, making them particularly useful in global contexts where text-based keywords may not translate effectively.

[0032] A significant limitation of image-based search is that relevant images must be accessible for the query. This becomes a problem in a variety of scenarios, such as when you encounter a product you like in a physical store but cannot take a photo to search online. Similarly, if you remember an image or scene but do not have a copy, text-based search becomes your only option. Furthermore, technical constraints may hinder the usefulness of image-based search; for example, limited device capabilities or poor internet connectivity may prevent users from uploading images, limiting them to text-based queries.

[0033] Search technology, including image-based search, is integral to the technical functionality of the Internet, primarily due to the sheer volume and variety of content available online. The Internet hosts billions of web pages, images, videos, and other forms of data. Without sophisticated search and ranking algorithms, it would be nearly impossible for users to find relevant information in this vast ocean of content. As an example, at the time of filing this application, a search for "Kansas City Current" returned more than 1.17 billion results. This demonstrates the importance of search and ranking technology to the functionality of the Internet, as it would be impossible for users to sift through each of these results and would have to rely on search engines.

[0034] As the Internet continues to grow, so too does the complexity and diversity of queries. Text-based search may not be sufficient to satisfy all types of queries, especially those that are visually or contextually complex. This is where specialized search technologies, such as image-based search, come into play, providing an alternative way to navigate the digital landscape and allowing search engines to provide users with useful results among trillions of possibilities.

[0035] In essence, search technologies are not just useful features, but essential to the Internet's function as a useful resource. They serve as the organizing principle that makes the Internet accessible and navigable, transforming vast amounts of data into a structured and user-accessible environment.

[0036] The techniques provided by the technology disclosed herein improve image-based searching. The present technology also provides a mechanism that allows users to take advantage of the convenience of text-based input while also providing the benefits of image-based searching.

[0037] An example method that can have some of the benefits of both text-based and image-based searches optimizes text input to generate images suitable for performing image searches. For example, a user can enter text-based input on a computing device (e.g., a desktop computer, a mobile phone, or even a smartwatch). The text-based input is provided to a language model, such as a text-based generative AI model.

[0038] The language model, in turn, outputs an optimized image model hint. An optimized image model hint is an extended hint that is optimized to provide an additional textual description of the text-based input. For example, if a user enters a text-based input to search for "black dress," the language model may output an optimized image model hint that includes a textual description of a black dress, such as a description including a location and a description of attributes of the dress, including attributes not listed in the text-based input. Furthermore, by using many language models, a user may switch back and forth during a conversation, interacting with a language model until a language model provides an optimized image model hint that describes an image that is close to the user's intent.

[0039] In this way, the language model is optimizing the text-based input for use by the image model, which will generate an image (e.g., an image of a black dress). For example, a diffusion model can be used to generate an image in response to a text description. In this case, the image model generates the image based on a textual description of the optimized image model prompt. The optimized image model prompt helps constrain the image model, making it more likely to produce relevant images based on the query. For example, a detailed description of a dress (including location and description of its various attributes) will constrain the image model more than the initial text-based input of just "black dress", and the resulting image is more likely to return a useful search result.

[0040] Once the image model has generated an image based on the optimized image model prompt, the image can be provided to a search engine for image-based searching. The search engine uses the image to identify and return search results.

[0041] Advantageously, this technique, as well as other techniques that will be further described, help address many of the problems inherent in conventional text-based and image-based search approaches while still gaining some of the benefits of these approaches, such as the convenience of text-based input and the robust search functionality provided by image-based search techniques.

[0042] For example, the method allows computing devices without cameras to have the benefits of image-based search. Due to several constraints, smaller devices (such as smart watches) are generally more suitable for text input rather than image input. Limited screen size makes viewing and selecting images challenging, while the reduced processing power and battery life of the device make text-based search more effective. In addition, the user interface on such devices is optimized for fast and simple interactions, making text input more practical. In addition, not all smart watches and small computing devices have cameras, and even devices with cameras may provide lower resolutions, which limits the effectiveness of image-based search. However, the method described in this article allows users to use such devices and still obtain the benefits of image-based search, because text-based input can be provided to these small devices, and text-based input can be converted into images for image-based search.

[0043] As described above, in some cases, network availability may be limited. In such cases, a device may not have the connection speed or bandwidth to upload images for image-based search. The techniques provided by the disclosed technology allow a user to enter text (including relatively small amounts of text) as the language model optimizes that text for image generation. The data transfer required to upload text is much less than the data transfer required to upload an image. This allows devices operating on poor connectivity to effectively use and gain the benefits of image-based search.

[0044] In addition, the technology provided herein can help improve some aspects of the computing device itself. For example, conventional methods (for example, strict text-based search) may require users to filter multiple pages of results, and may even require users to click on multiple websites to find the information they need. Compared with the disclosed technology (which can quickly produce desired results, thereby saving data transmission) of further optimizing and improving the search results provided to users, this may consume more bandwidth. In addition, because compared with conventional methods, the method provided herein can provide more targeted search results, this can make the buffer of equipment store less irrelevant pages, thereby obtaining more efficient use of the memory resources of equipment.

[0045] Many of the techniques described are not well known, routine, or conventional in the relevant art. For example, it is not considered conventional practice to use query expansion techniques for optimizing text-based input to generate hints that can be used to generate images. It is also not considered conventional to combine expanding text-based input into a textual description of an image of an item (which is used to generate an image) and performing an image-based search.

[0046] It will be appreciated that the previously described methods are merely examples that can be practiced in accordance with the following description and are provided to more easily understand the present technology and realize its benefits.Additional examples are now described with reference to the accompanying drawings.

[0047] Reference now Figure 1 , provides an example operating environment 100 in which aspects of the present technology may be employed. The operating environment 100 includes a server 102, a computing device 104, and a database 106 in communication with a search engine 110 via a network 108, as well as other components or engines not shown.

[0048] The database 106 generally stores information, including data, computer instructions (e.g., software program instructions, routines, or services), or models used in embodiments of the described technology. For example, the database 106 may store computer instructions for implementing functional aspects of the search engine 110. Although described as a single database component, the database 106 may be embodied as one or more databases or may be embodied in the cloud.

[0049] Network 108 may include one or more networks (eg, a public network or a virtual private network [VPN]), as shown as network 108. Network 108 may include, but is not limited to, one or more local area networks (LANs), wide area networks (WANs), or any other communication network or method.

[0050] In general, server 102 is a computing device that implements functional aspects of operating environment 100, such as one or more functions of search engine 110 to facilitate image-based searching by optimizing text input. One suitable example of a computing device that can be used as server 102 is described with respect to Fig.15 The computing device 1500. In an embodiment, the server 102 represents a back-end or server-side device.

[0051] The computing device 104 is typically a computing device that can be used to perform image-based searches using text input. For example, the computing device 104 can receive input from an input component corresponding to a text-based input, which can be transmitted to the search engine 110 for use by its components. The computing device 104 can receive and display items as search results from the search engine 110.

[0052] and Figure 1 As with other components of the , computing device 104 is intended to represent one or more computing devices. One suitable example of a computing device that can be used as computing device 104 is described with respect to Fig.151500. In an embodiment, computing device 104 is a client-side or front-end device. In addition to server 102, computing device 104 may also implement functional aspects of operating environment 100, such as one or more functions of search engine 110. It will be understood that some embodiments of the present technology will include client-side or front-end computing devices, back-end or server-side computing devices, or both that perform any combination of functions in search engine 110 and other functions or combinations of functions.

[0053] The search engine 110 typically receives text-based input and, in response, provides search results for items. The search results may be included in a search engine results page (SERP). Items may include any of web pages, images, videos, infographics, articles, research papers, and other types of files, and may include associated descriptions and hyperlinks. The search engine 110 may be configured for general Internet or network searches, or may be configured as a search engine for searching a specific database or website, such as a search engine that returns a list of items on an e-commerce platform.

[0054] Broadly speaking, the search engine 110 can use functions alone or in coordination with other components or systems to receive search queries, and in response, provide SERPs for items. In doing so, the search engine 110 can optimize the text-based input provided as the search query to generate optimized image model prompts. The optimized image model prompts can be used to generate photo-realistic images. Then, the search engine 110 can use the photo-realistic images to identify items and return these items as search results to a computing device, such as the computing device 104. To this end, the search engine 110 can use an optimized image model prompt generator 112, a photo-realistic image generator 114, a segmentation engine 116, and an image-based searcher 118. Note again that the search engine 110 is intended to be an example suitable for implementing the present technology. However, other arrangements and architectures of components and functions for optimizing text-based input to generate search images are intended to be within the scope of the present disclosure and understood by those who practice the present technology.

[0055] Typically, the optimized image model hint generator 112 generates optimized image model hints based on text-based input. To generate optimized image model hints, the optimized image model hint generator 112 uses a language model 120. The text-based input is provided to the language model 120 as input, and in response, the language model 120 outputs optimized image model hints.

[0056] In one aspect, the language model 120 is a machine learning model, such as a generative AI model. The generative AI model can be a text-based model because it outputs text that forms an optimized image model prompt. The optimized image model prompt expands the text-based input into a text description. The text description is a text-based description of an image (e.g., a photorealistic image to be described). The optimized image model prompt may include a textual description of the image corresponding to the object of the text-based input. For example, the language model 120 may expand the text-based input to include an additional text-based description of the image. When expanding the text-based input (e.g., the text-based input 202), the language model 120 may include additional attributes corresponding to the object of the text-based input, including features or characteristics and textual descriptions of these attributes.

[0057] Figure 2 An example use case of the language model 120 is provided. Here, a text-based input 202 is input to the language model 120, and the resulting output is an optimized image model prompt 204. The text-based input 202 is an example of a text-based input that can be received as a search query from a computing device (e.g., computing device 104). In this example, the text-based input 202 is "spring pink cocktail dress with lace trim." This is the input to the language model 120, which expands the text-based input 202 to generate a textual description of an image of an object that includes the text-based input 202 (in this example, "spring pink cocktail dress with lace trim"). The resulting optimized image model prompt 204 is "Spring pink cocktail dress. The dress should be made of a lightweight, breathable fabric such as chiffon or silk. The color should be a soft, pale pink to evoke the spirit of spring. The silhouette should be A-line, slightly cinched at the waist and flared to the hem, which should end just above the knee. Incorporate lace trim along the hem and short cap sleeves. The neckline should be a modest round neck. The lace trim should be a slightly lighter shade than the pink of the main fabric to create a harmonious contrast. The dress should look elegant and comfortable, perfect for a spring cocktail party."

[0058] The language model 120 may be trained to generate optimized image model prompts based on text-based input. Figure 1, the language model 120 can be trained at least in part using the project corpus 124. The project corpus can include projects and project descriptions corresponding to these projects. That is, the project corpus 124 can include a broad database of general descriptive information of text. For example, the project corpus 124 can include a variety of sources that can produce a broad understanding of human language, such as websites, books, Wikipedia, scientific articles, news media, technical manuals, movie scripts, programming codes, educational materials, and other texts. The language model 120 can be trained on a data set including these texts.

[0059] For example, for a model including a transformer architecture with multiple layers and millions of parameters, the training goal is to minimize the difference between the predicted value and the actual value of the next word under a given word sequence of training data (e.g., general text material). This can be achieved using a loss function (e.g., cross entropy). The parameters of the language model 120 can be optimized using, for example, a gradient-based optimization algorithm (e.g., Adam) or other stochastic optimization. This produces a pre-trained base model, which can be used as the language model 120 in some cases.

[0060] In one aspect, the language model 120 can be fine-tuned or further trained on a specific document category, which can be based on the use case of the language model 120. Fine-tuning can be performed using an algorithm similar to the algorithm described when training the initial base model. For example, in a broad use case (e.g., general Internet search), the language model 120 can be trained on the project corpus 124. In the case where the language model 120 is used for a specific task or a specific context, the language model 120 can be trained or fine-tuned using a corpus of documents related to the task or concept. As an example, for use by an e-commerce website, the language model 120 can be trained based on the items sold on the e-commerce website and the item descriptions of the items. By doing so, the language model 120 can better contextualize the text-based input provided in the context of searching for a list of items on the e-commerce website. The fine-tuned model can also be used as the language model 120.

[0061] It should be noted that the aforementioned training methods involving pre-training and fine-tuning are provided only as illustrative examples and are not intended to limit the range of possible training methods that can be employed. Other methods may include reinforcement learning based on human feedback (RLHF); transfer learning based on related tasks, in which the model is initially trained on a task similar to but not identical to the target task, and then fine-tuned on a specific task of interest; multi-task learning, in which the model is trained to perform multiple tasks simultaneously, sharing representations between them to improve overall performance. These training methods can be stand-alone schemes or integrated with other techniques to create more robust and versatile models, and new methods may be incorporated as they are developed.

[0062] Based on this training, the language model 120 may be adapted to determine the type of information required to generate an optimized image model prompt. That is, if additional information is required to generate an optimized image model prompt after providing the text-based input, the language model 120 will prompt the user computing device with the specific information required. This may be done via the optimized image model prompt generator 112 in the form of a dialog or back-and-forth information gathering session between the language model 120 and the user computing device.

[0063] In various aspects, contextual commands may be provided to the language model 120 to generate optimized image model hints, such as the purpose of providing the text-based input (e.g., “You are generating hints for a diffusion model”), the level of detail required (e.g., “Provide enough detail so that someone can draw the text-based input”), the knowledge lens (e.g., “Pretend you are a clothing designer”), and other contextual commands. These commands may be provided within the text-based input. In some cases, these commands are determined based on the context of the text-based input and provided by the search engine 110. For purposes of providing an example, if the text-based input includes a search for dresses, a command may be provided that indicates that the language model 120 should generate optimized image model hints like a tailor. Thus, more broadly, the search engine 110 may instruct the language model 120 to consider itself a manufacturer of the object of the text-based input. In these examples and more, the language model 120 contextualizes this information and formats the response (e.g., an optimized image model hint) accordingly. Commands such as these may be used to augment a text-based description of an object included within a text-based input, where the text-based description includes a text-based graphical description of the object.

[0064] As described above, the user can have a dialogue with the language model 120. In various aspects, the user can request modifications to the image, as will be discussed further. For example, the user can view an image generated by the language model and want to change the properties of the objects in the image. The text-based input can include modifications, i.e., text-based instructions for modifying the image in a particular manner. In doing so, the modification (which includes the text-based input) can be provided to the language model 120, which will modify the previous optimized image model prompt in a manner that includes the modification or an extended description thereof, thereby generating a modified optimized image model prompt. An example of this situation will be discussed further. The user can have this back-and-forth dialogue as many times as desired until the generated optimized image model prompt can be used to generate the image that the user desires to use as a search image.

[0065] Thus, as described above, the optimized image model hint generator 112 can use the language model 120 to generate an optimized image model hint based on the text-based input. It can also receive the text-based input including the modification, and modify the initial optimized image model hint based on it to generate a modified optimized image model hint, which can be used to modify the image to be used for image-based search.

[0066] In an embodiment, the modified optimized image model hint may be generated manually. For example, a text-based input may be provided to the language model 120 to generate the optimized image model hint. The optimized image model hint may be displayed at a user computing device (e.g., computing device 104). Such text-based input may be received that directly modifies the text description in the optimized image model hint, thereby generating a modified optimized image model hint that can be used by other components of the described search engine 110.

[0067] The search engine 110 typically uses a photorealistic image generator 114 to generate a photorealistic image based on the optimized image model hint. This may also include a modified optimized image model hint. As described above, the optimized image model hint generator 112 may be used to access or otherwise generate the optimized image model hint.

[0068] To generate a photorealistic image based on the optimized image model hint, the photorealistic image generator 114 may use the image model 122. That is, the image model 122 receives the optimized image model hint as input and outputs a photorealistic image based thereon that depicts the object described by the optimized image model hint.

[0069] Typically, the image model 122 is a machine learning model that can generate an image based on text input. In an example, the image model 122 is a generative AI model that receives text and, in response, outputs an image. In a specific example, the image model 122 is a diffusion model. Although generally referenced to diffusion models (or more broadly, generative AI models), it will be understood that other text-to-image models may be adopted or developed and are intended to be within the scope of the present disclosure. Some non-limiting examples may include generative adversarial networks (GANs), variational autoencoders (VAEs), and transformer models. The image model 122 may be a single AI model or a collaboration of various models that work to generate images based on text input.

[0070] In the context of the diffusion model, the training process for converting text to images can include a two-stage mechanism that first corrupts the original image by iteratively adding noise, and then reverses the process to generate new image samples based on the text input. In general, there are numerous image datasets that can be used. Some examples include Flickr30k, IMBD-Wiki, Berkeley Deep Drive, etc. The text description is encoded into a latent space using natural language processing techniques to provide conditions for the generative process. During the diffusion process, the image model 122 learns to map the potential text representation to a series of noisy image states, thereby effectively learning the transition dynamics between the text and the corresponding image. The image model 122 is trained to minimize the difference between the generated image and the actual image corresponding to the text description. As an example, mean square error, cross entropy, or other similar functions can be used as the loss function for training. The optimization is usually performed using a gradient-based algorithm (e.g., stochastic gradient descent (SGD), Adam, etc.). Once trained, the image model 122 can take the text description as input and iteratively improve the noisy image until it generates a new image that closely matches the text description, effectively converting the text to an image. This is an example method for training the image model 122 so that the image model 122 receives the text description provided by the optimized image model prompt and generates an image based on it that matches the description of the optimized image model prompt. Other training methods can be used as they are developed or can be used based on using a specific model.

[0071] Now turn to Figure 3 As shown in FIG. 1 , the image model 122 can be used to generate a photo-realistic image 302 based on an optimized image model hint 204. The optimized image model hint 204 is Figure 2 , generated by the language model 120. Generally speaking, a photo-realistic image is an image that closely resembles a photograph. It can obtain details, textures, lighting, and shadows in a way that makes it similar to an actual photograph.

[0072] In some cases, based on the resulting photorealistic image (e.g., photorealistic image 302), the user may wish to further modify the image. For example, the image depicts an object, and the object depicts attributes that are characteristics of the object. Figure 3In the example shown, the resulting photorealistic image 302 includes attributes such as waist, hem, sleeves, neckline, etc. These attributes can be modified via a modification process. This can be done as part of an iterative process to provide a photorealistic image that is close to the image envisioned or otherwise intended by the user. It should be noted that at any point in the process, the first output photorealistic image or any modified iteration thereof can be used to perform an image-based search, which will be further described.

[0073] In one aspect of the present technology, the modified image is a modification of a photograph. That is, although the modification in the described aspects is a photorealistic image, the modification process can also be performed on a photograph (e.g., a photograph taken with a camera) or a digital rendering of an image (which can be stored in various image file formats). That is, the initial image that is segmented and then modified (e.g., using the image model 122 to render the modified image) can be a photograph. Photographs include photorealistic images as well as photographs obtained using other known methods (e.g., cameras, computer editing, etc.). Once the modification is made by generating a modified image, the iterative process can continue, and if the user wishes to continue the modification, the modified image can be further used as the next input. Therefore, the method describing the use of the initial photorealistic image can also be applied to the initial photograph.

[0074] To modify the initial photorealistic image, a text-based modification may be input at a user computing device (e.g., computing device 104) and received by search engine 110. The modification may include attributes and changes to the attributes. As an example, the initial photorealistic image may be presented at a display of the user computing device. After the initial photorealistic image is generated and presented, a modification including a text-based input may be received. The modification may also be referred to as a text-based modification.

[0075] The text-based modifications may be provided to the language model 120. The language model 120 may also take as input a previously optimized image model prompt. In some cases, this is done via a back-and-forth dialogue. That is, the language model 120 may have previously generated an optimized image model prompt for generating a photorealistic image. The text-based modifications may be received based on the photorealistic image. The language model 120 modifies the previous optimized image model prompt according to the text-based modifications. The language model 120 outputs a modified optimized image model prompt that includes modifications to the attributes according to the text-based modifications.

[0076] Figure 4Here, the modification 402 is received and input to the language model 120. In this example, the language model 120 modifies the previous optimized image model hint (which in this example is Figure 2 202 ) to output a resulting modified optimized image model prompt 404. Here, modification 402 includes text describing "modify the dress to make the neckline lower." It can be seen that language model 120 outputs a modified optimized image model prompt 404 that includes additional language, such as "the neckline should be a modest round neck, low enough to increase appeal, but still remain within the range of modesty." As will be described, modified optimized image model prompt 404 can be used to modify the resulting image to generate a modified image using image model 122 using realistic image generator 114. As described above, in some embodiments, the computing device can receive a modification that directly modifies the text description within the optimized image model prompt to generate a modified optimized image model prompt.

[0077] In one aspect, the modified optimized image model hint is provided to the photorealistic image generator 114 to generate a photorealistic image according to the modified optimized image model hint using the image model 122. Thus, in some aspects, the modified optimized image model hint can be provided to the image model 122, where, in response, the image model 122 outputs a new photorealistic image according to the modified optimized image model hint.

[0078] In some aspects, the segmentation engine 116 can be utilized to identify an attribute in an image (e.g., a photorealistic image) and segment the attribute to modify the image. For example, this can be done in response to receiving a text-based modification that indicates a modification to a particular attribute included in an initial photorealistic image. The segmentation engine 116 can apply a segmentation mask to regions of the image (e.g., the initial photorealistic image) having the identified attribute (e.g., the attribute identified in the text-based modification). Thus, the segmentation search engine 110 can utilize semantic segmentation techniques to identify segments of an image and classify the segments as corresponding to particular attributes.

[0079] For example, the segmentation engine 116 can use segmentation techniques on an image to identify image segments, such as pixel regions within the image. As an example, for image segmentation, a convolutional neural network (CNN) can be used. The CNN can be trained using an image dataset (such as the image dataset already described). There are various methods for training CNNs for image segmentation, of which fully convolutional networks (FCNs) and U-Nets are examples. FCNs are designed to handle inputs of various sizes, making them suitable for use with the present technology. U-Nets can extend the capabilities of FCNs by incorporating jump connections, which helps to retain details of images (e.g., photorealistic images) from the input across the network architecture. In an example training method, a CNN is trained using a labeled image dataset (where each pixel in the image is assigned to a specific category or class). During training, the CNN network learns to recognize patterns and features by backpropagation, with the goal of minimizing the difference between its predicted segmentation and the true label. Image classification can be performed by the same or different networks to identify attributes in the image. CNNs can also be suitable for classification tasks. During training, each pixel can be labeled with a specific classification in the training image dataset. These classifications can be associated with many different objects. For example, pixel classifications can be associated with various attributes. In order to minimize these differences during training, mean square error, cross entropy loss or other similar algorithms can be used. This is just one example, and other models and other training methods can be used for image segmentation.

[0080] Thus, upon receiving a text-based modification that includes an attribute, the segmentation engine 116 may be utilized to identify the attribute in an image (eg, an initial photorealistic image). The segmentation engine 116 may apply a segmentation mask to the region having the attribute. Figure 5 , as an example, an initial photo-realistic image 502 is provided to the segmentation engine 116, which generates a segmented image 504 by identifying a segmentation mask 506 and applying it to the region including the identified attribute. As described above, the attribute can be identified based on its inclusion in the text-based modification. For example, returning to Figure 4 , modification 402 includes the attribute "neckline" in the text-based modification. Based on this, the segmentation engine 116 generates a segmented image 504 from the initial photorealistic image 502 by applying a segmentation mask 506. In this case, the initial photorealistic image 502 corresponds to Figure 3 A photo-realistic image 302 at.

[0081] In this manner, a user may generate a photorealistic image such as photorealistic image 302. The user may desire to change an attribute of photorealistic image 302 and, in doing so, provide a text-based input with a text-based modification (e.g., the text-based modification illustrated by modification 402). Segmentation engine 116 identifies and segments regions of an initial image (e.g., initial photorealistic image 502 corresponding to the same photorealistic image 302) to identify the attribute and applies segmentation mask 506 to the region of the image that includes the attribute.

[0082] In some aspects, to modify the property of the initial photo-realistic image, its corresponding segmented image having the identified property is provided to the image model 122 along with a modified optimized image model hint generated according to the text-based modification, as previously described. In doing so, pixels outside the segmented region may be rendered as transparent, causing the image model 122 to modify only the non-transparent pixels within the segmented region according to the modified optimized image model hint, thereby modifying the property within the image, and in response, generating a modified image. Pixels may be rendered as transparent by manipulating the alpha channel of the pixels of the image.

[0083] Figure 6 An example is shown. In this example, a modified optimized image model hint 404 and a segmented image 504 are input to the image model 122. In this case, pixels outside the segmentation mask 506 are rendered as transparent. Therefore, the image model 122 modifies only those pixels within the segmentation mask 506, such as non-transparent pixels, when generating the modified image 602. Transparent pixels between the input image (segmented image 504) and the output image (modified image 602) remain unchanged. The present technology allows attributes to be modified based on the entire modified optimized image model hint (e.g., the modified optimized image model hint 404), thereby rendering modifications that truly fit the rest of the image. For example, as shown in the modified image 602, the attribute neckline 606 has been modified to be cropped lower based on the text-based description in the modified optimized image model hint 404.

[0084] After the photorealistic image has been generated, the image-based searcher 118 may be used to identify search results for the item using an image-based search. That is, the generated photorealistic image may be used as a search query during an image-based search to return similar results. Note also that while reference is made to using a photorealistic image as the object of an image-based search, the photorealistic image being searched for may also be synonymous with a modified image that has been modified by a user to output a final photorealistic image that the user desires to search for. That is, the photorealistic image provided to the image-based searcher 118 to identify the search results may be the initial photorealistic image generated by the photorealistic image generator 114, or may include any number of modifications made by the photorealistic image generator 114, as previously described.

[0085] like Figure 7 As shown, the photo-realistic image 302 is provided to the image-based searcher 118 to identify search results 702. The search results 702 include one or more items, shown here as item A 704a, item B 704b, item C 704c, etc. In an embodiment, the image-based searcher 118 accesses an index to identify items during an image-based search. In response to the photo-realistic image being used as an image-based search query, the items are accessed from the index and provided as search results. Figure 7 In the context of the example shown, for example, the image-based searcher 118 accesses the index to retrieve one or more of item A 704a, item B 704b, item C 704c, or any other items, and returns them as search results 702. The items (e.g., Figure 7 ) as a search result for a photo-realistic image, because in the example shown, search result 702 has been provided as a search result for photo-realistic image 302.

[0086] Figure 8 and Fig. 9 The comparison results between the existing text-based search and the present technology are shown. Figure 8 Results of a text-based search are shown. Here, a common Internet search engine is used. Search query 802 is entered into a search engine (shown here as search engine 804). As shown, search query 802 is "spring pink cocktail dress with lace trim". The top results of search engine 804 in response to search query 802 are shown as search results 806. Fig. 91 shows the results of an image search performed using the techniques described in this application. Here, a text-based input including "spring pink cocktail dress with lace trim" is used to generate a photorealistic image, which is provided as a Fig. 9 902 in . The same common Internet search engine (shown here as search engine 904) is used, but this time is used to perform an image-based search using search query 902. The top results of this search are provided as search results 906. Search result 906 appears to include more items that match the intent of the input text "spring pink cocktail dress with lace trim" than the items of search result 806.

[0087] Reference now Fig.10 , a method 1000 for providing an item as a search result is provided. At block 1002, an optimized image model prompt is generated. This can be performed using an optimized image model prompt generator 112 employing a language model 120. The optimized image model prompt can be generated based on a text-based input (e.g., a text-based input received from a computing device at a search engine). In some cases, the language model generates the optimized image model prompt by expanding the text-based input into a text description. The text description can describe an object within a photorealistic image to be generated based on the description. The text-based input can be expanded into an optimized image model prompt, which includes text containing a textual description of the attributes of the object identified from the text-based input. In order to generate the optimized image model prompt based on the text-based input, a language model can be trained on an item corpus, which includes items and item descriptions corresponding to the items. If additional information is required to generate the optimized image model prompt, the optimized image model prompt generator 112 can transmit a question indicating the required additional information to the user computing device. Upon receiving the additional information, an optimized image model prompt can be generated.

[0088] At block 1004, a photorealistic image is generated. The photorealistic image may be generated by an image model (e.g., image model 122). For example, this may be performed using photorealistic image generator 114. The photorealistic image is generated according to the optimized image model prompt. That is, the optimized image model prompt is input to the image model, which outputs a photorealistic image, wherein the photorealistic image depicts the object described by the text description in the optimized image model prompt.

[0089] At block 1006, an item is accessed from an image-based search. Photorealistic images may be used for image-based searches. For example, the search may be performed using an image-based searcher 118. In an embodiment, the item may be accessed from an index identifying the item during the search.

[0090] At block 1008, the item is provided to the computing device as a search result for a photorealistic image. One or more items may be identified and returned as search results. In one aspect, the search results are returned to the SERP of the computing device from which the text-based input was received.

[0091] refer to Fig.11 , a method 1100 for providing search results to an item is shown. At box 1102, an optimized image model hint is accessed. The optimized image model hint may have been generated using a language model such as language model 120. For example, the optimized image model hint may have been generated by the language model in response to a text-based input provided at a search engine at a computing device. The optimized image model hint can be an extended text description of an object identified within the text-based input, for example, a literal text description of the object. In one aspect, a language model for generating optimized image model hints is trained on an item corpus including items and item descriptions corresponding to the items.

[0092] At block 1104, an optimized image model hint is provided to an image model (e.g., image model 122). This may be performed by photorealistic image generator 114. In response, the image model generates a photorealistic image based on a text description within the optimized image model hint (e.g., a text description of an object recognized in a text-based input).

[0093] At block 1106, an image-based search is performed. This may be performed, for example, by the image-based searcher 118. The image-based search may be performed to identify an item.

[0094] At block 1108, the item is provided to the computing device as a search result for a photorealistic image. One or more items may be identified and returned as search results. In one aspect, the search results are returned to the SERP of the computing device from which the text-based input was received.

[0095] Now turn to Fig.12, a method 1200 for providing an item as a search result is shown. At box 1202, an optimized image model hint is generated. The optimized image model hint can be generated by, for example, an optimized image model hint generator 112 using a language model 120. The language model outputs the optimized image model hint in response to a text-based input. The text-based input can be received from a computing device at a search engine. In order to generate the optimized image model hint, the language model expands the text-based input into a text description of an object included in the text-based input, which provides a textualized text description of the object included in a photo-realistic image to be generated based on the optimized image model hint. On the one hand, the language model is trained on a project corpus that includes projects and project descriptions corresponding to the projects.

[0096] At block 1204, the optimized image model hint is provided to the image model. For example, the image model may be the image model 122. In response to receiving the optimized image model hint, the image model generates a photorealistic image depicting the object according to the text description in the optimized image model hint.

[0097] At block 1206, an image-based search is performed. This may be performed, for example, by the image-based searcher 118. The image-based search may be performed to identify an item.

[0098] At block 1208, the items are provided to the computing device as search results for the photorealistic image. One or more items may be identified and returned as search results. In one aspect, the search results are returned to the SERP of the computing device from which the text-based input was received.

[0099] In any of methods 1000, 1100, and 1200, the methods may include generating a photorealistic image based on a segmented image. That is, the image model may also receive a segmented image as input, wherein the segmented image includes a segmentation mask that identifies regions of pixels in the segmented image and corresponds to properties of an object in the photorealistic image. Pixels outside the segmentation mask may be rendered as transparent, while pixels in the segmentation mask are opaque. In doing so, the image model may generate a photorealistic image by modifying only non-transparent pixels within the segmentation mask according to an optimized image model hint.

[0100] In some aspects, text-based modifications are received. The text-based modifications include text-based input, for example, including a text-based description of modifications to the attributes depicted in the photo-realistic image. Based on the text-based modifications, an optimized image model prompt generator 112 employing a language model 120 can be used to generate a modified optimized image model prompt. That is, the text-based modifications are input to the language model, and the language model outputs a modified optimized image model prompt. When generating the modified optimized image model prompt, the optimized image model prompt can also be input to the language model. In some cases, this is performed through a dialog.

[0101] In some cases, a modified image may be generated from a photorealistic image.The modified image may include modifications to the attributes identified within the text-based modification.

[0102] Turning to method 1300 and method 1400 , in general, these methods may be performed alone or in combination with other methods described (eg, methods 1000 , 1200 , and 1300 ).

[0103] Method 1300 illustrates an example method for segmenting an image for generating a photorealistic image (e.g., a modified image). At block 1302, an initial photorealistic image is accessed. In an embodiment, the initial photorealistic image is an initial image generated using an image model (e.g., image model 122). In one aspect, the initial photorealistic image is an image accessed from a data store.

[0104] At block 1304, the initial photorealistic image is segmented. This may be performed, for example, by the segmentation engine 116. The image may be segmented by applying one or more segmentation masks. The segmentation masks may be applied to regions of the initial photorealistic image that include attributes. The segmentation masks may be applied based on attributes included in the text-based input (e.g., the text-based modification).

[0105] At block 1306, a segmented image including transparent pixels is rendered. For example, the image segmented at block 1304 includes one or more segmentation masks corresponding to one or more attributes. Pixels outside the segmentation masks can be rendered as transparent, for example, by manipulating the alpha channel of the pixel until the pixel is completely transparent.

[0106] At block 1308, the segmented image is provided to an image model to generate a photorealistic image. The generated photorealistic image may be a photorealistic image corresponding to a photorealistic image generated in any of the aforementioned methods. In some cases, the generated photorealistic image corresponds to a modified image, e.g., a photorealistic image with modifications to an attribute generated by an image model according to a text-based input corresponding to the text-based modification.

[0107] Reference now Fig.14 , a method 1400 for performing an updated image-based search is shown. At block 1402, a text-based modification is received. The text-based modification can be a text-based input received from a computing device (e.g., computing device 104). The text-based modification can be received after an initial photo-realistic image is generated (e.g., any image generated by the aforementioned methods (including methods 1000, 1100, and 1200)). In some cases, the text-based modification describes attributes included in the initial photo-realistic image.

[0108] At box 1404, a modified optimized image model hint is generated. This can be performed using the optimized image model hint generator 112 that employs the language model 120. The modified optimized image model hint may include modifications to a previously generated optimized image model hint, such as any of the optimized image model hints generated by the aforementioned methods (including methods 1000, 1100, and 1200). The modified optimized image model hint may include a text description describing the initial photo-realistic image, and include a text description describing updates to the attributes based on the text-based input. Thus, in one aspect, the language model 120 receives the text-based modifications and the optimized image model hint as input, and generates a modified optimized image model hint.

[0109] At box 1406, a segmented image and a modified optimized image model hint are provided. For example, the segmented image can be provided to the image model to generate a modified image. The segmented image can be generated using the segmentation engine 116. The segmented image can include any one or more segmentation masks applied to the initial photorealistic image. The segmentation mask can include a region of the segmented image, the region including, for example, an attribute identified in the text-based modification. In some embodiments, the segmented image includes transparent pixels. The transparent pixel can be outside one or more segmentation masks about the attribute. The pixel can be rendered as transparent by adjusting the alpha channel of the pixel. The segmented image and the modified optimized image model hint can be provided in the same or separate communication with the image model. In response to receiving the segmented image and the modified optimized image model hint, the image model generates a modified image. The modified image includes a visual modification of the attribute within the segmentation mask of the segmented image according to the text description provided by the optimized image model hint. This process can be performed any number of times to generate a modified image that the user wishes to use as a search query.

[0110] At box 1408, an updated image-based search is performed. An updated image-based search is performed using the modified image. For example, this can be performed using the image-based searcher 118. During the updated image-based search, one or more items are identified and returned as search results corresponding to the modified image. In an embodiment, the updated image-based search is the first search performed by the search engine. That is, the user can use the previously described methods to generate an initial photo-realistic image, and then modify the image any number of times to generate a modified image (e.g., an image desired by the user), which can be used to generate a set of search results.

[0111] Return to reference Figures 10 to 14 , block diagrams are provided showing methods 1000, 1100, 1200, 1300, and 1400, respectively. Each block of these methods may include a computing process performed using any combination of hardware, firmware, or software. For example, various functions may be performed by a processor executing instructions stored in a memory. These methods may also be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or a plug-in for another product (just to name a few). These methods may be implemented in whole or in part by components of the operating environment 100.

[0112] Having described an overview of some embodiments of the present technology, an example computing environment in which embodiments of the present technology may be implemented is described below in order to provide a general context for various aspects of the present technology. Fig.15, an example operating environment for implementing embodiments of the present technology is shown and generally designated as computing device 1500. Computing device 1500 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the present technology. Computing device 1500 should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated.

[0113] The technology may be described in the general context of computer code or machine usable instructions, including computer executable instructions (e.g., program modules) executed by a computer or other machine (e.g., a cellular telephone, personal data assistant, or other handheld device). In general, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The technology may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The technology may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0114] refer to Fig.15 , computing device 1500 includes bus 1510 that directly or indirectly couples the following devices: memory 1512, one or more processors 1514, one or more presentation components 1516, input / output (I / O) ports 1518, input / output components 1520, and an example illustrative power supply 1522. Bus 1510 represents one or more buses (e.g., an address bus, a data bus, or a combination thereof). Although for clarity, Fig.15 The various boxes of FIG. 1 are represented by lines, but in reality, the boundaries between the various components are not so clear, and metaphorically, the lines would more accurately be gray and fuzzy. For example, a presentation component (e.g., a display device) can be considered an I / O component. In addition, a processor has memory. The inventors recognize that this is a feature of the art, and reiterate that Fig.15 The figures are merely schematic illustrations of example computing devices that can be used in conjunction with one or more embodiments of the present technology. No distinction is made between categories such as "workstation," "server," "laptop," "handheld device," etc., as all of these categories are in the same Fig.15 and are referred to as “computing devices”.

[0115] The computing device 1500 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1500, and includes volatile and non-volatile media, as well as removable and non-removable media. As an example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media (also referred to as communication components) include volatile and non-volatile, as well as removable and non-removable media implemented in any method or technology, which are used to store information, such as computer-readable instructions, data structures, program modules or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or, other storage technologies; CD-ROM, digital versatile disk (DVD), or other optical disk storage devices; cassettes, magnetic tapes, magnetic disk storage devices, or other magnetic storage devices; or any other media that can be used to store the desired information and can be accessed by the computing device 1500. The computer storage media itself does not include signals.

[0116] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in the form of a modulated data signal (e.g., a carrier wave or other transmission mechanism), and includes any information transmission medium. The term "modulated data signal" refers to a signal having one or more characteristics that are set or changed in a manner that encodes the information in the signal. As an example and not limitation, communication media include wired media (e.g., a wired network or a direct wired connection) and wireless media (e.g., an acoustic wireless medium, a radio frequency (RF) wireless medium, an infrared wireless medium, and other wireless media). Combinations of any of the above should also be included within the scope of computer-readable media.

[0117] Memory 1512 includes computer storage media in the form of volatile memory or non-volatile memory. Memory can be removable, non-removable, or a combination thereof. Example hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Computing device 1500 includes one or more processors that read data from various entities (e.g., memory 1512 or I / O components 1520). Presentation component 1516 presents data indications to a user or other device. Examples of presentation components include display devices, speakers, printing components, vibration components, etc.

[0118] I / O port 1518 allows computing device 1500 to be logically coupled to other devices including I / O component 1520, some of which may be built-in. Example illustrative components include microphones, joysticks, game controllers, satellite antennas, scanners, printers, wireless devices, etc. I / O component 1520 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, the input may be sent to an appropriate network unit for further processing. NUI may implement any combination of the following: voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, and air gestures, head and eye tracking, or touch recognition associated with the display of computing device 1500. Computing device 1500 may be equipped with a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, an RGB (red-green-blue) camera system, a touch screen technology, other similar systems, or a combination of these items. In addition, computing device 1500 may be equipped with an accelerometer or gyroscope capable of detecting motion. The output of the accelerometer or gyroscope may be provided to a display of computing device 1500 to render immersive augmented reality or virtual reality.

[0119] At a low level, a hardware processor executes instructions selected from a given processor's machine language (also referred to as machine code or native) instruction set. The processor recognizes native instructions and executes corresponding low-level functions associated with, for example, logic, control, and memory operations. Low-level software written in machine code can provide more complex functionality for higher-level software. As used herein, computer-executable instructions include any software, including low-level software written in machine code; high-level software, such as application software; and any combination thereof. In this regard, a component for optimizing text-based input to generate images for image-based search can manage resources and provide the described functionality. Any other variants and combinations thereof are contemplated within embodiments of the present technology.

[0120] Brief return reference Figure 1 , note and re-emphasize that any additional or fewer components in any arrangement may be used to achieve the desired functionality within the scope of the present disclosure. Although for clarity, Figure 1 The various components of are shown with lines, but in reality, the boundaries between the various components are not so clear, and metaphorically, the lines may more accurately be gray or blurred. Figure 1Some components of the operating environment 100 are depicted as single components, but these depictions are intended to be examples in nature and quantity, and for all embodiments of the present disclosure, this should not be interpreted as limiting. The functionality of the operating environment 100 can be further described based on the functions and features of its components. Other arrangements and elements (e.g., machines, interfaces, functions, commands, and functional groupings) can be used in addition to or instead of the arrangements and units shown, and some units can be omitted entirely.

[0121] In addition, regarding Figure 1 Some units described (e.g., units described with respect to the search engine 110) are functional entities that can be implemented as discrete components or distributed components or in combination with other components, and can be implemented in any suitable combination and location. The various functions described herein are performed by one or more entities and can be performed by hardware, firmware, or software. For example, the various functions can be performed by a processor executing computer executable instructions stored in a memory (e.g., database 106). In addition, the functions of the search engine 110, as well as other functions, can be performed by the server 102, the computing device 104, or any other component in any combination. For example, in a particular aspect, the search engine 110 can optimize text-based input at a server to generate an optimized image model prompt, and transmit the optimized image model prompt to another server that generates a photo-realistic image based on the optimized image model prompt, and then receive the photo-realistic image from the server to which the prompt is transmitted. This is just one example, and a person of ordinary skill in the art will understand other example combinations and configurations within the context of the contents of the present disclosure.

[0122] In general, by reference to the drawings and descriptions, after the various components in the present disclosure have been identified, it should be understood that any number of components and arrangements can be used to implement the desired functions within the scope of the present disclosure. For example, for the sake of conceptual clarity, the components in the embodiments depicted in the drawings are shown with lines. Other arrangements of these and other components can also be implemented. For example, although some components are depicted as single components, many units described herein can be implemented as discrete or distributed components or combined with other components, and implemented in any suitable combination and position. Some units can be omitted completely. In addition, the various functions performed by one or more entities described herein can be performed by hardware, firmware, or software. For example, various functions can be performed by a processor that executes instructions stored in a memory. Therefore, as a supplement or replacement for the arrangements and units shown, other arrangements and units (e.g., machines, interfaces, functions, commands, and groupings of functions, etc.) can also be used.

[0123] The above-described embodiments may be combined with one or more specifically described alternatives. Specifically, a claimed embodiment may include references to more than one other embodiment in an alternative. A claimed embodiment may specify additional limitations of the claimed subject matter.

[0124] The subject matter of the present technology is described in detail herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have contemplated that the claimed or disclosed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to the steps described in this document in conjunction with other prior art or future technologies. In addition, although the terms "step" or "box" may be used herein to refer to different elements of the method employed, the terms should not be interpreted as implying any particular order between or among the various steps disclosed herein, unless and in addition to the order of the various steps being explicitly described.

[0125] For purposes of this disclosure, the words "including," "having," and other similar words and their derivatives have the same broad meaning as the word "comprising," and the word "access" includes "receiving," "referencing," or "retrieving" or their derivatives. Additionally, the word "communicating" has the same broad meaning as the words "receiving" or "sending" as facilitated by a software or hardware based bus, receiver, or transmitter using the communications media described herein.

[0126] Furthermore, unless otherwise specified, words such as "a", "an", etc. include the plural as well as the singular. Thus, for example, where there are one or more features, the constraint on "one feature" is satisfied. Furthermore, the term "or" includes conjunctions, disjunctions, and both (a or b therefore includes a or b, and a and b).

[0127] For the purposes of the above detailed discussion, embodiments of the present technology are described with reference to a distributed computing environment. However, the distributed computing environment described herein is merely an example. Components may be configured to perform novel aspects of the embodiments, where the term "configured to" or "configured to" may refer to being "programmed to" perform a particular task or to implement a particular abstract data type using code. Furthermore, while embodiments of the present technology may generally refer to a distributed data object management system and the schematic diagrams described herein, it should be understood that the described technology may be extended to other implementation contexts.

[0128] As can be seen from the foregoing, the present technology is well suited to achieve all of the above-mentioned goals and objectives, including other advantages that are obvious or inherent to the structure. It will be understood that some features and sub-combinations are useful and can be used without reference to other features and sub-combinations. This is contemplated by the claims and is within the scope of the claims. Since many possible embodiments of the described technology can be made without departing from the scope, it should be understood that all content described herein or shown in the drawings should be interpreted as illustrative and not restrictive.

[0129] Some example aspects that may be practiced according to the foregoing description include, but are not limited to, the following examples:

[0130] Aspect 1: A method performed by one or more processors, the method comprising: using a language model to generate an optimized image model prompt, the language model outputting the optimized image model prompt in response to text-based input; using an image model to generate a photo-realistic image of an item, the image model outputting the photo-realistic image of the item in response to receiving the optimized image model prompt as input; accessing the item from an image-based search, the image-based search being performed using the photo-realistic image; and providing the item as a search result of the photo-realistic image to a computing device.

[0131] Aspect 2: A system comprising: at least one processor; and one or more computer storage media storing computer-readable instructions thereon, which, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising: accessing an optimized image model hint that has been generated by a language model in response to a text-based input; providing the optimized image model hint to an image model, which generates a photo-realistic image of an item based on the optimized image model hint; performing an image-based search at a search engine using the photo-realistic image; and providing the item as a search result identified by the image-based search to a computing device.

[0132] Aspect 3: One or more computer storage media storing computer-readable instructions which, when executed by a processor, cause the processor to perform a method comprising: generating an optimized image model hint using a language model, the language model outputting the optimized image model hint in response to a text-based input; providing the optimized image model hint to an image model, the image model generating a photo-realistic image of an item based on the optimized image model hint; performing an image-based search at a search engine using the photo-realistic image; and providing the item as a search result identified by the image-based search to a computing device.

[0133] Aspect 4: According to any one of aspects 1 to 3, the language model is trained on an item corpus, the item corpus comprising items and item descriptions corresponding to the items.

[0134] Aspect 5: According to any one of aspects 1 to 4, the language model generates the optimized image model hint by expanding the text-based input into a text description of the object in the text-based input.

[0135] Aspect 6: According to any one of aspects 1 to 5, it further includes providing the segmented image including transparent pixels to the image model, wherein the photo-realistic image is generated by the image model based on the segmented image.

[0136] Aspect 7: According to Aspect 6, further comprising determining transparent pixels based on a segmentation mask that identifies an attribute of the item.

[0137] Aspect 8: According to any one of Aspects 1 to 7, it also includes: accessing an initial photo-realistic image generated by the image model; segmenting the initial photo-realistic image to identify a segmentation mask including an attribute; generating a segmented image based on the initial photo-realistic image in the following manner: rendering pixels within the initial photo-realistic image as transparent, the rendered transparent pixels being outside the segmentation mask including the attribute; and providing the segmented image to the image model to generate a photo-realistic image.

[0138] Aspect 9: According to any one of Aspects 1 to 8, it also includes: receiving a text-based modification to an optimized image model hint, the text-based modification corresponding to an attribute of a photo-realistic image; using a language model to generate a modified optimized image model hint, the language model generating a modified optimized image model hint based on the text-based modification; providing a segmented image identifying the attribute and the modified optimized image model hint to an image model, wherein the image model generates a modified image with the modification to the attribute based on the modified optimized image model hint; and using the modified image to perform an updated image-based search.

Claims

1. A method for image-based searching performed by one or more processors, the method comprising: generating an optimized image model hint using a language model, the language model outputting the optimized image model hint in response to a text-based input; generating a photorealistic image of an item using an image model, the image model outputting a photorealistic image of the item in response to receiving as input the optimized image model prompt; accessing items from an image-based search performed using the photorealistic image; as well as The item is provided to a computing device as a search result for the photorealistic image.

2. The method according to claim 1, wherein: The language model is trained based on an item corpus, which includes items and item descriptions corresponding to the items.

3. The method according to claim 1, wherein: The language model generates the optimized image model hint by expanding the text-based input into a text description of an object in the text-based input.

4. The method of claim 1, further comprising providing the image model with a segmented image comprising transparent pixels, wherein: The photo-realistic image is generated by the image model based on the segmented image.

5. The method according to claim 4, further comprising: The transparent pixels are determined based on a segmentation mask that identifies attributes of the item.

6. The method according to claim 1, further comprising: accessing an initial photo-realistic image generated by the image model; segmenting the initial photorealistic image to identify a segmentation mask comprising attributes; Generating a segmented image from the initial photorealistic image by rendering pixels within the initial photorealistic image as transparent, the rendered transparent pixels being outside a segmentation mask including the attribute; as well as The segmented image is provided to the image model for generating the photo-realistic image.

7. The method according to claim 1, further comprising: receiving a text-based modification to the optimized image model hint, the text-based modification corresponding to an attribute of the photorealistic image; generating a modified optimized image model hint using the language model, the language model generating the modified optimized image model hint based on the text-based modifications; providing the segmented image identifying the attribute and the modified optimized image model hint to the image model, wherein the image model generates a modified image with a modification to the attribute according to the modified optimized image model hint; as well as An updated image-based search is performed using the modified image.

8. A system for image-based searching, the system comprising: at least one processor; as well as one or more computer storage media having computer readable instructions stored thereon, which when executed by the at least one processor cause the at least one processor to perform operations comprising: accessing an optimized image model hint that has been generated by a language model in response to a text-based input; providing the optimized image model hints to an image model, the image model generating a photorealistic image of an item based on the optimized image model hints; performing an image-based search at a search engine using the photorealistic image; and Items identified as search results by the image-based search are provided to a computing device.

9. The system according to claim 8, wherein: The language model is trained based on an item corpus, which includes items and item descriptions corresponding to the items.

10. The system according to claim 8, wherein: The language model generates the optimized image model hint by expanding the text-based input into a text description of an object in the text-based input.

11. The system of claim 8, further comprising providing the image model with a segmented image including transparent pixels, wherein: The photo-realistic image is generated by the image model based on the segmented image.

12. The system of claim 11, further comprising determining the transparent pixels based on a segmentation mask that identifies an attribute of the item.

13. The system of claim 8, further comprising: accessing an initial photo-realistic image generated by the image model; segmenting the initial photorealistic image to identify a segmentation mask comprising attributes; Generating a segmented image from the initial photorealistic image by rendering pixels within the initial photorealistic image as transparent, the rendered transparent pixels being outside a segmentation mask including the attribute; as well as The segmented image is provided to the image model for generating the photo-realistic image.

14. One or more computer storage media having computer readable instructions stored thereon, which when executed by a processor, cause the processor to perform a method for image-based searching, the method comprising: generating an optimized image model hint using a language model, the language model outputting the optimized image model hint in response to a text-based input; providing the optimized image model hints to an image model, the image model generating a photorealistic image of an item based on the optimized image model hints; performing an image-based search at a search engine using the photo-realistic image; as well as Items identified as search results by the image-based search are provided to a computing device.

15. The medium according to claim 14, wherein The language model is trained based on an item corpus, which includes items and item descriptions corresponding to the items.

16. The medium according to claim 14, wherein The language model generates the optimized image model hint by expanding the text-based input into a text description of an object in the text-based input.

17. The medium of claim 14, further comprising providing the image model with a segmented image including transparent pixels, wherein: The photo-realistic image is generated by the image model based on the segmented image.

18. The medium of claim 17, further comprising determining the transparent pixels based on a segmentation mask that identifies an attribute of the item.

19. The medium of claim 14, further comprising: accessing an initial photo-realistic image generated by the image model; segmenting the initial photorealistic image to identify a segmentation mask comprising attributes; Generating a segmented image from the initial photorealistic image by rendering pixels within the initial photorealistic image as transparent, the rendered transparent pixels being outside a segmentation mask including the attribute; as well as The segmented image is provided to the image model for generating the photo-realistic image.

20. The medium of claim 14, further comprising: receiving a text-based modification to the optimized image model hint, the text-based modification corresponding to an attribute of the photorealistic image; generating a modified optimized image model hint using the language model, the language model generating the modified optimized image model hint based on the text-based modifications; providing the segmented image identifying the attribute and the modified optimized image model hint to the image model, wherein the image model generates a modified image with a modification to the attribute according to the modified optimized image model hint; as well as An updated image-based search is performed using the modified image.

Images