Search using queries generated by machine learning models

Abstract

Systems and methods for searching using output generated by a machine learning model can provide a vehicle for a user to generate a theoretical dataset that can be matched to real-world examples. The systems and methods can include selecting a number of terms that can be utilized to generate prompt inputs, which can be processed by a dataset generation model to generate a number of model generation datasets. A selection can then be received to select a particular model generation database to utilize to query the database.

Description

[Technical field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 433,559, filed December 19, 2022, which is based on U.S. Provisional Patent Application No. 18 / 173,449, filed February 23, 2023. The application claims priority to and the benefit of each of such applications and incorporates by reference all such applications herein.

[0002] The present disclosure relates generally to searching leveraging machine learned model output, and more specifically to taking user interface selections and generating prompt inputs that can be processed by a machine learned model to generate outputs that a user can review and select to input into a search engine, and to receive search results associated with the selected outputs. [Background technology]

[0003] Searching for clothing, art, movies, and / or music can be difficult if a user does not have examples to provide to the search engine. Free-form text and / or Boolean strings provided to a search engine as a text query may provide mixed and / or unaligned search results that may be off-point and / or include only a portion of the search query. Refining those searches and / or reviewing those search results may be time intensive and non-intuitive. Image queries may provide more tailored results because images may contain features that cannot be descriptively described via succinct text. However, users may not have access to images of what they are looking for during a search and / or the user may base their search on real-world examples that they know based on real-world experience (e.g., the user may be searching for real-world examples of things they have imagined).

[0004] Additionally, using artificial intelligence techniques to generate images and / or other data sets can be non-intuitive, may be open-ended, and may be time consuming. Current image generation systems utilize a prompt input box to receive free-form text that is processed to generate one or more images. However, when a user uses the prompt input box, the user may struggle with which words to use and / or may be dissatisfied with the generated image because one or more of the entered words may not be used in the way the user desired (e.g., a user may enter "houndstooth" in reference to a pattern, but the model may generate an image that includes dog teeth). Summary of the Invention

[0005] Aspects and advantages of embodiments of the disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0006] An exemplary aspect of the present disclosure is directed to a computing system for searching using a data query generated by a machine learning model. The system can include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining a prompt input. The prompt input can include one or more terms. The operations can include processing the prompt input with an image generation model to generate one or more model-generated images. The one or more model-generated images can be generated based at least in part on the one or more terms. The operations can include determining one or more search results based on the one or more model-generated images. The one or more search results can be associated with one or more objects. The operations can include providing a search result interface. The search result interface can provide the one or more search results for display.

[0007] Another exemplary aspect of the present disclosure is directed to a computer-implemented method for searching using a data query generated by a machine learning model. The method can include obtaining, by a computing system including one or more processors, a prompt input. The prompt input can include one or more terms. The method can include processing, by the computing system, the prompt input with a data generation model to generate a plurality of model generation datasets. The plurality of model generation datasets can be generated based at least in part on the one or more terms. The method can include providing, by the computing system, the plurality of model generation datasets via a user interface. The method can include obtaining, by the computing system, a selection input. The selection input can describe a selection of a particular model generation dataset of the plurality of model generation datasets. The method can include determining, by the computing system, one or more search results based on the particular model generation dataset. The method can include providing, by the computing system, the one or more search results as output.

[0008] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations can include providing an image generation interface for display. The image generation interface can include a plurality of category user interface elements. In some implementations, each category user interface element can be associated with a different generation category. The operations can include obtaining first input data. The first input data can be associated with a selection of a particular category user interface element of the plurality of category user interface elements. In some implementations, the particular category user interface element can be associated with a particular category. The operations can include providing a plurality of descriptor user interface elements for display in the image generation interface. Each descriptor user interface element can be associated with a different descriptor. The operations can include obtaining second input data. The second input data can be associated with a selection of one or more particular descriptor user interface elements of the plurality of descriptor user interface elements. In some implementations, the one or more particular descriptor user interface elements can be associated with one or more particular descriptors. The operations can include processing data associated with one or more particular descriptors with a machine learning image generation model to generate one or more model-generated images. The operations can include providing the one or more model-generated images for display in the image generation interface.

[0009] Another exemplary aspect of the present disclosure is directed to a computing system. The system can include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include obtaining a prompt input. The prompt input can include one or more terms describing the absence of specific details. The operations can include processing the prompt input with an image generation model to generate one or more model-generated images. The one or more model-generated images can be generated based at least in part on the one or more terms. In some implementations, the one or more model-generated images can describe the generated objects without the specific details. The operations can include determining one or more search results based on the one or more model-generated images. The one or more search results can be associated with the one or more objects without the specific details. The operations can include providing a search result interface. The search result interface can provide the one or more search results for display.

[0010] Another exemplary aspect of the present disclosure is directed to a computer-implemented method. The method can include obtaining, by a computing system including one or more processors, a multimodal prompt input. The multimodal prompt input can include a prompt image and prompt text. The prompt image can describe a particular object. In some implementations, the prompt text can describe one or more particular details of the prompt image to augment. The method can include processing, by the computing system, the prompt image and the prompt text with an image generation model to generate a model-generated image. The model-generated image can describe a model-generated object. In some implementations, the model-generated object can describe a particular object that is augmented based on the prompt text. The method can include providing, by the computing system, the model-generated image to a search engine as a search query based on the one or more user inputs, and receiving, by the computing system, one or more search results from the search engine based on the model-generated image.

[0011] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations can include obtaining a first search query, processing the first search query to determine a plurality of first search results in response to the first search query, and providing the plurality of first search results for display. The operations can include obtaining a text input and an image search result selection of the plurality of first search results. The image search results can describe a particular object with one or more specific details. The operations can include generating a multimodal prompt input. The multimodal prompt input can include a prompt image and prompt text. The prompt image can be associated with the image search results. The prompt text can describe a request to render a particular object without one or more specific details. The operations can include processing the prompt image and the prompt text with an image generation model to generate a model-generated image. The model-generated image can describe the model-generated object. In some implementations, the model-generated object can describe the particular object without one or more specific details. The operations may include processing the model generated image to determine one or more second search results and providing the one or more second search results for display.

[0012] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0013] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the detailed description, serve to explain associated principles.

[0014] Detailed descriptions of embodiments directed to those skilled in the art are set forth herein with reference to the accompanying drawings. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates a block diagram of an exemplary model generation image retrieval system according to an exemplary embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates a block diagram of an exemplary machine learning model processing and search system according to an exemplary embodiment of the present disclosure. [Figure 3A] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3B] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3C] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3D] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3E] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3F] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3G] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3H] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3I] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3J] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3K] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3L] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 3M] 1 illustrates a diagram of an exemplary search interface according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates a block diagram of an exemplary machine learning model processing and search system according to an exemplary embodiment of the present disclosure. [Diagram 5] 1 shows a flowchart diagram of an exemplary method for performing image generation and retrieval according to an exemplary embodiment of the present disclosure. [Figure 6A] 1 illustrates a diagram of an example finder entry point according to an example embodiment of the present disclosure. [Figure 6B] 1 illustrates a diagram of an example finder entry point according to an example embodiment of the present disclosure. [Figure 6C] 1 illustrates a diagram of an example finder entry point according to an example embodiment of the present disclosure. [Figure 7A] 1 illustrates a diagram of an exemplary image generation and search interface according to an exemplary embodiment of the present disclosure. [Figure 7B] 1 illustrates a diagram of an exemplary image generation and search interface according to an exemplary embodiment of the present disclosure. [Figure 7C] 1 illustrates a diagram of an exemplary image generation and search interface according to an exemplary embodiment of the present disclosure. [Figure 7D] 1 illustrates a diagram of an exemplary image generation and search interface according to an exemplary embodiment of the present disclosure. [Figure 8A] 1 illustrates a diagram of an exemplary refined image generation and retrieval interface according to an exemplary embodiment of the present disclosure. [Figure 8B] 1 illustrates a diagram of an exemplary refined image generation and retrieval interface according to an exemplary embodiment of the present disclosure. [Figure 8C] 1 illustrates a diagram of an exemplary refined image generation and retrieval interface according to an exemplary embodiment of the present disclosure. [Figure 9A] 1 illustrates a diagram of an exemplary search interface entry point according to an exemplary embodiment of the present disclosure. [Figure 9B]1 illustrates a diagram of an exemplary search interface entry point according to an exemplary embodiment of the present disclosure. [Figure 9C] 1 illustrates a diagram of an exemplary search interface entry point according to an exemplary embodiment of the present disclosure. [Figure 9D] 1 illustrates a diagram of an exemplary search interface entry point according to an exemplary embodiment of the present disclosure. [Figure 9E] 1 illustrates a diagram of an exemplary search interface entry point according to an exemplary embodiment of the present disclosure. [Figure 10] 1 illustrates a diagram of an exemplary user collection interface, according to an exemplary embodiment of the present disclosure. [Figure 11A] 1 shows a diagram of an exemplary image retrieval system, in accordance with an exemplary embodiment of the present disclosure. [Figure 11B] 1 shows a diagram of an exemplary image retrieval system, in accordance with an exemplary embodiment of the present disclosure. [Figure 12] 1 illustrates a diagram of an exemplary collection interface, according to an exemplary embodiment of the present disclosure. [Figure 13] 1 illustrates a diagram of an exemplary collection interface, according to an exemplary embodiment of the present disclosure. [Figure 14] 1 illustrates a diagram of an exemplary collection interface, according to an exemplary embodiment of the present disclosure. [Figure 15] 1 illustrates a diagram of an exemplary suggestions interface, according to an exemplary embodiment of the present disclosure. [Figure 16A] 1 illustrates a diagram of an exemplary modification interface, according to an exemplary embodiment of the present disclosure. [Figure 16B] 1 illustrates a diagram of an exemplary modification interface, according to an exemplary embodiment of the present disclosure. [Figure 17A] 1 illustrates a diagram of an exemplary product suggestions interface, according to an exemplary embodiment of the present disclosure. [Figure 17B] 1 illustrates a diagram of an exemplary product suggestions interface, according to an exemplary embodiment of the present disclosure. [Figure 17C] 1 illustrates a diagram of an exemplary product suggestions interface, according to an exemplary embodiment of the present disclosure. [Figure 18] 1 illustrates a flowchart diagram of an exemplary method for performing machine learning model output generation and search in accordance with an exemplary embodiment of the present disclosure. [Figure 19] 1 illustrates a flowchart diagram of an exemplary method for performing prompt input generation according to an exemplary embodiment of the present disclosure. [Figure 20] FIG. 1 illustrates a block diagram of an exemplary model-generated image retrieval and customization system according to an exemplary embodiment of the present disclosure. [Figure 21] FIG. 1 illustrates a block diagram of an exemplary model generation image retrieval system according to an exemplary embodiment of the present disclosure. [Figure 22A] FIG. 1 illustrates a block diagram of an exemplary computing system for performing machine learning model output generation and retrieval in accordance with an exemplary embodiment of the present disclosure. [Figure 22B] 1 illustrates a block diagram of an exemplary computing device for performing machine learning model output generation and retrieval in accordance with an exemplary embodiment of the present disclosure. [Figure 22C] 1 illustrates a block diagram of an exemplary computing device for performing machine learning model output generation and retrieval in accordance with an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Reference numbers repeated among the drawings are intended to identify like features in the various embodiments.

[0017] In general, the present disclosure is directed to generating content (e.g., images) based on a user request (e.g., prompt input) to provide a visualization of the requested item, and then searching for the requested item to provide a directed search based on the visual information. The systems and methods disclosed herein can leverage one or more dataset generation models to generate a dataset, which can then be utilized to query a search engine to obtain search results. For example, a user may have an idea or concept for which the user desires to find one or more real-world examples of the idea or concept. Thus, the user may generate a prompt input that can be provided to the dataset generation model to generate one or more datasets. The user may then select a particular dataset, which can then be utilized to search one or more datasets to obtain search results. For example, a dataset generation module (e.g., an image generation model) may generate multiple datasets in response to the prompt input. The multiple model-generated datasets can be provided to a user interface that can include a displayed list, a carousel, and / or one or more other presentation methods. The user can then review the multiple model generation datasets (e.g., multiple model generation images) to determine one or more particular model generation datasets that can be used to search one or more databases. The one or more particular model generation datasets can be used to query one or more databases to obtain one or more search results.

[0018] Querying one or more databases with one or more particular model generation datasets can include processing the one or more particular model generation datasets with one or more machine learning models (e.g., one or more classification models and / or one or more embedding models). For example, the one or more particular model generation datasets can be processed by one or more embedding models to generate one or more features that can then be utilized to query the database for associated embeddings (e.g., one or more embedding neighborhoods) that can be associated with one or more candidate search results. Alternatively and / or additionally, the one or more particular model generation datasets can be processed by one or more classification models to determine one or more classification tags that can be utilized to generate queries to query one or more databases.

[0019] In particular, it can be difficult to articulate concepts and ideas for a search, and some concepts may not be articulated at all, which can lead to problems with search result coverage. Additional issues may include not knowing what terms to use, needing unique content, vocabulary boundaries between the user and the industry, only partial results, and / or irrelevant search results.

[0020] The systems and methods disclosed herein can leverage one or more machine learning diffusion models to generate images that can encapsulate a user request and then serve as image queries to determine real-world objects that are similar to the "imaginary" objects in the model-generated images. Artificial intelligence (AI) generative models can be used to generate images that can be considered and selected for use as search queries. In particular, images can provide more context for what a user is requesting during a search, thereby enabling a more targeted search than text alone.

[0021] The present disclosure is directed to systems and methods for searching using data queries generated by machine learning models. In particular, the systems and methods disclosed herein can leverage one or more machine learning models and one or more user interface elements to provide an interactive graphical user interface for suggesting, generating, and / or refining search queries based on a model-generated dataset. Thus, the generated images can be utilized to provide accurate search results since the generated dataset can provide a more detailed jump-off point for searching. For example, the systems and methods can include obtaining a prompt input. The prompt input can include one or more terms. The prompt input can be processed with an image generation model to generate one or more model-generated images. The one or more model-generated images can be generated based at least in part on the one or more terms. A selection input can then be obtained. The selection input can describe a selection of the one or more model-generated images. The systems and methods can include determining one or more search results based on the one or more model-generated images. The one or more search results can be associated with one or more objects (e.g., products such as clothing items). A search result interface can be provided for display (e.g., a search result page). The search result interface can provide one or more search results for display and can include a search result list and a viewport for viewing at least a portion of the resources associated with the one or more search results.

[0022] The systems and methods disclosed herein can be utilized to provide an interface for generating a proposed dataset, which can then be utilized to query the web for existing datasets that may be similar to and / or associated with the model generated dataset. For example, a model generated dataset may include a model generated image that describes an instance of an object. The model generated object can then query one or more datasets to identify resources associated with objects similar to the objects depicted in the model generated image.

[0023] The systems and methods can obtain prompt input (e.g., selection data describing one or more selections received from a user computing device). The prompt input can include one or more terms (e.g., one or more words (e.g., "jacket, feathered, brown, regal") that can describe an interpolation of a requested instance to request a view rendering of a brown feathered jacket having a regal aesthetic. In some implementations, the prompt input can include selection data describing one or more selection inputs associated with one or more selectable user interface elements and / or one or more text inputs including text input into a text input box.

[0024] In some implementations, obtaining the prompt input can include providing a plurality of selectable user interface elements for display within the graphical user interface. The plurality of selectable user interface elements can be associated with a plurality of candidate prompt terms (e.g., object types, categories, scene or object descriptors, and / or aesthetics). Selection data can then be obtained. The selection data can describe a first selectable user interface element (e.g., a first interactive tip) and a second selectable user interface element (e.g., a second interactive tip). The first selectable user interface element can be associated with a first prompt term (e.g., a noun, verb, adjective, and / or adverb associated with the requested concept), and the second selectable user interface element is associated with a second prompt term (e.g., a noun, verb, adjective, and / or adverb associated with the requested concept). For example, the prompt input can include a first prompt term and a second prompt term associated with the selected first user interface element and the selected second user interface element. Prompt terms may describe a subject (e.g., a landscape, an amusement park, a dress, and / or a purse), a quality (e.g., Tron-like, science fiction, made of plants, a particular video game aesthetic, baroque, cyborg, and / or covered in sequins), and / or an action (e.g., dancing, running, playing football, and / or cheering).

[0025] In some embodiments, the plurality of selectable user interface elements can be provided for display in response to obtaining a prompt selection request. The prompt selection request can describe an input for receiving a graphical user interface of a selectable user interface chip. The prompt selection request can be received by the user computing system during display of an entry point interface including a text input box for receiving user input data for generating a machine learning model output based on a user-provided text prompt. A plurality of candidate prompt terms associated with the plurality of selectable user interface chips can be predetermined. A first prompt term can be associated with a type of object. A second prompt term can be associated with a particular descriptive characteristic, and one or more model-generated images can describe a particular object of the type of object having the particular descriptive characteristic.

[0026] The prompt input can be processed with an image generation model to generate one or more model-generated images. The one or more model-generated images can be generated based at least in part on the one or more terms. The image generation model can be trained with multiple training images. The image generation model can be trained on a particular subject and / or a particular object type (e.g., a particular clothing item). Alternatively and / or in addition, the image generation model can be trained broadly. The training can include label training, where the labels can be utilized to determine and / or generate selectable user interface elements. For example, a particular label can be associated with multiple images (e.g., a "shirt" label can be associated with images of multiple different shirts, and / or a "fur" label can be associated with multiple images associated with multiple furs of clothing and / or home furnishings items). Descriptors of the labels can then be utilized to generate selectable user interface elements for the descriptors utilized as prompt terms.

[0027] In some implementations, one or more model-generated images can be provided for display with one or more terms in a graphical user interface. For example, multiple model-generated images can be generated and provided for display in an image carousel. One or more model-generated images can be provided for display for interaction. A user can select a portion of a particular model-generated image for augmentation. For example, a user can remove features (e.g., remove an object from the scene, remove an accessory, and / or match clothing), modify features (e.g., change texture and / or change color), and / or add features (e.g., add an object, add an ascent, and / or add an accessory) by providing one or more augmentation inputs.

[0028] A selection input may then be obtained. The selection input may describe a selection of one or more model-generated images. The selection input may describe a request to query one or more databases for content and / or items similar to the content and / or items in the selected model-generated images. The selection input may include one or more selections of one or more portions of the selected model-generated images that are of interest. The one or more portions may be segmented (or cropped) and then input to a search engine.

[0029] The system and method can determine one or more search results based on the one or more model-generated images. The one or more search results can be associated with one or more objects. In some implementations, the one or more search results can be associated with one or more products. Additionally and / or alternatively, the one or more search results can include one or more action links associated with the one or more products. The one or more action links can be associated with a purchasing interface for the one or more products. The one or more search results can be determined based on one or more labels associated with the model-generated images. Alternatively and / or additionally, the model-generated images can be processed using an embedding model to generate embeddings. The embeddings can then be utilized to determine similar embeddings, which can be associated with one or more search results. The one or more prompt terms can be utilized to determine one or more search engines. For example, the one or more search results can be obtained by generating a combined query using the prompt terms and the model-generated images.

[0030] In some implementations, determining one or more search results based on the one or more model-generated images can include providing the one or more model-generated images to a search engine and receiving one or more search results from the search engine, which can be a general search engine and / or can be a database-specific search engine (e.g., a shopping search engine).

[0031] A search result interface can then be provided for display. The search result interface may include one or more search results provided for display. The search result interface may be a search result page. The search result interface may include a list of search results, an augmented reality experience interface, and / or a viewport for viewing a preview of resources associated with one or more search results.

[0032] The systems and methods can be utilized to find images and products similar to the request. Additionally and / or alternatively, the systems and methods disclosed herein can be utilized to find other data types (e.g., songs that fit an aesthetic and / or theme). For example, a machine learning model can be trained to generate audio data based on one or more prompt inputs (e.g., "jazz, upbeat, saxophone solo" can be input to the machine learning model to generate a synthetic song, and the song can be presented to the user for selection and then search). For example, the systems and methods can include obtaining a prompt input. The prompt input can include one or more terms. The prompt input can be processed with the data generation model to generate a plurality of model generation datasets. The plurality of model generation datasets can be generated based at least in part on the one or more terms. The systems and methods can include providing a plurality of model generation datasets via a user interface and obtaining a selection input. The selection input can describe a selection of a particular model generation dataset of the plurality of model generation datasets. The systems and methods can include determining one or more search results based on the particular model generation dataset and providing the one or more search results as output.

[0033] The systems and methods can obtain a prompt input. The prompt input can include one or more terms. The prompt input can be generated based on one or more selections of one or more user interface tips that can include text characters and / or icons associated with the terms utilized to prompt the data generation model. In some implementations, the prompt input can include one or more images, one or more audio clips, and / or latent encoded data.

[0034] The prompt input can be processed with a data generation model to generate a plurality of model generation datasets. The plurality of model generation datasets can be generated based at least in part on one or more terms. In some implementations, each of the plurality of model generation datasets can be different. The data generation model can be trained to generate one or more datasets based on a plurality of learning parameters and adjusted based on the prompt input. The model generation datasets can include image data, audio data, multimodal data, text data, latent coding data, and / or sensor data. For example, the plurality of model generation datasets can include a plurality of images (e.g., a plurality of predicted depictions that describe the prompt input), a plurality of audio clips (e.g., a plurality of generated song clips predicted to describe the prompt input), and / or a plurality of video datasets (a plurality of predicted video clips generated based on the prompt input).

[0035] The multiple model generation datasets can then be provided for display via a user interface. Providing the multiple model generation datasets via the user interface may include providing the multiple model generation images in an image carousel. The multiple model generation datasets can be provided as a list of links to preview the model generation datasets. Alternatively and / or additionally, the multiple model generation datasets can be transmitted for local download.

[0036] The systems and methods can then obtain a selection input, which can describe a selection of a particular model generation dataset among the multiple model generation datasets, For example, a user can navigate through a carousel of model generation datasets and determine a particular model generation dataset of interest, and the user can then select the particular model generation dataset to utilize to query the database.

[0037] In some implementations, obtaining the selection input can include obtaining a selection of a particular model generation dataset of the plurality of model generation datasets and obtaining a cropping input. The cropping input can describe a portion of the particular model generation dataset. The portion of the particular model generation dataset can be segmented to generate a cropped model generation dataset. In some implementations, one or more search results can be determined based on the cropped model generation dataset.

[0038] The system and method can determine one or more search results based on a particular model generation dataset. The one or more search results can be determined based on an association with a resource dataset determined to be similar to the selected model generation dataset. For example, the resource dataset can be a song determined to be similar to the model generation audio clip.

[0039] The one or more search results can then be provided as output. The one or more search results can be provided in a search results page. In some embodiments, the one or more search results can be provided adjacent to one or more model-generating datasets. For example, the one or more search results can be provided in a panel of a user interface, and the one or more model-generating datasets can be provided in the same panel and / or a different panel.

[0040] The systems and methods disclosed herein can utilize a selection interface to generate prompt inputs that are processed for generation. The selection interface can include a plurality of user interface elements (e.g., chips or tiles) that can include words, symbols, and / or icons associated with a plurality of potential prompt terms. For example, the systems and methods can include providing an image generation interface for display. The image generation interface can include a plurality of category user interface elements. Each category user interface element can be associated with a different generation category. The systems and methods can include obtaining first input data. The first input data can be associated with a selection of a particular category user interface element of the plurality of category user interface elements. The particular category user interface element can be associated with a particular category. The systems and methods can include providing a plurality of descriptor user interface elements for display in the image generation interface. Each descriptor user interface element can be associated with a different descriptor. The systems and methods can include obtaining second input data. The second input data can be associated with a selection of one or more particular descriptor user interface elements of the plurality of descriptor user interface elements. In some implementations, the one or more particular descriptor user interface elements can be associated with one or more particular descriptors. The systems and methods may include processing data associated with one or more particular descriptors with a machine learning image generation model to generate one or more model-generated images, and providing the one or more model-generated images for display in an image generation interface.

[0041] The system and method can obtain an image generation interface for display. The image generation interface can include a plurality of categorical user interface elements. In some implementations, each categorical user interface element (e.g., chips, tiles, and / or drop-down elements) can be associated with a different generation category (e.g., scenes, murals, clothing, and / or video games).

[0042] Next, first input data can be obtained. The first input data can be associated with a selection of a particular category user interface element of the plurality of category user interface elements. The particular category user interface element can be associated with a particular category. In some implementations, the particular category can be associated with clothing. Additionally and / or alternatively, one or more particular descriptors can be associated with one or more clothing terms describing the clothing item.

[0043] A number of descriptor user interface elements (e.g., chips, tiles, and / or drop-down elements) can be provided for display in the image generation interface. Each descriptor user interface element can be associated with a different descriptor (e.g., an adjective and / or a complementary noun or verb associated with a particular category). A descriptor may be a general descriptor for a number of different categories. Alternatively and / or in addition, a number of descriptors may be determined and / or provided based on a selected category (e.g., clothing material and / or brand may be provided based on a clothing category being selected).

[0044] Second input data can then be obtained. The second input data can be associated with a selection of one or more particular descriptor user interface elements of the plurality of descriptor user interface elements. The one or more particular descriptor user interface elements can be associated with one or more particular descriptors. Additionally and / or alternatively, free-form text input can be obtained. For example, a text input box can be provided for display and available for receiving free-form text associated with the one or more additional descriptors.

[0045] Data associated with one or more particular descriptors can be processed using a machine learning image generation model to generate one or more model-generated images. In some implementations, a prompt can be generated based on the category selection and the descriptor selection(s). Additionally and / or alternatively, a particular machine learning image generation model can be obtained based on the selected category. The prompt can be a structured prompt based on a selection hierarchy (e.g., category of descriptors) and / or based on time of selection.

[0046] One or more model-generated images can be provided for display in an image generation interface. The one or more model-generated images can be provided in a carousel interface, in a list, in a grid, and / or in a slide show interface.

[0047] In some implementations, the systems and methods can receive third input data. The third input data can describe a selection of one or more model generated images. The systems and methods can then determine one or more search results based on the one or more model generated images. The one or more search results can be associated with one or more objects. A search result interface can then be provided for display. The search result interface can provide the one or more search results for display.

[0048] Additionally and / or alternatively, editing input data can be obtained. The editing input data can describe a request to replace one or more first features of one or more model generated images with one or more second features. One or more updated model generated images can then be generated based on the editing input data. The editing input data can be associated with a color change. In some implementations, the editing input data can be associated with a texture change.

[0049] The prompt generation interface and / or dataset generation and search interface may be provided to a search application, a browser application, a shopping application, a finder application, an image recognition application, an augmented reality application, a virtual reality application, a discovery application (e.g., a suggestion application), an image generation application, and / or a web application or platform.

[0050] The systems and methods may include a re-render option. For example, a user may deselect one or more prompt interface elements and select additional interface elements and then render a new dataset and / or a new portion of the generated dataset. A user may select a portion of a selected dataset to replace a portion of another dataset. Additionally and / or alternatively, a user may select a replacement color, a replacement material, a replacement texture, and / or a replacement design.

[0051] The generated dataset (e.g., model-generated images) and / or one or more search results can be saved. For example, the model-generated images and / or products associated with the search results can be added to a user-specific library, gallery, virtual closet, and / or collection. Subgroups and / or subcollections can be generated based on colors, determined aesthetics, and / or determined associations. The subcollections and / or subgroups can include data from other applications (e.g., social media applications). In some implementations, prompts can be suggested based on data from other applications and / or based on the generated collection. For example, media content and / or web content can be saved and / or interacted with, which can then be utilized to generate suggested prompts.

[0052] In some implementations, the systems and methods can be utilized to find real-world clothing, existing art, and / or potential travel locations. For example, a category (e.g., clothing, art, and / or locations) can be selected. A number of suggested prompt terms user interface elements (or a number of descriptor user interface elements) associated with the category can be provided for display. A user can select a number of suggested prompt terms to generate a prompt that can be provided to an image generation model to generate a model-generated image. A user can determine that the model-generated image matches a desired search. The model-generated image can then be searched to find clothing items, artwork, and / or travel locations that match the depicted features of the model-generated image.

[0053] The systems and methods may be performed based on cloud processing. Alternatively and / or in addition, processing may be performed locally on a user device and / or via a device at a retail store. The systems and methods may be embedded in a search interface.

[0054] The prompts may include a mood and / or aesthetic associated with a content item, a time period, a genre, and / or a location. The image generation model may include a text-to-image diffusion model (e.g., the text-to-image diffusion model of Imagen, Google Research (Nov. 25, 2022, 3:40 PM), https: / / imagen.research.google / ). The image generation model may include a transducer model (e.g., a T5-XXL encoder).

[0055] The systems and methods may utilize model-generated images as queries, utilize prompt inputs as queries, and / or utilize metadata and / or inputs associated with a user as queries. For example, the selected model-generated images and prompt inputs may be processed by a search engine to determine one or more search results. The multimodal search queries may include multimodal embeddings, feature recognition and text query generation, image-based search with text-based ranking, text-based search and image-based ranking, and / or conditional processing.

[0056] It can be difficult to articulate concepts and ideas for a search, and some concepts cannot be articulated at all, which can create problems with search result coverage. Additional issues may include not knowing what terms to use, needing unique content, vocabulary boundaries between the user and industry, only partial results, and / or irrelevant search results.

[0057] The systems and methods disclosed herein can leverage AI generative models to generate images that can be considered and selected to serve as search queries. In particular, images can provide more context for what a user is requesting during a search, thereby enabling a more targeted search than text alone.

[0058] Traditional searches for clothing, art, movies, and / or music can be difficult if the user does not have examples to provide to the search engine. Free-form text and / or Boolean strings provided to the search engine as text queries may provide mixed and / or unaligned search results that may be off-point and / or include only a portion of the search query. Refining those searches and / or reviewing those search results may be time intensive and non-intuitive. Image queries may provide more tailored results because images may contain features that cannot be descriptively described via succinct text. However, users may not have access to images of what they are looking for during a search and / or the user may base their search on real-world examples that they know based on real-world experience (e.g., the user may be searching for real-world examples of things they have imagined).

[0059] Additionally, utilizing artificial intelligence techniques to generate images and / or other datasets can be non-intuitive, may be open-ended, and may be time-consuming. Image generation systems such as DALLE ("DALL-E 2," OpenAI (April 6, 2022), https: / / openai.com / dall-e-2 / .) utilize a prompt input box to receive free-form text that is processed to generate one or more images. However, when a user uses the prompt input box, the user may struggle with which words to use and / or may be dissatisfied with the generated image because one or more of the entered words may not be utilized in the orientation desired by the user (e.g., a user may enter "fisheye" in relation to an image capture lens to describe a desired distortion, but the model may generate an image that includes a fish).

[0060] The systems and methods of the present disclosure provide several technical effects and advantages. As an example, the systems and methods can provide an interactive user interface that can utilize searches leveraging machine learning model outputs. In particular, the systems and methods disclosed herein can utilize machine learning models to generate outputs that can then be utilized as queries to query a search engine. For example, the systems and methods can provide prompt inputs to an image generation model that can generate a plurality of model-generated images. A user can then select one or more model-generated images that match a desired product or object. The selected model-generated image(s) can then be input to a search engine that can output one or more search results associated with products and / or objects that are determined to be similar to the provided image. The present disclosure can enable searching and retrieval of search results in a more efficient and / or faster manner. In particular, the present disclosure can enable more versatile searching and retrieval of search results based on different types of inputs. In the present disclosure, a selection of one or more images can be used to determine search results. Furthermore, in the present disclosure, one or more images can be model-generated by an image generation model. This essentially increases the versatility of searching and retrieval by expanding the range of inputs that can be provided as part of the searching and retrieval process. The systems and methods can enable searching and retrieval of search results based on images that did not previously exist but may have been newly generated for this purpose, thereby providing a mechanism for entering search queries that would not be possible without implementing an image generation model for the overall process as described herein. The present disclosure can thereby provide improved search and retrieval operations leveraging the image generation model in combination with determining search results.

[0061] Another technical advantage of the systems and methods of the present disclosure is the ability to leverage one or more user interface elements to provide suggested inputs to a machine learning model. For example, multiple category user interface elements may comprise user interface elements associated with different categories for dataset generation. Multiple descriptor user interface elements may be provided to enable more detailed prompt generation. Multiple descriptor user interface elements may be provided for display and / or refined based on the selection of a particular category. Different user interface elements may lead to more detailed prompt generation based on terms for which the model may be specifically trained. Additionally, the increased versatility of the search and retrieval process according to the present disclosure may enable faster and / or more accurate determination of desired search results. Text queries to a search engine may provide mixed and / or unaligned search results that may be off-target and / or may include only a portion of the search query. Repeated iterations of updating text queries and searching may lead to heavy use of processor power, heavy use of available bandwidth, and heavy drain on the battery of a user device. The present disclosure may enable more versatile inputs to a search engine based on model-generated images. This improves the accuracy of the input search query and allows the input search query to be tailored or targeted, which in turn allows for more efficient use of processor power, available bandwidth and battery in search and retrieval operations.

[0062] Another example of a technical effect and benefit relates to improved computational efficiency and improved capabilities of computing systems, for example, the systems and methods disclosed herein can utilize cloud computing to provide immersive artificial intelligence-based capabilities to user devices with limited computational capabilities.

[0063] Exemplary embodiments of the present disclosure will now be described in further detail with reference to the figures.

[0064] 1 illustrates a block diagram of an exemplary model-generated image retrieval system 10 in accordance with an exemplary embodiment of the present disclosure. In particular, the model-generated image retrieval system 10 can include obtaining user data 12, generating one or more model-generated images 16, and determining one or more search results 20 based on the one or more model-generated images 16.

[0065] For example, user data 12 can be obtained from a user computing system. The user data can include prompt inputs, historical data (e.g., data describing user search history, user purchase history, user browsing history, etc.), profile data, user selection data, and / or preference data. The prompt inputs can include free-form prompt inputs and / or generated prompt inputs generated based on one or more tile selections of a user interface. The prompt inputs can describe one or more attributes that the user requests to be rendered in the generated image. The prompt inputs can include the subject matter of the image (e.g., the environment and / or one or more objects) and one or more details of the subject matter (e.g., color, style, material, etc.).

[0066] The user data 12 can be processed using a diffusion model 14 to generate one or more model-generated images 16. The diffusion model 14 can be a machine learning image generation model and can be trained to process text data and / or images to generate one or more images. The one or more model-generated images 16 can include an object having one or more attributes and can be associated with one or more details of the object and a prompt input.

[0067] The one or more model generated images 16 can then be provided to a search engine 18 to determine one or more search results 20. The one or more model generated images 16 may be provided to the search engine 18 automatically upon generation and / or in response to one or more user inputs (e.g., selection of a search option and / or selection of a particular image). In some implementations, the search engine 18 may further process the user data 12 with the one or more model generated images 16 to determine one or more search results 20. The one or more search results 20 may be determined based on one or more visual similarities between the one or more model generated images 16 and one or more images associated with the one or more search results 20. The search results 20 may include image search results, website search results, and / or marketplace search results. For example, the search results 20 may include products determined to be visually similar to one or more objects depicted in the one or more model generated images 16.

[0068] In particular, the model-generated image retrieval system 10 can obtain user data 12 describing items that may be of interest to a user (e.g., based on explicit input, learned preferences, and / or usefulness). The model-generated image retrieval system 10 can generate visualizations of the items (e.g., one or more model-generated images 16). The user can select particular images that are of interest to the user. The model-generated images can then be provided to a search engine 18 to determine real-world products that are visually similar to the "imaginary" item.

[0069] 2 illustrates a block diagram of an exemplary machine learning model processing and search system 200 according to an exemplary embodiment of the present disclosure. In particular, the machine learning model processing and search system 200 includes generating a prompt input 206, providing the prompt input 206 to a machine learning model 208 (e.g., a dataset generation model) to receive a plurality of machine learning model outputs (e.g., a plurality of model generation datasets), obtaining a selection input, providing the selected machine learning model output to a search engine 216, and receiving one or more search results.

[0070] The prompt input 206 can be generated and / or determined based on one or more tip selections 202, one or more free-form text inputs 204, and / or one or more media file inputs. For example, a plurality of user interface tips associated with different candidate prompt terms can be provided for display. A user can then select a subset of the plurality of user input tips, which can then be utilized to generate a prompt input 206 that describes the plurality of selected prompt terms. In some implementations, the prompt input 206 can include a prompt term via the free-form text 204.

[0071] The prompt input 206 can then be processed with a machine learning model 208 (e.g., a dataset generation model (e.g., an image generation model)) to generate a plurality of machine learning model outputs (e.g., a plurality of images). The plurality of machine learning model outputs can include a first machine learning model output 210 (e.g., a first model-generated image), a second machine learning model output 212 (e.g., a second model-generated image), and a third machine learning model output 214 (e.g., a third model-generated image).

[0072] The user may then select a particular machine learning model output to leverage to search resources (e.g., to search for art, to search for clothing products, and / or to search for another type of product). For example, the first machine learning model output 210 may be input to a search engine 216 to obtain one or more first search results 218, the second machine learning model output 212 may be input to a search engine 216 to obtain one or more second search results 220, and the third machine learning model output 214 may be input to a search engine 216 to obtain one or more third search results 222. The search results may be determined based on the determined similarity scores.

[0073] 3A-3M show diagrams of an example search interface 300 according to an example embodiment of the present disclosure. In particular, FIG. 3A may show an example initial interface appearance with a free form input box 302 and an interactive prompt generation element 304. The interactive prompt generation element 304 can be selected to update the search interface 300 to include multiple interactive user interface elements for generating prompt input.

[0074] FIG. 3B shows an updated interface with a category indicator 306 indicating a selected category for creation, a number of category user interface elements 308 for selection to select a particular category for generation, a number of first descriptor user interface elements 310, a number of second descriptor user interface elements 312, a generation user interface element 314 for initiating dataset generation, and a preview window 316 for previewing the model generated dataset.

[0075] In response to a selection of a different one of the plurality of category user interface elements 308, the search interface 300 may again update to update the category indicator 306 to indicate the updated selected category (e.g., as shown in FIG. 3C ). Additionally and / or alternatively, the plurality of first descriptor user interface elements 310 and the plurality of second user interface elements 312 may be updated. The descriptor user interface elements may be updated based on the particular selected category to include object types and / or adjectives associated with the particular selected category.

[0076] In Figure 3D, the category is "Fashion Designer" associated with "imagining" an item of clothing. Figure 3D shows that a first descriptor (i.e., a dress descriptor) of the plurality of first descriptors 310 has been selected. Thus, the user is generating a request to generate an image including a dress.

[0077] 3E, the selected first descriptor includes an indication of selection, and further, a particular second descriptor user interface element of the plurality of second descriptor user interface elements 312 (i.e., the user interface element that includes the text "Baroque") is shown as being selected.

[0078] In Figure 3F, free form text (i.e., "feathered") is added to the input box and a generated user interface element 314 is selected, thereby initiating generation, and a buffer indicator is provided for display in preview window 316. The buffer indicator may indicate that the prompt input is being generated and / or processed with the image generation model to generate a number of model generated images.

[0079] In FIG. 3G, the search interface 300 is updated to provide a model image carousel 318 with a first model-generated image provided for display within a preview window 316. The updated search interface may include a copy image prompt interface element 320 that may be selected to utilize the currently previewed image as a query image for a search. The first model-generated image may describe the generated image based on the category "fashion designer", a first descriptor "dress", a second descriptor "baroque", and a free-form text input "feathered". The user may then navigate through the model-generated image carousel 318 to view various model-generated images in the preview window 316. Once the user has determined a particular model-generated image to utilize as a query, the user may select the copy image prompt interface element 320.

[0080] In Figure 3H, a second model generated image is provided for display in preview window 316. In Figure 3I, a fourth model generated image is provided for display in preview window 316. The user can then select a copy image prompt interface element 320. A selection input can be received and the selected model generated image can be utilized as an image query to query one or more databases.

[0081] 3J, a search results panel 322 can be provided for display in response to receiving a selection input. In some implementations, a cropping interface 324 can be provided to enable cropping of the selected model-generated image to refine the search results and / or expand the search query. Other interface options 326 can be provided for navigating between search options, optical character recognition options, and conversion options. The search results panel 322 can include a number of search results 328 that are provided for display in response to the model-generated image query. The number of search results 328 can be determined based on associations with the image determined to exceed a similarity threshold.

[0082] The user may scroll through multiple search results 328 within the search result panel 322 to determine a particular search result 330 of interest (e.g., as shown in FIG. 3K). A selection input may be received describing a selection of the particular search result 330. The image generation interface may then be replaced with a browser window 332 that displays at least a portion of the resources associated with the particular search result 330 (e.g., as shown in FIG. 3L). The user may then interact with the web resources within the browser window 332 (e.g., as shown in FIG. 3M). For example, the user may purchase a dress in the browser window 332, where the purchased dress resembles the dress shown in the selected model generated image.

[0083] 4 illustrates a block diagram of an exemplary machine learning model processing and retrieval system 400 in accordance with an exemplary embodiment of the present disclosure. In particular, one or more external inputs 404 and / or learned information 402 can be utilized to design prompts 408, which can be processed by an image generation model 410 to generate model-generated images, which can then be retrieved.

[0084] For example, stored and / or learned information 402 (e.g., fashion knowledge, personalization (e.g., based on stored data associated with a user), and / or trends (e.g., purchasing habits, social media trends, and / or search trends)) can be obtained and utilized to generate prompts and / or to suggest prompt inputs for selection via selectable user interface elements. One or more external inputs 404 (e.g., videos, images, music, and / or collections) can be obtained to generate prompt inputs. For example, image, video, and / or audio data can be provided to be processed to determine suggested prompts.

[0085] The learned information 402 and / or one or more external inputs 404 can be utilized to generate a palette 406 of data to be selected for prompt generation. For example, the palette 406 can include suggested images, suggested music, suggested videos, and / or suggested terms that can be selected. The selection data can then be utilized to design a prompt 408 that can be processed using an image generation model 410 to generate a number of model-generated images that can be provided for display in a canvas interface 412. One or more model-generated images can be selected. The one or more selected model-generated images can be processed using a search block 414 that can include object recognition, image matching, image embedding, and / or retrieval. The search block 414 can include feature recognition and can be searched to determine one or more resources that contain similar data. In some implementations, the search block 414 can utilize the Multitasking Unified Model (Pandu Nayak, “MUM: A new AI milestone for understanding information,” Google: the Keyword (May 18, 2021, https: / / blog.google / products / search / introducing-mum / ) for semantic and context-based understanding through search.

[0086] The output of the search block 414 may include a number of search results that may include products for sale. The product search results may be provided for display in an interface 416 that may enable a user to purchase products associated with the products they have "imagined" via the image generation model 410 and the prompt generation interface.

[0087] In some implementations, one or more of the search results can be utilized to edit (422) the palette 406 for future suggestions and prompt generation. Additionally and / or alternatively, a user can interact with the model-generated image displayed in the canvas interface 412 to modify (418) the model-generated image (e.g., to create a transformation of the depicted object and / or redo at least a portion of the image). In some implementations, one or more of the model-generated images can be saved (420) to a collection associated with the user.

[0088] 5 illustrates a flow chart diagram of an exemplary method for functioning in accordance with an exemplary embodiment of the present disclosure. Although FIG. 5 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the specifically depicted order or arrangement. Various steps of method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0089] At 502, the computing system can obtain a prompt input. The prompt input can include one or more terms (e.g., one or more words (e.g., “jacket, feathered, brown, majestic”) that can describe an interpolation of a requested instance to request a view rendering of a brown feathered jacket having a majestic aesthetic. In some implementations, the prompt input can include selection data describing one or more selection inputs associated with one or more selectable user interface elements and / or one or more text inputs including text input into a text input box. The prompt input can include one or more terms describing the absence of a particular detail. The one or more terms describing the absence of a particular detail can be associated with a request to generate an image without the particular detail. The particular detail can include an object, a material type, a color, a style, an attribute, and / or a shape.

[0090] In some implementations, obtaining the prompt input can include providing a plurality of selectable user interface elements for display within the graphical user interface. The plurality of selectable user interface elements can be associated with a plurality of candidate prompt terms (e.g., object types, categories, scene or object descriptors, and / or aesthetics). Selection data can then be obtained. The selection data can describe a first selectable user interface element (e.g., a first interactive tip) and a second selectable user interface element (e.g., a second interactive tip). The first selectable user interface element can be associated with a first prompt term (e.g., a noun, verb, adjective, and / or adverb associated with the requested concept), and the second selectable user interface element is associated with a second prompt term (e.g., a noun, verb, adjective, and / or adverb associated with the requested concept). For example, the prompt input can include a first prompt term and a second prompt term associated with the selected first user interface element and the selected second user interface element. The prompt terms may describe a subject (e.g., a landscape, an amusement park, a dress, and / or a purse), a quality (e.g., Tron-like, science fiction, made of plants, a particular video game aesthetic, baroque, cyborg, and / or covered in sequins), and / or an action (e.g., dancing, running, playing football, and / or cheering).

[0091] In some embodiments, the plurality of selectable user interface elements can be provided for display in response to obtaining a prompt selection request. The prompt selection request can describe an input for receiving a graphical user interface of a selectable user interface chip. The prompt selection request can be received by the user computing system during display of an entry point interface including a text input box for receiving user input data for generating a machine learning model output based on a user-provided text prompt. A plurality of candidate prompt terms associated with the plurality of selectable user interface chips can be predetermined. A first prompt term can be associated with a type of object. A second prompt term can be associated with a particular descriptive characteristic, and one or more model-generated images can describe a particular object of the type of object having the particular descriptive characteristic.

[0092] In some implementations, the prompt input may include a multimodal prompt input. The multimodal prompt input may include a prompt image and a prompt text. The prompt image may describe a particular object and / or a particular environment with one or more particular details. In some implementations, the prompt input may be an image search result selected by a user to augment the refined search. The image search result may comprise a plurality of other search results in response to obtaining a search query (e.g., a text query, an image query, and / or a multimodal query). Alternatively and / or in addition, the prompt image may include a user image and / or a previously generated model generated image. The prompt text may describe one or more particular details of the prompt image to augment. For example, the prompt text may describe a request to render a particular object and / or a particular environment without one or more particular details. The one or more particular details may be substituted for one or more other details and / or may be substituted for predicted background pixels. In some implementations, the prompt text may describe a request to include additional details (e.g., additional objects, additional colors, additional shapes, and / or additional materials).

[0093] At 504, the computing system can process the prompt input with the image generation model to generate one or more model-generated images. The one or more model-generated images can be generated based at least in part on the one or more terms. The image generation model can be trained with multiple training images. The image generation model can be trained on a particular subject and / or a particular object type (e.g., a particular clothing item). Alternatively and / or in addition, the image generation model can be trained broadly. The training can include label training, where the labels can be utilized to determine and / or generate selectable user interface elements. For example, a particular label can be associated with multiple images (e.g., a "shirt" label can be associated with images of multiple different shirts, and / or a "fur" label can be associated with multiple images associated with multiple furs of clothing and / or home furnishings items). Descriptors of the labels can then be utilized to generate selectable user interface elements for the descriptors utilized as prompt terms. The one or more model-generated images can describe the generated environment and / or the generated objects without specific details. For example, the prompt input may include terms describing a request for a particular object (e.g., a dress) without particular details (e.g., a ribbon and / or buttons), and the image generation model may process the prompt input to generate an image without the particular details (e.g., the dress without the ribbon and / or buttons).

[0094] In some implementations, one or more model-generated images can be provided for display with one or more terms in a graphical user interface. For example, multiple model-generated images can be generated and provided for display in an image carousel. One or more model-generated images can be provided for display for interaction. A user can select a portion of a particular model-generated image for augmentation. For example, a user can remove features (e.g., remove an object from the scene, remove an accessory, and / or match clothing), modify features (e.g., change texture and / or change color), and / or add features (e.g., add an object, add an ascent, and / or add an accessory) by providing one or more augmentation inputs.

[0095] In some multimodal prompting implementations, the prompt image and the prompt text can be processed using an image generation model to generate a model-generated image. The model-generated image can describe a model-generated object. The model-generated object can describe a specific object that is augmented based on the prompt text. In some implementations, the model-generated image can describe a specific object without one or more specific details.

[0096] At 506, the computing system can obtain a selection input. The selection input can describe a selection of one or more model generated images. The selection input can describe a request to query one or more databases for content and / or items similar to the content and / or items in the selected model generated images. The selection input can include one or more selections of one or more portions of the selected model generated images that are of interest. The one or more portions can be segmented (or cropped) and then input to a search engine.

[0097] At 508, the computing system can determine one or more search results based on the one or more model-generated images. The one or more search results can be associated with one or more objects. In some implementations, the one or more search results can be associated with one or more products. Additionally and / or alternatively, the one or more search results can include one or more action links associated with the one or more products. The one or more action links can be associated with a purchasing interface for the one or more products. The one or more search results can be determined based on one or more labels associated with the model-generated images. Alternatively and / or additionally, the model-generated images can be processed using the embedding model to generate embeddings. The embeddings can then be utilized to determine similar embeddings, which can be associated with the one or more search results. The one or more prompt terms can be utilized to determine one or more search engines. For example, the one or more search results can be obtained by generating a combined query using the prompt terms and the model-generated images.

[0098] In some implementations, determining one or more search results based on the one or more model-generated images can include providing the one or more model-generated images to a search engine and receiving one or more search results from the search engine, which can be a general search engine and / or can be a database-specific search engine (e.g., a shopping search engine).

[0099] At 510, the computing system can provide a search result interface. The search result interface can include one or more search results provided for display. The search result interface can be a search result page. The search result interface can include a list of search results, an augmented reality experience interface, and / or a viewport for viewing a preview of resources associated with one or more search results.

[0100] 6A-6C show diagrams of example finder entry points according to an example embodiment of the present disclosure. For example, a user may capture an image 602 (e.g., an image of an individual having multiple clothing items) for search (e.g., as shown in FIG. 6A). The finder interface may identify multiple objects in the captured image. The finder interface may then provide multiple indicators 604 associated with the multiple recognized objects in the image.

[0101] In Figure 6B, an indicator associated with the recognized sunglasses may be selected. A selection may be obtained and the selected object may be segmented and displayed along with one or more search results (606) that are determined to be associated with products similar to the depicted sunglasses. The user may then scroll to view more search results. A scroll input may be received and additional search results 608 may be provided for display.

[0102] In FIG. 6C, an indicator associated with the recognized coat may be selected. A selection may be obtained and the selected object may be segmented and displayed along with one or more search results (610) determined to be associated with products similar to the depicted coat. The user may then scroll to view more search results. A scroll input may be received and additional search results 612 may be provided for display. The additional search results 612 may be provided for display along with an entry point for utilizing an image generation model to generate a "fictional" clothing item.

[0103] 7A-7D show diagrams of an example search interface according to an example embodiment of the present disclosure. The image generation and search interface may include obtaining an initial object type selection to utilize as a starting point. A template object type may then be provided for display (e.g., 702 as shown in FIG. 7A). The image generation and search interface may then provide an initial set of options 704 for prompt generation (e.g., surprise options (e.g., for randomized process generation), trend options (e.g., trending views and selections for prompt generation), color options (for selecting colors for prompt generation), and / or material options (for selecting material for prompt generation).

[0104] In some implementations, a user can provide a prompt including a request for removal, absence, and / or replacement of one or more elements (e.g., one or more materials, one or more colors, one or more details, and / or one or more attributes). For example, the image generation and retrieval interface can include one or more user interface elements for receiving input to generate a prompt to request the absence of an element. A generated model-generated image can then be generated to describe the particular object that does not include the element requested to be absent. The absence request can be associated with augmenting a current image (e.g., a captured image, a downloaded image, and / or a previously generated model-generated image) and / or can be provided with a prompt input describing the inclusion and / or absence of the object along with details to include.

[0105] In Figure 7B, a color palette 706 and a material palette 708 are provided for display. The user can then select one or more colors and / or one or more materials to be utilized in generating prompts that are provided to an image generation model to "imagine" a product based on the request. Specifically, the user selected a light green user interface element from the color palette 706 and two materials (e.g., faux fur and patent leather) from the material palette 708.

[0106] Once the prompt input has been processed, an "imagine" buffer representation 710 can be provided for display (e.g., as shown in FIG. 7C). A model-generated image 712 can then be provided for display. The user can then modify the model-generated image, redo the prompt generation, save the model-generated image, and / or share the model-generated image.

[0107] In some implementations, the image generation and search interface may provide previously generated images to the model-generated images in the creations collection 714 (e.g., as shown in FIG. 7D). The model-generated images can then be searched to determine products similar to the depicted object. Links to resources associated with one or more search results can then be provided for display in the shop panel 716.

[0108] 8A-8C show diagrams of an exemplary search interface according to an exemplary embodiment of the present disclosure. For example, the model generated image including the clothing item of FIGS. 7A-7D may be modified. In FIG. 8A, a model generated image 802 is provided for display, and a camera roll popup 804 is provided for display in response to selection of a camera icon.

[0109] In FIG. 8B, an image has been selected and provided for display in the interface (e.g., as shown at 806), and an imagination buffer screen 808 is provided for display in response to an input describing a request to re-imagine an article of clothing based on the content of the selected image.

[0110] In Figure 8C, a first updated model generated image 810 and a second updated model generated image 812 can be generated and provided for display. The first updated model generated image 810 can include updated clothing based on the colors of the selected image that individually replicates the movements of the selected image (e.g., the selected image can be a GIF that includes a dancing movement). The second updated model generated image 812 can include updated clothing without the individual. The updated model generated images can be displayed together in a creation panel 814.

[0111] 9A-9E show diagrams of example search interface entry points according to an example embodiment of the present disclosure.

[0112] In particular, Figure 9A includes multiple entry points within the search interface. For example, a "Start Dreaming" tile can be provided for display adjacent to the image search results in the grid 902. A "Start Dreaming" chip can be provided for display below the image search results in the expanded image viewer 904. Additionally and / or alternatively, a "Start Dreaming" chip interface element can be provided in a search results pane of the image recognition interface 906. Users can interact with the "Start Dreaming" tile, the "Start Dreaming" chip, and / or the "Start Dreaming" chip interface element to initiate the prompt generation and image generation process.

[0113] 9B illustrates exemplary entry points for prompt generation and image generation displayed on a general search results page. For example, entry point interface elements can be provided in a related searches section 910, a refinement search tile carousel 912, and / or tiles in an image search results panel 914.

[0114] 9C illustrates exemplary entry points for prompt generation and image generation displayed in the finder and recognition applications. For example, entry point interface elements can be provided in a tip carousel adjacent to a recognition object tip 920, in a category function tab carousel 922, and / or in a search result popup 924.

[0115] 9D illustrates example entry points for prompt generation and image generation that are displayed for varying search result types. For example, on the video search results page 930, an entry point interface element may be provided below a segment identifier of the video search results. On the fashion search results page 932, an entry point interface element may be provided with specific search results to enable prompt generation. Additionally and / or alternatively, on the image search results page 934, an entry point interface element may be provided with specific search results to enable prompt generation.

[0116] 9E illustrates exemplary entry points for prompt generation and image generation displayed in a video player application. Entry point interface elements can be provided below the playing video with randomized model-generated images 940, below search results based on recognized objects in the video 942, and / or in a chip carousel adjacent to the recognized object chips 944.

[0117] FIG. 10 illustrates a diagram of an exemplary user collection interface, according to an exemplary embodiment of the present disclosure. For example, the user collection interface can provide a landing experience that offers various inspiration buckets and personalization methods. In particular, the interface can provide a "Recommended for You" panel 1002 that can be determined based on previous creations, based on preferences, and / or based on historical data (e.g., purchase history, search history, and / or browsing history). The user collection interface can include a dream closet 1004 that displays multiple previous model generation datasets. The user can interact with a preference interface 1006 to select various preferences that can be utilized for further "Recommended for You" suggestions, which can include automated prompt generation and processing for automatically generated model generation datasets.

[0118] 11A-11B show diagrams of an exemplary image retrieval system according to an exemplary embodiment of the present disclosure.

[0119] In FIG. 11A , an example image search system can include displaying a model generated image 1102 and a user can open a refinement search parameters interface 1104 that can enable the user to select various search refinement parameters (e.g., brand, manufacturer, price range, location, and / or quality). Search results can then be determined and provided for display. For example, product search results 1106 can comprise an image, one or more descriptors, and / or a link. Place search results 1108 can be provided for display to provide locations where similar products can be purchased. Place search results 1108 can comprise an image, one or more descriptors, an address, a rating, and / or a link.

[0120] In FIG. 11B, clothing items are "imagined" to be utilized for a search. A particular clothing item may be selected (e.g., a jacket may be selected). A customization interface may then be provided. The customization interface may include a trends tab, a types tab 1110, a colors tab 1112, a materials tab 1114, and / or a details tab. In response to each selection, an updated rendering preview may be provided. The trends tab may include trend styles provided for display. The types tab 1110 may provide selectable options for different types of jackets. The colors tab 1112 may provide selectable options for various colors to apply to the jacket and may include color subtabs (e.g., earth, popular, and / or pastel). The materials tab 1114 may provide selectable options for various materials to apply to the jacket (e.g., linen, wool, satin, cotton, and / or polyester). The user may pull up the popup to expand the popup 1116. The user can make various selections, which can then be processed to generate a model-generated image, which can then be retrieved (eg, similar to FIG. 11A).

[0121] FIG. 12 illustrates a diagram of an exemplary collection interface, according to an exemplary embodiment of the present disclosure. In particular, a user may create and label collections. For example, a user may create a “Picnic Day” collection, which may include a title label 1202, a number of saved images 1204 (e.g., a number of model-generated images), and a panel for selecting additional datasets (e.g., suggested model-generated datasets) to add to the collection. In some embodiments, collections may be generated automatically. For example, a collection may be generated 1208 that is associated with determined favorite content items.

[0122] FIG. 13 illustrates a diagram of an exemplary collection interface, according to an exemplary embodiment of the present disclosure. The collection interface may include multiple tabs 1306 (e.g., a “Your Zone” tab, a products tab 1302, a music tab 1304, and / or an entertainment tab). The products tab 1302 may include multiple products 1308 for customization (e.g., suggested products and / or products associated with one or more model-generated datasets), which may include generating new model-generated products and / or modifying existing model-generated products. The music tab 1304 may include suggested music and / or saved music. The suggested music and / or saved music may be found based on generating audio data using a dataset-generated model that processed the prompt input. The generated audio data may have been searched to determine music similar to the generated audio data.

[0123] 14 shows a diagram of an example collection interface according to an example embodiment of the present disclosure. The collection interface may include an entertainment tab 1402, which may include saved entertainment collections and / or suggested entertainment collections. A user may scroll the entertainment tab to a bottom 1404, which may include social media platform specific collections and / or may show a particular collection 1406.

[0124] FIG. 15 illustrates a diagram of an exemplary suggestion interface, according to an exemplary embodiment of the present disclosure. Specifically, at 1502, a moods tab is provided for display where the user can receive suggestions based on moods, which may include "creative," "cozy," and "relaxed." The suggestions may be tailored based on the user data and the selected mood. At 1504, a place tab is provided for display with multiple indicators associated with different places, which may include an initial aesthetic image associated with the place. At 1506, the user may have selected a selected Dumbo Street style indicator, and multiple suggested clothing items are provided for display. The suggested clothing items may be model-generated images of clothing based on the aesthetic and / or clothing style of the place with one or more user-specific preferences, which may be manually entered preferences and / or machine learning preferences based on the user's purchases, closet, browsing history, and / or search history. At 1508, a Peers tab is provided for display, which may include products, objects, and / or model-generated images that “peers” (e.g., social media friends and / or people with similar profile data (e.g., similar locations, similar clothing preferences, and / or similar hobbies)) have added to their own virtual collections.

[0125] 16A-16B show diagrams of an example modification interface, according to an example embodiment of the present disclosure. In particular, a user can modify a model-generated image after the image is generated. For example, FIG. 16A shows modification of a model-generated collar of a dress. A user can select a modification element that can provide multiple options for display, which may include indicators of different parts of the jacket that can be modified. At 1602, the collar, cuffs, and midsection are shown as being customizable. At 1604, a color is selected and a color palette is provided for display. At 1606, a new color for the color is selected and applied to the model-generated image rendering. At 1606, a complete updated image of the jacket is provided for display.

[0126] FIG. 16B illustrates modification of the collar of the model generated dress where the color is enhanced based on the design in the image. At 1610, the collar, cuffs, and midsection are indicated as being customizable. At 1612, the user selects the Cut and Craft interface element and the modification interface is updated to show the portions of the jacket that can be cut. At 1614, the collar is cut and the user is given the option to select an existing image and / or upload a new image via image capture. At 1616, an image containing a basketball is selected and additional customization options are provided for viewing. At 1618, a modified preview is provided for viewing showing a basketball print on the collar of the jacket. At 1620, the modifications are finalized and the modified jacket can be added to a collection.

[0127] 17A-17C show diagrams of an example product suggestions interface according to an example embodiment of the present disclosure.

[0128] 17A shows a shared creations interface 1702 and a mood / type customization interface 1704. Specifically, the product suggestion interface can include a shared creations interface 1702 that can enable a user to share their own model-generated images (e.g., model-generated clothing), view other users' model-generated images, and interact with other users' model-generated images (for example) (e.g., favorite posts, save posts, and / or share posts). Additionally and / or alternatively, the mood / type customization interface 1704 can enable a user to select example images of a style, aesthetic, and / or mood, which can be utilized to generate model-generated images associated with a particular style, aesthetic, and / or mood.

[0129] In Figure 17B, a tip selection interface 1706 is provided for display, providing tip interface elements for display that can be selected to generate prompt inputs. In particular, the tip selection interface 1706 is associated with selecting preferences that can be stored in association with a user profile that is utilized to tailor the generation and suggestions. Additionally and / or alternatively, a cut and craft interface 1708 may be provided to provide an interface for modifying the generated image.

[0130] 17C, where to find products similar to the "imaginary" product are provided as suggestions in a location-based interface 1710 that further includes a nearby creator suggestions pane. Additionally and / or alternatively, the mood board interface 1712 can provide an interface for users to save their own creations and the creations of others in collections for future viewing.

[0131] 18 illustrates a flow chart diagram of an exemplary method for functioning in accordance with an exemplary embodiment of the present disclosure. Although FIG. 18 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the order or arrangement specifically shown. Various steps of method 1800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0132] At 1802, the computing system can obtain a prompt input. The prompt input can include one or more terms. The prompt input can be generated based on one or more selections of one or more user interface tips that can include text characters and / or icons associated with the terms to utilize to prompt the data generation model. In some implementations, the prompt input can include one or more images, one or more audio clips, and / or latent encoded data.

[0133] At 1804, the computing system can process the prompt input with the data generation model to generate a plurality of model generation datasets. The plurality of model generation datasets can be generated based at least in part on one or more terms. In some implementations, each of the plurality of model generation datasets can be different. The data generation model can be trained to generate the one or more datasets based on a plurality of learning parameters and adjusted based on the prompt input. The model generation datasets can include image data, audio data, multimodal data, text data, latent coding data, and / or sensor data. For example, the plurality of model generation datasets can include a plurality of images (e.g., a plurality of predicted depictions that describe the prompt input), a plurality of audio clips (e.g., a plurality of generated song clips predicted to describe the prompt input), and / or a plurality of video datasets (a plurality of predicted video clips generated based on the prompt input).

[0134] At 1806, the computing system can provide the multiple model generation datasets via a user interface. Providing the multiple model generation datasets via the user interface can include providing the multiple model generation images in an image carousel. The multiple model generation datasets can be provided as a list of links to preview the model generation datasets. Alternatively and / or additionally, the multiple model generation datasets can be transmitted for local download.

[0135] At 1808, the computing system can obtain a selection input. The selection input can describe a selection of a particular model generation dataset among a plurality of model generation datasets. For example, a user can navigate through a carousel of model generation datasets and determine a particular model generation dataset of interest, and the user can then select the particular model generation dataset to utilize to query the database.

[0136] In some implementations, obtaining the selection input can include obtaining a selection of a particular model generation dataset of the plurality of model generation datasets and obtaining a cropping input. The cropping input can describe a portion of the particular model generation dataset. The portion of the particular model generation dataset can be segmented to generate a cropped model generation dataset. In some implementations, one or more search results can be determined based on the cropped model generation dataset.

[0137] At 1810, the computing system can determine one or more search results based on the particular model generation dataset. The one or more search results can be determined based on an association with a resource dataset determined to be similar to the selected model generation dataset. For example, the resource dataset can be a song determined to be similar to the model generation audio clip.

[0138] At 1812, the computing system can provide one or more search results as output. The one or more search results can be provided in a search results page. In some implementations, the one or more search results can be provided adjacent to one or more model generation datasets. For example, the one or more search results can be provided in a panel of a user interface, and the one or more model generation datasets can be provided in the same panel and / or a different panel.

[0139] 19 illustrates a flow chart diagram of an exemplary method for functioning in accordance with an exemplary embodiment of the present disclosure. Although FIG. 19 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the order or arrangement specifically shown. Various steps of method 1900 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0140] At 1902, the computing system can provide an image generation interface for display. The image generation interface can include multiple category user interface elements. In some implementations, each category user interface element (e.g., chips, tiles, and / or drop-down elements) can be associated with a different generation category (e.g., scenes, murals, clothing, and / or video games).

[0141] At 1904, the computing system can obtain first input data. The first input data can be associated with a selection of a particular category user interface element of the plurality of category user interface elements. The particular category user interface element can be associated with a particular category. In some implementations, the particular category can be associated with clothing. Additionally and / or alternatively, one or more particular descriptors can be associated with one or more clothing terms describing the clothing item.

[0142] At 1906, the computing system can provide a plurality of descriptor user interface elements for display in the image generation interface. Each descriptor user interface element can be associated with a different descriptor (e.g., an adjective and / or a complementary noun or verb associated with a particular category). The descriptors can be general descriptors for a plurality of different categories. Alternatively and / or in addition, a plurality of descriptors can be determined and / or provided based on a selected category (e.g., clothing material and / or brand can be provided based on a clothing category being selected).

[0143] At 1908, the computing system can obtain second input data. The second input data can be associated with a selection of one or more particular descriptor user interface elements of the plurality of descriptor user interface elements. The one or more particular descriptor user interface elements can be associated with one or more particular descriptors. Additionally and / or alternatively, free-form text input can be obtained. For example, a text input box can be provided for display and utilized to receive free-form text associated with the one or more additional descriptors.

[0144] At 1910, the computing system can process data associated with one or more particular descriptors using a machine learning image generation model to generate one or more model-generated images. In some implementations, a prompt can be generated based on the category selection and the descriptor selection(s). Additionally and / or alternatively, a particular machine learning image generation model can be obtained based on the selected category. The prompt can be a structured prompt based on a selection hierarchy (e.g., category of descriptors) and / or based on time of selection.

[0145] At 1912, the computing system can provide one or more model-generated images for display in an image generation interface. The one or more model-generated images can be presented in a carousel interface, in a list, in a grid, and / or in a slide show interface.

[0146] In some implementations, the computing system can obtain third input data. The third input data can describe a selection of one or more model generated images. The system and method can then determine one or more search results based on the one or more model generated images. The one or more search results can be associated with one or more objects. A search result interface can then be provided for display. The search result interface can provide the one or more search results for display.

[0147] Additionally and / or alternatively, editing input data can be obtained. The editing input data can describe a request to replace one or more first features of one or more model generated images with one or more second features. One or more updated model generated images can then be generated based on the editing input data. The editing input data can be associated with a color change. In some implementations, the editing input data can be associated with a texture change.

[0148] 20 illustrates a block diagram of an exemplary model-generated image search and customization system 2000 in accordance with an exemplary embodiment of the present disclosure. In particular, the model-generated image search and customization system 2000 can include obtaining user data 2012, generating one or more model-generated images 2016, and determining one or more search results 2020 based on the one or more model-generated images 2016.

[0149] For example, user data 2012 can be obtained from a user computing system. The user data can include prompt inputs, historical data (e.g., data describing user search history, user purchase history, user browsing history, etc.), profile data, and / or preference data. The prompt inputs can include free-form prompt inputs and / or generated prompt inputs generated based on one or more tile selections of a user interface. The prompt inputs can describe one or more attributes that the user requests to be rendered in the generated image. The prompt inputs can include the subject matter of the image (e.g., the environment and / or one or more objects) and one or more details of the subject matter (e.g., color, style, material, etc.).

[0150] The user data 2012 can be processed using a diffusion model 2014 to generate one or more model generated images 2016. The diffusion model 2014 can be a machine learning image generation model and can be trained to process the text data and / or images to generate one or more images. The one or more model generated images 2016 can include an object having one or more attributes and can be associated with one or more details of the object and a prompt input.

[0151] The one or more model generated images 2016 can then be provided to a search engine 2018 to determine one or more search results 2020. The one or more model generated images 2016 may be provided to the search engine 2018 automatically upon generation and / or in response to one or more user inputs (e.g., selection of a search option and / or selection of a particular image). In some implementations, the search engine 2018 may further process the user data 2012 with the one or more model generated images 2016 to determine the one or more search results 2020. The one or more search results 2020 may be determined based on one or more visual similarities between the one or more model generated images 2016 and one or more images associated with the one or more search results 2020. The search results 2020 may include image search results, website search results, and / or marketplace search results. For example, the search results 2020 may include products determined to be visually similar to one or more objects depicted in the one or more model generated images 2016.

[0152] In particular, the model-generated image search and customization system 2000 can obtain user data 2012 describing items that may be of interest to the user (e.g., based on explicit input, learned preferences, and / or usefulness). The model-generated image search and customization system 2000 can generate visualizations of the items (e.g., one or more model-generated images 2016). The user can select particular images that are of interest to the user. The model-generated images can then be provided to a search engine 2018 to determine real-world products that are visually similar to the "imaginary" item.

[0153] The model generated images 2016 may be provided for display in the user interface 2022 for selection. One or more of the model generated images 2016 may be provided via a carousel interface, a thumbnail interface, a slide show interface, and / or a collage interface. A user may select a particular model generated image to search for. Alternatively and / or in addition, a user may enter customization input 2024 to generate a new set of model generated images. The customization input 2024 may include adding one or more features to the generated model generated images, replacing one or more existing features, deleting one or more features, and / or augmenting the prompt input to include one or more additional prompt terms and / or prompt images. For example, a user may request model generated images of a dress to be augmented based on an input image of a particular pattern. The model generated images and the input image may be processed by the diffusion model 2014 to generate an augmented image, and the augmented image may then be provided for display and / or retrieved.

[0154] 21 illustrates a block diagram of an exemplary model-generated image retrieval system 2110 according to an exemplary embodiment of the present disclosure. In particular, FIG. 21 illustrates a model-generated image retrieval system 2110 that receives a prompt input 2112, generates one or more model-generated images 2116 using an image generation model 2114 based on the prompt input 2112, and performs a search based on one or more of the model-generated images 2116.

[0155] For example, the prompt input 2112 can be obtained from a user computing device. The prompt input 2112 can describe one or more terms and / or one or more images. The prompt input 2112 can be generated based on free-form text input, file upload, and / or based on multiple user interface tip selections. The prompt input 2112 can be processed using a prompt generation block to generate inputs for a particular machine learning model. Alternatively and / or additionally, the prompt input 2112 can be processed by an embedding model to generate text embeddings to be provided to a converter model trained to generate images based on the text embeddings.

[0156] The prompt input 2112 can be processed using an image generation model 2114 to generate one or more model generated images 2116. The one or more model generated images 2116 can be generated based on the prompt input 2112. For example, the one or more model generated images 2116 can show one or more features associated with one or more prompt terms (e.g., feathers on a dress can be shown in response to selection of a "dress" descriptor user interface element and a "feathered" descriptor user interface element).

[0157] The user may then select one or more of the one or more model generated images 2116 to be utilized as an image query. The selected image(s) may be provided to the search engine 18. One or more search results 2120 may then be received from the search engine 2118. The one or more search results 2120 may describe existing data that is similar to the model generated data.

[0158] One or more model generated images 2116 and / or one or more search results 2120 can be provided to the user as output 2122. The user can then store the output(s) 2122, store them in a collection, and / or share them with one or more other users.

[0159] The model generating image retrieval system 2110 can provide an interface for imagining and finding clothing, art, travel locations, videos, music, and / or other objects or content items.

[0160] In some implementations, the systems and methods can include utilizing model-generated data (e.g., one or more model-generated images) to generate an augmented reality rendering asset and / or a virtual reality experience. For example, a generative model (e.g., an image-generated model) can process to generate an augmented reality rendering asset and / or a virtual reality experience. In some implementations, a prompt can be processed by an image-generated model to generate one or more model-generated images, which can then be utilized to generate an augmented reality rendering asset and / or a virtual reality rendering experience. The augmented reality rendering asset can be utilized to render a model-generated object in a user's environment. For example, a user can utilize the augmented reality rendering asset to render a model-generated object in their room and / or on their body. The rendering can be performed on still images and / or a live camera feed. Additionally and / or alternatively, a virtual reality experience can be utilized to view one or more objects in a three-dimensional virtual space.

[0161] 22A illustrates a block diagram of an exemplary computing system 100 for performing machine learning model output generation and retrieval in accordance with an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0162] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0163] The computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0164] In some implementations, the user computing device 102 can store or include one or more dataset generation modules 120. For example, the dataset generation modules 120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks), or other types of machine learning models, including non-linear and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary dataset generation models 120 are described with reference to FIGS. 2 and 4.

[0165] In some implementations, one or more dataset generation models 120 may be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single dataset generation model 120 (e.g., to perform parallel dataset generation across multiple instances of a prompt input).

[0166] More specifically, one or more selections can be received to generate prompt inputs, which can then be processed by the dataset generation model 120 to generate a number of model generation datasets. A selection of a particular model generation dataset can be received. The particular model generation dataset can be input to a search engine to receive one or more search results.

[0167] Additionally and / or alternatively, one or more of the dataset generation models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the dataset generation module 140 may be implemented by the server computing system 140 as part of a web service (e.g., a search service). Thus, one or more of the models 120 may be stored and implemented on the user computing device 102 and / or one or more of the models 140 may be stored and implemented on the server computing system 130.

[0168] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the contact of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0169] The remote computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0170] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. When server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0171] As mentioned above, the server computing system 130 can store or otherwise include one or more machine learning dataset generation models 140. For example, the models 140 can be or otherwise include various machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are described with reference to FIGS. 2 and 4.

[0172] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 through interaction with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0173] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0174] The training computing system 150 can include a model trainer 160 that trains the machine learning models 120 and / or 140 stored on the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation of error. For example, a loss function can be backpropagated through the model(s) to update one or more parameters of the model(s) (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent can be used to iteratively update the parameters over several training iterations.

[0175] In some implementations, performing backpropagation of the error may include performing truncated backpropagation over time. The model trainer 160 may implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0176] In particular, model trainer 160 can train dataset generation model 120 and / or 140 based on a set of training data 162. Training data 162 can include, for example, training dataset examples (e.g., training images), training prompts, training labels, and / or training segmentations.

[0177] In some implementations, if a user provides consent, the training examples can be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 with user-specific data received from the user computing device 102. In some examples, this process may be referred to as personalizing the model.

[0178] Model trainer 160 includes computer logic utilized to provide desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some embodiments, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other embodiments, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or an optical or magnetic medium.

[0179] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 180 may occur over any type of wired or wireless link, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0180] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0181] In some implementations, the input to the machine learning model(s) of the present disclosure may be image data. The machine learning model(s) may process the image data to generate an output. As an example, the machine learning model(s) may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning model(s) may process the image data to generate an image segmentation output. As another example, the machine learning model(s) may process the image data to generate an image classification output. As another example, the machine learning model(s) may process the image data to generate an image data modification output (e.g., alteration of the image data, etc.). As another example, the machine learning model(s) may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model(s) may process the image data to generate an augmented image data output. As another example, the machine learning model(s) may process the image data to generate a prediction output.

[0182] In some implementations, the input to the machine learning model(s) of the present disclosure may be text or natural language data. The machine learning model(s) may process the text or natural language data to generate an output. As an example, the machine learning model(s) may process the natural language data to generate a language encoding output. As another example, the machine learning model(s) may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning model(s) may process the text or natural language data to generate a transformation output. As another example, the machine learning model(s) may process the text or natural language data to generate a classification output. As another example, the machine learning model(s) may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model(s) may process the text or natural language data to generate a semantic output. As another example, the machine learning model(s) can process text or natural language data to generate augmented text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language). As another example, the machine learning model(s) can process text or natural language data to generate predicted outputs.

[0183] In some implementations, the input to the machine learning model(s) of the present disclosure may be statistical data. The machine learning model(s) may process the statistical data to generate an output. As an example, the machine learning model(s) may process the statistical data to generate a recognition output. As another example, the machine learning model(s) may process the statistical data to generate a prediction output. As another example, the machine learning model(s) may process the statistical data to generate a classification output. As another example, the machine learning model(s) may process the statistical data to generate a segmentation output. As another example, the machine learning model(s) may process the statistical data to generate a segmentation output. As another example, the machine learning model(s) may process the statistical data to generate a visualization output. As another example, the machine learning model(s) may process the statistical data to generate a diagnostic output.

[0184] In some implementations, the input to the machine learning model(s) of the present disclosure may be sensor data. The machine learning model(s) may process the sensor data to generate an output. As an example, the machine learning model(s) may process the sensor data to generate a recognition output. As another example, the machine learning model(s) may process the sensor data to generate a prediction output. As another example, the machine learning model(s) may process the sensor data to generate a classification output. As another example, the machine learning model(s) may process the sensor data to generate a segmentation output. As another example, the machine learning model(s) may process the sensor data to generate a segmentation output. As another example, the machine learning model(s) may process the sensor data to generate a visualization output. As another example, the machine learning model(s) may process the sensor data to generate a diagnostic output. As another example, the machine learning model(s) may process the sensor data to generate a detection output.

[0185] In some cases, the machine learning model(s) can be configured to perform a task that includes encoding input data for reliable and / or efficient transmission or storage (and / or corresponding decoding). For example, the task may be an audio compression task; the input may include audio data and the output may include compressed audio data. In another example, the input may include visual data (e.g., one or more images or videos) and the output may include compressed visual data, and the task is a visual data compression task. In another example, the task may include generating an embedding for the input data (e.g., input audio or □ data).

[0186] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data of one or more images and the task is an image processing task. For example, the image processing task may be image classification and the output is a set of scores, each score corresponding to a different object class and representing the likelihood that one or more images show an object belonging to the object class. The image processing task may be object detection and the image processing output identifies one or more regions in one or more images and, for each region, the likelihood that the region shows an object of interest. As another example, the image processing task may be image segmentation and the image processing output defines, for each pixel of one or more images, a respective likelihood for each category of a set of predefined categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation and the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel of the input images, the motion of the scene depicted in that pixel between the images in the network input.

[0187] 22A illustrates one exemplary computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device 102 can include a model trainer 160 and a training dataset 162. In such implementations, the model 120 can be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 that personalizes the model 120 based on user-specific data.

[0188] 22B illustrates a block diagram of an exemplary computing device 40 for performing according to an exemplary embodiment of the present disclosure. The computing device 40 may be a user computing device or a server computing device.

[0189] Computing device 40 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine learning model(s). For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0190] 22B, each application can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0191] 22C illustrates a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0192] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the model(s) stored therein) using an API (e.g., a common API across all applications).

[0193] The central intelligence layer includes several machine learning models. For example, as shown in FIG. 22C, each machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications can share a single machine learning model. For example, in some embodiments, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some embodiments, the central intelligence layer is included within or otherwise implemented by the operating system of the computing device 50.

[0194] The central intelligence layer can communicate with a central device data layer, which may be a centralized repository of data for the computing device 50. As shown in FIG. 22C, the central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0195] The technology described herein refers to servers, databases, software applications, and other computer-based systems, as well as actions performed on and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of configurations, combinations, and divisions of tasks and functions among components. For example, the processes discussed herein may be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components may operate sequentially or in parallel.

[0196] Although the subject matter of the present disclosure has been described in detail with respect to various specific and exemplary embodiments thereof, each example is provided for the purpose of illustration and not for the purpose of limiting the present disclosure. Those skilled in the art, upon understanding the above content, can easily make modifications, variations, and equivalents of such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, changes, and / or additions to the subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment can be used with another embodiment to produce yet another embodiment. Thus, the present disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. 1. A computing system for searching using a data query generated by a machine learning model, comprising: one or more processors; one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations; and the operation comprises: obtaining a prompt input, the prompt input including one or more terms; processing the prompt input with an image generation model to generate one or more model-generated images, the one or more model-generated images being generated at least in part based on the one or more terms; determining one or more search results based on the one or more model generated images, the one or more search results being associated with one or more objects; providing a search result interface, the search result interface presenting the one or more search results for display; Including, the system.

2. obtaining the prompt input, providing a plurality of selectable user interface elements for display in a graphical user interface, the plurality of selectable user interface elements being associated with a plurality of candidate prompt terms; obtaining selection data, the selection data describing a first selectable user interface element and a second selectable user interface element, the first selectable user interface element being associated with a first prompt term and the second selectable user interface element being associated with a second prompt term; Including, the prompt input includes the first prompt term and the second prompt term; The system of claim 1 .

3. the plurality of selectable user interface elements are provided for display in response to obtaining a prompt selection request; the prompt selection request describes an input for receiving the graphical user interface of a selectable user interface chip; The system of claim 2 , wherein the plurality of candidate prompt terms is predetermined.

4. the first prompt term is associated with a type of object; the second prompt term is associated with a particular descriptive feature; The system of claim 2 , wherein the one or more model-generated images describe a particular object of the type of object having the particular descriptive characteristic.

5. determining the one or more search results based on the one or more model generated images; providing the one or more model generated images to a search engine; receiving the one or more search results from the search engine; The system according to any one of claims 1 to 4, comprising:

6. the one or more search results are associated with one or more products; The system of any one of claims 1 to 5, wherein the one or more search results include one or more action links associated with the one or more products.

7. The system of claim 6 , wherein the one or more action links are associated with a purchasing interface for the one or more products.

8. The operation is providing the one or more model generated images for display together with the one or more terms in a graphical user interface; The system of any one of claims 1 to 7, further comprising:

9. 9. The system of claim 1, wherein the prompt input comprises selection data describing one or more selection inputs associated with one or more selectable user interface elements and one or more texture inputs including text input into a text entry box.

10. 1. A computer-implemented method for searching using a data query generated by a machine learning model, comprising: obtaining, by a computing system having one or more processors, a prompt input, the prompt input including one or more terms; processing, by the computing system, the prompt input with a data generation model to generate a plurality of model generation data sets, the plurality of model generation data sets being generated based at least in part on the one or more terms; providing, by the computing system, the plurality of model generation data sets via a user interface; obtaining, by the computing system, a selection input, the selection input describing a selection of a particular model generation dataset among the plurality of model generation datasets; determining, by the computing system, one or more search results based on the particular model generation dataset; providing, by the computing system, the one or more search results as output; and A method comprising:

11. providing, by the computing system, the plurality of model generation datasets via the user interface; providing, by the computing system, a plurality of model generated images in an image carousel; The method of claim 10, comprising:

12. obtaining, by the computing system, the selection input; obtaining, by the computing system, the selection of the particular model generation dataset of the plurality of model generation datasets; obtaining, by the computing system, a cropping input, the cropping input describing a portion of the particular model generation data set; segmenting, by the computing system, the portion of the particular model generation dataset to generate a cropped model generation dataset; The method according to any one of claims 10 to 11, comprising:

13. The method of claim 12 , wherein the one or more search results are determined based on the cropped model generation data set.

14. The method of any one of claims 10 to 13, wherein each of the plurality of model generation data sets is different.

15. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations including: providing an image generation interface for display, the image generation interface including a plurality of categorical user interface elements, each categorical user interface element being associated with a different generated category; obtaining first input data, the first input data being associated with a selection of a particular category user interface element of the plurality of category user interface elements, the particular category user interface element being associated with a particular category; providing a plurality of descriptor user interface elements for display on the image generation interface, each descriptor user interface element being associated with a different descriptor; obtaining second input data, the second input data being associated with a selection of one or more particular descriptor user interface elements of the plurality of descriptor user interface elements, the one or more particular descriptor user interface elements being associated with one or more particular descriptors; processing data associated with the one or more particular descriptors with a machine learning image generation model to generate one or more model generated images; providing the one or more model generated images for display on the image generation interface; and [0023] In one embodiment, the present invention relates to a computer-readable medium comprising:

16. the particular category is associated with clothing; The one or more non-transitory computer-readable media of claim 15 , wherein the one or more particular descriptors are associated with one or more clothing terms that describe an article of clothing.

17. The operation is obtaining third input data, the third input data describing a selection of the one or more model generated images; determining one or more search results based on the one or more model generated images, the one or more search results being associated with one or more objects; providing a search result interface, the search result interface presenting the one or more search results for display; The one or more non-transitory computer readable media of any one of claims 15 to 16, further comprising:

18. The operation is obtaining editing input data, the editing input data describing a request to replace one or more first features of the one or more model generated images with one or more second features; generating one or more updated model generated images based on the edit input data; 18. The one or more non-transitory computer readable media of any one of claims 15 to 17, further comprising:

19. The one or more non-transitory computer-readable media of claim 18 , wherein the editing input data is associated with a color change.

20. The one or more non-transitory computer-readable media of claim 18 , wherein the editing input data is associated with a texture change.

Images