Machine learning model for multi-modal search and retrieval of images

Abstract

Systems and methods of the present disclosure relate to computer-implemented methods for machine learning multi-modal search refinement. The method includes obtaining a query image embedding of a query image and a textual query refinement associated with the query image. The method includes processing the query image embedding and the textual query refinement using a machine learning query refinement model to obtain a refined query image embedding incorporating the textual query refinement. The method includes evaluating a loss function that evaluates a distance between the refined query image embedding and an embedding of a true value image within an image embedding space. The method includes modifying a value of a parameter of the machine learning query refinement model based on the loss function.

Description

Technical Field

[0001] Aspects of the present disclosure relate to searching and retrieving images. More specifically, aspects of the present disclosure relate to machine learning models for allowing images to be retrieved from a database in a faster or more efficient manner. Background Art

[0002] Recently, visual search capabilities have been provided as features across a variety of applications (e.g., virtual assistant applications, camera applications, etc.). Conventionally, when performing a visual search, a user first provides an image to a search service. These search services will typically process the image using machine learning techniques to identify visually or semantically similar images and / or information associated with entities depicted in the image. However, because these conventional models are only trained to process image data, they cannot be refined in conjunction with text queries provided by the user. Summary of the invention

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] An example aspect of the present disclosure relates to a computer-implemented method for machine learning multimodal search refinement. The method includes: obtaining, by a computing system including one or more computing devices, a query image embedding of a query image and a text query refinement associated with the query image. The method includes: processing, by the computing system, the query image embedding and the text query refinement using a machine learning query refinement model to obtain a refined query image embedding combined with the text query refinement. The method includes: evaluating, by the computing system, a loss function that evaluates the distance between the refined query image embedding and an embedding of a true value image within an image embedding space. The method includes: modifying, by the computing system, one or more values ​​of one or more parameters of the machine learning query refinement model based at least in part on the loss function.

[0005] Another example aspect of the present disclosure relates to a computing system for machine learning multimodal search refinement. The computing system includes one or more processors. The computing system includes a machine learning query refinement model that is trained to refine a query image using text query refinement. The computing system includes one or more non-transitory computer-readable media that store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include: obtaining an image embedding of a query image provided by a user of a visual search application. The operations include: obtaining a text query refinement of the query image from the user of the visual search application, wherein the text query refinement is responsive to providing one or more initial result images of the query image to the user of the visual search application. The operations include: processing the image embedding and the text query refinement of the query image using the machine learning query refinement model to obtain a refined image embedding combined with the text query refinement. The operations include: determining one or more refined result images based at least in part on the refined image embedding combined with the text query refinement.

[0006] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause a computing system to perform operations. The operations include: obtaining an image embedding of a query image provided by a user of a visual search application. The operations include: obtaining a textual query refinement of the query image from the user of the visual search application, wherein the textual query refinement is responsive to providing one or more initial result images of the query image to the user of the visual search application. The operations include: processing the image embedding and the textual query refinement of the query image using a machine learning query refinement model to obtain a refined image embedding combined with the textual query refinement. The operations include: determining one or more refined result images based at least in part on the refined image embedding combined with the textual query refinement.

[0007] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0008] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0010] Figure 1AA block diagram of an example computing system that performs machine learning query refinement according to an example embodiment of the present disclosure is depicted.

[0011] Figure 1B A block diagram of an example computing device performing machine learning query refinement is depicted according to an example embodiment of the present disclosure.

[0012] Figure 1C A block diagram of an example computing device that performs training of a machine learning query refinement model is depicted according to an example embodiment of the present disclosure.

[0013] Figure 2 Depicted is a data flow diagram for training a machine learning query refinement model to refine a visual search query in accordance with some embodiments of the present disclosure.

[0014] Figure 3 Depicted is a data flow diagram for utilizing a machine learning query refinement model to refine a visual search query in accordance with some embodiments of the present disclosure.

[0015] Figure 4 Depicted is a data flow diagram for processing input to a machine learning query refinement model to refine a visual search query in accordance with some embodiments of the present disclosure.

[0016] Figure 5 Depicted is a communication flow diagram for providing a visual search service with query refinement according to some embodiments of the present disclosure.

[0017] Figure 6 Depicted is a flow diagram of an example method for performing training of a machine learning query refinement model according to an example embodiment of the present disclosure.

[0018] Figure 7 Depicted is a flow diagram of an example method for performing query refinement using a machine learning query refinement model according to an example embodiment of the present disclosure.

[0019] Reference numerals repeated in various figures are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0020] Overview

[0021] Aspects of the present disclosure relate to the technical task of searching and retrieving images using multimodal search. More specifically, the present disclosure relates to a machine learning model for textually refining an image query to form a multimodal query. As an example, a query image embedding of a query image may be obtained using a textual query refinement associated with a query image. For example, the query image may depict a specific person, and the textual query refinement may describe visual characteristics associated with the specific person that are different from the query image or that are not present in the query image (e.g., clothing items or facial features, such as beards). The query image embedding and the textual query refinement may be processed with a machine learning query refinement model (e.g., a transformer model, etc.) to obtain a refined query image embedding combined with the textual query refinement. To follow the example described previously, the refined query image embedding may be an image embedding of a specific (or at least visually similar) person having the characteristics described in the textual query refinement. A loss function may be evaluated that evaluates the distance between the refined query image embedding and the embedding of a true value image within the image embedding space. One or more values ​​of one or more parameters of the machine learning query refinement model may be modified based at least in part on the loss function. In this way, a machine learning query refinement model can be trained to refine image embeddings of an initial query image such that the refined image embeddings are combined with textual query refinement, thereby achieving a fast and efficient mechanism for retrieving images and / or other information associated with these images from a database.

[0022] Embodiments of the present disclosure provide a variety of technical effects and benefits. As an example technical effect and benefit, the technical task of searching and retrieving similar images can be performed in a faster and / or more efficient manner. For example, due to the incorrect determination of user intent by the visual search application, users of conventional visual search applications often need to recapture images of the query target. In turn, this may lead to a disappointing user experience and unnecessary use of resources to recapture the query target (e.g., power, computing cycles, memory, storage devices, bandwidth, etc.). However, embodiments of the present disclosure provide a machine learning query refinement model that can be utilized by a visual search application to provide text refinement features, so that users can quickly and efficiently utilize text data to refine their visual searches, thereby significantly improving the efficiency of the search by eliminating unnecessary resource usage associated with recapture of the query target. In addition, as will be appreciated, embodiments of the present disclosure can facilitate visual search-based retrieval of images that may not be easily retrieved using only the query image.

[0023] Referring now to the accompanying drawings, example embodiments of the present disclosure will be discussed in greater detail.

[0024] Example Apparatus and Systems

[0025] Figure 1A A block diagram of an example computing system 100 that performs machine learning query refinement according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 that are communicatively coupled via a network 180.

[0026] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0027] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing device 102 to perform operations.

[0028] In some implementations, the user computing device 102 may store or include one or more machine learning query refinement models 120. For example, the machine learning query refinement model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including non-linear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). Reference Figures 2 to 5 An example machine learning query refinement model 120 is discussed.

[0029] In some implementations, the one or more machine-learned query refinement models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 may implement multiple parallel instances of a single machine-learned query refinement model 120 (e.g., to perform parallel query refinement across multiple instances of the machine-learned query refinement model 120).

[0030] More specifically, the machine learning query refinement model 120 can be trained and utilized to refine query images provided for visual image search through text query refinement. For example, the machine learning query refinement model 120 can process a query image or a representation of a query image (e.g., an embedding of a query image) simultaneously with a text query refinement or a representation of the query refinement (e.g., a term embedding for text query refinement). Then, the machine learning query refinement model 120 can generate a refined query image embedding combined with the text query refinement. For example, if the query image depicts a specific person and the text query refinement describes the clothing item "hat", the refined image embedding can correspond to an image of a person (or a visually similar person) wearing a hat. The refined query image embedding can be used to retrieve images associated with image embeddings within a certain distance of the refined image embedding within an image embedding space (e.g., an embedding space for an image search service, etc.). In this way, the machine learning query refinement model 120 can be utilized to provide text refinement capabilities for visual search applications.

[0031] Additionally or alternatively, one or more machine-learned query refinement models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned query refinement models 140 may be implemented by the server computing system 140 as part of a web service (e.g., a query refinement service, a visual search service, etc.). Thus, one or more models 120 may be stored and implemented at the user computing device 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0032] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0033] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0034] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0035] As described above, the server computing system 130 may store or otherwise include one or more machine learning query refinement models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include a multi-head self-attention model (e.g., a transformer model). Reference Figures 2 to 5 Discuss example model 140 .

[0036] User computing device 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with training computing system 150 communicatively coupled via network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0037] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0038] The training computing system 150 may include a model trainer 160 that trains the machine learning models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques such as, for example, error back propagation. For example, a loss function may be back propagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques may be used to iteratively update the parameters in multiple training iterations.

[0039] In some implementations, performing error back-propagation may include performing truncated back-propagation through time.The model trainer 160 may perform a variety of generalization techniques (eg, weight decay, dropout, etc.) to improve the generalization capabilities of the model being trained.

[0040] Specifically, model trainer 160 can train machine learning query refinement models 120 and / or 140 based on a set of training data 162. Training data 162 can include various pairs of image data and associated textual query refinements. For example, in some embodiments, training data 162 can include a corpus of image search data. The corpus of image search data can describe interactions with search results from users, and can include search result images provided to users in response to queries, and refined search result images provided to users in response to selection of query refinement elements provided to users with search result images. The query image, textual query refinement, and true value image can be selected from the search result image, optional query refinement elements, and refined search result image.

[0041] For example, a corpus of image search data may include multiple search result images provided to a user by an image search service in response to a text query (e.g., a text query for "cars"). The corpus of image search data may indicate the search result images that users interact with most frequently (e.g., images depicting black cars, etc.). The corpus of image search data may also include refined search result images provided to the user in response to selection of a query refinement element provided to the user along with the search result image. For example, the image search service may provide multiple query refinement user interface elements while providing the search result image, which may be selected to refine the text query provided by the user (e.g., elements such as "truck", "red", "blue", "fast", "van", etc. for a text query for "cars", etc.). When a query refinement element is selected by the user, a refined search result image may be provided to the user making the selection in response to the text query and the selected query refinement element. For example, a user may provide an initial text query "cars" and then select a query refinement element "blue". The refined search result images may each depict a blue car.

[0042] The corpus of image search data may indicate the refined search result images that users interact with most frequently after selecting an associated query refinement element. Query images, textual query refinements, and ground truth images may be selected from the search result images, optional query refinement elements, and refined search result images to include in training data 162.

[0043] For example, multiple users may each provide the text query "car" to an image search application, and in response, the image search application may provide multiple search result images to the users. The corpus of image search data may indicate a first search result image of the multiple search result images as the one that has the most user interaction. Next, multiple users may each select the same query refinement element "blue". The image search application may provide multiple refined search result images depicting a blue car, and the corpus of image search data may indicate a first refined search result image of the multiple refined search result images as the one that has the most user interaction. The first search result image may be selected as the query image, the text content of the query refinement element (e.g., "blue") may be selected as the text query refinement, and the first refined search result image may be selected as the ground truth image. The query image, the text query refinement, and the ground truth image may be collectively included in the training data 162 as training examples for training the machine learning query refinement model 120 / 140 by the model trainer 160. In this manner, a corpus of image search data may be utilized to generate multiple training examples to include in the training data 162 for training the machine learning query refinement model 120.

[0044] In some implementations, if the user has provided consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 based on user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0045] The model trainer 160 includes computer logic for providing the desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software that controls a general purpose processor. For example, in some implementations, the model trainer 160 includes a program file stored on a storage device, loaded into a memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more computer executable instruction sets stored in a tangible computer readable storage medium such as RAM, a hard disk, or an optical or magnetic medium.

[0046] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be performed via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0047] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of image data, potential embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded representation and / or compressed representation of image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0048] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an upgraded (upscale) text or natural language output (e.g., text or natural language data with a quality higher than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0049] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data with a higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0050] In some implementations, the input to the machine learning model of the present disclosure may be latent coded data (e.g., a latent space representation of an input, an image embedding of an image, an embedding of text content (e.g., a word embedding), etc.). The machine learning model may process the latent coded data to generate an output. As an example, the machine learning model may process the latent coded data to generate a recognition output. As another example, the machine learning model may process the latent coded data to generate a reconstruction output. As another example, the machine learning model may process the latent coded data to generate a search output. As another example, the machine learning model may process the latent coded data to generate a re-clustering output. As another example, the machine learning model may process the latent coded data to generate a prediction output.

[0051] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that the one or more images depict an object belonging to an object class. The image processing task may be object detection, where the image processing output identifies one or more regions in the one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines the corresponding possibility of each category in a set of predetermined categories for each pixel in the one or more images. For example, a set of categories may be foreground and background. As another example, a set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in the one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines the motion of the scene depicted at the pixel between the images in the network input for each pixel of one of the input images.

[0052] Figure 1A An example computing system that can be used to implement the present disclosure is shown. Other computing systems may also be used. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training data set 162. In such implementations, the model 120 may be both trained and used locally at the user computing device 102. In some of such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0053] Figure 1BDepicted is a block diagram of an example computing device 10 that performs machine learning query refinement according to an example embodiment of the present disclosure. Computing device 10 may be a user computing device or a server computing device.

[0054] Computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and machine learning model. For example, each application can include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0055] like Figure 1B As shown, each application can communicate with multiple other components of the computing device (e.g., such as one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0056] Figure 1C A block diagram of an example computing device 50 performing training of a machine learning query refinement model according to an example embodiment of the present disclosure is depicted. The computing device 50 may be a user computing device or a server computing device.

[0057] The computing device 50 includes a plurality of applications (e.g., Application 1 to Application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0058] The central intelligence layer includes multiple machine learning models. For example, Figure 1C As shown, a corresponding machine learning model may be provided for each application, and the corresponding machine learning model may be managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model for all applications. In some implementations, the central intelligence layer is included in the operating system of the computing device 50 or is otherwise implemented by the operating system.

[0059] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for the computing device 50. Figure 1CAs shown, the central device data layer can communicate with multiple other components of the computing device (e.g., such as one or more sensors, context managers, device state components, and / or additional components). In some implementations, the central device data layer can use an API (e.g., a private API) to communicate with each device component.

[0060] Example Model Layout

[0061] Figure 2 A data flow diagram for training a machine learning query refinement model to refine a visual search query according to some embodiments of the present disclosure is depicted. Specifically, a query image embedding 202 of a query image and a textual query refinement 204 associated with the query image may be obtained for processing by a machine learning query refinement model 206. In some embodiments, the query image embedding 202 may be an image embedding of a visual query image depicting one or more entities (e.g., objects, scenes, people, etc.). Each of the one or more entities may have one or more characteristics. For example, an entity may be an object such as a car. The characteristics may be a brand, model, color, positioning, posture (e.g., the interior or exterior of a car), a condition of a vehicle, etc. Specifically, an entity depicted in a query image may have a first characteristic. For example, an entity may be a pair of pants, and the first characteristic may be blue (e.g., the query image depicts a pair of blue pants). The textual query refinement may describe a second characteristic (e.g., red) that is different from the second characteristic. The true value image may depict an entity (e.g., the pants) having the second characteristic (e.g., pants having red color).

[0062] In some implementations, the query image embedding 202 may be an image embedding of a portion of a visual query image. For example, a user may provide input selecting a portion of a query image. The query image embedding 202 may be generated for the portion of the query image selected by the user. For another example, a computing system generating the query image embedding 202 may automatically select the portion of the query image for which to generate the query image embedding 202 based on the content of the selected portion of the image and the unselected portion of the image. For example, the computing system may determine that the unselected portion of the image does not include any objects of interest, while the selected portion of the image includes multiple objects of interest.

[0063] In some embodiments, the text query refinement may describe one of the characteristics of the entity. For example, the entity depicted by the query image may be clothing. The characteristics of the clothing described by the text query refinement may be the color, brand, size, etc. of the clothing.

[0064] It should be noted that although the query image embedding 202 is an image embedding within an embedding space, the query image embedding 202 may refer to any encoding or latent representation of the query image 202 .

[0065] In some embodiments, the query image embedding 202 and the textual query refinement 204 may be obtained from a corpus of image search data. The corpus of image search data may describe interactions with search results from a user and may include search result images provided to the user in response to a query, and refined search result images provided to the user in response to selection of a query refinement element provided to the user with the search result image. The query image for the query image embedding 202, the textual query refinement 204, and the ground truth image for the ground truth image embedding 210 may be selected from the search result image, the optional query refinement element, and the refined search result image.

[0066] For example, multiple users may each provide a text query "car" to an image search application, and in response, the image search application may provide multiple search result images to the users. The corpus of image search data may indicate a first search result image of the multiple search result images as having the most interaction with the user. Next, multiple users may each select the same query refinement element "blue". The image search application may provide multiple refined search result images depicting a blue car, and the corpus of image search data may indicate a first refined search result image of the multiple refined search result images as having the most interaction with the multiple users. The first search result image may be selected as a query image for query image embedding 202, the text content of the query refinement element (e.g., "blue") may be selected as a text query refinement 204, and the first refined search result image may be selected as a true value image for true value image embedding 210.

[0067] The machine learning query refinement model 206 may process the text query refinement 204 and the query image embedding 202 to obtain a refined image embedding 208. The refined image embedding 208 is an image embedding combined with the text query refinement 204. For example, the query image embedding 202 may depict a blue dress. The text query refinement 204 may include the word "red". The refined image embedding 208 may correspond to an image embedding for an image of a red dress. In other words, the refined image embedding may be an alternative representation of an image of a red dress. As will be understood from this disclosure, the refined image embedding may be used to retrieve one or more images similar to the image represented by the refined image embedding based on low-level features of the images in the image corpus.

[0068] The loss function evaluator 212 may evaluate a loss function that evaluates the difference between the refined image embedding 208 and the true value image embedding 210. Specifically, the loss function evaluates the distance between the refined image embedding 208 and the true value image embedding 210 within the image embedding space 214.

[0069] Based at least in part on the loss function, modification determiner 216 may modify one or more values ​​of one or more parameters of machine learning query refinement model 206 via parameter value modification 218 .

[0070] Figure 3 A data flow diagram 300 for refining a visual search query using a machine learning query refinement model according to some embodiments of the present disclosure is depicted. Specifically, an image embedding can be obtained for a query image provided by a user of a visual search application. For example, the visual search application can be an application function provided by an operating system. For another example, the visual search application can be an application that is communicatively coupled to a second application accessed by a user. For yet another example, the visual search application can include a machine learning query refinement model 306, or can communicate with a service that provides a machine learning query refinement model 306.

[0071] A text query refinement 304 for a query image may be obtained from a user of the visual search application. The text query refinement may be responsive to providing one or more initial result images for the query image to the user of the visual search application. The text query refinement 304 and the image embedding 302 may be processed using a machine learning query refinement model 306 to obtain a refined image embedding 308. The refined image embedding may be combined with the text query refinement 304, such as with respect to Figure 2 The refined image embedding 208 is discussed.

[0072] The result image determiner 310 may determine one or more refined result images 314 based at least in part on the refined image embeddings 308. In some embodiments, determining the one or more refined result images includes determining one or more image embeddings within the image embedding space 312 that are within a threshold distance of the refined image embedding 308, and selecting the one or more refined result images 314 that respectively correspond to the one or more image embeddings within the threshold distance.

[0073] Figure 4 A data flow diagram 400 for processing input to a machine learning query refinement model to refine a visual search query according to some embodiments of the present disclosure is depicted. Specifically, a query image embedding 406 (e.g., Figure 2 The query image embedding 202, Figure 3In some embodiments, a machine learning model such as a machine learning image encoding model 404 can process the query image 402 to generate a query image embedding 406. For example, the machine learning image encoding model can be a model trained to generate image embeddings for an image embedding space. The image embedding space can be an image embedding space for visual search applications. The query image embedding can be processed using the machine learning query refinement model 206, such as with respect to Figure 2 as described.

[0074] In some embodiments, the text query refinement 408 may be processed using a machine learning model such as a machine learning text encoding model 410 to obtain a latent representation or encoding of the text query refinement 408. In some embodiments, the machine learning text encoding model 410 may be trained to process the text query refinement to generate a plurality of word-meta embeddings 412. Alternatively, in some embodiments, the machine learning text encoding model 410 may process the text query refinement 408 to generate some other type of latent representation 410 of the text query refinement.

[0075] In some embodiments, the machine learning image encoding model may be a sub-model or part of the machine learning query refinement model 206. Figure 2 When processing query image 402 and textual query refinement 408 as described, portions of machine learning query refinement model 206 including machine learning image encoding model 404 can process query image 402 to obtain query image embedding 406, which can then be processed as described with respect to Figure 3 The query image embedding 406 and the word-gram embedding 412 are processed as described to obtain a refined image embedding.

[0076] Similarly, in some embodiments, the machine learning text encoding model can be a sub-model or part of the machine learning query refinement model 206. Figure 2 When processing the query image 402 and the text query refinement 408 as described, the portion of the machine learning query refinement model 206 including the machine learning text encoding model 410 can process the text query refinement 408 to obtain a word element embedding 412, and then can process the query image embedding 406 and the word element embedding 412 to obtain a refined image embedding.

[0077] Figure 5A communication flow diagram for providing a visual search service with query refinement according to some embodiments of the present disclosure is depicted. Specifically, at step 504, a computing system 500 (e.g., computing system 100 of FIG. 1 , etc.) may receive a query image from a user of a user device 502 (e.g., user device 102 of FIG. 1 , etc.). It should be noted that communication entities 500 and 502 are only depicted to more easily illustrate information flow, and they may represent any other type of computing device or system. For example, computing system 500 and user device 502 may be user device 102 of FIG. 1 , and Figure 5 The information flows depicted in the figure may be or otherwise include inter-process communications on the same device. Alternatively, the computing system 100 may be the server computing system 130 of FIG. 1 and may provide services that facilitate the visual search application executed on the user device 102 of FIG. 1. Therefore, it should be broadly understood that embodiments of the present disclosure may be implemented across any configuration of computing devices of the present disclosure.

[0078] Once a query image is received, at step 506, computing system 500 may determine an image embedding corresponding to the query image, as previously described with respect to Figure 4 At step 508, an initial result image may be determined using the image embedding. For example, the computing system 500 may determine the image embedding of the initial result image that is closest to the image embedding of the query image in the image embedding space.

[0079] At step 510, computing system 500 may provide an initial result image to user device 502. In some embodiments, computing system may provide the initial result image within an interface of a visual search application (eg, executed by user device 502, etc.).

[0080] At step 512, in response to providing the initial result image, the computing system 500 may obtain a text query refinement in response to providing the initial result image at step 510. For example, the initial result image may be provided to a user of the visual search application within an interface of the visual search application. The visual search application may provide an indication to the user of the application to provide a text query refinement via a text input field presented within the interface. The user may provide a text query refinement via the text input field, and the text query refinement may be provided to the computing system 500.

[0081] At step 514, the computing system may utilize the machine learning query refinement model to process the image embedding of the query image and the text query refinement to obtain a refined image embedding combined with the text query refinement, as previously described with respect to Figure 3 and Figure 4 as described.

[0082] In some embodiments, the computing system may also process information associated with the image corresponding to the image embedding. For example, the image corresponding to the image embedding may be hosted on a website including a description of the image. The information associated with the image may include the description, or may be generated in other ways based on the description. For another example, the image corresponding to the image embedding may be processed using a semantic image processing model that is operable to generate a semantic output describing the image. The information may include the semantic output. For yet another example, the image corresponding to the image embedding may be hosted on a website or application that enables users to post text content (e.g., tags, comments, etc.) associated with the image. The information may include or otherwise describe the text content posted by the user. The machine learning query refinement model may process the information of the query image, the image embedding, and the text query refinement to obtain a refined image embedding.

[0083] At step 516, computing system 500 may determine a refined result image based on the refined image embedding, as previously described with respect to Figure 3 At step 518, the computing system 500 may provide the refined result image to the user device 502. In some embodiments, the refined result image may be provided to the user device 502 for display within an interface of a visual search application.

[0084] In some embodiments, at step 520, the computing system 500 may obtain a second text query refinement in response to providing the refined result image. For example, the refined result image may be provided to a user of the visual search application within an interface of the visual search application. The visual search application may provide an indication to the user of the application to provide the second text query refinement via a text input field presented within the interface. The user may provide the second text query refinement via the text input field, and the second text query refinement may be provided to the computing system 500.

[0085] In some embodiments, at step 522, computing system 500 may utilize a machine learning query refinement model to process the second text query refinement and the image embedding of the query image to obtain a second refined image embedding combined with the second text query refinement. Alternatively, in some embodiments, at step 22, computing system 500 may utilize a machine learning query refinement model to process the refined image embedding and the second text query refinement to obtain a second refined image embedding combined with the text query refinement and the second text query refinement.

[0086] Example Method

[0087] Figure 6 A flow chart of an example method 600 for performing training of a machine learning query refinement model is depicted in accordance with an example embodiment of the present disclosure. Figure 6 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. The various steps of method 600 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0088] At 602, the computing system may obtain a query image embedding and a textual query refinement. Specifically, the computing system may obtain a query image embedding for a query image and a textual query refinement associated with the query image. In some embodiments, obtaining the query image embedding includes determining a textual embedding for textual query refinement, and processing the query image embedding and the textual query refinement using a machine learning query refinement model includes processing the query image embedding and the textual embedding for textual query refinement using a machine learning query refinement model to obtain a refined query image embedding combined with the textual query refinement.

[0089] At 604, the computing system may process the query image embedding and the text query refinement. Specifically, the computing system may utilize a machine learning query refinement model to process the query image embedding and the text query refinement to obtain a refined query image embedding combined with the text query refinement.

[0090] At 606, the computing system may evaluate a loss function. Specifically, the computing system may evaluate a loss function that evaluates a distance between a refined query image embedding and an embedding of a ground-truth image within an image embedding space. In some embodiments, prior to evaluating the loss function, the computing system obtains a corpus of image search data, the corpus of image search data comprising search result images provided to a user in response to a query, and refined search result images provided to a user in response to a selection of a query refinement element provided to the user with the search result image. In some embodiments, the computing system selects a query image, a textual query refinement, and a ground-truth image from the search result image, the query refinement element, and the refined search result image.

[0091] At 608, the computing system may modify one or more values ​​of one or more parameters of the machine learning query refinement model based at least in part on the loss function.

[0092] In some embodiments, a computing system obtains a user query image and a textual query refinement for the user query image from a user. The textual query refinement is responsive to the user query image to provide one or more initial result images to the user. The computing system processes the user query image and the textual query refinement for the user query image using a machine learning query refinement model to obtain an image embedding of the user query image combined with a refinement of the textual query refinement.

[0093] In some embodiments, the computing system may obtain one or more refined result images in response to the refined image embedding of the user query image. In some embodiments, to obtain the refined result images, the computing system may select one or more image embeddings within the image embedding space that are within a threshold distance of the refined image embedding of the user query image. The one or more image embeddings may be associated with the one or more refined result images, respectively. In some embodiments, the computing system may provide the one or more refined result images. In some embodiments, to provide the refined result images, the computing system provides the one or more refined result images for display within an interface of a search application of a user device of the user.

[0094] In some embodiments, the computing system may receive data indicating a user selection of at least one refined result image of the one or more result images. In some embodiments, the computing system may modify one or more values ​​of the one or more parameters of the machine learning query refinement model based at least in part on the at least one refined result image.

[0095] Figure 7 A flow chart depicts an example method 700 for performing query refinement using a machine learning query refinement model according to an example embodiment of the present disclosure. Figure 7 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. The various steps of method 700 may be omitted, rearranged in various ways, combined and / or adjusted without departing from the scope of the present disclosure.

[0096] At 702, the computing system may obtain an image embedding for a query image provided by a user of a visual search application. In some embodiments, to obtain the image embedding, the computing system obtains a query image from the user of the visual search application and determines an image embedding based at least in part on the query image, wherein the image embedding represents the query image.

[0097] At 704, the computing system may obtain a textual query refinement of the query image from a user of the visual search application. The textual query refinement is responsive to providing one or more initial result images of the query image to the user of the visual search application. In some embodiments, the computing system determines one or more word-meta embeddings representing the textual query refinement.

[0098] At 706 , the computing system may utilize a machine learning query refinement model to process the image embedding of the query image and the textual query refinement to obtain a refined image embedding combined with the textual query refinement.

[0099] At 708, the computing system may determine one or more refined result images based at least in part on the refined image embeddings in combination with the text query refinement. In some embodiments, to determine the one or more refined result images, the computing system determines one or more image embeddings within the image embedding space that are within a threshold distance of the refined image embedding of the query image, and selects the one or more refined result images that respectively correspond to the one or more image embeddings.

[0100] Additional public content

[0101] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for multiple possible configurations, combinations, and partitioning of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0102] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation rather than limitation of the present disclosure. Those skilled in the art may easily produce changes, modifications, and equivalents to such embodiments after understanding the foregoing. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily apparent to those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

Claims

1. A computing system for machine learning multimodal search of images, comprising: one or more processors; a machine learning query refinement model trained to refine image queries using text query refinement; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining an image embedding of a query image provided by a user of a visual search application; obtaining, from the user of the visual search application, a textual query refinement of the query image, wherein the textual query refinement is responsive to providing one or more initial result images for the query image to the user of the visual search application; processing the image embedding of the query image and the textual query refinement using the machine learning query refinement model to obtain a refined image embedding combined with the textual query refinement; and One or more refined result images are determined based at least in part on the refined image embedding in combination with the textual query refinement.

2. The computing system of claim 1 , wherein determining the one or more refined result images comprises: determining one or more image embeddings within an image embedding space that are within a threshold distance of the refined image embedding of the query image; as well as The one or more refined result images corresponding to the one or more image embeddings, respectively, are selected.

3. The computing system of any one of claims 1 to 2, wherein obtaining the image embedding of the query image comprises: obtaining the query image from the user of the visual search application; as well as The image embedding is determined based at least in part on the query image, wherein the image embedding represents the query image.

4. The computing system of any one of claims 1 to 3, wherein obtaining the textual query refinement of the query image further comprises: determining one or more word-meta embeddings representing a refinement of the text query; and Wherein processing the image embedding and the text query refinement comprises: utilizing the machine learning query refinement model to process the image embedding and the one or more word element embeddings to obtain the refined image embedding of the query image combined with the text query refinement.

5. The computing system of any one of claims 1 to 4, wherein the machine learning query refinement model comprises a transformer model.

6. The computing system of any one of claims 1 to 5, wherein the operations further comprise: The one or more refined result images are provided to a user device for display within an interface of the visual search application.

7. The computing system of claim 6, wherein the operations further comprise: A second textual query refinement of the query image is obtained in response to providing the one or more refined result images.

8. The computing system of claim 7, wherein the operations further comprise: The second textual query refinement and the image embedding of the query image are processed using the machine learning query refinement model to obtain a second refined image embedding combined with the second textual query refinement.

9. The computing system of claim 7, wherein the operations further comprise: The refined image embedding and the second text query refinement are processed using the machine learning query refinement model to obtain a second refined image embedding that combines the text query refinement and the second text query refinement.

10. A computer-implemented method comprising: obtaining, by a computing system including one or more computing devices, a query image embedding of a query image and a textual query refinement associated with the query image; Processing the query image embedding and the textual query refinement by the computing system using a machine learning query refinement model to obtain a refined query image embedding combined with the textual query refinement; evaluating, by the computing system, a loss function that evaluates a distance between the refined query image embedding and an embedding of a ground-truth image within an image embedding space; as well as One or more values ​​of one or more parameters of the machine learning query refinement model are modified by the computing system based at least in part on the loss function.

11. The computer-implemented method of claim 10, wherein: The query image depicts an entity having a first characteristic; The text query refines a second characteristic describing the entity that is different from the first characteristic; and The true-value image depicts an entity having the second characteristic.

12. The computer-implemented method of any one of claims 10 to 11, wherein obtaining the query image embedding and the text query refinement further comprises: determining, by the computing system, a text embedding for refinement of the text query; and Wherein using the machine learning query refinement model to process the query image embedding and the text query refinement includes: the computing system uses the machine learning query refinement model to process the query image embedding and the text embedding used for the text query refinement to obtain a refined query image embedding combined with the text query refinement.

13. The computer-implemented method of claim 12, wherein the text embedding used for the text query refinement comprises a plurality of word-gram embeddings.

14. The computer-implemented method of any one of claims 10 to 13, wherein: Prior to evaluating the loss function, the method includes: obtaining, by the computing system, a corpus of image search data, the corpus of image search data comprising search result images provided to a user in response to a query and refined search result images provided to the user in response to selection of a query refinement element provided to the user with the search result images; and The query image, the textual query refinement, and the ground truth image are selected by the computing system from among the search result image, the query refinement element, and the refined search result image.

15. The computer-implemented method of any one of claims 10 to 14, wherein the machine learning query refinement model comprises a transformer model.

16. The computer-implemented method of any one of claims 10 to 15, wherein the method further comprises: obtaining, by the computing system, from a user a user query image and a textual query refinement of the user query image, wherein the textual query refinement is responsive to providing one or more initial result images to the user in response to the user query image; as well as The computing system processes the user query image and the text query refinement of the user query image using the machine learning query refinement model to obtain a refined image embedding of the user query image combined with the text query refinement.

17. The computer-implemented method of claim 16, wherein the method further comprises: obtaining, by the computing system, one or more refined result images in response to the refined image embedding of the user query image; as well as The one or more refined result images are provided by the computing system.

18. The computer-implemented method of claim 17, wherein providing the one or more refined result images comprises: The one or more refined result images are provided by the computing system for display within an interface of a search application of a user device of the user.

19. The computer-implemented method of claim 17, wherein obtaining the one or more refined result images comprises: One or more image embeddings within the image embedding space that are within a threshold distance of the refined image embedding of the user query image are selected by the computing system, wherein the one or more image embeddings are respectively associated with the one or more refined result images.

20. The computer-implemented method of claim 18, wherein the method further comprises: receiving, by the computing system, data indicative of a selection by the user of at least one of the one or more refined result images; as well as One or more values ​​of the one or more parameters of the machine learning query refinement model are modified by the computing system based at least in part on the at least one refined result image.

21. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations comprising: obtaining an image embedding of a query image provided by a user of a visual search application; obtaining, from the user of the visual search application, a textual query refinement of the query image, wherein the textual query refinement is responsive to providing one or more initial result images for the query image to the user of the visual search application; processing the image embedding of the query image and the textual query refinement using a machine learning query refinement model to obtain a refined image embedding combined with the textual query refinement; as well as One or more refined result images are determined based at least in part on the refined image embedding in combination with the textual query refinement.

Images