Data-driven automated visual consistency estimation for complementary item discovery in e-commerce
By identifying the frequent occurrence of the relationship and visual similarity of items in the image, we recommend complementary items that are consistent with the style of the user querying the item, solving the problem of unreliable recommendation of complementary items in the prior art, and improving the accuracy and user experience of recommendations.
Patent Information
- Application Number
- CN202410254229.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-03-06
- Publication Date
- 2024-11-26
AI Technical Summary
When recommending complementary items, the methods based on item titles and descriptions of existing e-commerce systems are unreliable and may be inconsistent with the user's query of items, resulting in the recommended items not meeting the user's aesthetic and style needs.
By identifying the frequent occurrence relationships of items in the image, a machine learning model is used to determine the complementary category relationship, and a complementary item that is consistent with the user's query for item style is recommended based on visual similarity.
Improve the accuracy and consistency of complementary items recommendations, ensuring that the recommended items are not only relevant, but also aesthetically match the user's query items, improving user experience and purchasing participation.
Smart Images

Figure CN119027209A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates generally to database systems and data processing and, more particularly, to data-driven automated visual consistency estimation for complementary item discovery in e-commerce. Background Art
[0002] Online marketplaces can use a variety of techniques to recommend items or products to users. Recommended products can include products that the user has explicitly queried or requested, and / or products that are "complementary" (e.g., related) to the queried product. For example, if a user submits a query for "sofa," the online marketplace can present the user with a product list of other sofas, as well as other complementary items related to sofas that the user may be interested in, such as coffee tables, pillows, etc. Some systems can be configured to identify complementary items based on the title of the item, item description, etc. However, such titles and item descriptions are typically entered by individual sellers, and therefore may vary widely between users, and may be vague or misleading. Summary of the invention
[0003] A method is described. The method may include: using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories in an online marketplace; using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency with which the items associated with the first product category and the second product category are depicted together in the set of multiple images; receiving a query image depicting a query item associated with the first product category in the set of multiple product categories from a client device; identifying the first product category as complementary to the second product category in the set of multiple product categories based on the complementary category relationship; identifying a first group of items associated with the first product category, the first group of items being depicted within the set of multiple images and being visually similar to the query item; identifying a second group of items associated with the second product category, based on the second product category being complementary to the first product category, the second group of items being depicted together with the first group of items in the set of multiple images; and sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that can be purchased from the online marketplace, wherein the one or more complementary items include one or more of the second group of items or one or more items that are visually similar to the second group of items.
[0004] A device is described. The device may include: at least one processor; at least one memory coupled to the at least one processor; and instructions stored in the at least one memory. The instructions may be executed by the at least one processor to cause the device to perform the following operations: using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories in an online marketplace; using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency with which the items associated with the first product category and the second product category are depicted together within the set of multiple images; receiving a query image depicting a query item associated with a first product category in a set of multiple product categories from a client device; identifying the first product category and the set of multiple product categories based on the complementary category relationship; The method comprises the steps of: identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query items; identifying a second group of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within a set of multiple images; and sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from an online marketplace, wherein the one or more complementary items include one or more of the second group of items or one or more items that are visually similar to the second group of items.
[0005] Another apparatus is described. The apparatus may include: means for using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories in an online marketplace; means for using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency with which the items associated with the first product category and the second product category are depicted together within the set of multiple images; means for receiving from a client device a query image depicting a query item associated with a first product category in the set of multiple product categories; means for identifying the first product category and the second product category in the set of multiple product categories based on the complementary category relationship. means for identifying a first group of items associated with a first product category that is complementary to the first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query items; means for identifying a second group of items associated with a second product category based on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within a set of multiple images; and means for sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from an online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
[0006] A non-transitory computer-readable medium storing code is described. The code may include instructions executable by a processor to: identify, using one or more processors, a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category from a set of multiple product categories of an online marketplace; determine, using one or more processors, a complementary category relationship between the first product category and the second product category based on a relative frequency with which items associated with the first product category and the second product category are depicted together within the set of multiple images; receive from a client device a query image depicting a query item associated with a first product category from a set of multiple product categories; identify, based on the complementary category relationship, the first product category and the set of multiple product categories; The method of claim 10 wherein the first product category is complementary to the second product category in the online marketplace; identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query items; identifying a second group of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within the set of multiple images; and sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
[0007] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for determining a set of multiple appearances indicating a frequency with which items associated with each of the set of multiple product categories may be depicted within a set of multiple images; and determining a relative frequency with which items associated with a first product category and a second product category may be depicted together within a set of multiple images based on normalizing the set of multiple appearances, wherein determining a complementary category relationship may be based on the relative frequency.
[0008] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for: generating one or more data objects using a machine learning model, the one or more data objects indicating a set of multiple relative frequencies at which items of respective product category pairs in a set of multiple product categories may be depicted together within a set of multiple images, the set of multiple relative frequencies including a relative frequency between a first product category and a second product category; and referencing the one or more data objects based on receiving a query image, wherein identifying the first product category may be based on referencing the one or more data objects.
[0009] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for performing the following operations: determining a set of multiple complementary category relationships between a first product category and other product categories in a set of multiple product categories, the set of multiple complementary category relationships including complementary category relationships; determining a set of multiple scores associated with the set of multiple complementary category relationships, wherein the set of multiple scores can be associated with the relative strength of the relationship between the first product category and other product categories in the set of multiple product categories; and selecting a complementary category relationship from the set of multiple complementary category relationships based on the set of multiple scores, wherein identifying the first group of items and the second group of items can be based on the complementary category relationship.
[0010] In some examples of the methods, devices, and non-transitory computer-readable media described herein, a complementary category relationship between a first product category and a second product category can be associated with a higher score than an additional complementary category relationship between the first product category and a third product category based on items of the first product category being more frequently depicted with items of the second product category than items of the first product category are depicted with items of the third product category.
[0011] In some examples of the methods, apparatus, and non-transitory computer-readable media described herein, a complementary category relationship may also be generated based on purchase history information associated with a relative frequency with which items in the first product category and items in the second product category may be purchased together via the online marketplace.
[0012] In some examples of the methods, apparatus, and non-transitory computer-readable media described herein, the first group of items may be visually similar to the query item based on the first group of items satisfying one or more visual similarity criteria, and the one or more visual similarity criteria include color criteria, style criteria, texture criteria, or any combination thereof.
[0013] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for determining one or more user preferences of a user associated with the client device, wherein identifying the second set of items may be based on the one or more user preferences.
[0014] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for receiving a request for complementary items associated with a third product category from a client device; identifying a third group of items associated with the third product category based on the request, the third group of items may be depicted with the first group of items within a set of multiple images; and sending search results to the client device in response to the query image, the search results indicating one or more additional complementary items associated with the third product category that are available for purchase from the online marketplace, wherein the one or more additional complementary items include one or more items in the third group of items or one or more items that are visually similar to the third group of items, or both.
[0015] Some examples of the methods, apparatus, and non-transitory computer-readable media described herein may also include operations, features, means, or instructions for performing the following operations: receiving, via a client device, a selection of a complementary item of one or more complementary items associated with a second product category; identifying a third group of items associated with the second product category, the third group of items can be depicted within a set of multiple images and can be visually similar to the complementary items; based on the second product category being complementary to the third product category, identifying a fourth group of items associated with the third product category, the fourth group of items can be depicted together with the third group of items within a set of multiple images; and sending additional search results to the client device based on the selection, the additional search results indicating one or more additional complementary items associated with the third product category that can be purchased from the online marketplace, wherein the one or more additional complementary items include one or more of the fourth group of items or one or more items that are visually similar to the fourth group of items, or both.
[0016] A method is described. The method may include: identifying a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace; determining a complementary category relationship between the first product category and the second product category based on a relative frequency that the items associated with the first product category and the second product category are depicted together in the set of multiple images; receiving a query image depicting a query item associated with the first product category in the set of multiple product categories from a client device; identifying the first product category as complementary to the second product category in the set of multiple product categories based on the complementary category relationship; identifying a first group of items associated with the first product category, the first group of items being depicted within the set of multiple images and visually similar to the query item; identifying a second group of items associated with the second product category, the second group of items being depicted together with the first group of items in the set of multiple images based on the second product category being complementary to the first product category; and sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
[0017] A device is described. The device may include: at least one processor; at least one memory coupled to the at least one processor; and instructions stored in the at least one memory. The instructions may be executable by the at least one processor to cause the device to perform the following operations: identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories in an online marketplace; determine a complementary category relationship between the first product category and the second product category based on a relative frequency at which the items associated with the first product category and the second product category are depicted together within the set of multiple images; receive a query image depicting a query item associated with a first product category in a set of multiple product categories from a client device; identify that the first product category is complementary to a second product category in a set of multiple product categories based on the complementary category relationship; and The method comprises: identifying a first group of items associated with a first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query items; identifying a second group of items associated with the second product category based on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within the set of multiple images; and sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from an online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items, or both.
[0018] Another apparatus is described. The apparatus may include: means for identifying a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace; means for determining a complementary category relationship between the first product category and the second product category based on a relative frequency with which items associated with the first product category and the second product category are depicted together within the set of multiple images; means for receiving from a client device a query image depicting a query item associated with a first product category in the set of multiple product categories; means for identifying that the first product category is complementary to a second product category in the set of multiple product categories based on the complementary category relationship; and means for identifying that the first product category is complementary to a second product category in the set of multiple product categories. A device for identifying a first group of items associated with a first product category, the first group of items being depicted within a set of multiple images and being visually similar to a query item; a device for identifying a second group of items associated with a second product category based on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within a set of multiple images; and a device for sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from an online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
[0019] A non-transitory computer-readable medium storing code is described. The code may include instructions executable by a processor to: identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories in an online marketplace; determine a complementary category relationship between the first product category and the second product category based on a relative frequency with which the items associated with the first product category and the second product category are depicted together in the set of multiple images; receive a query image depicting a query item associated with the first product category in the set of multiple product categories from a client device; identify the first product category as complementary to the second product category in the set of multiple product categories based on the complementary category relationship; identify a first group of items associated with the first product category, the first group of items being depicted within the set of multiple images and being visually similar to the query item; identify a second group of items associated with the second product category, based on the second product category being complementary to the first product category, the second group of items being depicted together with the first group of items in the set of multiple images; and send search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 An example of a complementary item discovery method system according to aspects of the present disclosure is shown, which supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce.
[0021] Figure 2 An example of a flow chart supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure.
[0022] Figure 3 and Figure 4 Examples of data objects supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce are shown in accordance with aspects of the present disclosure.
[0023] Figure 5 An example of an item recommendation graph supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure.
[0024] Figure 6 An example of a process flow supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure is shown.
[0025] Figure 7 A block diagram of an apparatus supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure.
[0026] Figure 8 A block diagram of an item recommendation component supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure.
[0027] Fig. 9 A diagram of a system including an apparatus supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure.
[0028] Fig.10 A flow chart illustrating a method of supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure. DETAILED DESCRIPTION
[0029] Online markets can use various techniques to recommend items or products to users. Recommended products can include products that the user explicitly queries or requests, and / or products that are "complementary" (e.g., related) to the queried products. For example, if a user submits a query for "sofa," the online market can present to the user a product list of other sofas, as well as other complementary items related to the sofa that the user may be interested in, such as coffee tables, pillows, etc. Some systems may define "complementary items" as items that have been purchased together by other users. However, such a definition may limit the number and variety of complementary items that can be recommended to users. Other systems may be configured to identify complementary items based on the title, item description, etc. of the item. However, such titles and item descriptions are typically entered by individual sellers, and therefore may vary greatly between users, and may be vague or misleading. Therefore, complementary items identified based on item titles and item descriptions may be unreliable or unrelated to the corresponding query items. In addition, even if the complementary items are fully related to the corresponding query items, the color or style of the returned complementary items may conflict with the corresponding query items or otherwise be inconsistent with the corresponding query items. For example, a user who submits an image of a “minimalist sofa” is unlikely to purchase a throw pillow decorated with ornate embroidery, even though a “throw pillow” is technically a complementary item to a “sofa.”
[0030] Thus, aspects of the present disclosure relate to techniques for identifying complementary products and product categories based on how often the respective products / product categories are depicted together in images. Specifically, the techniques described herein can be used to identify related product categories based on how often items of the respective product categories are depicted together in images, and use the related product categories to provide more suitable and visually appealing complementary products in response to a received query product.
[0031] For example, a machine learning model may be configured to perform object detection to identify items within an image, such as an image retrieved from a publication, a webpage, a social media platform, etc. The machine learning model may then be configured to identify "complementary category relationships" between different product categories based on how often items of the corresponding product categories are depicted together within the image. Specifically, product categories whose items are frequently depicted together may be associated with complementary category relationships with a higher strength or score than product categories whose items are rarely depicted together.
[0032] For example, a complementary category relationship between "sofa" and "coffee table" may be associated with a higher score than "sofa" and "gardening equipment" due to the fact that sofas are more often pictured with coffee tables than with gardening equipment. In some cases, the purchase history of items can also be used to generate complementary category relationships (e.g., the system identifies a relationship between "sofa" and "coffee table" based on the user previously purchasing the sofa and coffee table together).
[0033] Continuing with reference to the same example, the user may then submit a query image of a sofa as the query item. In this example, the system may identify complementary product categories based on the generated complementary category relationships. For example, the system may determine that "tables" is the most relevant or most complementary product category for "sofas." Therefore, the system may identify the sofa in the image that is most visually similar to the sofa from the query image. The system may then identify tables that are depicted with the most visually similar sofa, and may then provide a product list of the identified tables (or other tables that are visually similar to the identified tables) as recommended complementary products for the sofa from the query image.
[0034] By identifying complementary category relationships based on how often different types of items are shown together in the same image, the system may be able to identify the most relevant or complementary types of items that a user may be searching for. In other words, upon receiving a query depicting a sofa, complementary category relationships may enable the system to determine that the user is most likely to search for (or would be most likely to be interested in) complementary tables, pillows, chairs, curtains, etc. In addition, by identifying items that are visually similar to the queried item, and then identifying complementary items that are displayed with such visually similar items, the techniques described herein may enable the system to return complementary items that are not only related / related to the queried item, but are also more likely to be aesthetically and stylistically complementary to the queried item. In other words, the complementary table recommended to the user in the above example may exhibit a style, color, and texture that is complementary to the style, color, and texture of the queried sofa.
[0035] Aspects of the disclosure are initially described in the context of an environment supporting an on-demand database service. Additional aspects of the disclosure are described in the context of example flow charts, example item recommendation graphs, example data objects, and example process flows. Aspects of the disclosure are further illustrated by and described with reference to apparatus diagrams, system diagrams, and flow charts relating to data-driven automated visual consistency estimation for complementary item discovery in e-commerce.
[0036] Figure 1 An example of a system 100 for cloud computing according to various aspects of the present disclosure is shown, which supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce. System 100 includes cloud client 105, contact (e.g., client device 110), cloud platform 115, automated visual consistency system 145, and data center 120. Cloud platform 115 can be an example of a public or private cloud network. Cloud client 105 can access cloud platform 115 through network connection 135. The network can implement transmission control protocol and Internet protocol (TCP / IP), such as the Internet, or can implement other network protocols. Cloud client 105 can be an example of a computing device, such as a server (e.g., cloud client 105-a), a smart phone (e.g., cloud client 105-b), or a laptop (e.g., cloud client 105-c). In other examples, cloud client 105 can be a desktop computer, a tablet computer, a sensor, or another computing device or system capable of generating, analyzing, sending or receiving communications. In some examples, cloud client 105 may be operated by a user who is part of an enterprise, corporation, nonprofit organization, startup, or any other type of organization.
[0037] The cloud client 105 may interact with multiple contacts (e.g., client devices 110). The interactions 130 may include communications, opportunities, purchases, sales, or any other interactions between the cloud client 105 and the client device 110. Data may be associated with the interactions 130. The cloud client 105 may access the cloud platform 115 to store, manage, and process data associated with the interactions 130. In some cases, the cloud client 105 may have an associated security or permission level. The cloud client 105 may access certain applications, data, and database information within the cloud platform 115 based on the associated security or permission level, but may not access other applications, data, and database information.
[0038] The client device 110 may interact with the cloud client 105 in person or via telephone, email, web, text message, mail, or any other suitable form of interaction (e.g., interactions 130-a, 130-b, 130-c, and 130-d), which may include network-based communications (e.g., messages sent and received via a network). Interaction 130 may be a business-to-business (B2B) interaction or a business-to-consumer (B2C) interaction. The client device 110 may also be referred to as a customer, a potential customer, a lead, a client, or some other suitable term. In some cases, the client device 110 may be an example of a user device, such as a server (e.g., client device 110-a), a laptop (e.g., contact 110-b), a smartphone (e.g., contact 110-c), or a sensor (e.g., client device 110-d). In other cases, the client device 110 may be another computing system. In some cases, the client device 110 may be operated by a user or a user group. The user or user group may be associated with a business, a manufacturer, or any other suitable organization.
[0039] The cloud platform 115 can provide on-demand database services to the cloud client 105. In some cases, the cloud platform 115 can be an example of a multi-tenant database system. In this case, the cloud platform 115 can serve multiple cloud clients 105 with a single software instance. However, other types of systems can be implemented, including - but not limited to - client server systems, mobile device systems, and mobile network systems. In some cases, the cloud platform 115 can support CRM solutions. This can include support for sales, services, marketing, communities, analysis, applications, and the Internet of Things. The cloud platform 115 can receive data associated with contact interactions 130 from the cloud client 105 through a network connection 135, and can store and analyze the data. In some cases, the cloud platform 115 can directly receive data from the interaction 130 between the client device 110 and the cloud client 105. In some cases, the cloud client 105 can develop an application to be run on the cloud platform 115. The cloud platform 115 can be implemented using a remote server. In some cases, the remote server can be located at one or more data centers 120.
[0040] The data center 120 may include multiple servers. Multiple servers may be used for data storage, management, and processing. The data center 120 may receive data from the cloud platform 115 via a connection 140, or directly receive data from the cloud client 105 or the interaction 130 between the client device 110 and the cloud client 105. For security purposes, the data center 120 may utilize multiple redundancies. In some cases, the data stored at the data center 120 may be backed up at different data centers (not shown in the picture) via data copies.
[0041] Subsystem 125 may include cloud client 105, cloud platform 115, automated visual consistency system 145, and data center 120. In some cases, data processing may occur at any one of the components of subsystem 125, or at a combination of these components. In some cases, a server may perform data processing. The server may be cloud client 105 or located at data center 120. In some aspects, automated visual consistency system 145 may be included in an online market (e.g., an electronic market) or implemented by an online market (e.g., an electronic market). In addition, in some aspects, automated visual consistency system 145 may be communicatively coupled to various components of subsystem 125 (e.g., cloud client 105, cloud platform 115, data center 120), and may be configured to perform one or more of the steps, operations, or functions described herein.
[0042] As described previously herein, an online marketplace may use a variety of techniques to recommend items or products to a user. Recommended products may include products that the user has explicitly queried or requested, and / or products that are "complementary" (e.g., related) to the queried product. For example, if a user submits a query for "sofa," the online marketplace may present to the user a product list of other sofas, as well as other complementary items related to sofas that the user may be interested in, such as coffee tables, pillows, etc. Some systems may define "complementary items" as items that have been purchased together by other users. However, such a definition may limit the number and types of complementary items that can be recommended to a user.
[0043] Other systems may be configured to identify complementary items based on item titles, item descriptions, based on predefined manual attributes, and the like. However, such titles and item descriptions are typically entered by individual sellers and may therefore vary widely between users and may be ambiguous or misleading. Furthermore, defining complementary items based on predefined manual attributes may require manual supervision and labeling, which may be consuming and time consuming, may require expert-level knowledge for consistent annotation, and may not be scalable for different datasets. Furthermore, such manual labeling may result in category-specific labels and complementary relationships, which may not be scalable across other categories for unseen labels. Therefore, complementary items identified based on item titles and item descriptions may be unreliable or unrelated to the corresponding query item. Furthermore, even in cases where the complementary item is sufficiently related to the corresponding query item, the color or style of the returned complementary item may conflict with or otherwise be inconsistent with the corresponding query item. For example, a user who submits an image of a "minimalist sofa" is unlikely to purchase a throw pillow decorated with ornate embroidery, even though a "throw pillow" is technically a complementary item to a "sofa."
[0044] therefore, Figure 1 Various aspects of the system 100 shown and described in the text may support techniques for identifying complementary products and product categories based on how frequently corresponding products / product categories are depicted together within an image. Specifically, the automated visual consistency system 145 of the system 100 may support techniques for identifying related product categories based on how frequently items of corresponding product categories are depicted together within an image, and using the related product categories to provide more appropriate and visually appealing complementary products in response to a received query product. In particular, for items belonging to categories such as home decor and fashion, visual attributes may be more helpful or descriptive for determining complementary items than co-purchased item information (where functionality may be the focus). Thus, the techniques described herein may leverage visual similarity and consistency to drive complementary item recommendations.
[0045] For example, the automated visual consistency system 145 of the subsystem 125 associated with the online marketplace may utilize or implement a machine learning model to perform object detection to identify items within an image, such as an image retrieved from a publication, a webpage, a social media platform, etc. The machine learning model may then be configured to identify "complementary category relationships" between different product categories based on how often items of the corresponding product categories are depicted together within the image. Specifically, product categories whose items are frequently depicted together may be associated with complementary category relationships with a higher strength or score than product categories whose items are rarely depicted together.
[0046] For example, a complementary category relationship between "sofa" and "coffee table" may be associated with a higher score than "sofa" and "gardening equipment" due to the fact that sofas are more often pictured with coffee tables than gardening equipment. In some cases, the purchase history of items may also be used to generate complementary category relationships (e.g., an automated visual consistency system 145 implemented by an online marketplace identifies a relationship between "sofa" and "coffee table" based on a user previously purchasing the sofa and coffee table together).
[0047] Continuing with reference to the same example, the client device 110 may subsequently submit a query image of a sofa as a query item to an automated visual consistency system 145 associated with an online marketplace. In this example, the automated visual consistency system 145 may identify complementary product categories based on the generated complementary category relationships. For example, the automated visual consistency system 145 may determine that "tables" is the most relevant or most complementary product category for "sofas." Therefore, the automated visual consistency system 145 may identify a sofa in the image that is most visually similar to the sofa from the query image. The automated visual consistency system 145 may then identify a table that is depicted with the most visually similar sofa, and may then provide a product list of the identified tables (or other tables that are visually similar to the identified tables) to the client device 110 as recommended complementary products for the sofa from the query image.
[0048] By identifying complementary category relationships based on determining which types of items are often displayed together, the system 100 (e.g., the automated visual consistency system 145) may be able to identify the most relevant or complementary types of items that a user may be searching for. In other words, upon receiving a query depicting a sofa, the complementary category relationships may enable the system to determine that the user is most likely to search for (or would be most likely to be interested in) complementary tables, pillows, chairs, curtains, etc. In addition, by identifying items that are visually similar to the queried item, and then identifying complementary items that are displayed with such visually similar items, the techniques described herein may enable the system 100 to return complementary items that are not only related / related to the queried item, but are also more likely to be aesthetically and stylistically complementary to the queried item. In other words, the complementary table recommended to the user in the above example may exhibit a style, color, and texture that is complementary to the style, color, and texture of the queried sofa.
[0049] The techniques described herein can improve the quality of complementary items recommended to users. Specifically, aspects of the present disclosure can utilize visual similarity and visual consistency to recommend complementary items of other categories that visually and stylistically match or complement the queried item.
[0050] Those skilled in the art will appreciate that one or more aspects of the present disclosure may be implemented in the system 100 to additionally or alternatively address other issues beyond those described above. In addition, various aspects of the present disclosure may provide technical improvements to the "conventional" systems or processes described herein. However, the description and drawings include only example technical improvements resulting from implementing various aspects of the present disclosure, and therefore do not represent all technical improvements provided within the scope of the claims.
[0051] Figure 2 An example of a flowchart 200 that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure is shown. Aspects of flowchart 200 may implement or be implemented by aspects of system 100 (e.g., aspects of automated visual consistency system 145). Specifically, flowchart 200 illustrates steps and operations for recommending complementary items based on visual similarity and consistency, as previously described herein.
[0052] In some aspects, flowchart 200 illustrates example steps, operations, and functions that may be performed by one or more components of system 100, such as automated visual consistency system 145. In this regard, Figure 2 The automated visual consistency system 145 shown in FIG. 1 may include Figure 2The various steps and operations shown and / or may be configured to perform Figure 2 The various steps and operations shown in FIG.
[0053] Co-presented items depicted in the same image may provide clues to visual consistency. Therefore, the techniques described herein may be configured to quantify visual consistency (e.g., visual and / or style consistency) across multiple item images using machine learning models and other image processing techniques. In some cases, complementary item recommendations based on visual consistency may include two main operations, which are generally shown in the upper and lower halves of flowchart 200.
[0054] As a first major operation (or set of operations), as shown in the upper half of flowchart 200, and as will be further described herein, category relationships between different item / product categories can be defined. In this first major operation, an object detector can identify pairs of items within the same image. The item pairs can be used to generate or calculate data objects (e.g., two-dimensional (2D) graphs) based on the co-presentation of items of the corresponding categories in the analyzed image, which data objects are similar to the correlations of the corresponding categories. Using clustering, the data objects / graphs can be divided into groups or pairs of item / product categories that appear together in the image. In this regard, the generated data objects / graphs can specify or include relevant "complementary categories" with respect to the queried items. Item pairs that populate the relevant categories, representations, can be learned (e.g., by a processor, a machine learning model, etc.) to quantify the visual consistency of the items.
[0055] As a second major operation (or set of operations), as shown in the lower half of flowchart 200, and as will be further described herein, a system (e.g., automated visual concordance system 145) can receive a query image of a query item and perform a visually concordant complementary item retrieval for the query image / item. Specifically, items within the received query image can be identified or detected, and the resulting data objects can be used to identify relevant complementary categories. Visual concordance of the query item can be performed to identify similar-looking items in a catalog of images of related categories. Subsequently, items associated with categories that are related / complementary to the query item can be identified and returned as search results in response to the received query image.
[0056] In this regard, a system (eg, configured to implement Figure 2The automated visual consistency system 145 of the flowchart 200 shown in FIG. 1 can be configured to learn visual consistency (e.g., learn what item / product categories exhibit visual consistency) by analyzing a collection of object image pairs that are seen together within a multi-object image (e.g., instances where the same image depicts multiple objects / items together). The techniques described herein for retrieving complementary items based on visual consistency may require little to no supervision, compared to some conventional techniques for identifying complementary items. Furthermore, such techniques may be adaptable and adjustable over time, scalable to a large number of items and product categories, and applicable to a wide variety of product categories and / or different styles.
[0057] By way of example, we will now refer to Figure 2 200. At 205, a system (e.g., the automated visual consistency system 145) may collect or acquire a set of images to be analyzed for visual consistency. The automated visual consistency system 145 may be configured to retrieve images from relevant catalogs or web pages (e.g., fashion catalogs, home and garden magazines, etc.), social media platforms, etc.
[0058] At 210, the automated visual consistency system 145 may perform object detection to identify items in the image collected at 205. For example, the processor of the server may utilize one or more machine learning models to perform object detection to identify items within the analyzed image. Individual items identified within the image may be associated with (or belong to) individual item or product categories, such as product categories of an online marketplace, which may include different hierarchical categories of individual items or products (e.g., a "clothing" category, a "sunglasses" category, a "dresses" category, a "home and garden" category, a "furniture" category, a "coffee table" category, etc.).
[0059] Specifically, the automated visual consistency system 145 can be configured to identify multiple items that are co-depicted together in the same image. For example, when analyzing an image of a bedroom, the automated visual consistency system 145 can be configured to identify a bed, a mattress, a pillow, a side table, a rug, curtains, etc., so that these items are considered to be "co-depicted" together in the same image. In some aspects, the automated visual consistency system 145 can be configured to utilize one or more machine learning models (e.g., machine learning classifiers, algorithms, etc.) or computer vision techniques that are configured to identify and classify items detected within the analyzed image. Generally, the automated visual consistency system 145 can utilize any technology known in the art to perform item detection and recognition, including neural networks, deep learning, convolutional neural networks (CNNs), etc. In some implementations, separate machine learning models can be trained to recognize different categories of items (e.g., a first model is trained to recognize fashion / apparel items, a second model is trained to recognize furniture items, etc.).
[0060] In this regard, the automated visual consistency system 145 may be configured to identify groups or pairs of items that are frequently depicted together within the analyzed images (e.g., co-depicted group of items 215). The automated visual consistency system 145 may process each image of the image set to determine which group of items is detected in the image, and may count the number of times over the image set that groups or pairs of items are depicted together in a set of analyzed images (e.g., for 15 of the 90 images being analyzed, a couch and a pillow are depicted together in the same image). In other words, the automated visual consistency system 145 may be configured to “learn” or recognize the relative frequency with which different types of items (and / or different categories of items) are pictured together within the same image. For example, Figure 2 , the automated visual consistency system 145 can be configured to identify a first pair of commonly depicted items 215-a including a sofa and a pillow based on the sofa and the pillow being frequently depicted together within the same image of a set of analyzed images. Similarly, the automated visual consistency system 145 can be configured to identify a second pair of commonly depicted items 215-b including a coffee table and a sofa based on the coffee table and the sofa being frequently depicted together within the same image of a set of analyzed images. In another example, in a set of analyzed images, there may be multiple instances in the image set where a coffee table and a sofa are identified as being depicted together in the same image.
[0061] In an additional or alternative implementation, the automated visual consistency system 145 can utilize general object detection techniques to identify items / products in an image. In this case, when the system identifies items or objects without a category identification (e.g., the product category is unknown), the system can still be configured to generate or identify pairs of items / objects that are depicted together within the image, and can be configured to train a model for learning visual consistency. When such a visual consistency model is trained, for any given item / object pair, the system can be configured to calculate a score that estimates visual consistency. However, in such a case where the system cannot identify the corresponding product category of the identified item, the system may not know or recognize what product category the model is recommending.
[0062] At 220, a machine learning model may be trained (e.g., by automated visual consistency system 145) to recognize complementary category relationships between different product categories. Specifically, the machine learning model may be configured to determine which category groups or category pairs are related to each other (e.g., which categories are "complementary" to each other) based on the relative frequency with which items of the respective product categories are co-depicted together within a single image of a set of analyzed images. In an example, the machine learning model may recognize that in a set of images, fifteen of the images depict both a coffee table and a sofa in the same image.
[0063] For example, Figure 2 As shown in , the co-depicted item pairs 215-a, 215-b can be input into a machine learning model (e.g., a neural network, a deep learning model, a CNN), where the machine learning model is configured to identify complementary category relationships. For example, the machine learning model can be configured to identify a first complementary category relationship between "sofa" and "pillow" (as shown by the first pair of co-depicted items 215-a) based on the sofa and pillows being frequently depicted together in the analyzed image. Therefore, the first complementary category relationship can indicate that "sofa" and "pillow" are complementary to each other because they are frequently shown / positioned near each other in the same image, and it is logical that a user who purchases a sofa may also be interested in pillows.
[0064] Similarly, the machine learning model can be configured to identify a second complementary category relationship between “coffee table” and “sofa” based on the coffee table and the sofa being frequently depicted together in the analyzed image (as shown by the second pair of commonly depicted items 215-b). Therefore, the second complementary category relationship can indicate that “sofa” and “coffee table” are complementary to each other because they are frequently shown / positioned close to each other in a single image, and it would be logical that a user who purchases a sofa might also be interested in the coffee table. In this regard, the automated visual consistency system 145 (e.g., a machine learning model) can determine complementary category relationships between different product categories based on the relative frequency with which items of the corresponding product categories are depicted together within the analyzed image.
[0065] In some cases, the system / machine learning model can be configured to normalize the number of items identified in the image to generate a complementary category relationship that more accurately represents groups / pairs of complementary items. In other words, the automated visual consistency system 145 can be configured to determine complementary category relationships based on the relative frequency with which items are depicted together, rather than the total number of occurrences or occurrences of different items being depicted in the image.
[0066] For example, the automated visual consistency system 145 may be configured to determine the number of occurrences of different types of items or product categories in a set of analyzed images (e.g., the frequency with which "couch" and "curtains" are depicted in the same image in a set of analyzed images), and may be configured to normalize the absolute number of occurrences of different items / categories to determine the relative frequency with which the items / categories are depicted together. By normalizing the absolute number of occurrences of different items or categories, the techniques described herein may more effectively identify related / complementary category pairs. For example, if a sofa is depicted in an image far more often than any other type of item, the relationship between the sofa and the other product categories may be given a greater weight. Therefore, by normalizing the total number of sofas depicted in an image relative to the number of occurrences of other types of items, the automated visual consistency system 145 may alternatively be configured to determine complementary category relationships based on the relative frequency with which the sofa is depicted with the other items, which may result in more accurate identification of related / complementary categories.
[0067] In some aspects, at 220, the machine learning model can be configured to determine or assign scores to corresponding complementary category relationships. The scores of the complementary category relationships can indicate the relative strength of the relationships between the various product categories. For example, a first complementary category relationship between "sofas" and "coffee tables" can be assigned a higher score than a second complementary category relationship between "sofas" and "shovels" due to the fact that sofas are more frequently depicted (and sold) with coffee tables than with shovels.
[0068] In some cases, the system / machine learning model may be additionally configured to determine complementary category relationships between categories of items based on which types of items are frequently sold together. For example, at 225, sales metadata (e.g., purchase history information) from an online marketplace may be retrieved by the automated visual consistency system 145 (e.g., from the data center 120) and input into the machine learning model, where the machine learning model is configured to determine complementary category relationships based on both (1) the sales metadata and (2) the relative frequency with which items of different categories are depicted together within an image. For example, the machine learning model may use the sales metadata to determine a first complementary category relationship between "sofas" and "pillows" based on the fact that sofas and pillows are frequently co-depicted together within the same image of a set of images and that sofas and pillows are frequently purchased together from the online marketplace. The sales metadata 225 may also include other types of metadata, including counts of image views or list views that include one or more images in which complementary items are depicted together in the same image.
[0069] In some implementations, the automated visual consistency system 145 can be configured to generate one or more data objects that indicate or display complementary category relationships between various groups / pairs of product categories (e.g., data objects that indicate the relative frequency with which items from different categories are co-depicted and / or purchased together). The data objects that depict or show complementary category relationships can include network diagrams, 2D shading diagrams, etc., and can be referenced. Figure 3 and Figure 4 Further shown and described.
[0070] Figure 3 An example of a data object 300 that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce according to aspects of the present disclosure is shown. Aspects of the data object 300 may implement or be implemented by aspects of the system 100, the flowchart 200, or both.
[0071] Specifically, Figure 3 The data object 300 in FIG. 1 shows an example of a network diagram that can be generated (e.g., by a machine learning model) to illustrate or depict complementary category relationships between corresponding product categories. Figure 2 As described in, the automated visual consistency system 145 can be configured to generate data object 300 (e.g., a network graph) based on the relative frequency with which items from different product categories are co-depicted together within an image, based on purchase history information (e.g., sales metadata) indicating groups of items that are frequently sold together, or both.
[0072] The nodes of data object 300 may illustrate or represent different product categories, such as product categories of an online marketplace. In some cases, product categories may be grouped into different product category sets, where nodes of data object 300 having the same fill / shading may be included in the same group. Specifically, automated visual consistency system 145 may be configured to generate groups of related complementary category relationships (e.g., based on how often items of the categories in the group are depicted / sold together).
[0073] For example, example groupings of product categories may include: blankets, bunk beds, recliners, sofas, beds, futons, and mattresses. Other example groupings of product categories may include: (1) benches, chairs, desks, dining tables, stools, (2) armchairs, coffee tables, cushions, loveseats, sofas, (3) beds, headboards, pillows, tablecloths, (4) cabinets, dressers, rugs (carpets), (4) lamps, lampshades, tables, curtains, mirrors, (5) paintings, artwork, statues. In some cases, the system may perform category groupings across product categories using the Louvain algorithm for cross-category partitioning, a general graph partitioning algorithm, and the like.
[0074] like Figure 3 As shown in , the nodes of the data object 300 (which represent different product categories) can be connected to each other via one or more lines, where the lines indicate the relationship between the corresponding nodes / categories. Figure 3 The lines connecting the nodes in may represent complementary category relationships between the product categories represented by the corresponding nodes.
[0075] In some cases, the automated visual consistency system 145 (e.g., a machine learning model) can be configured to generate the data object 300 so that the properties of the corresponding lines indicate the relative strength of the relationship between the corresponding product categories. In other words, the properties of the line between a pair of product categories (e.g., line thickness, number of lines between nodes, dashed line type) can indicate or be associated with a score of the complementary category relationship between the corresponding product categories.
[0076] For example, product categories with a greater number of lines (and / or thicker lines, or lines with different dash patterns) may illustrate categories that are more closely related to each other (and therefore may illustrate a complementary category relationship with a higher score). Conversely, product categories that have fewer (or no) lines (and / or thinner lines, or lines with other dash patterns) between them may illustrate categories that are less closely related to each other (and therefore may illustrate a complementary category relationship with a lower score).
[0077] It should be noted that the relationship indicated by data object 300 can be updated over time based on the relative frequency with which items of different product categories are co-depicted together within an image, based on purchase history information indicating groups of items that are frequently sold together (e.g., sales metadata), or both. Over time, machine learning model 220 can receive one or more new images (e.g., update the relative frequency with which items of different product categories are co-depicted together within an image based on the one or more new images), new sales metadata 225, or both for updating the complementary category relationship depicted in data object 300.
[0078] Figure 4 An example of a data object 400 that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce according to aspects of the present disclosure is shown. Aspects of the data object 400 may implement or be implemented by aspects of the system 100, the flowchart 200, the data object 300, or any combination thereof. For example, the data object 400 may include, for example, Figure 3 300 and 400 are additional or alternative depictions of the identified complementary category relationships shown and described in data object 300. In this regard, in some cases, data object 300 and data object 400 are examples of two different ways of illustrating the identified complementary category relationships between items that are seen / sold together.
[0079] Figure 4 Data object 400 in illustrative example illustrates a 2D shading diagram that may be generated (e.g., by a machine learning model) to illustrate or depict complementary category relationships between corresponding product categories. In some cases, automated visual consistency system 145 may generate a 2D matrix describing the complementary category relationships, and may partition the 2D matrix to generate the diagram shown in data object 400. Figure 2 As described in, a system (e.g., automated visual consistency system 145) can be configured to generate data object 400 (e.g., a 2D shadow map) based on the relative frequency with which items from different product categories are co-depicted together within an image, based on purchase history information (e.g., sales metadata) indicating groups of items that are frequently sold together, or both.
[0080] Data object 400 illustrates different combinations of product categories, as indicated on the X-axis and Y-axis of data object 400. The shaded blocks at the intersections of the respective product category pairs may indicate the degree of relatedness (e.g., degree of complementarity) of the respective product category pairs, wherein darker blocks indicate product category pairs that are more related / complementary to each other, and lighter blocks indicate product category pairs that are less related / complementary to each other. In this regard, the darker blocks may indicate higher scores for the complementary category relationships between the respective category pairs, and the lighter blocks may indicate lower scores for the complementary category relationships between the respective product category pairs.
[0081] In addition, each shaded block may include a number, e.g. Figure 4 . It should be noted herein that, for simplicity, only a subset of the shaded blocks are depicted with numbers. In some aspects, the numbers within the shaded blocks may further indicate the degree of relevance (e.g., degree of complementarity) of the various product category pairs, where larger numbers indicate product category pairs that are more relevant / complementary to each other, and smaller numbers indicate product category pairs that are less relevant / complementary to each other. In this regard, the numbers may include or may represent relative "scores" between product category pairs, where the scores indicate the degree of relevance / complementarity of the various product categories to each other.
[0082] Thus, larger numbers may generally be associated with darker shaded blocks, while smaller numbers may generally be associated with lighter shaded blocks. In some cases, the automated visual consistency system 145 may generate numbers / scores between various product category pairs, and may generate shading for the corresponding blocks based on the determined numbers / scores (e.g., the shading for the block at the intersection of "bed" and "armchair" may be based on the determined number / score of "9").
[0083] In some aspects, the depicted numbers within the shaded blocks may be normalized on a scale of 0 to 100. Specifically, as described herein, the depicted numbers may be normalized based on the relative frequency with which items / products of different product categories are depicted together within an image in a set of images analyzed by the automated visual consistency system 145. For example, in some cases, the automated visual consistency system 145 may determine the number of occurrences / occurrences at which items of each category are depicted in an image and normalize the total number of occurrences / occurrences across categories. In this regard, where relative colors / shades are determined based on the determined numbers / scores, the corresponding colors / shades may therefore be normalized on a scale between 0 and 100 based on the relative frequency with which items of each product category pair are depicted together within an image (and / or the relative frequency with which items of each product category pair are sold together).
[0084] For example, due to the fact that blankets and armchairs are depicted and / or sold together at a relatively low frequency (e.g., approximately 7% of images depicting blankets also depict armchairs, and vice versa), the block at the intersection of “blanket” and “armchair” is represented by the number / score “7” and is therefore depicted as a light color, indicating a relatively low score of the complementary category relationship between “blanket” and “armchair.” In contrast, due to the fact that blankets and beds are depicted and / or sold together at a relatively high frequency (e.g., approximately 22% of images depicting blankets also depict beds, and vice versa), the block at the intersection of “blanket” and “bed” is represented by the number / score “22” and is therefore depicted as a darker color, indicating a relatively high score of the complementary category relationship between “blanket” and “bed.” In this regard, by comparing the numbers / scores (and / or colors / shading) between "blanket / armchair" and "blanket / bed", it can be seen that "blanket" and "bed" are more related / complementary than "armchair", due to the blocks at the intersection of "blanket / bed" depicting higher numbers (and darker colors) compared to the blocks at the intersection of "blanket / armchair"
[0085] Will refer again Figure 2 200. At 230, a system (e.g., automated visual consistency system 145) may receive a query image from a user (e.g., client device 110), wherein the query image depicts one or more query items (e.g., query items 240). At 235, automated visual consistency system 145 may perform object detection, as shown and described with respect to step 210 of flowchart 200. In this regard, automated visual consistency system 145 may utilize one or more machine learning models (e.g., the machine learning model used in step 210, including a neural network, CNN, etc.) to identify and classify query items 240 depicted in the query image.
[0086] For example, Figure 2 As shown in , a user can submit a query image depicting a sofa (query item 240) to find complementary items of the sofa. In this example, the automated visual consistency system 145 can identify the query item 240 within the query image and identify that the query item 240 belongs to the product category "sofas" of the online marketplace.
[0087] At 245, the automated visual consistency system 145 may perform visual consistency to identify other items that are visually similar to the query item 240. In other words, the automated visual consistency system 145 may identify other sofas (and / or other sofas sold in the online marketplace) depicted in the image analyzed at 205 that are visually similar to the sofa (query item 240) depicted in the query image. Figure 2As shown in , automated visual consistency system 145 may identify another sofa (visually similar item 250 ) that is visually similar to the sofa (query item 240 ) depicted in the query image.
[0088] When evaluating items that are "visually similar" to the query item 240, the automated visual consistency system 145 can analyze a set of visual similarity standards, including color standards, style standards, texture standards, etc. In this regard, a visually similar item 250 identified as being visually similar to the sofa from the query image (query item 240) can exhibit the same or similar color, style, texture, pattern, size, shape, and / or fabric as the sofa from the query image. In this regard, visual similarity can be evaluated by comparing one or more attributes. In additional or alternative implementations, visual similarity can be determined or assessed using a trained model. That is, a model for assessing visual similarity can be trained on a set of images, where similar items are grouped in different categories, where the model is trained to learn feature vectors about the categories. Using these feature vectors, the model can be configured to calculate a score that estimates the similarity of a pair of items, where a pair of items with a high score indicates a pair of items that are highly similar (or identical) to each other visually, and where a pair of items with a low score indicates a pair of items that are not visually similar to each other.
[0089] In some aspects, embedding generation can be performed using feature vectors (e.g., 2048-dimensional visual embeddings), dimensionality reduction (e.g., 200 dimensions using PCA), distances between item images (e.g., cosine (default) and Euclidean), Facebook AI Similarity Search (FAISS) for k-nearest neighbor (k-NN) search, etc. In addition, after retrieving complementary items 260 for the user, the system can perform post-processing operations, such as performing deduplication using perceptual hashing (e.g., removing duplicate copies of similar complementary items). In some cases, the system can be configured to perform deduplication after retrieving complementary items because some product listings on an online marketplace (or other image database) may include the same items. By removing duplicates, the suggested complementary items become more diverse and inclusive.
[0090] For example, after identifying an item / product in an image, the automated visual consistency system 145 may examine 2048 dimensions of visual feature attributes to generate a feature vector for the item / product that includes the 2048 values or dimensions. In some cases, the automated visual consistency system 145 may then select 200 values / dimensions from the 2048 values / dimensions that best represent the data or carry the most information, and then compare these 200 values / dimensions with the values / dimensions of other feature vectors associated with other items / products to determine a distance metric (e.g., Euclidean distance) between the feature vectors, where the distance metric is used to determine the level of visual similarity / consistency between the corresponding items / products. For example, if one of the dimensions is always the same, then even if that dimension has the highest value, that dimension may be considered not to represent the data (and therefore may not be selected as one of the 200 selected dimensions), while another dimension with a low value may still be very representative of the data (and therefore may be selected as one of the 200 selected dimensions).
[0091] In some cases, dimensionality reduction (e.g., selecting 200 values / dimension from 2048 values / dimension) may be performed to speed up the retrieval process of complementary items (e.g., instead of comparing 2048 values, the system may compare 200 values for assessing the similarity of two items to speed up the retrieval of results). Likewise, in some implementations, dimensionality reduction may not be performed.
[0092] At 255, the automated visual consistency system 145 may identify complementary category relationships associated with the query item 240 depicted in the query image to identify other product categories that are complementary to the product category of the query item 240. In other words, the automated visual consistency system 145 may identify complementary category relationships for "sofa" to determine other product categories that are complementary to sofa. For example, Figure 2 As shown in , the system can recognize that "table", "pillow", "chair", and "curtain" are complementary product categories for sofas (for example, based on the determined complementary category relationships between "sofa" and "table", between "sofa" and "pillow", etc.).
[0093] In some aspects, the automated visual consistency system 145 can be configured to identify relevant complementary category relationships associated with the product category of the query item 240 by referencing the one or more data objects generated at 220. In other words, the automated visual consistency system 145 can identify relevant complementary category relationships associated with the product category of the query item 240 by referencing the one or more data objects generated at 220. Figure 3 and Figure 4 The data objects 300 , 400 shown in are used to identify other product categories that are related to (eg, complementary to) “sofa”.
[0094] In the event that a query image includes multiple items of different categories, the automated visual consistency system 145 may be configured to identify complementary category relationships associated with all (or at least a subset) of the items / categories within the query image. In other words, for a query image that includes multiple items, the automated visual consistency system 145 may be configured to evaluate the complementary category relationships of all items in order to identify items that will be complementary to all items depicted in the query image. As a specific use case, a user may submit an image of their bedroom with a bed, nightstands, and rug, and request that the system "complete the room" by finding items of other categories (e.g., curtains, decorative items, etc.) that will be complementary to the depicted bed, nightstands, and rug.
[0095] In some cases, as previously described herein, the automated visual consistency system 145 can be configured to rank the relevant complementary category relationships according to scores corresponding to the complementary category relationships. In some cases, higher scores can indicate stronger complementary category relationships (e.g., categories that are more related or complementary to each other), while lower scores can indicate weaker complementary category relationships (e.g., categories that are less related or complementary to each other). For example, Figure 2 As shown in , a first complementary category relationship between "sofa" and "table" can be associated with a score of 0.48, while a second complementary category relationship between "sofa" and "pillow" can be associated with a score of 0.31. In some aspects, the scores of complementary category relationship pairs / groups (e.g., scores of 0.48, 0.31, etc.) can be normalized to some values (e.g., based on 300 or 400, the system normalizes scores between 0 and 1 and then sorts the scores from high to low). In this example, the automated visual consistency system 145 may have determined that tables are more frequently co-depicted (and / or purchased) with sofas than pillows, and therefore tables are more complementary to sofas than pillows.
[0096] In some cases, the automated visual consistency system 145 may be configured to return complementary items of other product categories that exhibit the highest scores. For example, in the case where a user submits a query image of a sofa (query item 240), the automated visual consistency system 145 may be configured to return an image / product list of tables due to the fact that the complementary category relationship between "sofas" and "tables" exhibits the highest score. In other cases, the automated visual consistency system 145 may be configured to return images / product lists of items belonging to product categories with the X highest scores (e.g., returning a product list of the three most complementary categories of tables, pillows, and chairs). In this case, the search results returned to the user may include complementary items of multiple categories, where the complementary items may be sorted or organized based on the corresponding scores (e.g., in descending order of applicable scores, where tables are provided or viewed first, followed by pillows, then chairs, etc.).
[0097] In additional or alternative situations, the user may be able to input or request the type of product category that the user is looking for. For example, where the user submits an image of a sofa (query item 240), the user may request that the automated visual consistency system 145 return a product list of curtains that would match / complement the sofa. In this example, the automated visual consistency system 145 may be configured to return a product list of curtains, despite the fact that curtains are less complementary to sofas than tables, pillows, etc.
[0098] Once the automated visual consistency system 145 identifies which product category(s) are complementary to the query item 240 , and therefore identifies which items / product listings to recommend to the user as complementary items, the flowchart 200 may proceed to step 265 .
[0099] At 265 , the automated visual consistency system 145 may identify complementary items 260 of a complementary product category that are depicted together with (or visually similar to) the visually similar item 250 that is visually similar to the query item 240 .
[0100] For example, as previously described herein, the system may identify other sofas (visually similar items 250) that are visually similar to the sofa from the query image (query item 240). In this example, and where the automated visual consistency system 145 is configured to identify tables that are complementary to the sofa from the query image, the system may identify other tables (complementary items 260) that are depicted together with the visually similar item 250 (and / or sold together with the visually similar item 250). That is, once the automated visual consistency system 145 has found other sofas that are visually similar to the sofa from the query image, the automated visual consistency system 145 may identify tables that are depicted together with the visually similar sofa (or identify tables that are visually similar to the tables that are depicted together with the visually similar sofa). In this regard, the automated visual consistency system 145 may be configured to identify tables (e.g., complementary items 260) that are more likely to be visually and stylistically complementary to the sofa from the query image (query item 240).
[0101] In some cases, the automated visual consistency system 145 may be configured to take into account the user's preferences when retrieving complementary items. User preferences may include preferences for certain styles, colors, fabrics, textures, etc., and may be obtained from multiple sources, including preferences explicitly indicated by the user, the user's previous purchase history, etc. For example, some users may prefer very neutral tones, such that the automated visual consistency system 145 is configured to identify and return most (or specific) complementary items that fit neutral tones. In contrast, other users may like trendy textures or patterns. In this case, the automated visual consistency system 145 may be configured to identify complementary items with trendy textures / patterns that are depicted together with visually similar items 250. For example, if the system is configured to return images / lists of throw pillows to complement the queried sofa, the system will find trendy throw pillows that are depicted together with a sofa that is visually similar to the sofa from the query image.
[0102] As previously described herein, when determining visual similarity between items, the automated visual consistency system 145 can generate feature vectors (e.g., 2048-dimensional feature vectors) for the corresponding items / products and compare the feature vectors between the items / products to generate some distance metric (e.g., Euclidean distance) between the feature vectors, where the distance metric indicates the relative degree of visual similarity between the corresponding items / products.
[0103] At 270, the automated visual consistency system 145 may, in response to the received query image, send search results indicating complementary items 260 (or items visually similar to complementary items 260) to the client device 110. In other words, after identifying other tables that are visually and stylistically complementary to the sofa from the query image, the automated visual consistency system 145 may return search results for sale via the online marketplace that include images of complementary tables and / or product listings that list complementary tables.
[0104] In some cases, the automated visual consistency system 145 may be configured to further train the machine learning model to identify complementary items by associating the query item 240 and the complementary item 260 as “complementary embeddings.” In other words, the system may associate the sofa from the query image with the table returned as a search result as complements of each other, and may input the associated sofa and complementary table back into the machine learning model to further train the machine learning model to identify complementary items. In this regard, as the automated visual consistency system 145 performs more searches and retrieves more search results for complementary items (and as users purchase or otherwise interact with the recommended complementary items), the machine learning model may become increasingly effective and efficient at identifying complementary items. Thus, the returned complementary items (and user interactions of selecting or purchasing the recommended complementary items) may be used as additional input to further train the machine learning model to retrieve complementary items across a wide range of product categories.
[0105] In some aspects, the Figure 2 200 . Specifically, the automated visual consistency system 145 can be configured to iteratively provide complementary items 260 as the user continues to browse the online marketplace and interact with the item / product listings. For example, continuing with the same example, the automated visual consistency system 145 can provide search results indicating complementary items 260 (complementary tables) in response to a query image depicting a sofa. In this example, the user can purchase, view, select, or otherwise interact with one of the tables (complementary items 260) returned to the user. In this example, the table selected or purchased by the user can be used as a new "query item" to repeat the process shown in flowchart 200.
[0106] In other words, the system (e.g., automated visual consistency system 145) can be configured to use the selected table to identify visually similar tables (step 245), identify complementary product categories for the table (step 255), identify items of other product categories that are complementary to the visually similar table (step 265), and return another search result including recommended items that are complementary to the table. In this case, the user can start with an image of a sofa, receive a search result with a table, select one of the received tables, and receive another search result with a chair that is complementary to the selected table. This technique can also be implemented in an "infinite scroll" implementation, where a user scrolling through an online marketplace's item / product list results in traversing between specific object categories based on relatedness. In other words, a user can begin a search for a table, where continued scrolling returns results for a chair (as a complementary item to the table), then returns results for a looper (as a complementary item to the chair), then returns results for table decorations (as a complementary item to the looper), and so on.
[0107] You can refer to Figure 5 Further shown and described are incidental advantages of techniques for returning complementary items using visual consistency.
[0108] Figure 5 An example of an item recommendation graph 500 is shown that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure. Aspects of the item recommendation graph 500 can implement or be implemented by aspects of the system 100, the flowchart 200, the data objects 300, 400, or any combination thereof.
[0109] Item recommendation graph 500 illustrates a different example in which a user submits a query image 510 (shown on the left) of a bed, and the system (e.g., automated visual consistency system 145) returns or recommends curtains that are complementary to the bed depicted in the query image 510. In this regard, the bed depicted in the query image 510 may be Figure 2 Another example of a query item 240 in , while the curtains depicted in the search result 515 may be in Figure 2 An example of complementary item 260 returned to the user in FIG.
[0110] The top row 505-a of the item recommendation graph 500 shows an example of a search query and results where the system returns complementary items based on conventional techniques. Specifically, the top row 505-a shows a prior art technique for retrieving complementary items based on purchase history and / or item titles or descriptions. As can be seen in the top row 505-a, the recommended "complementary" curtains returned in the search results 515-a may not aesthetically or stylistically match or complement the modern-looking bed depicted in the query image 510 on the left. In other words, a user who submitted the query image 510 of a modern-looking bed is unlikely to purchase the brightly printed curtains returned to the right in the search results 515-a.
[0111] In contrast, bottom row 505 - b illustrates techniques of the present disclosure that recommend complementary products based on visually similar products that are frequently depicted together in images. In this regard, bottom row 505 - b illustrates example search queries and results that may be performed by automated visual consistency system 145 according to techniques described herein.
[0112] As reference Figure 2 As described, upon receiving the query image 510, the system (e.g., the automated visual consistency system 145) can be configured to identify visually similar beds, identify curtains that are depicted with the visually similar bed within the image, and return the identified curtains (or other visually similar curtains) to the user via search results 515-b. By comparing the top row 505-a and the bottom row 505-b, it can be seen that the techniques described herein (as shown in the bottom row 505-b) return recommendations for curtains that are more aesthetically and stylistically matched to the modern bed depicted in the query image 510.
[0113] Thus, the techniques described herein can improve the quality of item recommendations provided to users, and can increase engagement (e.g., increase purchases) with returned complementary items. Thus, the complementary item recommendation techniques described herein can be used to find items that best match a set of query items (e.g., return complementary items that "complete a room" based on the query items in a room), and / or find items that are visually complementary to the query items (e.g., find matching items).
[0114] Figure 6 An example of a process flow 600 that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce according to aspects of the present disclosure is shown. Aspects of the process flow 600 may implement or be implemented by aspects of the system 100, the flowchart 200, the data objects 300, 400, the item recommendation graph 500, or any combination thereof.
[0115] Process flow 600 may include a client device 605 and a server device 610, which may be referred to herein as Figures 1 to 5 Examples of other devices described above. For example, the client device 605 may be Figure 2 The example of the client device 110 shown in FIG. 1 and the server device 610 may be as shown in FIG. Figure 2 Examples of automated visual consistency system 145, cloud client 105, subsystem 125, cloud platform 115, or any combination thereof shown in FIG.
[0116] In some examples, the operations shown in process flow 600 can be performed by hardware (e.g., including circuit systems, processing blocks, logic components, and other components), code executed by a processor (e.g., software), or any combination thereof. The following alternative examples can be implemented in which some of the steps are performed in a different order than described or not performed at all. In some cases, the steps may include additional features not mentioned below, or more steps may be added.
[0117] At 615, server device 610 may perform object detection to identify items depicted in a set of images. Figure 2 As shown in steps 205 and 210 of , server device 610 may receive a set of images (e.g., from a webpage, a social media platform, etc.) and identify items and corresponding product categories that are depicted together within the same image of the set of images. As previously described herein, server device 610 may be configured to utilize one or more machine learning models (e.g., neural networks, CNNs, etc.) to perform object detection and recognition / classification.
[0118] At 620, the server device 610 may identify complementary category relationships between various product category groups / pairs. Specifically, the server device 610 may identify complementary category relationships between various categories based on performing object detection / classification at 605. For example, Figure 2 As shown in step 220 of , the server device 610 can utilize one or more machine learning models to identify complementary category relationships between product categories based on (1) the relative frequency with which items of the corresponding product categories are depicted together in the analyzed images and / or (2) the relative frequency with which items of the corresponding product categories are sold together via the online marketplace.
[0119] As previously described herein, server device 610 may be configured to determine the number of times items of the respective product categories are depicted in the image, and to normalize the absolute number of occurrences to determine the relative frequency with which items of the respective product categories are co-depicted with one another. In some aspects, server device 610 may be configured to generate data objects (e.g., Figure 3 and Figure 4 3 and 400), the data objects indicate or represent the determined complementary category relationships between corresponding product category groups or pairs.
[0120] At 625, server device 610 may receive a query image from client device 605, wherein the query image depicts one or more query items. Figure 2 As shown in step 230 of , server device 610 may receive a query image of a sofa. In this example, server device 610 may be configured to identify and return items of other product categories that are associated with (eg, matching or complementary to) the sofa from the query image.
[0121] At 630, server device 610 may identify complementary category relationships that are related to or applicable to one or more items from the query image. Figure 2 As shown in step 255 of , server device 610 can determine and identify complementary category relationships between the sofa and (1) tables, (2) pillows, (3) chairs, and (4) curtains. In this regard, server device 610 can identify the table, pillows, chairs, and curtains as "complementary items" of the sofa from the query image.
[0122] In some cases, server device 610 may generate a data object by referring to a data object generated by server device 610 (e.g., by referring to Figure 3 and Figure 4 ) to identify applicable complementary category relationships (and therefore complementary categories / items).
[0123] At 635, server device 610 may determine a score for the identified complementary category relationship. Figure 2 As shown in step 255 of , server device 610 may determine that a first complementary category relationship between "sofas" and "tables" has a score of 0.48, while a second complementary category relationship between "sofas" and "pillows" may be associated with a score of 0.31. In this example, server device 610 may determine that tables are more frequently co-depicted (and / or purchased) with sofas than pillows, and therefore tables are more complementary to sofas than pillows. In some cases, server device 610 may determine that tables are more frequently co-depicted (and / or purchased) with sofas than pillows. Figure 3 and Figure 4 400) shown in FIG. 4A to determine or calculate a score associated with a complementary category relationship.
[0124] At 640, the server device 610 may select one or more other categories (e.g., a second category) that are complementary to the item in the query image, which will be returned to the user as complementary items. In some cases, the system may be configured to return complementary items of other product categories that exhibit the highest scores. For example, in the case where the user submits a query image of a sofa (query item 240), due to the fact that the complementary category relationship between "sofa" and "table" exhibits the highest score, the system may be configured to return images / product lists of tables. In other cases, the system may be configured to return images / product lists of items belonging to product categories with the X highest scores (e.g., returning a product list of the three most complementary categories of tables, pillows, and chairs).
[0125] For simplicity and illustration purposes, server device 610 may determine that items (e.g., tables) associated with the second product category will be returned to the user in process flow 600. However, as described herein, in other cases, server device 610 may select multiple product categories to be returned to the user as search results for complementary items.
[0126] At 645, the server device 610 may identify one or more user preferences of a user associated with the client device 605. In some cases, the user preferences may be obtained from a user profile associated with the user in an online marketplace, determined based on the user's prior purchase history, and the like. As previously described herein, the server device 610 may be configured to consider the user's preferences when retrieving complementary items. The user preferences may include preferences for certain styles, colors, fabrics, textures, and the like, and may be obtained from a variety of sources, including preferences explicitly indicated by the user, the user's prior purchase history, and the like.
[0127] At 650, server device 610 may identify a first set of items that are visually similar to the query item from the query image. Figure 2 As shown in step 245 of , if server device 610 receives a query image of a sofa, server device 610 can identify other sofas in the analyzed image that are visually similar to the sofa from the query image. When evaluating items that are "visually similar" to the query item, server device 610 can analyze a set of visual similarity criteria, including color criteria, style criteria, texture criteria, etc.
[0128] At 655, server device 610 may identify a second set of items (e.g., a complementary set of items) of a second product category. Specifically, server device 610 may identify items of the second product category that are depicted together with the first set of items that are visually similar to the query item (and / or items that are visually similar to items depicted together with the first set of visually similar items). For example, as in Figure 2 As shown in step 265, after identifying other sofas that are visually similar to the sofa from the query image (e.g., visually similar item 250), the server device 610 can be configured to identify tables that are depicted together with the visually similar sofa (or tables that are visually similar to the table that is depicted together with the visually similar sofa).
[0129] At 660, server device 610 can send search results to client device 605, wherein the search results include complementary items of the second product category that are complementary to the query item from the query image received at 625. Specifically, the search results can include items of the second product category identified at 655. For example, as in Figure 2 As shown in step 270 of , server device 610 may return search results including complementary tables (eg, complementary items 260 ) that are complementary to the sofa depicted in the query image.
[0130] At 665, server device 610 may receive from client device 605 a selection of an item or product listing included within the search results provided at 660. For example, a user may select, view, purchase, or otherwise interact with a complementary table provided via the search results.
[0131] At 670, server device 610 may return additional search results including images or product listings of items associated with a third product category that are complementary to the item selected at 665. In some cases, server device 610 may be configured to utilize the item selection at 665 as a new “query image / input” to repeat the query. Figure 6 The steps shown and described in order to return a second search result including complementary items in a third product category.
[0132] For example, continuing with the same example, server device 610 may provide a first search result indicating complementary tables in response to a query image depicting a sofa at 660. In this example, the user may purchase, view, select, or otherwise interact with one of the tables returned to the user at 665. In this example, the table selected or purchased by the user at 665 may be used as a new "query item" to repeat the process shown in process flow 600. In other words, server device 610 may be configured to use the selected table to identify complementary product categories for the table (step 630), identify visually similar tables (step 650), identify items of other product categories that are complementary to the visually similar table (step 655), and return another search result including recommended items that are complementary to the table (step 670).
[0133] Figure 7 A block diagram 700 of an apparatus 705 supporting data-driven automated visual consistency estimation for complementary item discovery in e-commerce is shown in accordance with aspects of the present disclosure. The apparatus 705 may include an input module 710, an output module 715, and an automated visual consistency system 720. In some aspects, Figure 7 The automated visual consistency system 720 shown in FIG. 7 may be Figure 1 An example of an automated visual consistency system 145 is shown in FIG. 705 . The device 705 may also include a processor. Each of these components may communicate with each other (eg, via one or more buses).
[0134] Input module 710 can manage input signals for device 705. For example, input module 710 can identify input signals based on interaction with a modem, keyboard, mouse, touch screen, or similar device. These input signals can be associated with user input or processing at other components or devices. In some cases, input module 710 can utilize such as or other known operating systems to process the input signals. The input module 710 can send aspects of these input signals to other components of the device 705 for processing. For example, the input module 710 can send the input signals to the automated visual consistency system 720 to support data-driven automated visual consistency estimation for complementary item discovery in e-commerce. In some cases, the input module 710 can be as described in reference Fig. 9 Components of I / O controller 910 are described.
[0135] Output module 715 can manage output signals of device 705. For example, output module 715 can receive signals from other components of device 705 (e.g., automated visual consistency system 720) and can send these signals to other components or devices. In some examples, output module 715 can send output signals for display in a user interface, for storage in a database or data store, for further processing at a server or server cluster, or for any other process at any number of devices or systems. In some cases, output module 715 can be a reference to Fig. 9 Components of I / O controller 910 are described.
[0136] For example, the automated visual consistency system 720 may include an object detection component 725, a complementary category relationship component 730, a query image receiving component 735, an item identification component 740, a search result component 745, or any combination thereof. In some examples, the automated visual consistency system 720 or its various components may be configured to perform various operations (e.g., receive, monitor, transmit) using or otherwise cooperating with the input module 710, the output module 715, or both. For example, the automated visual consistency system 720 may receive information from the input module 710, send information to the output module 715, or be integrated with the input module 710, the output module 715, or a combination of both to receive information, send information, or perform various other operations as described herein.
[0137] The object detection component 725 can be configured to support using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace. The complementary category relationship component 730 can be configured to support using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency that the items associated with the first product category and the second product category are depicted together within a set of multiple images. The query image receiving component 735 can be configured to support receiving a query image from a client device, the query image depicting a query item associated with a first product category in a set of multiple product categories. The complementary category relationship component 730 can be configured to support identifying that the first product category is complementary to a second product category in a set of multiple product categories based on the complementary category relationship. The item identification component 740 can be configured to support identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and visually similar to the query item. Item identification component 740 may be configured to support identifying a second set of items associated with the second product category based on the second product category being complementary to the first product category, the second set of items being depicted with the first set of items within a set of a plurality of images. Search result component 745 may be configured to support sending search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace to a client device in response to the query image, wherein the one or more complementary items include one or more items in the second set of items or one or more items that are visually similar to the second set of items.
[0138] Figure 8 A block diagram 800 of an automated visual consistency system 820 is shown that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure. The automated visual consistency system 820 can be an example of aspects of the item recommendation component 720 as described herein, or both. In some aspects, Figure 8 The automated visual consistency system 820 shown in FIG. 8 may be Figure 1Automated visual consistency system 820 or its various components may be examples of apparatus for performing various aspects of data-driven automated visual consistency estimation for complementary item discovery in e-commerce as described herein. For example, automated visual consistency system 820 may include object detection component 825, complementary category relationship component 830, query image receiving component 835, item identification component 840, search result component 845, machine learning model component 855, relationship score component 860, user preference component 865, user input component 870, or any combination thereof. Each of these components may communicate with each other directly or indirectly (e.g., via one or more buses).
[0139] The object detection component 825 may be configured to support using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace. The complementary category relationship component 830 may be configured to support using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency that the items associated with the first product category and the second product category are depicted together within a set of multiple images. The query image receiving component 835 may be configured to support receiving a query image from a client device, the query image depicting a query item associated with a first product category in a set of multiple product categories. In some examples, the complementary category relationship component 830 may be configured to support identifying that the first product category is complementary to a second product category in a set of multiple product categories based on the complementary category relationship. The item identification component 840 may be configured to support identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and visually similar to the query item. In some examples, item identification component 840 can be configured to support identifying a second set of items associated with the second product category based on the second product category being complementary to the first product category, the second set of items being depicted with the first set of items within a set of multiple images. Search results component 845 can be configured to support sending search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace to a client device in response to the query image, wherein the one or more complementary items include one or more items in the second set of items or one or more items that are visually similar to the second set of items.
[0140] In some examples, item identification component 840 can be configured to support determining a set of multiple appearances indicating a frequency at which items associated with respective ones of the multiple product categories are depicted within the set of multiple images. In some examples, item identification component 840 can be configured to support determining a relative frequency at which items associated with a first product category and a second product category are depicted together within the set of multiple images based on normalizing the set of multiple appearances, wherein determining the complementary category relationship is based on the relative frequencies.
[0141] In some examples, machine learning model component 855 can be configured to support using the machine learning model to generate one or more data objects indicating a set of multiple relative frequencies at which items of respective pairs of product categories in a set of multiple product categories are depicted together within a set of multiple images, the set of multiple relative frequencies including a relative frequency between a first product category and a second product category. In some examples, machine learning model component 855 can be configured to support referencing the one or more data objects based on receiving a query image, wherein identifying the first product category is based on referencing the one or more data objects.
[0142] In some examples, the complementary category relationships component 830 can be configured to support determining a set of multiple complementary category relationships between a first product category in a set of multiple product categories and other product categories, the set of multiple complementary category relationships including the complementary category relationship. In some examples, the relationship score component 860 can be configured to support determining a set of multiple scores associated with the set of multiple complementary category relationships, wherein the set of multiple scores is associated with the relative strength of the relationship between the first product category in the set of multiple product categories and other product categories. In some examples, the complementary category relationships component 830 can be configured to support selecting a complementary category relationship from a set of multiple complementary category relationships based on the set of multiple scores, wherein identifying the first group of items and the second group of items is based on the complementary category relationship.
[0143] In some examples, based on items of the first product category being more frequently depicted with items of the second product category than items of the first product category are depicted with items of the third product category, a complementary category relationship between the first product category and the second product category is associated with a higher score than an additional complementary category relationship between the first product category and the third product category.
[0144] In some examples, the complementary category relationship is also generated based on purchase history information associated with the relative frequency with which items in the first product category and items in the second product category are purchased together via the online marketplace.
[0145] In some examples, the first group of items is visually similar to the query item based on the first group of items satisfying one or more visual similarity criteria. In some examples, the one or more visual similarity criteria include color criteria, style criteria, texture criteria, or any combination thereof.
[0146] In some examples, user preference component 865 can be configured to support determining one or more user preferences of a user associated with the client device, wherein identifying the second group of items is based on the one or more user preferences.
[0147] In some examples, user input component 870 can be configured to support receiving a request from a client device for complementary items associated with a third product category. In some examples, item identification component 840 can be configured to support identifying a third group of items associated with the third product category based on the request, the third group of items being depicted with the first group of items within a set of multiple images. In some examples, search results component 845 can be configured to support sending search results to the client device in response to the query image, the search results indicating one or more additional complementary items associated with the third product category that are available for purchase from the online marketplace, wherein the one or more additional complementary items include one or more items in the third group of items or one or more items that are visually similar to the third group of items.
[0148] In some examples, user input component 870 may be configured to support receiving, via a client device, a selection of a complementary item of one or more complementary items associated with a second product category. In some examples, item identification component 840 may be configured to support identifying a third group of items associated with the second product category, the third group of items being depicted within a set of multiple images and being visually similar to the complementary item. In some examples, item identification component 840 may be configured to support identifying a fourth group of items associated with a third product category based on the second product category being complementary to the third product category, the fourth group of items being depicted together with the third group of items within a set of multiple images. In some examples, search result component 845 may be configured to support sending additional search results to the client device based on the selection, the additional search results indicating one or more additional complementary items associated with the third product category that can be purchased from an online marketplace, wherein the one or more additional complementary items include one or more items in the fourth group of items or one or more items that are visually similar to the fourth group of items.
[0149] Fig. 9A diagram of a system 900 including an apparatus 905 that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure is shown. Apparatus 905 may be an example of or include components of apparatus 705 as described herein. Apparatus 905 may include components for bidirectional data communication, including components for sending and receiving communications, such as an automated visual consistency system 920, an I / O controller 910, a database controller 915, a memory 925, a processor 930, and a database 935. These components may be in electronic communication or otherwise coupled (e.g., operatively, communicatively, functionally, electronically, electrically) via one or more buses (e.g., bus 940). In some aspects, Fig. 9 The automated visual consistency system 920 shown in FIG. 9 may be Figure 1 An example of an automated visual consistency system 145 is shown in FIG.
[0150] I / O controller 910 can manage input signals 945 and output signals 950 of device 905. I / O controller 910 can also manage peripheral devices that are not integrated into device 905. In some cases, I / O controller 910 can represent a physical connection or port to an external peripheral device. In some cases, I / O controller 910 can utilize an operating system, such as Or other known operating systems. In other cases, I / O controller 910 may represent or interact with a modem, keyboard, mouse, touch screen, or similar device. In some cases, I / O controller 910 may be implemented as part of processor 930. In some examples, a user may interact with device 905 via I / O controller 910 or via hardware components controlled by I / O controller 910.
[0151] Database controller 915 can manage data storage and processing in database 935. In some cases, a user can interact with database controller 915. In other cases, database controller 915 can operate automatically without user interaction. Database 935 can be an example of a single database, a distributed database, multiple distributed databases, a data store, a data lake, or an emergency backup database.
[0152] The memory 925 may include random access memory (RAM) and ROM. The memory 925 may store computer-readable, computer-executable software, including instructions that, when executed, cause the processor 930 to perform the various functions described herein. In some cases, the memory 925 may include a BIOS, etc., which may control basic hardware or software operations, such as interaction with peripheral components or devices.
[0153] The processor 930 may include an intelligent hardware device (e.g., a general purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof). In some cases, the processor 930 may be configured to operate a memory array using a memory controller. In other cases, the memory controller may be integrated into the processor 930. The processor 930 may be configured to execute computer-readable instructions stored in the memory 925 to perform various functions (e.g., functions or tasks that support data-driven automated visual consistency estimation for complementary item discovery in e-commerce).
[0154] For example, the automated visual consistency system 920 may be configured to support using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace. The automated visual consistency system 920 may be configured to support using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency that the items associated with the first product category and the second product category are depicted together within a set of multiple images. The automated visual consistency system 920 may be configured to support receiving a query image from a client device, the query image depicting a query item associated with a first product category in a set of multiple product categories. The automated visual consistency system 920 may be configured to support identifying that the first product category is complementary to a second product category in a set of multiple product categories based on the complementary category relationship. The automated visual consistency system 920 may be configured to support identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query item. The automated visual consistency system 920 may be configured to support identifying a second set of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second set of items being depicted with the first set of items within a set of the plurality of images. The automated visual consistency system 920 may be configured to support sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second set of items or one or more items that are visually similar to the second set of items.
[0155] For example, the automated visual consistency system 920 may be configured to support identifying a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace. The automated visual consistency system 920 may be configured to support determining a complementary category relationship between the first product category and the second product category based on a relative frequency with which the items associated with the first product category and the second product category are depicted together within a set of multiple images. The automated visual consistency system 920 may be configured to support receiving a query image from a client device, the query image depicting a query item associated with a first product category in a set of multiple product categories. The automated visual consistency system 920 may be configured to support identifying that the first product category is complementary to a second product category in a set of multiple product categories based on a complementary category relationship. The automated visual consistency system 920 may be configured to support identifying a first group of items associated with the first product category, the first group of items being depicted within a set of multiple images and being visually similar to the query item. The automated visual consistency system 920 may be configured to support identifying a second set of items associated with the second product category based on the second product category being complementary to the first product category, the second set of items being depicted together with the first set of items within a set of the plurality of images. The automated visual consistency system 920 may be configured to support sending search results to a client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second set of items or one or more items that are visually similar to the second set of items.
[0156] Fig.10 A flow chart illustrating a method 1000 is shown that supports data-driven automated visual consistency estimation for complementary item discovery in e-commerce in accordance with aspects of the present disclosure. The operations of the method 1000 may be implemented by a server device or components thereof as described herein. For example, the operations of the method 1000 may be implemented by a server device as described herein. Figures 1 to 9 Additionally or alternatively, the server device may use dedicated hardware to perform various aspects of the described functionality.
[0157] At 1005, the method may include using one or more processors to identify a set of multiple items depicted within a set of multiple images, wherein the set of multiple items is associated with a first product category and a second product category in a set of multiple product categories of an online marketplace. The operations of 1005 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1005 may be performed as described with reference to Figure 8 The object detection component 825 described above is performed.
[0158] At 1010, the method may include using one or more processors to determine a complementary category relationship between the first product category and the second product category based on a relative frequency at which items associated with the first product category and the second product category are depicted together within a set of multiple images. The operations of 1010 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1010 may be performed as described with reference to Figure 8 The complementary category relationship component 830 described above is executed.
[0159] At 1015, the method may include receiving a query image from a client device, the query image depicting a query item associated with a first product category in a set of multiple product categories. The operations of 1015 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1015 may be performed as described with reference to Figure 8 The query image receiving component 835 described above is executed.
[0160] At 1020, the method may include identifying that the first product category is complementary to a second product category in a set of a plurality of product categories based on a complementary category relationship. The operations of 1020 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1020 may be performed as described with reference to Figure 8 The complementary category relationship component 830 described above is executed.
[0161] At 1025, the method may include identifying a first group of items associated with a first product category, the first group of items depicted within a set of multiple images and visually similar to the query item. The operations of 1025 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1025 may be performed as described with reference to Figure 8 The described item identification component 840 is performed.
[0162] At 1030, the method may include identifying a second group of items associated with the second product category based on the second product category being complementary to the first product category, the second group of items being depicted together with the first group of items within a set of multiple images. The operations of 1030 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1030 may be performed as described with reference to Figure 8 The described item identification component 840 is performed.
[0163] At 1035, the method may include sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that can be purchased from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items and / or one or more items that are visually similar to the second group of items. The operations of 1035 may be performed according to examples as disclosed herein. In some examples, aspects of the operations of 1035 may be performed as described with reference to Figure 8 The search result component 845 described above is executed.
[0164] The following provides an overview of various aspects of the present disclosure:
[0165] Aspect 1: A computer-implemented method, comprising: using one or more processors to identify a plurality of items depicted within a plurality of images, wherein the plurality of items are associated with a first product category and a second product category among a plurality of product categories in an online marketplace; using the one or more processors to determine a complementary category relationship between the first product category and the second product category based at least in part on a relative frequency with which items associated with the first product category and the second product category are depicted together within the plurality of images; receiving a query image from a client device, the query image depicting a query item associated with the first product category among the plurality of product categories; identifying the first product category and the second product category among the plurality of product categories based at least in part on the complementary category relationship; The method comprises the steps of: identifying a first group of items associated with the first product category, the first group of items being depicted within the plurality of images and being visually similar to the query item; identifying a second group of items associated with the second product category, the second group of items being depicted within the plurality of images with the first group of items, based at least in part on the second product category being complementary to the first product category; and sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items and / or one or more items that are visually similar to the second group of items.
[0166] Aspect 2: According to the computer-implemented method of Aspect 1, it also includes: determining a plurality of appearances, wherein the plurality of appearances indicate the frequency with which items associated with each of the plurality of product categories are depicted within the plurality of images; and determining a relative frequency with which items associated with the first product category and the second product category are depicted together within the plurality of images based at least in part on normalizing the plurality of appearances, wherein determining the complementary category relationship is based at least in part on the relative frequency.
[0167] Aspect 3: The computer-implemented method according to any one of Aspects 1 to 2 further includes: using a machine learning model to generate one or more data objects, the one or more data objects indicating a plurality of relative frequencies at which items of respective product category pairs in the plurality of product categories are depicted together within the plurality of images, the plurality of relative frequencies including the relative frequency between the first product category and the second product category; and referencing the one or more data objects based at least in part on receiving the query image, wherein identifying the first product category is based at least in part on referencing the one or more data objects.
[0168] Aspect 4: The computer-implemented method according to any one of Aspects 1 to 3 further includes: determining multiple complementary category relationships between the first product category and other product categories in the multiple product categories, the multiple complementary category relationships including the complementary category relationship; determining multiple scores associated with the multiple complementary category relationships, wherein the multiple scores are associated with the relative strength of the relationship between the first product category and the other product categories in the multiple product categories; and selecting the complementary category relationship from the multiple complementary category relationships based at least in part on the multiple scores, wherein identifying the first group of items and the second group of items is based at least in part on the complementary category relationship.
[0169] Aspect 5: A computer-implemented method according to Aspect 4, wherein, based at least in part on the fact that items of the first product category are more frequently depicted with items of the second product category than items of the first product category are depicted with items of the third product category, a complementary category relationship between the first product category and the second product category is associated with a higher score than an additional complementary category relationship between the first product category and the third product category.
[0170] Aspect 6: A computer-implemented method according to any one of Aspects 1 to 5, wherein the complementary category relationship is also generated at least in part based on purchase history information associated with the relative frequency of purchasing items in the first product category and items in the second product category together via the online marketplace.
[0171] Aspect 7: A computer-implemented method according to any one of Aspects 1 to 6, wherein the first group of items are visually similar to the query items based at least in part on the first group of items satisfying one or more visual similarity criteria, the one or more visual similarity criteria including color criteria, style criteria, texture criteria, or any combination thereof.
[0172] Aspect 8: The computer-implemented method according to any one of aspects 1 to 7 further includes: determining one or more user preferences of a user associated with the client device, wherein identifying the second group of items is at least partially based on the one or more user preferences.
[0173] Aspect 9: The computer-implemented method according to any one of Aspects 1 to 8 further includes: receiving a request for complementary items associated with a third product category from the client device; identifying a third group of items associated with the third product category based at least in part on the request, the third group of items being depicted together with the first group of items within the multiple images; and sending search results to the client device in response to the query image, the search results indicating one or more additional complementary items associated with the third product category that can be purchased from the online market, wherein the one or more additional complementary items include one or more items in the third group of items or one or more items that are visually similar to the third group of items.
[0174] Aspect 10: The computer-implemented method according to any one of Aspects 1 to 9 further includes: receiving, via the client device, a selection of a complementary item from the one or more complementary items associated with the second product category; identifying a third group of items associated with the second product category, the third group of items being depicted within the multiple images and being visually similar to the complementary items; identifying a fourth group of items associated with the third product category, based at least in part on the second product category being complementary to the third product category, the fourth group of items being depicted together with the third group of items within the multiple images; and sending additional search results to the client device based at least in part on the selection, the additional search results indicating one or more additional complementary items associated with the third product category that can be purchased from the online market, wherein the one or more additional complementary items include one or more items in the fourth group of items or one or more items that are visually similar to the fourth group of items.
[0175] Aspect 11: A device comprising: at least one processor; a memory coupled to the at least one processor; and instructions stored in the memory and executable by the at least one processor to enable the device to perform the method described in any one of Aspects 1 to 10.
[0176] Aspect 12: An apparatus comprising at least one means for executing the method according to any one of aspects 1 to 10.
[0177] Aspect 13: A non-transitory computer-readable medium storing code, the code comprising instructions executable by at least one processor to cause a system or device to perform the method according to any one of aspects 1 to 10.
[0178] It should be noted that the above methods describe possible implementations, and that operations and steps may be rearranged or otherwise modified, and other implementations are possible. In addition, aspects from two or more methods may be combined in a method.
[0179] The descriptions set forth herein in conjunction with the accompanying drawings describe example configurations, but do not represent all examples that can be implemented or within the scope of the claims. The term "exemplary" as used herein means "used as an example, instance, or illustration," rather than "preferred" or "superior to other examples." For the purpose of providing an understanding of the described technology, the detailed description includes specific details. However, these technologies can be practiced without these specific details. In some cases, well-known structures and devices are shown in block diagram form to avoid obscuring the concepts of the described examples.
[0180] In the drawings, similar components or features may have the same reference number. In addition, various components of the same type may be distinguished by following the reference number with a dash and a second reference that distinguishes between the similar components. If only the first reference number is used in the specification, the description applies to any of the similar components having the same first reference number, regardless of the second reference number.
[0181] The information and signals described herein may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0182] The various illustrative blocks and modules described herein in conjunction with the present disclosure may be implemented or executed using a general purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration), wherein a processor or a group of processors may individually, separately, or collectively perform any of the functions discussed herein.
[0183] The functions described herein may be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. Other examples and implementations are within the scope of the present disclosure and the appended claims. For example, due to the nature of software, the functions described above may be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination of these. The features that implement the functions may also be physically located at various locations, including being distributed so that the various parts of the functions are implemented at different physical locations. In addition, as used herein, including in the claims, as in the list of items (e.g., a list of items beginning with a phrase such as "at least one of" or "one or more of"), the "or" used indicates a list of inclusions, such that, for example, a list of at least one of A, B, or C means A or B or C or AB or AC or BC or ABC (i.e., A and B and C). In addition, as used herein, the phrase "based on" should not be interpreted as referring to a closed set of conditions. For example, an exemplary step described as "based on condition A" may be based on both condition A and condition B without departing from the scope of the present disclosure. In other words, as used herein, the phrase "based on" should be interpreted in the same manner as the phrase "based at least in part on."
[0184] Computer-readable media include both non-transient computer storage media and communication media, and communication media include any medium that is convenient for transferring a computer program from one place to another.Non-transient storage media can be any available medium that can be accessed by a general or special computer.By way of example and not limitation, non-transient computer-readable media may include RAM, ROM, electrically erasable programmable ROM (EEPROM), optical disk (CD) ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or may be used to carry or store desired program code devices in the form of instructions or data structures and any other non-transient medium that can be accessed by a general or special computer or a general or special processor.In addition, any connection is properly referred to as computer-readable media.For example, if software is transmitted from a website, server or other remote source using coaxial cable, optical fiber cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave, coaxial cable, optical fiber cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and disc, as used herein, includes CD, laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above are also included within the scope of computer-readable media.
[0185] The description herein is provided to enable those skilled in the art to make or use the present disclosure. Various modifications to the present disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but should conform to the broadest scope consistent with the principles and novel features disclosed herein.
Claims
1. A computer-implemented method comprising: identifying, using one or more processors, a plurality of items depicted within the plurality of images, wherein the plurality of items are associated with a first product category and a second product category of a plurality of product categories of an online marketplace; determining, using the one or more processors, a complementary category relationship between the first product category and the second product category based at least in part on a relative frequency with which items associated with the first product category and the second product category are depicted together within the plurality of images; receiving, from a client device, a query image depicting a query item associated with the first of the plurality of product categories; identifying that the first product category is complementary to the second product category of the plurality of product categories based at least in part on the complementary category relationship; identifying a first group of items associated with the first product category, the first group of items being depicted within the plurality of images and being visually similar to the query item; identifying a second group of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within the plurality of images; and Sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
2. The computer-implemented method of claim 1 , further comprising: determining a plurality of appearances indicating a frequency with which items associated with respective product categories are depicted together within the plurality of images; as well as Determining a relative frequency at which items associated with the first product category and the second product category are depicted together within the plurality of images based at least in part on normalizing the plurality of appearances, wherein determining the complementary category relationship is based at least in part on the relative frequency.
3. The computer-implemented method of claim 1 , further comprising: generating, using a machine learning model, one or more data objects indicating a plurality of relative frequencies at which items of respective pairs of product categories in the plurality of product categories are depicted together within the plurality of images, the plurality of relative frequencies including a relative frequency between the first product category and the second product category; as well as The one or more data objects are referenced based at least in part on receiving the query image, wherein identifying the first product category is based at least in part on referencing the one or more data objects.
4. The computer-implemented method of claim 1 , further comprising: determining a plurality of complementary category relationships between the first product category and other product categories in the plurality of product categories, the plurality of complementary category relationships including the complementary category relationship; determining a plurality of scores associated with the plurality of complementary category relationships, wherein the plurality of scores are associated with relative strengths of relationships between the first product category and other product categories in the plurality of product categories; and The complementary categorical relationship is selected from the plurality of complementary categorical relationships based at least in part on the plurality of scores, wherein identifying the first group of items and the second group of items is based at least in part on the complementary categorical relationship.
5. The computer-implemented method of claim 4, wherein: Based at least in part on the fact that items of the first product category are more frequently depicted with items of the second product category than items of the first product category are depicted with items of the third product category, a complementary category relationship between the first product category and the second product category is associated with a higher score than an additional complementary category relationship between the first product category and the third product category.
6. The computer-implemented method of claim 1 , wherein: The complementary category relationship is also generated based at least in part on purchase history information associated with a relative frequency with which items in the first product category and items in the second product category are purchased together via the online marketplace.
7. The computer-implemented method of claim 1 , wherein: The first group of items is visually similar to the query item based at least in part on the first group of items satisfying one or more visual similarity criteria, wherein the one or more visual similarity criteria include color criteria, style criteria, texture criteria, or any combination thereof.
8. The computer-implemented method of claim 1 , further comprising: One or more user preferences of a user associated with the client device are determined, wherein identifying the second set of items is based at least in part on the one or more user preferences.
9. The computer-implemented method of claim 1 , further comprising: receiving a request from the client device for a complementary item associated with a third product category; Based at least in part on the request, identifying a third group of items associated with the third product category, the third group of items being depicted within the plurality of images with the first group of items; as well as The search results are sent to the client device in response to the query image, the search results indicating one or more additional complementary items associated with the third product category that are available for purchase from the online marketplace, wherein the one or more additional complementary items include one or more items in the third group of items or one or more items that are visually similar to the third group of items.
10. The computer-implemented method of claim 1 , further comprising: receiving, via the client device, a selection of a complementary item from among the one or more complementary items associated with the second product category; identifying a third group of items associated with the second product category, the third group of items being depicted within the plurality of images and being visually similar to the complementary items; identifying a fourth group of items associated with the third product category based at least in part on the second product category being complementary to the third product category, the fourth group of items being depicted with the third group of items within the plurality of images; as well as Sending, based at least in part on the selection, additional search results to the client device, the additional search results indicating one or more additional complementary items associated with the third product category that are available for purchase from the online marketplace, wherein the one or more additional complementary items include one or more items in the fourth group of items or one or more items that are visually similar to the fourth group of items.
11. A system comprising: at least one processor; a memory coupled to the at least one processor; as well as Instructions stored in the memory and executable by the at least one processor to cause the system to perform the following operations: identifying a plurality of items depicted within the plurality of images, wherein the plurality of items are associated with a first product category and a second product category of a plurality of product categories in an online marketplace; determining a complementary category relationship between the first product category and the second product category based at least in part on a relative frequency with which items associated with the first product category and the second product category are depicted together within the plurality of images; receiving, from a client device, a query image depicting a query item associated with the first of the plurality of product categories; identifying that the first product category is complementary to the second product category of the plurality of product categories based at least in part on the complementary category relationship; identifying a first group of items associated with the first product category, the first group of items being depicted within the plurality of images and being visually similar to the query item; identifying a second group of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within the plurality of images; and Sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
12. The system according to claim 11, wherein: The instructions are also executable by the at least one processor to cause the system to perform the following operations: determining a plurality of appearances indicating a frequency with which items associated with respective product categories are depicted together within the plurality of images; and Determining a relative frequency at which items associated with the first product category and the second product category are depicted together within the plurality of images based at least in part on normalizing the plurality of appearances, wherein determining the complementary category relationship is based at least in part on the relative frequency.
13. The system according to claim 11, wherein: The instructions are also executable by the at least one processor to cause the system to perform the following operations: generating, using a machine learning model, one or more data objects indicating a plurality of relative frequencies at which items of respective pairs of product categories in the plurality of product categories are depicted together within the plurality of images, the plurality of relative frequencies including a relative frequency between the first product category and the second product category; as well as The one or more data objects are referenced based at least in part on receiving the query image, wherein identifying the first product category is based at least in part on referencing the one or more data objects.
14. The system according to claim 11, wherein: The instructions are also executable by the at least one processor to cause the system to perform the following operations: determining a plurality of complementary category relationships between the first product category and other product categories in the plurality of product categories, the plurality of complementary category relationships including the complementary category relationship; determining a plurality of scores associated with the plurality of complementary category relationships, wherein the plurality of scores are associated with relative strengths of relationships between the first product category and other product categories in the plurality of product categories; and The complementary categorical relationship is selected from the plurality of complementary categorical relationships based at least in part on the plurality of scores, wherein identifying the first group of items and the second group of items is based at least in part on the complementary categorical relationship.
15. The system of claim 14, wherein: Based at least in part on the fact that items of the first product category are more frequently depicted with items of the second product category than items of the first product category are depicted with items of the third product category, a complementary category relationship between the first product category and the second product category is associated with a higher score than an additional complementary category relationship between the first product category and the third product category.
16. The system according to claim 11, wherein: The complementary category relationship is also generated based at least in part on purchase history information associated with a relative frequency with which items in the first product category and items in the second product category are purchased together via the online marketplace.
17. The system of claim 11, wherein: The first group of items is visually similar to the query item based at least in part on the first group of items satisfying one or more visual similarity criteria, and wherein the one or more visual similarity criteria include color criteria, style criteria, texture criteria, or any combination thereof.
18. The system of claim 11, wherein: The instructions are also executable by the at least one processor to cause the system to perform the following operations: One or more user preferences of a user associated with the client device are determined, wherein identifying the second set of items is based at least in part on the one or more user preferences.
19. The system of claim 11, wherein: The instructions are also executable by the at least one processor to cause the system to perform the following operations: receiving a request from the client device for a complementary item associated with a third product category; Based at least in part on the request, identifying a third group of items associated with the third product category, the third group of items being depicted within the plurality of images with the first group of items; as well as The search results are sent to the client device in response to the query image, the search results indicating one or more additional complementary items associated with the third product category that are available for purchase from the online marketplace, wherein the one or more additional complementary items include one or more items in the third group of items or one or more items that are visually similar to the third group of items.
20. A non-transitory computer readable medium storing code, the code comprising instructions executable by at least one processor to cause a system to: recognizing a plurality of objects depicted within a plurality of images, wherein: the plurality of items being associated with a first product category and a second product category of a plurality of product categories in the online marketplace; determining a complementary category relationship between the first product category and the second product category based at least in part on a relative frequency with which items associated with the first product category and the second product category are depicted together within the plurality of images; receiving, from a client device, a query image depicting a query item associated with the first of the plurality of product categories; identifying that the first product category is complementary to the second product category of the plurality of product categories based at least in part on the complementary category relationship; identifying a first group of items associated with the first product category, the first group of items being depicted within the plurality of images and being visually similar to the query item; identifying a second group of items associated with the second product category based at least in part on the second product category being complementary to the first product category, the second group of items being depicted with the first group of items within the plurality of images; as well as Sending search results to the client device in response to the query image, the search results indicating one or more complementary items associated with the second product category that are available for purchase from the online marketplace, wherein the one or more complementary items include one or more items in the second group of items or one or more items that are visually similar to the second group of items.
