Entropy-based machine learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for entropy-based machine learning. One of the methods includes providing, to one or more user devices, a graphical element representing an item on an exchange platform; receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform; generating, using the interaction data, an entropy ratio machine learning feature; and training one or more machine learning models using the entropy ratio machine learning feature.

Description

ENTROPY-BASED MACHINE LEARNINGBACKGROUND

[0001] This specification relates to improving efficiencies of machine learning models and improving a user interface of an exchange platform by utilizing entropy-based analysis.

[0002] An exchange platform enables the exchange of goods, content, and / or services between users and providers. Providers can list or provide their goods, content, and / or services on the exchange platform, and users obtain the goods, content, and / or services from the providers via the exchange platform.

[0003] The exchange platform can determine which goods, content, and / or services are displayed to the user. For example, a user can provide one or more search queries to the exchange platform and receive one or more web pages listing a set of goods, content, and / or services matching the search queries. However, existing systems for determining the set of goods, contents, and / or services for display may not factor in marketplace fitness or imbalance in the goods, contents, and / or services being offered, and thus cannot optimize listings of products / services or offerings of additional products / services based on demand and supply conditions on the exchange platform.SUMMARY

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of providing, to one or more user devices, a graphical element representing an item on an exchange platform; receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform; generating, using the interaction data, an entropy ratio machine learning feature; and training one or more machine learning models using the entropy ratio machine learning feature. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0005] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. In particular, one embodiment includes all thefollowing features in combination. Feature 1 : Training the one or more machine learning models using the entropy ratio machine learning feature comprises: providing one or more values representing a search query to the one or more machine learning models; providing, based on output of the one or more machine learning models processing the one or more values representing the search query, a set of one or more graphical elements to a set of user devices; providing the one or more values representing the search query to a modified version of the one or more machine learning models; providing, based on output of the modified version of the one or more machine learning models processing the one or more values representing the search query, a second set of one or more graphical elements to the set of user devices; receiving (i) first interaction data based on the provided set of one or more graphical elements and (ii) second interaction data based on the provided second set of one or more graphical elements; and adjusting one or more values of the one or more machine learning models using the first interaction data and the second interaction data. Feature 2: Adjusting the one or more values of the one or more machine learning models using the first interaction data and the second interaction data comprises: adjusting the one or more values of the one or more machine learning models to match one or more values of the modified version of the one or more machine learning models. Feature 3 : Training the one or more machine learning models using the entropy ratio machine learning feature comprises: training one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios. Feature 4: Training the one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios comprises: increasing an error term or decreasing a reward term used to modify one or more values of the one or more search engine models; or decreasing an error term or increasing a reward term used to modify one or more values of the one or more search engine models. Feature 5: Generating the entropy ratio machine learning feature comprises: generating (i) a first value representing entropy of a first type of interaction included in the interaction data and (ii) a second value representing entropy of a second type of interaction included in the interaction data; and generating the entropy ratio machine learning feature using a ratio of the first value and the second value. Feature 6: Providing, to the one or more user devices, the graphical element representing the item on the exchange platform comprises: providing a search query to the one or more machine learningmodels; and providing, using output of the one or more machine learning models, the graphical element representing the item on the exchange platform.

[0006] In general, a second innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of receiving, for an initial set of graphical elements provided on a display in response to a query submitted to a graphical user interface (GUI) of an exchange platform, interaction data representing interactions between a user and the graphical elements of the GUI provided on the display; determining, using the interaction data, an entropy-based metric indicating a degree of randomness in types of interactions performed by users with the initial set of graphical elements; and in response to determining the entropy-based metric, automatically positioning a first graphical element of a subsequent set of graphical elements closer to a top portion of the GUI compared to a second graphical element of the subsequent set. Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0007] In general, a third innovative aspect of the subject matter described in this specification can be embodied in methods including the following operations: receiving, for a first set of products provided in response to a query submitted to an exchange platform, interaction data; determining an impression entropy based on the impression data for the first set of products, wherein the impression entropy indicates a degree of concentration of impressions among the first set of products; determining a purchase entropy based on the purchase data for the first set of products, wherein the purchase entropy indicates a degree of concentration of purchases among the first set of products; computing an entropy ratio for the first set of products based on the impression entropy and the purchase entropy; determining an imbalance level based on a comparison of the entropy ratio to a threshold; and based on the determined imbalance level, modifying attributes of the exchange platform to compensate for the determined imbalance level.

[0008] These and other embodiments can each optionally include one or more of the following features.

[0009] In some implementations, modifying the attributes of the exchange platform includes modifying a search algorithm to surface a second set of products provided on the exchangeplatform in response to the query, wherein the second set of products is different from the first set of products.

[0010] In some implementations, the imbalance level of the second set of products can be lower than the imbalance level of the first set of products.

[0011] In some implementations, updating the first set of products to a second set of products can include replacing one or more products in the first set of products with one or more new products, wherein a corresponding entropy ratio of the one or more new products is lower than a pre-determined threshold.

[0012] In some implementations, the methods can include computing a conversion rate of the first set of products and a new conversion rate of the second set of products, where the new conversion rate is greater than the conversion rate.

[0013] In some implementations, determining the purchase entropy based on the purchase data for the first set of products can include computing a purchase entropy contribution for each product in the first set of products, including multiplying a corresponding share of the purchases of the product and a base-2 logarithm of the corresponding share of the purchases of the product; and aggregating the purchase entropy contributions computed for the first set of products, to obtain the purchase entropy.

[0014] In some implementations, determining the impression entropy based on the impression data for the first set of products can include computing an impression entropy contribution for each product in the first set of products, including multiplying a corresponding share of the impression of the product and a base-2 logarithm of the corresponding share of the impression of the product; and aggregating the impression entropy contributions computed for the first set of products, to obtain the impression entropy.

[0015] In some implementations, computing the entropy ratio can include dividing the purchase entropy by the impression entropy.

[0016] In some implementations, methods can include providing for display, at a first plurality of client devices and in response to the query submitted at a first time interval, the first set of products; and after modifying the attributes of the exchange platform, providing for display, at a second plurality of client devices and in response to the query submitted at a second time interval, the second set of products.

[0017] In some implementations, methods can include receiving, from the first plurality of client devices, the impression data and the purchase data obtained within a predetermined time window.

[0018] In some implementations, modifying the attributes of the exchange platform can include generating an instruction based on a machine learning algorithm, the instruction comprising an actionable item on the first set of products; and sending one or more providers of the exchange platform the generated instruction.

[0019] Other embodiments of the above-described aspects can include corresponding systems, devices, apparatuses, and computer programs configured to perform the actions of the methods. The computer programs (e.g., instructions) can be encoded on computer storage devices.

[0020] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages. For example, the innovations described in this specification can reduce memory and processing requirements of machine learning models — e.g., by using an entropy-based metric as a data feature as described in this document. The entropy-based metric can compress a set of two or more different features into a single feature. By using the single feature rather than the two or more different features, one or more processors training or utilizing a machine learning model can reduce memory or processing operations, as described herein.

[0021] The innovations described in this specification can utilize entropy-based techniques to improve a graphical user interface (GUI) of an exchange platform. In some cases, these improvements can include detecting imbalances on an exchange platform. Techniques can include receiving, using a GUI, user interactions. These interactions can be used to organize listings provided in the GUI, where the listings can include graphical interface elements such as clickable icons, product information, or the like. In some cases, interactions can be used to generate specific data that indicates purchase entropy where purchase entropy can represent if a type of item or a category of item corresponds to more or less randomness in purchase interactions. One or more processors can be used to generate the specific data using various input, such as the interactions performed by a user of the GUI. In response to a request of a user using the GUI, the one or more processors can automatically adjust the position of listings displayed in a query output of the GUI to highlight or make certain listings more easilyaccessible compared to others. More easily accessible can include promoting a first listing from number 20 in a list of results to number 1 thereby saving a user from scrolling or paging through the 19 other results.

[0022] Other improvements can include the automatic arrangement or inclusion of specific filters for filtering a results page based on data generated using interaction data. For example, using a search query combined with interaction data — e.g., of a current user or other users of a system — one or more processors can automatically display icons in a window of the GUI that allow for specific filtering by the current user. For example, for a given query, techniques described can include determining one or more relevant filters, such as size, color, material. Determined relevant filters can be displayed on a GUI for interaction by a user interface. Relevant filters can be determined, e.g., using a model trained on historical user data indicating filters manually navigated to and activated by a user for similar queries. In some cases, a current query can be provided to a machine learning model that is trained with training data that indicates filters activated by a user for similar queries — e.g., filters that are manually navigated to by a user or filters that are determined by the model and provided for interaction on a GUI. In some cases, a degree of similarity between a current query and one or more historical queries can be used to determine a degree of similarity between filters historically activated and filters automatically activated or positioned for the current query. Specific categories of item can have specific filter associations, e.g., for a query indicating a search for a t-shirt, relevant filters can include size, color, or material. For a jewelry related query, relevant filters can include material, purity, or diameter. For furniture, relevant filters can include dimensions, color, or material. Relevant filters can be determined using machine learning output, by static values, or by model output biased using static values — e.g., where model output is anchored to predetermined filters based on user knowledge.

[0023] In some cases, techniques can enable modification of listings of goods, content, and / or services on an exchange platform to mitigate imbalances imbalance between purchase activity (e g., actions users take to purchase items / services on the platform) and non-purchase activity (e.g., non-purchase actions or interactions with listings of items / services on the platform, including clicks on listings, searches for particular items, etc.). Imbalance can inform the fitness or health of the marketplace by indicating a positive or negative correlation between demand and supply of items / services on the platform.

[0024] Conventional techniques may detect imbalance by utilizing general performance metrics, e.g., that focus on whether the goods, content, and / or services are popular (e.g., receiving a lot of browsing activity) or the conversion rate of the goods, content, and / or services (e.g., rate at which user actually take positive actions with respect to listed items / services / content, including purchasing or subscribing to a particular item / service / content). However, in some instances, such performance metrics do not reflect imbalance. For example, higher conversion rate may be considered as reflecting good marketplace health, but may not accurately portray that the purchase activity is concentrated on a single item or a small portion of products — with many other products not being correlated with the users’ preference or demand.

[0025] In contrast, the entropy-based techniques described herein provide additional insight and complement the general performance metrics (e.g., conversion rate) by capturing the correlation between the degree of sales concentration on the exchange platform relative to user interaction (non-purchase) activity. The entropy-based techniques can be used to preemptively notify users of predicted shortages in material used to create products of a given user — e.g., allowing a given user to increase production prior to an increase of purchase requests.

[0026] For example, unlike conventional methods that utilize the conversion rate (or another performance metric) or rely on computer modeling of users’ preferences at the exchange platform level, the techniques described herein enable computation of an entropy ratio that measures the purchase entropy relative to the impression entropy of a set of goods, content, and / or services on the exchange platform. These techniques provide relatively more refined and complementary information about the marketplace’s fitness that conventional methods are silent or insensitive to. Not only can the entropy ratio identify instances of high conversion rates, it can also capture different degrees of sales concentration information, e g., whether the conversions are concentrated on a small number of products in a set of products. The entropyratio based analysis can complement traditional metrics like total number of sales and conversion rate to achieve more sensitive and accurate detection of marketplace fitness / imbalance, relative to conventional metrics, such as conversion rate.

[0027] The entropy-based techniques described herein can be implemented as a stand-alone solution on the exchange platform for imbalance detection and / or mitigation. Alternatively, the techniques described herein can be deployed in combination with existing marketplaceevaluation solutions. In this manner, the techniques described herein can provide more robust and rich information about the marketplace fitness that is otherwise not discernable from conventional performance metrics alone.

[0028] Moreover, the entropy-based techniques described herein are computationally efficient compared to conventional techniques for evaluating the marketplace and / or activity on an ecommerce platform. For example, some conventional techniques utilize computer models for evaluating marketplace conditions and activities. In contrast, the entropy-based techniques described herein are more computationally efficient and less computational complex than conventional models. For example, models can be trained using an entropy ratio that can represent a randomness of whether or not a given item or category of item will be associated with purchase interactions. The ratio metric can be used as a feature in one or more models for performing actions, e.g., in a GUI for an exchange platform. A model of the one or more models can be trained to rank results or identify results based on a query or other input, such as user search history, purchases or other stored interaction data.

[0029] By using an entropy-based metric as described in this document, data and processing can be saved in both the training and operation of these models compared to using other metrics or features to capture similar data. For example, an entropy-based metric can incorporate — e.g., using a summation of data, such as a summation compared to another summation in a ratio form — constituent input elements that, if stored and used for training or operation of a model separately, would increase storage and processing requirements of a system operating the models. To give one example, each value summed in an entropy-based metric — representing an interaction of impression or purchasing by a user — could be stored in 28 bytes of memory, where memory could include a value representing an interaction and an identifier of the value. To consider one case, a system could track 100 of these values resulting in 2.8 kilobytes of memory for just a single element of training data or runtime value. Over an entire system operating an exchange platform, such memory requirements could require thousands of terabytes in additional storage in addition to increased computation time in retrieving and processing this data. In contrast, the proposed techniques generate a combined metric representing entropy that can represented in a fraction of the bytes required for constituent elements, e.g., 4 bytes representing an integer value of an entropy-based metric. In some cases, techniques include aggregating individual interaction values into a single entropy -basedmetric. By aggregating individual interaction values into a single entropy-based metric, memory footprint can be reduced from 28 bytes per interaction to just 4 bytes for the aggregated metric — i.e., a reduction of over 85% in memory usage for each interaction.

[0030] The entropy -based techniques described herein can be scalable to provide fitness evaluation on an exchange platform with a large set of items without adding significant computational costs. Due to the flexibility of the computed metrics, complex trend analysis can be performed by daily computation of the entropy -based ratio without adding any extra computational cost. Additionally, given that components of the described techniques are based on metrics and statistics already tracked and measured by the exchange platform, the accessibility and subsequent computations described herein are minimal and can be made efficient by limiting bandwidth or resource consumption for the additional tasks in the data pipeline.

[0031] In addition to detecting the imbalance, the techniques described herein can be implemented to configure the exchange platform to execute one or more actionable items to improve or mitigate the imbalance. For example, the exchange platform can be configured to send the actionable item(s) on listings of products to provider(s) associated with the exchange platform for mitigating the imbalance — e.g., instructions configured to increase output of one or more machines configured to produce items for exchange on an exchange platform. As such, the techniques described herein enable provision of more responsive or relevant listing of goods, contents, and / or services based on users’ preferences, while enabling dynamic updates to the marketplace offerings that mitigate imbalance.

[0032] As noted below, the techniques described herein are applicable in any environment, and are not limited to an exchange platform environment. The entropy-based techniques described herein, however, enable an exchange service provider to proactively manage supply and demand on its exchange platform, despite not directly providing goods / content / services (and instead facilitating third parties to provide such goods / content / services on the platform). A traditional electronic platform — where the provider of the platform also supplies the goods / content / services — has a better view of the supply and demand issues developing on the platform. However, an exchange platform that does not directly provide goods / content / services (and instead facilitates third parties to provide such goods / content / services on the platform) has limited visibility and ability to control the ensuingimbalance issues developing on the platform. The entropy-based imbalance detection and mitigation techniques described herein enable such exchange platform providers to have increased visibility of imbalance issues developing on the platform and provides additional tools to enable proactive and automatic mitigation of such imbalance.

[0033] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG. l is a block diagram of an example environment in which an exchange platform facilitates an exchange of items between providers and users.

[0035] FIG. 2 is a block diagram that illustrates an example entropy -based method for detection and mitigation of imbalance on the exchange platform.

[0036] FIG. 3 are tables that illustrate example entropy ratios obtained using the entropy-based detection and mitigation methods.

[0037] FIG. 4 is a block diagram that illustrates example relationships of different values of the entropy ratio to levels of imbalance.

[0038] FIG. 5 is a flow diagram of an example process for machine learning model training using an entropy ratio machine learning feature.

[0039] FIG. 6 is a flow diagram of an example process for graphical user interface modifications using entropy-based metrics.

[0040] FIG. 7 is a flow diagram of an example process for detecting and mitigating imbalance using the entropy ratio.

[0041] FIG. 8 is a block diagram of computing devices that may be used to implement the systems and methods described in this disclosure.

[0042] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0043] This disclosure relates to computer-implemented methods and systems that facilitate detecting and mitigating imbalance on an exchange platform. The techniques and methods described in this specification are explained with reference to an example production environment of an E-commerce or exchange platform / website that provides a platform for providers or sellers who use the platform as a tool to provide or to sell the items and for buyers or users who use the platform to search for and purchase items on the platform. However, one skilled in the art will appreciate that the techniques described in this specification are applicable in any number of applications and systems (e.g., search applications, systems for recommending content, services, or items for provision to users, etc.). For brevity and ease of explanation, the following descriptions apply the techniques described in this specification with reference to an example exchange platform.

[0044] Providers (e.g., sellers) provide items (e.g., goods, services, and content) that users (e g., buyers) of the exchange platform (e.g., registered members of the platform or guests of the platform) can obtain (e.g., interact with, purchase, etc.). Some E-commerce platforms provide a website with search functionality that enables a user to input and execute a search query and retrieve one or more web pages that lists a set of items matching the search query. Each item on the web page(s) (and in general) has a set of attributes, which can include, e.g., common attributes such as color, size, shape, material, etc. or / and customized attributes such as craft type, occasion, wedding theme, etc. Users of the platform may interact with these items, for example, browse these items without further actions, click one or more items to get more information related to these items, add one or more items to a shopping cart, or purchase one or more items.

[0045] As described further in this specification, based on the user’s interactions with one or more items, the exchange platform can collect and store user interaction data with respect to a set of items displayed to the users. The interaction data can include, e.g., non-purchase impression data, which indicates data relating to user interactions with items (e.g., a number of total interactions users had with each item, types of interactions (clicks, viewing time, adding to cart, etc.), and purchase data, which indicates, e.g., purchase information for each item (e.g., the number of total purchases of the item, purchase quantity, purchase amount, etc.).

[0046] The impression data and purchase data can then be used for deriving an impression entropy and a purchase entropy, respectively. Purchase entropy represents a degree ofconcentration of purchases among a set of items, and impression entropy represents a degree of concentration of impressions with respect to a set of items. For example, a first set of items with a higher purchase entropy relative to a second set of items indicates that the first set of items includes more items that the users are interested in purchasing rather than the second set of items. As another example, a low impression entropy for a particular set of items indicates that the users’ interactions are focused on a small portion of items among the particular set of items.

[0047] The exchange platform can compute an entropy ratio using the impression entropy and the purchase entropy, e.g., by dividing the purchase entropy by the impression entropy. The entropy ratio can be specific to a set of items and the numerical value of the ratio can indicate a degree of market fitness with respect to the set of items. The market fitness level may indicate how well the set of items fit the preferences of the users. In other words, the market fitness level can indicate how well the selection of items correlates with the users’ browsing and purchasing of items. An imbalance is detected when the market fitness level indicates that the selection of items does not correlate with the users’ purchases and impressions.

[0048] To mitigate the detected imbalance level, the exchange platform can be configured to execute one or more actionable items or instruct the providers to execute the actionable items. For example, in response to a detected imbalance, the exchange platform can be configured to automatically update its search algorithm or ranking algorithm such that, in response to a user’s search query, a different set of items are being displayed to the users or a same set of items are sorted and displayed differently to the users.

[0049] These and additional features are described in more detail below.

[0050] FIG. 1 is a block diagram of an example environment 100 in which an exchange platform enables an exchange of items (such as goods, services, or content) between providers and users. The example environment 100 includes a network 104, such as a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof. The network 104 connects one or more user devices 102, one or more provider devices 106, and an exchange platform 110.

[0051] User device 102 and provider device 106 are electronic devices that are capable of requesting and receiving items over the network 104. Examples of such devices include,among others, personal computers, mobile communication devices, digital assistant devices, and other devices that can send and receive data over the network 104.

[0052] The exchange platform 110 can be a computing platform that is operated and maintained by an exchange service provider. The exchange platform 110 enables providers to list their items on the exchange platform 110 and enables users to obtain the items listed on the exchange platform 110. As shown in FIG. 1, the exchange platform 110 is depicted as a single block with various sub-blocks. The exchange platform 110 could be a single device or single set of devices, this specification contemplates that the exchange platform 110 could also be a group of devices, or even multiple different systems that communicate with each other to enable the exchange of goods, services, and / or content. In some implementations, the exchange service provider could be a provider of items, and in some such implementations, the provider device 106 and the exchange platform 110 can be integrated. Alternatively, the exchange service provider can be an entity different from the provider, such that the provider device 106 and the exchange platform 110 can be separate, as shown in FIG. 1.

[0053] A user of a user device 102 can use an application 102-A to communicate with the exchange platform 110 to, for example, view listings of items, search for items, obtain items, and / or perform other appropriate tasks related to the exchange platform 110 (e.g., inputting payment or other identifying credentials, managing orders for items, etc.). The application 102-A can transmit data to, and receive data from, the exchange platform 110 over the network 104. The application 102-A can be implemented as a native application developed for a particular platform or a particular device, a web browser that provides a web interface, or another appropriate type of application. The application 102-A can detect user interactions (e.g., user's touch, mouse clicks, etc.) with various interfaces that enable, for example, the user to view listings of items, search for items, obtain items, and / or perform other appropriate tasks related to the exchange platform 110. The application 102-A can store and communicate the detected user interaction to the exchange platform 110, e.g., via the network 104.

[0054] A provider uses an application 106-A executing on a provider device 106 to communicate with the exchange platform 110 to, for example, create or manage listings of items of provider on the exchange platform 110 and / or perform other appropriate tasks related to the exchange platform 110 (e.g., transfer an amount to the provider based on items obtained by users). The application 106-A can transmit data to, and receive data from, the exchangeplatform 110 over the network 104. The application 106-A can be implemented as a native application developed for a particular platform or a particular device, a web browser that provides a web interface, or another appropriate type of application. The application 106-A can present and detect user interactions (e.g., user’s touch, mouse clicks, etc.) with various interfaces that enable, for example, the provider to create and manage listings of the provider’s items on the exchange platform 110.

[0055] The exchange platform 110 includes one or more front-end servers 112 and one or more back-end servers 114. For ease of explanation, the following description assumes that the exchange platform 110 is implemented with multiple front-end servers 112 and multiple back- end servers 114.

[0056] The front-end servers 112 can transmit data to, and receive data from, user devices 102 and provider devices 106, over the network 104. For example, the front-end servers 112 can provide interfaces and / or data for presentation within the interfaces to applications 102-A and 106-A executing on user devices 102 and provider devices 106, respectively. The front-end servers 112 can also receive data specifying user interactions (e.g., data representing user clicks, time spent browsing different pages, keyboard entry, purchase, etc.) with the interfaces provided by the front-end servers 112 to user devices 102 and provider devices 106. The frontend servers 112 can update the interfaces, provide new interfaces, and / or update the data presented by the interfaces presented in applications 102-A and 106-A, respectively, based on user / provider interactions with user devices 102 and provider devices 106.

[0057] The front-end servers 112 can also communicate with the back-end servers 114. For example, the front-end servers 112 can identify data to be processed by the back-end servers 114 and communicate such data to the back-servers, e.g., data specifying information necessary to create listings requested by a provider 106, data specifying the quantity of a given item that a user of user device 102 is requesting to obtain, etc. The front-end servers 112 can also receive data from the back-end servers 114. For example, data for a particular user of a user device 102 or data for a particular provider of a provider device 106, and transmit the data to the appropriate user device 102 or provider device 106 over the network 104.

[0058] The back-end servers 114 can include multiple sub-components. In some implementations, and as shown in FIG. 1, the back-end server 114 can include the following sub -components: an item engine 116, a search engine 118, and an entropy -based engine 120.As used in this specification, the term “engine” refers to hardware, e.g., one or more data processing apparatuses, which execute software and / or programming instructions, to perform a set of tasks / operations. Although FIG. 1 depicts these three engines, the operations of these engines as described in this specification may be performed, wholly or in part, by one or more other engines. In other words, some implementations may include more than the three engines depicted in FIG. 1 to perform the operations described in this specification. Alternatively, some implementations may include fewer engines to perform the operations described in this specification. Further still, even if an implementation includes the same three engines depicted in FIG. 1, the operations performed by one of these engines, as described in this specification, may be performed by one or more of the other engines.

[0059] The item engine 116 manages the creation and modification of listings of items, e.g., as requested by a provider via application 106-A on a provider device 106. The item engine 116 can receive, from the front end-servers 112, data specifying a description of an item for a listing initiated by a provider. Based on this description, the item engine 116 can create the listing within the exchange platform 110. The description of the item can include, for example, one or more of a name for the item, a brief description of the item, a quantity of the item, an amount required to obtain the particular item, an amount required to deliver the item to a destination, a fulfillment time for the item to arrive at the destination, and one or more images of the item. The item engine 116 can use some or all of this information to create a listing for an item on the exchange platform 110. The item engine 116 can store the data for the listing, including the received information and user interactions with the listing for the item (e.g., clicks, add-to-favorite, add-to-cart, time spent viewing a particular listing, etc.), in an item data storage device 124. The item data storage device 124 can include one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., hard drive(s), flash memory, etc.).

[0060] The item engine 116 can also receive, from the front end-servers 112, data specifying features of an item listing that a provider 106 may want to modify. For example, provider 106, through application 106-A, may seek to modify one or more features of the provider’s item listed on the exchange platform 110. The modified features are communicated from the application 106-A to front-end server 112 over network 104. The item engine 116 in turn receives from the front end-servers 112, data specifying features of the item listing that theprovider 106 wants to modify. The features to be modified can include, for example, the quantity of available items and the monetary amount required to obtain the item. The item engine 116 can use the data about the modified features to modify the listing for the items on the exchange platform 110. The item engine 116 can then use the modified features to update the item’s features stored in the item data storage device 124.

[0061] The search engine 118 manages retrieval of listings of items, as requested by a user via application 102 -A on a user device 102. The search engine 118 can receive from the front-end servers 112, data specifying a user’s request to search for items (i.e., a search query), and generate listings of items that are responsive to the user’s request or query. If a user searches for an item or a type of item on the exchange platform, the user inputs a search query into a search bar of the exchange platform’s interface (e.g., website) displayed within application 102-A. The user’s search query is received by front-end servers 112, which in turn sends the search query to the search engine 118. The search engine 118 uses the data specified in the search query to identify the item listings stored in the item data storage device 124 in response to the search query.

[0062] The search engine 118 uses a search algorithm to identify the item listings. The search algorithm includes one or more parameters that can be modified or updated by the search engine 118 or by other engines of the exchange platform 110. Modifying one or more parameters of the search algorithm may result in a different search result even if the user’s search query and other conditions (e g., item listings stored in the item data storage device 124) remain unaltered.

[0063] The search algorithm may include a ranking algorithm and its parameters, and modifying the search algorithm can include modifying the proportion of results that explore new listings, the percentage of results are personalized, etc. The search engine 118 communicates the identified listing(s) to the front-end servers 112, which in turn provides a particular listing or a summary of listings for presentation on the application 102-A. If a summary of listings is presented to the user in application 102-A, the user can select a link for one listing from among the summary of listings. The user’s selection of the link is received by the front-end server 112, which interprets the user’s selection as a request for data about the particular listing. The front-end servers 112 request search engine 118 to provide data about the particular listing, which the search engine 118 obtains from the item data storage device124. The search engine 118 responds to the front-end servers 1 12 with the obtained data, which is then provided by the front-end servers 112 to the application 102-A in the form of a page showing a listing for the item. The page can be a content page in a native application, or a web page in a web browser.

[0064] The entropy-based engine 120 executes the computation of entropies (e.g., purchase and impression entropy) and entropy ratios. For such computation, the entropy-based engine 120 requests / receives data from one or more storage devices of the back-end servers 114, such as the item data storage device 124, the purchase data storage device 126, and / or the impression data storage device 128. The purchase data storage device 126 and the impression data storage device 128 can be included in a same storage device or be distinct storage devices. In some cases, the purchase data storage device 126 and the impression data storage device 128 are communicably connected. The impression data storage device 128 stores impression data indicating a number of total non-purchase interactions that the users of the ecommerce platform had with each item in the set of items. The purchase data storage device 126 stores purchase data for each item in the set of items. Other and / or additional information can be included in the impression data and purchase data. For example, a time-stamp corresponding to a time of the user’s interaction with a listed item, an identifier of the user’s device, e.g., an IP address of the user's device, an account information of the user associated with the exchange platform. In some embodiments, the purchase data and impression data may be preprocessed to eliminate possible interactions that are caused by user mistakes or network errors.

[0065] Each of the storage devices 124, 126, 128 includes one or more databases (or other appropriate data storage structures) stored in one or more non-transitory data storage media (e.g., hard drive(s), flash memory, etc.). In some implementations, the storage devices can be multiple devices as shown in FIG. 1 for storing different types of data. Alternatively, the data of the black-end server(s) 114 can be stored in a single storage device.

[0066] In some implementations, the entropy -based engine 120 passively receives data when a certain predetermined condition for receiving the data is met. For example, the entropybased engine can receive data in real-time or on a predefined frequency, e.g., every 48 hours, every week, or every two weeks, etc. The entropy-based engine may also actively request data from the storage devices, e.g., upon the occurrence of a certain condition, such as a request from an exchange service provider for an imbalance evaluation.

[0067] In some implementations, the entropy-based engine 120 can include computer algorithms for clustering impression data, purchase data, or other possible data received from the storage devices 126, 128. The clustering of data can be based on clustering of different queries. For example, the entropy-based engine 120 can include a machine learning algorithm that has been trained using a large number of queries to cluster or group subsets of queries together based on their similarities. Additionally, data corresponding to similar queries can be combined together by the entropy-based engine 120 to enable aggregation of the number of visits corresponding to such similar queries. In some implementations, the query clustering can be performed by using a pre-existing normalized grouping of queries, or by queries that follow segmentation mappings (e.g., category or frequency of query quantile bins from a preceding period such as from a previous year) used by other items (e.g., products, services), or by applying semantic clustering to the queries.

[0068] The entropy -based engine 120 calculates a purchase entropy and an impression entropy, which can be computed with respect to a set of items or products. In some implementations, each item / product in the set of items / products corresponds to an individual entropy contribution, and the individual entropy contributions of all items / products in the set of items / products are aggregated to generate the entropy corresponding to the set of items.

[0069] In some implementations, each entropy value is computed based on user interactions with the set of items, optionally within a time frame. For example, the individual purchase entropy contribution for a particular item is calculated based on the purchase interactions with the particular item in the past 2 weeks. The purchase entropy is then computed by aggregating the individual entropy contributions for all the items in the set. The set of items in these examples can be a combination of multiple goods, content, and / or services displayed on the exchange platform 110. For example, the set of items can include all the items in a search result when a user entered a search string of “ring.” As another example, the set of items includes items listed on the first page or any number of page(s) of the search result.

[0070] Using the purchase and impression entropy values, the entropy-based engine can determine an entropy ratio by dividing the purchase entropy by the impression entropy. Using the resulting entropy ratio value, the entropy-based engine 120 determines an imbalance level. In some implementations, the entropy-based engine determines an imbalance level by comparing the entropy ratio to a predetermined threshold. The predetermined threshold canbe, e.g., a static threshold that is defined by an exchange service provider or a dynamic threshold that can be based on, e.g., historical entropy ratios computed on the exchange platform 110 based on historical purchase and interact! on / impression data. As one skilled in the art will appreciate, there can be multiple different thresholds that are used to define different levels of imbalance.

[0071] In some implementations, when the imbalance level needs mitigation, e.g., the entropy ratio is lower than a predetermined threshold (e.g., is lower than a particular threshold in a single threshold system or, in a multi-threshold system, is lower than the threshold that indicates a high imbalance), the entropy -based engine 120 determines a mitigation plan, pursuant to which, it generates executable or actionable items for the exchange platform 110 or the providers 106 to act on. For example, the entropy-based engine 120 can sends executable or actionable items to other engines of the back-end server(s) to be executed for improving the imbalance level. In some implementations, in response to detecting high market place imbalance, the entropy -based engine 120 determines to put into a place a mitigation plan for adjusting search results, which then triggers the entropy-based engine 120 to send modification instructions (e.g., to modify certain search parameters) to the search engine 118 to modify and update the search algorithm, to enable generation and display of a modified set of items or products when a user enters the same or similar search string (e.g., “ring”) at a later time. One skilled in the art will appreciate that, in some implementations, any number of actionable / executable items can be determined and applied in response to determined imbalance levels (e.g., as determined per the above computations). In some implementations, new trending queries can be identified by detecting supply demand imbalance changes over time, and providers of items on the exchange platforms can be provided corresponding recommendations regarding certain inventory that such providers should focus upon in view of the supply imbalance changes.

[0072] In some implementations, the entropy-based engine 120 may additionally calculate a conversion rate of the set of products. In some implementations, the conversion rate is positively correlated with the entropy ratio. In other words, the conversion rate increases when the entropy ratio increases, or vice versa. In some implementations, the entropy -based engine can compute a conversion rate for the set of products and for additional products that may be used to substitute one or more products in the set of products (as described below).

[0073] Additional structural and operational details of the entropy-based engine 120 are described below with reference to FIG. 2.

[0074] FIG. 2 is a block diagram that illustrates an example entropy-based engine 120 for detecting and mitigating imbalance based on user interactions with a set of products displayed on an exchange platform, e.g., exchange platform 110 of FIG. 1. An online shopping session includes a sequence of interactions (e.g., search, click, view, add-to-cart, purchase, etc.) that the user performs while engaging with the exchange platform 110 in FIG. 1. The session ends when the user leaves the exchange platform 110 with a purchase or abandons the web session, e.g., after a predefined duration of inactivity (e.g., 30 minutes) or upon navigating to a different website.

[0075] As shown in FIG. 2, the entropy-based engine 120 communicates with storage devices 126 and 128 to obtain therefrom, data for performing the above-described entropy-based computations. The entropy-based engine 120 includes software and / or programming instructions that are executable by a data processing apparatus, which upon execution, perform a set of operations. The operations performed upon the execution of instructions associated with the entropy-based engine 120 are described below.

[0076] During an online shopping session, a user of a user device 102, while viewing a web interface for the exchange platform 110 within the application 102-A, may submit a search query (e.g., “ring”), and the exchange platform 110 in turn provides a first page 212 (the first page of multiple pages of search results) listing a set of items (e.g., a plurality of rings) that are responsive to the search query. The set of items can be all the search results responsive to the search query, or part of the search results in response to the search query (with additional search results available upon further navigation by the user). As disclosed herein, the set of items can be any number of items being displayed to the user on the exchange platform. For example, the set of items can be only the first page of search results, several pages of search results, or all the search results.

[0077] As shown in FIG. 2, the user interacts with one or more items on the first page 212. For example, user A clicks on two gemstone rings 212-A, 212-B, while user B adds a cartoon ring 212-C to the shopping cart for purchase. User A’s interactions are communicated to the exchange platform as non-purchase impression data and are stored in the impression data storage device 128. User B’s interaction with ring 212-C are communicated to the exchangeplatform as purchase data and are stored in purchase data storage device 126. The exchange platform 110 can determine if similar searches are performed, e.g., a different or same user performing a search with similar search queries. In response to determining a similar search is performed, the exchange platform 110 can obtain and store interactions, such as nonpurchase or purchase interactions, with an identifier indicating the particular search. The identifier can be used by the exchange platform 110 to provide adjustments to displayed listings or other graphical elements for similar or different searches. The interaction data can be used by one or more machine learning models to order and display listings of items. When other users use similar search strings and are provided these search results, their interactions with the set of products depicted in the search results, can be stored into the impression data and / or the purchase data storage devices. As described above, the entropy-based engine 120 can obtain the impression data and the purchase data for entropy computations.

[0078] In some implementations, the entropy-based engine 120 uses the purchase and impression data to compute the entropy ratio for a set of items / products. In some implementations as shown in FIG. 2, the entropy ratio is compared to one or more predetermined thresholds, and based on this comparison, the entropy -based engine 120 can determine a level of imbalance with respect to the set of items / products. For example, a single threshold (e.g., 0.7) can be defined such that if the computed ratio exceeds this threshold (e.g., 0.8), the entropy-based engine 120 can determine that an imbalance exists with respect to this set of items. As another example, multiple thresholds (e.g., 0.5, 1.5, and 1.75) can be defined that correspond to different imbalance thresholds (e g., high imbalance, medium imbalance, and low imbalance). In this example, if the computed entropy ratio (1.7) exceeds the threshold corresponding to medium imbalance but is less than the threshold corresponding to the low imbalance, the entropy-based engine 120 can determine that a medium imbalance exists with respect to this set of items.

[0079] As one skilled in the art will appreciate in view of these examples, imbalance is generally negatively correlated with entropy ratio. For example, a high entropy ratio generally indicates low imbalance (and instead indicates good balance of diversified inventory and access to the same), and a low entropy ratio indicates high imbalance.

[0080] Moreover, and as described above, the imbalance threshold can be static and predefined. Alternatively, the imbalance thresholds can be dynamic, and, e.g., can be determined based on a distribution of current and previously determined entropy ratios.

[0081] In some implementations, based on the determined imbalance, the entropy-based engine 120 determines an actionable or executable task and triggers performance of such a task. In some implementations, such actionable or executable tasks can be predefined and can correspond to the different levels of detected imbalance. For example, a certain set of tasks could be triggered upon detecting medium imbalance, while a different set of tasks could be triggered upon detecting high imbalance. As described above, the set of actionable or executable tasks include, e.g., adjustment to search algorithms to surface different items / products in searches, adjusting ranking of products based on user interactions with products, automatic notification to providers on the platform to provide additional goods / items, reports to providers indicating supply issues with ideas on how to rectify such issues. Actions can include providing computed entropy -based values as features for training or operating one or more machine learning models, such as one or more machine learning models for identifying relevant items for listing in a display in response to a search query or other graphical elements — e.g., filter settings or toggles.

[0082] In the example shown in FIG. 2, upon detecting that the imbalance level is high, the entropy-based engine 120 examines the products that are listed on the first page 212 and adjusts the products / items that are actually displayed on the first page. In some implementations, this includes the entropy -based engine 120 determining the items that did not receive any user interactions and substituting these two items from the first page 212 with two new items that have higher entropy ratios than the two removed items to generate a new page 213. The new page 213 is displayed to users when similar search strings are subsequently submitted by the users. In this manner, the imbalance detection techniques described herein can make real-time and dynamic adjustments to content / items presented to users of the platform and can proactively take steps to mitigate the imbalance as it develops on the platform. Examples of such actions include, but are not limited to, one or more of the following: (1) expanding results when supply of items is low and / or offering high entropy ratio listings on a first search results page provided in response to a search query, (2) using indicators of demand on the exchange platform to perform and orchestrate advertising or further promotions on the exchangeplatform or outside the exchange platform, or (3) systematic modification of search algorithms deployed on the exchange platform based on an analysis of changes in entropy ratio over a certain period (which in turn informs overall activity trend on the exchange platform).

[0083] FIG. 3 shows example tables depicting, for a set of products (represented by listing id), the impressions, purchases, shares of impressions, shares of purchases, purchase entropy (PE) contribution, and impression entropy (IE) contributions. As shown, each of tables 1, 2, and 3 provides a different set of data (which may be, e.g., from three different timeframes) for each of these quantities along with an aggregation that depicts the overall PE and IE contribution for the set of products.

[0084] In some implementations, an entropy calculation (for computing the purchase entropy or impression entropy) can be represented by the following equation:H(p) = - S”=i Pi log(Pi) (1)

[0085] where pi is the share of the zth product in the set of products, log() is the base-2 logarithm, and H() is the entropy. The set of products includes n products in total — where a set of products can be a set of one or more items that are displayed on a results page in response to a user query. When pt is the impression share of the zth product, then HQ is the impression entropy. When pt is the purchase share of the product, then H() is the purchase entropy. pdog(pi) represents individual entropy contribution of the corresponding product, while entropy H() is an aggregation of all the individual entropy contributions in the set of products.

[0086] In some implementations, the entropy ratio is calculated as the purchase entropy divided by the impression entropy. If the entropy ratio is 1, then both purchases and impressions are equally balanced or equally imbalanced. If the entropy ratio is larger than 1, purchase visits are more evenly distributed than non-purchase impression visits, and if the entropy ratio is smaller than 1 that impression visits are more evenly distributed than purchase visits.

[0087] The entropy ratio based on the data in Table 1 is 0.47 / 1.58 (total PE contribution / total IE contribution), which amounts to ~0.3 and indicates impression visits being more evenly distributed than purchase visits. The entropy ratio based on the data in Table 2 is 1 .57 / 1 .58 (total PE contribution / total IE contribution), which amounts to ~1 and indicates that the impression visits are evenly distributed with purchase visits. The entropy ratio based on thedata in Table 3 is 0.47 / 1 .25 (total PE contribution / total IE contribution), which amounts to ~0.4 and indicates that the impression visits are more evenly distributed than purchase visits.

[0088] In general, when entropy is close to zero there is concentration in purchasing or impressions on one single item (e.g., as p approaches 0, the term ptlog(pz) approaches zero which, because of the negative sign in H(p) results in low H(p) values). For a listing with no purchases or impressions, the share of H(p) is not calculated because log(0) is not defined. In a simple case of three listings, where only one listing has any impressions or purchases and the value of the impressions / purchases is one, H(p) for this set of listings would be expressed as — £3=ipi log(pi) = — l log(l) = 0. In general, the most interacted with, or purchased, item generates the lowest value H(p) value — e.g., a large negative value.

[0089] The following describes how values from the Table 1 in FIG. 3 can be generated from the equation (1) that expresses H(p). In Table 1, listings one and two have respectively nine and one purchases. Listing three has zero purchases and is therefore not included in the H(p) 9 generation. Listing one then has — shares of purchases for a p value of 0.9 and listing two has a — shares of purchases for a p value of 0.1. Applying the expression for H (p) results in the following, e.g., — £?=1ptlog(pj) = — (0.9 log2(0.9) + 0.1 log2(0.1)) = 0.47 in an example where base 2 is used — in other cases, other log bases can be used.

[0090] Table 2 has the highest entropy ratio among all three different sets of impression data and purchase data. The highest entropy ratio corresponds to more evenly distributed sales concentration and non-purchase visits among products, thereby indicating a market fitness that is better than the scenarios in Table 1 and 3. Tables 1 and 3, in contrast, indicate imbalance with respect to the set of products given the lower entropy ratio values computed for the data in these tables. Note that for Table 1 with entropy ratio -0.3, there is one example of imbalance in which while impressions for each of the three listings is balanced (100 for each), the purchases are concentrated around the first listing. This indicates that when the 3 listings are shown as an assortment, there is only one listing that shows an item that may be desirable to user(s). For Table 3 with entropy ratio -0.4, the impressions and purchase visits are imbalanced by being skewed towards listing 1. In this case, the listing assortment would be considered suboptimal because it shows non-desirable listings to users, but also indicates a reduce competitive landscape that emphasizes a much more known provider (e.g., provider ofitem in listing 1) over two other listings (e.g., by other providers) that could also be desirable and which would allow users to optimize their choices. Table 2 in contrast shows a balanced assortment of similarly exposed providers who match the actual demand of users / purchases active on the exchange platform. This balance allows users / purchases to examine their options for searched items while also enabling similar providers to compete evenly and fairly on the exchange platform.

[0091] FIG. 4 is a block diagram that shows the relationship between the purchase entropy, the impression entropy, the entropy ratio, and market fitness / health. As shown in FIG. 4, the entropy ratio (ER) can be used to categorize the purchase and user interaction activity on the platform into four regions, including a very high entropy ratio region 410, a very low entropy ratio region 430, and two regions in between 420, 440. In some implementations, the categorization depends on the distributional form of entropy ratios, which identify a high entropy ratio as one above the median of entropy ratios along the distribution of entropy ratios. Other categories can be identified or determined by applying heuristics to the identified distribution (e.g., 75% and above can be categorized as very high, 25-50% can be categorized as Medium, and below 25% can be categorized as low).

[0092] As shown in FIG. 4, entropy ratio is very high when the purchase entropy is high and the impression entropy is low. This indicates that the platform provides a vast product assortment and attracts buyers with different interests / tastes. The very high entropy ratio also indicates that the search algorithms are identifying the correct listings for the queries and displaying them consistently. In this scenario, the entropy-based engine 120 may not prescribe any corrective or actionable tasks.

[0093] As shown in FIG. 4, entropy ratio is high when the purchase entropy is high and the impression entropy is high as well. This indicates that the platform provides a vast product assortment and attracts buyers with different interests / tastes. The high entropy ratio further indicates that the items may be low in stock or that the browsing behavior may be spread out across more items / listings. In scenario, the entropy-based engine 120 may prescribe a corrective or actionable task in the form of notifications to sellers to restock their items to support the demand.

[0094] As shown in FIG. 4, entropy ratio is medium when the purchase entropy is low and the impression entropy is low as well. This indicates that the platform provides a few relevantitems / li stings that are capturing most of the purchase activity. The medium entropy ratio further indicates that users of the platform are staying within the main search results and not venturing out to other items offered on the platform. In this scenario, the entropy-based engine 120 may prescribe a corrective or actionable task in the form of, e.g., more diversification of items in search and provided listings, which in turn can result in more diverse sales / provision of items on the exchange platform.

[0095] As shown in FIG. 4, entropy ratio is low when the purchase entropy is low and the impression entropy is high. This indicates that the platform provides numerous items / listings but sales are concentrated in a few of the items. In this scenario, the entropy-based engine 120 may prescribe a corrective or actionable task in the form of, e.g., notifying providers to provide more items that are similar to the one where sales are currently concentrated.

[0096] As illustrated by these scenarios, for different sets of products, the set with the highest entropy ratio indicates that the level of imbalance is the lowest, e.g., 410 in FIG. 4. With this set of products, purchases are not concentrated on only a small fraction of the products but distributed over at least a majority of the products in the set. The distribution of purchases among the set of products is more even than the distribution of non-purchase impressions by the user. The highest entropy ratio can indicate that the set of products have given users good choices and the set of products is highly relevant to users’ demand or preferences. The set with the lowest entropy ratio indicates that the level of imbalance is the highest, e.g., 430 in FIG. 4. The distribution of purchase visits are more concentrated, e g., on a small number of products than the distribution of non-purchase visits to the set of products. In this case, the set of products has not given users good enough choices to meet their demand or preferences.

[0097] When the entropy ratio is between the highest and lowest points, e.g., 420, 440, the set of products indicates some level of imbalance that can be improved by updating the items included in the set.

[0098] FIG. 5 is a flow diagram of an example process 500 for machine learning model training using an entropy ratio machine learning feature. For convenience, the process 500 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, an exchange platform, e.g., the exchange platform 110 of FIG. 1, appropriately programmed, can perform the process 500.

[0099] The process 500 includes providing, to one or more user devices, a graphical element representing an item on an exchange platform (502). For example, the exchange platform 110 can provide graphical elements that represents items on an exchange platform. Providing graphical elements can include providing a results page, such as the first page 212, as described in reference to FIG. 2.

[0100] The process 500 includes receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform (504). For example, the exchange platform 110 can receive interaction data, such as the interaction data included in the storage devices 126 and 128. The storage devices 126 and 128 can be included in a single storage device or multiple storage devices, e.g., a set of communicably connected storage devices.

[0101] The process 500 includes generating, using the interaction data, an entropy ratio machine learning feature (506). For example, the entropy-based engine 120 of the exchange platform 110 can generate an entropy ratio machine learning feature. The entropy ratio machine learning feature can include one or more entropy ratios for one or more items included in a set of items.

[0102] The process 500 includes training one or more machine learning models using the entropy ratio machine learning feature (508). For example, the exchange platform 110 can provide one or more values representing a search query to one or more machine learning models, e.g., a feature vector. The exchange platform 110 can provide, based on output of one or more machine learning models processing one or more values representing a search query, a set of one or more graphical elements to a set of user devices, e.g., the exchange platform 110 can provide the first page 212 that includes graphical elements or other elements, such as elements used to control an interface or results page — such as search filtering settings. The exchange platform 110 can provide one or more values representing a search query to a modified version of one or more machine learning models, e.g., “gold ring” or a feature vector representing a search query. The exchange platform 110 can provide, based on output of a modified version of one or more machine learning models processing one or more values representing a search query, a one or more graphical elements — e.g., a second set of graphical elements — to a set of user devices. The exchange platform 110 can receive (i) first interaction data based on a provided set of one or more graphical elements and (ii) second interaction databased on a provided second set of one or more graphical elements. The exchange platform 110 can adjust one or more values of one or more machine learning models using first interaction data and second interaction data — e.g., values, such as one or more values in connected layers, can be adjusted using backpropagation techniques or other methods.

[0103] FIG. 6 is a flow diagram of an example process 600 for graphical user interface modifications using entropy-based metrics. For convenience, the process 600 will be described as being performed by a system of one or more computers, located in one or more locations, and programmed appropriately in accordance with this specification. For example, an exchange platform, e.g., the exchange platform 110 of FIG. 1, appropriately programmed, can perform the process 600.

[0104] The process 600 includes receiving — e.g., for an initial set of graphical elements provided on a display in response to a query submitted to a graphical user interface (GUI) of an exchange platform — interaction data representing interactions between a user and the graphical elements of the GUI provided on the display (602). For example, the exchange platform 110 can receive, for an initial set of graphical elements provided on a display in response to a query submitted to a graphical user interface (GUI) of an exchange platform, interaction data. The interaction data can include impression or purchase data, e.g., that can be stored in databases 126 and 128. The interaction data can include a mouse movement, a touch or scroll on a touch screen, eye attentiveness, stopping scrolling, hovering mouse, among others.

[0105] The process 600 includes determining, using the interaction data, an entropy-based metric indicating a degree of randomness in types of interactions performed by users with the initial set of graphical elements (604). For example, the exchange platform 110 can determine entropy-based metrics for one or more sets of items — such a set of one or more items that are related by one or more search terms and that are provided by a search engine in response to the exchange platform 110 obtaining a search query.

[0106] The process 600 includes automatically positioning a first graphical element of a subsequent set of graphical elements closer to a top portion of the GUI compared to a second graphical element of the subsequent set (606). For example, the exchange platform 110 can automatically position one or more elements via the user application 102 -A, e.g., in response to determining the entropy-based metric. The subsequent set of graphical elements can begenerated and provided by the exchange platform 1 10 in response to a subsequent query or action taken by a user in the application 102-A.

[0107] In some cases, the exchange platform 110 can generate one or more instructions to control display of one or more graphical elements displayed on the user application 102-A — e.g., listing graphical elements representing items available on the exchange platform 110, elements indicating search preferences or settings, navigational elements, input elements, or a combination of these or others. In some cases, elements indicating search preferences or settings can be adjusted — e.g., to promote search preferences, toggles, or other elements that are relevant for a given search query. For example, for a set of items that have low a calculated low entropy ratio, one or more machine learning models can position search preferences that include additional search filters to help filter out the numerous listings that are more likely to cause non-purchase interactions by a user. One or more machine learning models can use an entropy ratio to help maximize the entropy ratio for a given set of items to be provided in response to a search query, where the set of items are also similar and related to one or more search criteria input by a user.

[0108] FIG. 7 is a flow diagram of an example process 700 for detecting and mitigating imbalance. Operations of the process 700 are described below and for purposes of illustration only, these operations are described as being performed by the system depicted and described in FIG. 1. The operations of the process 700 can be performed by any appropriate device or system, e.g., any appropriate data processing apparatus (also referred to as “computing device” with reference to FIG. 8), which may include a device or system that is not part of the exchange platform (e.g., a proxy server through which all traffic to and from the exchange platform flows). Operations of the process 700 can also be implemented as instructions stored on a non- transitory computer-readable medium. Execution of the instructions causes one or more data processing apparatuses to perform operations of the process 700.

[0109] The process 700 includes receiving, for a first set of products provided in response to a query submitted to an exchange platform, impression data and purchase data (702). In some cases, the impression data indicates a number of non-purchase interactions with each product in the first set of products and the purchase data indicates a number of purchases for each product in the first set of products. For example, the exchange platform 110 can receive a query, e g., a search query, submitted by a user, in a first time interval. In response to thesearch query, a first set of products can be provided by the exchange platform to the user via the application. The first time interval can be predetermined or adjusted if needed. For example, it can be a week, two weeks, a month, or any number of days. The first set of products can be displayed in the first page 212 on a user device 102.

[0110] For the first set of products, the exchange platform 110, and more specifically, the entropy-based engine 120, can receive impression data and purchase data. As described with reference to FIG. 1-4, the purchase data and impression data can be obtained based on user interactions with the set of products, and can have a corresponding number of interactions by the user with each product in the first set. As an example, as shown in Table 1, the purchase data includes 9 purchases and 1 purchase of product 1 and product 2, respectively. A total number of purchases for the set of 3 products is 10.[0U1] The process 700 includes determining an impression entropy based on the impression data for the first set of products, wherein the impression entropy indicates a degree of concentration of impressions among the first set of products (704). For example, the entropybased engine 120 can determine an impression entropy of the first set of products based on the impression data. The impression entropy can indicate a degree of concentration of impressions among the first set of products. In some implementations, the impression entropy can be determined using equation (1), described above. The entropy -based engine 120 can compute an individual impression entropy contribution for each product in the first set of products by multiplying a corresponding share of the impressions (e g., non-purchase visits) of the product and a base-2 logarithm of the corresponding share of the impressions of the product (as described herein). The entropy-based engine 120 can then aggregate the individual impression entropy contributions computed for the set of products, to obtain the aggregate impression entropy (simply referred to as impression entropy). For example, as shown in Table 1, the share of impressions for each product is 0.33 and the individual impression entropy for each product determined is 0.53, which makes the impression entropy of the entire set of products 1.58.

[0112] The process 700 includes determining a purchase entropy based on the purchase data for the first set of products, wherein the purchase entropy indicates a degree of concentration of purchases among the first set of products (706). For example, the entropy-based engine 120 can determine a purchase entropy based on the purchase data for the first set of products. Thepurchase entropy can indicate a degree of concentration of purchases among the first set of products. In some implementations, purchase entropy is determined using equation (1). The entropy -based engine 120 can compute an individual purchase entropy contribution for each product in the first set of products by multiplying a corresponding share of the purchases of the product and a base-2 logarithm of the corresponding share of the purchases of the product, as described herein, e.g., with reference to FIGS. 1-4. The entropy-based engine 120 can then aggregate the individual purchase entropy contributions computed for the set of products, to obtain the aggregate purchase entropy (or simply the purchase entropy). For example, as shown in Tablel, the share of purchases for each product is 0.9, 0.1, and 0, respectively and the individual impression entropy for each product is 0.14, 0.33, and 0, respectively, which results in the purchase entropy of the entire set of products being 0.47.

[0113] The process 700 includes computing an entropy ratio for the first set of products based on the impression entropy and the purchase entropy (708). For example, the entropy-based engine 120 can compute an entropy ratio for the first set of products based on the impression entropy and the purchase entropy, e.g., from steps 704 and 706. In some implementations, the entropy ratio can be a ratio of the purchase entropy (as the numerator) to the impression entropy (as the denominator). For example, the entropy-based engine 120 can divide the purchase entropy by the impression entropy to obtain the entropy ratio. The entropy ratio can have a numerical value in the range of 0 to 1, or greater than 1.

[0114] The process 700 includes determining a market imbalance level based on a comparison of the entropy ratio to an entropy threshold (710). For example, the entropy -based engine 120 can determine an imbalance level based on a comparison of the entropy ratio to a threshold. As described above with reference to FIGS. 1-4, the threshold can be any predetermined value. For example, the entropy threshold is a historical entropy ratio on the exchange platform. As another example, the entropy threshold can be an average entropy ratio on the exchange platform over a pre-determined period, e.g., for the past 3 or 6 months. If the computed entropy ratio satisfies (e.g., meets or exceeds) the threshold, the entropy-based engine determines that a particular imbalance level (as described with reference to FIGS. 3-4).

[0115] The process 700 includes based on the determined market imbalance level, modifying attributes of the exchange platform to compensate for the determined market imbalance level (712). For example, the exchange platform 110 or entropy-based engine 120 can modifyattributes of the exchange platform 1 10 to compensate for the determined imbalance level based on the determined imbalance level. In some implementations, the entropy -based engine 120 generates and sends one or more instructions to the other engines of the exchange platform 110 to perform such compensation of imbalance (as described with reference to FIGS. 1-4).

[0116] For example, the exchange platform 110 sends instructions to the search engine 118 to modify a search algorithm to surface a different set of products provided on the exchange platform 110 when new queries are submitted by users. After modifying the attributes, the exchange platform can provide the different set of products for display at a second plurality of client devices in response to a subsequently submitted query. The different set of products can include new products to replace product(s) that had an individual entropy ratio contribution that is lower than a threshold.

[0117] As another example, the exchange platform 110 sends instructions to the search engine 118 to modify the search algorithm to surface the same set of products but to display the same set of products in the first page 212 in a different arrangement (e.g., using a different ranking), with different product descriptions, in different orientations, etc.

[0118] In some implementations, the instruction(s) can be generated based on computer algorithms such as a machine learning algorithm. The machine learning algorithm can be trained on entropy ratio data, impression data, purchase data, and item data on the exchange platform 110. The machine learning algorithm can be trained to learn features of the item(s) that contribute positively to the corresponding entropy ratio. After training, the machine learning algorithm can make inferences on features that can be changed on items to improve the entropy ratio and by extension, take actions that result in better overall market fitness. The instructions can include one or more actionable items on the first set of products. The one or more actionable item can be acted on by the providers 106 of the exchange platform 110. The entropy-based engine 120 can send the generated instructions to the provider device 106 or the application 106-A. The instructions can instruct the provider to update one or more attributes of an item that is provided on the exchange platform. The instructions can instruct the provider to add or remove items and their corresponding attribute information. For example, the machine learning algorithm can, based on the entropy ratio, predict a category of products that are better correlated with the users’ preferences, and the actionable item for the provider can be to change the description of their items to be similar to items in the category. As anotherexample, the actionable item can be to provide more products in the category of “ring” in the price range of $50-150. As yet another example, the actionable item can be to provide more images describing details of the product. As another example, if a particular item is in short supply on the exchange platform, the exchange platform’s search results can be modified to augment items sharing similarities in terms of query segment or belonging to a similar trend.

[0119] FIG. 8 is a diagram illustrating an example of a computing system used for semantic image adjustments. The computing system includes computing device 800 and a mobile computing device 850 that can be used to implement the techniques described herein. For example, one or more components of the environment 100 could be an example of the computing device 800 or the mobile computing device 850.

[0120] The computing device 800 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 850 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, mobile embedded radio systems, radio diagnostic computing devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

[0121] The computing device 800 includes a processor 802, a memory 804, a storage device 806, a high-speed interface 808 connecting to the memory 804 and multiple high-speed expansion ports 810, and a low-speed interface 812 connecting to a low-speed expansion port 814 and the storage device 806. Each of the processor 802, the memory 804, the storage device 806, the high-speed interface 808, the high-speed expansion ports 810, and the low-speed interface 812, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 802 can process instructions for execution within the computing device 800, including instructions stored in the memory 804 or on the storage device 806 to display graphical information for a GUI on an external input / output device, such as a display 816 coupled to the high-speed interface 808. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. In addition, multiple computing devices may be connected, with each device providing portions of the operations (e g., as a server bank, agroup of blade servers, or a multi-processor system). Tn some implementations, the processor 802 is a single threaded processor. In some implementations, the processor 802 is a multithreaded processor. In some implementations, the processor 802 is a quantum computer.

[0122] The memory 804 stores information within the computing device 800. In some implementations, the memory 804 is a volatile memory unit or units. In some implementations, the memory 804 is a non-volatile memory unit or units. The memory 804 may also be another form of computer-readable medium, such as a magnetic or optical disk.

[0123] The storage device 806 is capable of providing mass storage for the computing device 800. In some implementations, the storage device 806 may be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 802), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices such as computer- or machine readable mediums (for example, the memory 804, the storage device 806, or memory on the processor 802). The high-speed interface 808 manages bandwidth-intensive operations for the computing device 800, while the low-speed interface 812 manages lower bandwidthintensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface 808 is coupled to the memory 804, the display 816 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 810, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 812 is coupled to the storage device 806 and the low-speed expansion port 814. The low-speed expansion port 814, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0124] The computing device 800 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 820, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 822. It may also be implemented as part of a rack server system 824.Alternatively, components from the computing device 800 may be combined with other components in a mobile device, such as a mobile computing device 850. Each of such devices may include one or more of the computing device 800 and the mobile computing device 850, and an entire system may be made up of multiple computing devices communicating with each other.

[0125] The mobile computing device 850 includes a processor 852, a memory 864, an input / output device such as a display 854, a communication interface 866, and a transceiver 868, among other components. The mobile computing device 850 may also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 852, the memory 864, the display 854, the communication interface 866, and the transceiver 868, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

[0126] The processor 852 can execute instructions within the mobile computing device 850, including instructions stored in the memory 864. The processor 852 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 852 may provide, for example, for coordination of the other components of the mobile computing device 850, such as control of user interfaces, applications run by the mobile computing device 850, and wireless communication by the mobile computing device 850.

[0127] The processor 852 may communicate with a user through a control interface 858 and a display interface 856 coupled to the display 854. The display 854 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 856 may include appropriate circuitry for driving the display 854 to present graphical and other information to a user. The control interface 858 may receive commands from a user and convert them for submission to the processor 852. In addition, an external interface 862 may provide communication with the processor 852, so as to enable near area communication of the mobile computing device 850 with other devices. The external interface 862 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

[0128] The memory 864 stores information within the mobile computing device 850. The memory 864 can be implemented as one or more of a computer-readable medium or media, avolatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 874 may also be provided and connected to the mobile computing device 850 through an expansion interface 872, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 874 may provide extra storage space for the mobile computing device 850, or may also store applications or other information for the mobile computing device 850. Specifically, the expansion memory 874 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 874 may be provide as a security module for the mobile computing device 850, and may be programmed with instructions that permit secure use of the mobile computing device 850. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0129] The memory may include, for example, flash memory and / or NVRAM memory (nonvolatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier such that the instructions, when executed by one or more processing devices (for example, processor 852), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory 864, the expansion memory 874, or memory on the processor 852). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 868 or the external interface 862.

[0130] The mobile computing device 850 may communicate wirelessly through the communication interface 866, which may include digital signal processing circuitry in some cases. The communication interface 866 may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), LTE, 8G / 6G cellular, among others. Such communication may occur, for example, through the transceiver 868 using a radio frequency. In addition, short-range communication may occur,such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 870 may provide additional navigation- and location-related wireless data to the mobile computing device 850, which may be used as appropriate by applications running on the mobile computing device 850.

[0131] The mobile computing device 850 may also communicate audibly using an audio codec 860, which may receive spoken information from a user and convert it to usable digital information. The audio codec 860 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 850. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, among others) and may also include sound generated by applications operating on the mobile computing device 850.

[0132] The mobile computing device 850 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 880. It may also be implemented as part of a smart-phone 882, personal digital assistant, or other similar mobile device.

[0133] In general, use of “or” can refer to “and / or.” When providing a list of two or more items, the conjunction “or” can indicate any one of the items, any combination of a subset of the items, or all items in combination.

[0134] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0135] The subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter and the actions and operations described in this specification can be implemented as or in one or more computer programs, e.g., one or more modules of computer program instructions, encoded on a computer program carrier, for execution by, or to control the operation of, dataprocessing apparatus. The carrier can be a tangible non-transitory computer storage medium. Alternatively or in addition, the carrier can be an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be or be part of a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. A computer storage medium is not a propagated signal.

[0136] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. Data processing apparatus can include specialpurpose logic circuitry, e.g., an FPGA (field programmable gate array), an ASIC (applicationspecific integrated circuit), or a GPU (graphics processing unit). The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0137] A computer program can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program, e.g., as an app, or as a module, component, engine, subroutine, or other unit suitable for executing in a computing environment, which environment may include one or more computers interconnected by a data communication network in one or more locations.

[0138] A computer program may, but need not, correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code.

[0139] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or by a combination of special-purpose logic circuitry and one or more programmed computers.

[0140] Computers suitable for the execution of a computer program can be based on general or special-purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special-purpose logic circuitry.

[0141] Generally, a computer will also include, or be operatively coupled to, one or more mass storage devices, and be configured to receive data from or transfer data to the mass storage devices. The mass storage devices can be, for example, magnetic, magneto-optical, or optical disks, or solid state drives. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0142] To provide for interaction with a user, the subject matter described in this specification can be implemented on one or more computers having, or configured to communicate with, a display device, e.g., a LCD (liquid crystal display) monitor, or a virtual -reality (VR) or augmented-reality (AR) display, for displaying information to the user, and an input device by which the user can provide input to the computer, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball or touchpad. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback and responses provided to the user can be any form of sensory feedback, e.g., visual, auditory, speech, or tactile feedback or responses; and input from the user can be received in any form, including acoustic, speech, tactile, or eye tracking input, including touch motion or gestures, or kinetic motion or gestures or orientation motion or gestures. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smartphone or electronic tablet. Also, a computer can interact with a user by sending text messages or other forms of messageto a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0143] This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs the operations or actions.

[0144] The subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0145] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0146] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is defined by the claims themselves, but rather as descriptions of features that may be specific to particularembodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claim may be directed to a subcombination or variation of a subcombination.

[0147] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this by itself should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0148] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0149] What is claimed is:

Claims

CLAIMS1. A method comprising: providing, to one or more user devices, a graphical element representing an item on an exchange platform; receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform; generating, using the interaction data, an entropy ratio machine learning feature; and training one or more machine learning models using the entropy ratio machine learning feature.

2. The method of claim 1, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: providing one or more values representing a search query to the one or more machine learning models; providing, based on output of the one or more machine learning models processing the one or more values representing the search query, a set of one or more graphical elements to a set of user devices; providing the one or more values representing the search query to a modified version of the one or more machine learning models; providing, based on output of the modified version of the one or more machine learning models processing the one or more values representing the search query, a second set of one or more graphical elements to the set of user devices; receiving (i) first interaction data based on the provided set of one or more graphical elements and (ii) second interaction data based on the provided second set of one or more graphical elements; and adjusting one or more values of the one or more machine learning models using the first interaction data and the second interaction data.

3. The method of claim 2, wherein adjusting the one or more values of the one or more machine learning models using the first interaction data and the second interaction data comprises: adjusting the one or more values of the one or more machine learning models to match one or more values of the modified version of the one or more machine learning models.

4. The method of claim 1, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: training one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios.

5. The method of claim 4, wherein training the one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios comprises: increasing an error term or decreasing a reward term used to modify one or more values of the one or more search engine models; or decreasing an error term or increasing a reward term used to modify one or more values of the one or more search engine models.

6. The method of claim 1, wherein generating the entropy ratio machine learning feature comprises: generating (i) a first value representing entropy of a first type of interaction included in the interaction data and (ii) a second value representing entropy of a second type of interaction included in the interaction data; and generating the entropy ratio machine learning feature using a ratio of the first value and the second value.

7. The method of claim 1, wherein providing, to the one or more user devices, the graphical element representing the item on the exchange platform comprises: providing a search query to the one or more machine learning models; andproviding, using output of the one or more machine learning models, the graphical element representing the item on the exchange platform.

8. One or more computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: providing, to one or more user devices, a graphical element representing an item on an exchange platform; receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform; generating, using the interaction data, an entropy ratio machine learning feature; and training one or more machine learning models using the entropy ratio machine learning feature.

9. The storage media of claim 8, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: providing one or more values representing a search query to the one or more machine learning models; providing, based on output of the one or more machine learning models processing the one or more values representing the search query, a set of one or more graphical elements to a set of user devices; providing the one or more values representing the search query to a modified version of the one or more machine learning models; providing, based on output of the modified version of the one or more machine learning models processing the one or more values representing the search query, a second set of one or more graphical elements to the set of user devices; receiving (i) first interaction data based on the provided set of one or more graphical elements and (ii) second interaction data based on the provided second set of one or more graphical elements; and adjusting one or more values of the one or more machine learning models using the first interaction data and the second interaction data.

10. The storage media of claim 9, wherein adjusting the one or more values of the one or more machine learning models using the first interaction data and the second interaction data comprises: adjusting the one or more values of the one or more machine learning models to match one or more values of the modified version of the one or more machine learning models.

11. The storage media of claim 8, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: training one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios.

12. The storage media of claim 11, wherein training the one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios comprises: increasing an error term or decreasing a reward term used to modify one or more values of the one or more search engine models; or decreasing an error term or increasing a reward term used to modify one or more values of the one or more search engine models.

13. The storage media of claim 8, wherein generating the entropy ratio machine learning feature comprises: generating (i) a first value representing entropy of a first type of interaction included in the interaction data and (ii) a second value representing entropy of a second type of interaction included in the interaction data; and generating the entropy ratio machine learning feature using a ratio of the first value and the second value.

14. The storage media of claim 8, wherein providing, to the one or more user devices, the graphical element representing the item on the exchange platform comprises:providing a search query to the one or more machine learning models; and providing, using output of the one or more machine learning models, the graphical element representing the item on the exchange platform.

15. A system comprising: one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: providing, to one or more user devices, a graphical element representing an item on an exchange platform; receiving interaction data representing two or more users interacting with the graphical element representing the item on the exchange platform; generating, using the interaction data, an entropy ratio machine learning feature; and training one or more machine learning models using the entropy ratio machine learning feature.

16. The system of claim 15, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: providing one or more values representing a search query to the one or more machine learning models; providing, based on output of the one or more machine learning models processing the one or more values representing the search query, a set of one or more graphical elements to a set of user devices; providing the one or more values representing the search query to a modified version of the one or more machine learning models; providing, based on output of the modified version of the one or more machine learning models processing the one or more values representing the search query, a second set of one or more graphical elements to the set of user devices; receiving (i) first interaction data based on the provided set of one or more graphical elements and (ii) second interaction data based on the provided second set of one or more graphical elements; andadjusting one or more values of the one or more machine learning models using the first interaction data and the second interaction data.

17. The system of claim 16, wherein adjusting the one or more values of the one or more machine learning models using the first interaction data and the second interaction data comprises: adjusting the one or more values of the one or more machine learning models to match one or more values of the modified version of the one or more machine learning models.

18. The system of claim 15, wherein training the one or more machine learning models using the entropy ratio machine learning feature comprises: training one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios.

19. The system of claim 18, wherein training the one or more search engine models to increase a likelihood of search output sorted according to item-based entropy ratios comprises: increasing an error term or decreasing a reward term used to modify one or more values of the one or more search engine models; or decreasing an error term or increasing a reward term used to modify one or more values of the one or more search engine models.

20. The system of claim 15, wherein generating the entropy ratio machine learning feature comprises: generating (i) a first value representing entropy of a first type of interaction included in the interaction data and (ii) a second value representing entropy of a second type of interaction included in the interaction data; and generating the entropy ratio machine learning feature using a ratio of the first value and the second value.

Images