User Search Category Predictor

Abstract

Described herein are embodiments for improving search engine results of listings of items for sale (FSOs). Search engines can be improved by implementing rules that disambiguate between listings for different FSOs that match the same search input. An unsupervised machine learning module can evaluate candidate rules and identify improvements that may not be obvious to a human evaluator. E-commerce sites that combine the improved search engine with the unsupervised machine learning module can dynamically evaluate search results using different candidate rules and iteratively improve the search results.

Description

Technical Field

[0001] Technical Field

[0001] This disclosure generally relates to testing rules for improving search results and incorporating them into a search method.

Background Art

[0002] Background

[0002] E-commerce websites and applications provide buyers with means for purchasing various goods. However, the search for these goods can often introduce ambiguity into the search results. A buyer may attempt to minimize their input or search the list of free sales offers (FSOs) in a way that ambiguously conveys their intent. Multiple different FSOs may have features or names that match similar search terms. The search results may be inundated with a list of useless FSOs or may even exclude the list of FSOs that the buyer is looking for on the e-commerce site.

Summary of the Invention

Means for Solving the Problems

[0003] Summary

[0003] Provided herein are embodiments and / or combinations and sub-combinations thereof of a system, apparatus, product, method, and / or computer program product for improving the search engine results of an e-commerce site by testing rules for reducing the ambiguity of search inputs, identifying which rules are effective, and implementing those rules on the e-commerce site.

[0004]

[0004] Some embodiments operate by providing a baseline search result to a control group of buyers based on a search input and current rules, providing a filtered search result to a test group of buyers based on the search input, current rules, and a candidate rule corresponding to a particular test group, receiving a control response from the control group of buyers and a test response from the test group of buyers, determining, for each test group, a metric for the test group based on the control response and the test response, discarding a candidate rule corresponding to the test group in response to the metric being statistically significant and less than a threshold, and adding a candidate rule corresponding to the test group to the current rules in response to the metric being statistically significant and greater than a threshold.

[0005]

[0005] Some embodiments include receiving a search input from a buyer and assigning each buyer to one of a plurality of groups, including a control group and a test group, where each test group corresponds to one candidate rule; identifying search results from a plurality of FSO lists based on the search input; filtering the search results based on current rules to identify a first filtered search result; for each test group, filtering the search results based on the current rules and the corresponding candidate rule for the test group to identify a filtered search result for the test group; providing the first filtered search result to the control group; for each test group, providing the filtered search result for the test group; receiving a response metric from the buyer; determining a performance metric for each test group based on the response metric; determining the statistical significance for each test group based on the performance metric; for each test group, in response to the statistical significance for the test group being greater than a threshold and in response to the performance metric for the test group being less than a metric threshold, discarding the candidate rule corresponding to the test group; and in response to the performance metric for the test group being greater than the metric threshold, adding the candidate rule corresponding to the test group to the current rules.

[0006]

[0006] Further embodiments, features, and advantages of the present disclosure, as well as the structure and operation of the various embodiments of the present disclosure, are described in detail below with reference to the accompanying drawings.

[0007] Brief Description of the Drawings

[0007] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure and further to enable one of ordinary skill in the art to make and use the embodiments.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

[0009] FIG. 4 is a flowchart showing a method for testing candidate rules for improving search engine results, according to some embodiments.

Figure 3

[0010] FIG. 8 shows a block diagram of a general - purpose computer that can be used to perform various aspects of the present disclosure, according to some embodiments. DETAILED DESCRIPTION

[0009]

[0011] In the drawings, like reference numerals generally denote the same or similar elements. Additionally, generally, one or more digits at the left - hand end of a reference numeral identify the drawing in which that reference numeral first appears.

[0010] DETAILED DESCRIPTION

[0012] Provided herein are embodiments and / or combinations and sub - combinations thereof of systems, apparatuses, products, methods, and / or computer program products for improving search engine results of an e - commerce site by testing rules for reducing ambiguity of search inputs, identifying which rules are effective, and implementing those rules at the e - commerce site.

[0011]

[0013] FIG. 1 shows a block diagram of a computing environment 100 that includes an e - commerce site 102 by which a buyer 140 can view, search for, and purchase items and services (referred to herein as for - sale items or FSOs) offered for sale. The buyer 140 can access the e - commerce site 102 via the Internet 130 or any other network or communication medium, standard, protocol, or technology.

[0012]

[0014] The e-commerce site 102 has a list database 104 that includes a list of FSOs that a buyer 140 can search for using a search engine 110. When the buyer 140 finds a desired list, the buyer 140 may select, via a sales module 107, to purchase the FSOs in the desired list.

[0013]

[0015] The e-commerce site 102 has a machine learning module 120 that, in some embodiments, can monitor the interaction of the buyer 140 with the e-commerce site 102 and modify the search engine 110. The e-commerce site 102 also includes another database 106 for storing data and another module 109 for performing functions related to the e-commerce site 102.

[0014]

[0016] The search engine 110 can receive a search input from a buyer 140 via an input module 111 and use a result module 113 to search the list database 104 for a list that matches the search input. The result module 113 can, according to some embodiments, identify the search results and provide them to an output module 119. The output module 119 can provide the search results to the buyer 140.

[0015]

[0017] In some embodiments, the search engine 110 has a rule database 115 that includes rules for filtering search results. The rules can be a set of conditions or parameters for adding a list or removing a list from the search results. In some embodiments, the rules can be configured to resolve the ambiguity of the search input to filter out unwanted results from the search results. In some embodiments, the rules can be configured to push up certain results so that they appear higher or earlier in the overall results. These results can be pushed up based on attributes associated with the results. In some embodiments, the rules can rank the results based on items that have or do not have one or more attributes.

[0016]

[0018] The result filter 117 and the rule database 115 can operate together to filter search results based on a search input and rules. For example, the result filter 117 can filter the search results by applying rules from the rule database 115 and removing results that do not meet the rules. As another example, the result filter 117 can cause the result module 113 to identify only search results from a list that meet the provided rules by providing the rules from the rule database 115 to the result module 113. As yet another example, the result filter 117 can identify a filtered list by applying rules to a list within the list database 104, and then the result module 113 can search the filtered list to identify search results. As yet another example, the result filter 117 can emphasize or push up a particular result so that the particular result appears first in the search results by applying rules from the rule database 115.

[0017]

[0019] For example, a search input of "IPHONE" can match lists of both IPHONES and IPHONE cases. This is an ambiguity introduced by the search input because buyer 140 may enter "IPHONE" when searching for either item. This ambiguity can be resolved by buyer 140 entering additional information, such as adding the word "case" to the search input. Exemplary rules can resolve this ambiguity by differentiating between a list of IPHONES and a list of accessories such as cases. The rules can do this by differentiating categories that distinguish between IPHONES and IPHONE accessories. The rule can be that for an input of "IPHONE", IPHONE accessories should be excluded. That is, the rule can be that for an input of "IPHONE", the accessory category should be excluded.

[0018]

[0020] As another example, the rules can result in pushing "IPHONE" above "IPHONE CASE" or giving more importance to "IPHONE" than to "IPHONE CASE". In this case, the search results will place any search results for "IPHONE" before the search results for "IPHONE CASE".

[0019]

[0021] In some embodiments, the machine learning module 120 dynamically tests candidate rules to identify new rules for incorporation into the rule database 115. The candidate rules can resolve potential or known ambiguities in the search input. The group control 129 can control which buyer 140 is utilized to dynamically test the candidate rules by providing the candidate rules from the rule module 125 to the rule database 115 or the rule result filter 117. The result filter 117 can filter the search results by using candidate rules that include combinations with existing rules in the rule database 115. The group control 129 can configure the output module 119 to provide the filtered search results to a specified group of buyers 140.

[0020]

[0022] The e-commerce site 102 can monitor the responses of the buyers 140 to the search results provided by the output module 119. The response database 121 can store information regarding these responses, the statistical module 123 can perform statistical analysis on these responses, and the metric module 127 can calculate metrics of the responses. The machine learning module 120 can determine whether a candidate rule is effective in resolving ambiguities based on the statistical analysis and metrics. If the candidate rule is effective, the machine learning module 120 can add the rule to the rule database 115 for use in filtering search results. If the candidate rule is not effective, the machine learning module 120 can discard the rule.

[0021]

[0023] The machine learning module 120 may generate candidate rules using the rule module 125 based on the search input or the identified or recognized ambiguity of the search results. The candidate rules may be generated for the purpose of improving the search results identified or generated by the search engine 110. The machine learning module 120 may also receive candidate rules from the rule input module 150 via the Internet 130 or other sources.

[0022]

[0024] In some embodiments, the rule input module 150 may be incorporated into the e-commerce site 102. For example, the rule input module 150 may be incorporated into the machine learning module 120 or other modules 109.

[0023]

[0025] In some embodiments, the machine learning module 120 may perform a dynamic test of the candidate rules on a subset of the buyers 140. The group control 129 may divide the buyers 140 into groups such as buyers 140A, buyers 140B to buyers 140Z, etc. The machine learning module 120 may use a group of buyers 140 such as buyer 140A as a control group that either does not perform filtering or receives only the search results filtered based on the current rules in the rule database 115 rather than the candidate rules. The machine learning module 120 may use other groups of buyers 140 such as buyer 140B or 140Z as a test group that receives the search results filtered by the candidate rules or the current rules combined with the candidate rules. Each test group may be associated with a specific candidate rule.

[0024]

[0026] By using the control group and test group of the buyers 140, the machine learning module 120 can provide a comparison of the buyer responses to the search results from both the baseline search results and the search results filtered using the candidate rules. Statistical analysis and metrics may be performed on the buyer responses as described above and based on these comparisons as further described below.

[0025]

[0027] The machine learning module 120 can perform unsupervised learning, whereby the machine learning module 120 collects data and processes such data upon receipt. Through this process, the machine learning module 120 can advantageously identify and implement rules for improving search results independent of human input. The rules identified by the machine learning module 120 may not be obvious to a human observer, but the machine learning module 120 can identify those rules as valid by using a method (such as an embodiment of the following method 200) for evaluating the rules for determining which rules improve search results for the buyer.

[0026]

[0028] FIG. 2 is a flowchart showing a method 200 for testing candidate rules for improving search engine results according to some embodiments. The method 200 can be performed by processing logic that can include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed by a processing device), or a combination thereof. The steps of the method 200 can be performed by the e-commerce site 102 described above. The steps of the method 200 can be performed by the modules and engines of the e-commerce site 102 as described above and as made more clear in the following description of the steps. A subset of the steps of the method 200 may be sufficient for performing the improved techniques disclosed herein. Further, some of the steps of the method 200 can be performed simultaneously or in an order different from the order shown in FIG. 2, as will be understood by those skilled in the art.

[0027]

[0029] In some embodiments, the e-commerce site 102 provides communication between the modules, databases, and engines included in the e-commerce site 102. The e-commerce site 102 can receive input via the Internet 130 and provide such input to the modules, databases, and engines described above. The e-commerce site 102 can send data to the buyer 140 via the Internet 130.

[0028]

[0030] At 210, the input module 111 receives a search input from the buyer 140. For example, the search input can be received by the e-commerce site 102 and provided to the input module 111 of the search engine 110.

[0029]

[0031] In some embodiments, the search input is a character string representing the FSO that the buyer 140 is looking for in the list of the list database 104. The search input can be entered into the user interface of the website or application running on the e-commerce site 102. The search input can include search constraints such as boolean operators, search category selections, or other constraints.

[0030]

[0032] In some embodiments, the search input received from a specific buyer 140 for a specific FSO is the same. For example, for APPLE IPHONE, the search input received from a specific buyer is always the same character string when that buyer is looking for that specific FSO.

[0031]

[0033] At 215, the group control 129 assigns each buyer 140 to a control group or a test group. The test group corresponds to a candidate rule for improving the search results. There can be two or more test groups, and each test group corresponds to a different candidate rule. For example, the buyer 140A can be a control group, and the buyers 140B to 140Z can be test groups. The control group and the test groups may or may not include the same number of buyers 140.

[0032]

[0034] In some embodiments, the buyer 140 can provide the same search input two or more times. For example, the buyer 140 may repeat the search later. In this case, at 215, the buyer 140 has already been assigned to a group by the group control 129. When the group control 129 tries to assign the buyer 140 to a group, if the buyer 140 already belongs to a group, the group control 129 does not assign the buyer 140 to a new group. Instead, the group control 129 assigns the buyer 140 to the previously assigned group.

[0033]

[0035] At 220, the result module 113 identifies search results for the buyer 140 based on the search input. The result module 113 receives the search input from the input module 111, uses a search algorithm to search the list database 104 or a subset thereof for a list that matches or corresponds to the search input, and identifies them as search results. The e-commerce site 102 may store these search results for a particular search input in another database 106. The search results may be identified by accessing the stored search results that match the search input.

[0034]

[0036] At 230, the result filter 117 filters the search results based on the group to which the buyer 140 belongs and identifies the filtered search results. The result filter 117 can access both the current rules from the rule database 115 and the candidate rules from the rule module 125, or can use those rules. For the control group of buyers 140, the result filter 117 may use the current rules. For the test group of buyers 140, the result filter 117 may use the current rules and the candidate rules corresponding to the test group of buyers 140. Step 230 may identify different filtered search results for the control group and each test group. The e-commerce site 102 may store each filtered search result in another database 106. The filtered search results may be identified by accessing the stored filtered search results for the same input and rules.

[0035]

[0037] Rules in the current rules and candidate rules can be filtered based on various parameters included in both the search input and the list. For example, for a specific search input, a certain list parameter can be prioritized, and only the lists containing that parameter can be included. As another example, for a specific search input, a certain list parameter can be boosted, and that list parameter is listed with a higher score or priority than other lists. As a non-limiting example, the rule can be that a search for "IPHONE" corresponds to the "smartphone" category. This rule identifies the lists containing this category while removing the lists that do not contain this category. As another non-limiting example, the rule can be that a search for "IPHONE" corresponds to the "smartphone" category. This rule boosts the lists containing this category above other lists that do not contain this category.

[0036]

[0038] In some embodiments, steps 220 and 230 can be performed in an order different from the order shown in the example of FIG. 2. For example, as described above, step 220 identifies the search results, and step 230 filters the search results using the rules. This approach can be advantageous when the filtering step is more costly in terms of computer cycles or resources than the search step, as the search step reduces the number of lists to be filtered. In another example, step 230 filters the lists stored in the list database 104 using the rules, and step 220 searches for the filtered lists. This approach can be advantageous when the search step is more costly in terms of computer cycles or resources than the filtering step, as the filtering step reduces the number of lists to be searched.

[0037]

[0039] At 240, the output module 119 provides the filtered search results to the buyer 140 based on the group to which the buyer 140 belongs. The control group of buyers 140 receives the search results filtered based on the current rules. The test group of buyers 140 receives the search results filtered based on the current rules and the candidate rules corresponding to the test group. The filtered search results can be provided to the buyer 140 through the Internet 130.

[0038]

[0040] At step 250, the e-commerce site 102 receives a response from the buyer 140. The e-commerce site 102 stores the response in the response database 121 or other database 106, and may provide an indicator of what response was received or details of the response (such as the price paid for the purchased FSO) to the response database 121. The machine learning module 120 may retrieve the response indicator from other databases and store it in the response database 121.

[0039]

[0041] In some embodiments, the response is an action taken by the buyer 140 based on the provided filtered search results. Exemplary responses include, but are not limited to, the following. · The buyer 140 selects a list within the filtered search results. · The buyer 140 selects to add the FSOs in the list to a checkout system such as an online shopping cart. · The buyer 140 purchases an FSO within a certain period such as 30 days. · The buyer 140 does not purchase the FSOs in the shopping cart after a certain period. · The buyer 140 selects to view two or more lists in the filtered search results. · The buyer 140 enters a modified or different search (which indicates that the buyer 140 did not select any of the lists in the filtered search results). · Buyer 140 closes the browser, window, tab, or application in which the e-commerce site 102 is launched (this indicates that Buyer 140 did not select any of the lists in the filtered search results).

[0040]

[0042] In some embodiments, the e-commerce site 102 receives two or more responses from a single buyer 140. For example, Buyer 140 may view several lists and purchase an FSO from one of those lists, which can result in multiple responses. Step 250 may receive multiple responses over a period of time or simultaneously.

[0041]

[0043] At 260, the metric module 127 calculates a metric for the response. The metric module 127 may calculate metrics for the control group and each test group or subset of groups. The metric module 127 may calculate a single metric or several different metrics. The metric module 127 may combine metrics (including the use of weighted combinations). The weights in the weighted combination may be set based on the relative ranking of the different metrics in providing information about the effectiveness of the candidate rules for reducing the ambiguity between the search results and the filtered search results.

[0042]

[0044] The metric for the response may be the gross merchandise volume (GMV). The metric module 127 may determine or calculate the GMV for the control group as the total cost of the items sold to the buyers in the control group in response to the search results. The GMV may be calculated for the test group as the total cost of the items sold to the buyers in response to the filtered search results.

[0043]

[0045] The metric of the response can be the view rate. In the control group, the metric module 127 can determine or calculate the view rate as the number of buyers 140 who selected to view the FSO list among the filtered search results. In the test group, the metric module 127 can determine or calculate the view rate as the number of buyers 140 who selected to view the FSO list among the filtered search results.

[0044]

[0046] The metric of the response can be the conversion rate. For the control group, the metric module 127 can determine or calculate the conversion rate as the number of a specific type of FSO purchased by the buyers 140 in the control group divided by the number of FSO lists containing the above specific type of FSO in the filtered search results. For the test group, the metric module 127 can calculate the conversion rate as the number of a specific type of FSO purchased by the buyers 140 in the test group divided by the number of FSO lists containing the above specific type of FSO in the filtered search results.

[0045]

[0047] The metric of the response can be the click-through rate (CTR) of rate k. The metric module 127 can determine or calculate the CTR of rate k as the percentage of people who clicked on the items among the top k results. For example, in the case of a CTR of rate 36, if 100 people viewed the results and 32 people clicked on the results among the top 36 results, the CTR of rate 36 is 32%. In some embodiments, rate k is 3, 6, 12, 18, 36 or other values.

[0046]

[0048] The CTR of rate k can identify search results that are more useful or desirable to the user. In some embodiments, the CTR of rate k is used to identify results to be pushed higher or ranked as part of the filtering performed in other steps of method 200. In some embodiments, the results identified by the CTR of rate k are used to identify common item attributes having a high CTR of rate k (such as greater than 50%, greater than 75%, or other thresholds). Thereafter, items having these attributes can be pushed higher or ranked. In some embodiments, the CTR of rate k is used to create rules for pushing up or ranking items or item attributes.

[0047]

[0049] As an example of a combined metric, the metric can be GMV, view rate, and cell-through votes. If two or more of the metrics are higher for the test group than for the control group, the metric can be set to a value indicating that the candidate rule is valid. If two or more of the metrics are higher for the control group than for the test group, the metric can be set to a value indicating that the candidate rule is invalid. This combination can be weighted such that some of the votes are more valuable than others, as described above.

[0048]

[0050] At 270, the statistical module 123 determines the statistical significance of the metric. Each metric indicates whether the candidate rule is improving the results based on a comparison with a metric threshold. The metric module 127 can compare the metric with the metric threshold to determine what the metric indicates. The metric module 127 can provide this indication to the statistical module 123. Based on the indication, the statistical module 123 can formulate a hypothesis that the candidate rule is functioning as indicated.

[0049]

[0051] In some embodiments, the statistical module 123 may determine statistical significance by comparing a metric to a threshold. For example, one or more values can be calculated based on the metric, and each value can be compared to its respective threshold. The metric is statistically significant if each value is greater than its respective threshold.

[0050]

[0052] As an example, the determined value can be a p-value, i.e., the statistical module 123 can determine statistical significance by performing a p-test on the metric. The statistical module 123 can perform a p-test on the hypothesis that the candidate rule is functioning as shown (including determining a p-value for the hypothesis). As an example, the hypothesis for the p-test can compare the results of the test group to a control group, a combination of the control group and other test groups, or different test groups.

[0051]

[0053] As another example, the total sales of unfiltered search results can have a certain value, while the GMV of the filtered results can be somewhat higher in percentage. These two comparisons result in an increase rate, which indicates the statistical significance of the GMV if the increase rate is higher than a threshold such as 30% or 40%. In a specific non-limiting example, if the GMV is $2000 and the same item is sold for only $1000 without filtering, the raw value has increased by 100%. This increase is greater than the 30% threshold and is thus statistically significant.

[0052]

[0054] In some embodiments, the threshold takes into account both the increase rate and the aggregate value. For example, if the GMV is $13 and the unfiltered selling price is $10, there is a 30% increase, but the actual dollar increase is small. The threshold can be an increase rate and a dollar increase greater than a certain amount such as $50, $100, $500, $1000, or other amounts.

[0053]

[0055] At 275, the statistical module 123 checks whether the statistical significance of the metric is greater than a threshold. If one or more of the values determined at step 270 are greater than their respective thresholds, the metric is statistically significant. Based on this result, method 200 returns to step 210 to receive search inputs from other buyers 140.

[0054]

[0056] In some embodiments, the threshold can be set or changed based on the number of remaining candidate rules. For example, the threshold can increase as the number of candidate rules increases.

[0055]

[0057] At 280, the metric module 127 checks whether the metric is greater than a metric threshold. The metric threshold can be set, as described above, to identify whether a candidate rule improves the search results of a test group over a control group of rules, other candidate rules of a test group, or both. The metric, the metric threshold, or both can be scaled or normalized for comparison with each other.

[0056]

[0058] In some embodiments, the metric module 127 checks whether the metric is greater than a metric threshold to determine the hypothesis used at step 270. The metric module 127 can store the results in an internal or other database 106. When performing step 280, instead of repeating the check, the metric module 127 can access the results or pull them out to check it.

[0057]

[0059] If the metric is less than the metric threshold, the candidate rule is not valid and method 200 proceeds to step 285. At 285, machine learning module 120 discards the candidate rule. Machine learning module 120 can also discard the response data stored in the response database regarding the candidate rule and the corresponding test group. Group control 129 can remove the test group corresponding to the candidate rule. Future metrics and statistical calculations or decisions made by machine learning module 120 can be made without including the removed test group. Group control 129 can respond to future searches by buyers 140 from the removed test group by assigning these buyers 140 to other test groups or control groups.

[0058]

[0060] If the metric is greater than the metric threshold, the candidate rule is valid and method 200 proceeds to step 290.

[0059]

[0061] In some embodiments, the metric threshold can be set or varied based on the number of remaining candidate rules. For example, the metric threshold can increase as the number of candidate rules decreases.

[0060]

[0062] At 290, machine learning module 120 checks whether the evaluation of the candidate rule is complete. For example, the evaluation of the candidate rule can be complete when the number of remaining candidate rules falls below the rule number threshold, when the p-value for all of the remaining rules exceeds the p-value threshold, or when all of the p-values for the remaining candidate rules indicate that the hypothesis for that candidate rule is statistically significant.

[0061]

[0063] In some embodiments, machine learning module 120 causes statistical module 123 to update the p-test based on the removal of the candidate rule at step 285 to verify whether the evaluation of the candidate rule is complete.

[0062]

[0064] If the evaluation is not complete, method 200 returns to step 210 to receive further search inputs.

[0063]

[0065] If the evaluation is complete, method 200 proceeds to step 295. In 295, the machine learning module 120 updates the rule database 115 by adding candidate rules to the current rules. In some embodiments, the machine learning module 120 adds the remaining candidate rules in the rule module 125 to the current rules. The added candidate rules may be limited to candidate rules having a statistically significant metric indicating that the candidate rules are valid for the current rules in the rule module 125.

[0064]

[0066] In some embodiments, in step 295, the machine learning module 120 updates the metrics for the remaining rules. This update may occur by performing step 260. The machine learning module 120 may check the updated metrics against a metric threshold. This check may occur by performing step 280. The machine learning module 120 may remove candidate rules having updated metrics smaller than the metric threshold and add the remaining candidate rules to the current rules in the rule module 125.

[0065]

[0067] One of ordinary skill in the art will understand that method 200 may receive different search inputs or responses from different buyers 140 at different times. The e-commerce site 102, search engine 110, and machine learning module 120 may perform various steps of method 200 for different buyers 140, search inputs, or responses, either simultaneously or at different times. Method 200 may perform various steps actively, simultaneously, sequentially, or at different times as needed to handle different inputs or processes of method 200. Steps of method 200, such as discarding candidate rules, may, as described above, affect other steps of method 200, which may result in updates or changes to how some steps are performed between iterations.

[0066]

[0068] Method 200 can be performed simultaneously and independently for different search inputs. For example, e-commerce site 102 can receive multiple different search inputs from buyer 140 and perform Method 200 for each different search input to improve search results. Buyer 140 can be assigned to different control groups and test groups for each search input entered by buyer 140.

[0067]

[0069] With respect to a particular set of candidate rules, Method 200 can be performed on the search input until all candidate rules are discarded or a portion of the candidate rules are added to the current rules. The discarded candidate rules can be retested later as part of a different set of candidate rules. The different set of candidate rules can include the discarded candidate rules from previous iterations. One of ordinary skill in the art will understand that the effectiveness of candidate rules can change from invalid to valid over time depending on changes in the habits of buyer 140 or market forces.

[0068] Exemplary computer system

[0070] Various embodiments can be implemented using one or more computer systems, such as computer system 300 shown in FIG. 3, for example. One or more computer systems 300 can be used to implement any of the embodiments described herein, as well as combinations and subcombinations thereof.

[0069]

[0071] Computer system 300 can include one or more processors, such as processor 304 (also referred to as a central processing unit or CPU). Processor 304 can be connected to a bus or communication infrastructure 306.

[0070]

[0072] Computer system 300 can also include one or more input / output devices 303, such as a monitor, keyboard, pointing device, etc., that can communicate with communication infrastructure 306 via one or more user input / output interfaces 302.

[0071]

[0073] One or more of the processors 304 can be a Graphics Processing Unit (GPU). In some embodiments, the GPU can be a processor that is a dedicated electronic circuit designed to process computationally intensive applications. The GPU can have a parallel architecture that is efficient for parallel processing of large data blocks, such as computationally intensive data common to, for example, computer graphics applications, images, video, vector processing, array processing, etc., as well as encryption techniques (including brute force cracking), generating cryptographic hashes or hash sequences, solving partial hash inversion problems, and / or producing results of other proof-of-work calculations for some blockchain-based applications. Due to the General-Purpose Computing on Graphics Processing Units (GPGPU) capability, the GPU can be particularly useful, at least in the aspects of image recognition and machine learning described herein.

[0072]

[0074] In addition, one or more of the processors 304 can include a coprocessor or other implementation of logic (including a hardware-accelerated cryptographic coprocessor) for accelerating cryptographic calculations or other special mathematical functions. Such an accelerated processor can further include one or more instruction sets for a coprocessor and / or acceleration using other logic for such acceleration.

[0073]

[0075] The computer system 300 can also include main or primary memory 308, such as Random Access Memory (RAM). The main memory 308 can include one or more cache levels. The main memory 308 can store control logic (i.e., computer software) and / or data therein.

[0074]

[0076] The computer system 300 may also include one or more secondary storage devices or secondary memories 310. The secondary memory 310 may include, for example, a main storage drive 312 and / or a removable storage device or drive 314. The main storage drive 312 may be, for example, a hard disk drive or a solid state drive. The removable storage drive 314 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, a tape backup device, and / or any other storage device / drive.

[0075]

[0077] The removable storage drive 314 may interact with a removable storage unit 318. The removable storage unit 318 may include a computer-usable or readable storage device storing computer software (control logic) and / or data. The removable storage unit 318 may be a floppy disk, a magnetic tape, a compact disk, a DVD, an optical storage disk, and / or any other computer data storage device. The removable storage drive 314 can read from and / or write to the removable storage unit 318.

[0076]

[0078] The secondary memory 310 may include other means, devices, components, apparatuses, or other techniques to enable a computer program and / or other instructions and / or data to be accessed by the computer system 300. Such means, devices, components, apparatuses, or other techniques may include, for example, a removable storage unit 322 and an interface 320. Examples of the removable storage unit 322 and the interface 320 may include a program cartridge and a cartridge interface (such as those found in a video game console), a removable memory chip (such as an EPROM or PROM) and a related socket, a memory stick and a USB port, a memory card and a related memory card slot, and / or any other removable storage unit and a related interface.

[0077]

[0079] The computer system 300 may further include a communication or network interface 324. The communication interface 324 may enable the computer system 300 to communicate and interact with any combination of external devices, external networks, external entities, etc. (collectively and individually referred to by reference numeral 328). For example, the communication interface 324 may enable the computer system 300 to communicate with an external or remote device 328 through a communication path 326, which may be wired and / or wireless (or a combination thereof) and may include any combination such as a LAN, a WAN, the Internet, etc. Control logic and / or data may be transmitted to and from the computer system 600 via the communication path 326.

[0078]

[0080] The computer system 300 can be any of, by way of several non-limiting examples, a personal digital assistant (PDA), a desktop workstation, a laptop or notebook computer, a netbook, a tablet, a smartphone, a smartwatch or other wearable, an electrical product, a part of the Internet of Things (IoT) and / or an embedded system, or any combination thereof.

[0079]

[0081] It is understood that the framework described herein can be implemented as a product such as a method, a process, an apparatus, a system, or a non-transitory computer-readable medium or device. For illustrative purposes, this framework can generally be described in the context of a distributed ledger that is publicly available or at least accessible to untrusted third parties. One example of a modern use case is the use of a blockchain-based system. However, it is understood that this framework can also be applied in other situations where confidential or classified information may need to be passed into the hands of or through untrusted third parties, and that this technology is in no way limited to distributed ledger or blockchain applications.

[0080]

[0082] The computer system 300 can be a client or server that accesses or manages any applications and / or data via any delivery paradigm, including, but not limited to, remote or distributed cloud computing solutions, local or on-premises software (e.g., an "on-premises" cloud-based solution), a "service-type" model (e.g., Content as a Service (CaaS), Digital Content as a Service (DCaaS), Software as a Service (SaaS), Managed Software as a Service (MSaaS), Platform as a Service (PaaS), Desktop as a Service (DaaS), Framework as a Service (FaaS), Backend as a Service (BaaS), Mobile Backend as a Service (MBaaS), Infrastructure as a Service (IaaS), Database as a Service (DBaaS), etc.) and / or any combination of the above examples or other services or delivery paradigms, including hybrid models.

[0081]

[0083] Any applicable data structures, file formats, and schemas can be obtained from specifications that include, but are not limited to, JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations, either alone or in combination. Alternatively, custom data structures, formats, or schemas can be used exclusively or in combination with known or open specifications.

[0082]

[0084] Any suitable data, file, and / or database can be stored, retrieved, accessed, and / or transmitted in a human-readable format, such as, among many possible formats, numerical, text, graphic, or multimedia format (further including various types of markup languages). Alternatively or in combination with the above formats, the data, file, and / or database can be stored, retrieved, accessed, and / or transmitted in a binary, encoded, compressed, and / or encrypted format or any other machine-readable format.

[0083]

[0085] Various system and inter-layer interfacing or interconnecting can use any number of mechanisms, including, but not limited to, any number of protocols, program frameworks, floor plans, or application programming interfaces (APIs), such as the Document Object Model (DOM), Discovery Service (DS), NSUserDefaults, Web Services Description Language (WSDL), Message Exchange Pattern (MEP), Web Distributed Data Exchange (WDDX), Web Hypertext Application Technology Working Group (WHATWG) HTML5 Web Messaging, Representational State Transfer (REST or RESTful web services), Extensible User Interface Protocol (XUP), Simple Object Access Protocol (SOAP), XML Schema Definition (XSD), XML Remote Procedure Call (XML-RPC), or any other open or proprietary mechanism that can achieve similar functionality and results.

[0084]

[0086] Such interfacing or interconnecting can also utilize a Uniform Resource Identifier (URI), which may further include a Uniform Resource Locator (URL) or a Uniform Resource Name (URN). Other forms of unified and / or unique identifiers, locators, or names can be used exclusively or in combination with forms such as those described above.

[0085]

[0087] Any of the above protocols or APIs can interface with any procedural, functional, or object - oriented programming language, or can be implemented and compiled or interpreted in such a programming language. Non - limiting examples include C, C++, C#, Objective - C, Java, Scala, Clojure, Elixir, Swift, Go, Perl, PHP, Python, Ruby, JavaScript, WebAssembly, or virtually any other language, and include, without limitation, any kind of framework, runtime environment, virtual machine, interpreter, stack, engine, or similar mechanism such as Node.js, V8, Knockout, jQuery, Dojo, Dijit, OpenUI5, AngularJS, Express.js, Backbone.js, Ember.js, DHTMLX, Vue, React, Electron, etc., having any other library or schema.

[0086]

[0088] In some embodiments, a tangible non-transitory apparatus or article that includes a tangible non-transitory computer-usable or readable medium that stores control logic (software) may be referred to herein as a computer program product or a program storage device. This includes, but is not limited to, the computer system 300, main memory 308, secondary memory 310, and removable storage units 318 and 322, and tangible articles that embody any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system 300), can cause such data processing devices to operate as described herein.

[0087]

[0089] Based on the teachings contained in this disclosure, methods for manufacturing and using embodiments of this disclosure will be apparent to those skilled in the art using data processing devices, computer systems, and / or computer architectures other than those shown in FIG. 3. Specifically, embodiments may operate using implementations of software, hardware, and / or operating systems other than those described herein.

[0088]

[0090] It is understood that the detailed description section (and not the other sections) is intended to be used to interpret the claims. The other sections can describe one or more (but not all) exemplary embodiments as contemplated by the inventor, and are thus not intended to limit the present disclosure or the appended claims in any way.

[0089]

[0091] This disclosure describes exemplary embodiments related to exemplary fields and applications, but it is understood that the disclosure is not limited thereto. Other embodiments and modifications are possible and are within the scope and spirit of the disclosure. For example, without limiting the general discussion of this paragraph, embodiments are not limited to the software, hardware, firmware, and / or entities shown in the drawings and / or described herein. Further, embodiments have significant utility for fields and applications beyond the examples described herein (regardless of whether there is an explicit description herein).

[0090]

[0092] Embodiments are described herein using functional components that show the implementation of specific functions and their relationships. The boundaries of these functional components are arbitrarily defined herein for convenience of explanation. As long as the specific functions and relationships (or their equivalents) are appropriately performed, other boundaries may be defined. Also, alternative embodiments may perform functional blocks, steps, operations, methods, etc. using an ordering different from the ordering described herein.

[0091]

[0093] References in this specification to "one embodiment," "an embodiment," "exemplary embodiment," "some embodiments," or similar expressions indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. Also, such expressions do not necessarily refer to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, incorporating such feature, structure, or characteristic into other embodiments, whether or not explicitly stated or described herein, is within the knowledge of those skilled in the art.

[0092]

[0094] Additionally, some embodiments can be described using the terms "coupled" and "connected" together with derivatives thereof. These terms are not necessarily intended to be synonyms of each other. For example, in some embodiments, the terms "connected" and / or "coupled" can be used to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other but still cooperate or interact with each other.

[0093]

[0095] The breadth and scope of the present disclosure are not limited by any of the above exemplary embodiments, but are defined only in accordance with the following claims and their equivalents.

Claims

1. A computer-implemented method for dynamically testing candidate rules to improve the search results of a list of sellable items (FSOs) sold on an e-commerce site, comprising: Receiving a search input from one or more buyers; Assigning each of the one or more buyers to a group among a plurality of groups, wherein the plurality of groups includes a control group and one or more test groups, and each test group corresponds to a candidate rule among one or more candidate rules; Providing a reference search result based on the search input and the current rule to the control group of the buyers; Providing a filtered search result to the test group of the buyers based on the search input, the current rule, and the candidate rule corresponding to a specific test group among the test groups; Receiving a control response from the control group of the buyers and a test response from the test group of the buyers, wherein the control response and the test response are one or more of purchasing the FSO, viewing a list from the filtered search result or a filtered search result different from the search result, entering a further search input, or closing the e-commerce site; For each test group, Determining a metric for the test group based on the control response and the test response, wherein determining the metric comprises: Determining a total distribution transaction amount as the total cost of items sold to each buyer in response to each search result; Determining a conversion rate as the number of a specific type of FSO purchased by each buyer divided by the number of FSO lists including the specific type of FSO in each search result; Determining a click-through rate of rate k as the percentage of each buyer who clicks on the top k items in each search result, or Determining a view rate as the percentage of each buyer who receives an FSO list in each search result and selects to view the FSO list, Discarding the candidate rule corresponding to the test group in response to the metric being statistically significant and less than a threshold; and Adding the candidate rule corresponding to the test group to the current rule in response to the metric being statistically significant and greater than the threshold. A computer-implemented method including

2. The method according to claim 1, wherein determining the metric includes combining the total transaction volume in circulation, the conversion rate, the click-through rate of the rate k, and the view rate.

3. The method according to claim 2, wherein combining the total transaction volume in circulation, the conversion rate, the click-through rate of the rate k, and the view rate includes summing the total transaction volume in circulation, the conversion rate, the click-through rate of the rate k, and the view rate in a weighted combination.

4. Performing a p-test on a hypothesis based on the metric for each test group to determine a p-value, Comparing the p-value with a p-value threshold, Identifying that the metric is statistically significant in response to the p-value being greater than the p-value threshold The method according to claim 1, further comprising determining that the metric is statistically significant.

5. Calculating one or more values based on the metric, Comparing each of the one or more values with a respective threshold, Identifying that the metric is statistically significant in response to each of the one or more values being greater than the respective threshold The method according to claim 1, further comprising determining that the metric is statistically significant.

6. A system for dynamically testing candidate rules for improving search results of a list of for-sale items (FSOs) sold on an e-commerce site, One or more processors, One or more network interfaces communicatively coupled to the one or more processors, A memory communicatively coupled to the one or more processors and the one or more network interfaces, which when executed, causes the one or more processors to, Receive a search input from one or more buyers, Assigning each buyer among the one or more buyers to a group among a plurality of groups, wherein the plurality of groups includes a control group and one or more test groups, and each test group corresponds to a candidate rule among one or more candidate rules, Identifying search results from a plurality of FSO lists based on the search input Filtering the search results based on current rules to identify a first filtered search result; For each test group among the one or more test groups, filtering the search results based on the current rules and a corresponding candidate rule among the one or more candidate rules corresponding to the test group to identify a filtered search result corresponding to the test group; Providing the first filtered search result to the control group; For each test group among the one or more test groups, providing the filtered search result corresponding to the test group; Receiving one or more response metrics from the one or more buyers, where the one or more response metrics include one or more of purchasing the FSO, viewing a list from the filtered search results or a different filtered search result from the search results, entering further search input, or closing the e-commerce site; Determining a performance metric for each test group of the one or more test groups based on the one or more response metrics, where determining the performance metric includes: Determining a total distribution transaction amount as the total cost of items sold to each buyer in response to each search result; Determining a conversion rate as the number of a specific type of FSO purchased by each buyer divided by the number of FSO lists including the specific type of FSO in each search result; Determining a click-through rate of rate k as the percentage of each buyer who clicked on the top k items in each search result, or Determining a view rate as the percentage of each buyer who selected to receive and view the FSO list in each search result, selected from one or more of the above; Determining the statistical significance for each test group of the one or more test groups based on at least one of the one or more performance metrics; For each test group among the one or more test groups, in response to the statistical significance for the test group being greater than a threshold; In response to the performance metric for the test group being less than the metric threshold, discarding, among the one or more candidate rules, the candidate rule corresponding to the test group; In response to the performance metric for the test group being greater than the metric threshold, adding the candidate rule corresponding to the test group to the current rule; A memory storing instructions to cause; A system including.

7. The system according to claim 6, wherein the instructions further cause the one or more processors to determine the performance metric by combining the total circulation transaction amount, the conversion rate, the click-through rate of the rate k, and the view rate based on the one or more response metrics.

8. The system according to claim 7, wherein the instructions further cause the one or more processors to combine the total circulation transaction amount, the conversion rate, the click-through rate of the rate k, and the view rate by weighted combination.

9. The instructions Perform a p-test on a hypothesis based on the performance metric for each test group to determine a p-value; Compare the p-value with a p-value threshold; In response to the p-value being greater than the p-value threshold, identify that the performance metric is statistically significant; Thereby, the system according to claim 6, wherein the instructions further cause the one or more processors to determine the statistical significance for the test group.

10. The instructions Calculate one or more values based on the performance metric; Compare each of the one or more values with a respective threshold; In response to each of the one or more values being greater than the respective threshold, identify that the performance metric is statistically significant; Thereby, the system according to claim 6, wherein the instructions further cause the one or more processors to determine the statistical significance for the test group.

11. A non-transitory computer-readable storage medium having computer-readable code, Receiving a search input from one or more buyers; Assigning each buyer among the one or more buyers to a group among a plurality of groups, where the plurality of groups includes a control group and one or more test groups, and each test group corresponds to a candidate rule among one or more candidate rules; Identifying search results from a list of a plurality of sellable items (FSOs) based on the search input; Filtering the search results based on the current rule to identify a first filtered search result; For each test group among the one or more test groups, filtering the search results based on the current rule and the corresponding candidate rule among the one or more candidate rules corresponding to the test group to identify a filtered search result corresponding to the test group; Providing the first filtered search result to the control group; For each test group among the one or more test groups, providing the filtered search result corresponding to the test group; Receiving one or more response metrics from the one or more buyers, where the one or more response metrics are one or more of purchasing the FSO, viewing a list from the filtered search result or a filtered search result different from the search result, entering further search input, or closing the e-commerce site; Determining a performance metric for each test group among the one or more test groups based on the one or more response metrics, where determining the performance metric is Determining a total distribution transaction amount as the total cost of items sold to each buyer in response to each search result; Determining a conversion rate as the number of a specific type of FSO purchased by each buyer divided by the number of FSO lists including the specific type of FSO in each search result; Determining a click-through rate of rate k as the percentage of each buyer who clicks on the top k items in each search result, or Determining a view rate as the percentage of each buyer who receives an FSO list in each search result and selects to view the FSO list, being one or more selected from among these; Determining the statistical significance for each of the one or more test groups based on at least one performance metric; For each of the one or more test groups among the one or more test groups, in response to the statistical significance for the test group being greater than a threshold; In response to the performance metric for the test group being less than a metric threshold, discarding the candidate rule corresponding to the test group among the one or more candidate rules; In response to the performance metric for the test group being greater than the metric threshold, adding the candidate rule corresponding to the test group to the current rule A non-transitory computer-readable storage medium comprising instructions configured to cause a computer system to perform operations including the above.

12. The non-transitory computer-readable storage medium according to claim 11, wherein the operations further include determining the performance metric by combining the total circulation transaction amount, the conversion rate, the click-through rate of rate k, and the view rate based on the one or more response metrics.

13. The non-transitory computer-readable storage medium according to claim 12, wherein the operations further include combining the total circulation transaction amount, the conversion rate, the click-through rate of rate k, and the view rate using a weighted combination.

14. The operations include Calculating one or more values based on the performance metric; Comparing each of the one or more values with a respective threshold; Identifying that the performance metric is statistically significant in response to each of the one or more values being greater than the respective threshold The non-transitory computer-readable storage medium according to claim 11, further comprising determining that the performance metric is statistically significant by the above.

Images