Systems and methods for ranking objects

Abstract

To provide a system and a method for ranking objects.SOLUTION: A computer-implemented method for ranking a set of objects includes the steps of: receiving a set of objects and a set of objective functions; defining a decision space using a permutohedron having n decision variables, wherein n is the number of objects to be ranked, and vertices of the permutohedron represent exposure associated with corresponding ranks; deciding a Pareto set for the set of the objective functions; deciding a distribution for the ranks of the objects in the set, using the decision space, while deciding a Pareto optimal point in the Pareto set; selecting a sequence of ranks on the set of objects from the distribution for the ranks by proportion; and outputting the selected sequence of ranks.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application (and its counterpart U.S. application) claims priority pursuant to 35 U.S.C. 119 to French Patent Application No. FR2104801, filed May 6, 2021, the contents of which are incorporated herein by reference.

[0002] This application (corresponding U.S. application) further claims priority pursuant to 35 U.S.C. 119 to European Patent Application No. EP21306565, filed November 8, 2021, the contents of which are incorporated herein by reference.

[0003] The field of the invention is information retrieval and recommendation. More particularly, embodiments of the invention relate to computer-implemented methods and systems for unbiased ranking of retrieved objects. [Background technology]

[0004] Generally, information retrieval and recommendation consists of two steps: the first step focuses on searching a candidate set of multiple results, and the second step focuses on ranking the candidate set of multiple results.

[0005] The candidate set of results may include search results (e.g., a list of links to documents from search results in response to a query) and recommendations (e.g., a list of recommended points of interest in response to an identified location, or a list of recommended songs in response to a genre selection).

[0006] For example, for information retrieval, a query can be input to a first step searcher, which processes the query (e.g., based on relevance) to retrieve a set of documents. Then, a second step or ranker (or reranker) ranks the retrieved set of documents and outputs a ranked set of documents, which may be the same or even smaller in number than the first set.

[0007] Users (e.g., content consumers) expect the most relevant results to be exposed, while providers (e.g., content producers) seek fair (or impartial) exposure to their content. Thus, in the second step, when ranking the candidate sets of results, it is desirable for the ranking method to maintain a balance between utility (indicating users having access to the set of results) and fairness (indicating providers composing the set of results).

[0008] While some ranking methods exist that maintain a balance between utility and fairness, the complexity of the methods generally prohibits their use in realistic scenarios. Thus, there is a continuing need for optimal methods with reduced complexity relative to existing methods (e.g., capable of computing an exact optimal solution), e.g., to enable web-scale fairness-utility ranking. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Korean Patent Registration No. 10-1710465 Summary of the Invention [Problem to be solved by the invention]

[0010] An exemplary method of the present invention provides a computer-implemented method for ranking a set of multiple objects according to a first aspect, the method including the steps of receiving a set of multiple objects and a set of multiple objective functions; using a permutohedron to define a decision space having n decision variables, where n is the number of objects to be ranked and the vertices of the permutohedron represent permutations of exposure provided to objects in the set with corresponding ranks; determining a Pareto set for the set of multiple objective functions; determining a Pareto optimal point within the Pareto set and using the decision space to determine a distribution over ranks for the objects in the set, where each rank and proportion within the distribution are associated; selecting a sequence of ranks for the objects in the set from the distribution over ranks by proportion; and outputting the selected sequence of ranks.

[0011] A preferred but non-limiting embodiment of the method according to the first embodiment is as follows.

[0012] The set of multiple objective functions may include quadratic and linear functions; The quadratic function may be a fairness function, and the linear function may be a utility function; The fairness function may be a normalized difference, such as the L2-norm of the squared difference between the vector composed of decision variables and the target vector, The method may further include receiving a list of exposures to be provided to the objects in the set, each associated with a relevance score and a rank for each object in the set, a ranking fairness objective function, and a ranking utility objective function; Determining a Pareto set for a set of multiple objective functions may include calculating a Pareto set in a decision space given a ranking fairness objective function and a ranking utility objective function; and using the decision space to determine a distribution over ranks for the objects in the set may include receiving specific points in the Pareto set that are translated into target exposures across the objects in the set in the decision space; and using the target exposures to determine a distribution over ranks that on average achieves the target exposures for the objects in the set, wherein each rank in the distribution over ranks corresponds to a vertex in the decision space; The step of using the decision space to determine a distribution for ranks for the objects in the set includes determining a distribution for the number of objects in the set by: (i) determining an arbitrary vertex of the decision space; (ii) drawing a line starting from any vertex with the target exposure until the line intersects with a face of the decision space; and (iii) repeating (i) and (ii) on the intersected face of the decision space using the new intersection point in place of the target exposure until the new intersected face becomes a vertex; The fairness function can be based on meritocratic fairness or demographic fairness; The step of determining a distribution for ranks of the objects in the set using the decision space can use a GLS (Grotschel, Lovasz and Schrijver) procedure; selecting a sequence of rankings for the objects in the set may include one or more of stochastic sampling, low-discrepancy sequences, additive-recurrence sequences, stride scheduling, or m-balancing; An object in a set can contain multiple queries, The objects in the set can be identified in response to a query; An object in the set can contain multiple recommendations, The objects in the set can be identified according to map location; The outputting step can include providing at least a subset of the set of objects ranked according to the selected ranking sequence for display on a display of the device; At least a subset of the set of objects may be provided on a search engine results page.

[0013] According to each of the second, third, fourth and fifth aspects, the present invention may provide a computer program product including code instructions that, when executed on a computer, cause the computer to perform a method according to the first aspect of the present invention; a computer-readable medium having the computer program product stored thereon; a data processing device including a processor configured to perform a method according to the first aspect of the present invention; and / or a system for information retrieval including a computer-implemented first step searcher configured to receive a query and generate a set of objects, and a computer-implemented second step ranker configured to rank the set of objects according to the first aspect of the present invention. [Brief explanation of the drawings]

[0014] The accompanying drawings are incorporated herein to explain the principles of the embodiments. The drawings should not be construed as limiting the invention to the illustrated and described embodiments or to the manner in which they may be made and used. Additional features and advantages will become apparent in the following description and more particular section of the embodiments, as illustrated in the accompanying drawings. [Figure 1] FIG. 1 illustrates an example architecture capable of performing the disclosed methods. [Figure 2]1 is a flowchart illustrating one embodiment of the disclosed method. [Figure 3] FIG. 10 shows an example of an exemplary permutohedron for n=4. [Figure 4] FIG. 10 is a diagram showing an example of a procedure for decomposing a point in a permutohedron into a combination of up to n vertices of the permutohedron. [Figure 5] FIG. 1 shows an example of an exemplary generalized permutohedron (referred to as an expohedron in the exemplary method) for n=3. [Figure 6] FIG. 10 illustrates an example decomposition method intent code that can be used in some methods. [Figure 7] FIG. 1 shows examples of equi-utility curves and max-utility solutions in expohedron (n=3). [Figure 8] FIG. 1 shows an example of a maximum utility solution for an expohedron with relevance ties (n=3). [Figure 9] Figure 1 shows examples of optimal demographic and meritocratic fairness solutions and equivalent fairness curves in Expohedron (n=3). [Figure 10] Figure 1 shows examples of optimal demographic and meritocratic fairness solutions and equivalent fairness curves in Expohedron (n=3). [Figure 11] FIG. 10 is a diagram showing an example of a Pareto set for expohedron (n=3). [Figure 12] FIG. 10 illustrates intent codes for an exemplary Pareto set identification method. DETAILED DESCRIPTION OF THE INVENTION

[0015] 1. Architecture The disclosed computer-implemented methods and embodiments for ranking objects can be implemented within an architecture (e.g., network or system architecture) such as that shown in FIG. 1 , which includes a server 100 and one or more client devices 102 that communicate over a network 104 (wireless and / or wired), such as the Internet, for exchanging data. The server 100 and the client devices 102 include a data processor 112 and memory 113, such as, but not limited to, random access memory (RAM), read-only memory (ROM), a hard disk, a solid-state disk, or other non-volatile storage medium. Additionally, the memory 113 can be provided in whole or in part by external memory or storage that communicates with the data processor 112. The client devices 102 can be any device that communicates with the server 100.

[0016] The exemplary methods provided herein may be implemented by a processor, such as the data processor 112 or other processors in the server 100 and / or client device 102. The data processor 112 may include a single processor or multiple processors operating in serial or parallel. Memory used in the exemplary methods may be implemented, for example, by memory 113 and / or suitable storage within the server 100, the client devices 102b-102e, coupled remote storage, or any combination. Memory may include one or more memories or memory elements or structures, including a combination of memory types and / or locations. Data in memory may be stored in any suitable format for data retrieval and processing.

[0017] The server 100 may include, but is not limited to, a dedicated server, a cloud-based server, or a combination thereof (e.g., shared). Data streams may be communicated / communicated to, received / received by, or generated by the server 100 and / or the client devices 102b-102e.

[0018] The client devices 102b-102e may be any processor-based device, terminal, or the like, and may be embodied as a client application executable by a processor-based device, etc. The client devices may be located within the server 100 (locally or remotely, or any combination), or may be located outside the server and communicate with the server. Exemplary client devices 102b-102e include, but are not limited to, various processor-based devices not shown in FIG. 1 , such as an autonomous vehicle 102b, a robot 102c, a computer 102d, a mobile communication device (e.g., a smartphone or tablet PC), such as a smartphone 102e, a virtual reality (VR), an augmented reality (AR), or a mixed reality (MR) device, a wearable computer, etc. Client devices 102b-102e may be, but are not necessarily, configured to transmit / receive data to / from server 100 and may include one or more output devices such as, but not limited to, a display, speaker, printer, etc. for displaying, presenting, or printing the results of a particular method provided for display by the server. A client device may include a combination of client devices.

[0019] 2. Methods for ranking sets of objects The exemplary method provided herein solves the problem of efficiently designing an optimal fairness-utility ranking policy using a set of optimization / decision variables. A first feature of the exemplary method is to select, as an optimization / decision variable, a "listing exposure" variable that serves as a key link between important goals that constitute an ideal ranking, in particular, a utility goal that typically represents the user or consumer's perspective, and a fairness goal that typically represents the provider or supplier's perspective.

[0020] Advantageously, a second feature of the exemplary method is a generalized permutohedron, which allows optimization problems to be expressed in a decision space with only n decision variables (where n is the number of objects to be ranked), while maintaining policy expressibility for precise and complete control of utility and fairness goals. The vertices of the disclosed exemplary permutohedron can indicate exposures associated with corresponding rankings; such permutohedrons are referred to herein as "expohedrons." Any distribution (or convex combination) over rankings (or synonymously, over permutations) through the decision space provided by the permutohedron (e.g., expohedron) can be represented and geometrically reasoned to resolve utility-fairness tradeoffs in this space. In particular, the entire Pareto set of a multi-objective optimization (MOO) problem can be easily obtained without explicit scalarization techniques, thereby reducing required processing time and resources.

[0021] The exemplary method can determine optimal policies using unbiased estimates of relevance scores with uniform estimation quality for objects and exposure models with known structure and parameters.

[0022] An exemplary method is O(n 2 The method operates with complexity (log n). Such methods are amenable to geometric reasoning. Most steps of the method can be expressed by closed-form equations. Other steps of the method can be provided by aligning operations. Also, the global Pareto set can be generated analytically and geometrically, without relying on scalarization techniques that scan the global boundary, for example.

[0023] 2 is a flowchart illustrating one embodiment of the disclosed method for ranking a set of objects. The method according to the embodiment of FIG. 2 is performed by a server 100, and the exemplary method is described in connection with the server 100. However, it should be understood that the method may likewise be performed by a client device 102, or by a combination of the server 100 and the client device 102.

[0024] In step 202, server 100 receives from any suitable source(s), including external and / or internal sources, a set of objects, a (e.g., unbiased) relevance score for each object in the set of objects, a list of exposures each associated with a rank, and objective functions including a ranking fairness objective function and a ranking utility objective function. In step 204, server 100 defines a decision space using a permutohedron with n decision variables, where n is the number of objects to rank, and the vertices of the permutohedron indicate permutations of exposures to be provided to the objects in the set with corresponding ranks.

[0025] In step 206, using the ranked fairness objective function and the ranked utility objective function, server 100 calculates a Pareto set (i.e., a set of non-dominated solutions for which there are no better solutions for all objective functions, e.g., no exposures that simultaneously provide greater utility and greater fairness) within the decision space defined by the list of exposures between the first point exhibiting optimal fairness and the second point exhibiting optimal utility. From an exemplary method, optimal fairness types can include, for example, demographic fairness and meritocratic fairness. As described below, ideal exposures for demographic fairness allow for equal exposures, whereas for meritocratic fairness, exposures can be defined based on relevance (e.g., using a relevance vector). In other embodiments, the optimal fairness type can additionally or alternatively be customized based on a predetermined ratio different from fairness (e.g., to allocate prize money).

[0026] An exemplary fairness typology considers individual fairness as opposed to group fairness: individual fairness seeks to ensure fairness from the level of individual objects, whereas group fairness assumes that objects are associated with groups and fairness is ensured from the group level, typically preventing some groups from being disadvantaged.

[0027] In step 207, server 100 outputs the Pareto set to a decision maker (internally or externally). In step 208, server 100 receives (internally or externally) the points of the Pareto set, which are converted to target exposures in the decision space. This target exposure corresponds to the exposure across the objects in the set. In one embodiment, the target exposure is defined by a decision maker, internal or external to server 100, who sets a defined utility / fairness tradeoff, which is set by a case-by-case or general fixed constraint (e.g., setting a constraint that fairness does not fall below a predefined threshold). In some embodiments, the decision maker can be a human administrator. In other embodiments, the decision maker can be an automated system. A combination of a human administrator and an automated system can also be used.

[0028] In step 210, using the target exposure received by the decision maker as the optimal tradeoff from the Pareto set, server 100 determines a distribution for ranks (e.g., a set of weighted ranks) that will, on average, achieve the target exposure for the objects in the set, where each rank in the distribution for ranks corresponds to a vertex in the decision space. In one embodiment, determining the distribution for ranks can be performed as follows: (i) an arbitrary vertex of the decision space is determined; (ii) a line is drawn (e.g., calculated) starting from the arbitrary vertex with the target exposure received from the decision maker until the line intersects with a face of the decision space; and (iii) steps (i) and (ii) are repeated on the intersected face of the decision space, substituting the target exposure of the decision maker for the intersection point, until the newly intersected face becomes a vertex. Steps (i) through (iii) can be performed for as many objects in the set, with the intersection point in step (iii) being a vertex. Each vertex of the decision space has an associated ratio.

[0029] In step 212, the server 100 distributes the distribution of ranks by selecting a sequence of ranks for the set of objects from the distribution of ranks according to their proportions.

[0030] 2 may be performed partially or completely from the client device 102. In other embodiments, the method may be performed on different servers or client devices or multiple servers in a distributed manner, as described above.

[0031] In one exemplary embodiment, a non-personalized query (e.g., a query generated by an anonymous user) is repeatedly received for general points of interest from a predetermined map location. In this exemplary embodiment, the set of objects may be a list of general points of interest such as restaurants, museums, stores, and gas stations. In another exemplary embodiment, a non-personalized keyword query is repeatedly received by a search engine. In this exemplary embodiment, the set of objects may be a list of links to documents such as web pages. Repeated such queries over time result in a varying order of the set of objects with each ranking sequence to achieve target exposure.

[0032] In step 212, based on the selected ranking sequence, one, all, or a subset of the set of objects may be presented in a search engine results page (SERP) that may be provided, e.g., forwarded, to an external or internal device in preparation for presentation (e.g., displaying, publishing, printing, importing, exporting, saving, etc.). For example, a SERP including one, a subset, or all of the objects (such objects each positioned based on the determined ranking within the selected sequence) may be generated and forwarded to the server 100 or the client device 102 for display on a display. If one or a subset of the objects is presented in the SERP instead of all objects, such presented objects may be, for example, objects that each have a higher ranking in the selected sequence than other objects in the set of objects.

[0033] 3. Exposure (display) In the information retrieval (IR) field, exposure, also known as “attention” or “examination,” can be defined as the probability that a user examines an object (such as, but not limited to, a document) at a particular location on a search engine results page (SERP). Exposure values ​​(e.g., forming part of the list of exposures received in step 202(iii) of FIG. 2) can be captured by methods such as, but not limited to, eye-tracking studies or arbitration collection (e.g., placing the same document in two different rankings for the same query). Exposure values ​​can also be determined by methods using “click models” (i.e., user browsing models) known in the information retrieval technology field (e.g., as disclosed by Chuklin et al. in “Click models for web search,” Synthesis Lectures on Information Concepts, Retrieval, and Services, 7(3):1-115, July 2015). For the most part, such models attempt to explain clicking behavior through a natural sequence of two steps: first, inspect (or pay your attention to) a list or its representative snippets, and then click on it if it is interesting.

[0034] From a consumer (i.e., user) perspective, consumers desire more relevant objects to receive higher exposure so as not to waste time searching for relevant objects in less visible locations on the SERP. This consumer desire can be expressed in an exemplary way by defining ranking utility as the dot product between an exposure vector (i.e., a vector composed of the exposure values ​​provided by the rankings for each object) and a gain vector (i.e., a vector composed of the gains of each object, where the gains of an object are defined by an arbitrary monotonically increasing function of the object's relevance score). Without being limited thereto, well-known information search utility measures, such as Discounted Cumulative Gain (DCG) and Expected Reciprocal Rank (ERR), reflect this formula with specific selection of exposure model parameters. For example, DCG is expressed as follows: the exposure of an object from a rank (k) is given by 1 / log2(1+k), and the gain function g(d) is given by g(d)=2 rel(d) where rel(d) is the relevance score of d. The ERR measure is based on a cascade model, which assumes that exposure varies depending on the relevance of the previous object in the list; in particular, if a user is satisfied with an object, the exposure of the next object in the list will be 0.

[0035] From the supplier's perspective, they would like their objects to have higher visibility in the SERPs, however, due to the large number of suppliers, non-disparate treatment of the objects offered to users is desirable.

[0036] The consumer and provider perspectives, corresponding to the "utility" goal and the "fairness" criterion, respectively, can both be expressed in the exemplary method disclosed herein in terms of "exposure." Exposure serves as a link between the ideal ranking goal, i.e., the utility goal (e.g., representing the user or consumer's perspective) and the fairness criterion (e.g., representing the object provider or supplier's perspective).

[0037] 4. Permutohedron An exemplary permutohedron is a polytope, where each vertex corresponds to a particular ranking or permutation of n objects (e.g., documents), and the polytope is a convex hull of such vertices. This polytope is contained in n-dimensional space, but is actually (n-1)-dimensional. For example, Figure 3 shows a three-dimensional object 300 for n=4 objects, which appears like a sphere with a mixture of hexagonal and square facets. Consider vertex 4312 in 302. This shows a ranking that places d1 at rank 4, d2 at rank 3, d3 at rank 1, and d4 at rank 2. Further consider edge 306 connecting vertex 4312 in 302 with vertex 3412 in 304. This shows an arbitrary distribution (or convex combination) of the two rankings.

[0038] Also, all facets of the three-dimensional object 300 indicate a partial ordering of n objects into two groups. For example, facet 308, which includes vertices (4312), (3412), (2413), (2314), (3214), and (4213), indicates a partial ordering in which d3 is always first (i.e., d3@order 1), followed by three other documents in any order; facet 310, which includes vertices (3214), (2314), (1324), (1234), (2134), and (3124), indicates a partial ordering in which d4 is always last (i.e., d1 and d3 (i.e., d1 and d4 @ rank 4), indicating a partial order in which three other documents come before them out of order; regular square facet 312 containing vertices (2413), (1423), (1324), and (2314) indicates a partial order in which d1 and d3 are in any order for the first two positions, and d2 and d4 follow in any order for the last two positions (i.e., (d1 and d3) before (d2 and d4)).

[0039] In general, each facet of dimension (nk), which generalizes the concept of facets along a hierarchy of decreasing dimension (e.g., facet → edge → vertex for n=4), represents all possible distributions (or convex combinations) associated with a given partial ordering of n objects into k groups. Also, as shown in Figure 3, edge 306 connecting vertex (4312) in 302 with vertex (3412) in 304 corresponds to the following partial ordering: d3 first, d4 second, d1 and d2 following in any order.

[0040] 5.Disassembly According to Caratheodory's theorem, there are m points (v i )

[0041]

number

[0042] Considering a permutohedron as a special case, any point on a permutohedron (which is a d-dimensional object with d=(n-1), where n is the number of objects) can be decomposed into distributions for up to n ranks. In one embodiment, a procedure known as the GLS procedure (Grotschel, Lovasz, and Schrijver) can be used to determine one such decomposition (although generally more than one decomposition is possible). An example of a GLS procedure (see Grotschel et al., "Geometric Algorithms and Combinatorial Optimization," Springer Science & Business Media, published December 2012) is illustrated in FIG. 4, which shows an n=3 polytope applied to the method disclosed herein.

[0043] Figure 4(a) shows a point (x) to be decomposed in 402. As shown in Figure 4(b), the procedure begins by selecting an arbitrary vertex (v1) in 404 of polytope 400 (n=3) and extending a line segment from v1 toward x (the point to be decomposed 402) until it intersects with a face of dimension (n-2). This provides a new point (x'), where x is expressed as a convex combination of v1 and x' (i.e., x=λ1v1+(1-λ1)x').

[0044] As shown in Figure 4(c), a point (x') on an (n-2)-dimensional surface can be decomposed using the same procedure by selecting an arbitrary vertex (called v2) belonging to the x' surface and extending the line segment (v2-x') until it intersects with a surface of dimension (n-3). In the example shown in Figure 4(c), since n = 3, this surface is vertex (v3), and the procedure stops at this point, with the initial point (x) being represented as a convex combination of three vertices (i.e., x = λ1v1 + (1-λ1)λ2v2 + (1-λ1)(1-λ2)v3). In general, the higher the value of n, the more times such recursive steps are repeated, each time considering surfaces of decreasing dimension until a surface (i.e., a vertex) of dimension (0) is reached.

[0045] 6. Notation The exemplary ranking method described herein will now be described in more detail. The following notation will be used to formally describe the characteristics of the exemplary method: a) q is a query repeated infinitely by a single (e.g., anonymous) user (e.g., consumer); the query need not be a text query in the general "information search" sense, but should be understood in a broad sense; for example, the query can also include any trigger context in a "recommendation" setting.

[0046] b) Given a query (q), we have a fixed candidate set D = (d1,…,d n ), where the relevance score for a query (q) is ρ = (ρ1,…, ρ n ) is represented by; c) The ranking policy (π(q)) is a distribution (e.g., a convex combination) over m rankings, m<=n σ i (i = 1,…,m) where:

[0047]

number

[0048] 7. Optimizing ranking policies An exemplary method can use a ranking policy, denoted herein as π(q), that is useful and fair by expressing everything (e.g., utility for consumers and fairness for suppliers) in terms of a single set of decision / optimization variables, referred to herein as "control levers."

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053] Such optimization problems can be expressed as multi-objective optimization problems. A Pareto set (i.e., a set of feasible non-dominated solutions) can be determined by geometric reasoning. A particular tradeoff in this determined Pareto set can then be selected or determined, for example, by a decision maker, where the tradeoff is a single target point (i.e., target exposure) in the decision space. This point can be decomposed as a convex combination of up to n ranks, as known from Carathéodory's theorem mentioned above.

[0054] Once such combinations are determined, they (i.e., the ordering of the set of objects from the distribution relative to the rank) can be distributed by a fair scheduling strategy, which in one embodiment uses low-discrepancy sequences such as golden ratio low-discrepancy sequences (see "Weighted Round Robin (Weighted Random Integers) Using the Golden Ratio Low Discrepancy Sequence," published online at demofox.org in June 2020). Those skilled in the art should understand that other scheduling strategies can be used in alternative embodiments, such as, but not limited to, m-balanced word or similar algorithms similar to stride scheduling.

[0055] An advantage of the exemplary methods is that they can perform various steps, or most of the steps, using geometric reasoning, which in practice leads to simple algebraically closed-form solutions. An additional advantage of the exemplary methods is that they can be solved in O(n 2 Another advantage of the exemplary methods is that they provide a time complexity of, for example, n ! dimensional space and n 2 The difference is that instead of a one-dimensional space, it operates in an n-dimensional space. As a result, the number of decision / optimization variables used by the exemplary method is no greater than n, which allows any optimal solution to be embodied as a distribution over up to n ranks.

[0056] An exemplary method may be provided for a general type of exposure model referred to as a "Position-Based Model (PBM)." Such a family of models assumes that an object's exposure depends only on its rank. Each rank (k) is then associated with a parameter (γk), which indicates the probability that this rank will be viewed by a user. Other types of models can be processed using the exemplary method.

[0057] 8. PBM model and associated expohedrons In an embodiment, the exemplary method is

[0058]

number

[0059]

number

[0060] Any point within the expohedron shown in Figure 5 represents a specific exposure vector, which can be realized with up to three different permutation or rank distributions. The centroid 502 of the hexagon 500 corresponds to the same exposure (i.e., demographic fairness) to each object (e.g., document), but can be realized as a uniform distribution across the three different ranks.

[0061] There is a simple mathematical method to check whether a point belongs to an expohedron, and this method will be utilized as the exemplary method described herein. This is known in mathematics as the majorization condition, and when ∈ is majored by γ (which means

[0062]

number

[0063]

number

[0064] The mathematical definition of majorization is as follows:

[0065]

number

[0066]

number

[0067] As used herein, a "region" is defined as a set of points with the same ordered coordinates (i.e., exposure). More specifically, such points can correspond to vectors such that the indices of their components, sorted in ascending order, are identical. For example, in Python, this corresponds to the arrays in which the output of the argsort function is the same. An expohedron has as many regions as vertices, with each region containing only one vertex. An exemplary region can be viewed as an unbounded pyramid, with its vertices being its centroid and its semi-axes corresponding to the lines connecting the centroid of the expohedron to the centroid of each facet adjacent to a unique vertex that the region contains.

[0068]

number

[0069] 9. Usefulness in PBM-Expohedron An exemplary utility criterion specifies that objects with high relevance scores, or more generally high gain, should have higher exposure. Without loss of generality, ρ is

[0070]

number

[0071] Utility can be expressed, for example, as the dot product between the relevance vector and the exposure vector:

[0072]

number

[0073]

number

[0074] Given this mathematical expression of utility, the expohedron's maximum utility ranking policy is its projection onto ρ, i.e.

[0075]

number

[0076] If the relevance vectors have a tie (i.e., if at least two elements of the relevance vector are the same), then the overall surface 802 constitutes a set of maximum utility solutions, as shown in FIG. 8 with the tie (as shown by the relevance vector [0.7, 0.7, 0.4]).

[0077] 10. Fairness in PBM-Expohedron An exemplary fairness criterion is expressed in the expohedron framework. The individual demographic fairness criterion specifies that ideally, all objects (e.g., documents) should have the same exposure. Since the sum of exposures is a constant, this means that the target exposure of a demographic fairness policy is the centroid of the expohedron:

[0078]

number

[0079] Therefore, the fairness criterion can be defined as a quadratic function, e.g., proximity (or distance reduction) to the centroid:

[0080]

number

[0081] Considering meritocratic fairness, the ideal exposure vector should be proportional to the relevance vector or merit vector, commonly denoted ρ′, where the merit of an object is defined as a monotonically increasing function of this object's relevance score or equivalently its gain (

[0082]

number

[0083]

number

[0084] In an expohedron (n=3) 900 as shown in Figure 9, where ρ and ρ' are chosen to be identical, for demographic fairness 902 and meritocratic fairness 904, the equivalent fairness surface is a hypersphere centered at β or ρ, and the equivalent fairness curves for optimal demographic and meritocratic fairness are circles surrounding the optimal fairness points 902, 904, respectively, where vector 906 is the relevance vector.

[0085] If the merit vector is outside the expohedron, i.e.,

[0086]

number

[0087] Figure 10 shows an example where the optimal point 1004 identified by merit vector 1002 is outside of the expohedron 1000 and is therefore infeasible. This new feasible target vector is shown at point 1006, which can be calculated as follows:

[0088]

number

[0089] The intersection of the relevance vector axis and the boundary of the expohedron is the normalized merit

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094]

number

[0095] An alternative solution to the affine transformation for selecting a viable meritocratic fairness point is to compute an orthogonal projection, the results of which for the example shown in FIG.

[0096] 11. Pareto Set in PBM-Expohedron Once we determine how to compute and optimize utility and fairness separately, we can compute the complete Pareto set of a multi-objective utility-fairness problem.

[0097] To explain the method intuitively and geometrically, an exemplary computational method starts with one extreme of the Pareto set, i.e., a "totally fair" solution. Then, to plot (e.g., compute) the entire Pareto set, we (computationally) follow the payoff vector direction. It turns out that all these points are not dominated by other points in the expohedron and correspond to optimal tradeoffs between fairness and utility. If we always follow the payoff vector direction, the boundary of the expohedron will necessarily intersect at a certain point, meaning the solution is no longer feasible. Next, we (computationally) follow the payoff vector direction projected onto the directly intersected (n-2)-dimensional facet. Following this new direction results in a new facet, now (n-3)-dimensional, being intersected, and we once again follow the payoff vector direction projected onto the new facet. Finding the projection of the payoff vector on any facet corresponds to a closed-form expression. This path-tracing procedure is computationally repeated again until we finally arrive at a non-dominated maximum-utility solution. All points along the path correspond to strongly non-dominated solutions of the utility-fairness trade-off; in other words, the path is a Pareto set.

[0098] The path-tracing procedure described above is illustrated in FIG. 11 for n=3 and the gain and merit functions chosen as identity functions (i.e., a document's merit and gain are equal to its relevance score). Pareto sets are defined from 1102 to 1110 when the demographic fairness criterion is used, and from 1106 to 1110 when the meritocratic fairness criterion is used. When the demographic fairness criterion is used, the Pareto set starts from the graph's centroid 1102, follows the direction of line segment 1104 (which is the gain vector axis), passes through meritocratic fairness point 1106 on the graph, intersects with edge 1108, or facet, corresponding to the mixed ranking where objects (e.g., documents) d1 and d2 are initially permuted in two ranks, and then follows this facet until it reaches the maximum utility solution, represented on the graph by star 1110. When the meritocratic fairness criterion is used, the Pareto set should start from point 1106.

[0099] More formally, the Pareto optimal set is, for i = 1,...,(n-1),

[0100]

number

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105] One embodiment of a Pareto set construction method is illustrated in Figure 12 as an intention code. More specifically, Figure 12 illustrates an intention code for Pareto set identification, which (without loss of generality) is the payoff vector

[0106]

number

[0107]

number

[0108] 12. GLS Decomposition in PBM-Expohedron As described herein, for example, a point in the Pareto set that can be selected by a decision maker can be transformed into a target exposure in the decision space. This point in the Pareto set can be decomposed into a convex combination of up to n ranks. To address this decomposition problem, an exemplary method specializes the general GLS procedure described above and applies it to the construction of an exponential polytope.

[0109] One embodiment for implementing the decomposition is illustrated as the inference code in Figure 6. The decomposition method in Figure 6 takes advantage of the fact that if a point is on a facet (or a plane of lower dimensionality), then the vertices on this facet (or plane) can be found by selecting the vertices of the region in which the point is located (this can be done by sorting the components of the point) (see line 5 in Figure 6). Note that this step has a complexity of O(n log n).

[0110] The exemplary method in line 9 of FIG. 6 uses v (i) → x (i) Find the point where the line segment is extended and call this intersection point (x (i+1) ) defines a new point that is recursively decomposed. At each iteration of the loop, a new point (x (i+1) ) is on a plane of dimension (n i ), which means that the final point is always a vertex (dimension=0). Since finding the intersection point is the same as finding the root of a "majorization" condition (i.e., the point where the condition value goes from true to false), a bisection method (i.e., a numerical method for finding the root of a provided function) is applied using the majorization criterion in line 9 of FIG. 6 to determine when to stop the method. The majorization criterion can also be used to find an efficient upper endpoint at the intersection with the plane (λ) (here, efficient upper endpoint is used herein to mean "not too far from the true intersection point"). This endpoint calculation is performed in lines 6 to 8 of FIG. 6.

[0111] The bisection itself has a number of iterations (e.g., 5-10 iterations) that are independent of n. Checking the priming condition within the bisection requires O(n log n) complexity, so the overall complexity of the method is O(n 2 log n).

[0112] 13. Policy Distribution with Low Discrepancy Sequences When decomposing the target exposure into a rank distribution, any of a variety of methods can be used, alone or in combination, to distribute the distribution in the form of a rank sequence.

[0113] For example, stochastic sampling (i.e., a random number generator) can be used to distribute the distribution for the ranks. In other embodiments, low-discrepancy sequences can be used (see, e.g., Martin Roberts, "The unreasonable effectiveness of quasirandom sequences," Apr 2018).

[0114] A low discrepancy sequence (LDS) is a set of sequences with ranks R1,R2,…,R for all t. t The subsequences are provided to have low discrepancies (i.e., the ratio of ranks is close to the desired ratio, i.e., the ratio of an infinite sequence). A low discrepancy sequence is typically a pseudorandom sequence in the interval [0, 1] that is as close as possible to a uniform distribution, and such a floating-point sequence in [0, 1] can be converted into a ranked sequence with the desired ratio by comparing the generated floating-points with the cumulative value of the ratio.

[0115] The use of additive-recursive sequences based on irrational numbers (referred to as Kronecker, Weyl, or Richtmyer sequences) can be used in embodiments, particularly for golden numbers, which are in some sense the most irrational number. The general recursive form of the sequence is:

[0116]

number

[0117]

number

[0118] A range of efficient sampling strategies other than low discrepancy sequences can alternatively or additionally be used, for example, stride scheduling or equivalently strategies based on m-balanced words can also be used and provide very similar performance.

[0119] Expressed in terms of an exemplary problem, the m-balanced word generator generates a sequence of ranks from any pair of subsequences of the same length, where the frequency of any rank varies by at most m. In other words, the generator ensures that the generated sequence produces ranks with a ratio as close as possible to the target ratio. While not wishing to be bound by theory, the theoretically best achievable m is at most n-1 in the exemplary case. An exemplary algorithm that can efficiently generate an m-balanced sequence of ranks when a specific distribution of ranks is provided is provided in Algorithm (1) of the reference [Shinya Sano, Naoto Miyoshi, and Ryohei Kataoka. 2004. m-Balanced words: A generalization of balanced words. Theoretical Computer Science 314, 1-2 (Feb. 2004), 97-120. https: / / doi.org / 10.1016 / j.tcs.2003.11.021]. This generator is similar to known stride scheduling algorithms used to generate fair sequences in resource (CPU) management for concurrent processes, as described, for example, in [CA Waldspurger and E. Weihl. W. 1995. Stride Scheduling: Deterministic Proportional-Share Resource Management. Technical Report. Massachusetts Institute of Technology, USA].

[0120] 14. Computer Program Products The described exemplary method for ranking objects may be provided as a computer program product, which may include code instructions for executing the method (e.g., using the data processor 112 of the server 100 and the client device 102), and storage means readable by computer equipment provided with the computer program product (e.g., using the memory 113 of the server 100 and the client device 102) for storing such code instructions.

[0121] 15.General The foregoing description is merely exemplary in nature and is not intended to limit the present disclosure, its application, or uses. The broad teachings of the present disclosure can be embodied in a variety of forms. Accordingly, while the present disclosure includes specific examples, the true scope of the present disclosure should not be limited thereto. This is because other variations will become apparent upon study of the drawings, the specification, and the claims. It should be understood that one or more steps within a method may be performed in a different order (or simultaneously) without altering the principles of the present disclosure. Also, although each embodiment is described above as having certain features, any one or more of these features described in connection with any embodiment of the present disclosure may be embodied / embodied with or combined with features of any other embodiment, although such combinations are not expressly stated. In other words, the described embodiments are not mutually exclusive, and permutations of one or more embodiments with each other are within the scope of the present disclosure. Any references cited herein are incorporated herein by reference in their entirety without any admission that such references constitute prior art.

[0122] Each module may include one or more interface circuits. In some examples, the interface circuits may include wired or wireless interfaces connected via a local area network (LAN), the Internet, a wide area network (WAN), or a combination thereof. The functionality of any provided module in this disclosure may be distributed across multiple modules connected by interface circuits. For example, multiple modules may allow for load balancing. In other examples, a server (remote or cloud) module may perform some functionality on behalf of a client module. Each module may be embodied using code. The term code, as used herein, may include software, firmware, and / or microcode and may refer to programs, routines, functions, classes, data structures, and / or objects.

[0123] Memory circuits herein are a subset of computer-readable media. Computer-readable media herein do not include transient electrical or electromagnetic signals propagated through a medium (e.g., on a carrier wave); therefore, computer-readable media can be considered a type and non-transitory. Non-limiting examples of non-transitory and types of computer-readable media include non-volatile memory circuits (e.g., flash memory circuits, erasable programmable read-only memory circuits, or mask read-only memory circuits), volatile memory circuits (e.g., static random access memory circuits, dynamic random access memory circuits), magnetic storage media (e.g., analog or digital magnetic tape or hard disk drives), and optical storage media (e.g., CDs, DVDs, or Blu-ray discs).

[0124] The systems and methods described in this application may be implemented partially or completely by a special-purpose computer created by configuring a general-purpose computer to perform one or more specific functions embodied in a computer program. The functional blocks, flowchart components, and other elements described above serve as software specifications that can be converted into a computer program by the routine work of a skilled engineer or programmer.

[0125] A computer program includes at least one processor-executable instruction. A computer program may also include or depend on stored data. A computer program may include a basic input / output system (BIOS) that interacts with hardware in a special-purpose computer, device drivers that interact with specific devices in the special-purpose computer, one or more operating systems, user applications, background services, background applications, etc.

[0126] It should be understood that variations of the above-described embodiments and other features and functions, or alternatives thereof, may be preferably combined in many different systems or applications, and that various presently unforeseeable alternatives, modifications, variations, or improvements may subsequently be made by those skilled in the art, all of which are within the scope of the foregoing description and the appended claims.

Claims

1. 1. A computer-implemented method for ranking a set of objects, comprising: a processor of the computer receiving the set of objects and a set of objective functions; the processor using a permutohedron associated with a position-based model (PBM) to define a decision space having n decision variables, the decision variables being variables to be optimized for ranking the exposure of objects corresponding to information search results, the decision space being a space defined by the decision variables, n being the number of objects to be ranked; determining a Pareto set corresponding to a multi-objective optimization (MOO) problem for the set of multiple objective functions, the Pareto set being a path from a centroid of a facet determined based on a convex combination of associated exposure rankings from the decision space, intersecting edges and reaching a vertex of the facet, the determining step being determined using geometric reasoning; the processor using the decision space to determine a distribution for ranks for the objects in the set using a GLS (Grotschel, Lovasz and Schrijver) procedure, with Pareto points in the Pareto set, wherein a coefficient is associated with each rank in the distribution; the processor selecting a set of ranks for the objects in the set from a distribution for the ranks by a coefficient, wherein selecting a set of ranks for the objects in the set includes using one or more of stochastic sampling, low discrepancy sequences, additive repeat sequences, stride scheduling, or m-balance; the processor outputting the selected set of rankings; and the processor receiving a list of exposures to be provided to the objects in the set, each associated with a relevance score and a rank for each object in the set, a ranking fairness objective function, and a ranking utility objective function; the set of objective functions includes a quadratic function and a linear function; The computer-implemented method, wherein the quadratic function comprises a fairness function, the linear function comprises a utility function, the fairness function being a normalized function of the difference between a vector composed of the decision variables and a target vector, and the utility function being a linear function that indicates a high relevance score for a user accessing a set of results.

2. 2. The computer-implemented method of claim 1, wherein the set of multiple objective functions includes a fairness function and a utility function, the fairness function being a normalized function of the difference between a vector composed of the decision variables and a target vector, and the utility function being a linear function that indicates a higher relevance score accessed by a user.

3. The computer-implemented method of claim 1 or 2, wherein the fairness function is a normalized function.

4. 4. The computer-implemented method of claim 3, wherein the normalized function is a squared L2-norm function.

5. 2. The computer-implemented method of claim 1, wherein the processor determining the Pareto set for the set of objective functions comprises the processor calculating the Pareto set using Carathéodory's theorem within the decision space using the ranking fairness objective function and the ranking utility objective function.

6. The step of the processor using the decision space to determine a distribution for the ranks for the objects in the set includes, with respect to the number of objects in the set: (i) the processor determining an arbitrary vertex of the decision space; (ii) the processor drawing the line starting from the arbitrary vertex with a target exposure until the line intersects a face of the decision space; and (iii) the processor repeats (i) and (ii) on the intersected face of the decision space using the new intersection point in place of the target exposure until the newly intersected face becomes a vertex; The computer-implemented method of claim 5 , wherein each vertex of the decision space has an associated coefficient in the distribution for the rank.

7. the set of objective functions includes a fairness function and a utility function; The computer-implemented method of claim 1 , wherein the fairness function is based on meritocratic fairness, which can be defined as proximity to a normalized relevance vector.

8. the set of objective functions includes a fairness function and a utility function; The computer-implemented method of claim 1 , wherein the fairness function is based on demographic fairness corresponding to equal exposure to each object.

9. The computer-implemented method of claim 1 , wherein the objects in the set include multiple queries.

10. The computer-implemented method of claim 1 , wherein the objects in the set include a plurality of documents.

11. The computer-implemented method of claim 1 , wherein the objects in the set are identified in response to a query.

12. The computer-implemented method of claim 1 , wherein the objects in the set include multiple recommendations.

13. The computer-implemented method of claim 1 , wherein the objects in the set are identified according to map location.

14. The step of outputting by the processor includes:

2. The computer-implemented method of claim 1, further comprising the step of the processor providing at least a subset of the set of objects ranked according to the selected ranking sequence for display on a display of a device.

15. The computer-implemented method of claim 1 , wherein the at least a subset of the set of objects is provided on a search engine results page.

16. A computer program comprising non-transitory machine-readable instructions, The instructions, when executed on a computer, cause a processor of the computer to perform the following method, which method includes: receiving a set of a plurality of objects and a set of a plurality of objective functions; defining a decision space having n decision variables using a palmtohedron associated with a position-based model (PBM), the decision variables being variables to be optimized for ranking the exposure of objects, the decision space being a space defined by the decision variables corresponding to information search results, n being the number of objects to be ranked; determining a Pareto set corresponding to a multi-objective optimization (MOO) problem for the set of multiple objective functions, the Pareto set being a path from a centroid of a facet determined based on a convex combination of associated exposure rankings from the decision space, intersecting edges to reach a vertex of the facet, the determining step being determined using geometric reasoning; taking Pareto points within the Pareto set and using the decision space to determine a distribution for ranks for the objects in the set using a GLS (Grotschel, Lovasz and Schrijver) procedure, wherein a coefficient is associated with each rank in the distribution; selecting a set of ranks for the objects in the set from a distribution for the ranks by a coefficient, wherein selecting a set of ranks for the objects in the set includes using one or more of stochastic sampling, low discrepancy sequences, additive repeat sequences, stride scheduling, or m-balance; outputting the selected set of rankings; and receiving a list of exposures to be provided to the objects in the set, each associated with a relevance score and a ranking for each object in the set, a ranking fairness objective function, and a ranking utility objective function.

17. 1. A data processing device including a processor configured to perform a method, comprising: The method comprises: the processor receiving a set of a plurality of objects and a set of a plurality of goal functions; the processor using a palmtohedron associated with a position-based model (PBM) to define a decision space having n decision variables, the decision variables being variables to be optimized for ranking the exposure of objects corresponding to information search results, the decision space being a space defined by the decision variables, n being the number of objects to be ranked; determining a Pareto set corresponding to a multi-objective optimization (MOO) problem for the set of multiple objective functions, the Pareto set being a path from a centroid of a facet determined based on a convex combination of associated exposure rankings from the decision space, intersecting edges and reaching a vertex of the facet, the determining step being determined using geometric reasoning; the processor taking Pareto points within the Pareto set and using the decision space to determine a distribution for ranks for the objects in the set using a GLS (Grotschel, Lovasz and Schrijver) procedure, wherein a coefficient is associated with each rank in the distribution; the processor selecting a set of ranks for the objects in the set from a distribution for the ranks by a coefficient, wherein selecting a set of ranks for the objects in the set includes using one or more of stochastic sampling, low discrepancy sequences, additive repeat sequences, stride scheduling, or m-balance; the processor outputting the selected set of rankings; and A data processing device comprising: the processor receiving a list of exposures to be provided to the objects in the set, each associated with a relevance score and a ranking for each object in the set, a ranking fairness objective function, and a ranking utility objective function.

18. 20. The data processing device of claim 17, wherein the data processing device is integrated into a second step ranker for a two-step information retrieval system.

19. 19. A data processing device according to claim 17 or 18, wherein the data processing device comprises a server.

20. 19. A data processing device according to claim 17 or 18, wherein the data processing device comprises a client device.

21. An information retrieval system, comprising: a first-step searcher implemented on a computer configured to receive a query and generate a set of a plurality of objects; and a computer-implemented second step ranker; wherein the computer-implemented second step ranker comprises: receiving the set of objects and the set of objective functions; using a palmtohedron associated with a position-based model (PBM) to define a decision space having n decision variables, the decision variables being variables to be optimized for ranking the exposure of objects corresponding to information search results, the decision space being a space defined by the decision variables, n being the number of objects to be ranked; determining a Pareto set corresponding to a multi-objective optimization (MOO) problem for the set of multiple objective functions, the Pareto set being a path from a centroid of a facet determined based on a convex combination of associated exposure rankings from the decision space, intersecting an edge, and reaching a vertex of the facet, the determining being determined using geometric reasoning; determining a distribution for ranks for the objects in the set using a GLS (Grotschel, Lovasz and Schrijver) procedure, with Pareto points in the Pareto set and using the decision space, wherein a coefficient is associated with each rank in the distribution; selecting a set of ranks for the objects in the set from a distribution for the ranks by a coefficient, wherein selecting a set of ranks for the objects in the set includes using one or more of stochastic sampling, low discrepancy sequences, additive repeat sequences, stride scheduling, or m-balance; outputting the selected series of rankings; An information retrieval system configured to receive a list of exposures to be provided to the objects in the set, each associated with a relevance score and a rank for each object in the set, a ranking fairness objective function, and a ranking utility objective function.

22. 22. The information retrieval system of claim 21, wherein the outputting includes the ranker providing at least a subset of the set of objects ranked by the selected ranking sequence for display on a display of a device.

23. 23. The information retrieval system of claim 22, wherein the at least a subset of the set of objects is provided in a search engine results page for display on a server or a client device display.

Images