Method and device for recommending object, electronic equipment and medium

By using thresholds and candidate object probability in the recommendation system to determine the recommended object, the sorting fairness problem is solved, the diversity and accuracy of the sorting results are achieved, and the exposure opportunities of different objects are ensured.

CN119025746APending Publication Date: 2024-11-26FACE CUTE CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310605779.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-11-26

Smart Images

  • Figure CN119025746A_ABST
    Figure CN119025746A_ABST
Patent Text Reader

Abstract

The embodiment of the invention relates to a method and device for recommending objects, electronic equipment and a medium. The method includes determining a scored set of objects associated with a query, the scored set of objects including a plurality of objects and a plurality of scores corresponding thereto. The method further includes determining a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position. In addition, the method further comprises the step of determining a recommended object for the target sorting position based on the candidate object probability of the candidate object set. According to the method disclosed by the embodiment of the invention, while the overall accuracy of the sorting system is ensured, the sorting list generated by the sorting system has certain randomness and diversity, so that the sorting fairness can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure generally relates to the field of computers, and more particularly, to a method, apparatus, electronic device, and medium for recommending an object. Background Art

[0002] Recommendation systems are systems designed to recommend things to users based on many different factors. Recommendation systems can predict things that users are most likely to like or be interested in. For example, some movie information platforms are using recommendation systems to help users of the platform quickly identify movies that the user may like from a large number of movies, and then recommend the movie to the user by changing the order or display position of the movie, so that the user can quickly find and watch the movie he likes.

[0003] Sorting is an important part of the recommendation system, which aims to sort a set of objects according to a given query. The sorting result determines the order and arrangement of objects in the recommendation list. By sorting the objects most relevant to the query at the front, the probability of users finding and selecting objects of interest can be increased. How to improve the fairness and accuracy of sorting is the main problem that the recommendation system needs to face. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device and medium for recommending objects. When determining an object for a target ranking position, by screening a suitable candidate object set from an object set including more objects, and then determining the recommended object for the target ranking position according to the probability of each candidate object in the candidate object set, the fairness of the ranking can be improved while ensuring the accuracy of the ranking, and the diversity of the recommendation results can be increased to meet the needs and preferences of different users.

[0005] In a first aspect of an embodiment of the present disclosure, a method for recommending an object is provided. The method includes determining a scored object set associated with a query, the scored object set including a plurality of objects and a plurality of scores corresponding thereto. The method also includes determining a candidate object set for a target ranking position based on the scored object set and a threshold for the target ranking position. In addition, the method also includes determining a recommended object for the target ranking position based on a candidate object probability of the candidate object set.

[0006] In a second aspect of an embodiment of the present disclosure, a device for recommending an object is provided. The device includes a score determination module configured to determine a scored object set associated with a query, the scored object set including a plurality of objects and a plurality of scores corresponding thereto. The device also includes a candidate determination module configured to determine a candidate object set for a target ranking position based on the scored object set and a threshold for the target ranking position. In addition, the device also includes an object determination module configured to determine a recommended object for the target ranking position based on the candidate object probabilities of the candidate object set.

[0007] In a third aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement a method for recommending objects. The method includes determining a scored object set associated with a query, the scored object set including multiple objects and their corresponding multiple scores. The method also includes determining a candidate object set for a target ranking position based on the scored object set and a threshold for the target ranking position. In addition, the method also includes determining a recommended object for the target ranking position based on the candidate object probability of the candidate object set.

[0008] In a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, a method for recommending an object is implemented. The method includes determining a scored object set associated with a query, the scored object set including multiple objects and their corresponding multiple scores. The method also includes determining a candidate object set for a target ranking position based on the scored object set and a threshold for the target ranking position. In addition, the method also includes determining a recommended object for the target ranking position based on the candidate object probabilities of the candidate object set.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure may be implemented;

[0012] Figure 2 A flowchart of a method for recommending an object according to some embodiments of the present disclosure is shown;

[0013] Figure 3A-3B A schematic diagram showing a process for determining an object for a target ranking position according to some embodiments of the present disclosure;

[0014] Figures 4A-4B A schematic diagram illustrating a process for determining multiple objects for multiple target ranking positions according to some embodiments of the present disclosure is shown;

[0015] Figure 5 A schematic diagram showing a process for determining a suitable threshold sequence from a plurality of candidate threshold sequences according to some embodiments of the present disclosure;

[0016] Figure 6 A block diagram showing an apparatus for recommending an object according to some embodiments of the present disclosure; and

[0017] Figure 7 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0018] It is understood that all user-related data involved in this technical solution should be obtained and used after the user's authorization. This means that in this technical solution, if the user's personal information needs to be used, the user's explicit consent and authorization are required before obtaining this data, otherwise the relevant data will not be collected and used. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly observed during the collection, use and storage of data, and necessary technologies and measures should be taken to protect the user's data security and ensure the safe use of data.

[0019] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects, unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0021] A multi-party platform is a platform model that connects multiple different parties, where there can be three or more parties. The goal of a multi-party platform is to promote interaction and value exchange among multiple parties and achieve a win-win situation for all parties. For example, a multi-party platform can be a content distribution platform, where creators and readers are two participants. The content distribution platform, as another participant, can provide various functions and services to creators and readers to facilitate interaction between them. A multi-party platform needs to balance the interests and needs of different participants to ensure the sustainable development of the platform.

[0022] The platform can use a ranking system to display the content and services (also referred to as objects in this article) provided by one participant to another participant. The platform can use a ranking system to rank objects so that the most relevant options are presented to the user. The ranking system can be composed of a scoring model and a ranking model. For a given query and a set of objects that may be related to the query, the scoring model can predict a ranking score for each object, where the ranking score can indicate the relevance of the object to the given query. Then, the ranking model can use these ranking scores to sort the objects in the object set to obtain a ranked recommendation list. The ranking model in the related art sorts the objects from high to low according to the relevance score, thereby generating a ranked recommendation list. However, this traditional ranking model may lead to a lack of fairness in the exposure allocation of objects. For example, when two different objects have very close relevance scores for the same query, the traditional ranking model always generates the same ranking list, that is, the object with a higher relevance score is always ranked before the object with a lower score. In this case, although the relevance scores of the two are only slightly different, the object with a higher score will always get more exposure, which leads to a lack of fairness in the exposure allocation of the platform to the object.

[0023] To this end, according to various embodiments of the present disclosure, a scheme for recommending objects is provided. The scheme uses a deterministic scoring model to score the relevance of objects in an object set to a given query, and then determines a candidate object set for a target ranking position based on the relevance scores of the objects in the object set and a threshold for the target ranking position. The scheme can determine the recommended objects for the target ranking position based on the probability of the candidate objects in the candidate object set. By determining the candidate object set using the threshold for the target ranking position, the overall accuracy of the ranking system can be ensured. At the same time, by determining the recommended objects for the target ranking position based on the probability of the candidate objects, the ranking list generated by the ranking system can have a certain degree of randomness and diversity, thereby improving the fairness of the ranking.

[0024] Figure 1 1 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented. Figure 1 As shown, the environment 100 includes a user 102 and a platform 106, the platform 106 includes an object library 108, and the object library 108 stores a plurality of objects that can be used to recommend to the user 102. For example, the platform 106 can be a content distribution platform, the objects in the object library 108 can be news or articles, and the user 102 can browse the news or articles recommended to him on the platform 106. In the environment 100, the platform 106 also includes a search engine 110, a scoring model 114, and a ranking model 118. The user 102 can send a query 104 to the platform 106. After receiving the query 104, the platform 106 can use the search engine 110 to search the object library 108, so as to determine the objects 112-1, 112-2, ..., 112-N (collectively or individually referred to as objects 112) in the object library 108 that are related to the query 104. Then, the platform 106 uses the scoring model 114 to score the relevance of these objects 112 to the query 104, thereby generating scores 116-1, 116-2, ..., 116-N (collectively referred to as scores 116) and a scored object set 122, wherein object 112-1 has score 116-1, object 112-2 has score 116-2, ..., object 112-N has score 116-N. Objects with higher scores have higher relevance to the query 104. For example, in the scored object set 122, the score 116-1 of object 112-1 is the highest, the score 116-2 of object 112-2 is lower than the score 116-1 of object 112-1, the score 116-N of object 112-N is lower than the score 116-2 of object 112-2, and the scores of other objects in the scored object set 122 are all lower than the score 116-N of object 112-N.

[0025] exist Figure 1In the illustrated environment 100, the platform 106 uses the ranking model 118 to rank the objects in the scored object set 122. For example, the ranking model 118 may first determine the object ranked first (the frontmost ranking position) from all objects in the scored object set 122, and then may determine the next object from the remaining objects, and so on. In some traditional ranking systems, the ranking model will always rank the object with the highest relevance score first, and the object with the second highest relevance score second. However, in the examples provided in the present disclosure, the object with the highest relevance score is not necessarily always ranked first. For example, in Figure 1 In the example shown, the ranking model 118 outputs a recommendation list 120, in which the object 112-2 with a lower score is ranked first (the top position in the list), and the object 112-1 with a higher score is ranked second. Then, the platform 106 can expose the recommendation list 120 to the user 102 for the user 102 to interact with the object 112. In the embodiments provided by the present disclosure, the object ranking of the recommendation list generated by the ranking model 118 is not always the same as shown in the recommendation list 120. When the platform 106 receives two identical queries from different users, the ranking generated by the ranking model 118 may be different, and even when the platform 106 receives two identical queries from the same user, the ranking generated by the ranking model 118 may be different.

[0026] It should be understood that Figure 1 The platform 106 in the illustrated environment 100 may be any platform that uses a recommendation system to recommend objects to users, such as a content distribution platform, a movie information platform, a job search information platform, an e-commerce platform, etc. The objects in the object library 108 and the object 112 may be any objects that use a recommendation system to recommend objects to users, such as articles, movies, positions, items, etc.

[0027] Figure 2 FIG. 2 is a flow chart of a method 200 for recommending an object according to some embodiments of the present disclosure. Figure 2 As shown, at block 202, method 200 determines a scored set of objects associated with a query, the scored set of objects comprising a plurality of objects and a plurality of scores corresponding thereto. Figure 1 In the illustrated environment 100, the platform 106 uses the scoring model 114 to score each object 112, thereby generating a scored object set 122. In the scored object set 122, each object 112 has a corresponding score 116, wherein objects 112 with higher scores are more relevant to the query 104.

[0028] At block 204, method 200 determines a set of candidate objects for a target ranking position based on the set of scored objects and a threshold for the target ranking position. Figure 1 In the environment 100 shown, the scored object set 122 is input to the recommendation model 118. The recommendation model 118 can determine the target ranking position. In the recommendation system, objects that are often ranked first can get more exposure, so that users are more likely to interact with them. Therefore, in some examples, the target ranking position may be the first (the top ranking position). In other examples, one or more top ranking positions may be reserved by a specific service provider, in which case the target ranking position may be the second or third. For the target ranking position, the ranking model 118 needs to determine the object ranked at the target ranking position from the objects 112. In order to ensure the accuracy of the ranking, the ranking model 118 can use the threshold for the target ranking position to screen suitable candidate objects from the scored object set 122, thereby forming a candidate object set for the target ranking position.

[0029] At block 206, method 200 determines a recommended object for a target ranking position based on the candidate object probabilities of the candidate object set. Figure 1 In the environment 100 shown, the ranking model 118 needs to further determine the object for the target ranking position from the candidate object set determined at block 204. In order to prevent the ranking model 118 from always generating the same ranking, the probability of each candidate object being determined to be ranked at the target ranking position can be determined based on the scores of the objects in the candidate object set. The method 200 can determine the recommended object for the target ranking position based on these probabilities. Figure 1 In the example shown, although the score 116 - 2 of the object 112 - 2 is lower than the score 116 - 1 of the object 112 - 1 , since the score 116 - 2 is higher than the threshold for the first place, the object 112 - 2 also has a chance to be ranked first, thereby improving the fairness of the entire ranking.

[0030] By using the threshold for the target ranking position to determine the candidate object set, the overall accuracy of the ranking system can be ensured. At the same time, by determining the recommended object for the target ranking position based on the probability of the candidate objects, the ranking list generated by the ranking system can have a certain degree of randomness and diversity, thereby improving the fairness of the ranking.

[0031] Since the object ranked first can often obtain most of the exposure and user attention, in some embodiments, the improved ranking scheme may be adopted only for the top ranking position. Figure 3A-3BA schematic diagram of a process for determining an object for a target ranking position according to these embodiments is shown. In some other embodiments, the platform hopes to improve the exposure fairness of the top several ranking positions while ensuring the ranking accuracy, so an improved ranking scheme can be adopted for the top several ranking positions. Figures 4A-4B A schematic diagram of a process for determining multiple objects for multiple target ranking positions according to these embodiments is shown. Some embodiments of the present disclosure also provide a solution for determining a suitable threshold for several target ranking positions. Figure 5 A schematic diagram of a process for determining a suitable threshold sequence from a set of candidate threshold sequences according to some embodiments of the present disclosure is shown.

[0032] In order to determine a set of candidate objects for a target ranking position from a set of scored objects, in some embodiments, for an object whose ranking position has not been determined in the set of scored objects, the object is added to the candidate set in response to the score of the object being greater than or equal to a threshold for the target ranking position.

[0033] Figure 3A FIG. 3 is a schematic diagram showing a process 300 of determining a set of candidate objects for a target ranking position from a set of objects according to some embodiments of the present disclosure. Figure 3A As shown, the goal of process 300 is to determine the object at sort position 311 in recommendation list 310. Sorting position 311 may be, for example, the topmost position in recommendation list 310. Figure 3A In the example, object collection 306 is a collection of objects that have been retrieved from an object repository (e.g., Figure 1 The object library 108 in the query 308 is filtered by the object library 108, for example, the objects 301-305 in the object collection 306 can be associated with Figure 1 The object 112 in corresponds to. Figure 3A As shown, process 300 inputs query 308 and object set 306 into scoring model 314. Scoring model 314 can score each object in object set 306. The score indicates the relevance of the object to query 308. The higher the score, the higher the relevance of the object to query 308. Scoring model 314 can be any model that can generate or predict the relevance score of an object to a query, such as collaborative filtering, content-based recommendation, matrix decomposition, deep learning methods, combination methods, etc. Figure 3A In the example shown, the score of object 301 is 0.8, the score of object 302 is 0.6, the score of object 303 is 0.9, the score of object 304 is 0.7, and the score of object 305 is 0.5.

[0034] It should be understood that for the sake of simplicity, Figure 3AIn the example shown, only objects 301-305 are shown, but this example is not intended to limit the size of object set 306. For example, object set 306 may include a smaller size or a larger size (e.g., hundreds, thousands, or even more). In addition, for ease of understanding, Figure 3A The scores of objects 301-305 are assumed in the example, but the example is not intended to limit the data structure of the scores, the value range, the order of the scores of each object, the score distribution of each object, etc. For example, the score of an object can be an integer, a floating point number less than 1, any floating point number, a negative number, etc.

[0035] like Figure 3A As shown in FIG. 3 , after scoring each object in the object set 306, the process 300 inputs the scored object set 306 into the ranking model 318. The ranking model 318 can obtain a threshold 321 for the first ranking position 311 (also λ is used herein to represent the threshold, λ1 represents the threshold for the first ranking position), and then Figure 3A In the example shown, threshold 321 is 0.7. Process 300 can then compare the score of each object in scored object set 306 with threshold 321 and add the object to candidate object set 330 if the object's score is greater than or equal to threshold 321. Figure 3A In the example shown, the scores of object 301 (score is 0.8), object 303 (score is 0.9), and object 304 (score is 0.7) are greater than or equal to threshold 321 (0.7), and they are added to candidate object set 330. Since object 302 (score is 0.6) and object 305 (score is 0.5) are less than threshold 321, they are not added to candidate object set 330. Then, ranking model 318 determines the recommended object for ranking position 311 in candidate object set 330.

[0036] In some embodiments, if threshold 321 is greater than the scores of all objects in scored object set 306, or threshold 321 is equal to 0, process 300 may use a deterministic ranking algorithm to determine a recommended object for ranking position 311. For example, in this particular case, ranking model 318 may sort the scores of objects 301-305 in order from high to low, and determine the object with the highest score (e.g., object 303 with a score of 0.9) as the recommended object for ranking position 311.

[0037] In this way, the ranking model determines the recommended object for ranking position 311 only from objects that meet the score threshold condition (i.e., objects 301, 303, 304), which means that the recommended object determined for the top ranking position always has a sufficiently high relevance to the query 308, thereby ensuring that the recommendation result has a sufficiently high accuracy.

[0038] After generating a set of candidate objects for a target ranking position, it is necessary to determine the recommended objects for the target ranking position from the objects in the candidate object set. In some embodiments, the objects in the candidate object set may be randomly sampled to determine the recommended objects for the target ranking position. In other embodiments, the ranking model may determine the probability of candidate objects for the target ranking position based on the scores of the objects in the candidate object set, and then determine the recommended objects for the target ranking position by sampling the candidate object probabilities. In some embodiments, in order to determine the probability of candidate objects for the target ranking position, the ranking model may determine the ratio of the scores of the objects in the candidate object set to the total scores of all objects in the candidate object set, and then determine the ratio as the probability of the candidate object, wherein the probability of the candidate object indicates the probability of recommending the object at the target ranking position.

[0039] Figure 3B FIG. 3 is a schematic diagram showing a process 340 for determining a recommended object for a ranking position 311 based on a candidate object set 330 according to some embodiments of the present disclosure. Figure 3B As shown, after generating candidate object set 330 for ranking position 311, ranking model 318 determines the probability of each object being ranked at ranking position 311 based on the scores of objects 301, 303, and 304 (i.e., probability 341 for object 301, probability 343 for object 303, and probability 344 for object 304). In some embodiments, process 340 may calculate the total score of all objects in candidate object set 330. Figure 3B In the example shown, the total score is 2.4. The process 340 can then calculate the ratio of the score of each object to the total score respectively, and then determine the ratio as the probability that the object will be sorted in the sort position 311. For example, the probability that object 301 is sorted in the sort position 311 is approximately 0.33 (i.e., 0.8 / 2.4), the probability that object 303 is sorted in the sort position 311 is approximately 0.38 (i.e., 0.9 / 2.4), and the probability that object 304 is sorted in the sort position 311 is approximately 0.29 (i.e., 0.7 / 2.4). The process 340 can then sample 346 the probabilities 341, 343, and 344 to determine the recommended object for the sort position 311. Figure 3BIn the example shown, the result of sampling 346 is object 301 , so object 301 is sorted at sort position 311 .

[0040] By sampling the probabilities of multiple candidate objects in the candidate object set 330 to determine the ranking result, an object with a slightly higher score (e.g., object 303) is not always ranked in front of an object with a slightly lower score (e.g., object 301), thereby improving the fairness of the ranking. In addition, since the ranking model only determines the recommended object for the ranking position 311 from the candidate object set 330 that meets the score threshold condition, it is possible to ensure that the recommendation result has a sufficiently high accuracy. Therefore, the solution provided according to the embodiment of the present disclosure can improve the fairness of the ranking while ensuring the accuracy of the ranking system.

[0041] In some cases, the platform 106 may want to improve the fairness of multiple top ranking positions, so in some embodiments, a set of scored objects associated with the query can be determined, and then recommended objects for each of the multiple target ranking positions can be determined in turn based on the set of scored objects and a threshold sequence for the multiple target ranking positions. Figures 4A-4B A schematic diagram of a process for determining multiple objects for multiple target ranking positions according to some embodiments of the present disclosure is shown. Figures 4A-4B The process shown in Figure 3A-3B On the basis of the illustrated process (ie, on the basis of the determined recommended object at the top ranking position), the recommended object for the subsequent target ranking position is further determined.

[0042] Figure 4A A schematic diagram of a process 400 of determining a candidate object set for a next target ranking position from an object set is shown according to some embodiments of the present disclosure, when a recommended object for a top target ranking position among multiple target ranking positions has been determined. Figure 4A As shown, the goal of process 400 is to determine the recommended object for the next ranking position (i.e., the second ranking position 412) of ranking position 311 in recommendation list 310. Figure 3A-3B In the processes 300 and 340 shown, object 301 has been ranked to ranking position 311 , so process 400 may exclude object 301 from the scored object set 306 . Process 400 needs to determine a recommended object for ranking position 412 from objects 302 - 305 .

[0043] like Figure 4A As shown, the recommendation model 418 includes a threshold sequence 420, which includes multiple thresholds (i.e., thresholds 421, 422, etc.) for multiple target ranking positions, wherein the threshold 421 can be, for example, Figure 3A Since the goal of process 400 is to determine a recommended object for the second ranked position 412, process 400 may compare the scores of objects 302-305 with threshold 422. If the score of an object is greater than or equal to threshold 422, it is added to candidate object set 430. Figure 4A In the example shown, the scores of object 303 (score is 0.9) and object 304 (score is 0.7) are greater than or equal to threshold 422 (0.65), and they are added to candidate object set 430. Since object 302 (score is 0.6) and object 305 (score is 0.5) are less than threshold 422, they are not added to candidate object set 430. Then, ranking model 318 determines the recommended object for ranking position 412 in candidate object set 430.

[0044] Figure 4B FIG. 4 is a schematic diagram showing a process 440 for determining a recommended object for a ranking position 412 based on a candidate object set 430 according to some embodiments of the present disclosure. Figure 4B As shown, after generating candidate object set 430 for ranking position 412, ranking model 418 can re-determine the probability of each object being ranked in ranking position 412 (i.e., probability 443 for object 303 and probability 444 for object 304) based on the scores of objects 303 and 304. Similar to process 340, process 440 can calculate the total score of all objects in candidate object set 440. Figure 4B In the example shown, the total score is 1.6. Process 440 can then calculate the ratio of the score of each object to the total score, and then determine the ratio as the probability that the object will be sorted in sort position 412. For example, the probability that object 303 is sorted in sort position 412 is approximately 0.56 (i.e., 0.9 / 1.6), and the probability that object 304 is sorted in sort position 412 is approximately 0.44 (i.e., 0.7 / 1.6). Process 440 can then sample 446 the probabilities 443 and 444 to determine the recommended object for sort position 412. Figure 4B In the example shown, the result of sampling 446 is object 304 , so object 301 is sorted at sort position 412 .

[0045] The process of determining the recommended objects for the subsequent target ranking position can be referred to Figures 4A-4B The processes 400 and 440 shown in the figure are continued until the recommended objects for all target ranking positions are determined. For the kth ranking position among the multiple target ranking positions, the threshold for the ranking position is represented by λk, and the candidate object set for the ranking position can be determined by using the following equation (1):

[0046]

[0047] in represents the candidate object set, d represents the object, represents the relevance score of object d to a given query, y 1:k-1 Indicates that the sort position before the kth sort position has been determined.

[0048] In combination with equation (1), the following equation (2) can be used to determine the probability that object d is determined as the recommended object for the kth ranking position:

[0049]

[0050] where p(d|y 1:k-1 ,λ k ) the probability that object d is determined as the recommended object for the kth ranking position, is an indicator function, used to indicate the condition Is it satisfied? exp is an exponential function. Represents the temperature parameter used to adjust the degree of smoothing of the distribution.

[0051] By adopting the threshold sequence method, the threshold most suitable for each target ranking position can be set for the ranking position, which can ensure the accuracy of the recommendation results of each ranking position, thereby ensuring the accuracy of the entire ranking.

[0052] In the above embodiment, it is described that a set of candidate objects for each ranking position is determined by using a threshold. When determining a set of candidate objects for a target ranking position, selecting different thresholds may result in ranking results having different accuracy and fairness. In order to further ensure the accuracy of the ranking and improve fairness, in some embodiments, a plurality of candidate threshold sequences may be generated based on a scored object set, wherein each of the plurality of candidate threshold sequences includes one or more thresholds corresponding to one or more target ranking positions, and then the threshold sequence is determined based on the plurality of candidate threshold sequences and a calibration data set for the query.

[0053] When generating multiple candidate threshold sequences, in some embodiments, the highest score in the scored object set can be determined, and then a threshold (also called a first threshold) for the top ranking position (also called a first ranking position) in the candidate threshold sequence is determined based on the highest score, wherein the threshold is less than or equal to the highest score in the scored object set. In some embodiments, when the threshold sequence includes multiple thresholds, a threshold (also called a second threshold) for the next ranking position (also called a second ranking position) can be determined based on the threshold of the top ranking position.

[0054] In order to evaluate the impact of the candidate threshold sequence on the accuracy and fairness of the sorting, in some embodiments, the risk value of the candidate threshold sequence in the multiple candidate threshold sequences can be determined based on the scored object set and the calibration data set, and then the probability that the risk value is greater than a predetermined risk value threshold (also referred to as a first predetermined threshold) is determined. In response to the probability being less than a predetermined probability threshold (also referred to as a second predetermined threshold), the candidate threshold sequence is determined as a suitable threshold sequence.

[0055] Figure 5 FIG. 5 is a schematic diagram showing a process 500 for determining a suitable threshold sequence from a plurality of candidate threshold sequences according to some embodiments of the present disclosure. Figure 5 As shown, process 500 can generate multiple candidate threshold sequences 502-1, 502-2, ..., 502-N (collectively referred to as candidate threshold sequence 502). When generating candidate threshold sequence 502-1, process 500 can determine the highest score in the scored object set (e.g., the scored object set 306 in FIG. 3) (e.g., object 303 in FIG. 3 has the highest score of 0.9), and then process 500 determines threshold 504 of the top ranking position in candidate threshold sequence 502-1 based on the highest score, and threshold 504 can be less than or equal to the highest score in the scored object set, for example, threshold 504 can be 0.7. Then, process 500 can determine threshold 506 based on threshold 504, and threshold 506 can be less than or equal to threshold 504, for example, threshold 506 can be 0.65. Because the thresholds in the threshold sequence indicate the relevance requirements when screening the candidate object set, and the relevance requirements at the later positions should be lower than the relevance requirements at the earlier positions, generating a candidate threshold sequence in this way can reduce the number of unreasonable candidate threshold sequences, thereby improving the performance of the sorting model 518.

[0056] After generating multiple candidate threshold sequences 502, process 500 uses ranking model 518 to determine a suitable threshold sequence 526 from the multiple candidate threshold sequences 502 based on the multiple candidate threshold sequences 502 and the calibration data set 504. In some embodiments, process 500 may use a risk control method to determine the threshold sequence 526. The null hypothesis is a hypothesis proposed in statistical inference for hypothesis testing, which is usually used to compare with the alternative hypothesis to determine whether the actual observed data (i.e., the calibration data set 504) supports or rejects the null hypothesis. The null hypothesis usually indicates that there is no observed effect or association, or that there is no significant difference between the two sets of data. In process 500, the null hypothesis is first assumed to be true, and then inference is made based on the calibration data set 504 to determine whether there is sufficient evidence to reject the null hypothesis. Figure 5In the example shown, the null hypothesis can be that when the candidate threshold sequence is λ (e.g., candidate threshold sequence 502-1), the risk R(λ) of the ranking model 518 is greater than a predetermined risk threshold α. In some embodiments, the process 500 can evaluate the effect of the ranking model 518 using the normalized discounted cumulative gain (NDCG) and determine the risk R(λ) based on the NDCG. The NDCG indicates the utility of the ranking model 518, and the value of the NDCG is between 0 and 1. In some embodiments, the risk R(λ) can be the average 1-NDCG of the ranking model 518 when the candidate threshold sequence is λ, that is, when the NDCG of the ranking model 518 is higher, the risk is smaller.

[0057] In hypothesis testing, process 500 uses statistical methods to calculate a p-value, which represents the probability of observing a result consistent with or more extreme than the null hypothesis when the null hypothesis is true. If the p-value is very small (e.g., less than a predetermined probability threshold), process 500 can reject the null hypothesis and conclude that the calibration data set 504 provides sufficient evidence to support that for a given candidate threshold sequence, the risk of the ranking model 518 is less than a predetermined risk threshold. In other words, the goal of process 500 is to find a candidate threshold sequence 512 such that the probability that the risk of the ranking model 518 is greater than a predetermined risk threshold is less than a predetermined probability threshold, that is, P(R(λ)>α)<δ, where α is a predetermined risk threshold and δ is a predetermined probability threshold (e.g., 0.05). Figure 5 As shown, in the curve graph 520, area 522 indicates that for a given candidate threshold sequence 502, the probability that the risk is greater than the predetermined risk threshold is less than the predetermined probability threshold, and the candidate threshold sequence in this area is not a suitable threshold sequence. Area 524 indicates that for a given candidate threshold sequence 502, the probability that the risk is greater than the predetermined risk threshold is less than the predetermined probability threshold, and the candidate threshold sequence 502 in this area can be determined to be a suitable threshold sequence 526.

[0058] In some embodiments, process 500 tests the null hypothesis for each candidate threshold sequence on calibration data set 504, with the goal of obtaining a candidate threshold sequence 502 that can reject the null hypothesis through hypothesis testing. Process 500 can be expressed by the following equation (3):

[0059]

[0060] in represents the upper confidence bound of the risk.

[0061] In some embodiments, process 500 may select a candidate threshold sequence that satisfies the following equation (4):

[0062]

[0063] Where R fair A differential measure representing exposure-based fairness.

[0064] In some embodiments, process 500 may calculate an upper confidence bound for risk by using equation (5):

[0065]

[0066] Where R util represents the risk, and const. is a constant that depends on the set of loss values.

[0067] In some embodiments, process 500 may use the Hoeffding-Benktus inequality to calculate the p-value of the null hypothesis, i.e., the probability that the risk is greater than a predetermined risk threshold. For example, process 500 may calculate the p-value p by using the following equation (6): HB (λ):

[0068]

[0069] in represents the calibration dataset, represents the risk, h1 is a predetermined function and Bin(n, α) is the binomial distribution, It means rounding up the value of a, and p(c) represents the probability of satisfying condition c.

[0070] In this way, the impact of the candidate threshold sequence on the accuracy and fairness of the sorting can be accurately evaluated, so that a threshold sequence that makes the accuracy and fairness of the sorting model meet the expectations can be selected from multiple candidate threshold sequences. Figures 4A-4B The processes 400 and 440 described in the above can improve the fairness of the sorting while ensuring the accuracy of the sorting system.

[0071] Figure 6 FIG. 6 is a block diagram of an apparatus 600 for recommending an object according to some embodiments of the present disclosure. Figure 6As shown, the device 600 includes a score determination module 602, which is configured to determine a scored object set associated with the query, wherein the scored object set includes multiple objects and their corresponding multiple scores. The device 600 also includes a candidate determination module 604, which is configured to determine a candidate object set for a target ranking position based on the scored object set and a threshold for the target ranking position. In addition, the device 600 also includes a ranking determination module 606, which is configured to determine a recommended object for the target ranking position based on the candidate object probability of the candidate object set.

[0072] It can be understood that, through the device 600 of the present disclosure, at least one of the advantages that can be achieved by the method or process described above can be achieved. For example, by using the threshold for the target ranking position to determine the candidate object set, the overall accuracy of the ranking system can be ensured. At the same time, by determining the recommended object for the target ranking position based on the probability of the candidate object, the ranking list generated by the ranking system can have a certain degree of randomness and diversity, thereby improving the fairness of the ranking.

[0073] Figure 7 FIG. 1 is a block diagram of an electronic device 700 according to some embodiments of the present disclosure. The device 700 may be a device or apparatus described in an embodiment of the present disclosure. Figure 7 As shown, the device 700 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The CPU / GPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704. Although not shown in FIG. Figure 7 As shown in FIG. 7 , device 700 may further include a coprocessor.

[0074] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0075] The various methods or processes described above may be performed by the CPU / GPU 701. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the CPU / GPU 701, one or more steps or actions in the methods or processes described above may be performed.

[0076] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0077] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0078] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0079] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0081] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0082] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0084] Some example implementations of the present disclosure are listed below.

[0085] Example 1. A method for recommending an object, comprising:

[0086] determining a scored set of objects associated with the query, the scored set of objects comprising a plurality of objects and a plurality of scores corresponding thereto;

[0087] Determining a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position;

[0088] Based on the candidate object probabilities of the candidate object set, a recommended object for the target ranking position is determined.

[0089] Example 2. The method according to Example 1 further includes:

[0090] The scored object set is generated by scoring objects in the object set, wherein the scores indicate relevance of the objects to the query.

[0091] Example 3. The method of any one of Examples 1-2, wherein determining the set of candidate objects for the target ranking position comprises:

[0092] For an object whose ranking position is not determined in the scored object set, in response to the score of the object being greater than or equal to the threshold, the object is added to the candidate set.

[0093] Example 4. The method according to any one of Examples 1-3, wherein determining the recommended object for the target ranking position comprises:

[0094] determining the candidate object probability for the target ranked position based on the scores of the objects in the candidate object set; and

[0095] The recommended object for the target ranking position is determined by sampling the candidate object probabilities.

[0096] Example 5. The method of any one of Examples 1-4, wherein determining a candidate object probability for the target ranking position comprises:

[0097] determining a ratio of the score of the object in the candidate object set to the total score of all objects in the candidate object set; and

[0098] The ratio is determined as the candidate object probability, wherein the candidate object probability indicates a probability of recommending the object at the target ranking position.

[0099] Example 6. The method according to any one of Examples 1-5, further comprising:

[0100] Based on the scored set of objects, a threshold sequence is determined, the threshold sequence comprising a plurality of thresholds corresponding to a plurality of target ranking positions.

[0101] Example 7. The method of any one of Examples 1-6, wherein determining the threshold sequence comprises:

[0102] generating a plurality of candidate threshold sequences based on the scored object set, wherein each candidate threshold sequence in the plurality of candidate threshold sequences comprises a plurality of thresholds corresponding to the plurality of target ranking positions; and

[0103] The threshold sequence is determined based on the plurality of candidate threshold sequences and a calibration data set for the query.

[0104] Example 8. The method of any one of Examples 1-7, wherein generating the plurality of candidate threshold sequences comprises:

[0105] determining a highest score among the set of scored objects; and

[0106] Based on the highest score, determining a first threshold in a candidate threshold sequence, wherein the first threshold is less than or equal to the highest score;

[0107] Example 9. The method of any one of Examples 1-8, wherein generating the plurality of candidate threshold sequences further comprises:

[0108] determining a second threshold based on the first threshold,

[0109] The first threshold corresponds to a first sorting position, the second threshold corresponds to a second sorting position, the second sorting position is the next position of the first sorting position, and the second threshold is less than or equal to the first threshold.

[0110] Example 10. The method of any one of Examples 1-9, wherein determining the threshold sequence further comprises:

[0111] determining a risk value for a candidate threshold sequence from among the plurality of candidate threshold sequences based on the scored set of objects and the calibration data set;

[0112] determining a probability that the risk value is greater than a first predetermined threshold; and

[0113] In response to the probability being less than a second predetermined threshold, the candidate threshold sequence is determined as the threshold sequence.

[0114] Example 11. The method of any one of Examples 1-10, wherein determining the probability that the risk value is greater than the first predetermined threshold comprises:

[0115] The probability is determined based on the risk value and the first predetermined threshold by using the Hoefdingbentus inequality.

[0116] Example 12. A device for recommending an object, comprising:

[0117] determining a scored set of objects associated with the query, the scored set of objects comprising a plurality of objects and a plurality of scores corresponding thereto;

[0118] Determining a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position;

[0119] Based on the candidate object probabilities of the candidate object set, a recommended object for the target ranking position is determined.

[0120] Example 13. The apparatus of Example 12, further comprising:

[0121] The object scoring module is configured to generate the scored object set by scoring the objects in the object set, wherein the scores indicate the relevance of the objects to the query.

[0122] Example 14. The apparatus of any of Examples 12-13, wherein determining the set of candidate objects for the target ranking position comprises:

[0123] The threshold determination module is configured to add an object whose ranking position in the scored object set is not determined to the candidate set in response to the score of the object being greater than or equal to the threshold.

[0124] Example 15. The apparatus of any one of Examples 12-14, wherein determining the recommended object for the target ranking position comprises:

[0125] a probability determination module configured to determine the probability of the candidate object for the target ranking position based on the scores of the objects in the candidate object set; and

[0126] The probability sampling module is configured to determine the recommended object for the target ranking position by sampling the probability of the candidate objects.

[0127] Example 16. The apparatus of any of Examples 12-15, wherein determining a candidate object probability for the target ranking position comprises:

[0128] a ratio calculation module, configured to determine a ratio of the score of the object in the candidate object set to a total score of all objects in the candidate object set; and

[0129] The ratio using module is configured to determine the ratio as the candidate object probability, wherein the candidate object probability indicates a probability of recommending the object at the target ranking position.

[0130] Example 17. The apparatus of any one of Examples 12-16, further comprising:

[0131] The threshold sequence determination module is configured to determine a threshold sequence based on the scored object set, wherein the threshold sequence includes a plurality of thresholds corresponding to a plurality of target ranking positions.

[0132] Example 18. The apparatus of any of Examples 12-17, wherein determining the threshold sequence comprises:

[0133] a candidate sequence generation module configured to generate a plurality of candidate threshold sequences based on the scored object set, wherein each candidate threshold sequence in the plurality of candidate threshold sequences comprises a plurality of thresholds corresponding to the plurality of target ranking positions; and

[0134] The calibration data using module is configured to determine the threshold sequence based on the plurality of candidate threshold sequences and a calibration data set for the query.

[0135] Example 19. The apparatus of any of Examples 12-18, wherein generating the plurality of candidate threshold sequences comprises:

[0136] a highest score determination module configured to determine the highest score in the scored object set; and

[0137] A first threshold determination module, configured to determine a first threshold in a candidate threshold sequence based on the highest score, wherein the first threshold is less than or equal to the highest score;

[0138] Example 20. The apparatus of any of Examples 12-19, wherein generating the plurality of candidate threshold sequences further comprises:

[0139] A second threshold determination module is configured to determine a second threshold based on the first threshold,

[0140] The first threshold corresponds to a first sorting position, the second threshold corresponds to a second sorting position, the second sorting position is the next position of the first sorting position, and the second threshold is less than or equal to the first threshold.

[0141] Example 21. The apparatus of any of Examples 12-20, wherein determining the threshold sequence further comprises:

[0142] a threshold risk determination module configured to determine a risk value of a candidate threshold sequence among the plurality of candidate threshold sequences based on the scored object set and the calibration data set;

[0143] a risk threshold using module configured to determine a probability that the risk value is greater than a first predetermined threshold; and

[0144] The probability threshold using module is configured to determine the candidate threshold sequence as the threshold sequence in response to the probability being less than a second predetermined threshold.

[0145] Example 22. The apparatus of any of Examples 12-21, wherein determining the probability that the risk value is greater than the first predetermined threshold comprises:

[0146] The inequality using module is configured to determine the probability by using the Hoefdingbentus inequality based on the risk value and the first predetermined threshold.

[0147] Example 23. An electronic device comprising:

[0148] Processor; and

[0149] A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the electronic device to perform actions when executed by the processor, the actions comprising:

[0150] determining a scored set of objects associated with the query, the scored set of objects comprising a plurality of objects and a plurality of scores corresponding thereto;

[0151] Determining a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position;

[0152] Based on the candidate object probabilities of the candidate object set, a recommended object for the target ranking position is determined.

[0153] Example 24. The electronic device of Example 23, further comprising:

[0154] The scored object set is generated by scoring objects in the object set, wherein the scores indicate relevance of the objects to the query.

[0155] Example 25. The electronic device of any of Examples 23-24, wherein determining the set of candidate objects for the target sort position comprises:

[0156] For an object whose ranking position is not determined in the scored object set, in response to the score of the object being greater than or equal to the threshold, the object is added to the candidate set.

[0157] Example 26. The electronic device of any of Examples 23-24, wherein determining the recommended object for the target ranking position comprises:

[0158] determining the candidate object probability for the target ranked position based on the scores of the objects in the candidate object set; and

[0159] The recommended object for the target ranking position is determined by sampling the candidate object probabilities.

[0160] Example 27. An electronic device according to any of Examples 23-26, wherein determining a candidate object probability for the target sort position comprises:

[0161] determining a ratio of the score of the object in the candidate object set to the total score of all objects in the candidate object set; and

[0162] The ratio is determined as the candidate object probability, wherein the candidate object probability indicates a probability of recommending the object at the target ranking position.

[0163] Example 28. The electronic device of any of Examples 23-27, further comprising:

[0164] Based on the scored set of objects, a threshold sequence is determined, the threshold sequence comprising a plurality of thresholds corresponding to a plurality of target ranking positions.

[0165] Example 29. The electronic device of any of Examples 23-28, wherein determining the threshold sequence comprises:

[0166] generating a plurality of candidate threshold sequences based on the scored object set, wherein each candidate threshold sequence in the plurality of candidate threshold sequences comprises a plurality of thresholds corresponding to the plurality of target ranking positions; and

[0167] The threshold sequence is determined based on the plurality of candidate threshold sequences and a calibration data set for the query.

[0168] Example 30. An electronic device according to any of Examples 23-29, wherein generating the plurality of candidate threshold sequences comprises:

[0169] determining a highest score among the set of scored objects; and

[0170] Based on the highest score, determining a first threshold in a candidate threshold sequence, wherein the first threshold is less than or equal to the highest score;

[0171] Example 31. An electronic device according to any of Examples 23-30, wherein generating the plurality of candidate threshold sequences further comprises:

[0172] determining a second threshold based on the first threshold,

[0173] The first threshold corresponds to a first sorting position, the second threshold corresponds to a second sorting position, the second sorting position is the next position of the first sorting position, and the second threshold is less than or equal to the first threshold.

[0174] Example 32. An electronic device according to any of Examples 23-31, wherein determining the threshold sequence further comprises:

[0175] determining a risk value for a candidate threshold sequence from among the plurality of candidate threshold sequences based on the scored set of objects and the calibration data set;

[0176] determining a probability that the risk value is greater than a first predetermined threshold; and

[0177] In response to the probability being less than a second predetermined threshold, the candidate threshold sequence is determined as the threshold sequence.

[0178] Example 33. The electronic device of any of Examples 23-32, wherein determining the probability that the risk value is greater than the first predetermined threshold comprises:

[0179] The probability is determined based on the risk value and the first predetermined threshold by using the Hoefdingbentus inequality.

[0180] Although the disclosure has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Instead, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for recommending an object, comprising: determining a scored set of objects associated with the query, the scored set of objects comprising a plurality of objects and a plurality of scores corresponding thereto; Determining a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position; Based on the candidate object probabilities of the candidate object set, a recommended object for the target ranking position is determined.

2. The method according to claim 1, further comprising: The scored object set is generated by scoring objects in the object set, wherein the score of an object indicates the relevance of the object to the query.

3. The method of claim 1 , wherein determining the set of candidate objects for the target ranking position comprises: For an object whose ranking position is not determined in the scored object set, in response to the score of the object being greater than or equal to the threshold, the object is added to the candidate set.

4. The method according to claim 1, wherein determining the recommended object for the target ranking position comprises: Determining the candidate object probability for the target ranking position based on the scores of the objects in the candidate object set; as well as The recommended object for the target ranking position is determined by sampling the candidate object probabilities.

5. The method of claim 4, wherein determining the candidate object probability for the target ranking position comprises: determining a ratio of the score of the object in the candidate object set to the total score of all objects in the candidate object set; as well as The ratio is determined as the candidate object probability, wherein the candidate object probability indicates a probability of recommending the object at the target ranking position.

6. The method according to claim 1, further comprising: Based on the scored set of objects, a threshold sequence is determined, the threshold sequence comprising a plurality of thresholds corresponding to a plurality of target ranking positions.

7. The method of claim 6, wherein determining the threshold sequence comprises: generating a plurality of candidate threshold sequences based on the scored object set, wherein each candidate threshold sequence in the plurality of candidate threshold sequences comprises a plurality of thresholds corresponding to the plurality of target ranking positions; as well as The threshold sequence is determined based on the plurality of candidate threshold sequences and a calibration data set for the query.

8. The method of claim 7, wherein generating the plurality of candidate threshold sequences comprises: determining a highest score among the set of scored objects; as well as Based on the highest score, a first threshold in a candidate threshold sequence is determined, wherein the first threshold is less than or equal to the highest score.

9. The method according to claim 8, wherein generating the plurality of candidate threshold sequences further comprises: determining a second threshold based on the first threshold, The first threshold corresponds to a first sorting position, the second threshold corresponds to a second sorting position, the second sorting position is the next position of the first sorting position, and the second threshold is less than or equal to the first threshold.

10. The method of claim 7, wherein determining the threshold sequence further comprises: determining a risk value for a candidate threshold sequence from among the plurality of candidate threshold sequences based on the scored set of objects and the calibration data set; Determining a probability that the risk value is greater than a first predetermined threshold; as well as In response to the probability being less than a second predetermined threshold, the candidate threshold sequence is determined as the threshold sequence.

11. The method of claim 10, wherein determining the probability that the risk value is greater than the first predetermined threshold comprises: The probability is determined based on the risk value and the first predetermined threshold by using the Hoefdingbentus inequality.

12. A device for recommending an object, comprising: a score determination module configured to determine a scored object set associated with the query, the scored object set comprising a plurality of objects and a plurality of scores corresponding thereto; a candidate determination module configured to determine a set of candidate objects for the target ranking position based on the scored set of objects and a threshold for the target ranking position; The ranking determination module is configured to determine a recommended object for the target ranking position based on the candidate object probabilities of the candidate object set.

13. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 11.