Method and apparatus for rapid evaluation of ranking models

Abstract

The application discloses a method and device for quickly evaluating a ranking model, and belongs to the technical field of computers, in particular to the field of machine learning and data mining. The method comprises the following steps: obtaining a first score list of all items, wherein the score is obtained by a ranking model to be evaluated; obtaining a second score list of related items, wherein the score can be obtained from the first score list; dividing the first score list by taking the second score list as a demarcation value, and obtaining the ranking of the related items in the first score list; and calculating a ranking evaluation index and evaluating the ranking model according to the ranking of the related items. The application solves the problems of large calculation complexity and high calculation cost in the existing method for evaluating the ranking model, and improves the evaluation efficiency. By using the method, the ranking model can be quickly and accurately evaluated on the order of 100 million items by using a small amount of calculation resources, and the cost, resources and time are saved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically to the fields of machine learning and data mining, and particularly to a method and apparatus for rapidly evaluating ranking models. Background Technology

[0002] Learning to rank is a fundamental area of ​​research in machine learning and data mining, with wide applications in daily life, such as web search, image retrieval, recommender systems, and online advertising. Evaluating ranking models based on their ranking across a list of relevant items is fundamental to model training and selection. Existing evaluation metrics can be categorized into two types: full-item evaluation (e.g., AUC) and Top-k item evaluation (e.g., Precision and NDCG for the top k items). The former evaluates the model across the entire list of items, while the latter evaluates it only across the top k items.

[0003] In the evaluation process of existing ranking models, full-item evaluation requires ranking all items, while Top-k item evaluation requires partially ranking all items to obtain the top k items. Then, based on the ranking list, evaluation metrics are calculated to assess the ranking model. Therefore, the time complexity of existing methods is O(n log n). and With the development of the internet and information technology, modern websites and applications contain tens of millions or even hundreds of millions of items, such as web pages, music, images, and products. Faced with such massive amounts of data, existing methods suffer from high computational load, low efficiency, and high cost when evaluating models. These problems become increasingly apparent as informatization advances and data continues to increase. Summary of the Invention

[0004] The purpose of this invention is to address the problems faced by existing ranking model evaluation methods in large-scale projects by providing a method and apparatus for rapidly evaluating ranking models, thereby achieving fast and accurate evaluation of ranking models and saving costs, resources, and time.

[0005] In a first aspect, the present invention provides a method for rapidly evaluating a ranking model, the method comprising:

[0006] Obtain the first rating list of all items, where the ratings are obtained from the ranking model to be evaluated;

[0007] Obtain a second rating list for the relevant items, where the ratings can be derived from the first rating list;

[0008] Using the second rating list as a dividing line, the first rating list is divided to obtain the ranking of the relevant projects among all projects;

[0009] Based on the ranking of relevant projects, calculate ranking indicators and evaluate ranking models.

[0010] Secondly, the present invention also provides an apparatus for rapidly evaluating a ranking model, the apparatus comprising:

[0011] The rating list acquisition module is used to obtain the first and second rating lists of all items and related items, where the ratings in the first list are obtained by the model, and the ratings in the second list are obtained by the first rating list;

[0012] The relevant ranking acquisition module is used to quickly locate the ranking of a rating in the second rating list in the first rating list, and the obtained ranking is the ranking of the relevant item.

[0013] The ranking index calculation module calculates ranking evaluation indicators and evaluates the ranking model based on the ranking of relevant projects.

[0014] The method and apparatus of the present invention have the following characteristics: when evaluating the ranking model, the present invention does not require full or partial ranking of all items; the technical solution for quickly locating the ranking of relevant items in the method of the present invention can reduce the complexity of ranking evaluation; tests have shown that the method and apparatus of the present invention are highly efficient and can quickly evaluate the ranking model on hundreds of millions of items; therefore, the present invention can improve evaluation efficiency, reduce evaluation costs, and reduce resource consumption. Attached Figure Description

[0015] Figure 1 This is a flowchart of the fast evaluation and ranking model method provided in Embodiment 1 of the present invention;

[0016] Figure 2 This is a flowchart of the fast evaluation and ranking model method provided in Embodiment 2 of the present invention;

[0017] Figure 3 This is a schematic diagram of the structure of the fast evaluation and sorting model device provided in Embodiment 3 of the present invention.

[0018] Specific implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only for explaining the present invention and are not intended to limit the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0020] Example 1

[0021] Figure 1 This is a flowchart of a fast evaluation and ranking model method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of evaluating and ranking models based on the entire project. This method can be executed by a device for fast evaluation and ranking model methods, and this device can be implemented in software. Figure 1 As shown, this embodiment specifically includes:

[0022] S110: Obtain a first-level rating list for all items, where the ratings are derived from the ranking model to be evaluated. For example, for a given user or query, the ranking model can predict ranking ratings for all items (such as music and web pages), and the ranking ratings for all items form the first-level rating list.

[0023] S120: Obtain a second rating list for relevant items, wherein the ratings can be obtained from the first rating list. For example, the relevant items may be music that the user likes or related documents that the user queries. The ratings for the relevant items can be directly retrieved from the first rating list and used to form the second rating list.

[0024] S130: Using the second rating list as a dividing line, divide the first rating list and obtain the ranking of relevant items in the first rating list. Specifically:

[0025] S131: Sort the second rating list from largest to smallest;

[0026] S132: If the second rating list is not empty, select the median as the dividing value, divide the first rating list so that all ratings greater than the dividing value are located to the left of the dividing value, and all ratings less than the dividing value are located to the right of the dividing value.

[0027] S133: After the division is completed and the dividing point is obtained, execute S132 on the sublists to the left and right of the middle value (dividing value) of the second scoring list and the sublists to the left and right of the dividing point of the first scoring list to divide the sublists of the first scoring list.

[0028] After completing steps S132-S133 above, the division of the first scoring list is completed. At this point, for each dividing point, the scores to its left are all greater than the dividing value at the dividing point, and the scores to its right are all less than the dividing value at the dividing point. Therefore, the dividing point is the ranking of the relevant item in the entire item list.

[0029] S140: Based on the ranking of relevant projects, calculate the ranking evaluation index and evaluate the ranking model. For example, let n be the total number of projects, m be the number of relevant projects, and R represent the set of rankings of relevant projects. Then, the AUC evaluation index is calculated as follows:

[0030]

[0031] Through the above steps S110-S130, the method of the present invention can quickly locate the ranking of relevant projects without sorting all projects, and complete the evaluation of the ranking model based on all projects in step S140.

[0032] Example 2

[0033] Figure 2 This is a flowchart of the fast evaluation and ranking model method provided in Embodiment 2 of the present invention. This embodiment is applicable to the case of evaluating and ranking models based on Top-k items. Based on the characteristics of Top-k item evaluation, this embodiment further optimizes upon Embodiment 1. Specifically, ranking and evaluation based on Top-k items only requires the rankings of related items among the top k items. Therefore, this embodiment employs pruning techniques when quickly locating the rankings of related items to further reduce complexity and computational load. Figure 2 As shown, this embodiment specifically includes:

[0034] S210: Same as S110.

[0035] S220: Same as S120.

[0036] S230: Using the second rating list as a dividing line, a pruning technique is employed to divide the first rating list, obtaining the rankings of relevant items among the top k items in the first rating list. Specifically:

[0037] S231: Sort the second rating list from largest to smallest;

[0038] S232: If the second rating list is not empty, select the median as the dividing value to divide the first rating list; during the division process, if it is found that the dividing value is not in the first k items, pruning behavior is triggered and the division is stopped.

[0039] S233: If no pruning action occurs in step S232, then S232 continues to be executed in both the left and right sublists of the first and second rating lists; otherwise, if the median of the second rating list and the relevant items corresponding to its right sublist are not in the first k items, then S232 continues to be executed only in the left sublist.

[0040] After completing the above steps S232-S233, the effective dividing point obtained is the ranking of the relevant projects in the first k projects of the first scoring list.

[0041] S240: Based on the obtained rankings of relevant projects, calculate the evaluation metrics and ranking model based on the Top-k project assessment. For example, let n be the total number of projects, m be the number of relevant projects, and R represent the set of rankings of relevant projects. Then, the Precision and NDCG evaluation metrics for the top k projects are calculated as follows:

[0042]

[0043]

[0044] Through the above steps S210-S230, the method of the present invention can quickly locate the ranking of relevant projects among the top k projects, and complete the evaluation of the ranking model based on the Top-k projects in step S240.

[0045] Example 3

[0046] Figure 3 This is a schematic diagram of the device structure for rapidly evaluating ranking models provided by the present invention. This embodiment is applicable to situations requiring rapid evaluation of ranking models. The device for rapidly evaluating ranking models provided in this embodiment of the present invention can execute the method for rapidly evaluating ranking models provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Figure 3 As shown, the device includes a rating list acquisition module 310, a related item ranking and positioning module 320, and a ranking evaluation index calculation module 330, wherein:

[0047] The rating list acquisition model 310 is used to obtain a first rating list of all items and a second rating list of related items, wherein the ratings are obtained by the ranking model to be evaluated;

[0048] The relevant project ranking and positioning module 320 is used to quickly locate the ranking of relevant projects in the first rating list. This model can adaptively adjust the evaluation metrics based on both the total number of projects and the top-k projects using the value of k. Specifically, if k is the total number of projects, it is suitable for evaluation based on the total number of projects; if k is less than the total number of projects, it is suitable for evaluation based on the top-k projects.

[0049] The ranking index calculation module 330 is used to calculate ranking evaluation indexes and obtain model evaluation results.

[0050] Those skilled in the art will readily understand the embodiments of the present invention. The above description is only for preferred embodiments of the present invention and is not intended to limit the present invention.

Claims

1. A method for rapidly evaluating a ranking model, characterized in that... Includes the following steps: (1) Obtain a first rating list of all items, wherein the ratings are obtained by a ranking model to be evaluated, and the items are music or web pages; (2) Obtain the second rating list of relevant items from the first rating list; (3) Obtain the ranking set of relevant projects among all projects. The steps include: S131: Sort the second rating list from largest to smallest; S132: If the second rating list is not empty, select the median as the dividing value, divide the first rating list so that all ratings greater than the dividing value are located to the left of the dividing value, and all ratings less than the dividing value are located to the right of the dividing value. S133: After the division is completed and the dividing point is obtained, S132 is executed on the sublists to the left and right of the middle value (dividing value) of the second rating list and the sublists to the left and right of the dividing point of the first rating list, respectively, to divide the sublists of the first rating list, and the position of each dividing point in the first rating list is determined as the ranking of the relevant item. The rankings of all relevant items constitute the ranking set.

2. The method according to claim 1, characterized in that, The pruning technique includes two aspects: First, during the division of the first rating list, if the boundary value is found to be outside the first k items, pruning is triggered and the division stops; Second, if no pruning occurs, the division continues in both the left and right sublists of the first and second rating lists; otherwise, if the median of the second rating list and the relevant items corresponding to its right sublist are not in the first k items, the division continues only in the left sublist.

3. An apparatus for rapidly evaluating a ranking model, characterized in that... include: A rating list retrieval model is used to obtain a first rating list of all items and a second rating list of related items, where the ratings are obtained by a ranking model to be evaluated; The relevant project ranking and positioning module is used to locate the ranking of the relevant project in the first rating list by performing step (3) of the method as described in claim 1. The model can adaptively use the value of k to evaluate two evaluation indicators: the whole project and the Top-k project. Specifically, if k is the number of all projects, it can be applied to the model evaluation based on the whole project. If k is less than the number of all projects, it can be applied to the model evaluation based on the Top-k project. The ranking index calculation module is used to calculate ranking evaluation indicators and obtain model evaluation results.

Images