Recommendation system evaluation method, device, electronic device and storage medium

Abstract

The present application discloses a recommendation system evaluation method, device, electronic device and storage medium, the recommendation system evaluation method comprising: responding to a diversity evaluation trigger instruction of the recommendation system; evaluating the recommendation system according to the detection result of the diversity index of the set recommendation system, and updating the current recommendation strategy if it is determined that the recommendation strategy update condition is met, the detection result is obtained by detecting the diversity index in the current detection cycle, and the diversity index includes at least one of the following indicators: coverage, personalization, novelty and surprise, the coverage represents the coverage ability of the recommendation system to recommend information, the personalization represents the balance between the personalized recommendation and the non-personalized recommendation of the recommendation system, the novelty represents the degree of recommendation of the recommendation system for information that has not been recommended to the recommendation object, and the surprise represents the degree of recommendation of the recommendation system for information that has not been recommended to the recommendation object and is of interest to the recommendation object.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a recommendation system evaluation method, device, electronic device, and storage medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. No statement herein is admitted to be prior art by virtue of its inclusion in this section.

[0003] Diversity in a recommender system refers to the inclusion of different content sets in the information recommendations for a recommendee, which differ from the recommendee's past preferences. Given the wide range of interests of recommendees, on an information flow platform, to satisfy these diverse interests, the recommended content must cover their diverse areas of interest. This means that the recommendation results must be diverse. Although a recommendee's interests may not change much over time, at the moment they access the recommender system, their interests are often singular. If the recommendation results only cover a single interest point for the recommender, which is not their current interest, the recommendation results will not satisfy the recommender. Conversely, if the recommendation results are diverse, covering the vast majority of the recommender's interests or helping them discover previously undiscovered interests, the likelihood of them finding content of interest will increase, thereby improving their satisfaction with the recommended content and the information flow platform. Therefore, the content recommended to recommendees must also satisfy their broad interests and be able to help them discover unknown interests. This is called diversity.

[0004] In the related art, there is no applicable method for evaluating the diversity of recommendation systems, which results in the recommendation system's recommendation content being single. When the diversity of the recommendation system increases or decreases, it is impossible to locate the reasons for the increase or decrease, which affects the user experience. Summary of the Invention

[0005] To solve the above technical problems, embodiments of the present application provide a recommendation system evaluation method, device, electronic device, and storage medium.

[0006] On the one hand, an embodiment of the present application provides a recommendation system evaluation method, including:

[0007] triggering an instruction in response to a diversity evaluation of the recommender system;

[0008] The recommendation system is evaluated according to the detection results of the diversity indicators of the set recommendation system. If it is determined that the recommendation strategy update conditions are met, the current recommendation strategy is updated. The detection results are obtained by detecting the diversity indicators in the current detection cycle. The diversity indicators include at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system to recommend information, the personalization represents the balance between personalized recommendations and non-personalized recommendations of the recommendation system, the novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object, and the surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object and that the recommendation object is interested in.

[0009] In one aspect, an embodiment of the present application provides a recommendation system evaluation device, comprising:

[0010] a response unit, configured to trigger an instruction in response to a diversity evaluation of the recommendation system;

[0011] An evaluation unit is used to evaluate the recommendation system according to the detection results of the diversity indicators of the set recommendation system, and update the current recommendation strategy if it is determined that the recommendation strategy update conditions are met. The detection results are obtained by detecting the diversity indicators in the current detection cycle. The diversity indicators include at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system to recommend information, the personalization represents the balance between personalized recommendations and non-personalized recommendations of the recommendation system, the novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object, and the surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object and that the recommendation object is interested in.

[0012] On the one hand, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the recommendation system evaluation method described in the present application when executing the computer program.

[0013] On the one hand, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the recommendation system evaluation method described in the present application.

[0014] The recommendation system evaluation scheme provided in the embodiment of the present application responds to the diversity evaluation trigger instruction of the recommendation system, evaluates the recommendation system according to the detection results of the diversity indicators of the set recommendation system, and updates the current recommendation strategy if it is determined that the recommendation strategy update conditions are met, wherein the detection results are obtained by detecting the diversity indicators in the current detection cycle, and the diversity indicators include at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system to recommend information, the personalization represents the balance between personalized recommendations and non-personalized recommendations of the recommendation system, the novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object, and the surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object and that the recommendation object is interested in. In the embodiment of the present application, four dimensions of diversity indicators are set to evaluate the diversity of the recommendation system. The coverage indicator is used to measure the coverage of the recommended information of the recommendation system. The higher the coverage of the recommendation, the higher the diversity of the recommendation system. The personalization index is used to measure the balance between personalized recommendations and non-personalized recommendations of the recommendation system. Since the recommendation system is mostly composed of personalized recommendations, non-personalized recommendations (such as hot content, etc.) are also important needs of users in some specific scenarios. This indicator can enable the recommendation system to quickly understand the different needs of users and make timely adjustments to changes in user needs. The novelty index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users. Improving novelty is an important path to improving the diversity of the recommendation system, and it is also an important way to enhance users' desire to explore and obtain more complete user interests. The surprise index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users and that users are interested in. Users may feel bored when faced with information previously recommended by the recommendation system. The recommendation system needs to provide information that can make users feel novel and interested. Therefore, the surprise index is an important path to improve long-term user retention. Furthermore, when the recommendation system is evaluated based on the test results of the above-mentioned diversity indicators, if it is determined that the current recommendation strategy of the recommendation system needs to be updated, the current recommendation strategy is updated. In this way, while ensuring that the diversity indicators meet the requirements, the content recommended by the recommendation system can be more diversified. When the diversity of the recommendation system increases or decreases, the reasons for the increase or decrease can be accurately located, and the recommendation accuracy of the recommendation system is improved. A balance between the recommendation accuracy and diversity of the recommendation system can be achieved, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of an application scenario of the recommendation system evaluation method provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of the implementation process of the recommendation system evaluation method provided in an embodiment of the present application;

[0017] Figure 3 This is a schematic diagram of the implementation process of updating the current recommendation strategy when detecting coverage in an embodiment of the present application;

[0018] Figure 4 This is a schematic diagram of an implementation process for updating the current recommendation strategy when detecting the degree of personalization in an embodiment of the present application;

[0019] Figure 5 This is a schematic diagram of an implementation process for updating the current recommendation strategy when detecting novelty in an embodiment of the present application;

[0020] Figure 6 This is a schematic diagram of the implementation process of updating the current recommendation strategy when detecting the surprise level in an embodiment of the present application;

[0021] Figure 7 A schematic diagram of the structure of a recommendation system evaluation device provided in an embodiment of the present application;

[0022] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present application, and are not intended to limit the scope of the present application in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0024] Those skilled in the art will appreciate that the embodiments of the present application may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0025] It should be understood herein that any number of elements in the drawings is for illustration only and not for limitation, and any naming is only for distinction and does not have any limiting meaning.

[0026] The following is a detailed explanation of the principles and spirit of this application with reference to several representative embodiments of this application. For ease of understanding, the technical terms involved in the examples of this application are explained as follows:

[0027] 1. Content: refers to the general term for all readable information distributed on the Internet. It has various types of presentation forms, such as news articles, videos, dynamic short articles, e-commerce product information, advertisements, etc.

[0028] 2. Content Category: This refers to the classification of the content, which determines the ownership of the content. Categories are hierarchical and can be divided into primary, secondary, and tertiary categories. There is an inheritance relationship between high-level and low-level categories, which can usually be displayed through a category tree.

[0029] 3. Personalized Recommendation: Distributing different content to different users through computer algorithms to meet their individual information needs is called personalized recommendation. While respecting user privacy, privacy protection technologies are applied to safeguard the data provided by the personalized recommendation service for user-authorized personal information.

[0030] 4. Recommendation system: A system that implements personalized recommendation calculations. The recommendation process is generally divided into content recall, rough sorting, filtering, fine sorting, and re-sorting. In the recall, rough sorting, and fine sorting stages, machine learning algorithms in artificial intelligence are often used to score each user and content that currently needs to be recommended and perform content screening.

[0031] 5. Information flow recommendation: Information flow is a stream of content that can be scrolled and browsed. These contents will appear in sections that are similar in appearance and arranged in order. For example, an information flow can be a system-recommended information flow (such as an article or news list) or product details (such as a product list, a service list, etc.). The essence of information flow recommendation is an information distribution channel. Taking the news recommendation information flow as an example, information flow products capture users' reading behavior through personalized content recommendations, use continuously improved algorithms to analyze reading habits, and then actively recommend content that users are interested in. In addition, users can also actively filter content by marking content as "interested" or "not interested", etc., to help build personal preference data. In theory, if the content library is large enough and the recommendation mechanism is perfect enough, users can refresh the content they are interested in infinitely.

[0032] 6. Diversity in Recommender Systems: This refers to the inclusion of different types of content sets (or item sets) in the information recommendations for users (i.e., recommendation targets), which differ from the user's past preferences. Since users have a wide range of interests, on an information flow platform, a user might enjoy reading recent government news, sports news, and entertainment information. Therefore, to satisfy this broad range of users, the recommended content must cover their diverse interests, meaning the recommendation results must be diverse. Although user interests may not fluctuate significantly over time, at the moment a user accesses a recommendation system, their interests are often singular. If the recommendation results only cover a single interest point, which is not the user's current interest, the recommendation results will not satisfy the user. Conversely, if the recommendation results are diverse, covering the vast majority of the user's interests or helping the user discover previously undiscovered interests, the user will be more likely to find interesting content, improving their satisfaction with the recommended content and the information flow platform. Therefore, content recommendations must also satisfy a wide range of users' interests and help them discover unknown interests, meaning they must be diverse.

[0033] 7. User experience: Measures the overall user satisfaction with the recommendation system. Specific indicators may include recommendation accuracy, timeliness of information acquisition, content diversity, etc.

[0034] 8. Long-term and short-term interests: Users have accumulated a certain amount of historical behavior on information flow platforms or in specific scenarios. The longer the time span of historical behavior, the more stable the user's interests. Long-term interests refer to interests that are not easily changed over time and are relatively stable. Short-term interests refer to interests that change frequently. SUMMARY OF THE INVENTION

[0036] In the related art, there is no applicable method for evaluating the diversity of recommendation systems, which results in the recommendation system's recommendation content being single. When the diversity of the recommendation system increases or decreases, it is impossible to locate the reasons for the increase or decrease, which affects the user experience.

[0037] Based on this, in order to solve the above problems, the present application proposes a recommendation system evaluation method, device, electronic device and storage medium, which responds to the diversity evaluation trigger instruction of the recommendation system, evaluates the recommendation system according to the detection results of the diversity indicators of the set recommendation system, and updates the current recommendation strategy if it is determined that the recommendation strategy update conditions are met, wherein the detection results are obtained by detecting the diversity indicators in the current detection cycle, and the diversity indicators include at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system's recommended information, the personalization represents the balance between the personalized recommendation and non-personalized recommendation of the recommendation system, the novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object, and the surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object and is of interest to the recommendation object. In the embodiment of the present application, four dimensions of diversity indicators are set to evaluate the diversity of the recommendation system. The coverage indicator is used to measure the coverage of the recommended information of the recommendation system. The higher the coverage of the recommendation, the higher the diversity of the recommendation system. The personalization index is used to measure the balance between personalized recommendations and non-personalized recommendations of the recommendation system. Since the recommendation system is mostly composed of personalized recommendations, non-personalized recommendations (such as hot content, etc.) are also important needs of users in some specific scenarios. This indicator can enable the recommendation system to quickly understand the different needs of users and make timely adjustments to changes in user needs. The novelty index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users. Improving novelty is an important path to improving the diversity of the recommendation system, and it is also an important way to enhance users' desire to explore and obtain more complete user interests. The surprise index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users and that users are interested in. Users may feel bored when faced with information previously recommended by the recommendation system. The recommendation system needs to provide information that can make users feel novel and interested. Therefore, the surprise index is an important path to improve long-term user retention. Furthermore, when the recommendation system is evaluated based on the test results of the above-mentioned diversity indicators, if it is determined that the current recommendation strategy of the recommendation system needs to be updated, the current recommendation strategy is updated. In this way, while ensuring that the diversity indicators meet the requirements, the content recommended by the recommendation system can be more diversified. When the diversity of the recommendation system increases or decreases, the reasons for the increase or decrease can be accurately located, and the recommendation accuracy of the recommendation system is improved. A balance between the recommendation accuracy and diversity of the recommendation system can be achieved, thereby improving the user experience.

[0038] After introducing the basic principles of the present application, various non-limiting implementation methods of the present application are described in detail below.

[0039] Application Scenario Overview

[0040] refer to Figure 1 , which is a schematic diagram of an application scenario of the recommendation system evaluation method provided in an embodiment of the present application. The application scenario may include a recommendation system and a terminal. The terminal may include multiple terminals, such as terminal 1 to terminal N. The recommendation system is connected to terminals 1 to terminal N respectively through a communication network, and the recommendation system can recommend information to terminals 1 to terminal N. The recommendation system responds to a trigger instruction for diversity evaluation of the recommendation system and evaluates the recommendation system based on the inspection results of the set diversity indicators of the recommendation system. If the recommendation strategy update conditions are met, the current recommendation strategy is updated, and information is recommended to the corresponding target terminal according to the updated recommendation strategy.

[0041] Among them, the communication network can be a local area network, a wide area network, a mobile Internet, etc., the terminal can be but not limited to: smart phones, tablet computers, laptops, desktop computers and other devices, the recommendation system can include an independent physical server, and can also include a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, and cloud storage. The embodiments of this application do not limit this.

[0042] Exemplary Methods

[0043] The following describes the recommendation system evaluation method according to an exemplary embodiment of the present application in conjunction with the above application scenario. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0044] refer to Figure 2 , the embodiment of the present application provides a recommendation system evaluation method, which can be applied to Figure 1 The recommendation system shown may specifically include the following steps:

[0045] S11. Responding to a diversity evaluation trigger instruction of the recommendation system.

[0046] In specific implementation, the diversity index of the recommendation system is pre-set. The diversity index of the recommendation system is an index used to measure the diversity of the recommendation system. The diversity index can include but is not limited to at least one of the following indicators: coverage, personalization, novelty and surprise. Among them, coverage represents the coverage ability of the recommendation system's recommended information, personalization represents the balance between personalized recommendations and non-personalized recommendations, novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommended object, and surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommended object and that the recommended object is interested in. Among them, the recommended object is Figure 1 end users in .

[0047] In specific implementations, the recommendation system can initiate a diversity assessment trigger instruction for the recommendation system according to a set detection period, thereby triggering a test of the recommendation system's diversity indicators during the current detection period to obtain a test result. The detection period can be set as needed, for example, to one day, one week, or one month, and this embodiment of the application is not limited to this.

[0048] S12: Evaluate the recommendation system according to the detection results of the diversity index of the set recommendation system. If it is determined that the recommendation strategy update conditions are met, update the current recommendation strategy.

[0049] In specific implementation, the recommendation system evaluates the recommendation system according to the corresponding test results obtained by testing the diversity indicators of the recommendation system in the current test cycle. If it is determined that the recommendation strategy update conditions are met, the current recommendation strategy is updated.

[0050] In the embodiment of the present application, four dimensions of diversity indicators are set to evaluate the diversity of the recommendation system. The coverage indicator is used to measure the coverage of the recommended information of the recommendation system. The higher the coverage of the recommendation, the higher the diversity of the recommendation system. The personalization index is used to measure the balance between personalized recommendations and non-personalized recommendations of the recommendation system. Since the recommendation system is mostly composed of personalized recommendations, non-personalized recommendations are also an important need of users in some specific scenarios. This indicator can enable the recommendation system to quickly understand the different needs of users and make timely adjustments to changes in user needs. The novelty index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users. Improving novelty is an important path to improving the diversity of the recommendation system, and it is also an important way to enhance users' desire to explore and obtain more complete user interests. The surprise index is used to measure the ability of the recommendation system to recommend information that has not been recommended to users and that users are interested in. Users may feel bored when faced with information previously recommended by the recommendation system. The recommendation system needs to provide information that can make users feel novel and interested. The surprise index is an important path to improve long-term user retention. Furthermore, when the recommendation system is evaluated based on the test results of the above-mentioned diversity indicators, if it is determined that the current recommendation strategy of the recommendation system needs to be updated, the current recommendation strategy is updated. In this way, while ensuring that the diversity indicators meet the requirements, the content recommended by the recommendation system can be more diversified. When the diversity of the recommendation system increases or decreases, the reasons for the increase or decrease can be accurately located, and the recommendation accuracy of the recommendation system is improved. A balance between the recommendation accuracy and diversity of the recommendation system can be achieved, thereby improving the user experience.

[0051] As a possible implementation, if it is determined that the coverage is detected, it is determined that the recommended strategy update conditions are met, specifically including:

[0052] If it is determined that the per capita coverage of all recommended objects decreases by more than a first preset ratio compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0053] In specific implementations, coverage can include category coverage and long-term interest coverage. Category coverage represents the recommendation system's ability to cover the content categories it recommends, meaning it measures the extent to which the recommendation system covers the content categories it recommends to the recommended object. Long-term interest coverage represents the recommendation system's ability to cover content of long-term interest, meaning it measures the extent to which the recommendation system covers content of long-term interest for the recommended object. The first preset ratio can be set based on actual needs, for example, 5%, and this embodiment of the present application does not limit this.

[0054] Specifically, the recommendation system can calculate the category coverage for any recommended object using the following formula:

[0055]

[0056] Wherein, α represents the category coverage of the recommendation system for any of the recommended objects;

[0057] a represents the number of categories of content recommended to any of the recommended objects. The recommended content represents the content recorded in the log after information filtering. A represents the total number of categories of content recommended to all recommended objects.

[0058] Alternatively, a represents the number of categories of content to which any of the recommended objects are exposed, the exposed content represents the content displayed to any of the recommended objects, and the exposed content is displayed to any of the recommended objects in the form of a display list, and A represents the total number of categories of content to which all recommended objects are exposed;

[0059] Alternatively, a represents the number of categories of content clicked by any one of the recommendation objects from the display list, and A represents the total number of categories of content clicked by all the recommendation objects from their respective corresponding display lists.

[0060] The average category coverage of the recommendation system for each recommended object is the average category coverage for all recommended objects. During implementation, the average number of categories of content recommended to all recommended objects can also be calculated, and the ratio of the average number of categories of content recommended to all recommended objects to the total number of categories of content recommended to all recommended objects is the average category coverage for all recommended objects by the recommendation system; or, the average number of categories of content exposed to all recommended objects can be calculated, and the ratio of the average number of categories of content exposed to all recommended objects to the total number of categories of content exposed to all recommended objects is the average category coverage for all recommended objects by the recommendation system; or, the average number of categories of content clicked by all recommended objects from their respective display lists can be calculated, and the average number of categories of content clicked by all recommended objects from their respective display lists and the total number of categories of content clicked by all recommended objects from their respective display lists is the average category coverage for all recommended objects by the recommendation system.

[0061] The recommendation system includes an information filtering module. The content recorded in the log after information filtering is the content recorded in the log in the recommendation system background after filtering by the information filtering module. The level of the content category can be set by yourself. For example, the content category includes first-level categories, second-level categories, and third-level categories. When calculating category coverage, in order to obtain more detailed category statistics, the third-level category can be used. If the category level requirements are not high, the second-level category or the first-level category can also be used. This embodiment of the present application does not limit this.

[0062] In specific implementation, the recommendation system can calculate the long-term interest coverage for any recommended object using the following formula:

[0063]

[0064] Where β represents the long-term interest coverage of the recommendation system for any of the recommended objects;

[0065] b represents the content volume of the long-term interests of any of the recommended objects, and B represents the content volume of all the content recommended for any of the recommended objects;

[0066] Alternatively, b represents the recommendation volume of the content of long-term interest recommended to any of the recommendation objects, and B represents the recommendation volume of all the content recommended to any of the recommendation objects; the recommendation volume refers to the number of times it has been recommended;

[0067] Alternatively, b represents the exposure amount of the content of the long-term interest to which any of the recommended objects is exposed, and B represents the exposure amount of all the content to which any of the recommended objects is exposed; the exposure amount refers to the number of times the content is exposed;

[0068] Alternatively, b represents the number of clicks on the exposed long-term interests of any recommended object, and B represents the number of clicks on all contents clicked by any recommended object; click volume refers to the number of clicks;

[0069] Alternatively, b represents the browsing time of the exposed long-term interests content clicked by any of the recommended objects, and B represents the browsing time of all the clicked contents by any of the recommended objects.

[0070] Among them, the content of the long-term interests recommended to any recommendation object can be obtained through user portraits and other methods, and this embodiment of the present application does not limit this.

[0071] The average of the long-term interest coverage of each recommended object by the recommendation system is the average long-term interest coverage per person for all recommended objects by the recommendation system. During implementation, the average long-term interest coverage per person for all recommended objects by the recommendation system can also be calculated by the following method:

[0072] Calculate the average amount of content in the long-term interests recommended to all recommended objects and the average amount of content in all content recommended to all recommended objects. The ratio of the average amount of content in the long-term interests recommended to all recommended objects to the average amount of content recommended to all recommended objects is the average long-term interest coverage of the recommendation system for all recommended objects. Alternatively, calculate the average amount of recommendations for content in the long-term interests recommended to all recommended objects and the average amount of all content recommended to all recommended objects. The ratio of the average amount of recommendations for content in the long-term interests recommended to all recommended objects and the average amount of content recommended to all recommended objects is the average long-term interest coverage of the recommendation system for all recommended objects. Alternatively, calculate the average exposure of content in the long-term interests exposed to all recommended objects and the average exposure of all content exposed to all recommended objects. The ratio of the average amount of content in the long-term interests recommended to all recommended objects and the average amount of content recommended to all recommended objects is the average long-term interest coverage of the recommendation system for all recommended objects. The ratio of the exposure of the content to the exposure of all the content exposed to all the recommended objects is the per capita long-term interest coverage of the recommendation system for all the recommended objects; or, the number of clicks on the content of the exposed long-term interests clicked by all the recommended objects is calculated, and the number of clicks on the content of the exposed long-term interests clicked by all the recommended objects is calculated, and the ratio of the number of clicks on the content of the exposed long-term interests clicked by all the recommended objects to the number of clicks on all the content clicked by all the recommended objects is the per capita long-term interest coverage of the recommendation system for all the recommended objects; or, the browsing time of the content of the exposed long-term interests clicked by all the recommended objects is calculated, and the browsing time of all the content clicked by all the recommended objects is calculated, and the ratio of the browsing time of the content of the exposed long-term interests clicked by all the recommended objects to the browsing time of all the content clicked by all the recommended objects is the per capita long-term interest coverage of the recommendation system for all the recommended objects.

[0073] Furthermore, if the coverage test results meet the above-mentioned recommendation strategy update conditions, the following can be followed: Figure 3 The process shown here updates the current recommendation strategy and may include the following steps:

[0074] S201: Acquire first recommended objects whose coverage drops by more than a first preset ratio to generate a first object set.

[0075] In specific implementation, if the per capita category coverage or per capita long-term interest coverage of all recommended objects of the recommendation system drops by more than a first preset ratio compared with the previous detection cycle, the current recommendation strategy is updated.

[0076] Specifically, if it is determined that the per capita category coverage for all recommended objects has dropped by more than a first preset proportion compared with the previous detection cycle, the first recommended object whose category coverage has dropped by more than the first preset proportion is obtained to generate a first object set; if it is determined that the per capita long-term interest coverage for all recommended objects has dropped by more than a first preset proportion compared with the previous detection cycle, the first recommended object whose long-term interest coverage has dropped by more than the first preset proportion is obtained to generate a first object set.

[0077] S202. For each first recommended object in the first object set, starting from the next recommendation of the first recommended object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds a first set number of times, increase the recommendation of the first set number of unrecommended content.

[0078] In specific implementation, if it is determined that the per capita category coverage for all recommended objects has dropped by more than a first preset proportion compared with the previous detection cycle, then for each first recommended object in the first object set, starting from the next recommendation of the first object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds the first set number of times, the first set number of contents in the categories that have not been recommended will be recommended.

[0079] If it is determined that the per capita long-term interest coverage for all recommended objects has dropped by more than a first preset proportion compared with the previous detection cycle, then for each first recommended object in the first object set, starting from the next recommendation of the first object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds the first set number of times, the first set number of contents in the long-term interests of the first recommended object that have not been recommended will be recommended.

[0080] Among them, the first set number of times and the first set quantity can be set according to needs. For example, the first set number of times can be set to 10 times, and the first set quantity can be set to 5. The first set number of times and the first set quantity can be adjusted according to the detection results of multiple rounds of detection cycles. The embodiments of the present application are not limited to this.

[0081] During implementation, starting from the next recommendation to the first recommended object, when it is determined that the first recommended object has clicked on the recommended content more than a first set number of times, the first set number of unrecommended categories are recommended as an example. Specifically, the implementation can be carried out in the following manner:

[0082] Starting from the time when the click behavior (i.e., the number of recommended contents clicked) of the recommended contents for the first recommendation object exceeds the first number, the contents in the categories that have not been recommended can be gradually added from the next recommendation until the first set number of contents in the categories that have not been recommended is added. For example, assuming that the first set number is 5, 1 content in the category that has not been recommended can be added in the next recommendation, and 2 content in the category that has not been recommended can be added in the next recommendation, until a total of 5 content in the category that has not been recommended is added and no more recommendations are made. Similarly, the implementation method for adding the first set number of recommended contents in the long-term interests of the first recommendation object is similar to the above-mentioned method for adding the first set number of recommended contents in the category that have not been recommended, and will not be repeated.

[0083] In order to achieve a balance between the diversity index of the recommendation system and other existing indicators of the recommendation system (such as the recommendation accuracy index, etc.), the following steps can be further included:

[0084] S203: Determine whether the fluctuation range of other indicators of the recommendation system in the current detection period is within the set range. If yes, execute step S204; otherwise, execute step S205.

[0085] During specific implementation, the recommendation system determines whether the fluctuation range of other indicators of the recommendation system during the current detection period is within the set range, that is, whether other indicators of the recommendation system are stable. The set range can be set according to needs, and the embodiment of the present application does not limit this.

[0086] S204: Maintain the updated recommendation strategy.

[0087] S205: Return the recommended strategy before the update.

[0088] Among them, other indicators of the recommendation system may be, but are not limited to, one or more indicators of recommendation accuracy, timeliness of information acquisition, and experience fluency. The embodiments of the present application do not limit this. These other indicators of the recommendation system can be calculated according to existing calculation methods, and the embodiments of the present application do not elaborate on this.

[0089] In the embodiment of the present application, the higher the coverage of the recommendation system, the more diverse the recommendation system, the closer the connection between users and recommended content, and the higher the long-term stickiness.

[0090] As a possible implementation, if it is determined that the degree of personalization is to be detected, then it is determined that the recommendation strategy update conditions are met, specifically including:

[0091] If it is determined that the per capita personalized proportion of all recommended objects decreases by more than a second preset proportion compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0092] In specific implementation, the degree of personalization may include the personalized ratio and the non-personalized ratio. The personalized ratio represents the ratio of personalized recommended content of the recommendation system to all recommended content. The personalized ratio includes the personalized recommendation ratio, the personalized exposure ratio or the personalized consumption ratio. The non-personalized ratio represents the ratio of non-personalized recommended content of the recommendation system to all recommended content. The non-personalized ratio may include the non-personalized recommendation ratio, the non-personalized exposure ratio or the non-personalized consumption ratio.

[0093] The second preset ratio can be set according to actual needs, for example, it can be set to 5%, which is not limited in the embodiment of the present application.

[0094] Specifically, the recommendation system can calculate the personalization ratio for any recommended object using the following formula:

[0095]

[0096] Wherein, γ represents the proportion of personalized recommendations of the recommendation system for any of the recommended objects;

[0097] c represents the amount of content recommended to any of the recommendation objects for personalization, and C represents the amount of all content recommended to any of the recommendation objects; or c represents the amount of content recommended to any of the recommendation objects for personalization, and C represents the amount of all content recommended to any of the recommendation objects;

[0098] Alternatively, γ represents the personalized exposure ratio of the recommendation system to any of the recommended objects;

[0099] c represents the exposure of the content of any recommended object exposed by personalized recommendation, and C represents the exposure of all content exposed by any recommended object;

[0100] Alternatively, γ represents the personalized consumption ratio (also known as personalized click ratio) of the recommendation system for any of the recommended objects;

[0101] c represents the number of clicks on the personalized recommended content clicked by any of the recommended objects, and C represents the number of clicks on all the content clicked by any of the recommended objects; or, c represents the browsing time of the personalized recommended content clicked by any of the recommended objects, and C represents the browsing time of all the content clicked by any of the recommended objects.

[0102] The average of the personalized ratios of each recommendation object by the recommendation system is the average personalized ratio per person for all recommendation objects by the recommendation system.

[0103] Specifically, the average of the personalized recommendation ratios for each recommended object by the recommendation system is the average personalized recommendation ratio per capita for all recommended objects by the recommendation system; the average of the personalized exposure ratios for each recommended object by the recommendation system is the average personalized exposure ratio per capita for all recommended objects by the recommendation system; and the average of the personalized consumption ratios for each recommended object by the recommendation system is the average personalized consumption ratio per capita for all recommended objects by the recommendation system.

[0104] During implementation, the per capita personalized recommendation ratio of the recommendation system for all recommended objects can also be calculated in the following way:

[0105] Calculate the ratio of the average amount of personalized content recommended to all recommended objects to the average amount of all content recommended to all recommended objects, and obtain the average personalized recommendation ratio of the recommendation system for all recommended objects; or calculate the ratio of the average amount of personalized content recommended to all recommended objects to the average amount of all content recommended to all recommended objects, and obtain the average personalized recommendation ratio of the recommendation system for all recommended objects.

[0106] You can also calculate the average personalized exposure ratio of the recommendation system to all recommended objects in the following way:

[0107] Calculate the ratio of the average exposure of personalized recommended content for all recommended objects to the average exposure of all content for all recommended objects, and obtain the average personalized exposure ratio of the recommendation system for all recommended objects.

[0108] You can also calculate the per capita personalized consumption ratio of the recommendation system for all recommended objects in the following way:

[0109] Calculate the ratio of the number of clicks on personalized recommended content per person for all recommended objects to the number of clicks on all content per person for all recommended objects, and obtain the average personalized consumption ratio per person for all recommended objects in the recommendation system; or calculate the ratio of the browsing time of personalized recommended content per person for all recommended objects to the browsing time of all content per person for all recommended objects, and obtain the average personalized consumption ratio per person for all recommended objects in the recommendation system.

[0110] In specific implementation, the recommendation system can calculate the non-personalized ratio for any recommendation object using the following formula:

[0111]

[0112] Wherein, δ represents the proportion of non-personalized recommendations of the recommendation system for any of the recommended objects;

[0113] d represents the amount of non-personalized content recommended to any of the recommended objects, and D represents the amount of all content recommended to any of the recommended objects; or d represents the amount of non-personalized content recommended to any of the recommended objects, and D represents the amount of all content recommended to any of the recommended objects;

[0114] Alternatively, δ represents the non-personalized exposure ratio of the recommendation system to any of the recommended objects;

[0115] d represents the exposure of the content of any recommended object exposed by non-personalized recommendations, and D represents the exposure of all content exposed by any recommended object;

[0116] Alternatively, δ represents the proportion of non-personalized consumption of any of the recommended objects by the recommendation system;

[0117] d represents the number of clicks on the non-personalized recommended content clicked by any of the recommended objects, and D represents the number of clicks on all the content clicked by any of the recommended objects; or, d represents the browsing time of the non-personalized recommended content clicked by any of the recommended objects, and D represents the browsing time of all the content clicked by any of the recommended objects.

[0118] Furthermore, if the test results obtained by testing the degree of personalization meet the update conditions of the recommendation strategy, the following can be followed: Figure 4 The process shown here updates the current recommendation strategy and may include the following steps:

[0119] S301: Obtain second recommended objects whose personalized proportions decrease by more than the second preset proportion to generate a second object set.

[0120] In specific implementation, if the recommendation system's average personalized recommendation ratio for all recommended objects, average personalized exposure ratio for all recommended objects, or average personalized consumption ratio for all recommended objects drops by more than a second preset ratio compared to the previous detection cycle, the current recommendation strategy will be updated.

[0121] Specifically, if it is determined that the per capita personalized recommendation ratio for all recommended objects has dropped by more than a second preset ratio compared with the previous detection cycle, the second recommended object whose personalized recommendation ratio has dropped by more than the second preset ratio is obtained to generate a second object set; if it is determined that the per capita personalized exposure ratio for all recommended objects has dropped by more than a second preset ratio compared with the previous detection cycle, the second recommended object whose personalized exposure ratio has dropped by more than the second preset ratio is obtained to generate a second object set; if it is determined that the per capita personalized consumption ratio for all recommended objects has dropped by more than a second preset ratio compared with the previous detection cycle, the second recommended object whose personalized consumption ratio has dropped by more than the second preset ratio is obtained to generate a second object set.

[0122] S302. For each second recommended object in the second object set, determine whether the current non-personalized recommended content for the second recommended object meets its interest preference. If yes, execute step S303; otherwise, execute step S304.

[0123] S303: Maintain the current recommendation strategy.

[0124] S304: Starting from the next recommendation to the second recommended object, reduce the proportion of non-personalized recommended content.

[0125] In specific implementations, if the recommendation system determines that the non-personalized recommendations currently made to the second recommended subject do not match their interests and preferences, the system will reduce the proportion of non-personalized recommendations and increase the proportion of personalized recommendations accordingly, starting with the next recommendation made to the second recommended subject. The proportion of non-personalized recommendations reduced (i.e., the proportion of personalized recommendations increased) can be set as needed and is not limited in this embodiment.

[0126] Although most recommendations in the recommendation system are personalized, non-personalized recommendations (such as hot content, local content, etc.) are also important user needs in some specific scenarios. The personalization degree index enables the recommendation system to directly and quickly count the different aspects of user needs and make timely adjustments to changes in user needs.

[0127] As a possible implementation, if it is determined that the novelty is to be detected, then it is determined that the recommendation strategy update conditions are met, specifically including:

[0128] If it is determined that the average novelty of all recommended objects decreases by more than a third preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0129] In specific implementations, novelty includes category novelty and content novelty. Category novelty represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object, that is, category novelty is used to measure the recommendation system's ability to recommend content categories that have not been recommended to the recommendation object; content novelty represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object, that is, content novelty is used to measure the recommendation system's ability to recommend content that has not been recommended to the recommendation object. The third preset ratio can be set according to actual needs, for example, it can be set to 5%, and this embodiment of the application does not limit this.

[0130] Specifically, the recommendation system can calculate the category novelty of any recommended object using the following formula:

[0131]

[0132] Wherein, μ represents the category novelty of the recommendation system for any of the recommended objects;

[0133] e represents the daily average number of categories to which any of the recommended objects have not been recommended during the set period before the current day;

[0134] E represents the number of categories recommended to any recommended object on that day.

[0135] Among them, the set time period before the current day is the previous detection cycle, that is: e is the daily average number of categories for which any recommended object has not been recommended in the previous detection cycle, and the current day may be the last day of the current detection cycle.

[0136] The average category novelty of each recommended object in the recommendation system is the average category novelty per person for all recommended objects in the recommendation system.

[0137] During implementation, the average category novelty of all recommended items can be calculated by the following method:

[0138] Calculate the ratio of the daily average number of categories that have not been recommended to all recommended objects in the set period before the current day to the average number of categories recommended to all recommended objects on the current day, and obtain the average category novelty of the recommendation system for all recommended objects.

[0139] In specific implementation, the recommendation system can calculate the content novelty of any recommendation object using the following formula:

[0140]

[0141] Wherein, ρ represents the content novelty of the recommendation system for any of the recommended objects;

[0142] f represents the daily average of the content volume corresponding to the category to which any of the recommended objects have not been recommended during the set period before the current day;

[0143] F represents the amount of content corresponding to the category recommended to any of the recommended objects on that day.

[0144] The average value of the content novelty of each recommendation object in the recommendation system is the average content novelty per person for all recommendation objects in the recommendation system.

[0145] During implementation, the average content novelty of all recommended objects can be calculated by the following method:

[0146] Calculate the ratio of the daily average amount of content corresponding to categories that have not been recommended to all recommended objects in the set period before the day to the amount of content corresponding to categories that have been recommended to all recommended objects on the day, and obtain the average content novelty of the recommendation system for all recommended objects.

[0147] Furthermore, if the detection result obtained by detecting the novelty satisfies the above-mentioned recommendation strategy update conditions, the following can be followed: Figure 5 The process shown here updates the current recommendation strategy and may include the following steps:

[0148] S401: Obtain third recommended objects whose novelty decreases by more than a third preset ratio to generate a third object set.

[0149] In specific implementation, if the recommendation system's average category novelty for all recommended objects or average content novelty for all recommended objects drops by more than a third preset ratio compared to the previous detection cycle, the current recommendation strategy is updated.

[0150] Specifically, if it is determined that the average category novelty of all recommended objects has decreased by more than a third preset proportion compared with the previous detection cycle, the third recommended objects whose category novelty has decreased by more than the third preset proportion are obtained to generate a third object set; if it is determined that the average content novelty of all recommended objects has decreased by more than a third preset proportion compared with the previous detection cycle, the third recommended objects whose content novelty has decreased by more than the third preset proportion are obtained to generate a third object set.

[0151] S402. For each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content of the third recommended object without clicks exceeds the second set number of times, the second set number of new contents will be recommended starting from the next refresh behavior.

[0152] In specific implementation, if it is determined that the average category novelty for all recommended objects has dropped by more than a third preset proportion compared with the previous detection cycle, then for each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content by the third recommended object without clicks exceeds the second set number of times, then starting from the next refresh behavior, the content in the second set number of novel categories will be recommended.

[0153] If it is determined that the average content novelty for all recommended objects has dropped by more than a third preset ratio compared to the previous detection cycle, then for each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content by the third recommended object without clicks exceeds the second set number of times, the second set number of novel contents will be recommended starting from the next refresh behavior.

[0154] Among them, the second set number of times and the second set quantity can be set according to needs. For example, the second set number of times can be set to 2 times, and the second set quantity can be set to 5. The second set number of times and the second set quantity can be adjusted according to the detection results of multiple rounds of detection cycles. The embodiments of the present application are not limited to this.

[0155] During implementation, the specific implementation methods of adding the recommendation of the second set number of novel categories and the second set number of novel contents can refer to the aforementioned implementation methods of adding the recommendation of the first set number of unrecommended categories, which will not be repeated here.

[0156] In order to achieve a balance between the diversity index of the recommendation system and other indicators of the recommendation system (such as the recommendation accuracy index, etc.), the following steps can be further included:

[0157] S403: Determine whether the fluctuation range of other indicators of the recommendation system in the current detection period is within the set range. If yes, execute step S404; otherwise, execute step S405.

[0158] S404: Maintain the updated recommendation strategy.

[0159] S405: Return the recommended strategy before the update.

[0160] Among them, other indicators of the recommendation system may be, but are not limited to, one or more indicators of recommendation accuracy, timeliness of information acquisition, and experience fluency. The embodiments of the present application do not limit this. These other indicators of the recommendation system can be calculated according to existing calculation methods, and the embodiments of the present application do not elaborate on this.

[0161] In an embodiment of the present application, the novelty of the recommendation system is used to measure the ability of the recommendation system to recommend content or content categories that have not been recommended to the recommendation object. Improving the novelty of the recommendation system is an important path to improve the diversity of the recommendation system, and it is also an important way to enhance the user's desire to explore and obtain more complete content of interest to the user.

[0162] As a possible implementation, if it is determined that the surprise level is to be detected, then it is determined that the recommendation strategy update conditions are met, specifically including:

[0163] If it is determined that the average surprise degree per person for all recommended objects decreases by more than a fourth preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0164] In specific implementations, the surprise degree includes the category surprise degree and the content surprise degree. The category surprise degree indicates the degree to which the recommendation system recommends content categories that the recommendation target has not previously recommended and that the recommendation target is interested in. The content surprise degree indicates the degree to which the recommendation system recommends content that the recommendation target has not previously recommended and that the recommendation target is interested in. The fourth preset ratio can be set as needed, for example, 5%, and is not limited in this embodiment of the present application.

[0165] Specifically, the recommendation system can calculate the category surprise degree for any recommended object using the following formula:

[0166]

[0167] Wherein, σ represents the category surprise degree of the recommendation system for any of the recommended objects;

[0168] g represents the daily average number of categories that have not been recommended to any of the recommended objects and have been clicked during the set period before the current day;

[0169] G represents the number of categories that any recommended object was recommended and clicked on that day.

[0170] Among them, the set time period before the current day is the previous detection cycle, that is, g is the daily average number of categories that have not been recommended and clicked by any of the recommended objects in the previous detection cycle.

[0171] The average of the category surprise levels of each recommended object by the recommendation system is the average category surprise level per person for all recommended objects by the recommendation system.

[0172] During implementation, the average category surprise of the recommendation system for all recommended objects can also be calculated in the following way:

[0173] Calculate the ratio of the daily average number of categories that have not been recommended and clicked by all recommended objects in the set period before the current day to the average number of categories that have been recommended and clicked by all recommended objects on the current day, and obtain the average category surprise degree of the recommendation system for all recommended objects.

[0174] In specific implementation, the recommendation system can calculate the content surprise degree for any recommended object using the following formula:

[0175]

[0176] Wherein, ω represents the content surprise degree of the recommendation system for any of the recommended objects;

[0177] h represents the daily average of the content volume corresponding to the category that has not been recommended to any of the recommended objects and has been clicked during the set period before the current day;

[0178] G represents the amount of content recommended and clicked on any one of the recommended objects on that day.

[0179] The average value of the content surprise of each recommendation object by the recommendation system is the average value of the content surprise of all recommendation objects by the recommendation system.

[0180] During implementation, the average content surprise level of all recommended objects can be calculated by the following method:

[0181] Calculate the ratio of the daily average amount of content corresponding to categories that have not been recommended and clicked by all recommended objects in the set period before the current day to the average amount of content recommended and clicked by all recommended objects on the current day, and obtain the average content surprise degree of the recommendation system for all recommended objects.

[0182] Furthermore, if the test result of the surprise degree meets the recommendation strategy update condition, the following can be used: Figure 6 The process shown here updates the current recommendation strategy and may include the following steps:

[0183] S501: Obtain fourth recommended objects whose surprise degree decreases by more than a fourth preset ratio to generate a fourth object set.

[0184] In specific implementation, if the recommendation system's average category surprise degree for all recommended objects or average content surprise degree for all recommended objects drops by more than a fourth preset ratio compared to the previous detection cycle, the current recommendation strategy is updated.

[0185] Specifically, if it is determined that the average category surprise degree for all recommended objects has dropped by more than the fourth preset proportion compared with the previous detection period, the fourth recommended object whose category surprise degree has dropped by more than the fourth preset proportion is obtained to generate a fourth object set; if it is determined that the average content surprise degree for all recommended objects has dropped by more than the fourth preset proportion compared with the previous detection period, the fourth recommended object whose content surprise degree has dropped by more than the fourth preset proportion is obtained to generate a fourth object set.

[0186] S502. For each fourth recommended object in the fourth object set, starting from the next recommendation of the fourth recommended object, after the fourth recommended object performs the refresh behavior a third set number of times, start to increase the recommendation of the third set number of surprise contents preferred by the fourth recommended object.

[0187] During specific implementation, if it is determined that the average category surprise degree for all recommended objects has dropped by more than the fourth preset proportion compared with the previous detection cycle, then for each fourth recommended object in the fourth object set, starting from the next recommendation for the fourth recommended object, after the fourth recommended object executes the refresh behavior for the third set number of times, the recommendation of the third set number of contents in the surprise category preferred by the fourth recommended object will be increased.

[0188] Specifically, after the fourth recommendation object executes the refresh behavior for the third set number of times, the click duration of the content in the surprise category recommended by the recommendation system after the fourth recommendation object is refreshed the third set number of times minus 1 can be compared, and the category with a relatively long click duration of the fourth object can be determined as the surprise category preferred by the fourth recommendation object. Then, after the fourth recommendation object executes the refresh behavior for the third set number of times, the third set number of contents in the surprise category preferred by the fourth recommendation object can be recommended.

[0189] If it is determined that the average content surprise level for all recommended objects has dropped by more than the fourth preset ratio compared to the previous detection cycle, then for each fourth recommended object in the fourth object set, starting from the next recommendation for the fourth recommended object, after the fourth recommended object has been refreshed a third set number of times, the third set number of surprise contents preferred by the fourth recommended object will be recommended.

[0190] Specifically, after the fourth recommendation object executes the refresh behavior for the third set number of times, the click time of the surprise content recommended by the recommendation system after the fourth recommendation object is refreshed the third set number of times minus 1 can be compared, and the surprise content with a relatively long click time of the fourth object can be determined as the surprise content preferred by the fourth recommendation object. Then, after the fourth recommendation object executes the refresh behavior for the third set number of times, the third set number of surprise contents preferred by the fourth recommendation object can be recommended.

[0191] Among them, the third set number of times and the third set quantity can be set according to needs. For example, the third set number of times can be set to 3 times, and the third set quantity can be set to 1. The third set number of times and the third set quantity can be adjusted according to the detection results of multiple rounds of detection cycles. The embodiments of the present application are not limited to this.

[0192] In order to achieve a balance between the diversity index of the recommendation system and other indicators of the recommendation system (such as the recommendation accuracy index, etc.), the following steps can be further included:

[0193] S503: Determine whether the fluctuation range of other indicators of the recommendation system in the current detection period is within the set range. If yes, execute step S504; otherwise, execute step S505.

[0194] S504: Maintain the updated recommendation strategy.

[0195] S505: Return the recommended strategy before the update.

[0196] Among them, other indicators of the recommendation system may be, but are not limited to, one or more indicators of recommendation accuracy, timeliness of information acquisition, and experience fluency. The embodiments of the present application do not limit this. These other indicators of the recommendation system can be calculated according to existing calculation methods, and the embodiments of the present application do not elaborate on this.

[0197] In the embodiment of the present application, the surprise index of the recommendation system measures the ability of the recommendation system to provide users with a refreshing and unexpected experience. Since users will lose their sense of freshness about the previous recommendation results of the recommendation system and may even become increasingly bored, the recommendation system needs to provide users with information that makes them feel both novel and attractive. Improving the surprise index as one of the goals of optimizing the recommendation system is an important path to improving long-term user retention.

[0198] The recommendation system evaluation method provided in the embodiment of the present application can simultaneously measure the diversity of the recommendation system from four dimensions: coverage, personalization, novelty, and surprise. It provides a path to improve the diversity of the recommendation system based on the diversity indicators of these four dimensions. It not only improves the closeness and long-term stickiness of the connection between users and recommended content, but also meets the user's broader needs for different types of personalized content. At the same time, it enhances the user's desire to explore and enables the recommendation system to obtain more complete content of interest to users, and provides users with information that is both novel and attractive, thereby improving the user's ability to retain users for a long time. Moreover, while improving the diversity of the recommendation system based on the diversity indicators of these four dimensions, it can also not affect other indicators such as the recommendation accuracy of the recommendation system, thus achieving a balance between the diversity and recommendation accuracy of the recommendation system, and is more universal for various complex diversity evaluation scenarios.

[0199] Exemplary devices

[0200] After introducing the recommendation system evaluation method according to the exemplary embodiment of the present application, the recommendation system evaluation device according to the exemplary embodiment of the present application will be introduced next.

[0201] like Figure 7 FIG. 6 is a schematic diagram of a structure of a recommendation system evaluation device 60 provided in an embodiment of the present application. In one embodiment, the recommendation system evaluation device 60 includes: a response unit 601 and an evaluation unit 602 .

[0202] A response unit 601 is configured to respond to a diversity evaluation trigger instruction of the recommendation system;

[0203] An evaluation unit 602 is configured to evaluate the recommendation system according to a detection result of a diversity index of a set recommendation system. If it is determined that a recommendation strategy update condition is met, the current recommendation strategy is updated. The detection result is obtained by detecting the diversity index in a current detection cycle. The diversity index includes at least one of the following indicators: coverage, personalization, novelty, and surprise. The coverage represents the coverage capability of the recommendation system in recommending information. The personalization represents the balance between personalized and non-personalized recommendations of the recommendation system. The novelty represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object. The surprise represents the degree to which the recommendation system recommends information that has not been recommended to the recommendation object and that the recommendation object is interested in.

[0204] In a possible implementation, if it is determined to detect the coverage, the evaluation unit 602 is specifically configured to:

[0205] If it is determined that the per capita coverage of all recommended objects decreases by more than a first preset ratio compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0206] In a possible implementation, the evaluation unit 602 is specifically configured to:

[0207] Acquire first recommended objects whose coverage drops by more than the first preset ratio to generate a first object set;

[0208] For each first recommended object in the first object set, starting from the next recommendation of the first recommended object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds the first set number of times, the first set number of unrecommended content is added for recommendation.

[0209] In a possible implementation, the coverage includes category coverage and long-term interest coverage. The category coverage represents the coverage capability of the recommendation system in recommending content categories, and the long-term interest coverage represents the coverage capability of the recommendation system in recommending content of long-term interest.

[0210] If it is determined that the category coverage per capita for all recommended objects decreases by more than a first preset ratio compared to the previous detection period, the evaluation unit 602 is specifically configured to:

[0211] Acquire first recommended objects whose category coverage drops by more than the first preset ratio to generate a first object set; and

[0212] When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the content in the first set number of categories that have not been recommended is added for recommendation.

[0213] In a possible implementation, if it is determined that the per capita long-term interest coverage of all recommended objects decreases by more than a first preset ratio compared to the previous detection cycle, the evaluation unit 602 is specifically configured to:

[0214] Acquire first recommended objects whose long-term interest coverage decreases by more than the first preset ratio to generate a first object set; and

[0215] When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the first set number of contents of the first recommendation object's long-term interests that have not been recommended are added for recommendation.

[0216] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the category coverage for any recommended object using the following formula:

[0217]

[0218] Wherein, α represents the category coverage of the recommendation system for any of the recommended objects;

[0219] a represents the number of categories of content recommended to any of the recommended objects. The recommended content represents the content recorded in the log after information filtering. A represents the total number of categories of content recommended to all recommended objects.

[0220] Alternatively, a represents the number of categories of content to which any of the recommended objects are exposed, the exposed content represents the content displayed to any of the recommended objects, and the exposed content is displayed to any of the recommended objects in the form of a display list, and A represents the total number of categories of content to which all recommended objects are exposed;

[0221] Alternatively, a represents the number of categories of content clicked by any one of the recommendation objects from the display list, and A represents the total number of categories of content clicked by all the recommendation objects from their respective corresponding display lists.

[0222] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the long-term interest coverage for any recommended object using the following formula:

[0223]

[0224] Wherein, β represents the long-term interest coverage of the recommendation system for any of the recommended objects;

[0225] b represents the content volume of the long-term interests of any of the recommended objects, and B represents the content volume of all the content recommended for any of the recommended objects;

[0226] Alternatively, b represents the recommendation volume of the content of long-term interest recommended to any of the recommendation objects, and B represents the recommendation volume of all the content recommended to any of the recommendation objects;

[0227] Alternatively, b represents the exposure amount of the content of the long-term interest to which any of the recommended objects is exposed, and B represents the exposure amount of all the content to which any of the recommended objects is exposed;

[0228] Alternatively, b represents the number of clicks on the exposed long-term interests of any recommended object, and B represents the number of clicks on all contents clicked by any recommended object;

[0229] Alternatively, b represents the browsing time of the exposed long-term interests content clicked by any of the recommended objects, and B represents the browsing time of all the content clicked by any of the recommended objects.

[0230] In a possible implementation, the personalization degree includes a personalized ratio and a non-personalized ratio, wherein the personalized ratio represents the ratio of personalized recommended content of the recommendation system to all recommended content, and the non-personalized ratio represents the ratio of non-personalized recommended content of the recommendation system to all recommended content;

[0231] If it is determined to detect the personalization degree, the evaluation unit 602 is specifically configured to:

[0232] If it is determined that the per capita personalized proportion of all recommended objects decreases by more than a second preset proportion compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0233] In a possible implementation, the evaluation unit 602 is specifically configured to:

[0234] Obtaining second recommended objects whose personalized proportions decrease by more than the second preset proportion to generate a second object set;

[0235] For each second recommended object in the second object set, if it is determined that the content currently recommended to the second recommended object does not meet its interest preferences, starting from the next recommendation to the second recommended object, the proportion of non-personalized recommended content is reduced.

[0236] In a possible implementation, the personalized proportion includes a personalized recommendation proportion, a personalized exposure proportion, or a personalized consumption proportion; and

[0237] The evaluation unit 602 is specifically configured to calculate the personalized proportion of any recommended object using the following formula:

[0238]

[0239] Wherein, γ represents the proportion of personalized recommendations of the recommendation system for any of the recommended objects;

[0240] c represents the amount of content recommended to any of the recommendation objects for personalization, and C represents the amount of all content recommended to any of the recommendation objects; or c represents the amount of content recommended to any of the recommendation objects for personalization, and C represents the amount of all content recommended to any of the recommendation objects;

[0241] Alternatively, γ represents the personalized exposure ratio of the recommendation system to any one of the recommended objects;

[0242] c represents the exposure of the content of any recommended object exposed by personalized recommendation, and C represents the exposure of all content exposed by any recommended object;

[0243] Alternatively, γ represents the personalized consumption ratio of the recommendation system for any of the recommended objects;

[0244] c represents the number of clicks on the personalized recommended content clicked by any of the recommended objects, and C represents the number of clicks on all the content clicked by any of the recommended objects; or, c represents the browsing time of the personalized recommended content clicked by any of the recommended objects, and C represents the browsing time of all the content clicked by any of the recommended objects.

[0245] In a possible implementation, the non-personalized proportion includes a non-personalized recommendation proportion, a non-personalized exposure proportion, or a non-personalized consumption proportion; and

[0246] The evaluation unit 602 is specifically configured to calculate the non-personalized ratio of any recommended object using the following formula:

[0247]

[0248] Wherein, δ represents the proportion of non-personalized recommendations of the recommendation system for any of the recommended objects;

[0249] d represents the amount of non-personalized content recommended to any of the recommended objects, and D represents the amount of all content recommended to any of the recommended objects; or d represents the amount of non-personalized content recommended to any of the recommended objects, and D represents the amount of all content recommended to any of the recommended objects;

[0250] Alternatively, δ represents the non-personalized exposure ratio of the recommendation system to any of the recommended objects;

[0251] d represents the exposure of the content of any recommended object exposed by non-personalized recommendations, and D represents the exposure of all content exposed by any recommended object;

[0252] Alternatively, δ represents the non-personalized consumption ratio of the recommendation system for any of the recommended objects;

[0253] d represents the number of clicks on the non-personalized recommended content clicked by any of the recommended objects, and D represents the number of clicks on all the content clicked by any of the recommended objects; or, d represents the browsing time of the non-personalized recommended content clicked by any of the recommended objects, and D represents the browsing time of all the content clicked by any of the recommended objects.

[0254] In a possible implementation, if it is determined to detect the novelty, the evaluation unit 602 is specifically configured to:

[0255] If it is determined that the average novelty of all recommended objects decreases by more than a third preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0256] In a possible implementation, the evaluation unit 602 is specifically configured to:

[0257] Obtaining third recommended objects whose novelty drops by more than the third preset ratio to generate a third object set;

[0258] For each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content of the third recommended object without clicks exceeds the second set number of times, the second set number of new contents will be recommended starting from the next refresh behavior.

[0259] In a possible implementation, the novelty includes category novelty and content novelty. The category novelty represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object, and the content novelty represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object.

[0260] If it is determined that the average category novelty of all recommended objects has decreased by more than a third preset ratio compared to the previous detection cycle, the evaluation unit 602 is specifically configured to:

[0261] Obtaining third recommended objects whose category novelty decreases by more than the third preset ratio to generate a third object set; and

[0262] When it is determined that the third recommendation object has continuously refreshed the recommended content without clicking on it for more than a second set number of times, the second set number of contents in the novel category will be recommended starting from the next refresh behavior.

[0263] In a possible implementation, if it is determined that the average content novelty of all recommended objects has decreased by more than a third preset ratio compared to the previous detection period, the evaluation unit 602 is specifically configured to:

[0264] A third recommended object whose content novelty decreases by more than the third preset ratio is obtained to generate a third object set.

[0265] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the category novelty of any recommended object using the following formula:

[0266]

[0267] Wherein, μ represents the category novelty of the recommendation system for any of the recommended objects;

[0268] e represents the daily average number of categories to which any of the recommended objects have not been recommended during the set period before the current day;

[0269] E represents the number of categories recommended to any recommended object on that day.

[0270] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the content novelty of any recommended object using the following formula:

[0271]

[0272] Wherein, ρ represents the content novelty of the recommendation system for any of the recommended objects;

[0273] f represents the daily average of the content volume corresponding to the category to which any of the recommended objects have not been recommended during the set period before the current day;

[0274] F represents the amount of content corresponding to the category recommended to any of the recommended objects on that day.

[0275] In a possible implementation, if it is determined to detect the degree of surprise, the evaluation unit 602 is specifically configured to:

[0276] If it is determined that the average surprise degree per person for all recommended objects decreases by more than a fourth preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

[0277] In a possible implementation, the evaluation unit 602 is specifically configured to:

[0278] Obtaining fourth recommended objects whose novelty decreases by more than the fourth preset ratio to generate a fourth object set;

[0279] For each fourth recommended object in the fourth object set, starting from the next recommendation of the fourth recommended object, after the fourth recommended object performs the refresh behavior for the third set number of times, the third set number of surprise contents preferred by the fourth recommended object will be recommended.

[0280] In a possible implementation, the surprise degree includes a category surprise degree and a content surprise degree. The category surprise degree represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation recipient and that the recommendation recipient is interested in. The content surprise degree represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation recipient and that the recommendation recipient is interested in.

[0281] If it is determined that the average category surprise degree of all recommended objects decreases by more than a fourth preset ratio compared to the previous detection cycle, the evaluation unit 602 is specifically configured to:

[0282] Obtaining fourth recommended objects whose category surprise degree decreases by more than the fourth preset ratio to generate a fourth object set; and

[0283] After the fourth recommendation object executes the refresh behavior for the third set number of times, the third set number of contents in the surprise category preferred by the fourth recommendation object will be recommended.

[0284] In a possible implementation, if it is determined that the average content surprise degree per person for all recommended objects decreases by more than a fourth preset ratio compared to the previous detection period, the evaluation unit 602 is specifically configured to:

[0285] A fourth object set is generated by obtaining fourth recommended objects whose content surprise degree decreases by more than the fourth preset ratio.

[0286] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the category surprise degree of any recommended object using the following formula:

[0287]

[0288] Wherein, σ represents the category surprise degree of the recommendation system for any of the recommended objects;

[0289] g represents the daily average number of categories that have not been recommended to any of the recommended objects and have been clicked during the set period before the current day;

[0290] G represents the number of categories that any recommended object was recommended and clicked on that day.

[0291] In a possible implementation, the evaluation unit 602 is specifically configured to calculate the content surprise degree of any recommended object using the following formula:

[0292]

[0293] Wherein, ω represents the content surprise degree of the recommendation system for any of the recommended objects;

[0294] h represents the daily average of the content volume corresponding to the category that has not been recommended to any of the recommended objects and has been clicked during the set period before the current day;

[0295] G represents the amount of content recommended and clicked on any one of the recommended objects on that day.

[0296] Based on the same inventive concept as the above-mentioned recommendation system evaluation method, the embodiment of the present application further provides an electronic device 70. Figure 8 As shown, the electronic device 70 may include a processor 701 and a memory 702 .

[0297] The processor 701 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and can implement or execute the various recommendation system evaluation methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the recommendation system evaluation method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0298] Memory 702 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory can include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disc, etc. Memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 702 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0299] It should be noted that although several units or sub-units of the recommender system evaluation device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the present application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0300] Furthermore, although the operations of the recommendation system evaluation method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0301] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the specific embodiments disclosed, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A recommendation system evaluation method, characterized in that: include: triggering an instruction in response to a diversity evaluation of the recommender system; The recommendation system is evaluated according to the detection result of the diversity index of the set recommendation system. If it is determined that the recommendation strategy update condition is met, the current recommendation strategy is updated. The detection result is obtained by detecting the diversity index in the current detection cycle. The diversity index includes at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system to recommend information. The personalization represents the balance between personalized recommendation and non-personalized recommendation of the recommendation system. The novelty represents the degree of recommendation of the recommendation system to the information that has not been recommended to the recommendation object. The surprise represents the degree of recommendation of the recommendation system to the information that has not been recommended to the recommendation object and is of interest to the recommendation object. The coverage includes category coverage. The category coverage represents the coverage ability of the recommendation system to recommend content categories. The category coverage of any recommendation object is determined according to the ratio of the number of categories of content recommended to any recommendation object to the total number of categories of content recommended to all recommendation objects. The personalization degree includes a personalization ratio, and the personalization ratio represents the ratio of the personalized recommended content of the recommendation system to all the recommended content.

2. The method according to claim 1, characterized in that If it is determined that the coverage is to be detected, it is determined that the recommended strategy update conditions are met, specifically including: If it is determined that the per capita coverage for all recommended objects decreases by more than a first preset ratio compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

3. The method according to claim 2, characterized in that Update the current recommended strategy, including: Acquire the first recommended objects whose coverage drops by more than the first preset ratio to generate a first object set; For each first recommended object in the first object set, starting from the next recommendation of the first recommended object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds a first set number of times, increase the recommendation of the first set number of unrecommended content.

4. The method according to claim 3, characterized in that If it is determined that the per capita category coverage of all recommended objects has decreased by more than a first preset ratio compared with the previous detection period, obtaining the first recommended object whose coverage has decreased by more than the first preset ratio to generate a first object set specifically includes: Acquire the first recommended objects whose category coverage drops by more than the first preset ratio to generate a first object set; and When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, adding a first set number of unrecommended content to be recommended, specifically includes: When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the first set number of contents in the category that have not been recommended are added for recommendation.

5. The method according to claim 3, characterized in that The coverage also includes long-term interest coverage, and the long-term interest coverage represents the coverage capability of the recommendation system in recommending long-term interest content; If it is determined that the per capita long-term interest coverage of all recommended objects has decreased by more than a first preset ratio compared with the previous detection period, obtaining the first recommended object whose coverage has decreased by more than the first preset ratio to generate a first object set specifically includes: Acquire the first recommended object whose long-term interest coverage decreases by more than the first preset ratio to generate a first object set; and When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, adding a first set number of unrecommended content to be recommended, specifically includes: When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the first set number of contents of the first recommendation object's long-term interests that have not been recommended are added for recommendation.

6. The method according to claim 4, characterized in that The category coverage for any recommended object is calculated using the following formula: Wherein, α represents the category coverage of the recommendation system for any of the recommended objects; a represents the number of categories of content recommended to any of the recommended objects, the recommended content represents the content recorded in the log after information filtering, and A represents the total number of categories of content recommended to all the recommended objects; Alternatively, a represents the number of categories of content exposed to any of the recommended objects, the exposed content represents the content displayed to any of the recommended objects, the exposed content is displayed to any of the recommended objects in the form of a display list, and A represents the total number of categories of content exposed to all the recommended objects; Alternatively, a represents the number of categories of content clicked by any of the recommended objects from the display list, and A represents the total number of categories of content clicked by all the recommended objects from their respective corresponding display lists.

7. The method according to claim 5, characterized in that The long-term interest coverage for any recommended object is calculated using the following formula: Wherein, β represents the long-term interest coverage of the recommendation system for any of the recommended objects; b represents the content volume of the content of the long-term interest recommended to any of the recommendation objects, and B represents the content volume of all the content recommended to any of the recommendation objects; Alternatively, b represents the recommendation amount of the content of long-term interest recommended to any of the recommendation objects, and B represents the recommendation amount of all the content recommended to any of the recommendation objects; Alternatively, b represents the exposure amount of the content in the long-term interest to which any of the recommended objects is exposed, and B represents the exposure amount of all the content to which any of the recommended objects is exposed; Alternatively, b represents the number of clicks on the exposed long-term interests of any recommended object, and B represents the number of clicks on all contents clicked by any recommended object; Alternatively, b represents the browsing time of the exposed long-term interests content clicked by any of the recommended objects, and B represents the browsing time of all the content clicked by any of the recommended objects.

8. The method according to claim 1, characterized in that The personalization degree also includes a non-personalized ratio, where the non-personalized ratio represents the ratio of the non-personalized recommended content of the recommendation system to all the recommended content; If it is determined to detect the personalization degree, it is determined that the recommendation strategy update conditions are met, specifically including: If it is determined that the per capita personalized proportion of all recommended objects decreases by more than a second preset proportion compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

9. The method according to claim 8, characterized in that Update the current recommended strategy, including: Acquire second recommended objects whose personalized proportion decreases by more than the second preset proportion to generate a second object set; For each second recommended object in the second object set, if it is determined that the content currently recommended to the second recommended object does not meet its interest preference, starting from the next recommendation of the second recommended object, the proportion of non-personalized recommended content is reduced.

10. The method according to claim 9, characterized in that The personalized proportion includes personalized recommendation proportion, personalized exposure proportion or personalized consumption proportion; as well as The personalization ratio for any recommended object is calculated using the following formula: Wherein, γ represents the proportion of personalized recommendations of the recommendation system for any one of the recommended objects; c represents the amount of content recommended to any of the recommendation objects in a personalized manner, and C represents the amount of content recommended to any of the recommendation objects; or c represents the amount of content recommended to any of the recommendation objects in a personalized manner, and C represents the amount of content recommended to any of the recommendation objects; Alternatively, γ represents the personalized exposure ratio of the recommendation system for any one of the recommended objects; c represents the exposure amount of the content of any recommended object exposed by personalized recommendation, and C represents the exposure amount of all the content exposed by any recommended object; Alternatively, γ represents the personalized consumption ratio of the recommendation system for any one of the recommended objects; c represents the number of clicks on the personalized recommended content clicked by any of the recommended objects, and C represents the number of clicks on all the content clicked by any of the recommended objects; or, c represents the browsing time of the personalized recommended content clicked by any of the recommended objects, and C represents the browsing time of all the content clicked by any of the recommended objects.

11. The method according to claim 9, characterized in that The non-personalized proportion includes the non-personalized recommendation proportion, the non-personalized exposure proportion or the non-personalized consumption proportion; and The non-personalized proportion for any recommendation object is calculated using the following formula: Wherein, δ represents the proportion of non-personalized recommendations of the recommendation system for any one of the recommended objects; d represents the amount of non-personalized recommended content for any of the recommended objects, and D represents the amount of all content recommended for any of the recommended objects; or, d represents the amount of non-personalized recommended content for any of the recommended objects, and D represents the amount of all content recommended for any of the recommended objects; Alternatively, δ represents the non-personalized exposure ratio of the recommendation system to any one of the recommended objects; d represents the exposure amount of the content of any recommended object exposed by non-personalized recommendation, and D represents the exposure amount of all the content exposed by any recommended object; Alternatively, δ represents the non-personalized consumption ratio of the recommendation system for any one of the recommended objects; d represents the number of clicks on the non-personalized recommended content clicked by any of the recommended objects, and D represents the number of clicks on all the content clicked by any of the recommended objects; or, d represents the browsing time of the non-personalized recommended content clicked by any of the recommended objects, and D represents the browsing time of all the content clicked by any of the recommended objects.

12. The method according to claim 1, characterized in that If it is determined to detect the novelty, it is determined that the recommended strategy update conditions are met, specifically including: If it is determined that the average novelty of all recommended objects decreases by more than a third preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

13. The method according to claim 12, characterized in that Update the current recommended strategy, including: Acquire third recommended objects whose novelty decreases by more than the third preset ratio to generate a third object set; For each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content of the third recommended object without clicks exceeds the second set number of times, the second set number of new contents will be recommended starting from the next refresh behavior.

14. The method according to claim 13, characterized in that The novelty includes category novelty and content novelty. The category novelty represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object, and the content novelty represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object. If it is determined that the average category novelty of all recommended objects has decreased by more than a third preset ratio compared with the previous detection cycle, obtaining third recommended objects whose novelty has decreased by more than the third preset ratio to generate a third object set specifically includes: Acquire third recommended objects whose category novelty decreases by more than the third preset ratio to generate a third object set; and When it is determined that the third recommendation object has continuously refreshed the recommended content without clicking on it for more than a second set number of times, the second set number of novel contents will be recommended starting from the next refresh behavior, specifically including: When it is determined that the third recommendation object has continuously refreshed the recommended content without clicking on it for more than a second set number of times, the second set number of contents in novel categories will be recommended starting from the next refresh behavior.

15. The method according to claim 14, characterized in that If it is determined that the average content novelty of all recommended objects decreases by more than a third preset ratio compared with the previous detection cycle, obtaining third recommended objects whose novelty decreases by more than the third preset ratio to generate a third object set specifically includes: The third recommended objects whose content novelty decreases by more than the third preset ratio are obtained to generate a third object set.

16. The method according to claim 14, characterized in that The category novelty for any recommended object is calculated using the following formula: Wherein, μ represents the category novelty of the recommendation system for any of the recommended objects; e represents the daily average number of categories that have not been recommended to any of the recommended objects in the set period before the current day; E represents the number of categories recommended to any recommended object on that day.

17. The method according to claim 14, characterized in that The content novelty for any recommendation object is calculated using the following formula: Wherein, ρ represents the content novelty of the recommendation system for any of the recommended objects; f represents the daily average value of the content volume corresponding to the category to which any of the recommended objects have not been recommended in the set period before the current day; F represents the amount of content corresponding to the category recommended for any of the recommended objects on that day.

18. The method of claim 1, wherein: If it is determined to detect the degree of surprise, it is determined that the recommendation strategy update conditions are met, specifically including: If it is determined that the average surprise degree per capita for all recommended objects decreases by more than a fourth preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

19. The method according to claim 18, characterized in that Update the current recommended strategy, including: Acquire fourth recommended objects whose novelty decreases by more than the fourth preset ratio to generate a fourth object set; For each fourth recommended object in the fourth object set, starting from the next recommendation of the fourth recommended object, after the fourth recommended object executes the refresh behavior a third set number of times, the third set number of surprise contents preferred by the fourth recommended object will be recommended.

20. The method of claim 19, wherein: The surprise degree includes category surprise degree and content surprise degree. The category surprise degree represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object and that the recommendation object is interested in. The content surprise degree represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object and that the recommendation object is interested in. If it is determined that the average category surprise degree of all recommended objects decreases by more than a fourth preset ratio compared with the previous detection cycle, obtaining a fourth recommended object whose surprise degree decreases by more than the fourth preset ratio to generate a fourth object set specifically includes: Acquire fourth recommended objects whose category surprise degree decreases by more than the fourth preset ratio to generate a fourth object set; and Starting from the fourth recommendation object executing the refresh behavior for the third set number of times, the third set number of surprise contents preferred by the fourth recommendation object are recommended, specifically including: After the fourth recommendation object executes the refresh behavior for the third set number of times, the third set number of contents in the surprise category preferred by the fourth recommendation object will be recommended.

21. The method of claim 20, wherein: If it is determined that the average content surprise degree of all recommended objects decreases by more than a fourth preset ratio compared with the previous detection cycle, obtaining a fourth recommended object whose surprise degree decreases by more than the fourth preset ratio to generate a fourth object set specifically includes: The fourth recommended objects whose content surprise degree decreases by more than the fourth preset ratio are obtained to generate a fourth object set.

22. The method of claim 20, wherein: The category surprise degree for any recommended object is calculated using the following formula: Wherein, σ represents the category surprise degree of the recommendation system for any of the recommended objects; g represents the daily average number of categories that have not been recommended and clicked by any of the recommended objects in the set period before the current day; G represents the number of categories that any recommended object is recommended and clicked on that day.

23. The method of claim 20, wherein: The content surprise degree for any recommended object is calculated using the following formula: Wherein, ω represents the content surprise degree of the recommendation system for any of the recommended objects; h represents the daily average of the content volume corresponding to the category that has not been recommended and clicked by any of the recommended objects in the set period before the current day; G represents the amount of content that was recommended and clicked on any one of the recommended objects on that day.

24. A recommendation system evaluation device, characterized in that: include: A response unit, configured to respond to a diversity evaluation trigger instruction of the recommendation system; An evaluation unit is used to evaluate the recommendation system according to the detection result of the diversity index of the set recommendation system, and update the current recommendation strategy if it is determined that the recommendation strategy update condition is met. The detection result is obtained by detecting the diversity index in the current detection cycle. The diversity index includes at least one of the following indicators: coverage, personalization, novelty and surprise. The coverage represents the coverage ability of the recommendation system to recommend information, the personalization represents the balance between personalized recommendation and non-personalized recommendation of the recommendation system, the novelty represents the degree of recommendation of the recommendation system for information that has not been recommended to the recommendation object, and the surprise represents the degree of recommendation of the recommendation system for information that has not been recommended to the recommendation object and is of interest to the recommendation object; wherein the coverage includes category coverage, the category coverage represents the coverage ability of the recommendation system to recommend content categories, and the category coverage of any recommendation object is determined according to the ratio of the number of categories of content recommended to any recommendation object to the total number of categories of content recommended to all recommendation objects; The personalization degree includes a personalization ratio, and the personalization ratio represents the ratio of the personalized recommended content of the recommendation system to all the recommended content.

25. The device according to claim 24, characterized in that If it is determined to detect the coverage, the evaluation unit is specifically used to: If it is determined that the per capita coverage for all recommended objects decreases by more than a first preset ratio compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

26. The device according to claim 25, characterized in that The evaluation unit is specifically used for: Acquire the first recommended objects whose coverage drops by more than the first preset ratio to generate a first object set; For each first recommended object in the first object set, starting from the next recommendation of the first recommended object, when it is determined that the click behavior of the first recommended object on the recommended content exceeds a first set number of times, increase the recommendation of the first set number of unrecommended content.

27. The device according to claim 26, characterized in that If it is determined that the per capita category coverage for all recommended objects decreases by more than a first preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: Acquire the first recommended objects whose category coverage drops by more than the first preset ratio to generate a first object set; and When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the first set number of contents in the category that have not been recommended are added for recommendation.

28. The device according to claim 26, characterized in that The coverage also includes long-term interest coverage, and the long-term interest coverage represents the coverage capability of the recommendation system in recommending long-term interest content; If it is determined that the per capita long-term interest coverage of all recommended objects has decreased by more than a first preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: Acquire the first recommended object whose long-term interest coverage decreases by more than the first preset ratio to generate a first object set; and When it is determined that the click behavior of the first recommendation object on the recommended content exceeds a first set number of times, the first set number of contents of the first recommendation object's long-term interests that have not been recommended are added for recommendation.

29. The device according to claim 27, characterized in that The evaluation unit is specifically used to calculate the category coverage for any recommended object by using the following formula: Wherein, α represents the category coverage of the recommendation system for any of the recommended objects; a represents the number of categories of content recommended to any of the recommended objects, the recommended content represents the content recorded in the log after information filtering, and A represents the total number of categories of content recommended to all the recommended objects; Alternatively, a represents the number of categories of content exposed to any of the recommended objects, the exposed content represents the content displayed to any of the recommended objects, the exposed content is displayed to any of the recommended objects in the form of a display list, and A represents the total number of categories of content exposed to all the recommended objects; Alternatively, a represents the number of categories of content clicked by any of the recommended objects from the display list, and A represents the total number of categories of content clicked by all the recommended objects from their respective corresponding display lists.

30. The device according to claim 28, characterized in that The evaluation unit is specifically used to calculate the long-term interest coverage for any recommended object by using the following formula: Wherein, β represents the long-term interest coverage of the recommendation system for any of the recommended objects; b represents the content volume of the content of the long-term interest recommended to any of the recommendation objects, and B represents the content volume of all the content recommended to any of the recommendation objects; Alternatively, b represents the recommendation amount of the content of long-term interest recommended to any of the recommendation objects, and B represents the recommendation amount of all the content recommended to any of the recommendation objects; Alternatively, b represents the exposure amount of the content in the long-term interest to which any of the recommended objects is exposed, and B represents the exposure amount of all the content to which any of the recommended objects is exposed; Alternatively, b represents the number of clicks on the exposed long-term interests of any recommended object, and B represents the number of clicks on all contents clicked by any recommended object; Alternatively, b represents the browsing time of the exposed long-term interests content clicked by any of the recommended objects, and B represents the browsing time of all the content clicked by any of the recommended objects.

31. The device of claim 24, wherein: The personalization degree also includes a non-personalized ratio, where the non-personalized ratio represents the ratio of the non-personalized recommended content of the recommendation system to all the recommended content; If it is determined to detect the personalization degree, the evaluation unit is specifically used to: If it is determined that the per capita personalized proportion of all recommended objects decreases by more than a second preset proportion compared with the previous detection cycle, it is determined that the recommendation strategy update condition is met.

32. The device according to claim 31, characterized in that The evaluation unit is specifically used for: Acquire second recommended objects whose personalized proportion decreases by more than the second preset proportion to generate a second object set; For each second recommended object in the second object set, if it is determined that the content currently recommended to the second recommended object does not meet its interest preference, starting from the next recommendation of the second recommended object, the proportion of non-personalized recommended content is reduced.

33. The device according to claim 32, characterized in that The personalized proportion includes personalized recommendation proportion, personalized exposure proportion or personalized consumption proportion; and The evaluation unit is specifically used to calculate the personalized proportion for any recommended object by the following formula: Wherein, γ represents the proportion of personalized recommendations of the recommendation system for any one of the recommended objects; c represents the amount of content recommended to any of the recommendation objects in a personalized manner, and C represents the amount of content recommended to any of the recommendation objects; or c represents the amount of content recommended to any of the recommendation objects in a personalized manner, and C represents the amount of content recommended to any of the recommendation objects; Alternatively, γ represents the personalized exposure ratio of the recommendation system for any one of the recommended objects; c represents the exposure amount of the content of any recommended object exposed by personalized recommendation, and C represents the exposure amount of all the content exposed by any recommended object; Alternatively, γ represents the personalized consumption ratio of the recommendation system for any one of the recommended objects; c represents the number of clicks on the personalized recommended content clicked by any of the recommended objects, and C represents the number of clicks on all the content clicked by any of the recommended objects; or, c represents the browsing time of the personalized recommended content clicked by any of the recommended objects, and C represents the browsing time of all the content clicked by any of the recommended objects.

34. The device according to claim 32, characterized in that The non-personalized proportion includes the non-personalized recommendation proportion, the non-personalized exposure proportion or the non-personalized consumption proportion; and The evaluation unit is specifically used to calculate the non-personalized proportion of any recommended object by using the following formula: Wherein, δ represents the proportion of non-personalized recommendations of the recommendation system for any one of the recommended objects; d represents the amount of non-personalized recommended content for any of the recommended objects, and D represents the amount of all content recommended for any of the recommended objects; or, d represents the amount of non-personalized recommended content for any of the recommended objects, and D represents the amount of all content recommended for any of the recommended objects; Alternatively, δ represents the non-personalized exposure ratio of the recommendation system to any one of the recommended objects; d represents the exposure amount of the content of any recommended object exposed by non-personalized recommendation, and D represents the exposure amount of all the content exposed by any recommended object; Alternatively, δ represents the non-personalized consumption ratio of the recommendation system for any one of the recommended objects; d represents the number of clicks on the non-personalized recommended content clicked by any of the recommended objects, and D represents the number of clicks on all the content clicked by any of the recommended objects; or, d represents the browsing time of the non-personalized recommended content clicked by any of the recommended objects, and D represents the browsing time of all the content clicked by any of the recommended objects.

35. The device of claim 24, wherein: If it is determined to detect the novelty, the evaluation unit is specifically configured to: If it is determined that the average novelty of all recommended objects decreases by more than a third preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

36. The device according to claim 35, characterized in that The evaluation unit is specifically used for: Acquire third recommended objects whose novelty decreases by more than the third preset ratio to generate a third object set; For each third recommended object in the third object set, starting from the next recommendation of the third recommended object, when it is determined that the continuous refresh of the recommended content of the third recommended object without clicks exceeds the second set number of times, the second set number of new contents will be recommended starting from the next refresh behavior.

37. The device according to claim 36, characterized in that The novelty includes category novelty and content novelty. The category novelty represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object, and the content novelty represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object. If it is determined that the average category novelty of all recommended objects has decreased by more than a third preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: Acquire third recommended objects whose category novelty decreases by more than the third preset ratio to generate a third object set; and When it is determined that the third recommendation object has continuously refreshed the recommended content without clicking on it for more than a second set number of times, the second set number of contents in novel categories will be recommended starting from the next refresh behavior.

38. The device according to claim 37, characterized in that If it is determined that the average content novelty of all recommended objects decreases by more than a third preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: The third recommended objects whose content novelty decreases by more than the third preset ratio are obtained to generate a third object set.

39. The device according to claim 37, characterized in that The evaluation unit is specifically used to calculate the category novelty of any recommended object by using the following formula: Wherein, μ represents the category novelty of the recommendation system for any of the recommended objects; e represents the daily average number of categories that have not been recommended to any of the recommended objects in the set period before the current day; E represents the number of categories recommended to any recommended object on that day.

40. The device according to claim 37, characterized in that The evaluation unit is specifically used to calculate the content novelty of any recommended object by using the following formula: Wherein, ρ represents the content novelty of the recommendation system for any of the recommended objects; f represents the daily average value of the content volume corresponding to the category to which any of the recommended objects have not been recommended in the set period before the current day; F represents the amount of content corresponding to the category recommended for any of the recommended objects on that day.

41. The device of claim 24, wherein: If it is determined to detect the degree of surprise, the evaluation unit is specifically configured to: If it is determined that the average surprise degree per capita for all recommended objects decreases by more than a fourth preset ratio compared to the previous detection cycle, it is determined that the recommendation strategy update condition is met.

42. The device according to claim 41, characterized in that The evaluation unit is specifically used for: Acquire fourth recommended objects whose novelty decreases by more than the fourth preset ratio to generate a fourth object set; For each fourth recommended object in the fourth object set, starting from the next recommendation of the fourth recommended object, after the fourth recommended object executes the refresh behavior a third set number of times, the third set number of surprise contents preferred by the fourth recommended object will be recommended.

43. The device according to claim 42, characterized in that The surprise degree includes category surprise degree and content surprise degree. The category surprise degree represents the degree to which the recommendation system recommends content categories that have not been recommended to the recommendation object and that the recommendation object is interested in. The content surprise degree represents the degree to which the recommendation system recommends content that has not been recommended to the recommendation object and that the recommendation object is interested in. If it is determined that the average category surprise degree of all recommended objects decreases by more than a fourth preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: Acquire fourth recommended objects whose category surprise degree decreases by more than the fourth preset ratio to generate a fourth object set; and After the fourth recommendation object executes the refresh behavior for the third set number of times, the third set number of contents in the surprise category preferred by the fourth recommendation object will be recommended.

44. The device according to claim 43, characterized in that If it is determined that the average content surprise degree of all recommended objects decreases by more than a fourth preset ratio compared with the previous detection cycle, the evaluation unit is specifically configured to: The fourth recommended objects whose content surprise degree decreases by more than the fourth preset ratio are obtained to generate a fourth object set.

45. The device according to claim 43, characterized in that The evaluation unit is specifically used to calculate the category surprise degree for any recommended object by using the following formula: Wherein, σ represents the category surprise degree of the recommendation system for any of the recommended objects; g represents the daily average number of categories that have not been recommended and clicked by any of the recommended objects in the set period before the current day; G represents the number of categories that any recommended object is recommended and clicked on that day.

46. ​​The device according to claim 43, characterized in that The evaluation unit is specifically used to calculate the content surprise degree of any recommended object by using the following formula: Wherein, ω represents the content surprise degree of the recommendation system for any of the recommended objects; h represents the daily average of the content volume corresponding to the category that has not been recommended and clicked by any of the recommended objects in the set period before the current day; G represents the amount of content that was recommended and clicked on any one of the recommended objects on that day.

47. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the recommendation system evaluation method according to any one of claims 1 to 23 are implemented.

48. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the recommendation system evaluation method according to any one of claims 1 to 23 are implemented.

Images