Recommendation system evaluation method, device, electronic device and storage medium

Abstract

The present application discloses a recommendation system evaluation method, device, electronic device, medium and computer program product, which relates to the field of artificial intelligence, and in particular to the field of big data. The specific implementation scheme is: mining the user's negative feedback result instances; screening the negative feedback result instances to obtain at least one target negative feedback result instance; and determining the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance. In the embodiment of the present application, based on the recommendation reason corresponding to the recommendation information in the feedback result instance, the problems existing in the recommendation system can be accurately mined, which provides a guarantee for the subsequent optimization of the recommendation system.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, in particular to the field of big data, and specifically to a method, device, electronic device, storage medium and computer program product for evaluating system results. Background Art

[0002] Personalized recommendations are based on user characteristics and preferences. By collecting, analyzing and defining their historical behaviors on the terminal, we ultimately understand and derive user characteristics and preferences that comply with platform rules, and recommend information and products of interest to users.

[0003] At present, personalized recommendations have been widely used in content, advertising, product and other scenario recommendations. In this recommendation system that is tailored to each individual, large-scale machine learning and other technologies are used, making the recommendation system extremely black-box and complex. As a result, how to find problems in the recommendation system and how to explain the recommendation effects have become urgent issues to be resolved. Summary of the invention

[0004] The present application provides a recommendation system evaluation method, device, electronic device, storage medium and computer program product.

[0005] According to one aspect of the present application, a recommendation system evaluation method is provided, comprising:

[0006] Mining users’ negative feedback results instances;

[0007] Screening the negative feedback result instances to obtain at least one target negative feedback result instance;

[0008] According to the recommendation reasons of the recommendation information in the target negative feedback result instance, the corresponding problem of the recommendation system is determined.

[0009] According to another aspect of the present application, a recommendation system evaluation device is provided, comprising:

[0010] A mining module, used to mine negative feedback result instances of users;

[0011] A screening module, used for screening the negative feedback result instances to obtain at least one target negative feedback result instance;

[0012] The analysis module is used to determine the corresponding problem of the recommendation system according to the recommendation reasons of the recommendation information in the target negative feedback result instance.

[0013] According to another aspect of the present application, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to at least one processor; wherein,

[0016] The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that the at least one processor can execute the recommendation system evaluation method of any embodiment of the present application.

[0017] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to enable a computer to execute the recommendation system evaluation method of any embodiment of the present application.

[0018] According to another aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the recommendation system evaluation method of any embodiment of the present application.

[0019] According to the technology of this application, the problems existing in the recommendation system can be accurately discovered, providing a guarantee for the subsequent optimization of the recommendation system.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0022] Figure 1 is a schematic diagram of a recommendation system evaluation method according to an embodiment of the present application;

[0023] Figure 2a is a schematic diagram of a recommendation system evaluation method according to an embodiment of the present application;

[0024] Figure 2b is a schematic diagram of determining a recommendation reason according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of a recommendation system evaluation method according to an embodiment of the present application;

[0026] Figure 4 is a logical schematic diagram of a recommendation system evaluation method according to an embodiment of the present application;

[0027] Figure 5 is a logical schematic diagram of a recall problem in-depth analysis according to an embodiment of the present application;

[0028] Figure 6 It is a logical schematic diagram of the in-depth analysis of the model estimation problem according to the embodiment of the present application;

[0029] Figure 7is a schematic diagram of a recommendation system evaluation device according to an embodiment of the present application;

[0030] Figure 8 It is a block diagram of an electronic device used to implement the recommendation system evaluation method of the embodiment of the present application. DETAILED DESCRIPTION

[0031] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0032] In the embodiment of the present application, user satisfaction with the recommendation system can be divided into three types: satisfied / average / disliked. The most headache-inducing aspect of the recommendation system product is the dislike of advertising, which directly affects the most important word-of-mouth and experience of the user for the product. Therefore, the embodiment of the present application first needs to start with the recommendations that users dislike, reduce the proportion of recommendations that users dislike, and thus improve the overall effect; secondly, user satisfaction with the recommendation is a subjective behavior, so there is no so-called accuracy rate for the recommendation effect, so it is necessary to collect recommendation instances that users are subjectively dissatisfied with (i.e., negative feedback result instances); finally, it is necessary to recommend explainable issues that users are dissatisfied with, and then find the problem depression or short board of the recommendation system through batch problems. For the specific implementation process, please refer to the following specific embodiments.

[0033] Figure 1 This is a flowchart of a recommendation system evaluation method according to an embodiment of the present application. This embodiment can be applied to discover problems existing in the recommendation system and how to explain the recommendation effect. The method can be performed by a recommendation system evaluation device, which is implemented in software and / or hardware and integrated in an electronic device.

[0034] For details, see Figure 1 ,The recommendation system evaluation method is as follows:

[0035] S101. Mining users' negative feedback result instances.

[0036] Among them, the negative feedback result instance includes at least the recommended information that the user is dissatisfied with and the reasons for the dissatisfaction in the recommendation system. The recommended information can be selected as advertisements, products, content, etc. What the recommendation system does is to match the recommended information that is most relevant to the user's current interests and needs, and then display it to the user, and then collect user feedback and continue to recommend. The recommendation system needs to go through multiple stages to push information, such as user portrait recognition, recommendation information model recognition, recall, click-through rate estimation and sorting (coarse sorting, fine sorting, re-sorting), and strategy filtering, and the calculations in most stages rely on complex machine learning models.

[0037] In the embodiments of the present application, the user's negative feedback result instances can be mined from any of the following feedback sources: 1) Internal evaluation data: relying on the ability of the self-built evaluation platform, the product's internal PM (Product Manager), RD (Research and Development engineer) and QA (Qualtiy Assurance) are encouraged to conduct self-evaluation and immersive satisfaction evaluation feedback on a biweekly basis. 2) Data from authoritative organizations: monthly public testing of products is initiated, and questionnaire surveys and collections are conducted on users with different regional portraits. 3) Dislike negative feedback data: derived from the experience feedback channel of the application software online, all users can provide feedback on dislike of recommended information and the reasons for disliking it. 4) Complaint data: the same as Dslike negative feedback data. 5) Public opinion data: derived from the commercial public opinion data group, public opinion is collected on external evaluations of commercial products.

[0038] Each feedback source has its own advantages and disadvantages, see Table 1 for details.

[0039] Table 1 Feedback sources

[0040]

[0041] On the basis of comprehensive consideration, internal self-evaluation, Dislike negative feedback, and data from authoritative institutions can be used as feedback sources.

[0042] Furthermore, if the feedback source is Dislike negative feedback, since the Dislike feedback source has the characteristic of large data noise, when mining the user's negative feedback result instance, the user's negative feedback result instance can be obtained by performing noise processing and behavior recognition on the negative feedback result instance included in the feedback source to ensure that the obtained negative feedback result instance is real. Specifically, first remove the data suspected of machine behavior, for example, if the user clicks on the dislike ratio is too high (machine), then filter out the user; secondly, filter out the feedback result instances for demand or interest satisfaction, and filter out the negative feedback result instances caused by repetition, material quality and other factors; finally, perform smooth weighted processing based on historical user behavior, and finally retain the real data of the corresponding feedback according to the threshold.

[0043] When authoritative agency data and internal evaluation data are used as feedback sources, these two types of data are high-quality data for user satisfaction evaluation. By distributing designed questionnaires to users, we can obtain users' true evaluations of all aspects of the advertisement. At the same time, based on the principle that users' needs and interests will not shift significantly in the short term, we can continue to track the advertising recommendation content and performance of the recommendation system in subsequent product recommendations for these users, and obtain more negative feedback result examples through posterior data mining.

[0044] S102: Screen the negative feedback result instances to obtain at least one target negative feedback result instance.

[0045] In the embodiment of the present application, after the negative feedback result instances of users are mined in S101, in order to perform statistical analysis on them, the negative feedback result instances of prominent problems or industries are screened out, so as to subsequently optimize the recommendation system in a targeted manner.

[0046] In an optional implementation, a large-scale statistical analysis method may be used to screen negative feedback result instances for different recommendation stages to obtain at least one target negative feedback result instance.

[0047] S103: Determine the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance.

[0048] In the embodiments of the present application, the reason for recommendation information refers to the explanation of why the recommendation system pushes recommendation information that the user is not satisfied with to the user. In order to determine the reason for recommendation information, the recommendation system needs to have the ability to make explainable recommendations. However, complex systems and machine learning methods make it difficult to understand the recommendation system. Based on this, the inventor proposes a method for explainable recommendations. Specifically, the embedding technology is applied to the recommendation system, and a comprehensive tracking link is collected for the online recommendations and offline models of the recommendation system, so as to achieve information collection and tracking of each step of each information recommendation from portrait recognition -> advertising model -> advertising recall -> click estimation sorting -> strategy filtering, and then determine the reason for recommendation based on the collected information.

[0049] After obtaining the reason for recommendation, the corresponding problem of the recommendation system can be determined, where the corresponding problem of the recommendation system refers to the problem to be optimized in the recommendation system. For example, the reason for recommendation is based on the user's search keyword. Since the user is not satisfied with the recommended information, it is considered that the corresponding problem of the recommendation system is the poor correlation between the search keyword and the recommended information. The correlation threshold of the recommendation system can be increased later to optimize the system.

[0050] In the embodiment of the present application, through the screening operation, negative feedback result instances with prominent problems or industries can be selected so as to subsequently optimize the recommendation system in a targeted manner; and based on the recommendation reasons corresponding to the recommendation information in the feedback result instances, the corresponding problems existing in the recommendation system can be accurately discovered, providing a guarantee for the subsequent optimization of the recommendation system.

[0051] Figure 2a is a flow chart of the recommendation system evaluation method according to an embodiment of the present application. This embodiment is optimized based on the above embodiment. Figure 2a ,The recommendation system evaluation method is as follows:

[0052] S201. Mining users' negative feedback result instances.

[0053] The negative feedback result instance at least includes the recommendation information that the user is dissatisfied with in the recommendation system and the reasons for the dissatisfaction.

[0054] S202: Determine the negative feedback rates of at least two dimensions of the recommended information in the negative feedback result instance in the recall phase, and use the negative feedback result instance with the largest dimension of negative feedback rate as the target negative feedback result instance.

[0055] In an embodiment of the present application, the dimensions of the recall stage include different industries (such as games, novels, beauty, etc.), product channels (such as keywords, interests, LBS, etc.) and recall branches (such as keywords-first-order time series model, keywords-ernie trigger, etc.), and when determining the negative feedback rate, preference degree (TGI) analysis can be performed under three dimensions, and the analysis results are used as the negative feedback rate of the corresponding dimension. Furthermore, the negative feedback result instance where the dimension with the maximum negative feedback rate is located is used as the target negative feedback result instance, thereby finding the recall channel that generates the most negative feedback result instances, so that the system optimization can be performed for the recall channel later.

[0056] S203: Determine the recommendation reason of the recommendation information in the target negative feedback result instance according to the key information of the recommendation information in the recall phase, and determine the corresponding problem of the recommendation system according to the recommendation reason.

[0057] In the embodiment of the present application, the key information of the recall stage can be determined by tracking the tracking points, wherein the key information includes the trigger branch of the recommendation information, the recall word and the original signal of the recall word (i.e., the user search keyword); and then the recommendation reason is determined based on the key information. For example, see Figure 2b , which shows a schematic diagram of determining the reason for recommendation, where the user has the attribute "male, 35-44 years old, IT communication electronics, professional technician", and the recommended information pushed is an advertisement for a hot air furnace. When analyzing the reason for recommendation, the key information of the recall stage obtained through point tracking is: the recall branch is "broad search keywords", the recall word is "gas boiler", and the original signal of the recall word is "gas payment" and "Beijing Gas Company". It can be seen that because the user has searched for the word "gas", the recommendation system recalls "gas boiler" through the expansion operation. It can be seen that the recommendation system recommends based on relevance. Since the user is not satisfied with the recommended information, it is believed that the recommendation system has a relevance problem, for example, the correlation between the recall word and the original signal of the recall word is poor.

[0058] In the embodiment of the present application, the reason for recommendation is accurately determined through key information in the recall stage, so that the recommendation is explainable, and problems in the recommendation system can be quickly discovered based on the reason for recommendation.

[0059] Furthermore, the corresponding problems include at least one of the following: the problem that the trigger word of the recommendation information is not related to the user input signal, the problem that the trigger word of the recommendation information is not related to the purchaser of the trigger word, and the problem that the user input signal is not related to the purchaser of the trigger word. It should be noted here that by classifying the corresponding problems of the recommendation system, it is convenient for subsequent evaluation and analysis.

[0060] Furthermore, after determining the corresponding problems of the recommendation system, further in-depth analysis is required to measure the user's satisfaction with the recommendation system and then provide a measurement indicator. Optionally, after obtaining the trigger word (i.e., the recall word) of the recommendation information, the original signal output by the user (i.e., the original signal of the recall word) is determined, and then the first correlation between the trigger word of the recommendation information and the original signal input by the user is calculated. Since some trigger words cannot find the corresponding original signal input by the user, the second correlation between the trigger word of the recommendation information and the browsing and clicking data of the user in the historical period (e.g., within a week) can be calculated; subsequently, the correlation between the trigger word and the original signal of the user can be enhanced by adjusting the thresholds of the first correlation and the second correlation to optimize the recommendation system.

[0061] In order to measure the relevance of the trigger word purchase to the buyer, the third correlation between the trigger word of the recommended information and the recommended information entity, recommended information industry, recommended information title and exempted keywords can be calculated; subsequently, the third correlation can be adjusted to ensure that the buyer purchases trigger words that are as relevant to themselves as possible.

[0062] Furthermore, in order to measure the correlation between the original signal input by the user and the title of the recommended information, a fourth correlation between the original signal input by the user and the title of the recommended information needs to be calculated.

[0063] The first correlation, the second correlation, the third correlation and the fourth correlation are used as the recommendation effect measurement indicators of the recommendation system. It should be noted that since the calculation and analysis are performed in batches, the output recommendation effect measurement indicators are expressed in the form of curves, and thus the user's satisfaction with the recommendation system can be intuitively determined by outputting the recommendation effect measurement indicators.

[0064] Furthermore, after discovering the problem, the recommendation system needs to be optimized. Specifically, during the iteration of the recommendation system, the project can be intercepted according to the recommendation effect measurement indicators. For example, for a certain project, if a certain indicator is found not to meet the preset conditions, the project will be taken offline to reduce the effect loss. Moreover, after the recommendation system is launched, the measurement indicators of the recommendation effect can be monitored in real time so as to understand the user's satisfaction with the recommendation system in real time.

[0065] Figure 3 is a flow chart of the recommendation system evaluation method according to an embodiment of the present application. This embodiment is optimized based on the above embodiment. Figure 3 ,The recommendation system evaluation method is as follows:

[0066] S301, mining users' negative feedback result instances.

[0067] The negative feedback result instance at least includes the recommendation information that the user is dissatisfied with in the recommendation system and the reasons for the dissatisfaction.

[0068] S302: Determine the degree to which the recommended information in the negative feedback result instance is overestimated in at least two dimensions in the model estimation stage, and use the negative feedback result instance with an overestimated degree greater than a threshold as a target negative feedback result instance.

[0069] In the embodiment of the present application, the problem of model overestimation in the recommendation scenario will cause users to be dissatisfied with the high winning rate of the recommended information. Therefore, the overestimation rule calculation is performed for the model to determine the degree of overestimation of at least two dimensions of the recommended information in the negative feedback result instance in the model estimation stage, so as to filter the negative feedback result instances where the model estimates overestimation according to the overestimation degree. It should be noted that the overestimation degree can be selected as a preset threshold, and the overestimation degree can be used to quickly filter out the overestimated negative feedback instances.

[0070] In the embodiment of the present application, the overestimation of the model may be caused by factors such as user historical preferences and the beauty of the picture. For example, through the overestimation rule calculation, the user's historical click-through rate is 3.7%, the recommended information's historical click-through rate is 0.94%, and the model-estimated click-through rate is 3.26%. It can be seen that the estimated click-through rate is significantly higher than the historical click-through rate of the recommended information, that is, the user's historical preferences lead to an overestimation of the click-through rate.

[0071] S303: Determine the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance.

[0072] Among them, the corresponding problems of the recommendation system also include the problem of model overestimation.

[0073] Furthermore, in order to further explore the problems of the recommendation system, the method also includes: for the target negative feedback result instance, determining the feature data used in the model estimation stage, wherein the feature data is determined by tracking, and the feature data is exemplarily the user's gender, age, etc., and can also be other features; replacing the feature data, re-estimating based on the replaced feature data, and sorting the feature data according to importance according to the re-estimation results, wherein importance is used to measure the impact of feature data on model estimation, and the more important the feature data, the greater the impact on model estimation; performing commonality mining based on the sorting results to obtain deviation problems in the model estimation stage in order to optimize the model. It should be noted here that not all recommended features have been applied to the model. As the environment changes, features not included in the model estimation will affect important features. Therefore, it is necessary to conduct commonality mining on negative feedback instances under similar important features to find model problems.

[0074] Figure 4is a logical diagram of a recommendation system evaluation method according to an embodiment of the present application. This embodiment is optimized based on the above embodiment. Figure 4 ,The evaluation logic of the recommendation system mainly involves the mining of effect problems. Specifically, firstly, a statistical analysis of the market is carried out, and the negative feedback problem analysis rate of different dimensions is calculated for different stages such as "advertising recall" and "model estimation". For example, in the advertising recall stage, TGI analysis is carried out under three important characteristics: different advertising industries (games, novels, beauty, etc.), product channels (such as keywords, interests, LBS, etc.), and recall branches (keywords-first-order time series model, keywords-ernie trigger, etc.), so as to find the advertising recall channels with the most problems. In the model estimation stage, the rules of overestimation of the model are calculated in the dimensions of the market, users, and advertisements, and the negative feedback result instances with overestimation of the model are screened. After two stages of screening, a set of negative feedback results is obtained.

[0075] Then, we recommend and explain each negative feedback result instance in the negative feedback result set, obtain the recommendation reason corresponding to each negative feedback result instance, and then classify the problem according to the recommendation reason, that is, determine the problem of the recommendation system. Furthermore, we conduct further evaluation and mining analysis to produce recommendation effect indicators and product closed loop.

[0076] Furthermore, due to new risk issues caused by traffic fluctuations and system fluctuations, it is also necessary to mine and recall such popular issues. Specifically, based on the feedback data, through manual analysis, we can obtain the evaluation strategy, and then through machine exploration, we can mine common issues to obtain new risk issues.

[0077] Figure 5 is a logical diagram of the in-depth analysis of the recall problem according to the embodiment of the present application, see Figure 5 It specifically shows a strategy mining and analysis method for explicit recall in advertising recall. Through market statistical analysis, it conducts in-depth analysis on the prominent problems of trigger word products (Query first order / ernie translation, etc.) placed in prominent industries such as games / novels, and finds that there are three major problems: signal transformation correlation, advertiser targeting correlation, and end-to-end correlation.

[0078] When conducting machine evaluation mining, for signal transformation correlation, based on the trigger word, check the original signal. If the original signal is not found, obtain the user's historical browsing and clicking data for one week; then calculate the correlation between the word and the original signal, and the correlation between the word and the user's historical browsing and clicking data for one week. When optimizing the subsequent recommendation system project, you only need to adjust the thresholds of these two correlations to solve the signal transformation correlation problem.

[0079] For advertiser targeting relevance, the relevance between the word and the advertising industry, advertising entity, title, and exempted keywords is calculated respectively. Since the relevance between the word and the advertising industry has the greatest impact on the advertiser targeting relevance problem, the industry threshold can be adjusted during subsequent recommendation system project optimization.

[0080] For end-to-end correlation, we can reverse the original signal and the ad title based on the word, and then calculate the correlation between the original signal and the ad title. After calculating the above correlation, we will use the settlement result as the recommendation effect measurement indicator of the recommendation system, and then we can close the product loop based on the effect indicator.

[0081] Figure 6 is a logical diagram of the in-depth analysis of the model estimation problem according to the embodiment of the present application, see Figure 6 Through the statistical analysis of the market, we screen out the negative feedback result instances that are overestimated by the model. Through in-depth analysis, we find that the overestimation of the model is mainly caused by the misleading of user historical preferences and the beauty of pictures. For each overestimated negative feedback result instance, we explain it: relying on the plain text feature interpretation tool of the model, we randomly replace features or replace them according to customized rules to obtain the most important feature set that affects this estimation. Then, we conduct commonality analysis through batches of user dissatisfied negative feedback result instances to find the deviation problems in the model estimation, so as to optimize the model. Finally, not all recommended features have been applied to the model. As the environment changes, features not included in the model estimation will affect important features. Therefore, it is necessary to conduct commonality mining for negative feedback result instances under similar important features to find model problems. Finally, the government effect set of negative feedback instance evaluation is output, and the collection data can be used to evaluate user satisfaction with the recommendation system in the future.

[0082] Figure 7 1 is a schematic diagram of the structure of the recommendation system evaluation device according to an embodiment of the present application. This embodiment can be applied to the problem of mining the recommendation system and how to explain the recommendation effect. Figure 7 As shown, the device specifically includes:

[0083] Mining module 701, used for mining negative feedback result instances of users;

[0084] A screening module 702 is used to screen the negative feedback result instances to obtain at least one target negative feedback result instance;

[0085] The analysis module 703 is used to determine the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance.

[0086] Based on the above embodiment, optionally, the screening module includes:

[0087] The first screening unit is used to determine the negative feedback rate of at least two dimensions of the recommended information in the negative feedback result instance in the recall phase, and take the negative feedback result instance where the dimension with the maximum negative feedback rate is located as the target negative feedback result instance.

[0088] Based on the above embodiment, optionally, the analysis module is used to:

[0089] According to the key information of the recommended information in the target negative feedback result instance in the recall phase, the recommendation reason of the recommended information in the target negative feedback result instance is determined, and the corresponding problem of the recommendation system is determined according to the recommendation reason.

[0090] Based on the above embodiments, optionally, the corresponding problems of the recommendation system include at least one of the following: the problem that the trigger word of the recommendation information is irrelevant to the user input signal, the problem that the trigger word of the recommendation information is irrelevant to the purchaser of the trigger word, and the problem that the user input signal is irrelevant to the purchaser of the trigger word.

[0091] Based on the above embodiment, optionally, the device further includes:

[0092] A first calculation module, used to calculate a first correlation between a trigger word of the recommendation information and an original signal input by a user, and to calculate a second correlation between the trigger word of the recommendation information and browsing click data of the user in a historical period;

[0093] A second calculation module is used to calculate the third correlation between the trigger word of the recommended information and the recommended information entity, the recommended information industry, the recommended information title and the exemption keyword;

[0094] A third calculation module, used to calculate a fourth correlation between the original signal input by the user and the title of the recommended information;

[0095] The indicator determination module is used to use the first correlation, the second correlation, the third correlation and the fourth correlation as the recommendation effect measurement indicators of the recommendation system.

[0096] Based on the above embodiment, optionally, the method further includes:

[0097] The monitoring and interception module is used to monitor the measurement indicators of recommendation effects in real time; or to intercept projects based on the measurement indicators of recommendation effects.

[0098] Based on the above embodiment, optionally, the screening module further includes:

[0099] The second screening unit is used to determine the overestimation degree of at least two dimensions of the recommended information in the negative feedback result instance in the model estimation stage, and take the negative feedback result instance with an overestimation degree greater than a threshold as the target negative feedback result instance.

[0100] Based on the above embodiment, optionally, the corresponding problem of the recommendation system also includes a model overestimation problem; the device further includes:

[0101] The feature data alkyne module is used to determine the feature data used in the model estimation stage for the target negative feedback result instance;

[0102] A replacement sorting module is used to replace the feature data, re-estimate the feature data based on the replaced feature data, and sort the feature data according to importance according to the re-estimated result;

[0103] The commonality mining module is used to conduct commonality mining based on the sorting results to obtain the deviation problems in the model estimation stage.

[0104] Based on the above embodiment, optionally, the mining module is used to:

[0105] Noise processing and behavior recognition are performed on the negative feedback result instance included in the feedback source to obtain the negative feedback result instance of the user.

[0106] The recommendation system evaluation device provided in the embodiment of the present application can execute the recommendation system evaluation method provided in any embodiment of the present application, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, please refer to the description in any method embodiment of the present application.

[0107] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0108] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0109] like Figure 8As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0110] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as the recommendation system evaluation method. For example, in some embodiments, the recommendation system evaluation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the recommendation system evaluation method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the recommendation system evaluation method in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0119] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A recommendation system evaluation method, comprising: Mining users’ negative feedback results instances; Screening the negative feedback result instances to obtain at least one target negative feedback result instance; Determine the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance; wherein the recommendation reason of the recommendation information refers to the explanation of why the recommendation system pushes the recommendation information that the user is not satisfied with to the user; wherein the corresponding problem includes at least one of the following: the problem that the trigger word of the recommendation information is not related to the user input signal, the problem that the trigger word of the recommendation information is not related to the purchaser of the trigger word, and the problem that the user input signal is not related to the purchaser of the trigger word; Calculating a first correlation between the trigger word of the recommendation information and the original signal input by the user, and calculating a second correlation between the trigger word of the recommendation information and browsing click data in the user's historical period; Calculate the third correlations between the trigger word of the recommended information and the recommended information entity, the recommended information industry, the recommended information title and the exempted keyword respectively; calculating a fourth correlation between the original signal input by the user and the title of the recommended information; The first correlation, the second correlation, the third correlation and the fourth correlation are used as recommendation effect measurement indicators of the recommendation system.

2. The method according to claim 1, wherein: Screening the negative feedback result instance to obtain at least one target negative feedback result instance includes: The negative feedback rates of at least two dimensions of the recommended information in the negative feedback result instance in the recall phase are determined, and the negative feedback result instance where the dimension with the maximum negative feedback rate is located is used as the target negative feedback result instance.

3. The method according to claim 2, wherein: Determining the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance includes: According to the key information of the recommended information in the target negative feedback result instance in the recall phase, the recommendation reason of the recommended information in the target negative feedback result instance is determined, and the corresponding problem of the recommendation system is determined according to the recommendation reason.

4. The method according to claim 1, further comprising: Performing real-time monitoring on the metrics of the recommendation effect; Alternatively, project interception is performed based on the recommendation effect measurement indicator.

5. The method according to claim 1, wherein: Screening the negative feedback result instance to obtain at least one target negative feedback result instance includes: Determine the degree of overestimation of at least two dimensions of the recommended information in the negative feedback result instance in the model estimation stage, and use the negative feedback result instance with an overestimation degree greater than a threshold as the target negative feedback result instance.

6. According to the method of claim 5, the corresponding problem of the recommendation system also includes a model overestimation problem; the method further includes: For the target negative feedback result instance, determining feature data used in the model estimation stage; replacing the characteristic data, re-estimating based on the replaced characteristic data, and sorting the characteristic data according to importance according to the re-estimation result; Based on the sorting results, commonality mining is carried out to identify the deviation problems in the model estimation stage.

7. The method according to claim 1, wherein: Examples of mining negative feedback results from users include: Noise processing and behavior recognition are performed on the negative feedback result instance included in the feedback source to obtain the negative feedback result instance of the user.

8. A recommendation system evaluation device, comprising: A mining module, used to mine negative feedback result instances of users; A screening module, used for screening the negative feedback result instance to obtain at least one target negative feedback result instance; An analysis module is used to determine the corresponding problem of the recommendation system according to the recommendation reason of the recommendation information in the target negative feedback result instance; wherein the recommendation reason of the recommendation information refers to the explanation of why the recommendation system pushes the recommendation information that the user is not satisfied with to the user; the corresponding problem includes at least one of the following: the problem that the trigger word of the recommendation information is not related to the user input signal, the problem that the trigger word of the recommendation information is not related to the purchaser of the trigger word, and the problem that the user input signal is not related to the purchaser of the trigger word; A first calculation module, used to calculate a first correlation between the trigger word of the recommendation information and the original signal input by the user, and to calculate a second correlation between the trigger word of the recommendation information and browsing click data of the user in a historical period; A second calculation module is used to calculate the third correlation between the trigger word of the recommended information and the recommended information entity, the recommended information industry, the recommended information title and the exemption keyword respectively; A third calculation module, used to calculate a fourth correlation between the original signal input by the user and the title of the recommended information; The indicator determination module is used to use the first correlation, the second correlation, the third correlation and the fourth correlation as recommendation effect measurement indicators of the recommendation system.

9. The device according to claim 8, wherein: The screening modules include: The first screening unit is used to determine the negative feedback rate of at least two dimensions of the recommended information in the negative feedback result instance in the recall phase, and take the negative feedback result instance where the dimension with the maximum negative feedback rate is located as the target negative feedback result instance.

10. The device according to claim 9, wherein: The analysis module is used to: According to the key information of the recommended information in the target negative feedback result instance in the recall phase, the recommendation reason of the recommended information in the target negative feedback result instance is determined, and the corresponding problem of the recommendation system is determined according to the recommendation reason.

11. The apparatus according to claim 8, further comprising: The monitoring and interception module is used to monitor the measurement indicators of the recommendation effect in real time; or to intercept projects according to the measurement indicators of the recommendation effect.

12. The device according to claim 8, wherein: The screening module also includes: The second screening unit is used to determine the overestimation degree of at least two dimensions of the recommended information in the negative feedback result instance in the model estimation stage, and use the negative feedback result instance with an overestimation degree greater than a threshold as the target negative feedback result instance.

13. The device according to claim 12, wherein the corresponding problem of the recommendation system further includes a model overestimation problem; the device further includes: A feature data alkyne module, for determining feature data used in a model estimation phase for the target negative feedback result instance; A replacement sorting module, used to replace the feature data, re-estimate based on the replaced feature data, and sort the feature data according to importance according to the re-estimated result; The commonality mining module is used to conduct commonality mining based on the sorting results to obtain the deviation problems in the model estimation stage.

14. The device according to claim 8, wherein: The mining module is used to: Noise processing and behavior recognition are performed on the negative feedback result instance included in the feedback source to obtain the negative feedback result instance of the user.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images