Recommendation model determination method and device and data recommendation method and device

Abstract

The invention provides a recommendation model determination method and device, a data recommendation method and device, electronic equipment, a storage medium and a computer program product, relates to the technical field of artificial intelligence and smart finance, in particular to the technical field of data recommendation, model evaluation and model testing, and can be applied to a data recommendation scene. The specific implementation scheme is as follows: for training sample data in a training sample data set, weighting the training sample data based on a weight corresponding to a category of a user to which the training sample data belongs, so as to obtain a weighted data set; and fitting the weighted training sample data in the weighted data set by using a regression model to obtain a recommendation model. According to the method and the device, the problem that the model targets of different links in the recommendation scene are inconsistent, so that the combination between the models has obstacles is avoided; moreover, based on weighted training sample data, the accuracy of the recommendation model is improved.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and smart financial technology, specifically to the fields of data recommendation, model evaluation, and model testing technology, and in particular to a method and device for determining a recommendation model, a data recommendation method and device, an electronic device, a storage medium, and a computer program product, which can be applied in data recommendation scenarios. Background Art

[0002] There are four marketing links in the marketing scenario, namely, reaching (sending), clicking (responding), completing (applying), and granting credit. The goals of multiple links are inconsistent, and the goals of multiple links may be mutually exclusive. In the existing technical solutions, the main method of determining potential marketing users is to superimpose the intention model on the qualification model. The goal of the intention model is to identify people who have the intention to purchase or click on credit products, while the qualification model mainly identifies people with good credit status. There are large differences between these two groups of people. For example, people who usually have a high intention for loan services generally have poor economic conditions and generally poor credit status. Due to the above-mentioned contradiction between the intention model and the qualification model, the marketing users determined by superimposing the intention model on the qualification model are not accurate. Summary of the invention

[0003] The present disclosure provides a method, device, electronic device, storage medium and computer program product for determining a recommendation model.

[0004] According to the first aspect, a method for determining a recommendation model is provided, comprising: weighting the training sample data in a training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs to obtain a weighted data set; and fitting the weighted training sample data in the weighted data set using a regression model to obtain a recommendation model.

[0005] According to a second aspect, a data recommendation method is provided, comprising: determining a recommended user set through a recommendation model, wherein the recommendation model is determined through any implementation method of the first aspect; and recommending target recommended data to users in the recommended user set.

[0006] According to a third aspect, a device for determining a recommendation model is provided, comprising: a weighting unit, configured to weight the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set; and a fitting unit, configured to fit the weighted training sample data in the weighted data set using a regression model to obtain a recommendation model.

[0007] According to a fourth aspect, a data recommendation device is provided, comprising: a user determination unit, configured to determine a recommended user set through a recommendation model, wherein the recommendation model is determined through any implementation method of the third aspect; and a recommendation unit, configured to recommend target recommendation data to users in the recommended user set.

[0008] According to the fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any implementation of the first aspect or the second aspect.

[0009] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a seventh aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation of the first aspect or the second aspect when executed by a processor.

[0011] According to the technology disclosed in the present invention, a method and device for determining a recommendation model are provided. First, the training sample data are weighted based on the weight corresponding to the category of the user to which the training sample data belongs; then, the weighted training sample data are fitted with a regression model to obtain a recommendation model, thereby obtaining a recommendation model based on an adapted recommendation scenario, thereby avoiding the problem of obstacles to the combination of models caused by inconsistent model objectives in different links of the recommendation scenario; and the accuracy of the recommendation model is improved based on the weighted training sample data.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0015] Figure 2 is a flow chart of an embodiment of a method for determining a recommendation model according to the present disclosure;

[0016] Figure 3This is a schematic diagram of user category classification in the credit marketing recommendation scenario according to this embodiment.

[0017] Figure 4 is a schematic diagram of an application scenario of the method for determining a recommendation model according to this embodiment;

[0018] Figure 5 is a flowchart of another embodiment of a method for determining a recommendation model according to the present disclosure;

[0019] Figure 6 is a flow chart of an embodiment of the data recommendation method according to this embodiment;

[0020] Figure 7 is a structural diagram of an embodiment of a device for determining a recommendation model according to the present disclosure;

[0021] Figure 8 is a structural diagram of an embodiment of a data recommendation device according to the present disclosure;

[0022] Fig. 9 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0024] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Figure 1 An exemplary architecture 100 is shown to which the recommendation model determination method and apparatus, and data recommendation method and apparatus of the present disclosure can be applied.

[0026] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The communication connection between the terminal devices 101, 102, 103 constitutes a topological network, and the network 104 is used to provide a medium for the communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0027] Terminal devices 101, 102, 103 may be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, 103 are hardware, they may be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smart phones, tablet computers, e-book readers, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they may be installed in the electronic devices listed above. They may be implemented as multiple software or software modules for providing distributed services, or they may be implemented as a single software or software module. No specific limitation is made here.

[0028] The server 105 may be a server that provides various services, for example, a background processing server that determines a recommended user set through a recommendation model and recommends target recommended data to the terminal devices 101, 102, and 103 used by users in the recommended user set. In addition, the server may weight the training sample data based on the weight corresponding to the category of the user to which the training sample data belongs, and then use a regression model to fit the weighted training sample data to obtain a recommendation model. As an example, the server 105 may be a cloud server.

[0029] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0030] It should also be noted that the recommendation model determination method and data recommendation method provided in the embodiments of the present disclosure are generally executed by a server, but the possibility of being executed by a terminal device, or being executed by a server and a terminal device in cooperation with each other is not excluded. Accordingly, the various parts (such as various units) included in the recommendation model determination device can be all set in the server, or all set in the terminal device, or can be set in the server and the terminal device respectively.

[0031] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is only illustrative. Any number of terminal devices, networks, and servers may be provided as required. When the electronic device on which the method for determining a recommendation model and the method for data recommendation are run does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (e.g., terminal device or server) on which the method for determining a recommendation model and the method for data recommendation are run.

[0032] Please refer to Figure 2 , Figure 2 A flowchart of a method for determining a recommendation model provided in an embodiment of the present disclosure. In process 200, the following steps are included:

[0033] Step 201 : weighting the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set.

[0034] In this embodiment, the execution subject of the method for determining the recommendation model (for example, Figure 1 The server in the data processing unit (server) can obtain the training sample data set remotely or locally through a wired network connection or a wireless network connection, and weight the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set.

[0035] The recommendation model can be a model applied to various recommendation scenarios and used to determine recommended users, including but not limited to credit marketing recommendation scenarios, information content (such as news) recommendation scenarios, e-commerce platform recommendation scenarios, and O2O (Online To Offline) service platform recommendation scenarios.

[0036] The training sample data set includes a large amount of training sample data. One training sample data corresponds to one user, including user data and labels that characterize the user's feedback behavior in the recommendation scenario. User data includes basic information such as age, gender, region, interests and preferences, behavioral habits, and other data.

[0037] Based on the degree of influence of the training sample data on the training process of the recommendation model, it can be divided into different categories and different weights can be set for it. As an example, users can be divided into different categories according to the links involved in the recommendation process in the recommendation scenario. Taking the credit marketing recommendation scenario as an example, it includes four marketing links: sending, responding, applying for loans, and granting credit. It can be divided into different categories according to the user's feedback behavior in the above links. Among them, the sending link is used to determine the user of the recommended credit product and send the credit product data to the user. The user's feedback behavior includes receiving the credit product data; the response link is used to determine whether the user responds to the credit product data. The user's feedback behavior includes responding or not responding to the credit product data. For example, clicking on the credit product is considered to be a user's response; the loan application link is used to determine whether the user triggers the credit product and applies for a loan. The user's feedback behavior includes applying for or not applying for a loan; the credit granting link is used to verify the user's loan application information and issue the loan if the verification is passed. The user's feedback behavior includes receiving the loan.

[0038] Taking the recommendation scenario of an e-commerce platform as an example, it includes four links: sending, responding, purchasing, and returning. It can be divided into different categories according to the user's feedback behavior in the above links.

[0039] In this embodiment, for each training sample data in the training sample data set, first, the category of the user to which the training sample data belongs (the user represented by the training sample data) is determined; then, based on the weight corresponding to the user category, the training sample data is weighted to obtain weighted training sample data. Furthermore, the weighted training sample data corresponding to each training sample data is combined to obtain a weighted data set.

[0040] In some optional implementations of this embodiment, before executing the above step 201, the above execution subject may further perform the following operations:

[0041] In the first step, users are divided into multiple categories based on their feedback behaviors on recommendation data in multiple links involved in the target recommendation scenario.

[0042] Continue to refer Figure 3 , which shows a schematic diagram of user category division in a credit marketing recommendation scenario.

[0043] Based on the four links of sending, responding, applying for loans, and granting credit involved in the credit marketing recommendation scenario, users are divided into the following five categories: 1. Users who fail to pass the screening, that is, users to whom no credit products are recommended; 2. Users who pass the screening but do not respond after being contacted; 3. Users who do not apply for a loan after responding; 4. Users who apply for a loan but are ultimately not approved due to qualification reasons; 5. Users who apply for a loan and are ultimately approved.

[0044] The second step is to determine the weights corresponding to the multiple categories of users according to the conversion rates corresponding to the multiple links.

[0045] The conversion rate refers to the percentage or fraction of a certain response object being converted. In different recommendation scenarios, the response objects are different. Taking the credit marketing scenario as an example, when a user who has not applied for a loan applies for and is approved for a loan because of a recommended credit product, it means that the user has been converted.

[0046] The conversion rate corresponding to a link represents the conversion rate of users who enter this link. Taking the response link in the credit marketing recommendation scenario as an example, users who are recommended credit products can be considered as users who enter the response link. If the number of users who enter the response link is 10,000, and the number of users who click on the credit product, apply for a loan and are approved is 200 among the 10,000 users, then the conversion rate corresponding to the response link is considered to be 2%.

[0047] In this implementation, a one-to-one correspondence between multiple links and multiple user categories can be preset. Continuing with the credit marketing recommendation scenario as an example, a one-to-one correspondence between the five links of unmarketing, sending, responding, applying for loans, and credit granting and the above five categories can be preset.

[0048] For each of the multiple categories, the weight corresponding to the users in the category is determined according to the conversion rate of the link corresponding to the category and the principle that the conversion rate is positively correlated with the weight.

[0049] In this implementation, the feedback behavior of the recommendation data in multiple links involved in the user target recommendation scenario is categorized, and the weights corresponding to the categories are determined according to the conversion rates corresponding to the links, so that the weight determination process is closely adapted to the recommendation scenario, thereby improving the accuracy of the weight determination process.

[0050] In some optional implementations of this embodiment, the above-mentioned execution entity can perform the above-mentioned second step in the following manner: for at least one link other than the preset link among the multiple links, determine the weight corresponding to the category of users whose feedback behavior ends at the link according to the conversion rate corresponding to the link, the conversion rate corresponding to the preset link, and the weight corresponding to the category of users whose feedback behavior ends at the preset link.

[0051] The weight corresponding to the category of the user whose feedback behavior ends at the preset stage is the preset weight. In order to improve the referenceability of the preset stage, the stage that is earlier in the time sequence is generally selected. For example, the preset stage is the sending stage.

[0052] Taking a link other than the preset links among multiple links as an example, this link can be any link other than the preset links among the multiple links. The conversion rate corresponding to the link is divided by the conversion rate corresponding to the preset link, and multiplied by the weight corresponding to the category of the user whose feedback behavior ends at the preset link, to determine the weight corresponding to the category of the user whose feedback behavior ends at this link.

[0053] Furthermore, for categories whose weights have not yet been determined, the weights corresponding to the categories can be determined based on the time series relationship between the links corresponding to different categories, in accordance with the principle that time series is positively correlated with weights, and based on the weights calculated in the above manner. Among them, there is a corresponding relationship between the category of users whose feedback behavior ends at this link and the link. Continuing with the credit marketing recommendation scenario as an example, if the weight corresponding to the category of users whose feedback behavior ends at the response link is determined to be 15 based on the above calculation method, on this basis, the weight of the category of users whose feedback behavior ends at the loan application link is obtained by increasing the preset value or increasing the preset ratio.

[0054] In some implementations, for each link other than a preset link in multiple links, the weight corresponding to the category of users whose feedback behavior ends at the link is determined based on the conversion rate corresponding to the link, the conversion rate corresponding to the preset link, and the weight corresponding to the category of users whose feedback behavior ends at the preset link.

[0055] Continuing to refer to the following table, it shows the weight calculation method in the credit marketing scenario.

[0056]

[0057]

[0058] In the credit marketing scenario, the corresponding data push has not been carried out in the pre-marketing stage, so the conversion rate and weight are 0.

[0059] In this example, the preset link is the sending link, the conversion rate of users entering this link is 0.08%, and its weight is set to the preset weight 1; for the response link, the conversion rate of users entering this link is 2%, the conversion rate corresponding to the preset link is 0.08% and the weight corresponding to the category of users whose feedback behavior ends at the preset link (credit products are not recommended) is 1, and the weight corresponding to the category of users whose feedback behavior ends at the response link (credit products are recommended but do not respond) is 2% ÷ 0.08% × 1 = 15.

[0060] Similarly, for the loan application stage, the conversion rate of users entering this stage is 15%, the conversion rate corresponding to the preset stage is 0.08%, and the weight corresponding to the category of users whose feedback behavior ends at the preset stage (no credit product is recommended) is 1. The weight corresponding to the category of users whose feedback behavior ends at the loan application stage (who applied for a loan but was not approved) is 15% ÷ 0.08% × 1 = 187.5.

[0061] For the credit granting stage, the conversion rate of users entering this stage is 100%, the conversion rate corresponding to the preset stage is 0.08%, and the weight corresponding to the category of users whose feedback behavior ends at the preset stage (no credit product is recommended) is 1. The weight corresponding to the category of users whose feedback behavior ends at the credit granting stage (who have applied for a loan and have been approved) is 100% ÷ 0.08% × 1 = 1250.

[0062] In this implementation, the weights corresponding to the categories are determined based on the conversion rates corresponding to the links, so that the weight determination process is closely adapted to the recommendation scenarios. The weights of the categories corresponding to other links are further calculated based on the weights and conversion rates corresponding to the preset links, thereby improving the accuracy of the weights.

[0063] Step 202: Use a regression model to fit the weighted training sample data in the weighted data set to obtain a recommendation model.

[0064] In this embodiment, the execution entity may use a regression model to fit the weighted training sample data in the weighted data set to obtain a recommendation model.

[0065] The regression model includes but is limited to a linear regression model, a multivariate linear regression model, a ridge regression model, a Lasso (Least Absolute Shrinkage and Selection Operator) regression model, a decision tree regression model, a random forest regression model, a gradient boosting regression model, and an XGBoost (Extreme Gradient Boosting) regression model.

[0066] As an example, first, according to the characteristics of the weighted training sample data in the weighted data set and the recommended scenario, a suitable regression model is selected. Then, for the selected regression model, its mathematical formula is set. Taking decision tree regression as an example, the decision tree constructs a tree model by recursively dividing the data set into smaller subsets. Each internal node represents a feature, each branch represents a value of the feature, and each leaf node represents a category or predicted value. At each segmentation, it is necessary to select the optimal feature for segmentation according to a certain indicator (such as information gain, Gini coefficient), which requires setting the mathematical calculation formula of information gain and Gini coefficient. Then, parameter estimation is performed to find a set of optimal parameter values ​​so that the model can best fit the data. Common parameter estimation methods include least squares method, gradient descent method, etc. Finally, after determining the parameter estimation method, the weighted training sample data in the weighted data set is used to train the model. For example, taking the weighted user data in the weighted training sample data as input and the weighted labels in the weighted training sample data as expected output, a regression model is trained so that the regression model fits the weighted training sample data in the weighted data set to obtain a recommendation model.

[0067] Continue to see Figure 4 , Figure 4400 is a schematic diagram of an application scenario of the method for determining a recommendation model according to the present embodiment. First, the server obtains a training sample data set 401 from a database. One training sample data in the training sample data set corresponds to a user, including user data and a label representing the user's feedback behavior in each link of the recommendation scenario. Then, according to the label representing the feedback behavior, the users are divided into multiple categories, and different weights are set for users of multiple categories. Then, for the training sample data in the training sample data set, the training sample data is weighted based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set 402. Finally, the weighted training sample data in the weighted data set is fitted using a regression model to obtain a recommendation model 403.

[0068] In this embodiment, a method for determining a recommendation model is provided. First, the training sample data is weighted based on the weight corresponding to the category of the user to which the training sample data belongs. Then, a regression model is used to fit the weighted training sample data to obtain a recommendation model, thereby obtaining a recommendation model based on an adapted recommendation scenario. This avoids the problem of inconsistent model objectives in different links of the recommendation scenario, which leads to obstacles in the combination of models. Furthermore, the accuracy of the recommendation model is improved based on the weighted training sample data.

[0069] In some optional implementations of this embodiment, the above execution subject may also perform the following operations:

[0070] In the first step, for a target link among multiple links involved in the target recommendation scenario, perform the following operations:

[0071] Firstly, for a first recommended user set obtained by the recommendation model based on the evaluation sample data set, evaluation index data of the first recommended user set under the target link is determined.

[0072] Among them, the target link can be a link with evaluation value, such as the response link, loan application link and credit granting link in the credit recommendation scenario.

[0073] The evaluation sample data in the evaluation sample data set is input into the evaluation sample data set, and the recommendation model selects the first recommended user set from the user set corresponding to the evaluation sample data set, and recommends the target recommendation data to the users in the first recommended user set, obtains the feedback behavior of the users in the first recommended user set in each target link, and determines the evaluation index data under each target link. Different evaluation sample data sets can be used for different target links.

[0074] The evaluation index data include, for example, prediction accuracy, recall rate, precision rate and other index data.

[0075] Then, the evaluation index data is weighted according to the weight corresponding to the category of the user whose feedback behavior ends at the target stage, so as to obtain the weighted evaluation index data.

[0076] For each target link, the evaluation index data under the target link is weighted according to the weight corresponding to the category of the user whose feedback behavior ends at the target link, so as to obtain the weighted evaluation index data of the recommendation model under the target link.

[0077] Continuing with the credit marketing recommendation scenario as an example, for the evaluation index data in the response stage, it is weighted according to the weight 25 corresponding to the category of users whose feedback behavior ends at the response stage (no loan application was made after the response); for the evaluation index data in the loan application stage, it is weighted according to the weight 187.5 corresponding to the category of users whose feedback behavior ends at the loan application stage (loan application was made but not approved); for the evaluation index data in the credit granting stage, it is weighted according to the weight 1250 corresponding to the category of users whose feedback behavior ends at the response stage (loan application was approved).

[0078] The second step is to determine the evaluation index data corresponding to the recommendation model according to the weighted evaluation index data corresponding to each of the multiple target links.

[0079] As an example, the weighted evaluation index data corresponding to each of the multiple target links are added together to obtain the evaluation index data corresponding to the recommendation model.

[0080] In this implementation, for the evaluation process of the recommendation model, weighted indicator data is used to evaluate the recommendation model, thereby improving the accuracy of the evaluation indicator data.

[0081] The execution subject may determine whether to continue training the recommendation model according to the evaluation index data. In response to determining that the recommendation model needs to be continued to be trained, the recommendation model is continued to be trained through the above steps 201-202.

[0082] For example, in response to determining that the indicator value represented by the evaluation indicator data is lower than a preset threshold, it indicates that the recommendation model is relatively lacking in prediction accuracy, recall rate, precision rate, etc., and needs to continue training.

[0083] In some optional implementations of the present embodiment, the execution entity may execute the process of determining the evaluation index data in the following manner: taking users in the first recommended user set who make feedback in the target link as actual positive samples, taking users in the previous link of the first recommended user set whose feedback ends at the target link as actual negative samples, taking a preset proportion of users ranked at the front in the first recommended user set as predicted positive samples, and taking users other than the preset proportion of users in the first recommended user set as predicted negative samples, to determine the evaluation index data of the recommendation model in the target link.

[0084] The preset ratio may be set according to actual conditions, for example, 5%.

[0085] Continuing with the response phase in the credit marketing recommendation scenario as an example, users who make feedback in the response phase (for example, click on the recommended credit product) of the first recommended user group are taken as actual positive samples, users in the previous phase of the target phase (for example, users who did not respond and users to whom no credit products were recommended) of the first recommended user group are taken as actual negative samples, a preset proportion of users ranked at the top of the first recommended user group are taken as predicted positive samples, and users other than the preset proportion of users in the first recommended user group are taken as predicted negative samples, to determine the evaluation index data of the recommendation model in the response phase.

[0086] The evaluation index data is, for example, the F1 score. The calculation formula is as follows:

[0087] F1=2×(Precision×Recall) / (Precision+Recall).

[0088] Among them, Precision represents the accuracy rate and Recall represents the recall rate.

[0089] The F1 score is the harmonic mean of precision and recall, and is used to comprehensively evaluate the performance of the model. When both precision and recall are high, the F1 score will also be high. A higher F1 score indicates that the model has achieved a good balance between precision and recall.

[0090] In this implementation, a specific method for determining evaluation index data is provided, which improves the accuracy of the evaluation index data.

[0091] In some optional implementations of this embodiment, the above execution subject may perform the following operations:

[0092] In the first step, the test sample data in the test sample data set is input into a plurality of models to be tested including a recommendation model, and a plurality of recommended user subsets corresponding one-to-one to the plurality of models to be tested are determined.

[0093] The test sample data includes user data.

[0094] In this implementation, the purpose is to test multiple models to be tested simultaneously. The test sample data in the test sample data set are respectively input into multiple models to be tested, and each model to be tested selects a recommended user subset from the user set corresponding to the test sample data set.

[0095] The second step is to merge multiple recommended user subsets to obtain a second recommended user set.

[0096] The third step is to combine at least one model to be tested from the multiple models to be tested to obtain multiple model groups.

[0097] The models to be tested include A, B, and C, and the multiple model groups include (A), (B), (C), (A, B)(A, C), (B, C), and (A, B, C).

[0098] In the fourth step, for each model group in the multiple model groups, the recommended users determined by all the models in the model group in the second recommended user set are divided into one user group to obtain multiple user groups corresponding to the multiple model groups one by one.

[0099] Continuing to refer to the following table, the divided user groups are shown.

[0100] Grouping Model A to be tested Model B to be tested Model C to be tested 1 0 0 1 2 0 1 0 3 0 1 1 4 1 0 0 5 1 0 1 6 1 1 0 7 1 1 1

[0101] For the model groups (A), (B), and (C) that include only one model to be tested, the recommended users in the second recommended user set determined by the single model to be tested in the model group are divided into one user group, and user groups 4, 2, and 1 corresponding to the model groups (A), (B), and (C) are obtained respectively; for the model groups (A, B)(A, C), and (B, C) that include two models to be tested, the recommended users in the second recommended user set determined by both models to be tested in the model group are divided into one user group, and user groups 6, 5, and 3 corresponding to the model groups (A, B)(A, C), and (B, C) are obtained respectively; for the model group (A, B, C) that includes three models to be tested, the recommended users in the second recommended user set determined by all three models to be tested in the model group are divided into one user group, and user group 7 corresponding to the model group (A, B, C) is obtained.

[0102] The fifth step is to determine the test results corresponding to the multiple models to be tested according to the conversion rates corresponding to the multiple user groups.

[0103] In this implementation, first, the conversion rates corresponding to the multiple user groups are determined; then, the test results corresponding to the multiple models to be tested are determined according to the conversion rates corresponding to the user groups respectively related to the multiple models to be tested.

[0104] Taking the model A to be tested as an example, its related user groups include 4 and 5. The test result of the model A to be tested can be determined according to the conversion rates corresponding to the user groups 4 and 5. For example, the conversion rates corresponding to the user groups 4 and 5 are added together as the test result of the model A to be tested.

[0105] In this implementation, a parallel testing method is provided for testing multiple models to be tested at the same time. Users in the merged second recommended user set are grouped based on different model groups, so as to determine the test results corresponding to each of the multiple models to be tested based on the conversion rates corresponding to each of the multiple user groups, thereby reducing the number of tests and the testing cost; and, based on the same test sample data set, the sample deviation in the testing process is reduced.

[0106] The execution subject may determine whether to continue training the recommendation model according to the test result. In response to determining that the recommendation model needs to be continued to be trained, the recommendation model is continued to be trained through the above steps 201-202.

[0107] In some optional implementations of the present embodiment, the execution entity may perform the fifth step as follows: for each model to be tested among multiple models to be tested, determine a target user group determined by the model to be tested from multiple user groups, and determine a test result corresponding to the model to be tested based on a conversion rate corresponding to the target user group.

[0108] Continuing to refer to the above table, the target user groups determined by the model to be tested A include 4, 5, 6, and 7; the target user groups determined by the model to be tested B include 2, 3, 6, and 7; and the target user groups determined by the model to be tested C include 1, 3, 5, and 7.

[0109] As an example, for each of the multiple models to be tested, multiple target user groups determined by the model to be tested are determined from multiple user groups, and the average of the conversion rates corresponding to the multiple target user groups is calculated and determined as the test result corresponding to the model to be tested.

[0110] In this implementation, the test result of the model to be tested is determined based on the conversion rates of multiple target user groups determined by the model to be tested, thereby improving the accuracy of the test result.

[0111] Continue to refer Figure 5 , shows a schematic process 500 of another embodiment of a method for determining a recommendation model according to the present disclosure. In process 500, the following steps are included:

[0112] Step 501 , based on the user's feedback behavior on the recommendation data in multiple links involved in the target recommendation scenario, the users are divided into multiple categories.

[0113] Step 502: Determine the weights corresponding to the multiple categories of users according to the conversion rates corresponding to the multiple links.

[0114] Step 503 : weighting the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set.

[0115] Step 504: Use the regression model to fit the weighted training sample data in the weighted data set to obtain a recommendation model.

[0116] Step 505, for the target link among the multiple links involved in the target recommendation scenario, perform the following operations: for the first recommended user set obtained by the recommendation model based on the evaluation sample data set, determine the evaluation index data of the first recommended user set in the target link; according to the weight corresponding to the category of the user whose feedback behavior ends at the target link, weight the evaluation index data to obtain the weighted evaluation index data.

[0117] Step 506: Determine the evaluation index data corresponding to the recommendation model according to the weighted evaluation index data corresponding to each of the multiple target links.

[0118] Step 507: input the test sample data in the test sample data set into a plurality of models to be tested including the recommendation model, and determine a plurality of recommended user subsets corresponding one-to-one to the plurality of models to be tested.

[0119] Step 508: merge multiple recommended user subsets to obtain a second recommended user set.

[0120] Step 509: combine at least one model to be tested among the multiple models to be tested to obtain multiple model groups.

[0121] Step 510 : for each model group in the multiple model groups, divide the recommended users determined by all the models in the model group in the second recommended user set into one user group, so as to obtain multiple user groups corresponding to the multiple model groups one by one.

[0122] Step 511 , for each of the multiple models to be tested, a target user group determined by the model to be tested is determined from multiple user groups, and a test result corresponding to the model to be tested is determined according to a conversion rate corresponding to the target user group.

[0123] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiments, process 500 of the method for determining the recommendation model in this embodiment specifically illustrates the weight determination process, the evaluation process and the testing process of the recommendation model, and improves the accuracy of the recommendation model, the accuracy of the evaluation process and the convenience of the testing process based on the weighted training sample data.

[0124] Continue to refer Figure 6 , shows a schematic process 600 of an embodiment of a method for recommending data according to the present disclosure. In process 600, the following steps are included:

[0125] Step 601: determine a recommended user set through a recommendation model.

[0126] In this embodiment, the execution subject of the recommended data method (for example, Figure 1 The server in the example 200 may determine the recommended user set through a recommendation model. The recommendation model may be determined through any implementation method in the above embodiments 200 and 500.

[0127] The recommendation model can be a model applied to various recommendation scenarios and used to determine recommended users, including but not limited to credit marketing recommendation scenarios, information content (such as news) recommendation scenarios, e-commerce platform recommendation scenarios, and O2O (Online To Offline) service platform recommendation scenarios.

[0128] Taking the credit marketing recommendation scenario as an example, the above-mentioned execution entity can input the user data of a large number of candidate users into the recommendation model, determine whether each candidate user is a recommended user, and obtain a recommended user set.

[0129] Step 602: recommend target recommendation data to users in the recommendation user set.

[0130] In this embodiment, the above-mentioned execution entity may recommend the target recommendation data to the users in the recommendation user set.

[0131] As an example, the above execution entity may send the target recommendation data to terminal devices used by users in the recommendation user set.

[0132] In this embodiment, a method for determining a recommendation model is provided, which improves the accuracy of the determined recommended users based on the recommendation model.

[0133] Continue to refer Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a device for determining a recommendation model. The system embodiment is Figure 2 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0134] like Figure 7 As shown, the device 700 for determining the recommendation model includes: a weighting unit 701 is configured to weight the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set; and a fitting unit 702 is configured to fit the weighted training sample data in the weighted data set using a regression model to obtain a recommendation model.

[0135] In some optional implementations of the present embodiment, the above-mentioned device also includes: a weight determination unit (not shown in the figure) is configured to: divide users into multiple categories based on their feedback behavior on the recommendation data in multiple links involved in the target recommendation scenario; and determine the weights corresponding to each of the multiple categories of users according to the conversion rates corresponding to each of the multiple links.

[0136] In some optional implementations of this embodiment, the weight determination unit (not shown in the figure) is further configured to: for at least one link other than the preset link among the multiple links, determine the weight corresponding to the category of users whose feedback behavior ends at the link based on the conversion rate corresponding to the link, the conversion rate corresponding to the preset link, and the weight corresponding to the category of users whose feedback behavior ends at the preset link, wherein the weight corresponding to the category of users whose feedback behavior ends at the preset link is the preset weight.

[0137] In some optional implementations of the present embodiment, the above-mentioned device also includes: an evaluation unit (not shown in the figure) is configured to: for a target link among multiple links involved in a target recommendation scenario, perform the following operations: for a first recommended user set obtained by the recommendation model based on an evaluation sample data set, determine the evaluation index data of the first recommended user set under the target link; weight the evaluation index data according to the weight corresponding to the category of the user whose feedback behavior ends at the target link to obtain weighted evaluation index data; and the evaluation unit (not shown in the figure) is also configured to: determine the evaluation index data corresponding to the recommendation model according to the weighted evaluation index data corresponding to each of the multiple target links.

[0138] In some optional implementations of the present embodiment, the evaluation unit (not shown in the figure) is further configured to: use users in the first recommended user set who make feedback behavior in the target link as actual positive samples, use users in the previous link of the recommendation results whose feedback behavior ends at the target link as actual negative samples, use a preset proportion of users ranked at the front in the first recommended user set as predicted positive samples, and use users other than the preset proportion of users in the first recommended user set as predicted negative samples, to determine the evaluation index data of the recommendation model in the target link.

[0139] In some optional implementations of the present embodiment, the above-mentioned device also includes: a testing unit (not shown in the figure) is configured to: input the test sample data in the test sample data set into multiple models to be tested including the recommendation model, and determine multiple recommended user subsets corresponding one-to-one to the multiple models to be tested; merge the multiple recommended user subsets to obtain a second recommended user set; combine at least one model to be tested among the multiple models to be tested to obtain multiple model groups; for each model group in the multiple model groups, divide the recommended users determined by all models in the model group in the second recommended user set into a user group to obtain multiple user groups corresponding one-to-one to the multiple model groups; determine the test results corresponding to each of the multiple models to be tested according to the conversion rates corresponding to each of the multiple user groups.

[0140] In some optional implementations of the present embodiment, the testing unit (not shown in the figure) is further configured to: for each model to be tested among multiple models to be tested, determine the target user group determined by the model to be tested from multiple user groups, and determine the test result corresponding to the model to be tested based on the conversion rate corresponding to the target user group.

[0141] In this embodiment, a device for determining a recommendation model is provided. First, the training sample data is weighted based on the weight corresponding to the category of the user to which the training sample data belongs. Then, a regression model is used to fit the weighted training sample data to obtain a recommendation model, thereby obtaining a recommendation model based on an adapted recommendation scenario. This avoids the problem of inconsistent model objectives in different links in the recommendation scenario, which leads to obstacles in the combination of models. Furthermore, the accuracy of the recommendation model is improved based on the weighted training sample data.

[0142] Continue to refer Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a data recommendation device. The system embodiment is Figure 6 Corresponding to the method embodiment shown, the system can be specifically applied to various electronic devices.

[0143] like Figure 8 As shown, the determination device 800 of the recommendation model includes: a user determination unit 801 is configured to determine a recommended user set through a recommendation model, wherein the recommendation model is determined through any implementation method of embodiment 600; a recommendation unit 802 is configured to recommend target recommendation data to users in the recommended user set.

[0144] In this embodiment, a device for determining a recommendation model is provided, which improves the accuracy of the determined recommended users based on the recommendation model.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the method for determining a recommendation model and the method for data recommendation described in any of the above embodiments when executing.

[0146] According to an embodiment of the present disclosure, the present disclosure further provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the method for determining a recommendation model and the method for data recommendation described in any of the above embodiments when executed.

[0147] The embodiments of the present disclosure provide a computer program product. When the computer program is executed by a processor, the computer program can implement the method for determining a recommendation model and the method for data recommendation described in any of the above embodiments.

[0148] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0149] like Fig. 9 As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0150] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0151] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as the determination method of the recommendation model. For example, in some embodiments, the determination method of the recommendation model may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the determination method of the recommendation model described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the determination method of the recommendation model in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable recommendation model determination device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0157] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services; it may also be a server for a distributed system, or a server combined with a blockchain.

[0158] According to the technical solution of the embodiment of the present disclosure, a method for determining a recommendation model is provided. First, the training sample data is weighted based on the weight corresponding to the category of the user to which the training sample data belongs; then, the weighted training sample data is fitted with a regression model to obtain a recommendation model, thereby obtaining a recommendation model based on an adapted recommendation scenario, thereby avoiding the problem of inconsistent model objectives in different links in the recommendation scenario, which leads to obstacles in the combination of models; and the accuracy of the recommendation model is improved based on the weighted training sample data.

[0159] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for determining a recommendation model, comprising: For the training sample data in the training sample data set, weighting the training sample data based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set; The weighted training sample data in the weighted data set is fitted using a regression model to obtain the recommendation model.

2. The method according to claim 1, wherein: Before weighting the training sample data based on the weight corresponding to the category of the user to which the training sample data belongs, the method further includes: Based on the user's feedback behavior on the recommended data in multiple links involved in the target recommendation scenario, users are divided into multiple categories; The weights corresponding to the multiple categories of users are determined according to the conversion rates corresponding to the multiple links.

3. The method according to claim 2, wherein: Determining the weights corresponding to the multiple categories of users according to the conversion rates corresponding to the multiple links respectively includes: For at least one link other than the preset link among the multiple links, the weight corresponding to the category of users whose feedback behavior ends at the link is determined according to the conversion rate corresponding to the link, the conversion rate corresponding to the preset link, and the weight corresponding to the category of users whose feedback behavior ends at the preset link, wherein the weight corresponding to the category of users whose feedback behavior ends at the preset link is the preset weight.

4. The method according to any one of claims 1 to 3, wherein: Also includes: For a target link among the multiple links involved in the target recommendation scenario, perform the following operations: For a first recommended user set obtained by the recommendation model based on the evaluation sample data set, determining evaluation index data of the first recommended user set under the target link; The evaluation index data is weighted according to the weight corresponding to the category of the user whose feedback behavior ends at the target stage to obtain the weighted evaluation index data; as well as Also includes: The evaluation index data corresponding to the recommendation model is determined according to the weighted evaluation index data corresponding to each of the plurality of target links.

5. The method according to claim 4, wherein: The step of determining evaluation index data of the first recommended user set obtained by the recommendation model based on the evaluation sample data set under the target link includes: The users who have made feedback in the target link as the first recommended users are taken as actual positive samples, the users who have made feedback in the previous link of the target link as the first recommended users are taken as actual negative samples, the users who are ranked at the top in a preset proportion in the first recommended users are taken as predicted positive samples, and the users other than the preset proportion in the first recommended users are taken as predicted negative samples, so as to determine the evaluation index data of the recommendation model in the target link.

6. The method according to any one of claims 1 to 3, wherein: Also includes: Inputting the test sample data in the test sample data set into a plurality of models to be tested including the recommendation model, and determining a plurality of recommended user subsets corresponding one-to-one to the plurality of models to be tested; Merging the multiple recommended user subsets to obtain a second recommended user set; Combining at least one model to be tested among the multiple models to be tested to obtain multiple model groups; For each model group in the multiple model groups, dividing the recommended users determined by all models in the second recommended user set into one user group, so as to obtain multiple user groups corresponding to the multiple model groups one by one; According to the conversion rates corresponding to the multiple user groups, the test results corresponding to the multiple models to be tested are determined.

7. The method according to claim 6, wherein: The step of determining the test results corresponding to the multiple models to be tested respectively according to the conversion rates corresponding to the multiple user groups respectively includes: For each model to be tested among the multiple models to be tested, a target user group determined by the model to be tested is determined from the multiple user groups, and a test result corresponding to the model to be tested is determined according to a conversion rate corresponding to the target user group.

8. A data recommendation method, comprising: Determining a recommended user set by a recommendation model, wherein the recommendation model is determined by any one of claims 1 to 7; The target recommendation data is recommended to users in the recommendation user set.

9. A device for determining a recommendation model, comprising: a weighting unit configured to weight the training sample data in the training sample data set based on the weight corresponding to the category of the user to which the training sample data belongs, so as to obtain a weighted data set; The fitting unit is configured to fit the weighted training sample data in the weighted data set using a regression model to obtain the recommendation model.

10. The device according to claim 9, wherein: Also includes: The weight determination unit is configured to: Based on the feedback behaviors of users on the recommendation data in multiple links involved in the target recommendation scenario, the users are divided into multiple categories; and according to the conversion rates corresponding to the multiple links, the weights corresponding to the users in the multiple categories are determined.

11. The device according to claim 10, wherein: The weight determination unit is further configured to: For at least one link other than the preset link among the multiple links, the weight corresponding to the category of users whose feedback behavior ends at the link is determined according to the conversion rate corresponding to the link, the conversion rate corresponding to the preset link, and the weight corresponding to the category of users whose feedback behavior ends at the preset link, wherein the weight corresponding to the category of users whose feedback behavior ends at the preset link is the preset weight.

12. The device according to any one of claims 9 to 11, wherein: Also includes: Evaluation unit, configured as: For a target link among the multiple links involved in the target recommendation scenario, the following operations are performed: for a first recommended user set obtained by the recommendation model based on the evaluation sample data set, determining evaluation index data of the first recommended user set under the target link; weighting the evaluation index data according to the weight corresponding to the category of the user whose feedback behavior ends at the target link to obtain weighted evaluation index data; and The evaluation unit is further configured to: The evaluation index data corresponding to the recommendation model is determined according to the weighted evaluation index data corresponding to each of the plurality of target links.

13. The device according to claim 12, wherein: The evaluation unit is further configured to: The users who have made feedback in the target link among the first recommended users are taken as actual positive samples, the users who have made feedback in the previous link of the target link in the recommendation results are taken as actual negative samples, the users who are ranked at the top in a preset proportion in the first recommended user set are taken as predicted positive samples, and the users other than the preset proportion in the first recommended user set are taken as predicted negative samples, so as to determine the evaluation index data of the recommendation model in the target link.

14. The device according to any one of claims 9 to 11, wherein: Also includes: The test unit is configured as follows: Input the test sample data in the test sample data set into a plurality of models to be tested including the recommendation model, determine a plurality of recommended user subsets corresponding to the plurality of models to be tested one by one; merge the plurality of recommended user subsets to obtain a second recommended user set; Combining at least one model to be tested among the multiple models to be tested to obtain multiple model groups; For each model group in the multiple model groups, the recommended users determined by all models in the model group in the second recommended user set are divided into a user group to obtain multiple user groups corresponding to the multiple model groups one by one; and the test results corresponding to each of the multiple models to be tested are determined according to the conversion rates corresponding to each of the multiple user groups.

15. The device according to claim 14, wherein: The test unit is further configured to: For each model to be tested among the multiple models to be tested, a target user group determined by the model to be tested is determined from the multiple user groups, and a test result corresponding to the model to be tested is determined according to a conversion rate corresponding to the target user group.

16. A data recommendation device, comprising: A user determination unit, configured to determine a recommended user set by a recommendation model, wherein the recommendation model is determined by any one of claims 9 to 15; The recommendation unit is configured to recommend target recommendation data to users in the recommendation user set.

17. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images