Model determination method, device, electronic device and storage medium in recommendation system

Abstract

The present application provides a model determination method, device, electronic device and storage medium in a recommendation system, which belongs to the field of sandbox technology. The method includes: obtaining a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event, and the recommendation request log carries the real-time characteristics of the recommendation request event; sending the recommendation request log into a sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model; analyzing the real-time characteristics through the target recommendation model to obtain the recommendation result output by the target recommendation model; and determining that the monitoring index of the recommendation result meets the preset conditions, determining that the target recommendation model can be applied to the recommendation system. The present application improves the consistency between the model recommendation results and the actual online application effects.

Description

Technical Field

[0001] The present application relates to the field of sandbox technology, and in particular to a model determination method, device, electronic device and storage medium in a recommendation system. Background Art

[0002] In the application process of recommendation systems in industrial scenarios, the business side often continuously iterates and updates the model in order to continuously optimize the recommendation effect. The current update mode is as follows: the technician simulates the online traffic and stores the simulated traffic. Then the technician makes a model traffic request based on the offline stored traffic data, records the model's prediction results, analyzes the prediction results and compares them with the online analysis results to determine the prediction effect of the offline model. If the iterative model verification effect is better than the online one, the online model is replaced.

[0003] During the offline model prediction process, due to differences in real-time features and samples, the offline prediction results of the model cannot be guaranteed to be completely consistent with the actual online application effects. If the online model is replaced with an offline model and an online small-flow test is started, the actual online business will fluctuate due to differences in the model. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for determining a model in a recommendation system to solve the problem that the model prediction results are inconsistent with the actual application effects. The specific technical solution is as follows:

[0005] In a first aspect, a model determination method in a recommendation system is provided, the method comprising:

[0006] Obtaining a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event and carries a real-time feature of the recommendation request event;

[0007] Sending the recommendation request log into a sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model;

[0008] Analyzing the real-time features through the target recommendation model to obtain a recommendation result output by the target recommendation model;

[0009] When it is determined that the monitoring index of the recommendation result meets the preset condition, it is determined that the target recommendation model can be applied to the recommendation system.

[0010] Optionally, analyzing the real-time features by the target recommendation model to obtain a recommendation result output by the target recommendation model includes:

[0011] Obtaining an initial recommended item list obtained by the target recommendation model based on the real-time features in the recommendation request log;

[0012] Performing information analysis on the recommended item list through an operation strategy to obtain an updated first recommended item list, wherein the information analysis includes query aggregation, re-ranking, and crowd targeting;

[0013] Processing the first recommended item list through a processing solution to obtain a second recommended item list, wherein the processing solution includes risk control, business guarantee, and financial analysis;

[0014] The second recommended item list is outputted through the target recommendation model.

[0015] Optionally, determining that the target recommendation model can be applied to the recommendation system when it is determined that the monitoring indicator of the recommendation result meets a preset condition includes:

[0016] Performing data analysis on the second recommended item list through a monitoring system to obtain multiple monitoring indicators;

[0017] If the plurality of monitoring indicators meet the preset indicator conditions within the preset time period, marking the target recommendation model;

[0018] Conduct AB testing in actual business processes through the labeled target recommendation model;

[0019] If the labeled target recommendation model passes the test, the labeled target recommendation model is applied to the recommendation system.

[0020] Optionally, sending the recommendation request log into a sandbox through the recommendation engine includes:

[0021] Sending the recommendation request log to a log aggregation system through the recommendation engine to obtain a plurality of recommendation request logs to be selected, wherein the log aggregation system is used to collect, aggregate and move the plurality of recommendation request logs;

[0022] Sending the plurality of candidate recommendation request logs into a message queue through the log aggregation system, so as to input the candidate recommendation request logs into a target recommendation model of the sandbox through the message queue;

[0023] A traffic ratio used by the sandbox is configured, wherein the traffic ratio is used to select a recommendation request event from a plurality of recommendation request events to be selected.

[0024] Optionally, before obtaining the recommendation request log through the recommendation engine, the method further includes:

[0025] In the case of monitoring a recommendation request event, performing recommendation preprocessing on the recommendation request data carried by the recommendation request event to obtain a recommendation request event after recommendation preprocessing, wherein the recommendation preprocessing includes risk decision-making, online data assembly and model feature preparation;

[0026] In the case where the traffic of the recommendation request event after the recommendation preprocessing passes through the recommendation engine, the recommendation request log is obtained by the recommendation engine in an asynchronous manner, wherein the recommendation request event after the recommendation preprocessing carries the recommendation request log.

[0027] Optionally, the real-time feature includes an offline feature, and before obtaining the recommendation request log through the recommendation engine, the method further includes:

[0028] In the case of monitoring a recommendation request event, performing data preprocessing on the recommendation request data carried by the recommendation request event, wherein the data preprocessing includes extraction, conversion and loading;

[0029] Extract offline features from the recommendation request data after the data preprocessing, wherein the offline features include consumption features, risk features, demographic attributes, and income features.

[0030] Optionally, before analyzing the real-time features by the target recommendation model, the method further includes:

[0031] Inputting the real-time features of the sample into the initial recommendation model to obtain an estimated recommended item list output by the initial recommendation model;

[0032] When the estimated recommended item list is inconsistent with the actual recommended item list, adjusting the parameters of the initial recommendation model until the estimated recommended item list is consistent with the actual recommended item list;

[0033] The initial recommendation model after adjusting the parameters is used as the target recommendation model.

[0034] In a second aspect, a model determination device in a recommendation system is provided, the device comprising:

[0035] an acquisition module, configured to acquire a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event and carries a real-time feature of the recommendation request event;

[0036] A sending module, used for sending the recommendation request log into a sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model;

[0037] An analysis module, used to analyze the real-time features through the target recommendation model to obtain a recommendation result output by the target recommendation model;

[0038] The determination module is used to determine whether the target recommendation model can be applied to the recommendation system when it is determined that the monitoring index of the recommendation result meets the preset conditions.

[0039] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0040] Memory, used to store computer programs;

[0041] The processor is used to implement the steps of the model determination method in any one of the recommendation systems when executing the program stored in the memory.

[0042] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the model determination method in any one of the recommendation systems are implemented.

[0043] Beneficial effects of the embodiments of the present application:

[0044] The present application relates to the field of recommendation system architecture in recommendation technology. The embodiment of the present application provides a model determination method in a recommendation system. The server obtains the recommendation request log through the recommendation engine, and then sends the recommendation request log to the sandbox through the recommendation engine. The real-time features are analyzed by the target recommendation model to obtain the recommendation results output by the target recommendation model. Finally, when the monitoring indicators of the recommendation results meet the preset conditions, it is determined that the target recommendation model can be applied to the recommendation system. In the present application, the server sets the business process in the sandbox to be the same as the actual business process, and uses the recommendation request log, so that all the traffic of the sandbox environment comes from the results of real-time collection of the recommendation request log in the online business, and uses the same system processing logic as the online environment in the model verification link of the sandbox, ensuring that the request data and business process of the sandbox environment are consistent with the online environment, improving the authenticity of the model experiment in the sandbox environment, and improving the consistency between the model recommendation results and the actual application effects online.

[0045] Of course, implementing any product or method of the present application does not necessarily require achieving all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A schematic diagram of the hardware environment of a model determination method in a recommendation system provided in an embodiment of the present application;

[0048] Figure 2 A flow chart of a method for determining a model in a recommendation system provided in an embodiment of the present application;

[0049] Figure 3 A flow chart of a method for obtaining a recommendation result output by a target recommendation model provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a model determination system in a recommendation system provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a model determination device in a recommendation system provided in an embodiment of the present application;

[0052] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0055] In order to solve the problem mentioned in the background technology, according to one aspect of the embodiments of the present application, an embodiment of a model determination method in a recommendation system is provided.

[0056] Optionally, in the embodiment of the present application, the model determination method in the above recommendation system can be applied to Figure 1 In the hardware environment composed of the terminal 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the terminal 101 via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the terminal 101 includes but is not limited to a PC, a mobile phone, a tablet computer, etc.

[0057] A model determination method in a recommendation system in an embodiment of the present application may be executed by the server 103 or by the terminal 101 .

[0058] An embodiment of the present application provides a method for determining a model in a recommendation system, which can be applied to a server to use a model in a sandbox as a model in the recommendation system.

[0059] The following will describe in detail a method for determining a model in a recommendation system provided by an embodiment of the present application in combination with a specific implementation manner. Figure 2 As shown, the specific steps are as follows:

[0060] Step 201: Obtain recommendation request logs through the recommendation engine.

[0061] The recommendation request log is generated based on the recommendation request event, and the recommendation request log carries the real-time characteristics of the recommendation request event.

[0062] In an embodiment of the present application, each request behavior of a user on an app or mini program is regarded as a recommendation request event. Each recommendation request event will generate a recommendation request log. The recommendation request log records the real-time characteristics of the recommendation request event, including user offline characteristics, current environment characteristics, transaction scenarios and other information.

[0063] For example, each time a user clicks on a web page, it is regarded as a recommendation request event. The recommendation request log corresponding to the recommendation request event records the user's offline characteristics, current environment characteristics and transaction scenarios. The user's offline characteristics include the user's consumption characteristics, credit card attributes, risk characteristics, demographic attributes, property characteristics and income characteristics. The current environment characteristics include the current web page name and click time, and the transaction scenario includes the transaction amount, transaction merchant information, transaction order, etc.

[0064] The recommendation engine is an information network used to proactively discover current or potential needs of users and proactively push information to users. The server obtains the recommendation request log through the recommendation engine. Since the recommendation request log records the real-time characteristics of the recommendation request event, the recommendation request log can completely replicate the scenario in which the actual recommendation request event occurs.

[0065] Step 202: Send the recommendation request log to the sandbox through the recommendation engine.

[0066] The business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model.

[0067] In the embodiment of the present application, the business process in the sandbox is the same as the actual business process, that is, the sandbox environment reuses the recommendation process of the online environment, so the same system processing logic as the online environment can be used in the model verification link of the sandbox. Since the recommendation engine sends the recommendation request log to the sandbox, all the traffic of the sandbox environment comes from the results of real-time collection of the recommendation request log in the online business, ensuring that the request data of the sandbox environment is consistent with the request data of the online business, so the recommendation request log can completely replicate the scene where the recommendation request event occurs. The authenticity of the model experiment in the sandbox environment is maximized.

[0068] The actual business process includes the recommendation model, so the business process in the sandbox also includes the target recommendation model.

[0069] Step 203: Analyze the real-time features through the target recommendation model to obtain the recommendation results output by the target recommendation model.

[0070] In an embodiment of the present application, in an actual business scenario, a user's clicking behavior on a web page will result in a list of recommended items fed back by the website. The business process of the sandbox environment is exactly the same as the actual business process, so the sandbox environment will also output a list of recommended items.

[0071] After the server sends the recommendation request log to the sandbox, since the recommendation request log carries the real-time characteristics of the recommendation request event, and the business process in the sandbox includes a target recommendation model, the server can analyze the real-time characteristics through the target recommendation model to obtain the recommendation results output by the target recommendation model, where the recommendation results are a list of item recommendations.

[0072] Step 204: When it is determined that the monitoring index of the recommendation result meets the preset conditions, it is determined that the target recommendation model can be applied to the recommendation system.

[0073] In an embodiment of the present application, the server analyzes the recommendation results through the monitoring system to obtain multiple monitoring indicators, and then the server determines whether the monitoring indicators meet the preset conditions. If the server determines that the monitoring indicators do not meet the preset conditions, it means that the recommendation model in the sandbox environment cannot be applied to the recommendation system and needs further optimization.

[0074] If the server determines that the monitoring indicators meet the preset conditions, it means that the recommendation model in the sandbox environment can be applied to the recommendation system. The server can use the recommendation model to conduct online small-traffic model experiments to reduce the risk of model production and improve the stability of the recommendation system.

[0075] In this application, the server sets the business process in the sandbox to be the same as the actual business process and adopts the recommendation request log, so that all the traffic of the sandbox environment comes from the results of real-time collection of the recommendation request log in the online business, and uses the same system processing logic as the online environment in the model verification phase of the sandbox to ensure that the request data and business processes in the sandbox environment are consistent with the online ones, thereby improving the authenticity of the model experiment in the sandbox environment and improving the consistency between the model recommendation results and the actual online application effects.

[0076] As an optional implementation, Figure 3 As shown, the target recommendation model is used to analyze the real-time features, and the recommendation results output by the target recommendation model include:

[0077] Step 301: Obtain an initial recommended item list obtained by a target recommendation model based on real-time features in a recommendation request log.

[0078] In the embodiment of the present application, the server inputs the real-time features into the target recommendation model to obtain the initial recommended item list output by the target recommendation model, wherein the target recommendation model may include multiple sub-models, illustratively including business model 1, business model 2, and business model 3. The target recommendation model needs to be trained in advance, and the training method can adopt deep learning, graph algorithm, NLP algorithm, model evaluation, machine learning, hyperparameter search, CV algorithm, and test management.

[0079] Step 302: Analyze the recommended item list based on the operation strategy to obtain an updated first recommended item list.

[0080] Among them, information analysis includes query aggregation, re-ranking, and crowd targeting.

[0081] In an embodiment of the present application, after the server obtains the initial recommended item list, it can use the operation strategy to perform information analysis on the recommended item list, and the information analysis includes query aggregation, reordering, crowd targeting, specifying policy rules, etc., to obtain an updated first recommended item list. Reordering is to reorder the recommended items in the initial recommended item list so that the items of greater user interest are arranged at the front. Crowd targeting is to divide items with similar or identical tags into the same group and configure corresponding recommended items for each group. If the server determines that the user who initiated the recommendation request event also belongs to the group, the corresponding recommended item will be included in the recommended item list.

[0082] Step 303: Process the first recommended item list using the processing solution to obtain a second recommended item list.

[0083] Among them, the treatment solutions include risk control, business guarantee and financial analysis.

[0084] In an embodiment of the present application, after the server obtains the first recommended item list, it processes the first recommended item list through a processing scheme, which includes risk control, business guarantee and financial analysis, to obtain a second recommended item list. For financial products, risk control should be carried out to improve the property safety of users. Use FICO (Finance, Controlling, the financial module in SAP) credit score to evaluate the credit value of users and improve financial security. Use business guarantee to provide users with reliable recommended items.

[0085] Step 304: Output a second recommended item list through the target recommendation model.

[0086] In the embodiment of the present application, after obtaining the second recommended item list, the server outputs the second recommended item list through the target recommendation model.

[0087] In this application, after the server obtains the initial list of recommended items, it further optimizes the recommended items through operational strategies and processing solutions, so that the recommended items can meet user needs and interests while also improving risk safety.

[0088] As an optional implementation, determining that the target recommendation model can be applied to the recommendation system when it is determined that the monitoring indicators of the recommendation results meet preset conditions includes: performing data analysis on the second recommended item list through the monitoring system to obtain multiple monitoring indicators; labeling the target recommendation model when multiple monitoring indicators meet preset indicator conditions within a preset time period; performing AB testing in actual business processes on the labeled target recommendation model; and applying the labeled target recommendation model to the recommendation system when the labeled target recommendation model passes the test.

[0089] In an embodiment of the present application, after the server obtains the second recommended item list, the second recommended item list is analyzed by the monitoring system. The analysis can be performed by drawing a monitoring graph or the like to obtain multiple monitoring indicators, including but not limited to the improvement rate of the model effect indicators, the stability of the recommendation effect, the distribution of model scores, the model AUC, and the model response time. The server then sends the monitoring indicators to Grafana for real-time model effect monitoring. The server can compare the monitoring indicators with the online model indicators horizontally, so that the technical staff can intuitively compare the improvement effect of the recommendation model. Among them, the improvement rate of the model effect indicators includes but is not limited to click-through rate, conversion rate, purchase rate, repurchase rate, and length of stay, and the stability of the recommendation effect includes but is not limited to the standard deviation and mean of the model prediction score.

[0090] Among them, Grafana is an open source data visualization tool developed in Go language, which can be used for data monitoring and data statistics. It has the characteristics of visualization, alarm, notification, dynamic dashboard, hybrid data source, annotation and filter.

[0091] The server can also set a preset time and observe whether multiple monitoring indicators meet the preset indicator conditions within the preset time. If the server determines that the monitoring indicators do not meet the preset indicator conditions, it means that the target recommendation model cannot be used for the recommendation system. If the server determines that the monitoring indicators meet the preset indicator conditions, the target recommendation model is labeled, and then according to the actual business, the labeled target recommendation model is used to perform AB testing in the actual business process. If the AB test passes, it is determined that the target recommendation model is stable, and then the target recommendation model is used to conduct online small-traffic model experiments to reduce the risk of model commissioning and improve the stability of the recommendation system.

[0092] As an optional implementation, sending the recommendation request log into the sandbox through the recommendation engine includes: sending the recommendation request log into the log aggregation system through the recommendation engine to obtain multiple recommendation request logs to be selected, wherein the log aggregation system is used to collect, aggregate and move the multiple recommendation request logs; sending the multiple recommendation request logs to be selected into the message queue through the log aggregation system, so as to input the selected recommendation request logs into the target recommendation model of the sandbox through the message queue; configuring the traffic ratio used by the sandbox, wherein the traffic ratio is used to select the recommendation request event from the multiple recommendation request events to be selected.

[0093] In the embodiment of the present application, after the recommendation engine obtains the recommendation request log, the recommendation request log is sent to the log aggregation system to obtain multiple recommendation request logs to be selected. The log aggregation system can be Flume, which is a distributed, reliable and available service for effectively collecting, aggregating and moving a large number of recommendation request logs. It has a simple and flexible architecture based on streaming data flow, has a reliable reliability mechanism and many failover and recovery mechanisms, has strong fault tolerance, uses a simple scalable data model, and allows online analysis of applications.

[0094] The server sends multiple candidate recommendation request logs to the message queue through the log aggregation system, so as to input the candidate recommendation request logs into the target recommendation model of the sandbox through the message queue. The message queue can be Kafka, which is a distributed, divisible, redundant and persistent log service. Kafka can reduce the complexity of system networking and programming, and play the role of a high-speed data bus; it also provides high throughput for publishing and subscription; it can perform persistent operations; it is a distributed system and easy to expand outward; the status of the message being processed is maintained on the consumer side, not on the server side, and it can automatically balance when it fails; it supports online and offline scenarios.

[0095] The server also configures the traffic ratio used by the sandbox, which is used to select a recommendation request event from multiple recommendation request events to be selected. The selection method can be random selection or fixed-time selection. This application does not limit the specific selection method. Since a powerful server is required to predict the results of all recommendation request events, and the sandbox environment cannot support such a large server, some recommendation request events are selected to reduce the server's computing workload.

[0096] As an optional implementation, before obtaining the recommendation request log through the recommendation engine, the method also includes: when a recommendation request event is monitored, performing recommendation preprocessing on the recommendation request data carried by the recommendation request event to obtain a recommendation request event after recommendation preprocessing, wherein the recommendation preprocessing includes risk decision-making, online data assembly and model feature preparation; when the traffic of the recommendation request event after recommendation preprocessing passes through the recommendation engine, obtaining the recommendation request log through the recommendation engine in an asynchronous manner, wherein the recommendation request event after recommendation preprocessing carries the recommendation request log.

[0097] In an embodiment of the present application, after monitoring a recommendation request event, the server first performs recommendation preprocessing on the recommendation request data carried by the recommendation request event, including risk decision-making, online data assembly, and model feature preparation, to obtain a recommendation request event after recommendation preprocessing. Risk decision-making can reduce the risk of recommendation request events and improve model security. Since the model needs to be trained with sample features in advance, the server obtains sample features based on the recommendation request data, and then uses the sample features to train the model. Then, when the traffic of the recommendation request event passes through the recommendation engine, the recommendation engine obtains the recommendation request log in an asynchronous manner, wherein the recommendation request event after recommendation preprocessing carries the recommendation request log.

[0098] As an optional implementation, the real-time features include offline features. Before obtaining the recommendation request log through the recommendation engine, the method also includes: when a recommendation request event is monitored, performing data preprocessing on the recommendation request data carried by the recommendation request event, wherein the data preprocessing includes extraction, conversion and loading; extracting offline features from the recommendation request data after data preprocessing, wherein the offline features include consumption characteristics, risk characteristics, demographic attributes and income characteristics.

[0099] The real-time features used by the target recommendation model include offline features. Offline features are acquired by the server in advance. The specific acquisition method is: after the server detects the recommendation request event, the recommendation request data carried by the recommendation request event is landed, and the server performs data preprocessing ETL on the recommendation request data. Data preprocessing includes extraction, conversion and loading, and then the recommendation request data after data preprocessing is used to construct offline features, and the offline features are used for subsequent offline model training. Offline features include but are not limited to consumption characteristics, risk characteristics, property characteristics, demographic attributes, credit card attributes and income characteristics.

[0100] As an optional implementation, before analyzing the real-time features through the target recommendation model, the method also includes: inputting the real-time features of the sample into the initial recommendation model to obtain an estimated recommended item list output by the initial recommendation model; when the estimated recommended item list is inconsistent with the actual item recommendation list, adjusting the parameters of the initial recommendation model until the obtained estimated recommended item list is consistent with the actual item recommendation list; and using the initial recommendation model with adjusted parameters as the target recommendation model.

[0101] In an embodiment of the present application, the server inputs the real-time features of the sample into the initial recommendation model to obtain an estimated recommended item list output by the initial recommendation model. If the estimated recommended item list is inconsistent with the actual item recommendation list, the server adjusts the parameters of the initial recommendation model until the estimated recommended item list is consistent with the actual item recommendation list. The server uses the initial recommendation model with adjusted parameters as the target recommendation model.

[0102] Optionally, the embodiment of the present application also provides a schematic diagram of determining a model in a recommendation system, such as Figure 4 As shown, the specific steps are as follows.

[0103] Step 1: The user clicks on the app, web or mini program, and the server detects a recommendation request event, which carries the recommendation request data.

[0104] Step 2: The server performs recommendation preprocessing on the recommendation request data, including risk decision-making, online data assembly, and model feature preparation.

[0105] Step 3: When online traffic passes through the recommendation engine, the recommendation engine obtains the recommendation request log.

[0106] Step 4: The recommendation engine sends the recommendation request log to the sandbox through Flume and Kafka, and performs exception filtering and traffic distribution on the recommendation request log in the sandbox environment.

[0107] Step 5: In the sandbox environment, the server obtains real-time features and inputs the real-time features into the target recommendation model, and then sequentially applies the operation strategy and processing solution to the initial recommended item list output by the target recommendation model to obtain a second recommended item list.

[0108] Step 6: The server uses the monitoring system to perform data analysis on the second recommended item list to obtain monitoring indicators.

[0109] Step 7: If the server determines that the monitoring indicators meet the preset indicator conditions, it will conduct AB testing in the actual business process through the target recommendation model, and then use the target recommendation model to conduct an online small-traffic model experiment.

[0110] The offline evaluation process of existing technical models is relatively complicated, the model verification cycle is long, and it is easy to make mistakes, resulting in inaccurate model verification data and difficulty in troubleshooting. Offline models cannot simulate online application scenarios well, resulting in inaccurate estimates.

[0111] The sandbox environment of this application can simulate the online request environment to the maximum extent without affecting the online recommendation results, and can achieve a comprehensive and realistic offline model effect evaluation, minimize the business losses during online experiments caused by the distortion of the offline model evaluation, and improve the stability of the recommendation system model service. At the same time, it improves the efficiency of model verification, reduces the workload of offline model verification, and improves accuracy.

[0112] Based on the same technical concept, the embodiment of the present application also provides a model determination device in a recommendation system, such as Figure 5 As shown, the device comprises:

[0113] A first acquisition module 501 is used to acquire a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event and carries a real-time feature of the recommendation request event;

[0114] The sending module 502 is used to send the recommendation request log to the sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model;

[0115] The analysis module 503 is used to analyze the real-time features through the target recommendation model to obtain the recommendation results output by the target recommendation model;

[0116] The determination module 504 is used to determine whether the target recommendation model can be applied to the recommendation system when it is determined that the monitoring index of the recommendation result meets the preset conditions.

[0117] Optionally, the analysis module 503 includes:

[0118] An acquisition unit, used for acquiring an initial recommended item list obtained by a target recommendation model based on real-time features in a recommendation request log;

[0119] A first analysis unit is used to perform information analysis on the recommended item list through an operation strategy to obtain an updated first recommended item list, wherein the information analysis includes query aggregation, re-ranking, and crowd targeting;

[0120] A processing unit, configured to process the first recommended item list through a processing solution to obtain a second recommended item list, wherein the processing solution includes risk control, business guarantee and financial analysis;

[0121] The output unit is used to output a second recommended item list through the target recommendation model.

[0122] Optionally, the determining module 504 includes:

[0123] A second analysis unit, configured to perform data analysis on the second recommended item list through a monitoring system to obtain a plurality of monitoring indicators;

[0124] A labeling unit, used to label the target recommendation model when multiple monitoring indicators meet preset indicator conditions within a preset time period;

[0125] The testing unit is used to conduct AB testing in actual business processes through the annotated target recommendation model;

[0126] The application unit is used to apply the labeled target recommendation model to the recommendation system if the labeled target recommendation model passes the test.

[0127] Optionally, the feeding module 502 includes:

[0128] A first input unit is used to input the recommendation request log into the log aggregation system through the recommendation engine to obtain a plurality of recommendation request logs to be selected, wherein the log aggregation system is used to collect, aggregate and move the plurality of recommendation request logs;

[0129] A second input unit is used to input the multiple candidate recommendation request logs into the message queue through the log aggregation system, so as to input the candidate recommendation request logs into the target recommendation model of the sandbox through the message queue;

[0130] The configuration unit is used to configure the traffic ratio used by the sandbox, wherein the traffic ratio is used to select a recommended request event from a plurality of recommended request events to be selected.

[0131] Optionally, the device comprises:

[0132] A first processing module is used to perform recommendation preprocessing on the recommendation request data carried by the recommendation request event when a recommendation request event is detected, so as to obtain a recommendation request event after recommendation preprocessing, wherein the recommendation preprocessing includes risk decision making, online data assembly and model feature preparation;

[0133] The second acquisition module is used to obtain the recommendation request log in an asynchronous manner through the recommendation engine when the traffic of the recommendation request event after recommendation preprocessing passes through the recommendation engine, wherein the recommendation request event after recommendation preprocessing carries the recommendation request log.

[0134] Optionally, the real-time feature includes an offline feature, and the apparatus includes:

[0135] A second processing module is used to perform data preprocessing on the recommendation request data carried by the recommendation request event when a recommendation request event is detected, wherein the data preprocessing includes extraction, conversion and loading;

[0136] The extraction module is used to extract offline features from the recommendation request data after data preprocessing, wherein the offline features include consumption features, risk features, demographic attributes and income features.

[0137] Optionally, the device comprises:

[0138] An input module is used to input the real-time features of the samples into the initial recommendation model to obtain an estimated recommended item list output by the initial recommendation model;

[0139] An adjustment module, used for adjusting the parameters of the initial recommendation model when the estimated recommended item list is inconsistent with the actual recommended item list, until the estimated recommended item list is consistent with the actual recommended item list;

[0140] As a module, it is used to use the initial recommendation model after adjusting the parameters as the target recommendation model.

[0141] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 6 As shown, it includes a memory 603, a processor 601, a communication interface 602 and a communication bus 604. The memory 603 stores a computer program that can be run on the processor 601. The memory 603 and the processor 601 communicate through the communication interface 602 and the communication bus 604. When the processor 601 executes the computer program, the steps of the above method are implemented.

[0142] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0143] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0144] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0145] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0146] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store a program code for the processor to execute the above method:

[0147] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0148] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0149] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0150] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0151] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0157] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A model determination method in a recommendation system, characterized in that: The method comprises: Obtaining a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event and carries a real-time feature of the recommendation request event; Sending the recommendation request log into a sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model; Analyzing the real-time features through the target recommendation model to obtain a recommendation result output by the target recommendation model; In the case where it is determined that the monitoring index of the recommendation result meets the preset conditions, determining that the target recommendation model can be applied to the recommendation system; The analyzing the real-time features by the target recommendation model to obtain the recommendation results output by the target recommendation model includes: Obtaining an initial recommended item list obtained by the target recommendation model based on the real-time features in the recommendation request log; Performing information analysis on the recommended item list through an operation strategy to obtain an updated first recommended item list, wherein the information analysis includes query aggregation, re-ranking, and crowd targeting; Processing the first recommended item list through a processing solution to obtain a second recommended item list, wherein the processing solution includes risk control, business guarantee, and financial analysis; The second recommended item list is outputted through the target recommendation model.

2. The method according to claim 1, characterized in that Determining that the target recommendation model can be applied to the recommendation system when the monitoring index of the recommendation result meets the preset conditions includes: Performing data analysis on the second recommended item list through a monitoring system to obtain a plurality of monitoring indicators; If the plurality of monitoring indicators meet the preset indicator conditions within the preset time period, marking the target recommendation model; Conduct AB testing in actual business processes through the labeled target recommendation model; When the labeled target recommendation model passes the test, the labeled target recommendation model is applied to the recommendation system.

3. The method according to claim 1, characterized in that Sending the recommendation request log into the sandbox through the recommendation engine includes: Sending the recommendation request log to a log aggregation system through the recommendation engine to obtain a plurality of recommendation request logs to be selected, wherein the log aggregation system is used to collect, aggregate and move the plurality of recommendation request logs; Sending the plurality of candidate recommendation request logs into a message queue through the log aggregation system, so as to input the candidate recommendation request logs into a target recommendation model of the sandbox through the message queue; A traffic ratio used by the sandbox is configured, wherein the traffic ratio is used to select a recommendation request event from a plurality of recommendation request events to be selected.

4. The method according to claim 1, characterized in that: Before obtaining the recommendation request log through the recommendation engine, the method further includes: In the case of monitoring a recommendation request event, performing recommendation preprocessing on the recommendation request data carried by the recommendation request event to obtain a recommendation request event after recommendation preprocessing, wherein the recommendation preprocessing includes risk decision-making, online data assembly and model feature preparation; In the case where the traffic of the recommendation request event after the recommendation preprocessing passes through the recommendation engine, the recommendation request log is obtained by the recommendation engine in an asynchronous manner, wherein the recommendation request event after the recommendation preprocessing carries the recommendation request log.

5. The method according to claim 1, characterized in that: The real-time feature includes an offline feature. Before obtaining the recommendation request log through the recommendation engine, the method further includes: In the case of monitoring a recommendation request event, performing data preprocessing on the recommendation request data carried by the recommendation request event, wherein the data preprocessing includes extraction, conversion and loading; Extract offline features from the recommendation request data after the data preprocessing, wherein the offline features include consumption features, risk features, demographic attributes, and income features.

6. The method according to claim 1, characterized in that Before analyzing the real-time features by the target recommendation model, the method further includes: Inputting the real-time features of the sample into the initial recommendation model to obtain an estimated recommended item list output by the initial recommendation model; When the estimated recommended item list is inconsistent with the actual recommended item list, adjusting the parameters of the initial recommendation model until the estimated recommended item list is consistent with the actual recommended item list; The initial recommendation model after adjusting the parameters is used as the target recommendation model.

7. A model determination device in a recommendation system, characterized in that: The device comprises: an acquisition module, configured to acquire a recommendation request log through a recommendation engine, wherein the recommendation request log is generated based on a recommendation request event and carries a real-time feature of the recommendation request event; A sending module, used for sending the recommendation request log into a sandbox through the recommendation engine, wherein the business process in the sandbox is the same as the actual business process, and the business process in the sandbox includes a target recommendation model; An analysis module, used to analyze the real-time features through the target recommendation model to obtain a recommendation result output by the target recommendation model; A determination module, configured to determine whether the target recommendation model can be applied to the recommendation system when it is determined that the monitoring index of the recommendation result meets a preset condition; Wherein, the analysis module is used for: Obtaining an initial recommended item list obtained by the target recommendation model based on the real-time features in the recommendation request log; Performing information analysis on the recommended item list through an operation strategy to obtain an updated first recommended item list, wherein the information analysis includes query aggregation, re-ranking, and crowd targeting; Processing the first recommended item list through a processing solution to obtain a second recommended item list, wherein the processing solution includes risk control, business guarantee, and financial analysis; The second recommended item list is outputted through the target recommendation model.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images