Recommendation model selection method and device, electronic equipment and program product

Abstract

The invention provides a recommendation model selection method and device, electronic equipment and a program product, and relates to the field of recommendation models. The method comprises the steps of determining a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model based on recommendation effect evaluation parameters of each recommendation model in a plurality of recommendation models in a historical unit duration; and based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model, determining a comprehensive recommendation probability set corresponding to each recommendation model. And for a target customer group in the target flow channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determining a target recommendation model corresponding to the target customer group from the plurality of recommendation models. By calculating the model traffic channel recommendation probability and the customer group recommendation probability, the selection range of the recommendation model can be refined, and the recommendation model traffic distribution is performed for different customer groups, so that the accuracy of model selection can be effectively improved.

Description

Technical Field

[0001] The present application relates to the field of recommendation models, and in particular to a recommendation model selection method, device, electronic device and program product. Background Art

[0002] With the development of the Internet and communication technology, recommendation models have occupied a key position in various online platforms such as e-commerce and content information. They conduct in-depth analysis based on user behavior habits and preference characteristics, and then push content that meets user needs. This not only improves the user experience, but also helps the platform improve user retention and business conversion rates.

[0003] In addition, in order to achieve better recommendation results, the same online platform may deploy multiple recommendation models. These models can output different types of recommendation content due to different algorithm logic and business focus, thereby adapting to the needs of diverse user groups.

[0004] However, while the coexistence of multiple recommendation models enriches the recommendation content, it also brings up the problem of selecting recommendation models. Summary of the invention

[0005] The present application provides a recommendation model selection method, device, electronic device and program product. By calculating the model traffic channel recommendation probability and customer group recommendation probability, the selection range of the recommendation model can be refined, and the recommendation model traffic can be allocated for different customer groups, thereby improving the accuracy of model selection.

[0006] In a first aspect, the present application provides a recommendation model selection method, the method comprising: obtaining recommendation effect evaluation parameters of each recommendation model in a historical unit time length among multiple recommendation models, and determining a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model. Based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model, determine a comprehensive recommendation probability set corresponding to each recommendation model. For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determine the target recommendation model corresponding to the target customer group from multiple recommendation models.

[0007] The recommendation model selection method provided in the present application divides the traffic into different traffic channels, and each traffic channel is divided into multiple customer groups according to user characteristics. By obtaining the traffic channel recommendation probability set of each model in different traffic channels, the horse racing mechanism in the segmented traffic channels is realized, and the performance effect of each model in different traffic channels is clarified according to the traffic channel recommendation probability. Furthermore, the customer group recommendation probability of the model in each customer group is calculated, and the adaptability of the model in different customer groups can also be determined. The comprehensive recommendation probability set combines the traffic channel probability and customer group recommendation probability of the recommendation model to measure the competitiveness of the recommendation model from multiple aspects. Compared with the relatively single decision-making basis and decision-making process of the prior art, the present application refines the traffic into different traffic channels and refines users into different customer groups, breaking the previous general allocation model, evaluating the adaptability of the recommendation model from different aspects, and can improve the accuracy of the recommendation model selection.

[0008] A possible implementation method is to determine the channel recommendation probability set corresponding to each recommendation model for any first recommendation model among multiple recommendation models based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length, including: for each traffic channel, determine the future income of the first recommendation model in the traffic channel and the historical reference income of the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time length. The future income is determined based on the income of the first recommendation model in the historical unit time length, and the historical reference income is the sum of the income of multiple models in the historical unit time length, and the income is determined based on the recommendation effect evaluation parameters of the recommendation model. For each traffic channel, normalize the future income of the first recommendation model in the traffic channel and the historical reference income of the traffic channel to obtain the channel recommendation probability corresponding to the traffic channel. Based on the multiple channel recommendation probabilities corresponding to the multiple traffic channels, determine the channel recommendation probability set corresponding to the first recommendation model.

[0009] Another possible implementation method is to determine the future revenue of the first recommendation model in each traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, including: for each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, determine the basic revenue of the first recommendation model in the historical unit time of the traffic channel. Determine the future revenue of the first recommendation model in the traffic channel according to the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

[0010] Another possible implementation method is to determine the revenue adjustment amount, including: determining a proportional coefficient based on the ratio of the difference between the estimated stimulated traffic value of the first recommendation model in the future unit time and the actual stimulated traffic value in the historical unit time. Determine a reference adjustment amount based on the product of the basic revenue of the first recommendation model in the historical unit time and the proportional coefficient. Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel based on the sum of the basic revenue adjustment amount of the first recommendation model in the traffic channel and the reference adjustment amount, the basic revenue adjustment amount being the revenue adjustment amount of the first recommendation model in the historical unit time.

[0011] Another possible implementation method is to determine the estimated stimulated traffic value of the first recommendation model for a future unit duration, including: obtaining historical traffic sequences of multiple recommendation models, the historical traffic sequences include historical stimulated traffic values ​​corresponding to multiple historical unit durations, and the historical stimulated traffic values ​​are the total traffic values ​​stimulated by recommendations by multiple recommendation models in the corresponding historical unit durations. Based on the historical traffic sequence and the traffic prediction model, the total estimated stimulated traffic value stimulated by recommendations by multiple recommendation models in the future unit duration is predicted. The estimated stimulated traffic value of the recommendation model for the future unit duration is determined based on the product of the total estimated stimulated traffic value and the historical revenue ratio of the first recommendation model, and the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit duration of the traffic channel to the total basic revenue of multiple recommendation models in the historical unit duration of the traffic channel.

[0012] Another possible implementation method is to determine the customer group recommendation probability set corresponding to each recommendation model for any first recommendation model among multiple recommendation models based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length, including: for each customer group, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time length, determine the inference sample of the first recommendation model, the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model. For each customer group, based on the inference sample of the first recommendation model and the recommendation effect evaluation model, predict the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the recommendation effect evaluation model is used to predict the future recommendation effect evaluation parameters based on the historical recommendation effect evaluation parameters and the model hyperparameters. For each customer group, determine the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameters of the first recommendation model in the sum of the future recommendation effect evaluation parameters of all recommendation models. Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, obtain the customer group recommendation probability set corresponding to the first recommendation model.

[0013] In another possible implementation, a training sample set is obtained, the training sample set includes multiple training samples, each training sample includes a recommendation effect evaluation parameter when the recommendation model makes recommendations for a customer group in a first period, a model hyperparameter of the recommendation model, and a recommendation effect evaluation parameter label when the recommendation model makes recommendations for the customer group in a second period after the first period. A preset initial model is trained based on the training sample set to obtain a recommendation effect evaluation model.

[0014] In a second aspect, the present application provides a recommendation model selection device, which includes various functional modules used in the method described in the first aspect above.

[0015] In a third aspect, the present application provides a computer program product, including: computer instructions; when the computer instructions are executed on an electronic device, the electronic device implements the method described in the first aspect above.

[0016] In a fourth aspect, the present application provides an electronic device, comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0017] In a fifth aspect, the present application provides a readable storage medium, which includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0018] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect and will not be elaborated on again. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a recommendation model selection method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a method for obtaining evaluation parameters of recommendation effect of a recommendation model provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a method for determining a channel recommendation probability set of a recommendation model provided in an embodiment of the present application;

[0023] Figure 4A flowchart of a method for determining future benefits of a recommendation model provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a method for determining future income adjustment amount of a recommendation model provided in an embodiment of the present application;

[0025] Figure 6 A flowchart of a method for determining an estimated excitation flow value of a recommended model provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of a method for determining a total estimated excitation flow value provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of a method for determining a set of recommendation probabilities for customer groups of a recommendation model provided in an embodiment of the present application;

[0028] Fig. 9 A flowchart of a recommendation effect evaluation model training method provided in an embodiment of the present application;

[0029] Fig.10 An overall architecture diagram of a recommendation model selection method provided in an embodiment of the present application;

[0030] Fig.11 A schematic diagram of the composition of a recommended model selection device provided in an embodiment of the present application;

[0031] Fig.12 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] It should be noted that, in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0034] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. are not limiting the quantity and execution order.

[0035] In the field of recommendation, in pursuit of more efficient recommendation results, the same online platform may deploy multiple recommendation models. These recommendation models may have different business focuses and algorithm logics, and thus can output diverse recommendation content to meet the needs of diverse user groups, thereby improving the user experience and the platform's commercial conversion rate.

[0036] However, while the coexistence of multiple models enriches the recommended content, it also brings up the issue of model selection. Existing technologies use online data as the basis for model selection, but due to real-time requirements, the decision-making process of this type of solution is relatively simple. For example, the scale of traffic stimulated by the recommendation model and the feedback data of the recommended content of the recommendation model are used as the basis for model allocation. This may result in the selected model being unable to push content that fits the user, resulting in a waste of traffic. In view of this, how to improve the accuracy of recommendation model selection has become an urgent problem to be solved in the current recommendation field.

[0037] Based on this, this application provides a method for selecting a recommendation model, which can segment traffic into different traffic channels and divide users into different customer groups based on user characteristics. It can refine the model's adaptation scenarios, measure the model's adaptation degree from different dimensions, and improve the accuracy of recommendation model selection.

[0038] The recommendation model selection method provided in this application can be applied to a computing device on a network platform. The computing device can be a server cluster composed of multiple servers, or a single server, or a computer, or a processor or processing chip in a server or computer. The embodiments of this application do not limit the specific device form of the computing device.

[0039] In order to better understand the technical solution provided by this application, the following professional terms are explained.

[0040] Horse racing mechanism: refers to a competitive selection model. It puts multiple similar entities on the same competition track, sets unified phased goals, assessment standards and time limits, and each entity competes in a relatively fair environment. During this period, resource allocation, development trends, etc. are mostly controlled by each entity independently. Ultimately, based on the results of the competition, those with outstanding performance will be rewarded, supported or further promoted, while those with poor performance may face adjustments or even elimination. In this application, the horse racing mechanism is used to apply multiple recommendation models to the same traffic channel, which is a smaller traffic division range. The traffic allocation of multiple recommendation models is determined by measuring the value indicators of multiple recommendation models in the traffic channel per unit time.

[0041] Click-through rate (CTR): refers to the proportion of users who click on the recommended content after the recommended content is displayed to the user. The formula is CTR = number of times the recommended content is clicked / number of times the recommended content is displayed. The click-through rate directly reflects the attractiveness of the recommended content.

[0042] Conversion rate (CVR): refers to the percentage of users who complete the platform's expected conversion behavior among the users who click on the recommended content. The formula is CVR = number of users who complete the conversion / number of users who click on the recommended content. The conversion rate measures the effectiveness of the second half of the recommendation link and tests the conversion ability of the recommended content from interest stimulation to action promotion.

[0043] Average order value (AOV): refers to the average amount of consumption per order by users who complete the desired conversion behavior. The calculation formula is AOV = total conversion sales / number of conversion orders. The average order value is related to the revenue scale and profit margin of the recommendation business.

[0044] Normalization: Data preprocessing method, which scales the data to a fixed interval according to specific rules, usually [0,1] or [-1,1]. The purpose is to eliminate the problem of large differences in the dimensions and value ranges of different features, so that each feature has an equal status in subsequent analysis and modeling, and avoid model training deviations caused by different data scales.

[0045] Go receiver: Go language is known for its excellent concurrent performance. In the recommendation system, the Go receiver can be used as a lightweight and efficient data receiving component. When a large amount of recommendation interaction data (such as clicks and conversion information) is transmitted back in real time, the Go receiver can quickly receive the data, avoid data accumulation and loss, and maintain the smoothness of the recycling link.

[0046] Netty receiver: Netty is a high-performance network programming framework. In the scenario of recommendation effect recovery, especially when transmitting data across networks and processes, the Netty receiver can efficiently process data inflows under protocols such as TCP and UDP. In the face of high-concurrency and large-volume recommendation feedback data, it can reduce thread blocking and improve overall receiving efficiency based on the asynchronous non-blocking I / O model, ensuring that data can quickly flow from the network into subsequent processing links.

[0047] Kafka: An open source distributed stream processing platform. It has the characteristics of high throughput, persistent storage, partition fault tolerance, etc. It is often used in decoupled systems and real-time data stream processing scenarios. Different applications can communicate asynchronously through Kafka, send and receive messages, just like a data "transfer station".

[0048] Clickhouse: A column-based database management system for online analytical processing. It has extremely fast read and write speeds when processing massive amounts of data, has a high data compression ratio, can efficiently support complex queries, and is suitable for data analysis, aggregation, and other operational scenarios under large amounts of data.

[0049] xxl-job: A distributed task scheduling platform. It can schedule various tasks across platforms and languages, such as scheduled tasks and data synchronization tasks, and provides a visual task management interface, allowing operation and maintenance personnel and developers to easily control the execution and scheduling rules of tasks, ensuring that tasks in business processes are executed on time and as needed.

[0050] Software development kit (SDK): SDK is a collection of software tools, documents, sample codes and related resources, which is designed to help developers create applications for specific platforms, operating systems, frameworks or hardware more efficiently. For example, JavaScript SDK, Vue SDK and React SDK mentioned below are SDKs for specific front-end development technology frameworks, and iOSSDK and Android SDK are SDKs for mobile operating system platforms. It is like a "treasure box" that includes commonly used code snippets, function libraries, debugging tools, etc. in the development process.

[0051] Autoregressive moving average (ARMA): ARMA is a statistical model used for time series analysis and forecasting, which combines two key parts: autoregressive (AR) and moving average (MA). The model assumes that the time series data is stationary, that is, the statistical characteristics of the data (such as mean and variance) do not change over time.

[0052] Autoregressive integrated moving average model (ARIMA): ARIMA is an extended version of the ARMA model, specially designed to handle non-stationary time series data. In actual scenarios, many time series have non-stationary characteristics such as trends and seasonality. ARIMA solves this problem by introducing an integrated link.

[0053] eXtreme gradient boosting (XGBoost) algorithm: Based on the gradient boosting framework, it builds strong learners by iteratively adding weak learners (usually decision trees). Each iteration focuses on fitting the residuals of the previous round of models, so that the overall prediction error is gradually reduced. It uses second-order derivative information to optimize the objective function and find the best split point. Compared with the traditional gradient boosting algorithm that only relies on the first-order derivative, it converges faster. For example, when predicting house prices, the first round of models obtains a preliminary estimate, and subsequent models correct the previous errors in turn.

[0054] Logistic regression (LR) algorithm: LR is a generalized linear regression model used to deal with classification problems. It introduces a logistic function to map the continuous values ​​obtained by linear regression to [0,1] to represent the probability of an event.

[0055] Light gradient boosting machine (LightGBM) algorithm: LightGBM is a machine learning algorithm based on the gradient boosting framework, known for its high efficiency, speed and accuracy. It continuously iterates to build multiple decision tree weak learners based on leaf node growth, and combines them to form a powerful model for processing tasks such as classification, regression and sorting. It is widely used in e-commerce recommendations, advertising click-through rate prediction, financial risk control and medical image analysis.

[0056] Figure 1 A flow chart of a method for selecting a recommendation model provided in an embodiment of the present application. The method for selecting a recommendation model provided in the present application can be applied to the above-mentioned computing device, such as Figure 1 As shown, the specific steps include:

[0057] S101. Based on the recommendation effect evaluation parameters of each recommendation model in a historical unit time, determine a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model.

[0058] Among them, the recommendation effect evaluation parameters can be obtained through the user's interactive behavior data on the recommended content, and the effect evaluation parameters include at least: click-through rate, conversion rate and average order value. The channel recommendation probability set includes multiple channel recommendation probabilities that correspond one-to-one to the recommendation model and multiple traffic channels. The channel recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel. Each of the multiple traffic channels is divided into multiple customer groups. The customer group recommendation probability set includes multiple customer group recommendation probabilities that correspond one-to-one to the multiple customer groups. The customer group recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group.

[0059] For example, taking the first recommendation model as an example, the channel recommendation probability set corresponding to the first recommendation model can be expressed as x 1,i =[x 1,1 ,x 1,2 ,x 1,3 ], where x 1,1 represents the probability that the first recommendation model is assigned traffic for recommendation in the first traffic channel, x 1,2 represents the probability that the first recommendation model is assigned traffic for recommendation in the second traffic channel, x 1,3 It indicates the probability that the first recommendation model is assigned traffic for recommendation in the third traffic channel.

[0060] For another example, taking the second recommendation model as an example, the channel recommendation probability set corresponding to the second recommendation model can be expressed as x 2,i =[x 2,1 ,x 2,2 ,x 2,3 ], where x 2,1 represents the probability that the second recommendation model is assigned traffic for recommendation in the first traffic channel, x 2,2 represents the probability that the second recommendation model is assigned traffic for recommendation in the second traffic channel, x 2,3 It indicates the probability that the second recommendation model is assigned traffic for recommendation in the third traffic channel.

[0061] For another example, taking the first recommendation model as an example, the customer group recommendation probability set corresponding to the first recommendation model can be expressed as p 1,i =[p 1,1 ,p 1,2 ,p 1,3 ], where p 1,1 represents the probability that the first recommendation model is assigned traffic for recommendation in the first customer group, p 1,2 represents the probability that the first recommendation model is assigned traffic for recommendation in the second customer group, p 1,3 It indicates the probability that the first recommendation model is assigned traffic for recommendation in the third customer group.

[0062] For another example, taking the second recommendation model as an example, the customer group recommendation probability set corresponding to the first recommendation model can be expressed as p 2,i =[p 2,1 ,p 2,2 ,p 2,3 ], where p 2,1 represents the probability that the second recommendation model is assigned traffic for recommendation in the first customer group, p 2,2 represents the probability that the second recommendation model is assigned traffic for recommendation in the second customer group, p 2,3 It indicates the probability that the second recommendation model is assigned traffic for recommendation in the third customer group.

[0063] It should be understood that obtaining the channel recommendation probability set of each recommendation model can help us understand the degree of advantage of each recommendation model in different traffic channels. The higher the degree of advantage of the recommendation model, the higher its online revenue. Furthermore, obtaining the customer group recommendation probability set of each recommendation model in different customer groups can help us understand the degree of adaptation of each recommendation model to different customer groups. A recommendation model with a higher degree of adaptation can push out recommended content that better meets user needs, which can improve the utilization rate of traffic.

[0064] In one possible implementation, the computing device can divide the traffic channels by converting the mobile phone number bound to the platform user or the platform user number into a hexadecimal number, and then taking the last two digits of the hexadecimal number as the classification number. Further, the user traffic with the same classification number is divided into one traffic channel. It should be noted that there are 100 combinations of the last two digits of a decimal number, and 256 combinations of the last two digits of a hexadecimal number. Converting decimal numbers into hexadecimal numbers can expand the number of traffic channels and refine the selection range of more recommendation models. In addition, the hexadecimal conversion can also conceal the digital characteristics of the mobile phone number, making the division more equal.

[0065] In another possible implementation, the computing device can divide the customer groups according to the information characteristics of the user. For example, in the user's age range, every 10 years is used as a division interval, thereby forming a total of 9 age intervals [≤10, 11-20, 21-30, 31-40, 41-50, 51-60, 61-70, 71-80, ≥80]. And then divide the users into two categories according to gender, and combine the 9 age intervals with the 2 gender categories, so there are a total of 9×2=18 customer groups.

[0066] S102: Determine a comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model.

[0067] The comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group in different traffic channels of the corresponding recommendation model.

[0068] It should be understood that the comprehensive recommendation probability is obtained by combining the channel recommendation probability and customer group recommendation probability of the recommendation model. From the perspective of channel recommendation probability, it is considered that the recommendation model has an advantage in online performance for the traffic channel, and from the perspective of customer group recommendation probability, it is considered that the recommendation model is suitable for the customer group. The comprehensive recommendation probability can combine the two aspects of the recommendation model data and comprehensively evaluate the competitiveness of the recommendation model from different dimensions.

[0069] In some embodiments, the comprehensive recommendation probability of the recommendation model may be obtained by multiplying each channel recommendation probability in its channel recommendation probability set by each customer group recommendation probability in its customer group recommendation probability set.

[0070] Exemplarily, taking the first recommendation model as an example, the channel recommendation probability set corresponding to the first recommendation model is expressed as x 1,i =[x 1,1 ,x 1,2 ,x 1,3 ], the customer group recommendation probability set corresponding to the first recommendation model is expressed as p 1,i =[p 1,1 ,p 1,2 ,p 1,3 ]. Then the comprehensive recommendation probability set of the first recommendation model can be y1=[x 1,1 p 1,1 ,x 1,1 p 1,2 ,x 1,1 p 1,3 ,x 1, 2p 1,1 ,x 1,2 p 1,2 ,x 1,2 p 1,3 ,x 1,3 p 1,1 ,x 1,3 p 1,2 ,x 1,3 p 1,3 ], where x 1,1 p 1,1 represents the probability that the first recommendation model is assigned traffic for recommendation in the first customer group under the first traffic channel, x 1,1 p 1,2 represents the probability that the first recommendation model is assigned traffic for recommendation in the second customer group under the first traffic channel, x 1,1 p 1,3 represents the probability that the first recommendation model is assigned traffic for recommendation in the third customer group under the first traffic channel, x 1,2p 1,1 It represents the probability that the first recommendation model is allocated traffic for recommendation in the third customer group under the second traffic channel. By analogy, we can obtain the comprehensive recommendation probability set of multiple customer groups corresponding to the recommendation model under multiple traffic channels.

[0071] S103. For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determine a target recommendation model corresponding to the target customer group from multiple recommendation models.

[0072] Among them, the target traffic channel is one of the multiple traffic channels, the target customer group is one of the multiple customer groups in the target traffic channel, and the target recommendation model recommends the target customer group in the target traffic channel within the future unit time.

[0073] In one possible implementation, the computing device may directly select, from multiple recommendation models, a recommendation model with the highest comprehensive recommendation probability for a target customer group in a target traffic channel as a target recommendation model based on a set of comprehensive recommendation probabilities corresponding to each recommendation model.

[0074] For example, taking the comprehensive recommendation probability set y1 of the first recommendation model as an example, if the target traffic channel is the first traffic channel and the target customer group is the first customer group in the first traffic channel, the computing device can compare the comprehensive recommendation probabilities for the first customer group in the first traffic channel in the comprehensive recommendation probability sets corresponding to different recommendation models (for example, the comprehensive recommendation probability x1 of the first recommendation model for the first customer group in the first traffic channel is 1,1 p 1,1 , the comprehensive recommendation probability x of the second recommendation model for the first customer group in the first traffic channel 2,1 p 2,1 , the comprehensive recommendation probability x of the third recommendation model for the first customer group of the first traffic channel 3,1 p 3,1 ), and select the recommendation model with the highest comprehensive recommendation probability as the target recommendation model.

[0075] In another possible implementation, the computing device may also normalize the comprehensive recommendation probabilities of the target customer groups in the target traffic channel in different comprehensive recommendation probability sets according to the comprehensive recommendation probability sets corresponding to each recommendation model, and select the recommendation model with the largest normalization result as the target recommendation model.

[0076] Exemplarily, the calculation process of the computing device performing normalization processing on the comprehensive recommendation probability can be expressed as:

[0077]

[0078] in:

[0079] y i,k,j It represents the normalized result of the comprehensive recommendation probability of the j-th recommendation model in the k-th customer group under the i-th traffic channel.

[0080] x i,j It represents the channel recommendation probability of the j-th recommendation model under the i-th traffic channel.

[0081] p i,k,j It represents the customer recommendation probability of the j-th recommendation model in the k-th customer group under the i-th traffic channel.

[0082] It represents the exponential power operation result of the product of the channel recommendation probability of the j-th recommendation model in the i-th traffic channel and the recommendation probability of the k-th customer group in the j-th recommendation model in the i-th traffic channel.

[0083] The sum of the exponential power calculation results of the product of the channel recommendation probability and the customer group recommendation probability corresponding to all recommendation models in the kth customer group under the i-th traffic channel, where n is the total number of recommendation models.

[0084] It should be noted that each comprehensive recommendation probability in the comprehensive recommendation probability set of the recommendation model represents the adaptation probability of the recommendation model for a certain customer group in a certain traffic channel. To determine the target recommendation model corresponding to the target customer group from multiple recommendation models, it is only necessary to compare the comprehensive recommendation probabilities of multiple recommendation models in the corresponding target customer group one by one, and select the recommendation model with the larger comprehensive recommendation probability as the recommendation model for the target customer group.

[0085] It should also be noted that the target recommendation model is a recommendation model for recommending the target customer group in the target traffic channel in the future unit time. It can be allocated traffic to recommend to the target customer group in the future unit time. And, at the end of the future unit time, steps S101-S103 of this application can be executed again to determine the recommendation model of the target customer group in the target traffic channel in the next future unit time.

[0086] Exemplarily, the unit duration in the future unit duration may be a shorter period such as 3 minutes, 5 minutes or 10 minutes. The shorter unit duration may enable the selected recommendation model to have certain timeliness advantages, and may reduce the risk of lagging recommended content and being out of touch with the user's current interests due to time differences.

[0087] A recommendation model selection method provided in an embodiment of the present application can refine the platform traffic into multiple traffic channels, and determine the channel recommendation probability of the recommendation model in each traffic channel through a horse racing mechanism, so as to screen out recommendation models with excellent online performance in the same traffic channel. Furthermore, among the customer groups divided under the traffic channel, by evaluating the customer group recommendation probabilities of multiple recommendation models in the same customer group, the recommendation model that best fits the customer group can be identified. This method can comprehensively evaluate the potential value of the recommendation model from multiple factors, and can improve the accuracy of the recommendation model selection, thereby giving full play to the value of traffic.

[0088] The following is a detailed introduction to the process of obtaining the recommendation effect evaluation parameters in the above step S101.

[0089] In some embodiments, Figure 2 As shown, the process of obtaining the recommendation effect evaluation parameters in step S101 also includes the following steps before step S101:

[0090] S201: Data collection.

[0091] Specifically, data collection is responsible for collecting the original behavior data of front-end users. The original behavior data can be the interactive behavior of users on the recommended content of the recommendation model. The original behavior at least includes: click, favorite, follow, share, comment, add to shopping cart, place an order, etc.

[0092] like Figure 2 As shown in the figure, data collection can be achieved by accessing SDK. Currently supported access methods include JavaScript SDK, Vue SDK, iOSSDK, Android SDK and React SDK. These SDKs can be embedded in visual interfaces such as web pages, apps and mini-programs to collect users' original behavior data and encapsulate it into a standard data format and upload it to the data receiver.

[0093] S202: Data reception.

[0094] Specifically, data reception is responsible for preprocessing and forwarding the original behavior data uploaded by the front end. Preprocessing can verify the validity and clean the uploaded data, and remove invalid and abnormal data. Forwarding is to transmit the preprocessed data to data processing.

[0095] like Figure 2 As shown, the data receiver can include a go receiver and a netty receiver. The go receiver has efficient concurrent processing capabilities, while the netty receiver has rich functions, supports multiple network protocols and data formats, and can flexibly cope with various complex network environments.

[0096] S203: Data transmission.

[0097] Specifically, data transmission is used to transmit the original behavior data from the receiver. First, data transmission plans the transmission path from data reception to data processing based on the business logic and flow of the data. Secondly, the received data is pushed to the transmission channel in sequence according to the established plan to ensure orderly transmission.

[0098] like Figure 2 As shown, Kafka can be used as a high-speed transmission channel for user behavior data, which can effectively ensure the timeliness of the data.

[0099] S204: data processing.

[0100] Specifically, data processing is used to store and analyze raw behavioral data.

[0101] like Figure 2 As shown in the figure, in terms of storage, the clickhouse column-based database is used to reduce redundancy while ensuring data integrity and traceability. In terms of analysis, materialized views are built based on the original behavioral data, and then the recommendation effect evaluation parameters such as click-through rate, conversion rate and average order value are analyzed to quickly feedback the current model performance. At the same time, xxl-job is used to process the dimension table data, and this type of data used to supplement the main data attributes and dimensions is processed into a usable state for accurate analysis and deep insight, assisting the final formation of each indicator.

[0102] The specific process of determining the channel probability set corresponding to each recommendation model in the above step S101 is introduced below.

[0103] In some embodiments, Figure 3 As shown, taking any first recommendation model among multiple recommendation models as an example, the step of determining the channel recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameter of each recommendation model in the historical unit time length in step S101 by the computing device may specifically include:

[0104] S301. For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time of the traffic channel, determine the future benefit of the first recommendation model in the traffic channel and the historical reference benefit of the traffic channel.

[0105] Among them, the future profit is determined based on the profit of the first recommendation model within the historical unit time, the historical reference profit is the sum of the profits of multiple models within the historical unit time, and the profit is determined based on the recommendation effect evaluation parameters of the recommendation model.

[0106] It should be noted that, as mentioned above, the unit duration used in this application is relatively short. This feature can make the revenue data of the recommendation model within two adjacent unit durations have a certain consistency and stability. At a certain time node, the revenue of the past adjacent historical unit duration of the time node can be approximately the revenue of the future unit duration of the node.

[0107] S302 . For each traffic channel, normalize the future revenue of the traffic channel and the historical reference revenue of the traffic channel according to the first recommendation model to obtain a channel recommendation probability corresponding to the traffic channel.

[0108] It should be understood that the computing device normalizing the historical returns of multiple recommendation models can calibrate the historical returns of different recommendation models to the same scale for comparison, which can provide a more intuitive understanding of the performance of the first recommendation model among multiple recommendation models.

[0109] In some embodiments, the normalization process may perform an exponential operation on the historical returns of multiple recommendation models under a traffic channel, and divide the exponential operation result of the historical returns of the first recommendation model by the sum of the exponential operation results of the historical returns of multiple recommendation models to obtain the channel recommendation probability of the first recommendation model.

[0110] Exemplarily, the calculation process of the channel recommendation probability can be expressed as:

[0111]

[0112] in:

[0113] represents the exponential power result of the benefit of the jth recommendation model in the i-th traffic channel in the future unit time t, C i,t,j represents the basic revenue of the jth recommendation model in the i-th traffic channel in the future unit time t, and the revenue adjustment amount α i,t,j It represents the revenue adjustment of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0114] It represents the exponential power result of all recommendation model benefits in the future unit time t under the i-th traffic channel.

[0115] S303: Determine a channel recommendation probability set corresponding to the first recommendation model based on a plurality of channel recommendation probabilities corresponding to each of the plurality of traffic channels.

[0116] Specifically, the computing device can obtain multiple channel recommendation probabilities by calculating the channel recommendation probabilities of the first recommendation model under different traffic channels, and aggregate the multiple channel recommendation probabilities to obtain a channel recommendation probability set of the first recommendation model.

[0117] In the above steps S301-S303, a method for determining a recommendation model channel recommendation probability set is provided in an embodiment of the present application, and the channel probability set of the recommendation model is calculated by calculating the online effect of the recommendation model, wherein the better the online effect of the recommendation model, the greater the probability of obtaining channel traffic. This method strengthens the correlation between traffic distribution and recommendation effect, effectively stimulates the competitive vitality between recommendation models, and thus realizes the horse racing mechanism of recommendation model selection.

[0118] The specific process of determining the future benefit of the first recommendation model in the traffic channel in the above step S301 is introduced below.

[0119] In some embodiments, Figure 4 As shown, taking any one of the multiple recommendation models as an example, the step of determining the future benefit of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameter of the first recommendation model in the historical unit time of the traffic channel by the computing device in step S301 may specifically include:

[0120] S401. For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time of the traffic channel, determine the basic revenue of the first recommendation model within the historical unit time of the traffic channel.

[0121] In some embodiments, the recommendation effect evaluation parameters of each recommendation model within the historical unit time of multiple traffic channels can be obtained by online recovery and processing of the user's original behavior data.

[0122] For example, the specific calculation process of basic income can be expressed as:

[0123] C i,t,j =CTR i,t,j ×CVR i,t,j ×AOV i,t,j Formula (3)

[0124] in:

[0125] C i,t,j It represents the basic revenue of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0126] CTR i,t,j It represents the click rate of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0127] CVR i,t,j It represents the conversion rate of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0128] AOVi,t,j It represents the average order value of the j-th recommendation model under the i-th traffic channel within the future unit time t.

[0129] It should be noted that, as can be seen from the aforementioned step S301, the recommendation effect evaluation parameter of the future unit duration t in the above formula is actually the recommendation effect evaluation parameter of its adjacent historical unit duration.

[0130] S402: Determine the future revenue of the first recommendation model in the traffic channel according to the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the historical unit time of the traffic channel.

[0131] It should be understood that the basic revenue represents the actual online revenue of the recommendation model in the historical unit time in the traffic channel. The revenue adjustment also takes into account the revenue fluctuations caused by traffic differences in the entire historical time of the recommendation model, and then the potential value of the recommendation model can be excavated. By combining the basic revenue and the revenue adjustment, the future revenue of the recommendation model can be effectively evaluated.

[0132] The specific process of obtaining the revenue adjustment amount of the first recommendation model within the historical unit time of the traffic channel in the above step S402 is introduced below.

[0133] In some embodiments, Figure 5 As shown, taking any one of the multiple traffic channels and any one of the multiple recommendation models as an example, the step of determining the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel in step S402 may specifically include:

[0134] S501. Determine a proportionality coefficient according to the difference ratio between the estimated excitation flow value per unit time of the first recommendation model in the future and the actual excitation flow value per unit time of the history.

[0135] It should be noted that the flow difference between the estimated excitation flow value of the first recommended model in the future unit time and its actual excitation flow value in the historical unit time is calculated, and the flow difference represents the flow value that should be allocated to the first recommended model but is not allocated. The ratio of the flow difference to the estimated excitation flow value is calculated to obtain the proportional coefficient, which represents the proportion of the flow difference to the estimated excitation flow value.

[0136] In some embodiments, after determining the proportionality coefficient, the proportionality coefficient may be multiplied by a step size parameter of the recommendation model, and the step size parameter is used to control the iteration amplitude of the model. This method can take into account the influence of iterative update of the recommendation model while calculating the flow difference.

[0137] S502. Determine a reference adjustment amount according to the product of the basic income of the first recommendation model within a historical unit time and the proportional coefficient.

[0138] Specifically, the reference adjustment amount can be understood as the revenue fluctuation caused by the aforementioned traffic difference.

[0139] S503: Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel according to the sum of the basic revenue adjustment amount of the first recommendation model in the traffic channel and the reference adjustment amount.

[0140] For example, the calculation process of the revenue adjustment amount of the recommendation model in the future unit time can be expressed as:

[0141]

[0142] in:

[0143] I i,t,j It represents the estimated stimulated traffic value of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0144] ∑x i,t-1,j It represents the total traffic obtained by the j-th recommendation model in the i-th traffic channel within the historical unit time t-1.

[0145] C t,i,j It represents the basic revenue of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0146] α i,t-1,j It represents the revenue adjustment of the j-th recommendation model in the i-th traffic channel within the historical unit time t-1.

[0147] α i,t,j It represents the revenue adjustment of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0148] In some embodiments, Figure 6 As shown, taking any one of the multiple traffic channels and any one of the multiple recommendation models as an example, the method for determining the estimated excitation traffic value of the first recommendation model per unit time in the future in step S501 may specifically include:

[0149] S601: Obtain historical traffic sequences of multiple recommendation models.

[0150] The historical traffic sequence includes historical stimulated traffic values ​​corresponding to multiple historical unit time lengths, and the historical stimulated traffic value is the total traffic value stimulated by recommendations from multiple recommendation models within the corresponding historical unit time length.

[0151] S602: Based on the historical traffic sequence and the traffic prediction model, predict the total estimated stimulated traffic value stimulated by the recommendations of multiple recommendation models within a future unit time period.

[0152] S603: Determine the estimated stimulation flow value of the recommendation model per unit time in the future according to the product of the total estimated stimulation flow value and the historical revenue ratio of the first recommendation model.

[0153] Among them, the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit time of the traffic channel to the total basic revenue of multiple recommendation models in the historical unit time of the traffic channel.

[0154] Exemplarily, the calculation process of estimating the excitation flow value can be expressed as:

[0155]

[0156] in:

[0157] I i,t,j It represents the estimated stimulated traffic value of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0158] V i,t It represents the total estimated stimulated traffic value stimulated by the recommendations of multiple recommendation models in the future unit time t under the i-th traffic channel.

[0159] C i,t,j It represents the basic revenue of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0160] It represents the sum of the basic benefits of multiple recommendation models in the future unit time t under the i-th traffic channel.

[0161] In some embodiments, Figure 7 As shown, the traffic prediction model in step S602 may be an ARMA model, and the method for constructing the model may specifically include:

[0162] S701. Perform differential processing on the historical traffic sequence.

[0163] It should be noted that the historical traffic series may be non-stationary, that is, its statistical characteristics such as mean, variance or autocovariance will change over time. The ARMA model requires that the time series data is stationary, so the historical traffic series needs to be differentiated. Difference is to subtract the historical excitation traffic value of the previous historical unit time from the historical excitation traffic value of the current historical unit time (first-order difference), or the historical excitation traffic value of the current historical unit time from the historical excitation traffic value of several previous historical unit time (high-order difference). In this way, non-stationary factors such as trend and seasonality in the data are removed to make it stable for subsequent modeling.

[0164] S702. Draw a fixed-order heat map based on the historical traffic sequence.

[0165] It should be noted that the fixed-order heat map is a visualization tool used to determine the combined effects of different values ​​of the autoregressive order p and the moving average order q in the ARMA model.

[0166] Specifically, a maximum order L is determined based on the length or time characteristics of the historical traffic sequence, where L represents the maximum length of the historical traffic sequence that can be used. The value range of p and q is determined based on the maximum order L, that is, (0, L]. Within the value range, the value combinations of p and q are traversed, and the Akaike information criterion (AIC) value is calculated for each combination. The AIC values ​​for different combinations of p and q are displayed on a two-dimensional plane (the horizontal axis is p and the vertical axis is q), and the size of the indicator is represented by the depth of color.

[0167] It should be noted that the autoregressive order p represents the number of historical traffic series used by the autoregressive part of the model. Similarly, the sliding average order q represents the number of historical traffic series used by the sliding average part of the model.

[0168] S703: Determine the order of ARMA based on the determined-order heat map.

[0169] Specifically, the p-value and q-value of the darkest area in the fixed-order heat map are retrieved, and the combination of the p and q values ​​is used as the order of ARMA.

[0170] It should be noted that the smaller the AIC value in the fixed-order heat map, the better the model performs in fitting the data, and the complexity of the model is relatively low. The order of ARMA is determined based on the p-value and q-value corresponding to the darkest color area in the fixed-order heat map.

[0171] S704. Perform residual normality test and residual independence test on the ARMA model.

[0172] Specifically, ARMA modeling is performed based on the determined values ​​of p and q, and the model is tested using residual normality test and residual independence test.

[0173] It should be noted that the purpose of the residual normality test is to check whether the residuals after the ARMA model is fitted obey the normal distribution. Because the model assumes that the residuals are normally distributed white noise sequences, if the normality is not satisfied, it may mean that there are problems with the model and it cannot fit the data well.

[0174] It should also be noted that the purpose of the residual independence test is to determine whether the residuals are independent of each other, that is, there is no autocorrelation in the residual sequence. If there is autocorrelation in the residuals, it means that the model does not fully capture the information in the data and may need to be re-adjusted.

[0175] In some embodiments, if the model constructed by the above p and q cannot satisfy the residual value normality test or the residual value independence test, it is necessary to re-execute steps S701-S703 to determine the values ​​of p and q again.

[0176] S705 . Predict the total estimated excitation flow value within a future unit time period by using the ARMA model.

[0177] Specifically, an ARMA model that satisfies both the residual normality test and the residual independence test is used to predict future traffic excitation values.

[0178] In other embodiments, the traffic prediction model in step S602 may also be an ARIMA model.

[0179] The specific process of determining the customer group recommendation probability set corresponding to each recommendation model in the above step S101 is introduced below.

[0180] In some embodiments, Figure 8 As shown, taking any first recommendation model among the multiple recommendation models as an example, the step of determining the customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameter of each recommendation model in the historical unit time length in step S101 by the computing device may further include:

[0181] S801. For each customer group, based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time, determine the inference samples of the first recommendation model.

[0182] Specifically, the inference sample includes historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and model hyperparameters of the first recommendation model.

[0183] In some embodiments, the historical recommendation effect evaluation parameters may include at least one of the following: click-through rate, conversion rate, and average order value.

[0184] In some embodiments, the model hyperparameters of the first recommendation model include at least: category of the recommendation model, sample size, number of features, learning rate, and number of batch samples.

[0185] S802: For each customer group, based on the inference samples of the first recommendation model and the recommendation effect evaluation model, predict future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group.

[0186] Specifically, the recommendation effect evaluation model is used to predict future recommendation effect evaluation parameters based on historical recommendation effect evaluation parameters and model hyperparameters.

[0187] It should be understood that the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model. Using the inference sample as the input of the recommendation effect evaluation model can enable the recommendation effect evaluation model to understand the internal logic of the first recommendation model and the performance effect of the recommended content, thereby allowing the recommendation effect evaluation model to predict future recommendation effect evaluation parameters that are more in line with the actual performance of the recommendation model.

[0188] S803. For each customer group, determine the customer group recommendation probability of the first recommendation model for the customer group according to the proportion of the future recommendation effect evaluation parameter of the first recommendation model in the total future recommendation effect evaluation parameters of all recommendation models.

[0189] It should be noted that after predicting the future recommendation effect evaluation parameters of the first recommendation model, it is still necessary to compare them with the future recommendation effect evaluation parameters of other recommendation models under the same customer group. The degree of advantage of the first recommendation model over other recommendation models under the same customer group can be determined by comparing the future recommendation effect evaluation parameters of the first recommendation model to the total future recommendation effect evaluation parameters of all recommendation models.

[0190] S804 . Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, a customer group recommendation probability set corresponding to the first recommendation model is obtained.

[0191] It should be understood that the channel recommendation probability set of the recommendation model calculated in the aforementioned steps S201-S203 is calculated from the perspective of the online revenue of the recommendation model, while steps S801-803 are calculated from the perspective of the customer group, taking into account the degree of adaptation of the recommendation model to the customer group. Under the same customer group, a recommendation model with a higher customer recommendation probability has greater potential than other recommendation models to push content that meets user needs.

[0192] In some embodiments, Fig. 9 As shown, the process of obtaining the recommendation effect evaluation model in step S802 further includes before step S802:

[0193] S901: Obtain a training sample set.

[0194] Specifically, the training sample set includes multiple training samples, each training sample includes recommendation effect evaluation parameters when the recommendation model makes recommendations for a customer group in a first time period, model hyperparameters of the recommendation model, and recommendation effect evaluation parameter labels when the recommendation model makes recommendations for the customer group in a second time period after the first time period.

[0195] It should be noted that the recommendation effect evaluation parameters when making recommendations for a customer group recommendation model in the first period or the second period are all obtained from the historical recommendation effect evaluation parameters of the recommendation model.

[0196] In some embodiments, the historical recommendation effect evaluation parameters of the recommendation model can be obtained by executing the aforementioned steps S401-S404, which will not be described in detail here.

[0197] S902: Train the preset initial model based on the training sample set to obtain a recommendation effect evaluation model.

[0198] Exemplarily, taking the process of training the initial model using a training sample as an example, the recommendation effect evaluation parameters when making recommendations for a customer group in the first period and the model hyperparameters of the recommendation model are input into the initial model, and the recommendation effect evaluation parameters when making recommendations for the customer group in the second period after the first period are used as the true label. The recommendation effect evaluation parameters when the initial model makes recommendations for the customer group in the second period after the first period are predicted, and after obtaining the output of the initial model, the true value deviation between the output content and the true label is measured using the true label. The initial model is adjusted according to the true deviation, and the next training sample is used for the next round of training.

[0199] In some embodiments, the recommendation effect evaluation model can be trained using the XGBoost algorithm.

[0200] In other embodiments, the LR algorithm or the LightGBM algorithm may also be used to train the recommendation effect evaluation model.

[0201] It should be understood that steps S901-S902 train the preset model through the model hyperparameters of multiple recommendation models and the recommendation effect evaluation parameters within the historical unit time. It can adaptively learn the customer groups that each recommendation model is suitable for through the artificial intelligence algorithm, and because of the continuous data inflow from the online platform, it will also effectively improve the prediction accuracy of the recommendation effect evaluation model. Fig.10This is an overall architecture diagram of a recommendation model selection method provided in an embodiment of the present application. Fig.10 As shown, where:

[0202] Online effect recovery. The computing device collects the online effects of multiple recommendation models within the historical unit time at the current moment and processes them into recommendation effect evaluation parameters as the data source for subsequent calculations.

[0203] Obtain a set of channel recommendation probabilities corresponding to each recommendation model. The computing device calculates the future benefits of the recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time under each traffic channel, and then calculates the channel recommendation probability of each recommendation model. Aggregate the channel recommendation probabilities of multiple traffic channels corresponding to each recommendation model into the channel recommendation probability set of the recommendation model.

[0204] A set of customer group recommendation probabilities corresponding to each recommendation model is obtained. The computing device uses the recommendation effect evaluation parameter model to predict the future effect evaluation parameters of each recommendation model in each customer group, and then calculates a set of customer group recommendation probabilities for each recommendation model in different customer groups.

[0205] Based on the channel recommendation probability set and customer group recommendation probability set of each recommendation model, the comprehensive recommendation probability set of each recommendation model is calculated. The calculation process is as described in the above step S102 and will not be repeated here.

[0206] The target recommendation model is determined based on the comprehensive recommendation probabilities of the target customer groups under the target traffic channel by multiple recommendation models. The comprehensive recommendation probability of each recommendation model corresponds to a customer group under a traffic channel. When calculating the target recommendation model for the target customer group under the target traffic channel, the comprehensive recommendation probabilities of multiple recommendation models corresponding to the target customer group are compared, and the recommendation model with the higher comprehensive recommendation probability is selected as the recommendation model for the target customer group.

[0207] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to achieve the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed in this article, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0208] In an exemplary embodiment, the present application also provides a recommendation model selection device, which can be applied to the above computing device. Fig.11 As shown, the device includes: an acquisition module 110 and a processing module 120.

[0209] The acquisition module 110 is used to determine the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length. The channel recommendation probability set includes multiple channel recommendation probabilities corresponding to multiple traffic channels one by one, and the channel recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel. Each of the multiple traffic channels is divided into multiple customer groups, and the customer group recommendation probability set includes multiple customer group recommendation probabilities corresponding to multiple customer groups one by one, and the customer group recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group.

[0210] The processing module 120 is used to determine the comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model, and the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group of the corresponding recommendation model in different traffic channels. For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, the target recommendation model corresponding to the target customer group is determined from multiple recommendation models, the target traffic channel is one of the multiple traffic channels, the target customer group is one of the multiple customer groups in the target traffic channel, and the target recommendation model recommends the target customer group in the target traffic channel within the future unit time.

[0211] In a possible implementation, the acquisition module 110 is specifically used to, for any first recommendation model among multiple recommendation models, determine the channel recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length, including: for each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time length of the traffic channel, determine the future income of the first recommendation model in the traffic channel and the historical reference income of the traffic channel. The future income is determined based on the income of the first recommendation model in the historical unit time length, the historical reference income is the sum of the income of multiple models in the historical unit time length, and the income is determined based on the recommendation effect evaluation parameters of the recommendation model. For each traffic channel, normalization is performed according to the future income of the first recommendation model in the traffic channel and the historical reference income of the traffic channel to obtain the channel recommendation probability corresponding to the traffic channel. Based on the multiple channel recommendation probabilities corresponding to the multiple traffic channels, determine the channel recommendation probability set corresponding to the first recommendation model.

[0212] In another possible implementation, the processing module 110 is specifically used to determine, for each traffic channel, the future revenue of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, including: for each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, determine the basic revenue of the first recommendation model in the historical unit time of the traffic channel. Determine the future revenue of the first recommendation model in the traffic channel based on the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

[0213] In another possible implementation, the processing module 120 is specifically used to determine the revenue adjustment amount, including: determining a proportional coefficient based on the ratio of the difference between the estimated excitation flow value of the first recommendation model in the future unit time and the actual excitation flow value of the historical unit time. Determine a reference adjustment amount based on the product of the basic revenue of the first recommendation model in the historical unit time and the proportional coefficient. Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel based on the sum of the basic revenue adjustment amount of the first recommendation model in the traffic channel and the reference adjustment amount, the basic revenue adjustment amount being the revenue adjustment amount of the first recommendation model in the historical unit time.

[0214] In another possible implementation, the processing module 120 is specifically used to determine the estimated excitation traffic value of the first recommendation model for a future unit duration, including: obtaining historical traffic sequences of multiple recommendation models, the historical traffic sequences include historical excitation traffic values ​​corresponding to multiple historical unit durations, and the historical excitation traffic value is the total traffic value stimulated by recommendations by multiple recommendation models in the corresponding historical unit duration. Based on the historical traffic sequence and the traffic prediction model, the total estimated excitation traffic value stimulated by recommendations by multiple recommendation models in the future unit duration is predicted. The estimated excitation traffic value of the recommendation model for the future unit duration is determined based on the product of the total estimated excitation traffic value and the historical revenue ratio of the first recommendation model, and the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit duration of the traffic channel to the total basic revenue of multiple recommendation models in the historical unit duration of the traffic channel.

[0215] In another possible implementation, the acquisition module 110 is specifically used to, for any first recommendation model among the multiple recommendation models, determine the customer group recommendation probability set corresponding to each recommendation model among the multiple recommendation models, including: for each customer group, determine the inference sample of the first recommendation model based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time length, the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model. For each customer group, predict the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group based on the inference samples of the first recommendation model and the recommendation effect evaluation model, the recommendation effect evaluation model is used to predict the future recommendation effect evaluation parameters based on the historical recommendation effect evaluation parameters and the model hyperparameters. For each customer group, determine the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameters of the first recommendation model in the sum of the future recommendation effect evaluation parameters of all recommendation models. Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, obtain the customer group recommendation probability set corresponding to the first recommendation model.

[0216] In another possible implementation, the processing module 120 is further used to obtain a training sample set, the training sample set includes multiple training samples, each training sample includes a recommendation effect evaluation parameter when the recommendation model makes a recommendation for a customer group in a first period, a model hyperparameter of the recommendation model, and a recommendation effect evaluation parameter label when the recommendation model makes a recommendation for the customer group in a second period after the first period. The preset initial model is trained based on the training sample set to obtain a recommendation effect evaluation model.

[0217] It should be noted that Fig.11 The division of modules in the example is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0218] In an exemplary embodiment, as described above, the computing device may be a computer or a server or other electronic device having a computing function. In this case, the present application also provides an electronic device, Fig.12 The following is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application. Fig.12 As shown, the electronic device includes: a processor 10 , a memory 20 , a communication line 30 , a communication interface 40 , and an input / output interface 50 .

[0219] The processor 10 , the memory 20 , the communication interface 40 and the input / output interface 50 may be connected via a communication line 30 .

[0220] The processor 10 is used to execute the instructions stored in the memory 20 to implement the recommended model selection method provided in the above embodiments of the present application. The processor 10 can be a CPU, a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / single-chip microcomputer, a programmable logic device (PLD) or any combination thereof. The processor 10 can also be any other device with processing functions, such as a circuit, a device or a software module, which is not limited in the embodiments of the present application. In one example, the processor 10 may include one or more CPUs, such as Fig.12 As an optional implementation, the electronic device may include multiple processors, for example, in addition to the processor 10, it may also include a processor 60 ( Fig.12 The dashed line is used as an example.

[0221] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage devices, etc., and the embodiments of the present application are not limited to this.

[0222] It should be noted that the memory 20 may exist independently of the processor 10, or may be integrated with the processor 10. The memory 20 may be located inside the electronic device, or may be located outside the electronic device, which is not limited in the embodiment of the present application.

[0223] The communication line 30 is used to transmit information between various components included in the electronic device.

[0224] The communication interface 40 is used to communicate with other devices or other communication networks. The other communication networks may be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 40 may be a module, a circuit, a transceiver or any device capable of achieving communication.

[0225] The input / output interface 50 is used to implement human-computer interaction between a user and an electronic device, for example, to implement action interaction or information interaction between a user and an electronic device.

[0226] Exemplarily, the input / output interface 50 may be a mouse, a keyboard, a display screen, or a touch display screen, etc. Action interaction or information interaction between a user and an electronic device may be achieved through a mouse, a keyboard, a display screen, or a touch display screen, etc.

[0227] It should be noted that Fig.12 The structure shown in the figure does not constitute a limitation on the electronic device, except Fig.12 In addition to the components shown, the electronic device may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0228] In an exemplary embodiment, the embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment.

[0229] In an exemplary embodiment, the present application also provides a readable storage medium, which includes software instructions. When the software instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment. The computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device.

[0230] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When loading and executing computer-executable instructions on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer-executable instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center.

[0231] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other changes to the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0232] Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the drawings are merely exemplary illustrations of the present application as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, those skilled in the art may make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A recommendation model selection method, characterized in that: The method comprises: Based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length, determine the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model; the channel recommendation probability set includes multiple channel recommendation probabilities corresponding to multiple traffic channels one by one; the channel recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel; each traffic channel in the multiple traffic channels is divided into multiple customer groups; the customer group recommendation probability set includes multiple customer group recommendation probabilities corresponding to the multiple customer groups one by one; the customer group recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group; Based on the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model, determine the comprehensive recommendation probability set corresponding to each recommendation model; the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group of the corresponding recommendation model in different traffic channels; For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, the target recommendation model corresponding to the target customer group is determined from the multiple recommendation models; the target traffic channel is a traffic channel among the multiple traffic channels; the target customer group is a customer group among the multiple customer groups of the target traffic channel; the target recommendation model recommends the target customer group in the target traffic channel within a future unit time.

2. The method according to claim 1, characterized in that For any first recommendation model among the multiple recommendation models, determining a channel recommendation probability set corresponding to each recommendation model based on a recommendation effect evaluation parameter of each recommendation model in a historical unit time length among the multiple recommendation models includes: For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time length of the traffic channel, determine the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel; the future revenue is determined based on the revenue of the first recommendation model in the historical unit time length; the historical reference revenue is the sum of the revenues of the multiple models in the historical unit time length; the revenue is determined based on the recommendation effect evaluation parameters of the recommendation model; For each traffic channel, normalize the future revenue of the traffic channel and the historical reference revenue of the traffic channel according to the first recommendation model to obtain a channel recommendation probability corresponding to the traffic channel; Based on a plurality of channel recommendation probabilities corresponding to each of the plurality of traffic channels, a channel recommendation probability set corresponding to the first recommendation model is determined.

3. The method according to claim 2, characterized in that The determining, for each traffic channel, the future benefit of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameter of the first recommendation model in the historical unit time of the traffic channel includes: For each traffic channel, based on the recommendation effect evaluation parameter of the first recommendation model in the historical unit time of the traffic channel, determine the basic revenue of the first recommendation model in the historical unit time of the traffic channel; The future revenue of the first recommendation model in the traffic channel is determined according to the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

4. The method according to claim 3, characterized in that The income adjustment amount is determined in the following manner: Determine a proportionality coefficient according to a difference ratio between the estimated excitation flow value per unit time of the first recommendation model in the future and the actual excitation flow value per unit time of the history; Determine a reference adjustment amount according to the product of the basic income of the first recommendation model in the historical unit time and the proportional coefficient; The revenue adjustment amount of the first recommendation model in the traffic channel in the future unit time is determined based on the sum of the basic revenue adjustment amount and the reference adjustment amount of the first recommendation model in the traffic channel; the basic revenue adjustment amount is the revenue adjustment amount of the first recommendation model in the historical unit time.

5. The method according to claim 3, characterized in that: The estimated excitation flow value per unit time of the first recommendation model in the future is determined by: Acquire the historical traffic sequences of the multiple recommendation models; the historical traffic sequences include historical stimulated traffic values ​​corresponding to multiple historical unit time lengths; the historical stimulated traffic values ​​are the total traffic values ​​stimulated by the recommendations of the multiple recommendation models within the corresponding historical unit time lengths; Based on the historical traffic sequence and the traffic prediction model, predict the total estimated stimulated traffic value stimulated by the recommendations of the multiple recommendation models within a future unit time length; The estimated stimulated traffic value of the recommendation model in the future unit time is determined based on the product of the total estimated stimulated traffic value and the historical revenue ratio of the first recommendation model; the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit time of the traffic channel to the total basic revenue of the multiple recommendation models in the historical unit time of the traffic channel.

6. The method according to claim 1, characterized in that For any first recommendation model among the multiple recommendation models, determining the customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameter of each recommendation model in the multiple recommendation models within the historical unit time length includes: For each customer group, based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time length, determine the inference samples of the first recommendation model; the inference samples include the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model; For each customer group, based on the inference samples of the first recommendation model and the recommendation effect evaluation model, predict the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group; the recommendation effect evaluation model is used to predict the future recommendation effect evaluation parameters based on the historical recommendation effect evaluation parameters and model hyperparameters; For each customer group, determining the customer group recommendation probability of the first recommendation model for the customer group according to the proportion of the future recommendation effect evaluation parameter of the first recommendation model in the sum of the future recommendation effect evaluation parameters of all recommendation models; Based on the customer group recommendation probabilities for different customer groups of the first recommendation model, a customer group recommendation probability set corresponding to the first recommendation model is obtained.

7. The method according to claim 6, characterized in that The method further comprises: Obtain a training sample set; the training sample set includes multiple training samples, each training sample includes a recommendation effect evaluation parameter when the recommendation model makes a recommendation for a customer group in a first period, a model hyperparameter of the recommendation model, and a recommendation effect evaluation parameter label when the recommendation model makes a recommendation for the customer group in a second period after the first period; The preset initial model is trained based on the training sample set to obtain the recommendation effect evaluation model.

8. The method according to any one of claims 1 to 7, characterized in that: The recommendation effect evaluation parameter includes at least one of the following: click-through rate, conversion rate, and average order value.

9. A recommendation model selection device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to determine the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length; the channel recommendation probability set includes multiple channel recommendation probabilities corresponding to multiple traffic channels one by one; the channel recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel; each traffic channel in the multiple traffic channels is divided into multiple customer groups; the customer group recommendation probability set includes multiple customer group recommendation probabilities corresponding to the multiple customer groups one by one; the customer group recommendation probability is used to indicate the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group; The processing module is used to determine the comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model; the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group of the corresponding recommendation model in different traffic channels; for the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determine the target recommendation model corresponding to the target customer group from the multiple recommendation models; the target traffic channel is one of the multiple traffic channels; the target customer group is one of the multiple customer groups in the target traffic channel; the target recommendation model recommends the target customer group in the target traffic channel within a future unit time.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: include: Software instructions; When the software instructions are executed in an electronic device, the electronic device implements the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 8.

Images