A recommendation model selection method and apparatus, an electronic device, and a program product

Abstract

The application provides a recommendation model selection method and device, electronic equipment and program product, and relates to the field of recommendation models. The method comprises: determining a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model based on a recommendation effect evaluation parameter of each recommendation model in a historical unit time period; determining a comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model; and determining a target recommendation model corresponding to a target customer group in a target traffic channel from a plurality of recommendation models based on the comprehensive recommendation probability set corresponding to each recommendation model. By calculating the traffic channel recommendation probability and the customer group recommendation probability, the selection range of the recommendation model can be refined, and the recommendation model traffic can be allocated according to different customer groups, thereby effectively improving the accuracy of model selection.

Description

Technical Field

[0001] The present application relates to the field of recommendation models, and in particular to a recommendation model selection method, device, electronic device, and program product. Background Art

[0002] With the development of the Internet and communication technologies, recommendation models have occupied a key position in various online platforms such as e-commerce and content information. They conduct in-depth analysis based on user behavior habits and preference characteristics, and then push content that meets user needs. This not only improves the user experience, but also helps the platform improve user retention and business conversion rates.

[0003] In addition, in order to achieve better recommendation results, the same online platform may deploy multiple recommendation models. These models can output different types of recommendation content due to different algorithm logic and business focus, thereby adapting to the needs of diverse user groups.

[0004] However, while the coexistence of multiple recommendation models enriches the recommendation content, it also brings about the problem of selecting the recommendation model. Summary of the Invention

[0005] The present application provides a recommendation model selection method, device, electronic device and program product. By calculating the model traffic channel recommendation probability and customer group recommendation probability, it can refine the selection range of the recommendation model and distribute the recommendation model traffic for different customer groups, thereby improving the accuracy of model selection.

[0006] In a first aspect, the present application provides a recommendation model selection method, the method comprising: obtaining recommendation effect evaluation parameters of each recommendation model in a historical unit time length for a plurality of recommendation models, and determining a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model. Based on the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model, a comprehensive recommendation probability set corresponding to each recommendation model is determined. For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, a target recommendation model corresponding to the target customer group is determined from multiple recommendation models.

[0007] The recommendation model selection method provided in the present application divides traffic into different traffic channels, and each traffic channel is divided into multiple customer groups according to user characteristics. By obtaining the traffic channel recommendation probability set of each model in different traffic channels, a horse racing mechanism is implemented within the segmented traffic channels. The performance of each model in different traffic channels is clarified according to the traffic channel recommendation probability. Furthermore, the customer group recommendation probability of the model in each customer group is calculated, and the adaptability of the model in different customer groups can also be determined. The comprehensive recommendation probability set combines the traffic channel probability and customer group recommendation probability of the recommendation model to measure the competitiveness of the recommendation model from multiple aspects. Compared with the relatively single decision-making basis and decision-making process of the prior art, the present application refines traffic into different traffic channels and users into different customer groups, breaking the previous general distribution model, evaluating the adaptability of the recommendation model from different aspects, and improving the accuracy of the recommendation model selection.

[0008] A possible implementation method is to determine the channel recommendation probability set corresponding to each recommendation model for any first recommendation model among multiple recommendation models based on the recommendation effect evaluation parameters of each recommendation model in the multiple recommendation models within a historical unit time, including: for each traffic channel, determine the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel. The future revenue is determined based on the revenue of the first recommendation model in the historical unit time, and the historical reference revenue is the sum of the revenues of multiple models in the historical unit time, and the revenue is determined based on the recommendation effect evaluation parameters of the recommendation model. For each traffic channel, normalization is performed based on the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel to obtain the channel recommendation probability corresponding to the traffic channel. Based on the multiple channel recommendation probabilities corresponding to the multiple traffic channels, determine the channel recommendation probability set corresponding to the first recommendation model.

[0009] Another possible implementation method is to determine, for each traffic channel, the future revenue of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, including: for each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, determining the basic revenue of the first recommendation model in the historical unit time of the traffic channel. Determine the future revenue of the first recommendation model in the traffic channel based on the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

[0010] Another possible implementation method is to determine the revenue adjustment amount, including: determining a proportional coefficient based on the ratio of the difference between the estimated stimulated traffic value of the first recommendation model in the future unit time and the actual stimulated traffic value in the historical unit time. Determine a reference adjustment amount based on the product of the basic revenue of the first recommendation model in the historical unit time and the proportional coefficient. Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel based on the sum of the basic revenue adjustment amount of the first recommendation model in the traffic channel and the reference adjustment amount, where the basic revenue adjustment amount is the revenue adjustment amount of the first recommendation model in the historical unit time.

[0011] Another possible implementation method is to determine the estimated stimulated traffic value of the first recommendation model for the future unit duration, including: obtaining historical traffic sequences of multiple recommendation models, the historical traffic sequences including historical stimulated traffic values ​​corresponding to multiple historical unit durations, and the historical stimulated traffic values ​​being the total traffic values ​​stimulated by recommendations from multiple recommendation models within the corresponding historical unit durations. Based on the historical traffic sequences and the traffic prediction model, the total estimated stimulated traffic value stimulated by recommendations from multiple recommendation models within the future unit durations is predicted. The estimated stimulated traffic value of the recommendation model for the future unit duration is determined based on the product of the total estimated stimulated traffic value and the historical revenue ratio of the first recommendation model, the historical revenue ratio being the ratio of the basic revenue of the first recommendation model within the historical unit duration of the traffic channel to the total basic revenue of multiple recommendation models within the historical unit duration of the traffic channel.

[0012] Another possible implementation method is to determine, for any first recommendation model among multiple recommendation models, a set of customer group recommendation probabilities corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model within a historical unit time period, including: for each customer group, determining an inference sample of the first recommendation model based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time period, the inference sample including the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group and the model hyperparameters of the first recommendation model. For each customer group, predicting the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group based on the inference sample of the first recommendation model and the recommendation effect evaluation model, the recommendation effect evaluation model being used to predict the future recommendation effect evaluation parameters based on the historical recommendation effect evaluation parameters and the model hyperparameters. For each customer group, determining the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameters of the first recommendation model in the sum of the future recommendation effect evaluation parameters of all recommendation models. Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, a set of customer group recommendation probabilities corresponding to the first recommendation model is obtained.

[0013] Another possible implementation involves obtaining a training sample set, where the training sample set includes multiple training samples, each of which includes recommendation effectiveness evaluation parameters for a recommendation model when performing recommendations for a customer group in a first time period, model hyperparameters of the recommendation model, and recommendation effectiveness evaluation parameter labels for the recommendation model when performing recommendations for the same customer group in a second time period after the first time period. A preset initial model is trained based on the training sample set to obtain a recommendation effectiveness evaluation model.

[0014] In a second aspect, the present application provides a recommendation model selection device, which includes various functional modules used in the method described in the first aspect above.

[0015] In a third aspect, the present application provides a computer program product, comprising: computer instructions; when the computer instructions are executed on an electronic device, the electronic device implements the method described in the first aspect above.

[0016] In a fourth aspect, the present application provides an electronic device comprising: a processor and a memory; the memory stores instructions executable by the processor; when the processor is configured to execute the instructions, the electronic device implements the method described in the first aspect above.

[0017] In a fifth aspect, the present application provides a readable storage medium, which includes: software instructions; when the software instructions are executed in an electronic device, the electronic device implements the method described in the first aspect above.

[0018] The beneficial effects of the second to fifth aspects mentioned above can be referred to the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 A flowchart of a recommendation model selection method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a method for obtaining evaluation parameters of a recommendation model recommendation effect provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of a method for determining a channel recommendation probability set of a recommendation model provided in an embodiment of the present application;

[0023] Figure 4A flowchart of a method for determining future returns of a recommendation model provided in an embodiment of the present application;

[0024] Figure 5 A flowchart of a method for determining future income adjustment amounts for a recommendation model provided in an embodiment of the present application;

[0025] Figure 6 A flowchart of a method for determining an estimated excitation flow value based on a recommended model provided in an embodiment of the present application;

[0026] Figure 7 A flowchart of a method for determining a total estimated excitation flow value provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of a method for determining a set of recommendation probabilities for customer groups in a recommendation model provided in an embodiment of the present application;

[0028] Figure 9 A flowchart of a recommendation effect evaluation model training method provided in an embodiment of the present application;

[0029] Figure 10 An overall architecture diagram of a recommendation model selection method provided in an embodiment of the present application;

[0030] Figure 11 A schematic diagram of the composition of a recommended model selection device provided in an embodiment of the present application;

[0031] Figure 12 A schematic diagram of the composition of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] It should be noted that in the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in the embodiments of this application as "exemplarily" or "for example" should not be interpreted as being more preferred or advantageous than other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0034] In order to facilitate a clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with basically the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0035] In the field of recommendations, driven by the pursuit of more efficient recommendations, a single online platform may deploy multiple recommendation models. These models may have different business focuses and algorithmic logic, enabling them to output diverse recommendations to meet the needs of diverse user groups. This improves the user experience and the platform's commercial conversion rate.

[0036] However, while the coexistence of multiple models enriches recommendation content, it also raises the issue of model selection. Existing technologies rely on online data as the basis for model selection. However, due to real-time requirements, the decision-making process of such solutions is relatively simple. For example, model selection is based on the scale of traffic generated by the recommendation model and the feedback data of the recommended content. This may result in the selected model failing to push content that is relevant to the user, resulting in wasted traffic. Therefore, improving the accuracy of recommendation model selection has become a pressing issue in the current recommendation field.

[0037] Based on this, this application provides a method for selecting a recommendation model. This method can segment traffic into different traffic channels and divide users into different customer groups based on their characteristics. It can also refine the model's adaptation scenarios, measure the model's adaptability from different dimensions, and improve the accuracy of recommendation model selection.

[0038] The recommendation model selection method provided in this application can be applied to computing devices on a network platform. The computing device can be a server cluster consisting of multiple servers, a single server, a computer, or a processor or processing chip in a server or computer. The embodiments of this application do not limit the specific form factor of the computing device.

[0039] In order to better understand the technical solutions provided by this application, the following professional terms are explained.

[0040] Racehorse mechanism: refers to a competitive selection mode. It puts multiple subjects of the same kind in the same competitive track, sets unified phased goals, assessment standards and time limits, and each subject carries out competition in a relatively fair environment. During this period, resource allocation, development trend, etc. are mainly controlled by each subject. Finally, the excellent performers are rewarded, supported or further promoted, and the underperformers may face adjustment or even elimination. In this application, the racehorse mechanism is used to apply multiple recommendation models to the same traffic channel. The traffic channel is a smaller traffic division range. By measuring the value indicators of multiple recommendation models in the traffic channel within a unit of time, the traffic distribution of multiple recommendation models is determined.

[0041] Click-through rate (CTR): refers to the proportion of users who click on the recommended content after the recommended content is displayed to the users. The formula is CTR = number of times of clicking on the recommended content / number of times of displaying the recommended content. The click-through rate directly reflects the attractiveness of the recommended content.

[0042] Conversion rate (CVR): refers to the proportion of users who complete the expected conversion behavior in the user group who click on the recommended content. The formula is CVR = number of users who complete conversion / number of users who click on the recommended content. The conversion rate measures the effectiveness of the second half of the recommended link and tests the conversion ability of the recommended content from interest stimulation to action promotion.

[0043] Average order value (AOV): refers to the average consumption amount per order of users who complete the expected conversion behavior. The calculation formula is AOV = total conversion sales / number of conversion orders. The average order value is related to the revenue and profit space of the recommendation business.

[0044] Normalization processing: a data preprocessing method that scales data to a fixed interval according to certain rules, commonly [0, 1] or [-1, 1]. The purpose is to eliminate the problem of different feature dimensions and large value range differences, so that each feature has an equal status in subsequent analysis and modeling, and avoids model training bias caused by different data scales.

[0045] Go receiver: Go language is known for its excellent concurrency performance. In the recommendation system, the go receiver can be used as a lightweight and efficient data receiving component. When massive recommendation interaction data (such as click and conversion information) is returned in real time, the go receiver can quickly receive data, avoid data accumulation and loss, and maintain the smoothness of the recycling link.

[0046] Netty Receiver: Netty is a high-performance network programming framework. In recommendation feedback scenarios, especially when transferring data across networks and processes, the Netty Receiver efficiently handles data inflows using protocols such as TCP and UDP. For high-concurrency and large-volume recommendation feedback, Netty leverages an asynchronous non-blocking I / O model to reduce thread blocking, improve overall receiving efficiency, and ensure data flows quickly from the network to subsequent processing.

[0047] Kafka: An open-source distributed stream processing platform. It offers high throughput, persistent storage, and partitioned fault tolerance. It's commonly used in decoupled systems and real-time data stream processing scenarios. Different applications can communicate asynchronously through Kafka, sending and receiving messages, acting as a data "transfer station."

[0048] ClickHouse: A column-based database management system for online analytical processing. It handles massive amounts of data with extremely fast read and write speeds, high data compression ratios, and efficient support for complex queries. It's suitable for data analysis, aggregation, and other operations involving large data volumes.

[0049] xxl-job: A distributed task scheduling platform. It can schedule various tasks across platforms and languages, such as scheduled tasks and data synchronization tasks. It provides a visual task management interface, allowing operations and developers to easily manage task execution and scheduling rules, ensuring that tasks in business processes are executed on time and as needed.

[0050] Software Development Kit (SDK): An SDK is a collection of software tools, documentation, sample code, and related resources designed to help developers more efficiently create applications for specific platforms, operating systems, frameworks, or hardware. For example, the JavaScript SDK, Vue SDK, and React SDK mentioned below are SDKs for specific front-end development frameworks, while the iOS SDK and Android SDK are SDKs for mobile operating system platforms. It's like a "treasure chest," housing commonly used code snippets, function libraries, debugging tools, and more during development.

[0051] Autoregressive Moving Average (ARMA): ARMA is a statistical model used for time series analysis and forecasting that combines two key components: autoregressive (AR) and moving average (MA). This model assumes that the time series data is stationary, meaning that the statistical properties of the data (such as mean and variance) do not change over time.

[0052] Autoregressive Integrated Moving Average (ARIMA) model: ARIMA is an extension of the ARMA model, specifically designed to handle non-stationary time series data. In real-world scenarios, many time series exhibit non-stationary characteristics such as trend and seasonality. ARIMA addresses this by introducing an integrated process.

[0053] eXtreme Gradient Boosting (XGBoost) algorithm: Based on the gradient boosting framework, it iteratively adds weak learners (typically decision trees) to build strong learners. Each iteration focuses on fitting the residuals of the previous model, gradually reducing the overall prediction error. It uses second-order derivative information to optimize the objective function and find the optimal split point. Compared with traditional gradient boosting algorithms that rely solely on first-order derivatives, it converges faster. For example, when predicting housing prices, the first model generates a preliminary estimate, and subsequent models correct the previous errors.

[0054] Logistic regression (LR) algorithm: LR is a generalized linear regression model used for classification problems. It uses a logistic function to map the continuous values ​​obtained from linear regression to the range [0, 1], thereby representing the probability of an event occurring.

[0055] Light Gradient Boosting Machine (LightGBM) algorithm: LightGBM is a machine learning algorithm based on the gradient boosting framework, known for its efficiency, speed, and accuracy. It continuously iteratively constructs multiple decision tree weak learners based on leaf node growth and combines them to form a powerful model for processing tasks such as classification, regression, and sorting. It is widely used in e-commerce recommendations, advertising click-through rate prediction, financial risk control, and medical image analysis.

[0056] Figure 1 The flow chart of a method for selecting a recommendation model provided in an embodiment of the present application is shown in FIG. The method for selecting a recommendation model provided in the present application can be applied to the above-mentioned computing devices, such as Figure 1 As shown, the specific steps include:

[0057] S101. Based on the recommendation effect evaluation parameters of each recommendation model in a historical unit time, determine the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model.

[0058] Among them, the recommendation effect evaluation parameters can be obtained through the user's interactive behavior data on the recommended content, and the effect evaluation parameters include at least: click-through rate, conversion rate, and average order value. The channel recommendation probability set includes multiple channel recommendation probabilities that correspond one-to-one between the recommendation model and multiple traffic channels. The channel recommendation probability is used to represent the probability that the corresponding recommendation model will be allocated traffic for recommendation in a traffic channel. Each of the multiple traffic channels is divided into multiple customer groups. The customer group recommendation probability set includes multiple customer group recommendation probabilities that correspond one-to-one to the multiple customer groups. The customer group recommendation probability is used to represent the probability that the corresponding recommendation model will be allocated traffic for recommendation in a customer group.

[0059] For example, taking the first recommendation model as an example, the channel recommendation probability set corresponding to the first recommendation model can be expressed as x 1,i =[x 1,1 ,x 1,2 ,x 1,3 ], where x 1,1 represents the probability that the first recommendation model is assigned traffic for recommendation in the first traffic channel, x 1,2 represents the probability that the first recommendation model is assigned traffic for recommendation in the second traffic channel, x 1,3 Indicates the probability that the first recommendation model is assigned traffic for recommendation in the third traffic channel.

[0060] For another example, taking the second recommendation model as an example, the channel recommendation probability set corresponding to the second recommendation model can be expressed as x 2,i =[x 2,1 ,x 2,2 ,x 2,3 ], where x 2,1 represents the probability that the second recommendation model is assigned traffic for recommendation in the first traffic channel, x 2,2 represents the probability that the second recommendation model is assigned traffic for recommendation in the second traffic channel, x 2,3 It represents the probability that the second recommendation model is assigned traffic for recommendation in the third traffic channel.

[0061] For another example, taking the first recommendation model as an example, the customer group recommendation probability set corresponding to the first recommendation model can be expressed as p 1,i =[p 1,1 ,p 1,2 ,p 1,3 ], where p 1,1 It represents the probability that the first recommendation model is assigned traffic for recommendation in the first customer group, p 1,2 It represents the probability that the first recommendation model is allocated traffic for recommendation in the second customer group, p 1,3 Indicates the probability that the first recommendation model is allocated traffic for recommendation in the third customer group.

[0062] For another example, taking the second recommendation model as an example, the customer group recommendation probability set corresponding to the first recommendation model can be expressed as p 2,i =[p 2,1 ,p 2,2 ,p 2,3 ], where p 2,1 It represents the probability that the second recommendation model is allocated traffic for recommendation in the first customer group, p 2,2 It represents the probability that the second recommendation model is allocated traffic for recommendation in the second customer group, p 2,3 It represents the probability that the second recommendation model is allocated traffic for recommendation in the third customer group.

[0063] It should be understood that obtaining the channel recommendation probability set for each recommendation model can help determine the degree of dominance of each recommendation model in different traffic channels. Models with higher dominance tend to generate higher online revenue. Furthermore, obtaining the customer group recommendation probability set for each recommendation model in different customer groups can help determine the degree of suitability of each recommendation model for these different customer groups. Recommendation models with higher suitability can deliver recommended content that better meets user needs, thereby improving traffic utilization.

[0064] One possible implementation involves converting the platform user's mobile phone number or platform user ID into a hexadecimal number. The last two digits of the hexadecimal number are then used as a classification code. Furthermore, user traffic with the same classification code is grouped into a single traffic channel. It should be noted that there are 100 possible combinations of the last two digits of a decimal number, while there are 256 possible combinations of the last two digits of a hexadecimal number. Converting decimal numbers to hexadecimal numbers can expand the number of traffic channels and refine the selection range for recommendation models. Furthermore, the hexadecimal conversion can conceal the numeric characteristics of the mobile phone number, making the classification more equitable.

[0065] In another possible implementation, the computing device can divide customer groups based on user information characteristics. For example, within the user's age range, using every 10 years as a division interval, this can form a total of nine age intervals: [≤10, 11-20, 21-30, 31-40, 41-50, 51-60, 61-70, 71-80, ≥80]. Furthermore, users are divided into two categories based on gender. Combining the nine age intervals with the two gender categories, the total number of customer groups is 9 × 2 = 18.

[0066] S102 : Determine a comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model.

[0067] Among them, the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group in different traffic channels of the corresponding recommendation model.

[0068] It should be understood that the comprehensive recommendation probability is derived by combining the channel recommendation probability and customer group recommendation probability of the recommendation model. The channel recommendation probability considers the online performance advantage of the recommendation model for the traffic channel, while the customer group recommendation probability considers the degree of adaptability of the recommendation model to the customer group. The comprehensive recommendation probability combines these two aspects of the recommendation model to comprehensively evaluate the competitiveness of the recommendation model from different dimensions.

[0069] In some embodiments, the comprehensive recommendation probability of the recommendation model may be obtained by multiplying each channel recommendation probability in its channel recommendation probability set by each customer group recommendation probability in its customer group recommendation probability set.

[0070] For example, taking the first recommendation model as an example, the channel recommendation probability set corresponding to the first recommendation model is expressed as x 1,i =[x 1,1 ,x 1,2 ,x 1,3 ], the customer group recommendation probability set corresponding to the first recommendation model is expressed as p 1,i =[p 1,1 ,p 1,2 ,p 1,3 ]. Then the comprehensive recommendation probability set of the first recommendation model can be y1=[x 1,1 p 1,1 ,x 1,1 p 1,2 ,x 1,1 p 1,3 ,x 1, 2p 1,1 ,x 1,2 p 1,2 ,x 1,2 p 1,3 ,x 1,3 p 1,1 ,x 1,3 p 1,2 ,x 1,3 p 1,3 ], where x 1,1 p 1,1 represents the probability that the first recommendation model is assigned traffic for recommendation in the first customer group under the first traffic channel, x 1,1 p 1,2 represents the probability that the first recommendation model is assigned traffic for recommendation in the second customer group under the first traffic channel, x 1,1 p 1,3 represents the probability that the first recommendation model is assigned traffic for recommendation in the third customer group under the first traffic channel, x 1,2p 1,1 It represents the probability that the first recommendation model is allocated traffic for recommendation in the third customer group under the second traffic channel. By analogy, we can obtain the comprehensive recommendation probability set of multiple customer groups corresponding to the recommendation model under multiple traffic channels.

[0071] S103 . For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determine a target recommendation model corresponding to the target customer group from multiple recommendation models.

[0072] Among them, the target traffic channel is one of the multiple traffic channels, the target customer group is one of the multiple customer groups of the target traffic channel, and the target recommendation model recommends the target customer group in the target traffic channel within the future unit time.

[0073] In one possible implementation, the computing device may directly select the recommendation model with the highest comprehensive recommendation probability for the target customer group in the target traffic channel from multiple recommendation models as the target recommendation model based on the comprehensive recommendation probability set corresponding to each recommendation model.

[0074] For example, taking the comprehensive recommendation probability set y1 of the first recommendation model as an example, if the target traffic channel is the first traffic channel and the target customer group is the first customer group in the first traffic channel, the computing device can compare the comprehensive recommendation probabilities for the first customer group in the first traffic channel in the comprehensive recommendation probability sets corresponding to different recommendation models (for example, the comprehensive recommendation probability x1 of the first recommendation model for the first customer group in the first traffic channel is 1,1 p 1,1 , the comprehensive recommendation probability x of the second recommendation model for the first customer group in the first traffic channel 2,1 p 2,1 , the comprehensive recommendation probability x of the third recommendation model for the first customer group in the first traffic channel 3,1 p 3,1 ), and select the recommendation model with the highest comprehensive recommendation probability as the target recommendation model.

[0075] In another possible implementation method, the computing device can also normalize the comprehensive recommendation probabilities of the target customer groups under the target traffic channel in different comprehensive recommendation probability sets according to the comprehensive recommendation probability sets corresponding to each recommendation model, and select the recommendation model with the largest normalization result as the target recommendation model.

[0076] For example, the calculation process of the computing device normalizing the comprehensive recommendation probability can be expressed as:

[0077]

[0078] in:

[0079] y i,k,j It represents the normalized result of the comprehensive recommendation probability of the j-th recommendation model in the k-th customer group under the i-th traffic channel.

[0080] x i,j It represents the channel recommendation probability of the j-th recommendation model under the i-th traffic channel.

[0081] p i,k,j It represents the customer recommendation probability of the j-th recommendation model in the k-th customer group under the i-th traffic channel.

[0082] It represents the exponential power operation result of the product of the channel recommendation probability of the j-th recommendation model in the i-th traffic channel and the recommendation probability of the k-th customer group in the j-th recommendation model in the i-th traffic channel.

[0083] The sum of the exponential power calculation results of the product of the channel recommendation probability and the customer group recommendation probability corresponding to all recommendation models in the k-th customer group under the i-th traffic channel, where n is the total number of recommendation models.

[0084] It should be noted that each comprehensive recommendation probability in a recommendation model's set represents the probability of that model being suitable for a specific customer segment within a specific traffic channel. Determining the target recommendation model for a target customer segment from multiple recommendation models simply involves comparing the comprehensive recommendation probabilities of the multiple recommendation models for the target customer segment and selecting the model with the highest comprehensive recommendation probability as the recommendation model for that target customer segment.

[0085] It should also be noted that the target recommendation model is a recommendation model that recommends the target customer group in the target traffic channel within the future unit time. It can be allocated traffic to recommend to the target customer group within the future unit time. Moreover, at the end of the future unit time, steps S101-S103 of this application can be executed again to determine the recommendation model for the target customer group in the target traffic channel within the next future unit time.

[0086] For example, the unit duration in the future unit duration can be a shorter period of time such as 3 minutes, 5 minutes or 10 minutes. The shorter unit duration can give the selected recommendation model a certain timeliness advantage, and can reduce the risk of recommended content lagging behind and being out of touch with the user's current interests due to time difference.

[0087] A recommendation model selection method provided in an embodiment of the present application can refine the platform traffic into multiple traffic channels, and determine the channel recommendation probability of the recommendation model in each traffic channel through a horse racing mechanism, so as to screen out recommendation models with excellent online performance in the same traffic channel. Furthermore, among the customer groups divided under the traffic channel, by evaluating the customer group recommendation probabilities of multiple recommendation models in the same customer group, the recommendation model that best suits the customer group can be identified. This method can comprehensively evaluate the potential value of the recommendation model based on multiple factors, improve the accuracy of the recommendation model selection, and thus give full play to the value of traffic.

[0088] The following is a detailed introduction to the process of obtaining the recommendation effect evaluation parameters in the above step S101.

[0089] In some embodiments, as Figure 2 As shown, the process of obtaining the recommendation effect evaluation parameters in step S101 further includes the following steps before step S101:

[0090] S201: Data collection.

[0091] Specifically, data collection is responsible for collecting raw behavioral data from front-end users. Raw behavioral data can be the user's interactive actions on the recommended content of the recommendation model. Raw actions include at least: click, favorite, follow, share, comment, add to cart, order, etc.

[0092] like Figure 2 As shown, data collection can be achieved by integrating with SDKs. Currently supported integration methods include JavaScript SDK, Vue SDK, iOS SDK, Android SDK, and React SDK. These SDKs can be embedded in visual interfaces such as web pages, apps, and mini-programs to collect raw user behavior data, encapsulate it in a standard data format, and upload it to the data receiver.

[0093] S202: Data reception.

[0094] Specifically, data reception is responsible for preprocessing and forwarding the raw behavioral data uploaded by the frontend. Preprocessing verifies the validity of the uploaded data and cleans it, removing invalid and abnormal data. Forwarding transmits the preprocessed data to data processing.

[0095] like Figure 2 As shown, data receivers can include Go receivers and Netty receivers. The Go receiver has efficient concurrent processing capabilities, while the Netty receiver has rich functions, supports multiple network protocols and data formats, and can flexibly cope with various complex network environments.

[0096] S203: Data transmission.

[0097] Specifically, data transmission is used to transmit raw behavioral data from the receiver. First, data transmission plans the transmission path from data reception to data processing based on the business logic and flow of the data. Second, the received data is pushed to the transmission channel in sequence according to the established plan to ensure orderly transmission.

[0098] like Figure 2 As shown, Kafka can be used as a high-speed transmission channel for user behavior data, which can effectively ensure the timeliness of the data.

[0099] S204: Data processing.

[0100] Specifically, data processing is used to store and analyze raw behavioral data.

[0101] like Figure 2 As shown, for storage, the ClickHouse column-based database is used to reduce redundancy while ensuring data integrity and traceability. For analysis, materialized views are constructed based on raw behavioral data to analyze recommendation effectiveness evaluation parameters such as click-through rate, conversion rate, and average order value, providing rapid feedback on current model performance. Furthermore, xxl-job is used to process dimension table data, transforming this data, which supplements the attributes and dimensions of the main data, into a usable state for precise analysis and in-depth insights, assisting in the final development of various indicators.

[0102] The following describes the specific process of determining the channel probability set corresponding to each recommendation model in step S101.

[0103] In some embodiments, as Figure 3 As shown, taking any first recommendation model among multiple recommendation models as an example, the step of determining, by the computing device in step S101, a channel recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model within a historical unit time period among the multiple recommendation models may specifically include:

[0104] S301. For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time of the traffic channel, determine the future benefit of the first recommendation model in the traffic channel and the historical reference benefit of the traffic channel.

[0105] Among them, future profits are determined based on the profits of the first recommendation model within the historical unit time, the historical reference profits are the sum of the profits of multiple models within the historical unit time, and the profits are determined based on the recommendation effect evaluation parameters of the recommendation model.

[0106] It should be noted that, as mentioned above, the unit durations used in this application are relatively short. This feature ensures that the revenue data of the recommendation model within two adjacent unit durations has a certain degree of consistency and stability. At a certain time point, the revenue of the past adjacent historical unit durations at that time point can be roughly equivalent to the revenue of the future unit duration at that time point.

[0107] S302 : For each traffic channel, normalize the future revenue of the traffic channel and the historical reference revenue of the traffic channel according to the first recommendation model to obtain a channel recommendation probability corresponding to the traffic channel.

[0108] It should be understood that the computing device normalizes the historical returns of multiple recommendation models and can calibrate the historical returns of different recommendation models to the same scale for comparison, which can more intuitively understand the performance of the first recommendation model among multiple recommendation models.

[0109] In some embodiments, the normalization process may perform an exponential power operation on the historical returns of multiple recommendation models under the traffic channel, and divide the exponential power operation result of the historical returns of the first recommendation model by the sum of the exponential power operation results of the historical returns of multiple recommendation models to obtain the channel recommendation probability of the first recommendation model.

[0110] For example, the calculation process of the channel recommendation probability can be expressed as:

[0111]

[0112] in:

[0113] C represents the exponential power of the benefits of the j-th recommendation model in the i-th traffic channel within the future unit time t. i,t,j Represents the basic revenue of the j-th recommendation model in the future unit time t under the i-th traffic channel, and the revenue adjustment amount α i,t,j It represents the revenue adjustment of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0114] It represents the exponential power result of all recommendation model benefits in the future unit time t under the i-th traffic channel.

[0115] S303 : Determine a channel recommendation probability set corresponding to the first recommendation model based on the multiple channel recommendation probabilities corresponding to the multiple traffic channels.

[0116] Specifically, the computing device can obtain multiple channel recommendation probabilities by calculating the channel recommendation probabilities of the first recommendation model under different traffic channels, and aggregate the multiple channel recommendation probabilities to obtain a channel recommendation probability set of the first recommendation model.

[0117] Steps S301-S303 above provide a method for determining a channel recommendation probability set for a recommendation model, as provided in an embodiment of this application. This method calculates the channel probability set for a recommendation model by calculating the online performance of the recommendation model. The better the online performance of the recommendation model, the greater its probability of obtaining channel traffic. This method strengthens the correlation between traffic distribution and recommendation effectiveness, effectively stimulating competition among recommendation models and thus implementing a horse racing mechanism for recommendation model selection.

[0118] The following describes the specific process of determining the future revenue of the first recommendation model in the traffic channel in step S301.

[0119] In some embodiments, as Figure 4 As shown, taking any one of the multiple recommendation models as an example, the step of determining the future benefit of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameter of the first recommendation model in the historical unit time of the traffic channel in step S301 may specifically include:

[0120] S401. For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time of the traffic channel, determine the basic revenue of the first recommendation model within the historical unit time of the traffic channel.

[0121] In some embodiments, the recommendation effect evaluation parameters of each recommendation model within the historical unit time of multiple traffic channels can be obtained by online recycling and processing the user's original behavior data.

[0122] For example, the specific calculation process of basic income can be expressed as follows:

[0123] C i,t,j =CTR i,t,j ×CVR i,t,j ×AOV i,t,j Formula (3)

[0124] in:

[0125] C i,t,j It represents the basic revenue of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0126] CTR i,t,j It represents the click-through rate of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0127] CVR i,t,j It represents the conversion rate of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0128] AOVi,t,j represents the single-user price of the jth recommendation model in the ith traffic channel in the future unit time t.

[0129] It should be noted that, according to the foregoing step S301, the recommendation effect evaluation parameter of the future unit time t in the above formula is actually the recommendation effect evaluation parameter of the adjacent historical unit time.

[0130] S402, according to the basic income of the first recommendation model in the historical unit time of the traffic channel, and the income adjustment amount of the first recommendation model in the historical unit time of the traffic channel, determine the future income of the first recommendation model in the traffic channel.

[0131] It should be understood that the basic income represents the actual online income of the recommendation model in the historical unit time of the traffic channel. The income adjustment amount also considers the income fluctuation of the recommendation model in the entire historical time due to traffic difference, and thus the potential value of the recommendation model can be mined. By combining the basic income and the income adjustment amount, the future income of the recommendation model can be effectively evaluated.

[0132] The specific acquisition process of the income adjustment amount of the first recommendation model in the historical unit time of the traffic channel in the above step S402 is introduced as follows.

[0133] In some embodiments, as shown in Figure 5 For any one of the traffic channels and any one of the recommendation models, the determination step of the income adjustment amount of the first recommendation model in the future unit time of the traffic channel in step S402 can specifically include:

[0134] S501, according to the difference ratio between the estimated stimulated traffic value of the first recommendation model in the future unit time and the actual stimulated traffic value in the historical unit time, determine the proportion coefficient.

[0135] It should be noted that the traffic difference between the estimated stimulated traffic value of the first recommendation model in the future unit time and the actual stimulated traffic value in the historical unit time is calculated, which represents the traffic value that should be allocated to the first recommendation model but is not allocated. The ratio of the traffic difference to the estimated stimulated traffic value is calculated to obtain the proportion coefficient, which represents the proportion of the traffic difference in the estimated stimulated traffic value.

[0136] In some embodiments, after the proportion coefficient is determined, the proportion coefficient can be multiplied by the step parameter of the recommendation model, and the step parameter is used to control the iteration amplitude of the model. This method can consider the influence of the recommendation model iteration update while calculating the traffic difference.

[0137] S502. Determine a reference adjustment amount based on the product of the basic income of the first recommendation model within a historical unit time period and the proportional coefficient.

[0138] Specifically, the reference adjustment amount can be understood as the revenue fluctuation caused by the aforementioned traffic difference.

[0139] S503: Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel according to the sum of the basic revenue adjustment amount and the reference adjustment amount of the first recommendation model in the traffic channel.

[0140] For example, the calculation process of the revenue adjustment amount of the recommendation model in the future unit time can be expressed as:

[0141]

[0142] in:

[0143] I i,t,j It represents the estimated stimulated traffic value of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0144] ∑x i,t-1,j It represents the total traffic obtained by the j-th recommendation model in the i-th traffic channel within the historical unit time t-1.

[0145] C t,i,j It represents the basic revenue of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0146] α i,t-1,j It represents the revenue adjustment of the j-th recommendation model in the i-th traffic channel within the historical unit time t-1.

[0147] α i,t,j It represents the revenue adjustment of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0148] In some embodiments, as Figure 6 As shown, taking any one of the multiple traffic channels and any one of the multiple recommendation models as an example, the method for determining the estimated stimulated traffic value per unit time of the first recommendation model in step S501 may specifically include:

[0149] S601: Obtain historical traffic sequences of multiple recommendation models.

[0150] The historical traffic sequence includes historical stimulated traffic values ​​corresponding to multiple historical unit time lengths. The historical stimulated traffic value is the total traffic value stimulated by recommendations from multiple recommendation models within the corresponding historical unit time length.

[0151] S602: Based on the historical traffic sequence and the traffic prediction model, predict the total estimated stimulated traffic value stimulated by recommendations from multiple recommendation models within a future unit time period.

[0152] S603: Determine the estimated stimulating traffic value per unit time of the recommendation model in the future according to the product of the total estimated stimulating traffic value and the historical revenue ratio of the first recommendation model.

[0153] Among them, the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit time of the traffic channel to the total basic revenue of multiple recommendation models in the historical unit time of the traffic channel.

[0154] For example, the calculation process of the estimated excitation flow value can be expressed as:

[0155]

[0156] in:

[0157] I i,t,j It represents the estimated stimulated traffic value of the j-th recommendation model in the i-th traffic channel within the future unit time t.

[0158] V i,t It represents the total estimated traffic value stimulated by the recommendations of multiple recommendation models in the future unit time t under the i-th traffic channel.

[0159] C i,t,j It represents the basic revenue of the j-th recommendation model in the future unit time t under the i-th traffic channel.

[0160] It represents the sum of the basic benefits of multiple recommendation models in the future unit time t under the i-th traffic channel.

[0161] In some embodiments, as Figure 7 As shown, the traffic prediction model in step S602 may be an ARMA model, and the method for constructing the model may specifically include:

[0162] S701: Perform differential processing on the historical traffic sequence.

[0163] It should be noted that historical traffic series may be non-stationary, that is, its statistical characteristics such as mean, variance or autocovariance will change over time. The ARMA model requires that the time series data is stationary, so the historical traffic series needs to be differentiated. Difference is to use the historical excitation traffic value of the current historical unit time minus the historical excitation traffic value of the previous historical unit time (first-order difference), or the historical excitation traffic value of the current historical unit time minus the historical excitation traffic value of several previous historical unit time (higher-order difference). In this way, non-stationary factors such as trend and seasonality in the data are removed, making it stable for subsequent modeling.

[0164] S702: Draw a fixed-order heat map based on the historical traffic sequence.

[0165] It should be noted that the fixed-order heat map is a visualization tool used to determine the combined effects of different values ​​of the autoregressive order p and the moving average order q in the ARMA model.

[0166] Specifically, a maximum order L is determined based on the length or time characteristics of the historical traffic sequence, where L represents the maximum length of the historical traffic sequence that can be used. Based on the maximum order L, the value range of p and q is determined, that is, (0, L]. Within the value range, the value combinations of p and q are traversed, and the Akaike Information Criterion (AIC) value of each combination is calculated. The AIC values ​​for different combinations of p and q values ​​are displayed on a two-dimensional plane (the horizontal axis is p, the vertical axis is q), and the size of the indicator is represented by the depth of the color.

[0167] It should be noted that the autoregressive order p represents the number of historical traffic series used in the autoregressive part of the model. Similarly, the sliding average order q represents the number of historical traffic series used in the sliding average part of the model.

[0168] S703: Determine the order of ARMA based on the fixed-order heat map.

[0169] Specifically, the p-value and q-value of the darkest area in the fixed-order heat map are retrieved, and the combination of the p and q values ​​is used as the order of the ARMA.

[0170] It should be noted that the smaller the AIC value in the fixed-order heatmap, the better the model fits the data and the lower the model complexity. The order of the ARMA is determined based on the p-value and q-value corresponding to the darkest area in the fixed-order heatmap.

[0171] S704. Perform residual normality test and residual independence test on the ARMA model.

[0172] Specifically, ARMA modeling is performed based on the determined values ​​of p and q, and the model is tested using residual normality test and residual independence test.

[0173] It should be noted that the purpose of the residual normality test is to check whether the residuals after fitting the ARMA model follow a normal distribution. Because the model assumes that the residuals are a normally distributed white noise sequence, if normality is not satisfied, it may mean that there is a problem with the model and it cannot fit the data well.

[0174] It is also important to note that the purpose of the residual independence test is to determine whether the residuals are independent of each other, that is, whether there is autocorrelation in the residual series. If there is autocorrelation in the residuals, it means that the model does not fully capture the information in the data, and the model may need to be re-adjusted.

[0175] In some embodiments, if the model constructed by the above p and q cannot satisfy the residual normality test or the residual independence test, it is necessary to re-execute steps S701-S703 to determine the values ​​of p and q again.

[0176] S705 : Predict the total estimated excitation flow value within a future unit time period using the ARMA model.

[0177] Specifically, an ARMA model that satisfies both the residual normality test and the residual independence test is used to predict future traffic excitation values.

[0178] In other embodiments, the traffic prediction model in step S602 may also be an ARIMA model.

[0179] The following describes the specific process of determining the customer group recommendation probability set corresponding to each recommendation model in step S101.

[0180] In some embodiments, as Figure 8 As shown, taking any first recommendation model among the multiple recommendation models as an example, the step of determining the customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time in step S101 by the computing device may further include:

[0181] S801. For each customer group, determine an inference sample of the first recommendation model based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time.

[0182] Specifically, the inference sample includes historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and model hyperparameters of the first recommendation model.

[0183] In some embodiments, the historical recommendation effect evaluation parameters may include at least any one of the following: click-through rate, conversion rate, and average order value.

[0184] In some embodiments, the model hyperparameters of the first recommendation model include at least: category of the recommendation model, sample size, number of features, learning rate, and number of batch samples.

[0185] S802 : For each customer group, based on the inference samples of the first recommendation model and the recommendation effect evaluation model, predict future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group.

[0186] Specifically, the recommendation effect evaluation model is used to predict future recommendation effect evaluation parameters based on historical recommendation effect evaluation parameters and model hyperparameters.

[0187] It should be understood that the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, as well as the model hyperparameters of the first recommendation model. Using the inference sample as the input of the recommendation effect evaluation model can enable the recommendation effect evaluation model to understand the internal logic of the first recommendation model and the performance effect of the recommended content, thereby allowing the recommendation effect evaluation model to predict future recommendation effect evaluation parameters that are more in line with the actual performance of the recommendation model.

[0188] S803. For each customer group, determine the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameter of the first recommendation model in the total future recommendation effect evaluation parameters of all recommendation models.

[0189] It should be noted that after predicting the future recommendation effect evaluation parameters of the first recommendation model, it is also necessary to compare them with the future recommendation effect evaluation parameters of other recommendation models under the same customer group. By measuring the proportion of the future recommendation effect evaluation parameters of the first recommendation model in the total future recommendation effect evaluation parameters of all recommendation models, the degree of advantage of the first recommendation model over other recommendation models under the same customer group can be determined.

[0190] S804 : Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, obtain a customer group recommendation probability set corresponding to the first recommendation model.

[0191] It should be understood that the channel recommendation probability set of the recommendation model calculated in the aforementioned steps S201-S203 is calculated from the perspective of the online revenue of the recommendation model, while steps S801-803 are calculated from the perspective of the customer group, taking into account the degree of adaptation of the recommendation model to the customer group. Under the same customer group, a recommendation model with a higher customer recommendation probability has greater potential than other recommendation models to push content that meets user needs.

[0192] In some embodiments, as Figure 9 As shown, the process of obtaining the recommendation effect evaluation model in step S802 further includes the following steps before step S802:

[0193] S901: Obtain a training sample set.

[0194] Specifically, the training sample set includes multiple training samples, each training sample includes the recommendation effect evaluation parameters when the recommendation model makes recommendations to a customer group in the first time period, the model hyperparameters of the recommendation model, and the recommendation effect evaluation parameter labels when the recommendation model makes recommendations to the customer group in the second time period after the first time period.

[0195] It should be noted that the recommendation effect evaluation parameters when making recommendations for a customer group recommendation model in the first period or the second period are all obtained from the historical recommendation effect evaluation parameters of the recommendation model.

[0196] In some embodiments, the historical recommendation effect evaluation parameters of the recommendation model can be obtained by executing the aforementioned steps S401-S404, which will not be repeated here.

[0197] S902: Train the preset initial model based on the training sample set to obtain a recommendation effect evaluation model.

[0198] For example, taking the process of training the initial model using a training sample as an example, the recommendation effect evaluation parameters when making recommendations to a customer group in the first time period and the model hyperparameters of the recommendation model are input into the initial model, and the recommendation effect evaluation parameters when making recommendations to the customer group in the second time period after the first time period are used as the true label. The recommendation effect evaluation parameters when the initial model makes recommendations to the customer group in the second time period after the first time period are predicted, and after obtaining the output of the initial model, the true value deviation between the output content and the true label is measured using the true label. The initial model is adjusted according to the true deviation, and the next round of training is carried out using the next training sample.

[0199] In some embodiments, the recommendation effect evaluation model can be trained using the XGBoost algorithm.

[0200] In other embodiments, the LR algorithm or the LightGBM algorithm may also be used to train the recommendation effect evaluation model.

[0201] It should be understood that steps S901-S902 train the preset model through the model hyperparameters of multiple recommendation models and the recommendation effect evaluation parameters within the historical unit time. It can adaptively learn the customer group that each recommendation model is suitable for through artificial intelligence algorithms, and because of the continuous influx of data from the online platform, it will also effectively improve the prediction accuracy of the recommendation effect evaluation model. Figure 10This is a diagram showing the overall architecture of a recommendation model selection method provided in an embodiment of the present application. Figure 10 As shown, where:

[0202] Online effect recovery. The computing device collects the online effects of multiple recommendation models within the current historical unit time and processes them into recommendation effect evaluation parameters as the data source for subsequent calculations.

[0203] Obtain a set of channel recommendation probabilities corresponding to each recommendation model. The computing device calculates the future benefits of each recommendation model for each traffic channel based on the recommendation effectiveness evaluation parameters of each recommendation model within a historical unit of time, and then calculates the channel recommendation probability for each recommendation model. The channel recommendation probabilities corresponding to multiple traffic channels for each recommendation model are aggregated to form a set of channel recommendation probabilities for that recommendation model.

[0204] Obtain a set of customer group recommendation probabilities corresponding to each recommendation model. The computing device uses the recommendation effect evaluation parameter model to predict the future effect evaluation parameters of each recommendation model in each customer group, and then calculates a set of customer group recommendation probabilities for each recommendation model in different customer groups.

[0205] Based on the channel recommendation probability set and customer group recommendation probability set of each recommendation model, the comprehensive recommendation probability set of each recommendation model is calculated. The calculation process is as described in step S102 above and will not be repeated here.

[0206] The target recommendation model is determined based on the combined recommendation probabilities of multiple recommendation models for the target customer segment within the target traffic channel. Each recommendation model's combined recommendation probability corresponds to a customer segment within the target traffic channel. When calculating the target recommendation model for the target customer segment within the target traffic channel, the combined recommendation probabilities of multiple recommendation models for that target customer segment are compared, and the model with the highest combined recommendation probability is selected as the recommendation model for that target customer segment.

[0207] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. It should be easy to realize that the technical goals in this field are combined with the units and algorithm steps of each example described in the embodiments disclosed herein, and the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional technical goals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0208] In an exemplary embodiment, the present application also provides a recommendation model selection device, which can be applied to the above computing device. Figure 11 As shown, the apparatus comprises an acquisition module 110 and a processing module 120.

[0209] The acquisition module 110 is configured to determine, based on the recommendation effect evaluation parameters of each recommendation model in the plurality of recommendation models within a historical unit time length, a channel recommendation probability set and a customer group recommendation probability set corresponding to each recommendation model respectively. The channel recommendation probability set comprises a plurality of channel recommendation probabilities corresponding to a plurality of traffic channels one by one, and the channel recommendation probability is used to represent the probability of the corresponding recommendation model being allocated traffic for recommendation in a traffic channel. Each of the plurality of traffic channels is divided into a plurality of customer groups, and the customer group recommendation probability set comprises a plurality of customer group recommendation probabilities corresponding to the plurality of customer groups one by one, and the customer group recommendation probability is used to represent the probability of the corresponding recommendation model being allocated traffic for recommendation in a customer group.

[0210] The processing module 120 is configured to determine, based on the channel recommendation probability set and the customer group recommendation probability set corresponding to each recommendation model respectively, a comprehensive recommendation probability set corresponding to each recommendation model respectively, and the comprehensive recommendation probability set comprises a comprehensive recommendation probability of the corresponding recommendation model in each customer group in different traffic channels. For a target customer group in a target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model respectively, a target recommendation model corresponding to the target customer group is determined from the plurality of recommendation models, the target traffic channel is one of the plurality of traffic channels, the target customer group is one of the plurality of customer groups of the target traffic channel, and the target recommendation model recommends the target customer group in the target traffic channel within a future unit time length.

[0211] In a possible implementation, the acquisition module 110 is specifically configured to, for any first recommendation model in the plurality of recommendation models, determine, based on the recommendation effect evaluation parameters of each recommendation model in the plurality of recommendation models within a historical unit time length, a channel recommendation probability set corresponding to each recommendation model respectively, comprising: for each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time length in the traffic channel, determining the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel. The future revenue is determined based on the revenue of the first recommendation model within the historical unit time length, the historical reference revenue is the sum of the revenues of the plurality of models within the historical unit time length, and the revenue is determined based on the recommendation effect evaluation parameter of the recommendation model. For each traffic channel, the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel are normalized to obtain the channel recommendation probability corresponding to the traffic channel. Based on the plurality of channel recommendation probabilities corresponding to the plurality of traffic channels respectively, the channel recommendation probability set corresponding to the first recommendation model is determined.

[0212] In another possible implementation, the processing module 110 is specifically configured to, for each traffic channel, determine the future revenue of the first recommendation model in the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel, including: for each traffic channel, determining the basic revenue of the first recommendation model in the historical unit time of the traffic channel based on the recommendation effect evaluation parameters of the first recommendation model in the historical unit time of the traffic channel. Determine the future revenue of the first recommendation model in the traffic channel based on the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

[0213] In another possible implementation, the processing module 120 is specifically used to determine the revenue adjustment amount, including: determining a proportional coefficient based on the ratio of the difference between the estimated excitation flow value of the first recommendation model in the future unit time and the actual excitation flow value of the historical unit time. Determine a reference adjustment amount based on the product of the basic revenue of the first recommendation model in the historical unit time and the proportional coefficient. Determine the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel based on the sum of the basic revenue adjustment amount of the first recommendation model in the traffic channel and the reference adjustment amount, where the basic revenue adjustment amount is the revenue adjustment amount of the first recommendation model in the historical unit time.

[0214] In another possible implementation, the processing module 120 is specifically used to determine the estimated stimulated traffic value of the first recommendation model for the future unit duration, including: obtaining historical traffic sequences of multiple recommendation models, the historical traffic sequences including historical stimulated traffic values ​​corresponding to multiple historical unit durations, and the historical stimulated traffic values ​​are the total traffic values ​​stimulated by recommendations made by multiple recommendation models within the corresponding historical unit durations. Based on the historical traffic sequences and the traffic prediction model, the total estimated stimulated traffic value stimulated by recommendations made by multiple recommendation models within the future unit durations is predicted. The estimated stimulated traffic value of the recommendation model for the future unit duration is determined based on the product of the total estimated stimulated traffic value and the historical revenue ratio of the first recommendation model, and the historical revenue ratio is the ratio of the basic revenue of the first recommendation model within the historical unit duration of the traffic channel to the total basic revenue of multiple recommendation models within the historical unit duration of the traffic channel.

[0215] In another possible implementation, the acquisition module 110 is specifically used to, for any first recommendation model among the multiple recommendation models, determine the customer group recommendation probability set corresponding to each recommendation model in the multiple recommendation models based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time length for each customer group, wherein the inference sample of the first recommendation model is determined, and the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model. For each customer group, based on the inference sample of the first recommendation model and the recommendation effect evaluation model, predict the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the recommendation effect evaluation model is used to predict the future recommendation effect evaluation parameters based on the historical recommendation effect evaluation parameters and the model hyperparameters. For each customer group, determine the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameters of the first recommendation model in the total of the future recommendation effect evaluation parameters of all recommendation models. Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, a customer group recommendation probability set corresponding to the first recommendation model is obtained.

[0216] In another possible implementation, the processing module 120 is further configured to obtain a training sample set, where the training sample set includes multiple training samples, each of which includes a recommendation effect evaluation parameter when the recommendation model makes recommendations for a customer group in a first time period, a model hyperparameter of the recommendation model, and a recommendation effect evaluation parameter label when the recommendation model makes recommendations for the customer group in a second time period after the first time period. A preset initial model is trained based on the training sample set to obtain a recommendation effect evaluation model.

[0217] It should be noted that Figure 11 The module division described is illustrative and represents only one logical functional division. Actual implementations may employ different divisions. For example, two or more functions may be integrated into a single processing module. These integrated modules may be implemented as either hardware or software functional modules.

[0218] In an exemplary embodiment, as described above, the computing device may be a computer or a server or other electronic device with computing and processing functions. In this case, the present application also provides an electronic device, Figure 12 This is a schematic diagram of the composition of an electronic device provided in an embodiment of the present application. Figure 12 As shown, the electronic device includes: a processor 10 , a memory 20 , a communication line 30 , a communication interface 40 , and an input / output interface 50 .

[0219] The processor 10 , the memory 20 , the communication interface 40 , and the input / output interface 50 may be connected via a communication line 30 .

[0220] The processor 10 is used to execute the instructions stored in the memory 20 to implement the recommended model selection method provided in the above embodiment of the present application. The processor 10 can be a CPU, a general-purpose processor network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / single-chip microcomputer, a programmable logic device (PLD) or any combination thereof. The processor 10 can also be any other device with processing functions, such as a circuit, a device or a software module, which is not limited in the embodiment of the present application. In one example, the processor 10 may include one or more CPUs, such as Figure 12 As an optional implementation, the electronic device may include multiple processors, for example, in addition to the processor 10, it may also include a processor 60 ( Figure 12 The dashed line is used as an example.

[0221] The memory 20 is used to store instructions. For example, the instruction may be a computer program. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions, or a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, etc., and the embodiments of the present application are not limited thereto.

[0222] It should be noted that the memory 20 may exist independently of the processor 10 or may be integrated with the processor 10. The memory 20 may be located inside the electronic device or outside the electronic device, which is not limited in the embodiment of the present application.

[0223] The communication line 30 is used to transmit information between the components included in the electronic device.

[0224] Communication interface 40 is used to communicate with other devices or other communication networks. Such other communication networks may be Ethernet, radio access networks (RAN), wireless local area networks (WLAN), etc. Communication interface 40 may be a module, circuit, transceiver, or any other device capable of communication.

[0225] The input / output interface 50 is used to implement human-computer interaction between a user and the electronic device, for example, to implement action interaction or information interaction between the user and the electronic device.

[0226] For example, the input / output interface 50 may be a mouse, keyboard, display screen, or touch screen screen, etc. Action interaction or information interaction between a user and the electronic device may be achieved through the mouse, keyboard, display screen, or touch screen screen, etc.

[0227] It should be noted that Figure 12 The structure shown in the figure does not constitute a limitation on the electronic device, except Figure 12 In addition to the components shown, the electronic device may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components.

[0228] In an exemplary embodiment, the present application also provides a computer program product, which includes computer instructions. When the computer instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment.

[0229] In an exemplary embodiment, the present application also provides a computer-readable storage medium including software instructions. When the software instructions are executed in an electronic device, the electronic device implements the method in the aforementioned method embodiment. The computer-readable storage medium can be a non-transitory computer-readable storage medium, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0230] In the above embodiments, all or part can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part can be implemented in the form of a computer program product. The computer program product includes one or more computer-executable instructions. When the computer-executable instructions are loaded and executed on a computer, all or part generates the flow or function according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer-executable instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).

[0231] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0232] Although the present application is described herein in conjunction with various embodiments, it is understood that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed application, from an inspection of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0233] The above description is merely illustrative of the application, and the scope of the application should be determined by the appended claims.

Claims

1. A recommendation model selection method, characterized in that: The method comprises: Based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time, determine the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model; the channel recommendation probability set includes multiple channel recommendation probabilities corresponding to multiple traffic channels one by one; the channel recommendation probability is used to represent the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel; each traffic channel in the multiple traffic channels is divided into multiple customer groups; the customer group recommendation probability set includes multiple customer group recommendation probabilities corresponding to the multiple customer groups one by one; the customer group recommendation probability is used to represent the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group; Based on the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model, determine the comprehensive recommendation probability set corresponding to each recommendation model; the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group in different traffic channels corresponding to the recommendation model; For the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, the target recommendation model corresponding to the target customer group is determined from the multiple recommendation models; the target traffic channel is a traffic channel among the multiple traffic channels; the target customer group is a customer group among the multiple customer groups of the target traffic channel; the target recommendation model recommends the target customer group in the target traffic channel within a future unit time.

2. The method according to claim 1, characterized in that For any first recommendation model among the multiple recommendation models, determining a channel recommendation probability set corresponding to each recommendation model based on a recommendation effect evaluation parameter of each recommendation model in a historical unit time period includes: For each traffic channel, based on the recommendation effect evaluation parameters of the first recommendation model within the historical unit time length of the traffic channel, determine the future revenue of the first recommendation model in the traffic channel and the historical reference revenue of the traffic channel; the future revenue is determined based on the revenue of the first recommendation model within the historical unit time length; the historical reference revenue is the sum of the revenues of the multiple recommendation models within the historical unit time length; the revenue is determined based on the recommendation effect evaluation parameters of the recommendation model; For each traffic channel, normalize the future revenue of the traffic channel and the historical reference revenue of the traffic channel using the first recommendation model to obtain a channel recommendation probability corresponding to the traffic channel; Based on the multiple channel recommendation probabilities corresponding to the multiple traffic channels, a channel recommendation probability set corresponding to the first recommendation model is determined.

3. The method according to claim 2, characterized in that The determining, for each traffic channel, based on the recommendation effect evaluation parameter of the first recommendation model within the historical unit time of the traffic channel, of the future revenue of the first recommendation model in the traffic channel includes: For each traffic channel, determining a basic revenue of the first recommendation model within the historical unit time length of the traffic channel based on a recommendation effect evaluation parameter of the first recommendation model within the historical unit time length of the traffic channel; The future revenue of the first recommendation model in the traffic channel is determined based on the basic revenue of the first recommendation model in the historical unit time of the traffic channel and the revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel.

4. The method according to claim 3, characterized in that The income adjustment amount is determined as follows: Determining a proportionality coefficient based on a ratio of a difference between an estimated excitation flow rate value per unit time in the future and an actual excitation flow rate value per unit time in the history of the first recommendation model; Determine a reference adjustment amount based on the product of the basic income of the first recommendation model within the historical unit time period and the proportional coefficient; The revenue adjustment amount of the first recommendation model in the future unit time of the traffic channel is determined based on the sum of the basic revenue adjustment amount and the reference adjustment amount of the first recommendation model in the traffic channel; the basic revenue adjustment amount is the revenue adjustment amount of the first recommendation model in the historical unit time.

5. The method according to claim 4, characterized in that The estimated excitation flow value per unit time of the first recommendation model in the future is determined by: Obtaining historical traffic sequences of the multiple recommendation models; the historical traffic sequences include historical stimulated traffic values ​​corresponding to multiple historical unit time periods; the historical stimulated traffic values ​​are total traffic values ​​stimulated by recommendations of the multiple recommendation models within the corresponding historical unit time periods; Based on the historical traffic sequence and the traffic prediction model, predict the total estimated stimulated traffic value stimulated by the recommendations of the multiple recommendation models within a future unit time length; The estimated stimulated traffic value of the recommendation model in the future unit time is determined based on the product of the total estimated stimulated traffic value and the historical revenue ratio of the first recommendation model; the historical revenue ratio is the ratio of the basic revenue of the first recommendation model in the historical unit time of the traffic channel to the total basic revenue of the multiple recommendation models in the historical unit time of the traffic channel.

6. The method according to claim 1, characterized in that For any first recommendation model among the multiple recommendation models, determining a set of customer group recommendation probabilities corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model within a historical unit time period includes: For each customer group, based on the recommendation effect evaluation parameters of the first recommendation model within a historical unit time period, determine an inference sample of the first recommendation model; the inference sample includes the historical recommendation effect evaluation parameters of the first recommendation model when making recommendations for the customer group, and the model hyperparameters of the first recommendation model; For each customer group, based on the inference samples of the first recommendation model and the recommendation effect evaluation model, predict the future recommendation effect evaluation parameters of the first recommendation model when making recommendations for that customer group; the recommendation effect evaluation model is used to predict future recommendation effect evaluation parameters based on historical recommendation effect evaluation parameters and model hyperparameters; For each customer group, determining the customer group recommendation probability of the first recommendation model for the customer group based on the proportion of the future recommendation effect evaluation parameter of the first recommendation model in the total future recommendation effect evaluation parameters of all recommendation models; Based on the customer group recommendation probabilities of the first recommendation model for different customer groups, a customer group recommendation probability set corresponding to the first recommendation model is obtained.

7. The method according to claim 6, characterized in that The method further comprises: Obtaining a training sample set; the training sample set includes multiple training samples, each training sample including a recommendation effect evaluation parameter of the recommendation model when making recommendations for a customer group in a first time period, a model hyperparameter of the recommendation model, and a recommendation effect evaluation parameter label when the recommendation model makes recommendations for the customer group in a second time period after the first time period; The preset initial model is trained based on the training sample set to obtain the recommendation effect evaluation model.

8. The method according to any one of claims 1 to 7, characterized in that The recommendation effect evaluation parameter includes at least one of the following: click-through rate, conversion rate, and average order value.

9. A recommendation model selection device, characterized in that: The device comprises: an acquisition module and a processing module; The acquisition module is used to determine the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model based on the recommendation effect evaluation parameters of each recommendation model in the historical unit time length; the channel recommendation probability set includes a plurality of channel recommendation probabilities corresponding one-to-one to a plurality of traffic channels; the channel recommendation probability is used to represent the probability that the corresponding recommendation model is allocated traffic for recommendation in a traffic channel; each of the plurality of traffic channels is divided into a plurality of customer groups; the customer group recommendation probability set includes a plurality of customer group recommendation probabilities corresponding one-to-one to the plurality of customer groups; the customer group recommendation probability is used to represent the probability that the corresponding recommendation model is allocated traffic for recommendation in a customer group; The processing module is used to determine the comprehensive recommendation probability set corresponding to each recommendation model based on the channel recommendation probability set and customer group recommendation probability set corresponding to each recommendation model; the comprehensive recommendation probability set includes the comprehensive recommendation probability of each customer group of the corresponding recommendation model in different traffic channels; for the target customer group in the target traffic channel, based on the comprehensive recommendation probability set corresponding to each recommendation model, determine the target recommendation model corresponding to the target customer group from the multiple recommendation models; the target traffic channel is one of the multiple traffic channels; the target customer group is one of the multiple customer groups in the target traffic channel; the target recommendation model recommends the target customer group in the target traffic channel within a future unit time.

10. An electronic device, characterized in that: include: processor and memory; The memory stores instructions executable by the processor; When the processor is configured to execute the instructions, the electronic device implements the method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: include: Software instructions; When the software instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that include: Computer instructions; When the computer instructions are executed in an electronic device, the electronic device is enabled to implement the method according to any one of claims 1 to 8.

Images