Recommendation system feature fusion method and device based on reinforcement learning agent

By adopting reinforcement learning agents and DQN networks in the recommendation system, the integration of dynamic adjustment of click-through rate and duration estimate models has been solved, and the real-time and accuracy of the recommendation system has been improved, and the users have been improved. Experience and recommend effects.

CN119089390AInactive Publication Date: 2024-12-06BEIJING WANGZHI TIANYUAN BIG DATA TECH CO LTD +1
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411455609.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-18
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In the process of integrating click-through rate estimate and duration estimate models, traditional recommendation systems cannot adjust dynamically according to the latest status in a timely manner, resulting in a deviation in recommendation effect.

Method used

Adopt reinforcement learning-based agents to build a click-through rate and reading time estimate model by obtaining user logs, and use DQN network for online services, dynamically adjust the fusion model to adapt to changes in user behavior, and use T-power form to fusion click-through rate and time Predicted value.

Benefits of technology

It improves the real-time and accuracy of the recommendation system, improves the total number of clicks, daily active users and reading time, and enhances the diversity and user experience of recommendation results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089390A_ABST
    Figure CN119089390A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field related to intelligent recommendation, in particular to a recommendation system feature fusion method and device based on a reinforcement learning agent. The method comprises the following steps: acquiring a user log; respectively constructing a click rate estimation model and a reading duration estimation model based on the user log; wherein the click rate estimation model is used for estimating the click rate of an article or a video, and the reading duration estimation model is used for estimating the reading duration of the article or the video; constructing a fusion model of an output result of the click rate prediction model and an output result of the reading duration prediction model; wherein the fusion model is further used for adjusting the internal parameters based on the latest operation of the user, so that the fusion value output by the fusion model better conforms to the expectation of the user; wherein the click rate is positively correlated with the fusion value, and the reading duration is positively correlated with the fusion value; and performing recommendation based on the click rate estimation model, the reading duration estimation model and the fusion model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field related to intelligent recommendation, and specifically to a feature fusion method and device for a recommendation system based on reinforcement learning intelligent agent. Background Art

[0002] Recommendation system is a very popular product form in the industry, widely used in APP service scenarios such as information channels, video channels, application stores, shopping channels, etc. Recommendation technology has been very popular in recent years, and there are many technical methods involved, which basically involve multi-objective estimation and multi-objective fusion.

[0003] In the multi-objective estimation phase, the most commonly used models are the click-through rate estimation model and the duration estimation model. The multi-objective fusion phase is to fuse the estimated values ​​produced by the click-through rate estimation, duration estimation and other models to form a unified estimated value. So, how to achieve more effective fusion has a greater impact on the final recommendation effect. Summary of the invention

[0004] In view of this, the embodiments of the present application are committed to providing a feature fusion method and device for a recommendation system based on reinforcement learning agents.

[0005] The present application provides a feature fusion method for a recommendation system based on a reinforcement learning agent, comprising: Get user logs; Based on the user logs, a click rate prediction model and a reading time prediction model are respectively constructed; The click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; Construct a fusion model of the output results of the click rate prediction model and the reading time prediction model; The fusion model is further used to adjust internal parameters based on the latest operation of the user, so that the fusion value output by the fusion model better meets the user's expectations; Among them, the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; Recommendations are made based on the click rate prediction model, the reading time prediction model and the fusion model.

[0006] In some embodiments, the click rate prediction model uses a dual-tower deep neural network model to predict the click rate. The click rate prediction model is inputted on one side with features of user operations in user logs, and on the other side with features of content in user logs. The click logs of real online recommendation results of users are trained offline in advance.

[0007] In some embodiments, the reading time estimation model uses a dual-tower deep neural network commonly used in the industry to estimate reading time. One side inputs the features of user operations in the user log, and the other side inputs the content features in the user log. Offline training is performed in advance based on the click log of the user's real online recommendation results.

[0008] In some embodiments, the click rate prediction model and the reading time prediction model are retrained at intervals of a preset duration based on the latest data.

[0009] In some embodiments, the fusion model uses T-power form to fuse the click rate and duration to obtain the fused prediction value q_final; The fusion formula is: q_final=p_ctrT ∗ p_duration(1-T); The value range of T is from 0.40 to 1.0, with a step size of 0.01. p_ctr is the estimated click rate; p_duration is the estimated reading time.

[0010] In some embodiments, it also includes: the fusion model adopts the DQN network model for online application, implements the Markov decision process, and adopts the ϵ--greed method for optimization.

[0011] The present application also provides a recommendation system feature fusion device based on reinforcement learning agent, comprising: Acquisition module, used to obtain user logs; A construction module is used to construct a click rate prediction model and a reading time prediction model based on the user log, respectively; wherein the click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; a fusion model of the output results of the click rate prediction model and the output results of the reading time prediction model is constructed; wherein the fusion model is also used to adjust internal parameters based on the user's latest operation, so that the fusion value output by the fusion model is more in line with the user's expectations; wherein the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; A recommendation module is used to make recommendations based on the click rate prediction model, the reading time prediction model and the fusion model.

[0012] The present application also provides an electronic device, including: A processor, and a memory for storing a program executable by the processor; The processor is used to implement the above-mentioned feature fusion method of the recommendation system based on reinforcement learning agent by running the program in the memory.

[0013] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the processor executes the feature fusion method for a recommendation system based on a reinforcement learning agent as described above.

[0014] The present application provides a method for fusion of recommendation system features based on reinforcement learning agent, obtains user logs; constructs click rate prediction model and reading time prediction model based on the user logs; wherein the click rate prediction model is used to estimate the click rate of articles or videos, and the reading time prediction model is used to estimate the reading time of articles or videos; constructs a fusion model of the output results of the click rate prediction model and the output results of the reading time prediction model; wherein the fusion model is also used to adjust internal parameters based on the latest operation of the user, so that the fusion value output by the fusion model is more in line with the user's expectations; wherein the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; and makes recommendations based on the click rate prediction model, the reading time prediction model and the fusion model. In this way, reinforcement learning is adopted, so that each time the user refreshes, the recommendation system can obtain the best candidate recommendation result in the current state by requesting the online service of the DQN network model. It solves the problem that the traditional method learns according to the historical environment or historical samples of the recommendation system, and cannot make dynamic adjustments in time to the current latest state of the recommendation system, thereby generating deviations. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0016] Figure 1 It is a flowchart of a feature fusion method for a recommendation system based on a reinforcement learning agent provided in one embodiment of the present application.

[0017] Figure 2 This is a schematic diagram of reinforcement learning provided by an embodiment of the present application.

[0018] Figure 3 This is a system implementation process provided by an embodiment of the present application.

[0019] Figure 4 This is a structural diagram of a click rate prediction model provided by an embodiment of the present application.

[0020] Figure 5 This is a structural diagram of a reading time estimation model provided by an embodiment of the present application.

[0021] Figure 6 This is a DQN network model structure diagram provided by an embodiment of the present application.

[0022] Figure 7 It is a structural diagram of a feature fusion device of a recommendation system based on reinforcement learning agent provided by an embodiment of the present application.

[0023] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Application Overview Recommendation system is a very popular product form in the industry, widely used in APP service scenarios such as information channels, video channels, application stores, shopping channels, etc. Recommendation technology has been very popular in recent years, and there are many technical methods involved, which basically involve multi-objective estimation and multi-objective fusion.

[0026] In the multi-objective estimation phase, the most commonly used models are the click-through rate estimation model and the duration estimation model. The multi-objective fusion phase is to fuse the estimated values ​​produced by the click-through rate estimation, duration estimation and other models to form a unified estimated value. So, how to achieve more effective fusion has a greater impact on the final recommendation effect.

[0027] Currently, there are two main methods for integrating the estimated values ​​produced by models such as click-through rate estimation and duration estimation.

[0028] One method is to manually design formulas and find parameters using multiple groups of small flows.

[0029] Another method is to use the XGBoost Point-wise supervised learning fusion method.

[0030] Existing methods such as multi-group small-flow parameter search and XGBoost Point-wise supervised learning fusion all perform machine learning based on the historical environment or historical samples of the recommendation system. They are unable to make dynamic adjustments in a timely manner based on the latest status of the recommendation system, which will result in fusion bias.

[0031] This patent uses reinforcement learning technology to solve the problem of how to merge the estimated click rate and the estimated duration. Each time a user refreshes, the recommendation system can obtain the best candidate recommendation result in the current state by requesting the online service of the DQN network model. This solves the problem that the traditional method learns based on the historical environment or historical samples of the recommendation system, and cannot make dynamic adjustments in time to the latest state of the recommendation system, thus causing deviations.

[0032] After introducing the basic principles of the present application, various non-limiting embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0033] Exemplary Methods Figure 1 FIG. 1 is a flow chart of a feature fusion method for a recommendation system based on a reinforcement learning agent provided by an embodiment of the present application. Figure 1 As shown, the method includes the following contents.

[0034] Step S110, obtaining user logs; Specifically, the data of user interaction with the recommendation system is collected, and this data will be used to train and optimize the recommendation model. User logs include basic user attributes (such as activity, points of interest, age, occupation, etc.), user refresh status (such as refresh times, whether in a WIFI environment, refresh type, etc.), and user operation sequence information (such as the number of recently displayed articles or videos, clicked articles or videos, and the reading time of clicked articles or videos, etc.).

[0035] Step S120, constructing a click rate prediction model and a reading time prediction model based on the user log respectively; The click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; Specifically, the click-through rate prediction model: uses a dual-tower DNN model, with features in user logs (such as user operation features) on one side and content features (such as article or video content features) on the other side. The model is trained offline based on the click logs of real online recommendation results of users to estimate the click-through rate of articles or videos. Reading time estimation model: also uses a dual-tower DNN model structure, and the input and training methods are similar to the click-through rate prediction model, but focuses on estimating the reading time of users for article or video content.

[0036] Step S130, constructing a fusion model of the output results of the click rate prediction model and the output results of the reading time prediction model; The fusion model is also used to adjust internal parameters based on the user's latest operation, so that the fusion value output by the fusion model is more in line with the user's expectations; the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; Specifically, the results of click-through rate prediction and reading time prediction are effectively integrated to form a comprehensive recommendation score. The fusion model can dynamically adjust the internal parameter T according to the user's latest operation to optimize the fusion value to make it more consistent with the user's expectations and behavior patterns.

[0037] Step S140, making recommendations based on the click rate prediction model, the reading time prediction model and the fusion model.

[0038] Specifically, based on the fusion value output by the fusion model, this value combines the estimates of click-through rate and reading time to reflect the user's interest in the recommended content and the possible degree of interaction. The recommendation system sorts the candidate recommended content according to the fusion value and prioritizes content with high scores to users. Continuous optimization: The recommendation system continuously collects new user feedback, retrains and adjusts the click-through rate estimation model, reading time estimation model and fusion model to adapt to changes in user behavior and improve recommendation quality.

[0039] In some embodiments, the click-through rate prediction model uses a dual-tower deep neural network model to predict click-through rate. One side of the click-through rate prediction model inputs the features of user operations in the user log, and the other side inputs the features of content in the user log. The click log based on the user's real online recommendation results is trained offline in advance.

[0040] Specifically, the relevant contents of the click rate prediction model include: 1. Model structure: Dual-tower structure: The model consists of two parts, namely two independent neural networks, which process different types of input features respectively.

[0041] User Feature Tower: Processes user-related features, such as user click history, browsing behavior, user preferences, etc.

[0042] Content feature tower: processes features related to recommended content, such as metadata, text descriptions, tags, etc. of articles or videos.

[0043] 2. Input features: User operation characteristics: including user behavior data on the recommendation system, such as clicks, browsing, and dwell time.

[0044] Content features: include attributes of the recommended content, such as the article or video title, description, publisher information, article or video category, etc.

[0045] 3. Offline training: The model is trained using historical data, which is usually the real click logs of users on the recommendation system.

[0046] By analyzing users’ click behavior on different recommended content, the model learns which features are associated with high click-through rates.

[0047] 4. Model training objectives: The training goal is to minimize the difference between the predicted click-through rate and the actual click-through rate, which is usually achieved using the cross entropy loss function.

[0048] 5. Model output: The output of the model is an estimated click-through rate for each recommendation, which is a value between 0 and 1 that indicates the probability that a user will click on the recommendation.

[0049] 6. Model optimization: The model is regularly retrained with the latest user click data to adapt to changes in user behavior and improve prediction accuracy.

[0050] 7. Application scenarios: In the recommendation system, the output of the model can be used as part of the recommendation algorithm to help the system decide which content is more likely to be clicked by users, thereby improving the personalization and accuracy of the recommendation.

[0051] Through this dual-tower structure, the model can capture both the user's behavior patterns and the content's features, thereby more accurately predicting the user's click behavior. The advantage of this structure is that it can flexibly process and learn large-scale heterogeneous data, providing powerful prediction capabilities for the recommendation system.

[0052] In some embodiments, the reading time estimation model uses a dual-tower deep neural network commonly used in the industry to estimate reading time. One side inputs the features of user operations in the user log, and the other side inputs the content features in the user log. The reading time log based on the user's real online recommendation results is trained offline in advance.

[0053] Specifically, the relevant contents of the reading time estimation model are as follows: 1. Model structure: Dual-tower design: Similar to the click-through rate prediction model, the reading time prediction model also uses two independent neural networks to process user features and content features respectively.

[0054] User Feature Tower: Analyze user behavior patterns, such as historical reading time, user preferences, active time periods, etc.

[0055] Content Feature Tower: Analyzes the properties of content, such as the length, complexity, and theme of articles or videos.

[0056] 2. Input features: User operation characteristics: including user’s reading history, interactive behaviors (such as comments, sharing), and user’s personal settings.

[0057] Content features: include detailed attributes of the recommended content, such as article or video content, image information, article or video length, etc.

[0058] 3. Offline training: The model is trained using real reading time data of users on the recommendation system, which usually comes from the user's actual reading behavior records of the content.

[0059] By analyzing how long users spend reading different recommended content, the model learns which features are associated with longer reading times.

[0060] 4. Model training objectives: The training goal is to minimize the difference between the predicted reading time and the actual reading time, which is usually achieved using the mean squared error (MSE) or other related loss functions.

[0061] 5. Model output: The output of the model is an estimated reading time for each recommended content, which is a continuous value indicating the time the user is expected to spend on the recommended content.

[0062] 6. Model optimization: The model is regularly retrained using the latest user reading data to adapt to changes in user behavior and improve prediction accuracy.

[0063] 7. Application scenarios: In the recommendation system, the output of the model can be used as part of the recommendation algorithm to help the system decide which content is more likely to attract users to read for a long time, thereby improving the attractiveness of the content and user engagement.

[0064] Through this dual-tower structure, the model can capture both the user's behavior patterns and the characteristics of the content, thereby more accurately predicting the user's reading time. The advantage of this structure is that it can flexibly process and learn large-scale heterogeneous data, providing powerful prediction capabilities for the recommendation system. In addition, through continuous optimization and adjustment, the model can better adapt to changes in user behavior, thereby improving the overall performance of the recommendation system and user experience.

[0065] In some embodiments, the click rate prediction model and the reading time prediction model are retrained at intervals of a preset duration based on the latest data.

[0066] In some specific implementations, the click rate prediction model and reading time prediction model will be retrained based on the latest data at a pre-set interval (e.g., one day). This is to enable the model to learn the latest data features and patterns in a timely manner, thereby improving the accuracy and adaptability of the model to better adapt to the current state of the recommendation system and changes in user needs.

[0067] In some embodiments, the fusion model uses T-power form to fuse the click rate and duration to obtain the fused prediction value q_final; The fusion formula is: q_final=p_ctr T ∗ p_duration (1-T); The value range of T is from 0.40 to 1.0, with a step size of 0.01. p_ctr is the estimated click rate; p_duration is the estimated reading time.

[0068] Using T-power to fuse click rate prediction and reading time prediction is an innovative approach. This approach allows the model to dynamically adjust the score of recommendation results based on different user behavior characteristics (such as click tendency and reading time). The following is a detailed explanation of this fusion method: In the fusion model mentioned in the patent briefing, using T-power to fuse click-through rate estimation and reading time estimation is an innovative method. This method allows the model to dynamically adjust the score of the recommendation results based on different user behavior characteristics (such as click tendency and reading time). The following is a detailed explanation of the fusion method: The value of T determines the relative importance of click-through rate and reading time in the final fusion value.

[0069] When T is close to 1, the click rate dominates the fusion value, which is suitable for scenarios that place more emphasis on user click behavior.

[0070] When T is close to 0.4, the impact of reading time is greater, which is suitable for scenarios that focus more on user engagement and content consumption depth.

[0071] 3. Value step: The step size of T is 0.01, which means that there are 60 possible T values ​​in the range of 0.40 to 1.0.

[0072] This fine-tuning capability enables the model to be flexibly adjusted according to different user groups and content types to achieve the best recommendation effect.

[0073] Application of Fusion Model Dynamic adjustment: By adjusting the value of T, the model can dynamically adjust the recommendation strategy according to different time periods, different user groups or different business goals.

[0074] Optimizing recommendation effects: The fusion model can more comprehensively evaluate the attractiveness of recommended content by comprehensively considering users' click behavior and content consumption depth, thereby improving the accuracy of recommendations and user satisfaction.

[0075] Experimentation and optimization: In practical applications, the optimal T value can be determined through methods such as A / B testing to achieve the best business results.

[0076] The core advantage of this fusion method lies in its flexibility and adaptability, which can be adjusted according to different business needs and user behavior characteristics, thereby optimizing the overall performance of the recommendation system.

[0077] In some embodiments, the fusion model uses a DQN network model for online application, implements a Markov decision process, and uses a ϵ--greed approach for optimization.

[0078] Using the Deep Q Network (DQN) model for online applications is an efficient method for implementing the Markov decision process and optimizing the performance of the recommendation system. The following is a detailed explanation of the method: Deep Q Network (DQN) Model 1. Introduction to DQN model: DQN is a technique that combines deep learning and reinforcement learning to solve decision-making problems with high-dimensional inputs.

[0079] DQN makes decisions by learning a policy that maps input states to expected rewards (i.e., Q values) for each possible action.

[0080] 2. Markov Decision Process: The present invention adopts reinforcement learning, so that each time the user refreshes, the recommendation system can obtain the best candidate recommendation result in the current state by requesting the online service of the DQN network model. This solves the problem that the traditional method learns based on the historical environment or historical samples of the recommendation system, and cannot make dynamic adjustments to the current latest state of the recommendation system in a timely manner, thus causing deviations.

[0081] From a macro-benefit perspective, the total number of clicks, daily active users, and total reading time of the recommendation system have all increased by about 1%. From a user experience perspective, the diversity of categories displayed and clicked on in the recommendation results has increased by more than 0.5%.

[0082] A Markov decision process (MDP) is a mathematical framework for modeling the decision-making process of decision makers in uncertain environments.

[0083] In the recommendation system, each user's state (such as user attributes, points of interest, refresh times, etc.) and possible actions (such as different recommended content) constitute the elements of the MDP.

[0084] 3. ϵ-greedy strategy: The ϵ-greedy strategy is a commonly used exploration and exploitation strategy that is used to strike a balance between exploration (trying new, unknown actions) and exploitation (selecting the best known action).

[0085] In the ϵ-greedy strategy, the current best action is chosen with probability \( 1 \varepsilon \) and an action is chosen randomly with probability \( \varepsilon \). The parameter \( \varepsilon \) is usually gradually reduced over time to reduce exploration and increase exploitation.

[0086] Implementation of the fusion model 1. DQN network structure: A DQN network typically consists of an input layer, several hidden layers, and an output layer.

[0087] The input layer receives features of user status and recommended content.

[0088] The hidden layer processes the input data through a nonlinear activation function to extract features.

[0089] The output layer contains multiple neurons, one for each possible action (in this case, a different value of T).

[0090] 2. Training process: Use real user feedback (such as clicks and reading time) as reward signals to train the DQN model.

[0091] By continuously updating the network weights, the model is able to predict the expected reward for each action in a given state.

[0092] 3. Online Application: When a user refreshes the recommended content, the DQN model predicts the reward value of each possible action based on the current user state and content features.

[0093] The action is selected according to the ϵ-greedy strategy, that is, the value of T that maximizes \( q_{\text{final}} \) is selected, or a value of T is randomly selected for exploration.

[0094] 4. Optimization and adjustment: Over time, the DQN model is continuously trained and optimized by collecting more user feedback data.

[0095] Adjust the \( \varepsilon \) value in the ϵ-greedy strategy to adapt to different business needs and changes in user behavior.

[0096] By combining DQN and ϵ-greedy strategies, the fusion model can dynamically adjust the recommendation strategy to adapt to changes in user behavior, thereby improving the performance of the recommendation system and user experience.

[0097] The following is a specific example and Figure 2-Figure 6 Description of this application: First, let’s explain reinforcement learning: Reinforcement learning is used to describe how an agent or actor learns during interaction with an environment. For a recommendation system, the user is the environment, and the recommendation system is an actor with a policy.

[0098] like Figure 2 As shown in the figure, the action role decides action a1 for the user in the recommendation result refresh 1 according to the user's state s1 (operation state, basic attributes, points of interest, etc.). After browsing the results of this action, the user gives a reward (Reward) r1 (such as opening the landing page, staying to read, etc.). The action role then decides the next action a2 according to the current user's state s2, and so on. The interaction continues until the user leaves, thus reaching the end of a user's operation sequence. This sequence forms a Markov decision process. ; The goal of learning can be simply abstracted into finding the maximum The strategy π.

[0099] The system implementation process of the present invention is as follows Figure 3 As shown, the following steps are included: Step 1: Collect user feedback Collect user status features, including: basic user attributes, such as user activity, user interests, age, occupation, etc.; user refresh status, such as the number of refreshes, whether WIFI is used, refresh type, etc.; user operation sequence information, such as how many articles or videos have been displayed recently, which articles or videos have been clicked, and the reading time of clicked articles or videos.

[0100] Step 2: User log stitching Samples are organized based on the granularity of each refresh action of the user on the result page of the recommendation system. The characteristics of the user status are recorded in the user log, and the log files are processed and spliced ​​to form samples that can be used by the machine learning model.

[0101] Step 3: Click-through rate estimation The industry-standard dual-tower DNN (deep neural network) model is used to estimate click-through rate. The name "dual towers" vividly explains the structural composition of the model. The model structures on both sides are like two tall towers, and the top interoperability layer is like an aerial corridor built by two tower spires, responsible for the communication of information on both sides. In the dual-tower model for click-through rate estimation, one side inputs the features obtained by splicing user logs, and the other side inputs the content features of the results. This model is trained offline in advance using the click logs of real online recommendation results, and then applied to the online for estimation. The output is the estimated click-through rate value p_ctr of the recommended result. The dual-tower model is updated daily. The structure of the dual-tower model for click-through rate estimation is as follows: Figure 4 shown.

[0102] Step 4: Estimated reading time The industry-wide dual-tower DNN (deep neural network) model structure is used to estimate reading time. One side inputs the features obtained by splicing user logs, and the other side inputs the content features of the results. This model is trained offline in advance using the reading time logs of real online recommendation results, and then used online for estimation. The output is the estimated reading time p_duration to be recommended. The dual-tower model is updated daily. The structure of the dual-tower model for reading time estimation is as follows: Figure 5 shown.

[0103] Step 5: Click duration fusion value prediction Use the T-power form to fuse the click rate and duration to get the fused prediction value q_final. The fusion formula is: q_final=p_ctr T ∗ p_duration (1-T) The value range of T is from 0.40 to 1.0, and the step size is 0.01, so a total of 60 possible actions are selected as candidates.

[0104] Use the Deep Q-learning method commonly used in the industry in reinforcement learning to learn, and train a DQN network model (Deep Q-learning Network) offline in advance. The model structure is as follows Figure 6 As shown. The action reward function value It is the model's prediction of the action to be used in the current state s. The reward value that will be obtained; 𝜃 is the weight.

[0105] For each set of recommendation results of the recommendation system, the user will give a set of real feedback. By recording the different states of the user, abstracting the feature representation, and then forming an operation sequence training sample with refresh as the granularity, a DQN model can be trained. The input of the model is the user's current state, including the user's attributes, points of interest, refresh times, recent operation sequence information, etc. The output of the model is the estimated reward value of the result launched to the user by each candidate action in this state. In this way, by traversing the output of all candidate actions of the DQN model, taking the candidate action corresponding to the largest reward value as the T fusion parameter of this recommendation, and applying it to the line, a closed loop of continuous learning and continuous optimization is formed. Of course, in order to ensure the exploration of unknown possibilities, 1% of the actions will be randomly selected.

[0106] The DQN network model uses a three-layer network structure. The last layer is a fully connected layer with 60 output nodes, corresponding to the estimated reward value of each action. The model organizes samples at a refresh granularity. First, 1% of users are randomly selected to obtain samples. No simulator is used. The industry-common Off-policy algorithm is used for training, and the model is updated daily.

[0107] Reward design: Current reward value Immediate Reward(t) = ln(1 + total number of clicks within the refresh at time t * average stay time on the landing page within the refresh at time t) Total Reward(t) = Immediate Reward(t) + r * ImmediateReward(t+1) + r 2 * Immediate Reward(t+2) + r 3 * Immediate Reward(t+3) + … The attenuation coefficient is r = 0.5 How the DQN network model works online: Each time a user refreshes, the recommendation system requests the online service of the DQN network model. The online service calculates the reward value corresponding to all candidate actions (60 candidates) in this state through the DQN network model according to the user's basic attributes, current refresh status, and operation sequence characteristics (i.e., the reward value for each action). The estimated value of the corresponding reward , and then use the industry's commonly used method (i.e., 1% probability of randomly selecting T, 99% probability of selecting the maximum estimated reward value) Corresponding T) returns the T value to the recommendation system for fusion q_final=p_ctr T ∗ p_duration (1-T) .

[0108] Step 6: Overall sorting of results The recommendation system performs overall sorting of the recommended results based on the predicted value q_final which is a fusion of the click rate estimate and the duration estimate. The sorted results are then orchestrated and processed at the business layer of the recommendation system and inserted into the cache queue of the results to be displayed.

[0109] Step 7: Result refresh recommendation The user refreshes the result display interface of the recommendation system, and the recommendation system takes the result from the cache queue to be displayed and displays it to the user. Then the process of step 1 is repeated.

[0110] In this way, the present invention adopts reinforcement learning, so that each time the user refreshes, the recommendation system can obtain the best candidate recommendation result in the current state by requesting the online service of the DQN network model. This solves the problem that the traditional method learns based on the historical environment or historical samples of the recommendation system, and cannot make dynamic adjustments to the current latest state of the recommendation system in a timely manner, thus causing deviations.

[0111] From a macro-benefit perspective, the total number of clicks, daily active users, and total reading time of the recommendation system have all increased by about 1%. From a user experience perspective, the diversity of categories displayed and clicked on in the recommendation results has increased by more than 0.5%.

[0112] Exemplary Devices The device embodiments of the present application can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0113] Figure 7 FIG. 1 is a block diagram of a feature fusion device for a recommendation system based on a reinforcement learning agent provided by an embodiment of the present application. Figure 7 As shown, the device comprises: An acquisition module 71 is used to acquire user logs; A construction module 72 is used to construct a click rate prediction model and a reading time prediction model based on the user log, respectively; wherein the click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; a fusion model of the output results of the click rate prediction model and the output results of the reading time prediction model is constructed; wherein the fusion model is further used to adjust internal parameters based on the user's latest operation, so that the fusion value output by the fusion model is more in line with the user's expectations; wherein the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; The recommendation module 73 is used to make recommendations based on the click rate prediction model, the reading time prediction model and the fusion model.

[0114] Exemplary Electronic Devices Below, reference Figure 8 To describe an electronic device according to an embodiment of the present application. Figure 8 A block diagram of an electronic device according to an embodiment of the present application is illustrated.

[0115] like Figure 8 As shown, electronic device 800 includes one or more processors 810 and memory 820 .

[0116] The processor 810 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0117] The memory 820 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 810 may run the program instructions to implement the feature fusion method of the recommendation system based on the reinforcement learning agent of each embodiment of the present application described above and / or other desired functions. Various contents such as category correspondences may also be stored in the computer-readable storage medium.

[0118] In one example, the electronic device 800 may further include: an input device 830 and an output device 840 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0119] In addition, the input device 830 may also include, for example, a keyboard, a mouse, an interface, etc. The output device 840 may output various information to the outside, including analysis results, etc. The output device 840 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0120] Of course, to simplify, Figure 8 Only some of the components in the electronic device related to the present application are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0121] Exemplary computer program products and computer-readable storage media In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the feature fusion method of the recommendation system based on reinforcement learning agents according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0122] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0123] In addition, an embodiment of the present application may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the feature fusion method for a recommendation system based on a reinforcement learning agent according to various embodiments of the present application described in the above “Exemplary Method” section of this specification.

[0124] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0125] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present application to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A feature fusion method for a recommendation system based on reinforcement learning agent, characterized in that: include: Get user logs; Based on the user logs, a click rate prediction model and a reading time prediction model are respectively constructed; The click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; Construct a fusion model of the output results of the click rate prediction model and the reading time prediction model; The fusion model is a reinforcement learning model, and is also used to adjust internal parameters based on the user's latest operation, so that the fusion value output by the fusion model better meets the user's expectations; Among them, the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; Recommendations are made based on the click rate prediction model, the reading time prediction model and the fusion model.

2. The feature fusion method of recommendation system based on reinforcement learning agent according to claim 1, characterized in that: The click rate prediction model uses a dual-tower deep neural network model to predict the click rate; The CTR prediction model is inputted on one side with features of user operations in user logs, and on the other side with features of content in user logs, and is obtained by offline training in advance based on click logs of real online recommendation results of users.

3. The feature fusion method of recommendation system based on reinforcement learning agent according to claim 1, characterized in that: The click rate prediction model and reading time prediction model adopt the industry-common dual-depth neural network to estimate reading time. One side inputs the features of user operations in the user log, and the other side inputs the features of content in the user log. They are obtained by offline training in advance based on the click logs of the user's real online recommendation results.

4. The feature fusion method of recommendation system based on reinforcement learning agent according to claim 1, characterized in that: The click rate prediction model and reading time prediction model are retrained based on the latest data at each preset interval.

5. The feature fusion method of recommendation system based on reinforcement learning agent according to claim 1, characterized in that: The fusion model uses the T-power form to fuse the click rate and the duration to obtain the fused prediction value q_final; The fusion formula is: q_final=p_ctr T ∗ p_duration (1-T); The value range of T is from 0.40 to 1.0, with a step size of 0.

01. p_ctr is the estimated click-through rate; p_duration is the estimated reading time.

6. The feature fusion method of recommendation system based on reinforcement learning agent according to claim 1, characterized in that: Also includes: The fusion model adopts the DQN network model for online application, implements the Markov decision process, and uses the ϵ--greed method for optimization.

7. A feature fusion device for a recommendation system based on reinforcement learning agent, characterized in that: include: Acquisition module, used to obtain user logs; A construction module is used to construct a click rate prediction model and a reading time prediction model based on the user log, respectively; wherein the click rate prediction model is used to estimate the click rate of an article or video, and the reading time prediction model is used to estimate the reading time of an article or video; a fusion model of the output results of the click rate prediction model and the output results of the reading time prediction model is constructed; wherein the fusion model is also used to adjust internal parameters based on the user's latest operation, so that the fusion value output by the fusion model is more in line with the user's expectations; wherein the click rate is positively correlated with the fusion value, and the reading time is positively correlated with the fusion value; A recommendation module is used to make recommendations based on the click rate prediction model, the reading time prediction model and the fusion model.

8. An electronic device, characterized in that: include: A processor, and a memory for storing a program executable by the processor; The processor is used to implement the feature fusion method of the recommendation system based on reinforcement learning agent according to any one of claims 1 to 6 by running the program in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to execute the feature fusion method for a recommendation system based on a reinforcement learning agent according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Information recommendation method and device, electronic equipment and computer readable storage medium

    CN115858911A

  • Search engine accurate recommendation method based on user behavior analysis

    CN117851689A

  • User data mining method and system for various customized meteorological services

    CN118445483A