A method and system for information pushing based on reinforcement learning, and a computer readable storage medium

Abstract

This application provides an information push method and system based on reinforcement learning, as well as a computer-readable storage medium, belonging to the field of artificial intelligence technology. The solution includes: constructing a multi-dimensional user emotion vector representing the user's psychological state, the vector containing at least information on fatigue, surprise, and demand; dynamically generating a reward function based on the multi-dimensional user emotion vector, where the weights of various factors related to business benefits, changes in fatigue, and changes in surprise are determined based on the multi-dimensional user emotion vector; and a reinforcement learning agent selecting the action with the highest expected total reward from a hierarchical candidate action space based on the reward function and executing it. The hierarchical candidate action space includes multiple action space layers, with different action space layers containing actions for different purposes or with different strategies.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an information push method and system based on reinforcement learning, and a computer-readable storage medium. Background Technology

[0002] In online commercial operation systems such as e-commerce platforms and content platforms, the goal is typically to maximize short-term commercial profits. Therefore, these systems often employ marketing strategies based on preset rules or simple models. In recent years, reinforcement learning technology has been introduced into this field, attempting to learn optimal operational strategies through the interaction between intelligent agents and the user environment.

[0003] However, existing applications still have the following shortcomings: First, the characterization of user states is overly simplistic. Existing reinforcement learning operation systems typically only consider a single, general state dimension, such as user activity or purchase history, failing to accurately and comprehensively depict the complex psychological changes of users during continuous interactions. For example, the system struggles to distinguish whether a user churns due to disinterest in content or fatigue caused by excessive marketing activities. Second, reward mechanisms are usually static. In existing reinforcement learning schemes, the trade-off coefficient between rewards representing commercial gains and penalties representing negative user feedback (such as churn) is usually a pre-set fixed value. This rigid decision-making logic cannot adapt to the dynamic changes in user emotional states. When users are fatigued, the system may still execute high-intensity marketing actions to pursue short-term gains, thereby exacerbating the user's negative experience. Finally, the system's decision-making action space is incomplete. Most existing operation systems are limited to the two actions of "pushing marketing content" or "not pushing," lacking strategic actions that can proactively maintain and improve the long-term user experience. This limitation leads to the accumulation of negative user emotions in the process of pursuing short-term profit maximization, ultimately damaging the total lifetime value of users. Therefore, how to achieve a dynamic and optimal balance between pursuing short-term business goals and maintaining long-term user value is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] The purpose of this application is to provide an information push method and system based on reinforcement learning, as well as a computer-readable storage medium, which aims to solve the technical problem that existing commercial operation systems based on reinforcement learning cannot achieve a dynamic optimal balance between short-term business goals and long-term user value due to overly simplistic user psychological state models, rigid reward mechanisms, and incomplete decision-making action spaces.

[0005] To achieve the above objectives, this application provides an information push method based on reinforcement learning, comprising:

[0006] Construct a multidimensional user emotion vector that represents the user's current comprehensive psychological state. The multidimensional user emotion vector contains information from at least three dimensions: fatigue, surprise, and need.

[0007] Based on the multidimensional user emotion vector, a reward function for reinforcement learning decision-making is dynamically generated. The reward function includes at least terms related to business benefits, changes in user fatigue, and changes in user surprise, and the weight coefficients of each term are determined based on the multidimensional user emotion vector.

[0008] Based on the reward function, the reinforcement learning agent selects the action with the highest expected total reward as the optimal action and executes it in the hierarchical candidate action space. The hierarchical candidate action space includes multiple action space layers, and different action space layers include actions with different purposes or different strategies.

[0009] In one possible implementation, constructing a multidimensional user emotion vector representing the user's current comprehensive psychological state includes:

[0010] The system acquires multi-dimensional state characteristics of users, including interactive behavior characteristics, historical behavior characteristics, marketing feedback characteristics, and social network characteristics.

[0011] The fatigue prediction sub-model is used to process the multi-dimensional state features and obtain the fatigue level output by the fatigue prediction sub-model.

[0012] The multi-dimensional state features are processed using the surprise prediction sub-model, and the surprise level output by the surprise prediction sub-model is obtained.

[0013] The multi-dimensional state features are processed using the demand prediction sub-model, and the demand degree output by the demand prediction sub-model is obtained.

[0014] The multidimensional user emotion vector is determined using the fatigue level, the surprise level, and the demand level.

[0015] In one possible implementation, the fatigue prediction sub-model includes a gradient boosting decision tree model or a deep neural network model.

[0016] In one possible implementation, the weighting coefficient corresponding to the change in user fatigue is determined using a monotonically increasing function with fatigue as the variable.

[0017] In one possible implementation, the weighting coefficient corresponding to business revenue is determined using a monotonically increasing function with demand as the variable.

[0018] In one possible implementation, the hierarchical candidate operational action space includes a high-intensity marketing action layer, a low-intensity marketing action layer, a value-supplementing cooling action layer, and a strategic blank cooling action layer.

[0019] Among them, the high-intensity marketing action layer includes forced pop-up actions and homepage banner pushes of high discounts;

[0020] The low-intensity marketing action layer includes in-feed advertising actions and red dot notification actions;

[0021] The value-added cooling action layer includes information push actions related to user interests;

[0022] The strategic white space cooling action layer includes actions such as not pushing information within a preset time window and reducing the information push frequency to below a preset value.

[0023] In one possible implementation, it also includes:

[0024] The fatigue prediction sub-model, surprise prediction sub-model, demand prediction sub-model, and reinforcement learning agent are iteratively updated using user feedback data generated after the optimal action is performed; the user feedback data includes user click-through rate, purchase rate, or fatigue changes.

[0025] This application also provides an information push system based on reinforcement learning, including:

[0026] The multidimensional user emotion vector generation module is used to construct a multidimensional user emotion vector that represents the user's current comprehensive psychological state. The multidimensional user emotion vector contains information in at least three dimensions: fatigue, surprise, and need.

[0027] An adaptive reward shaping module is used to dynamically generate a reward function for reinforcement learning decision-making based on the multidimensional user emotion vector. The reward function includes at least terms related to business benefits, changes in user fatigue, and changes in user surprise, and the weight coefficients of each term are determined based on the multidimensional user emotion vector.

[0028] A reinforcement learning agent is used to select the action with the highest expected total reward as the optimal action and execute it in a hierarchical candidate action space based on the reward function. The hierarchical candidate action space includes multiple action space layers, and different action space layers include actions with different purposes and strategies.

[0029] In one possible implementation, the multidimensional user emotion vector generation module is configured to perform the following operations:

[0030] The system acquires multi-dimensional state characteristics of users, including interactive behavior characteristics, historical behavior characteristics, marketing feedback characteristics, and social network characteristics.

[0031] The fatigue prediction sub-model is used to process the multi-dimensional state features and obtain the fatigue level output by the fatigue prediction sub-model.

[0032] The surprise level prediction sub-model is used to process multi-dimensional state features and obtain the surprise level output by the surprise level prediction sub-model.

[0033] The demand prediction sub-model is used to process the multi-dimensional state features and obtain the demand degree output by the demand prediction sub-model.

[0034] The multidimensional user emotion vector is determined using the fatigue level, the surprise level, and the demand level.

[0035] This application also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the information push method based on reinforcement learning as described in any of the preceding claims.

[0036] Compared with the prior art, the beneficial effects of this application are as follows:

[0037] 1. Achieved dynamic balance and proactive planning in decision-making. By designing the weight coefficients of the reward function as a function based on the user's emotion vector, this application can dynamically adjust the trade-off logic between short-term business benefits and long-term user experience according to the user's real-time psychological state. Simultaneously, by incorporating the estimated emotion change into the reward function, the system can shift from passive response to proactive planning of action sequences to maximize the user's lifecycle value.

[0038] 2. Improved precision and accuracy of decision-making. Compared to the single and general state perception of existing technologies, this application provides a richer and more accurate user profile for decision-making through a multi-dimensional vector composed of fatigue, surprise, and demand, making operational decisions more contextualized and reasonable, and avoiding static "one-size-fits-all" strategies.

[0039] 3. Quantify user experience and incorporate it into optimization goals. This application models psychological indicators such as "surprise level" that represent positive user experience and incorporates them as positive reward items into the optimization goals of reinforcement learning. This incentivizes the system to proactively create and improve user experience while pursuing commercial benefits, thus serving the growth of long-term user value.

[0040] Furthermore, this application pioneers a new operating model. By defining a layered action space that includes "value-added cooling actions" and "strategic blank cooling actions," this application transforms operational behavior from a simple "marketing disturbance" into an "intelligent service." Without conducting direct marketing, the system can enhance user stickiness by providing value or giving users quiet time. This represents a fundamental improvement compared to existing technologies that only include an "push / no push" action space. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the structure of an information push system based on reinforcement learning, provided in an embodiment of this application.

[0042] Figure 2 This is a flowchart illustrating an information push method based on reinforcement learning, provided as an embodiment of this application.

[0043] Figure 3 This is a schematic diagram of the signaling timing for system-user interaction in one embodiment of this application.

[0044] Figure 4 This is a schematic diagram showing the relationship between the weights of the adaptive reward function in the embodiments of this application. Detailed Implementation

[0045] To better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is to be understood that the specific embodiments described herein are only for explaining this application and not for limiting this application.

[0046] Example 1

[0047] This embodiment provides an information delivery method and system based on reinforcement learning. (Refer to...) Figure 1 This method can be implemented by a reinforcement learning-based information push system, which can be deployed on a server, such as the backend server of a live streaming platform. The system includes a data acquisition module 20, a multi-dimensional user emotion vector generation module 30, an adaptive reward shaping module 40, a reinforcement learning agent 50, and an action execution module 60. These modules can be software program units, hardware circuits, or combinations thereof.

[0048] Figure 2 This illustration shows the overall flow of an information push method based on reinforcement learning provided in an embodiment of this application. This embodiment will combine... Figure 1 The system architecture shown is Figure 2The method and process illustrated are explained in detail in a "casual live streaming" application scenario. This scenario is used to illustrate how, when users are in a complex state of "high fatigue and high demand," the technical solution of this application can simultaneously meet user needs and improve their experience by selecting "value-added cooling actions" rather than direct marketing, thereby achieving a win-win situation for commercial conversion and long-term value.

[0049] The execution process of this method in a specific decision-making cycle is described as follows:

[0050] First, step S101 is executed to acquire user state characteristics, that is, to acquire multi-dimensional user state characteristics. Specifically, the data acquisition module 20 continuously collects multi-dimensional state characteristics related to a specific user (e.g., user A) from the user terminal 10. These characteristics together constitute a comprehensive user behavior profile, rather than being limited to a single interaction event. In this embodiment, the data acquisition module 20 detected that user A continuously watched live streams online for the past 3 hours, which is a signal indicating potential fatigue. At the same time, the system recorded that user A swiped or closed the application push notification for gift items three times in the last 15 minutes, which further strengthens the judgment that the user is in a fatigued state. On the other hand, the data acquisition module 20 also captured that user A repeatedly entered the mall and browsed the details page of a newly launched virtual item "Starship" during the intervals of watching the live stream, with a total browsing time of more than 5 minutes, but did not place an order. These raw data, including session duration, pop-up interaction history, browsing depth and duration of specific pages, historical purchase records, etc., are integrated into a set of structured multi-dimensional state characteristics.

[0051] Subsequently, step S102 is executed to construct a multi-dimensional user emotion vector. The aforementioned multi-dimensional state features are transmitted to the multi-dimensional user emotion vector generation module 30. This module integrates multiple parallel emotion state prediction models, whose function is to map raw, high-dimensional behavioral data to a low-dimensional, quantified psychological state space. Specifically, the fatigue prediction sub-model 31, the surprise prediction sub-model 32, and the need prediction sub-model 33 are responsible for predicting different dimensions of the user emotion vector, respectively.

[0052] In one embodiment of this application, the fatigue prediction sub-model 31 can be configured as a gradient boosting decision tree model. This model receives features such as "continuous online time," "number of pop-up window closures per unit time," and "time since the last payment" as input. Through ensemble learning of a series of decision trees, it outputs a fatigue score between [0, 1]. Since user A has a long online time and frequently closes pop-ups, the fatigue score F output by the model is very high, for example, F = 0.9.

[0053] The demand prediction sub-model 33 can be configured as a deep neural network model, which receives features such as "number of times a specific product page is viewed", "viewing duration", "whether it is added to the shopping cart", and "historical purchase preferences for similar products" as input. User A's repeated browsing of "Starship" is interpreted by the model as a strong signal of immediate demand, so the demand N value output by the model is also high, for example, N=0.8.

[0054] The surprise level prediction sub-model 32 can also be a deep neural network model, which analyzes the diversity of recent content consumed by users, the degree of deviation from historical interests, and the interaction with new features or content. Since user A's recent behavior pattern is monotonous and has not interacted with novel content, the surprise level S value output by the model is low, for example, S=0.1.

[0055] Finally, the multidimensional user emotion vector generation module 30 integrates the outputs of each sub-model to generate a multidimensional user emotion vector V_sentiment = [fatigue level F=0.9, surprise level S=0.1, need level N=0.8] representing user A's current comprehensive psychological state. This vector is simultaneously passed to the adaptive reward shaping module 40 and the reinforcement learning agent 50.

[0056] Next, step S103 is executed to dynamically generate an adaptive reward function. After receiving the V_sentiment vector, the adaptive reward shaping module 40 dynamically constructs a reward function suitable for the current decision step. This reward function can be defined as follows: in, It is the estimated immediate business revenue after performing a certain action. and These are the estimated changes in user fatigue and surprise after performing the action.

[0057] It should be noted that the weighting coefficients , , It is a function of the current multidimensional user sentiment vector V_sentiment, not a fixed value. (See reference...) Figure 4 The figure schematically illustrates the functional relationship between the weighting coefficients and the emotion dimension. In this embodiment, the fatigue penalty weighting... Designed as a monotonically increasing function of fatigue factor F, such as an exponential function. , where k is a hyperparameter. When F=0.9, The value will become very large, meaning the system will impose a huge penalty on any action that might further increase user fatigue. (Reward weight) Designed as a monotonically increasing function of demand N, such as a linear function. a and b are both hyperparameters. When N=0.8, The value is also at a high level, indicating that the system is still focused on meeting users' high demands to obtain potential benefits. (Surprise reward weight) This is a more complex function of S and F, whose weights might be moderately increased to encourage exploration when S is currently low. This dynamic reward function, "tailor-made" for user A's current state, is then sent to reinforcement learning agent 50.

[0058] Subsequently, steps S104 (evaluating candidate actions in the hierarchical action space) and S105 (selecting the action with the highest expected total reward) are executed. The hierarchical action space is the aforementioned hierarchical candidate action space. The reinforcement learning agent 50, as a reinforcement learning agent, evaluates the expected total reward of each selectable action in its preset hierarchical candidate action space based on the current state (i.e., the V_sentiment vector) and the reward function provided by the adaptive reward shaping module 40, and selects the optimal one.

[0059] Candidate Action A (High-Intensity Marketing Action): Immediately pop up a window on user A's terminal 10, pushing a "50% off Starship coupon". For this action, the value network or model within the reinforcement learning agent 50 makes the following prediction: Estimated Business Benefits Due to high user demand and the incentive of discounts, the probability of purchase is relatively high, and the estimated revenue is a high positive value, such as 50. Estimated change in fatigue level. If an intrusive pop-up is received again while the user is in a state of high fatigue, the estimated fatigue level will increase sharply. A large positive value, such as +0.5. Estimated change in surprise level. This conventional promotional tactic lacks originality, and the expected level of surprise varies. It is 0 or slightly negative. Substituting it into the reward function, because... At this point, it is extremely large. The term will become a huge negative value, even if Higher, with a higher expected total reward. It is also very likely to be a negative number, such as -200.

[0060] Candidate Action B (Value-Added Cooling Action): Push an article titled "Interview with the Designer of 'Starship': Unveiling the Story Behind its Creation" to user A via in-site messaging (a non-intrusive method). The model's prediction for this action is as follows: Estimated Commercial Revenue This action is not direct marketing and does not directly generate revenue, therefore... Predicting changes in fatigue The content pushed is highly relevant and valuable to users' interests, rather than being marketing harassment. This is expected to alleviate user annoyance and reduce fatigue. A negative value, such as -0.3. Estimated change in surprise level. This way of providing in-depth information may exceed user expectations, bringing a sense of novelty and anticipation. Let it be a positive value, for example, +0.4. Substituting this into the reward function, since... Negative, The item is transformed into a positive reward. Meanwhile, It is also a positive reward. Although It is 0, but the final expected total reward is 0. It is a significantly positive value, such as +150.

[0061] After comparing Total_Reward(A) and Total_Reward(B), the reinforcement learning agent 50 selects the action B with the highest expected total reward as the optimal action to be executed in step S105.

[0062] Finally, steps S106 (execute the optimal action) and S107 (collect user feedback and iteratively update the model) are executed. The reinforcement learning agent 50 passes the optimal action "push the designer interview article" to the action execution module 60. The action execution module 60 is responsible for calling the content library and message push service to send the article via in-site message to user A's user terminal 10. This interaction process is as follows: Figure 3 As shown, after making a decision, the information push system based on reinforcement learning requests valuable supplementary content from the content / product library and pushes it to the user.

[0063] User A received an in-app message and was attracted by the title, clicking to read the article. The data acquisition module 20 captured this click behavior and the user's dwell time on the article page as positive "actual user feedback data." More importantly, the system monitored in a subsequent observation window (e.g., within 1 hour) that about half an hour after reading the article, User A actively returned to the store and purchased the "Starship" item at the original price.

[0064] This series of feedback data is used for iterative updates in step S107. On one hand, the complete experience trajectory (current state V_sentiment, action B, actual reward obtained, next state) is stored in the experience replay pool of the reinforcement learning agent 50 for subsequent training and optimization of its policy network. This increases the probability that the reinforcement learning agent will choose a value-replenishing cooling action when it encounters a similar "high fatigue + high demand" state again in the future. On the other hand, this successful transformation can provide a supervisory signal for the emotion state prediction model. For example, the user's final purchase behavior verifies that the initial demand level N=0.8 prediction is accurate; and the fact that the user made the purchase after receiving the cooling action can generate a pseudo-label indicating that the action did indeed effectively reduce fatigue. This information can be used to fine-tune the fatigue prediction sub-model 31 and the demand prediction sub-model 33, continuously improving their predictive capabilities.

[0065] Through the steps above, this embodiment demonstrates how the method, through precise emotion perception, dynamic reward trade-offs, and a rich action space, prioritizes solving users' negative emotion problems without sacrificing ultimate business goals, thereby achieving synergistic growth in user experience and business value.

[0066] Example 2

[0067] This embodiment illustrates the application of the technical solution of this application in another scenario, specifically how the system uses "strategic white space cooling-off actions" to proactively reduce disturbances and maintain long-term user activity and goodwill when the user exhibits moderate fatigue and has no clear needs. The scenario in this embodiment is set as a news and information application, whose backend also deploys... Figure 1 The image shows an information delivery system based on reinforcement learning.

[0068] The process also follows the same principle. Figure 2 The steps are shown.

[0069] In step S101, the data acquisition module 20 detected a change in user B's recent behavioral patterns. Specifically, user B's average daily app launches decreased from 5 times last week to 2 times this week, and the total in-app usage time decreased by 40%. Furthermore, user B's click-through rate on daily news notifications pushed by the system continued to decline from 20% to less than 5%. These data collectively point to a negative trend.

[0070] In step S102, the multidimensional user sentiment vector generation module 30 processes these features. The fatigue prediction sub-model 31, based on the continuously declining push notification open rate and decreasing login frequency, determines that user B is in a state of marketing fatigue, outputting a fatigue level F=0.6. The demand prediction sub-model 33 analyzes the user's recent browsing behavior and finds that they have not shown sustained interest in any specific topic or paid content, therefore outputting a demand level N=0.1. The surprise prediction sub-model 32 analyzes and finds that the content categories consumed by the user are very fixed, and interactions (likes, comments, shares) are scarce, outputting a surprise level S=0.2. Thus, a multidimensional user sentiment vector V_sentiment = [F=0.6, S=0.2, N=0.1] representing user B's current state is generated.

[0071] In step S103, the adaptive reward shaping module 40 generates a reward function based on this vector. Since F=0.6 is at a moderate level, the fatigue penalty weight... It is set to a moderately high value. However, because N=0.1 is extremely low, the reward weight is... It is set to a very low value. This means that in this state, the primary goal of the system's decision-making is to avoid increasing fatigue, while the priority of pursuing immediate gains is very low.

[0072] In steps S104 and S105, the reinforcement learning agent 50 evaluates candidate actions in the hierarchical action space.

[0073] Candidate Action A (Low-Intensity Marketing Action): Based on user B's historical profile (e.g., likes sports news), push a recent trending sports news article. Model Prediction: The direct benefit of a single read in news and information applications is extremely low, close to zero. Even though the content might be of interest to users, any proactive push notifications could be perceived as intrusive in the current context. It is a small positive value, such as +0.1. Regular content recommendations, with variations in surprise level. Close to 0. Because Medium to high and If positive, the expected total reward Total_Reward(A) will be a negative value.

[0074] Candidate Action B (Strategic Retention and Cooling-Off Action - Absolute Silence): Executes the strategy of "doing no action," and does not send any proactive push notifications to the user within a preset 24-hour time window. Model Prediction: , , The estimates at the current time step are all 0. Therefore, the expected total reward Total_Reward(B) is 0.

[0075] Candidate Action C (Strategic Retention and Cooling-Off Action - Reduce Frequency): Execute the "Reduce Global Push Frequency" strategy, temporarily lowering User B's push notification level from the default "3 times per day" to "1 time per day". Model Prediction: The current time step is 0. and The value of this action lies not in the instantaneous change, but in its positive impact on the future state. The policy network of the reinforcement learning agent 50 is capable of long-term value prediction. It predicts that by reducing the push frequency, user B's fatigue caused by push notifications will gradually subside over the next few days, meaning that over several time steps... Sum (cumulative) The total reward (C) will be a significantly negative value. Therefore, although the total reward (C) may also be close to 0 at the current time step, its expected long-term cumulative reward (Q value) will be significantly higher than that of actions A and B.

[0076] Ultimately, the reinforcement learning agent 50 selected action C, which had the highest expected long-term value, and modified the push strategy for user B in the user configuration system through the action execution module 60.

[0077] In steps S106 and S107, the action is executed. Over the following week, the data acquisition module 20 observed that although user B received fewer push notifications, their open rate of these notifications gradually recovered, from 5% to 15%. Simultaneously, their frequency of actively opening applications and browsing news also slightly increased. This positive user feedback data was collected and used to iteratively update the model. This successful interaction validated the effectiveness of "strategic white space" in handling moderately fatigued users, reinforcing the agent's tendency to choose a "proactive retreat" strategy in similar situations. This demonstrates that the proposed solution can proactively plan and reserve resources for future effective marketing, thereby protecting and enhancing the long-term value of users.

[0078] Example 3

[0079] This embodiment aims to illustrate how, when users are in a "dormant" state of "low fatigue and low demand," the proposed solution can incentivize "exploratory" actions by adjusting the adaptive reward function, thereby creating surprises, uncovering potential user interests, breaking down information cocoons, and increasing user engagement. The scenario for this embodiment is a short video application.

[0080] Similarly, refer to Figure 1 and Figure 2 The architecture and process.

[0081] Step S101: The data acquisition module 20 detected that user C's recent behavior was stable but lacked deep engagement. Specifically, user C's daily usage time was stable at around 30 minutes, but the frequency of their interactive behaviors such as liking, commenting, and sharing was extremely low. In addition, the system found through content analysis that over 90% of the video content consumed by user C was highly concentrated in a single area, such as "food preparation".

[0082] Step S102: The multi-dimensional user sentiment vector generation module 30 analyzes these data. The fatigue prediction sub-model 31, based on the user's stable usage time and normal response to push notifications, judges the user's fatigue level to be very low, outputting F=0.2. The demand prediction sub-model 33, because the user did not show interest in any paid features or e-commerce links, outputs a demand level of N=0.2. The surprise prediction sub-model 32, based on the user's extremely narrow content consumption range and sparse interaction behavior, judges that the user may be in an "information cocoon" or "content fatigue" state, lacking novelty, and therefore outputs a surprise level of S=0.1. The final generated multi-dimensional user sentiment vector is V_sentiment = [F=0.2, S=0.1, N=0.2].

[0083] Step S103: After receiving this "three lows" vector, the adaptive reward shaping module 40's internal logic is triggered, recognizing that the extremely low surprise level S=0.1 is the main contradiction in the current user state. To solve this problem, the surprise level reward weight in the reward function is adjusted. The calculation method is dynamically adjusted. As an optional implementation method, It can be a function inversely proportional to the surprise level S, such as... , where c and It is a constant. When S is extremely low, this will lead to... The temporary and significant increase indicates that the system prioritizes enhancing user surprise in the current decision-making cycle. Accordingly, and Since both F and N are at low levels, they remain at a low baseline level.

[0084] Steps S104 and S105: The reinforcement learning agent 50 evaluates candidate actions under the guidance of a temporarily adjusted reward function.

[0085] Candidate Action A (Standard Recommendation Action): Following standard collaborative filtering or content recommendation logic, continue recommending a high-quality "food" video to user C. Model Prediction: : Close to 0. Users do not object, and the change is close to zero. The content was as expected, but the level of surprise varied. It is also close to 0. Therefore, the expected total reward Total_Reward(A) is close to 0.

[0086] Candidate Action B (Exploratory Action): The system selects an exploratory action from the hierarchical action space, aiming to break with convention and push content that differs significantly from the user's historical preferences but has recently become popular among other user groups. For example, pushing a short science video about the "latest astronomical telescope observations." Model Prediction: Uncertainty; expected value is 0. The novelty of the content may slightly increase the cognitive load on users, but since users are not fatigued, The estimated value is very small and can be ignored. This action has a certain probability of being ignored by the user, but it also has a considerable probability of bringing the user a strong sense of "surprise". Therefore, the estimated change in surprise level... Expected value E[ ] is a significantly positive value, for example, +0.5. Substituting this into the reward function, since... The value was temporarily increased, and the term +w_s(S) * E[ΔS] became very substantial, making the expected total reward Total_Reward(B) a positive value.

[0087] The reinforcement learning agent 50 therefore chose the exploratory action B, which has a higher expected total reward.

[0088] Steps S106 and S107: Action execution module 60 pushes the technology video to user C. User C sees this unexpected video while swiping through the video stream, is attracted by its visual effects, watches the entire video, and gives it a thumbs up.

[0089] The data acquisition module 20 captures this "like" action, which is a very strong positive feedback signal. In step S107, this feedback is used for system iteration. First, it updates the user C's interest profile, adding the interest tag "technology" or "popular science". Second, this successful "exploration" experience (state, exploration action, high reward, new state) is used to train the policy network of the reinforcement learning agent 50, enabling it to learn to more proactively explore across domains when the user's surprise level is low. Simultaneously, after the user likes, the subsequent evaluation value of their surprise level S will be improved, which will then be further enhanced by the adaptive reward shaping module 40. The weights are reverted to normal levels to prevent the system from engaging in endless, excessive exploration.

[0090] This embodiment demonstrates that the proposed solution can not only handle negative emotions, but also proactively identify and resolve the problem of "experience gaps," creating novel experiences for users by dynamically incentivizing exploratory actions, thereby enhancing users' long-term engagement and platform stickiness.

[0091] Example 4

[0092] This embodiment applies the technical solution provided in this application to a typical e-commerce scenario to demonstrate its adaptability under different business models, and specifically explains the calculation method of the "demand level" dimension and how to make differentiated marketing decisions based on it.

[0093] The scenario is set as the back-end operation system of a large e-commerce website, which also integrates... Figure 1 The modules shown.

[0094] Step S101: The data acquisition module 20 detected a series of shopping behaviors of user D. User D added a pair of "active noise-canceling headphones" from a well-known brand to their shopping cart and proceeded to the checkout page. However, after staying on the checkout page for about 2 minutes, user D abandoned the payment and closed the page. In addition, system records show that the user has received 3 promotional text messages from the online store about the "618 Shopping Festival" in the past two days.

[0095] Step S102: The multi-dimensional user emotion vector generation module 30 interprets these complex behavioral signals. The demand prediction sub-model 33 identifies the behavior pattern of "adding a product to the cart and then abandoning it on the checkout page" as an extremely high demand signal, which usually means that the user is very interested in the product itself, but may hesitate due to external factors such as price, discounts, and delivery. Therefore, the model outputs an extremely high demand level N=0.9. The fatigue prediction sub-model 31 interprets the behavior of "receiving three consecutive promotional text messages in a short period of time" and the user's failure to access the mall through the text message link as the user having developed a high degree of fatigue with the marketing activities, and outputs F=0.7. The surprise prediction sub-model 32 outputs a neutral to low value S=0.3 based on the user's recent regular browsing behavior. Finally, the system generates the current state vector of user D, V_sentiment = [F=0.7, S=0.3, N=0.9], which accurately depicts the complex psychological state of the user with both high purchase intention and high marketing fatigue.

[0096] Step S103: The adaptive reward shaping module 40 generates a reward function based on this vector. Here, a high F=0.7 results in a fatigue penalty weight. Setting it to a high value, and a high N=0.9, results in a higher reward weight. It is also set to a high value. This forces the reward function to make a delicate trade-off between the "high reward incentive" and the "high churn risk".

[0097] Steps S104 and S105: The reinforcement learning agent 50 evaluates multiple candidate actions in its hierarchical action space.

[0098] Candidate Action A (High-Intensity Marketing Action): Immediately push a pop-up window to user D's phone via the app, stating, "Your noise-canceling headphones in your shopping cart are eligible for a ¥10 exclusive coupon!" Model Prediction: There is a high probability of conversion, and the estimated returns are high. When users are already fatigued with marketing, sending highly intrusive pop-up notifications is very likely to cause user resentment or even lead to them uninstalling the application. Very high. Ultimately, the high fatigue penalty may cause Total_Reward(A) to be negative or low.

[0099] Candidate Action B (Low-Intensity Marketing Action): Instead of immediately disturbing the user, display a card in a natural and unobtrusive way in the homepage feed the next time the user naturally opens the app. The card reads, "The noise-canceling headphones you're interested in have been reduced by 20 yuan." Model Prediction: Due to the gentle approach, the conversion rate may be slightly lower than that of Action A, but the estimated revenue is still considerable. This minimally invasive approach presents information when the user actively uses the app, estimating... It is very low, and may even be negative due to the acquisition of useful information. Ultimately, Total_Reward(B) is expected to be a significantly positive value.

[0100] Candidate Action C (Value-Added Cooling Action): Instead of mentioning the price, push a high-quality in-site article or video to the user, titled "In-Depth Review of Five Flagship Noise-Canceling Headphones on the Market," naturally including the headphones in user D's shopping cart and highlighting their advantages. Model Prediction: The immediate return is 0. It provides users with decision-making assistance information; it is a service, not an intrusion. It is negative. This unexpected level of professional content may bring surprises, as predicted. It is positive. Ultimately, Total_Reward(C) is also expected to be a positive value.

[0101] At this point, the reinforcement learning agent 50 faces a choice between two actions (B and C) with positive expected rewards. As an alternative implementation, the agent can further query the user's segmented profile. If historical data shows that user D is a "price-sensitive" consumer (e.g., over 80% of past orders used coupons), the system will determine that action B (direct price reduction information) has the highest conversion efficiency. Conversely, if data shows that user D is a "research-oriented" consumer (e.g., browsing reviews and Q&A communities multiple times before purchasing high-priced items), the system will determine that action C (providing in-depth reviews) is more likely to appeal to the user, build trust, and thus facilitate a purchase. Assuming user D is a research-oriented consumer, the system ultimately chooses action C.

[0102] Steps S106 and S107: Action execution module 60 executes action C. User D received a push notification for the review article, clicked to read it, gained stronger confidence in the headphones' performance and reputation, eliminated their last doubts, and then returned to their shopping cart to complete the purchase. This series of actions and the final conversion result are recorded by the system and used to iteratively update the reinforcement learning strategy and sentiment prediction model, enabling the system to more accurately select value-added cooling actions when facing "research-oriented" and "high-demand, high-fatigue" users in the future.

[0103] This embodiment verifies that the proposed solution can not only perform multi-dimensional emotional state characterization in complex e-commerce scenarios, but also conduct multi-level and differentiated refined operations based on this, thereby effectively improving conversion rates and user satisfaction.

[0104] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An information push method based on reinforcement learning, characterized in that, include: Construct a multidimensional user emotion vector that represents the user's current comprehensive psychological state. The multidimensional user emotion vector contains information from at least three dimensions: fatigue, surprise, and need. Based on the multidimensional user emotion vector, a reward function for reinforcement learning decision-making is dynamically generated. The reward function includes at least terms related to business benefits, changes in user fatigue, and changes in user surprise, and the weight coefficients of each term are determined based on the multidimensional user emotion vector. Based on the reward function, the reinforcement learning agent selects the action with the highest expected total reward as the optimal action in the hierarchical candidate action space. The hierarchical candidate action space includes multiple action space layers, and different action space layers include actions with different purposes or different strategies.

2. The information push method based on reinforcement learning according to claim 1, characterized in that, The construction of a multidimensional user emotion vector representing the user's current comprehensive psychological state includes: The system acquires multi-dimensional state characteristics of users, including interactive behavior characteristics, historical behavior characteristics, marketing feedback characteristics, and social network characteristics. The fatigue prediction sub-model is used to process the multi-dimensional state features and obtain the fatigue level output by the fatigue prediction sub-model. The multi-dimensional state features are processed using the surprise prediction sub-model, and the surprise level output by the surprise prediction sub-model is obtained. The multi-dimensional state features are processed using the demand prediction sub-model, and the demand degree output by the demand prediction sub-model is obtained. The multidimensional user emotion vector is determined using the fatigue level, the surprise level, and the demand level.

3. The information push method based on reinforcement learning according to claim 2, characterized in that, The fatigue prediction sub-model includes a gradient boosting decision tree model or a deep neural network model.

4. The information push method based on reinforcement learning according to claim 1, characterized in that, The weighting coefficients corresponding to the changes in user fatigue are determined using a monotonically increasing function with fatigue as the variable.

5. The information push method based on reinforcement learning according to claim 1, characterized in that, The weighting coefficients corresponding to business revenue are determined using a monotonically increasing function with demand as the variable.

6. The information push method based on reinforcement learning according to claim 1, characterized in that, The tiered candidate operational action space includes a high-intensity marketing action layer, a low-intensity marketing action layer, a value-supplementing cooling action layer, and a strategic blank cooling action layer. Among them, the high-intensity marketing action layer includes forced pop-up actions and homepage banner pushes of high discounts; The low-intensity marketing action layer includes in-feed advertising actions and red dot notification actions; The value-added cooling action layer includes information push actions related to user interests; The strategic white space cooling action layer includes actions such as not pushing information within a preset time window and reducing the information push frequency to below a preset value.

7. The information push method based on reinforcement learning according to claim 2, characterized in that, Also includes: The fatigue prediction sub-model, surprise prediction sub-model, demand prediction sub-model, and reinforcement learning agent are iteratively updated using user feedback data generated after the optimal action is performed. The user feedback data includes changes in user click-through rate, purchase rate, or fatigue level.

8. An information push system based on reinforcement learning, characterized in that, include: The multidimensional user emotion vector generation module is used to construct a multidimensional user emotion vector that represents the user's current comprehensive psychological state. The multidimensional user emotion vector contains information in at least three dimensions: fatigue, surprise, and need. An adaptive reward shaping module is used to dynamically generate a reward function for reinforcement learning decision-making based on the multidimensional user emotion vector. The reward function includes at least terms related to business benefits, changes in user fatigue, and changes in user surprise, and the weight coefficients of each term are determined based on the multidimensional user emotion vector. A reinforcement learning agent is used to select the action with the highest expected total reward as the optimal action in a hierarchical candidate action space based on the reward function. The hierarchical candidate action space includes multiple action space layers, and different action space layers include actions with different purposes and strategies.

9. The information push system based on reinforcement learning according to claim 8, characterized in that, The multi-dimensional user emotion vector generation module is configured to perform the following operations: The system acquires multi-dimensional state characteristics of users, including interactive behavior characteristics, historical behavior characteristics, marketing feedback characteristics, and social network characteristics. The fatigue prediction sub-model is used to process the multi-dimensional state features and obtain the fatigue level output by the fatigue prediction sub-model. The surprise level prediction sub-model is used to process multi-dimensional state features and obtain the surprise level output by the surprise level prediction sub-model. The demand prediction sub-model is used to process the multi-dimensional state features and obtain the demand degree output by the demand prediction sub-model. The multidimensional user emotion vector is determined using the fatigue level, the surprise level, and the demand level.

10. A computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the information push method based on reinforcement learning as described in any one of claims 1 to 7.

Images