Reinforcement learning-based recommendation method, system and storage medium

Abstract

Embodiments of the present application provide a recommendation method and system based on reinforcement learning and a storage medium, relating to the technical field of artificial intelligence. The method comprises: obtaining state information of a target object, the state information comprising static features and interaction features; inputting the state information into a reinforcement learning model to generate an initial recommendation action; obtaining marketing strategy information, the marketing strategy information comprising product types and strategy weights corresponding to each product type; weighting and adjusting the initial recommendation action according to the strategy weights to generate a target recommendation action; executing the target recommendation action to output a recommended product; generating a reward signal according to interaction feedback corresponding to the recommended product, and updating the reinforcement learning model. The embodiments of the present application realize the collaborative optimization of intelligent decision-making and business rules by fusing marketing strategy weights with reinforcement learning recommendation results, while meeting the preferences of target objects and taking into account business marketing goals.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a recommendation method, system, and storage medium based on reinforcement learning. Background Technology

[0002] In financial recommendation scenarios such as mobile banking, users' demand for personalized services is growing. How to meet user preferences while also taking into account the bank's marketing goals has become an urgent problem to be solved in the current fintech field.

[0003] Currently, mainstream financial product recommendation solutions mainly fall into two categories: one is rule-based static recommendation, which filters recommended content through predefined business rules (such as product type priority and customer tag matching), and the system directly calls the corresponding recommended product list based on customer tags; the other is recommendation based on traditional machine learning models, which makes recommendations through collaborative filtering or deep learning prediction results.

[0004] However, all of the above solutions have limited intelligence and cannot adapt recommendation strategies to changes in user preferences. Summary of the Invention

[0005] This application provides a recommendation method, system, and storage medium based on reinforcement learning to solve the technical problem that existing recommendation schemes cannot adapt to changes in user preferences due to limited intelligence.

[0006] In a first aspect, embodiments of this application provide a recommendation method based on reinforcement learning, the method comprising:

[0007] Obtain the state information of the target object, which includes static features and interactive features;

[0008] The state information is input into the reinforcement learning model, which calculates the expected recommendation value of each product and generates an initial recommendation action based on the expected recommendation value.

[0009] Obtain marketing strategy information, which includes product types and the corresponding strategy weights for each product type; adjust the expected value of the initial recommendation action based on the strategy weights to generate the target recommendation action;

[0010] Perform the target recommendation action and output recommended products;

[0011] Reward signals are generated based on the interactive feedback of the recommended products, and the reinforcement learning model is updated based on the reward signals.

[0012] In this embodiment, the state information of the target object is processed by a reinforcement learning model, and the sequential decision-making capability of reinforcement learning is used to generate the initial recommendation action. This achieves an upgrade from static rule filtering to intelligent dynamic decision-making, solving the problems of limited intelligence and reliance on simple rule filtering in existing recommendation mechanisms, which lack in-depth analysis capabilities. At the same time, by acquiring preset marketing strategy information, the expected value of the initial recommendation action is weighted and adjusted according to the strategy weight corresponding to the product type to generate the target recommendation action. This ensures that the final recommendation result satisfies the target object's preferences while taking into account business marketing objectives. This avoids the problem of pure AI recommendation models deviating from marketing strategies due to the lack of business objective constraints, resulting in uncontrollable recommendation results. This achieves an effective integration of intelligent decision-making and business rules.

[0013] Based on this, a reward signal is generated by collecting the interaction feedback of the target user with the recommended product. The reinforcement learning model is then continuously updated based on this reward signal, enabling the recommendation strategy to adapt to changes in the target user's preferences and actual feedback. This overcomes the problems of poor adaptability and inability to capture real-time changes in user preferences inherent in traditional static recommendation models. Thus, this embodiment of the application achieves an upgrade from static rule filtering to intelligent dynamic decision-making, resulting in improved recommendation accuracy and target user response rate.

[0014] In one possible embodiment, the expected recommendation value of the initial recommendation action is weighted and adjusted according to the policy weights to generate the target recommendation action, including:

[0015] The weight coefficient for each product type is determined based on the strategy weight corresponding to each product type.

[0016] The expected recommendation value of each product in the initial recommendation action is weighted according to the weight coefficient to obtain the weighted expected recommendation value.

[0017] Based on the weighted expected value of recommendations, the recommendation priority of each product is determined, and the target recommendation action is generated.

[0018] In this implementation, the marketing strategy weights at the business level are transformed into calculable weight coefficients. These coefficients are then used to weight and reorder the expected values ​​of product recommendations output by the reinforcement learning model. This allows the strategy weights to quantitatively influence the final recommendation results, achieving an upgrade from qualitative weighting to quantitative calculation. Consequently, the strategy fusion process possesses clear calculability and reproducibility. The recommendation results accurately reflect the relative levels of business strategy weights, improving the precision and consistency of the strategy fusion results, as well as the accuracy of the business strategies' control over the recommendation results.

[0019] In one possible embodiment, obtaining the state information of the target object includes:

[0020] Based on the unique identifier of the target object, the raw data is extracted from multiple business systems, including at least the core business system, the credit system, the wealth management system, and the credit card system.

[0021] Perform feature alignment and stitching on the raw data to build a unified customer view;

[0022] Static and interactive features are extracted based on a unified customer view.

[0023] This implementation method associates and extracts raw customer data from multiple independent business systems, such as the bank's core business system, credit system, wealth management system, and credit card system, through the unique identifier of the target object. This solves the problem of data silos, where customer data is scattered and cannot be effectively integrated within the bank. Through feature alignment and splicing, the differences in data format, field definition, and encoding rules between different systems are eliminated, and multi-source heterogeneous data is integrated into a unified customer view. On this basis, static features and interactive features are extracted from the unified customer view, providing a complete customer panoramic data foundation for reinforcement learning models.

[0024] In this way, the input state of the reinforcement learning model no longer depends on the local data of a single system, but is based on a unified customer view after multi-source fusion, which improves the completeness and accuracy of the state information, provides comprehensive data support for subsequent recommendation decisions, and solves the problem of insufficient recommendation accuracy caused by data silos.

[0025] In one possible embodiment, the state information also includes context features;

[0026] Obtaining the state information of the target object also includes:

[0027] Obtain the scene information of the target object. The scene information includes at least one of the following: login time, geographical location, device type, and current operation page.

[0028] Extract contextual features based on scene information;

[0029] State information is constructed by combining static features, interactive features, and contextual features.

[0030] In this implementation, by introducing contextual features into the state information, the state input of the reinforcement learning model is expanded from a two-dimensional representation of user attributes and historical behavior to a three-dimensional representation of user attributes, historical behavior, and the real-time scene. By acquiring multi-dimensional scene information such as the target object's login time, geographical location, device type, and current operating page, and extracting contextual features, the recommendation system can perceive the target object's current environmental context and capture the differences in needs under different scenarios. Thus, the generation of recommendation results no longer relies solely on the target object's historical behavior and static attributes, but can also adapt to its current real-time scene. This solves the problem of traditional recommendation systems lacking scene awareness, leading to a mismatch between recommendation results and real-time needs, and further improves recommendation accuracy and user experience.

[0031] In one possible embodiment, a reward signal is generated based on the interactive feedback corresponding to the recommended product, including:

[0032] Obtain the types of feedback from the interaction, including positive feedback and negative feedback;

[0033] Generate a reward signal by matching the corresponding reward value to the feedback type;

[0034] In this context, the reward value for positive feedback is positive, and the reward value for negative feedback is negative, with different reward values ​​corresponding to different feedback types.

[0035] In this implementation, by qualitatively classifying the interactive feedback of the target object into positive and negative categories, the reinforcement learning model can clearly distinguish between positive incentives and negative penalties, ensuring that the model training always iterates in the correct optimization direction. By matching different reward values ​​to different feedback types, positive feedback behaviors such as purchasing, clicking, and saving receive differentiated positive reward values, while rejection behaviors and quick skipping behaviors receive negative or zero reward values. This further distinguishes the importance differences of different feedback behaviors in the same direction, enabling the model to prioritize high-value behaviors (such as actual purchases) and reduce overfitting to low-value behaviors.

[0036] In this way, the reward signal not only provides clear guidance for optimization but also has differentiated value weights, enabling precise quantification of user feedback. This provides a more accurate basis for strategy optimization for reinforcement learning models, improving the convergence speed of model training and the recommendation effect.

[0037] In one possible embodiment, positive feedback includes at least one of click behavior, favorite behavior, share behavior, rating behavior and purchase behavior; negative feedback includes at least one of rejection behavior and quick skip behavior.

[0038] In this implementation, behaviors expressing user interest and positive responses, such as clicking, saving, sharing, rating, and purchasing, are categorized as positive feedback, while behaviors expressing user rejection and negative responses, such as refusing and quickly skipping, are categorized as negative feedback. This clarifies the source boundaries of positive and negative reward signals in the reinforcement learning model. In this way, the model can accurately identify the feedback tendency based on the specific type of user interaction behavior. Positive feedback indicates that the recommended product meets or exceeds user expectations, and the model should strengthen this type of recommendation strategy; negative feedback indicates that the recommended product does not meet user expectations, and the model should adjust or avoid this type of recommendation strategy in a timely manner. By establishing clear boundaries for positive and negative behavior classification, the consistency of reward signal generation is improved, enabling the model to obtain clear and stable optimization direction guidance during training, further enhancing the stability and convergence efficiency of model training.

[0039] In one possible embodiment, a reward signal is generated based on the interactive feedback corresponding to the recommended product, and the reinforcement learning model is updated based on the reward signal, including:

[0040] During the target user's usage, continuously collect interaction feedback corresponding to the recommended products;

[0041] Update the state information of the target object based on interactive feedback;

[0042] Recommended products are regenerated based on the updated status information, and the reinforcement learning model is continuously updated iteratively.

[0043] In this implementation, a real-time data channel is established between the recommendation system and the target object by continuously collecting interactive feedback during the target object's use, enabling the system to perceive every user behavior; by updating the status information in real time based on interactive feedback, the latest user behavior is incorporated into the status information, so that the status information can reflect the user's current preferences; by regenerating recommended products based on the updated status and iteratively updating the model, a complete closed loop of continuous collection, dynamic updating, regeneration, and iterative optimization is formed.

[0044] In this way, the recommendation strategy can evolve with the real-time changes in user preferences, realizing an upgrade from static recommendation to continuous evolution. This solves the problem that traditional static models cannot capture the dynamic changes in user preferences, enabling the recommendation system to have adaptive continuous optimization capabilities and further improving the accuracy of recommendation results.

[0045] In one possible embodiment, the reinforcement learning model is a Q-learning-based neural network model.

[0046] In this implementation, a Q-learning-based neural network model is used as the reinforcement learning model. The Q-learning algorithm provides a mature theoretical framework, enabling the model to progressively optimize the recommendation strategy through continuous interaction with the target object. The neural network, acting as a Q-function approximator, can effectively extract and generalize features from high-dimensional state information, including static features, interaction features, and contextual features, solving the problem that traditional Q-learning lookup methods cannot effectively handle high-dimensional state information. Thus, the combination of Q-learning's policy optimization capability and the neural network's feature representation capability gives the reinforcement learning model stable policy optimization capabilities and adaptability to complex recommendation scenarios, improving the training efficiency and generalization ability of the recommendation system.

[0047] Secondly, embodiments of this application provide a recommendation system based on reinforcement learning, the system comprising:

[0048] The state acquisition module is used to acquire the state information of the target object, which includes static features and interactive features.

[0049] The reinforcement learning module is used to receive state information, calculate the expected recommendation value of each product, and generate initial recommendation actions based on the expected recommendation value.

[0050] The strategy fusion module is used to acquire marketing strategy information, which includes product types and the corresponding strategy weights for each product type. Based on the strategy weights, the expected value of the initial recommendation action is weighted and adjusted to generate the target recommendation action.

[0051] The output module is used to perform the target recommendation action and output recommended products;

[0052] The update module is used to generate reward signals based on the interactive feedback of the recommended products, and to update the reinforcement learning module based on the reward signals.

[0053] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided above.

[0054] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method provided above. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0056] Figure 1 The system core logic flowchart provided in the embodiments of this application;

[0057] Figure 2 This is a flowchart illustrating the reinforcement learning-based recommendation method provided in an embodiment of this application.

[0058] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0059] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0060] It should be noted that the recommendation method, system, and storage medium based on reinforcement learning provided in this application can be used in the field of artificial intelligence technology, or in any field other than artificial intelligence technology. This application does not limit the application field of the recommendation method, system, and storage medium based on reinforcement learning.

[0061] Specific application scenarios for this application include product recommendations, intelligent marketing, and customer guidance in financial service terminals such as mobile banking. In these scenarios, users' demand for personalized financial services is increasing, while banks and other financial institutions need to balance meeting user preferences with their own marketing goals. Finding a balance between intelligent recommendations and business objectives is a major challenge currently facing financial recommendation scenarios.

[0062] For example, in mobile banking homepage recommendation scenarios, users expect to receive customized product recommendations based on their asset status, risk appetite, and real-time operational behavior when logging into the mobile banking app; in promotional marketing scenarios, users expect to receive product pushes that match their current needs and align with the bank's marketing strategies when visiting promotional pages or product zones; and in wealth management advisory scenarios, users expect to receive advisory services with reasons for recommendation. In these scenarios, technical solutions capable of intelligent decision-making and collaborative optimization of business rules are needed.

[0063] For the above application scenarios, existing technologies typically employ the following two methods for recommending financial products.

[0064] In rule-based static recommendation, the system pre-configures a standardized business rule library, containing recommendation rules and corresponding triggering conditions (such as product type priority and customer tag matching) for different product types (e.g., deposits, wealth management, loans, credit cards). When the system detects that a user has accessed a specific page (e.g., mobile banking homepage, activity page, product zone) or matched specific customer tags, it matches the corresponding recommended product list according to a preset mapping relationship and directly retrieves standardized recommendation content from the product library to return to the user. In this approach, the recommendation rules are configured during the product design phase, and the rule content remains fixed after deployment, ensuring that users receive the same recommendation results every time the same scenario is triggered.

[0065] In traditional machine learning-based recommendation methods, the system inputs the user's historical transaction data, basic attribute information, and product holdings into a collaborative filtering or deep learning model. The model then calculates the user's preference rating for each product based on predictions obtained through offline training, and outputs a recommendation list according to the ratings. In this method, the recommendation logic relies on offline training on historical data. Once the model is trained, its parameters are fixed and cannot be learned and adjusted in real time during user interaction.

[0066] Regarding the aforementioned existing technologies, rule-based static recommendation methods rely on statically configured business rules and customer tag matching for recommendation logic, failing to comprehensively handle real-time changes in user behavior and fluctuations in preferences. When a user's asset status, browsing behavior, or stage-specific needs change, the system still outputs fixed recommendation results according to preset rules, making it difficult to reflect the user's immediate needs in different scenarios in a timely manner. Furthermore, static recommendation rules, once fixed during the product design phase, cannot be dynamically adjusted, outputting the same list of recommended products each time the same scenario is triggered, making it difficult to adapt to the differences in preferences among different users.

[0067] In traditional machine learning-based recommendation methods, the recommendation logic relies on offline training using historical data. Once the model is trained, its parameters are fixed and cannot be learned and adjusted in real time during user interaction. User interactions such as clicking, ignoring, and purchasing recommended products are difficult to continuously translate into optimization criteria for subsequent recommendation strategies. This results in recommendation outcomes that tend to remain at the historical judgment level, failing to evolve with changes in user preferences and exhibiting insufficient adaptability.

[0068] Furthermore, neither of the above two methods fully utilizes customer data from multiple banking systems (such as core business systems, credit systems, wealth management systems, and credit card systems). Customer information is stored in a scattered manner, making it difficult to build a complete and unified customer view. This results in recommendations being based on a single criterion and causing a discrepancy between the recommended content and the user's actual needs.

[0069] Furthermore, existing recommendation schemes typically focus solely on user satisfaction while neglecting the bank's business and marketing objectives. This makes it difficult to synergize recommendation results with business goals, resulting in ineffective coordination between recommendations and bank marketing strategies. Consequently, this not only reduces recommendation accuracy but also increases invalid exposure, impacting users' decision-making efficiency and product delivery effectiveness in mobile banking scenarios.

[0070] In view of this, this application provides a recommendation method based on reinforcement learning. Its core concept is to simultaneously consider user preferences and business marketing goals in recommendation decision-making by combining policy weight fusion with reinforcement learning feedback updates, thereby achieving collaborative optimization of intelligent decision-making and business rules.

[0071] Specifically, in traditional recommendation schemes, user preference modeling and business objective constraints are relatively independent. Rule-driven approaches focus on achieving business objectives but neglect user preferences, while pure model-driven approaches focus on matching user preferences but neglect business objectives; neither can achieve synergistic optimization. This application's embodiment introduces marketing strategy weights into the reinforcement learning decision-making chain, enabling business objectives to participate in the generation of recommendation actions in a computable manner at the decision-making source, rather than being post-processed constraints after the recommendation results are produced. This achieves a deep integration of intelligent decision-making and business rules. Simultaneously, through a reward signal-driven feedback update mechanism, user interaction behavior is continuously transformed into the basis for model optimization, allowing the recommendation strategy to dynamically evolve with changes in user preferences.

[0072] In this way, the strategy weighting integration ensures that the recommendation results meet both user preferences and business marketing goals; the feedback and update mechanism enables the recommendation strategy to have continuous optimization and dynamic adaptation capabilities, thereby balancing recommendation accuracy with the achievement of business objectives.

[0073] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0074] First, combine Figure 1 The system architecture and core logic flow of the embodiments of this application are described. Figure 1 This is a flowchart illustrating the core logic of the system provided in an embodiment of this application. Figure 1 As shown, the system uses a reinforcement learning framework to generate and continuously optimize recommendation strategies. Its core modules include a state acquisition module, a reinforcement learning module, a policy fusion module, an output module, and an update module. Figure 1In the diagram, S represents the state information of the target object, which includes static features and interactive features; a represents the recommendation action, i.e., the candidate product; r represents the reward signal, used to quantify the target object's interactive feedback to the recommended product; S′ represents the next state that the target object transitions to after executing the recommendation action; θ represents the parameters of the current Q network; and θ′ represents the parameters of the target Q network.

[0075] Specifically, in the real-time recommendation phase, the state information S of the target object is input into the reinforcement learning system. The current Q-network calculates the expected recommendation value of each product and generates recommendation actions based on the expected recommendation value. The system then outputs the results to the target audience. Simultaneously, it retrieves pre-defined marketing strategy information, including product types and corresponding strategy weights for each product type. Based on these strategy weights, it weights and adjusts the expected value of the initial recommendation action to generate the target recommendation action.

[0076] The target user's interaction with the recommended product generates feedback. The system generates a reward signal *r* based on this feedback, which is used to calculate the error function of the current Q-network. The error function is based on... The difference between the current value and the target value is used to calculate the loss, which is then used to update the current Q-network parameters θ. Experience samples from each interaction... The parameters are stored in the experience storage pool for subsequent sampling of mini-batch samples and network updates. The parameters θ of the current Q-network are periodically synchronized to the target Q-network θ′ to stabilize the training process.

[0077] Thus, the system forms a complete closed loop from state perception, recommendation generation, strategy fusion, feedback collection, experience storage to model updating.

[0078] The following is combined Figure 2 The recommendation method based on reinforcement learning, as described in the embodiments of this application, will be introduced. Figure 2 This is a flowchart illustrating the reinforcement learning-based recommendation method provided in an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0079] S201. Obtain the state information of the target object, which includes static features and interactive features.

[0080] Static features are used to characterize the relatively stable long-term attributes of the target object (such as age, risk level, registration duration, etc.), while interactive features are used to characterize the target object's historical behavior and recent operation records (such as historical transactions, product holdings, past behavioral trajectories, etc.). In this embodiment, after receiving a recommendation request, the recommendation service program extracts relevant data from the bank's big data platform based on the target object's unique identifier, constructs state information containing both static and interactive features, and uses this information as input to the reinforcement learning model.

[0081] S202. Input the state information into the reinforcement learning model, which calculates the expected recommendation value of each product and generates the initial recommendation action based on the expected recommendation value.

[0082] The reinforcement learning model is a Q-learning-based neural network model. This model consists of an input layer, several hidden layers, and an output layer. The dimension of the input layer matches the dimension of the state information, and the dimension of the output layer is the same as the number of candidate products. The output is the expected recommendation value of each product. The expected recommendation value represents the predicted response of the target object to each candidate product in the current state. The higher the value, the more the model predicts that the product better meets the target object's needs in the current state. The initial recommendation action is the basic recommendation result output by the reinforcement learning model before the application of business marketing constraints.

[0083] In this embodiment of the application, after the state information constructed in S201 is input into the reinforcement learning model, the model outputs the expected recommendation value corresponding to each candidate product, and the system generates an initial recommendation action based on the expected recommendation value.

[0084] S203. Obtain marketing strategy information, which includes product types and the corresponding strategy weights for each product type; adjust the expected value of the initial recommendation action based on the strategy weights to generate the target recommendation action.

[0085] Among them, product type is used to identify the business category to which the candidate product belongs (such as deposit, wealth management, loan, credit card, etc.), and strategy weight is used to characterize the recommendation priority of each product type in the current marketing cycle. In this embodiment of the application, when the recommendation service program receives a recommendation request, it reads the currently valid marketing strategy information. The system performs weighted processing on the expected recommendation value of each product in the initial recommendation action according to the strategy weight corresponding to each product type, obtains the weighted expected recommendation value, and determines the recommendation priority of each product based on the weighted expected recommendation value, thereby generating the target recommendation action.

[0086] S204. Execute the target recommendation action and output the recommended product.

[0087] The recommendation service program converts the target recommendation action generated by S203 into a displayable recommendation result, and outputs the recommended products in the order determined by the target recommendation action, which are then displayed on the mobile banking interface.

[0088] S205. Generate a reward signal based on the interactive feedback corresponding to the recommended product, and update the reinforcement learning model based on the reward signal.

[0089] Interactive feedback includes behaviors such as clicking, browsing, purchasing, ignoring, or rejecting recommended products by the target user. Reward signals are used to quantify the interactive feedback into numerical values ​​that can be used for model learning, driving the continuous updating of the reinforcement learning model.

[0090] In this step, the system acquires the interactive feedback corresponding to the recommended product, matches the corresponding reward value according to the feedback type, and generates a reward signal. Based on the reward signal, the loss function is calculated, and the network parameters of the reinforcement learning model are updated using gradient descent, allowing the model output to gradually approach the target Q-value. Experience samples from each interaction are stored in an experience storage pool. During training, small batches of samples are randomly sampled from the experience storage pool to update the network, enabling the recommendation strategy to continuously optimize as the target object's behavior changes.

[0091] Through the above steps, this application embodiment achieves dynamic coordination between recommendation strategies, user preferences, and business objectives. Specifically, by acquiring state information containing static and interactive features, a data foundation reflecting the true needs of the target audience is provided for recommendation decisions; by mapping state information to the expected recommendation value of each product and generating initial recommendation actions through a reinforcement learning model, an upgrade from static rule matching to intelligent dynamic decision-making is achieved; by acquiring marketing strategy information and adjusting the expected recommendation value according to strategy weights, the recommendation results satisfy the preferences of the target audience while taking into account business marketing objectives, achieving collaborative optimization of intelligent decision-making and business rules; by executing target recommendation actions and outputting recommended products, a traceable recommendation display chain is formed; by continuously updating the model through reward signals driven by interactive feedback, the recommendation strategy can dynamically evolve with changes in the behavior of the target audience, improving the problem that static models are difficult to adapt to real-time changes in preferences.

[0092] In summary, the embodiments of this application enable the recommendation results to be continuously corrected during the interaction process and to maintain a response to the current state, thereby achieving the technical effect of improving the accuracy of recommendations and the response rate of the target object.

[0093] Based on the aforementioned embodiments, the strategy weight fusion process will be further described in detail.

[0094] The fusion process includes three stages: determining weight coefficients, weighted processing, and priority ranking. The product types in the marketing strategy information can include deposits, wealth management, loans, and credit cards. The corresponding strategy weights are pre-configured by the bank based on the activity cycle, business objectives, and product inventory status, and written into the strategy parameter table. After receiving the initial recommendation action output by the reinforcement learning model, the system first reads the corresponding strategy weights according to the product type, and then converts the strategy weights into weight coefficients through normalization mapping for multiplication with the expected recommendation value. For multiple candidate products under the same product type, the weight coefficients can remain consistent, or they can be subdivided based on product availability, term limits, or activity tags; this embodiment does not limit this.

[0095] The system then performs a weighted processing on the expected recommendation value of each product in the initial recommendation action, multiplying the original expected recommendation value of each product by its corresponding weight coefficient to obtain the weighted expected recommendation value. If a product has a high original expected recommendation value but its product type has a low strategy weight, the weighted expected recommendation value will decrease accordingly; conversely, if a product aligns with the current marketing focus and has a high weight coefficient, its weighted expected recommendation value will increase.

[0096] After weighting, the system determines the recommendation priority of each product based on the weighted expected recommendation value. Products with higher priority are placed in the recommendation position on the mobile banking homepage, the first screen of the activity page, or at the top of the product zone, thereby generating the target recommendation action.

[0097] Using the above method, the initial recommendation action is no longer directly used as the final output. Instead, it is adjusted through weight coefficients corresponding to product types, allowing the ranking of different products in the recommendations to be updated as the marketing strategy changes. Therefore, the target recommendation action can maintain the model's predictive ability while reflecting the current strategy orientation, ensuring that the recommendation results are consistent with the product promotion goals.

[0098] In one possible implementation, when obtaining the state information of the target object, raw data is first extracted from multiple business systems based on the unique identifier of the target object. These business systems include at least the core business system, the credit system, the wealth management system, and the credit card system.

[0099] Because of differences in field naming and data granularity across various business systems, the system first aligns the fields in the original data according to a pre-defined mapping relationship. For example, customer levels in the core business system might be identified as "A / B / C / D," risk assessment results in the wealth management system might be identified as "R1 / R2 / R3 / R4 / R5," and credit limits in the lending system might be in ten thousand yuan. The naming and coding rules for the same customer identifier differ across systems. The system then uniformly converts these fields to the same standard based on the pre-defined mapping relationship, standardizing dimensions such as account status, amount, term, and frequency into a unified format and unit.

[0100] After feature alignment is complete, the system then concatenates the aligned data by customer ID, merging data from different business systems into a single complete customer record, forming a unified customer view covering multiple business domains. This unified customer view can be stored in the customer profile table of the data platform for direct retrieval later.

[0101] Based on this unified customer view, the system can further extract two types of features: static features, which reflect long-term stable customer attributes such as age, gender, education level, and risk level; and interactive features, which reflect recent changes in customer behavior such as login frequency, product click records, browsing duration, and purchase or redemption operations of products for sale within the past week. These two types of features together constitute the state information of the reinforcement learning model.

[0102] Through the above processing, scattered data in multiple business systems are linked and integrated under the same identifier, and information from different sources is transformed into state information that can be directly used for recommendation modeling. This enables subsequent models to utilize both basic customer attributes and real-time interactive changes, improving the completeness and consistency of state information. It can more accurately represent the current state of the target object and provide more sufficient data basis for generating recommendation actions.

[0103] Building upon the foregoing embodiments, the state information further includes contextual features. When acquiring the state information of the target object, the scene information of the target object is also acquired, and contextual features are extracted based on the scene information. The static features, interaction features, and contextual features are then combined to construct the state information.

[0104] In this embodiment, after receiving an access request from the target object, the system first collects scenario information from the session log and terminal-reported information, and then preprocesses the collected scenario information. The login time is converted into the corresponding time period identifier (such as "weekday daytime" or "weekday nighttime"), the geographical location is converted into geocoding, the device type is converted into the terminal category identifier (such as mobile terminal or tablet terminal), and the current operation page is converted into the page identifier (such as homepage, finance page, loan page, or credit card page, etc.).

[0105] After preprocessing, the system extracts features from the scene information according to preset mapping rules, converting the discretized scene identifiers into context feature vectors that can be used as input to the model. For example, the time period identifier "weekday daytime" can be mapped to a fixed-length numerical vector, and the page identifier "financial management page" can be mapped to another numerical vector. These vectors together constitute the context feature vector.

[0106] When combining state information, the system aligns and concatenates static features, interaction features, and contextual features in a preset order to form complete state information of the target object. Contextual features describe the current access environment (e.g., access time, location, device type, and currently viewed page category). All three serve as input to the reinforcement learning model, and the dimension of the model's input layer matches the total dimension after concatenation.

[0107] By incorporating contextual features into state information, reinforcement learning models can simultaneously perceive changes in the current scenario when calculating the expected recommendation value of each product. For example, when a user visits a financial management page during the daytime on a weekday, the model can prioritize recommending financial management-related products based on their current time preferences and page category; when a user visits the homepage on a weekend night, the model can recommend products more suitable for a leisure browsing scenario based on their time characteristics and device type. In this way, state information not only includes historical attributes and interaction records but also current environmental factors, enabling the recommendation model to distinguish the differences in user needs across different times, locations, devices, and pages, thereby improving the consistency between recommendation results and the current scenario.

[0108] Building upon the aforementioned embodiments, further, when generating a reward signal based on the interactive feedback corresponding to the recommended product, the feedback type of the interactive feedback is first obtained, and then the corresponding reward value is matched according to the feedback type to generate the reward signal. The feedback types include positive feedback and negative feedback. The reward value for positive feedback is positive, and the reward value for negative feedback is negative, with different reward values ​​corresponding to different feedback types.

[0109] In this implementation, after the recommended product is output to the target user, the system obtains interactive feedback based on front-end tracking or server-side feedback information and identifies the type of feedback behavior. During identification, the system matches key user behaviors within a preset time window with a feedback rule base to determine whether the interactive feedback is positive or negative. Positive feedback includes user actions such as clicking, saving, sharing, rating, or purchasing the recommended product, while negative feedback includes user actions such as explicitly rejecting or quickly skipping the recommended product.

[0110] After determining the feedback type, the system matches the corresponding reward value based on the feedback type and generates a reward signal. The reward value for positive feedback is set to a value greater than zero, and the reward value for negative feedback is set to a value less than zero. Different feedback types correspond to different reward values. For example, the reward value for a purchase is greater than the reward value for a click, and the reward value for an explicit rejection is negative, with its absolute value greater than the absolute value of the negative reward for a quick skip. The reward value can be obtained directly from a preset parameter table; this embodiment does not limit this.

[0111] After the reward signal is generated, the system outputs it to the reinforcement learning model to adjust the value assessment result corresponding to the current recommended action. Since positive feedback corresponds to positive values, the model will increase the preference for actions related to the recommended product; since negative feedback corresponds to negative values, the model will decrease the selection probability of the corresponding action. By assigning different reward values ​​to different feedback types, the system can distinguish the importance of various interactive behaviors, making the model update process more closely aligned with the differences in product clicks, browsing, and purchases in mobile banking scenarios.

[0112] In this way, interactive feedback is stably converted into a unified numerical reward signal. The reinforcement learning model then continuously adjusts the recommendation strategy accordingly, ensuring that subsequent recommendation results are consistent with the user's true preferences and improving the ability to distinguish between positive and negative responses.

[0113] Based on the foregoing embodiments, positive feedback further includes at least one of click behavior, favorite behavior, share behavior, rating behavior and purchase behavior; negative feedback includes at least one of rejection behavior and quick skip behavior.

[0114] In this implementation, the mobile banking front-end interface encapsulates interactive events such as clicks, favorites, shares, ratings, purchases, rejections, and quick skips of target objects into unified feedback messages, which are then transmitted to the back-end reward calculation unit via a message queue. The reward calculation unit configures corresponding reward values ​​based on different feedback types. The positive reward value for purchases is higher than that for clicks, for favorites it is higher than that for ratings, and for sharing and ratings it can be configured with moderate positive reward values. The absolute value of the negative reward value for rejections is higher than that for quick skips. This mapping relationship can be stored in a configuration table and updated synchronously with marketing strategies.

[0115] After the reward signal is generated, the reinforcement learning model accumulates user interest signals based on positive reward values ​​and receives rejection signals based on negative reward values, adjusting the recommendation strategy accordingly. Since high-value behaviors such as purchasing and adding to favorites increase the expected recommendation value of corresponding products, while behaviors such as rejection and quickly skipping inhibit subsequent exposure to similar products, the model can more stably align with the target user's current preferences.

[0116] In this way, the recommendation system can distinguish between explicit preferences and explicit rejections of the target object, giving the reward signal a more granular discriminative ability, providing a more sufficient basis for model updates, thereby improving the matching degree between the recommendation results and the user's true interests, and reducing the output of recommendation content that is irrelevant to the user's needs.

[0117] Based on the aforementioned embodiments, the real-time update process of the reinforcement learning model will be further explained.

[0118] In this implementation, after the target user logs into the mobile banking app, the system continuously records their interactions with recommended products, such as clicking, adding to favorites, purchasing, rejecting, or quickly skipping them, and writes the corresponding events to the interaction log. The interaction log is then timestamped and aggregated by the recommendation service to generate a feedback feature vector, which is then fused with the static features, interaction features, and contextual features in the original state information to obtain the updated state information.

[0119] The updated state information is re-input into the reinforcement learning model, which then recalculates the expected recommendation value of each product and generates a new set of recommended products, which is then output to the mobile banking interface again. Each complete cycle of "state update → recommendation recalculation → result output" represents one round of real-time interaction.

[0120] During this process, the system matches the corresponding reward value based on the type of interactive feedback and generates a reward signal. Positive feedback (such as clicks, favorites, and purchases) corresponds to positive rewards, while negative feedback (such as rejection and quick skipping) corresponds to negative rewards. Different feedback types correspond to different reward values.

[0121] After each interaction, the system organizes the current state S, the action a (i.e., the recommended product), the reward r, and the next state S′ after the action into an experience sample. The experience sample is then stored in the experience storage pool.

[0122] During model updates, the system randomly samples a small batch of experience samples from the experience storage pool. For each sample, it calculates the error between the Q-value of the current Q-network performing action a in state S and the target Q-value. Based on this error, it calculates the loss function and then updates the parameters θ of the current Q-network using gradient descent, gradually bringing the model output closer to the target Q-value. The parameters θ of the current Q-network are periodically synchronized to the target Q-network θ′ to stabilize the training process. If a new round of recommendations continues to generate interactive feedback, the system repeats the state update, experience storage, and model parameter correction, forming a continuous iterative optimization closed loop.

[0123] In this way, interactive feedback is continuously transformed into state increments, and the recommendation strategy is continuously evolved according to the cyclical process of state update, recommendation recalculation, experience storage, and model update. The reinforcement learning model can continuously correct recommendation preferences based on real-time behavioral changes in mobile banking scenarios, so that subsequent recommendations are consistent with the current needs of the target object.

[0124] Based on the aforementioned embodiments, the specific types of reinforcement learning models will be further explained.

[0125] In this embodiment, the reinforcement learning model is a neural network model based on Q-learning. Q-learning is an offline reinforcement learning algorithm that does not rely on an environment model. Its core idea is to learn a Q-function to evaluate the expected cumulative reward of performing each action in a given state. In this embodiment, the Q-function is represented as... Q is used to characterize the expected long-term reward that a target user can obtain when choosing product a as the recommendation result in the current state s, where s represents the state information and a represents the recommendation action. The higher the Q value, the better the user feedback is expected to be obtained when recommending the product in the current state.

[0126] The model comprises an input layer, several hidden layers, and an output layer. The input layer receives state information, including static features, interaction features, and contextual features. These features are normalized and vectorized, then concatenated into a fixed-length input vector. The dimension of the input layer is the same as the dimension of this input vector. The hidden layers employ a fully connected structure, using non-linear activation functions to perform layer-by-layer feature transformations on the input, extracting high-dimensional abstract features from the state information. The output layer's dimension is the same as the number of candidate products, and each output node corresponds to the Q-value of a candidate product, i.e., the expected recommendation value.

[0127] After the system inputs the current state information into the model, the model calculates and outputs the Q-value of each candidate product through forward propagation, which serves as the expected recommendation value. The system then adjusts the expected recommendation value according to the policy weights to generate the target recommendation action. After each recommendation is completed, the system generates a reward signal based on the interaction feedback of the target object, calculates the loss function based on the reward signal, and updates the connection weights and bias parameters of each layer of the model through backpropagation and gradient descent, so that the estimated Q-value gradually approaches the true target Q-value.

[0128] By iterating through the parameters of a neural network to achieve continuous updates, reinforcement learning models can process high-dimensional state information, including static features, interactive features, and contextual features, and express the non-linear relationship between states and actions. This allows recommendation results to change dynamically with real-time user feedback and facilitates integration with marketing strategy weights to form adjustable recommendation outputs.

[0129] In this application embodiment, a recommendation system based on reinforcement learning is also provided, including a state acquisition module, a reinforcement learning module, a policy fusion module, an output module, and an update module.

[0130] The state acquisition module is used to execute the state information acquisition step in the aforementioned method embodiment; the reinforcement learning module is used to execute the recommendation expected value calculation and initial recommendation action generation steps in the aforementioned method embodiment; the strategy fusion module is used to execute the marketing strategy information acquisition and weighted adjustment steps in the aforementioned method embodiment; the output module is used to execute the target recommendation action execution step in the aforementioned method embodiment; and the update module is used to execute the reward signal generation and model update steps in the aforementioned method embodiment.

[0131] The specific implementation methods of each module are consistent with the descriptions of the corresponding steps in the aforementioned method embodiments, and will not be repeated here.

[0132] This application also provides a computer-readable storage medium for storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the methods in the above method embodiments. The implementation principle and technical effects are similar, and will not be repeated here.

[0133] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0134] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the equivalent technology of this application, this application also intends to include these modifications and variations.

[0138] In this application, the term "comprising" and its variations can refer to non-limiting inclusion; the term "or" and its variations can refer to "and / or". The terms "first", "second", etc., in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. In this application, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0139] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0140] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0141] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0142] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0143] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0144] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A recommendation method based on reinforcement learning, characterized in that, The method includes: Obtain the state information of the target object, the state information including static features and interactive features; The state information is input into the reinforcement learning model, which calculates the expected recommendation value of each product and generates an initial recommendation action based on the expected recommendation value. Obtain marketing strategy information, which includes product types and strategy weights corresponding to each product type; adjust the expected recommendation value of the initial recommendation action according to the strategy weights to generate a target recommendation action; Perform the target recommendation action and output recommended products; A reward signal is generated based on the interactive feedback corresponding to the recommended product, and the reinforcement learning model is updated based on the reward signal.

2. The method according to claim 1, characterized in that, The step of weighting and adjusting the expected recommendation value of the initial recommendation action according to the strategy weight to generate the target recommendation action includes: The weight coefficient for each product type is determined based on the strategy weight corresponding to each product type. The expected recommendation value of each product in the initial recommendation action is weighted according to the weighting coefficient to obtain the weighted expected recommendation value. Based on the weighted expected recommendation value, the recommendation priority of each product is determined, and the target recommendation action is generated.

3. The method according to claim 1, characterized in that, The acquisition of the target object's state information includes: Based on the unique identifier of the target object, raw data is extracted from multiple business systems, including at least the core business system, the credit system, the wealth management system, and the credit card system. The original data is subjected to feature alignment and splicing processing to construct a unified customer view; The static features and the interactive features are extracted based on the unified customer view.

4. The method according to claim 1 or 3, characterized in that, The status information also includes context features; The process of obtaining the state information of the target object also includes: Obtain scene information of the target object, wherein the scene information includes at least one of the following: login time, geographical location, device type, and current operation page; Extract the context features based on the scene information; The static features, the interactive features, and the context features are combined to construct the state information.

5. The method according to claim 1, characterized in that, The step of generating a reward signal based on the interactive feedback corresponding to the recommended product includes: Obtain the feedback type of the interactive feedback, wherein the feedback type includes positive feedback and negative feedback; The reward signal is generated by matching the corresponding reward value according to the feedback type. The reward value for positive feedback is positive, and the reward value for negative feedback is negative, with different reward values ​​corresponding to different feedback types.

6. The method according to claim 5, characterized in that, The positive feedback includes at least one of the following behaviors: clicking, saving, sharing, rating, and purchasing; the negative feedback includes at least one of the following behaviors: rejection and quickly skipping.

7. The method according to claim 1, characterized in that, The step of generating a reward signal based on the interaction feedback corresponding to the recommended product, and updating the reinforcement learning model based on the reward signal, includes: During the use of the target user, continuous collection of interactive feedback corresponding to the recommended products is conducted. Update the state information of the target object based on the interactive feedback; Recommended products are regenerated based on the updated status information, and the reinforcement learning model is continuously updated iteratively.

8. The method according to claim 1, characterized in that, The reinforcement learning model is a neural network model based on Q-learning.

9. A recommendation system based on reinforcement learning, characterized in that, The system includes: A state acquisition module is used to acquire the state information of a target object, the state information including static features and interactive features; The reinforcement learning module is used to receive the state information, calculate the expected recommendation value of each product, and generate an initial recommendation action based on the expected recommendation value. The strategy fusion module is used to acquire marketing strategy information, which includes product types and strategy weights corresponding to each product type. Based on the strategy weights, the expected recommendation value of the initial recommendation action is weighted and adjusted to generate the target recommendation action. The output module is used to execute the target recommendation action and output recommended products; An update module is used to generate a reward signal based on the interactive feedback corresponding to the recommended product, and update the reinforcement learning module based on the reward signal.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

11. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images