A multi-wheel adaptive fund recommendation method and a reinforcement learning model training method

Abstract

The application belongs to the technical field of artificial intelligence and financial technology, and particularly relates to a multi-round self-adaptive fund recommendation method and a reinforcement learning model training method. The model training method comprises the following steps: a Markov decision process oriented to fund recommendation is constructed, a state is constructed by a user interaction history and corresponding fund overall score and component stock label and is dynamically calibrated through a multi-dimensional market state, an action is output by a strategy network to embed a continuous action and is mapped into a recommended fund; a double critic network is constructed, a first critic evaluates a difference between fund overall scores of adjacent rounds as an incremental reward, a second critic evaluates a double-head value after mask processing to evaluate component stocks liked and disliked by a user and filter missing labels; and the strategy network is updated based on an evaluation result. The fund recommendation method generates a recommendation based on the trained strategy network, in combination with a current state and a market state, and updates the state according to feedback to enter a next round of recommendation.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and financial technology, specifically involving a multi-round adaptive fund recommendation method and a reinforcement learning model training method. Background Technology

[0002] Fund recommendations involve the efficient matching of investors and fund products in intelligent wealth management. Unlike ordinary product recommendations, funds exhibit significant multi-source heterogeneity: their multi-view attributes include at least numerical views (returns, volatility, drawdown, beta, etc.), category views (fund type, theme tags, risk levels, etc.), textual views (fund introduction, investment strategy description, etc.), and hierarchical structure views (fund-constituent stock holdings). Furthermore, users' investment preferences dynamically shift with market conditions, and real-world interaction data generally suffers from sparsity, fragmentation, and privacy compliance limitations. User feedback is often subjective, noisy, and partially labeled (e.g., only overall star ratings, only indicating a few favorite / disliked stocks).

[0003] Traditional recommendation methods based on collaborative filtering or static logs rely on large amounts of historical behavioral data, making it difficult to proactively obtain feedback through multiple rounds of interaction, and even more difficult to characterize the dynamic migration and multi-weighting of user preferences. To address these issues, existing technologies have introduced interactive recommendation methods based on reinforcement learning, modeling the multi-round "recommendation-feedback-re-recommendation" process as a Markov decision process to optimize long-term rewards. However, existing reinforcement learning methods still have the following technical limitations: Defect 1: Lack of multi-view and hierarchical structure representation. Existing methods struggle to uniformly represent the multi-view attributes of funds and the fund-constituent stock hierarchical structure, lacking an effective fusion mechanism that can simultaneously capture the overall characteristics of funds and the fine-grained characteristics of constituent stocks, resulting in insufficient recommendation matching accuracy; Defect 2: Reward design biases towards maximizing absolute scores. Existing methods often bias towards maximizing absolute scores in their reward design. In multi-round scoring feedback scenarios, this can easily lead to an overemphasis on "high-scoring star funds," resulting in insufficient exploration, decreased recommendation diversity, and poor generalization stability across years / preference distributions. Defect 3: Poor robustness to noise and partially labeled feedback. Real user feedback is subjective, noisy, and partially labeled. Direct training with existing conventional learning objectives is easily affected by missing feedback and selective labeling bias, resulting in unstable training and low sample efficiency.

[0004] Therefore, existing fund recommendation schemes based on Markov decision processes have long been plagued by three major problems: lack of representation fusion, reward design bias towards absolute scores, and poor robustness to noise and partial annotations. There is a lack of a technical solution that can systematically solve the above defects at the same time. Summary of the Invention

[0005] This application proposes a multi-round adaptive fund recommendation scheme, which aims to address the three major technical shortcomings of existing fund recommendation schemes based on Markov decision processes: lack of representation fusion, bias of reward design towards absolute scores, and poor robustness to noise and partial annotations.

[0006] A first aspect of this application provides a reinforcement learning model training method, including: A Markov decision process for fund recommendation is constructed, wherein the state includes the user state constructed based on the user's interaction history and the corresponding overall fund score and constituent stock label, and the action includes the continuous action embedding of the strategy network based on the current user state, which is used to map to specific recommended funds in the candidate fund pool through similarity retrieval. The user state is dynamically calibrated through multi-dimensional market state representation. Constructing a dual-critic network, the dual-critic network includes: The first-rate critic network is used to evaluate incremental rewards, which are the differences between the overall fund score in the current round and the overall fund score in the previous round. A second commentator network is used to evaluate masked two-head values, which include: a positive value output head for evaluating the positive value of constituent stocks that the user likes, and a negative value output head for evaluating the negative value of constituent stocks that the user dislikes. The overall fund score and the like and dislike labels of the constituent stocks are obtained based on user feedback on the recommended funds, and then input into the dual critic network for evaluation; Based on the evaluation results of the first and second critic networks, the parameters of the policy network are updated, and the user state is updated based on the user's feedback on the recommended funds to enter the next round of interaction, thereby optimizing the long-term cumulative return of the Markov decision process.

[0007] In some embodiments of this application, the policy network includes a hierarchical hybrid expert structure, which comprises: Multiple expert networks, each designed to handle a different preference dimension; A gated routing network is used to dynamically assign weights to the multiple expert networks based on the input state. The outputs of the multiple expert networks are fused according to the weights to output an action embedding, which is used to map to a specific recommended fund in the candidate fund pool through similarity retrieval.

[0008] In some embodiments of this application, the recommended funds in the candidate fund pool generate a joint fund representation for similarity retrieval in the following manner: Encode the multi-view attribute set of the fund separately to obtain the encoded representation of each view; The encoded representations of each view are input into the view gating network, and the fusion weights of each view are obtained according to the following formula:

[0009] in, The learnable weight matrix of the view gating network. For bias terms, , , These are the encoded representations for the Fund ID view, Numeric view, and Text view, respectively. The multi-view fused representation of the fund is obtained by weighting and fusing the coded representations of each view according to the following formula: , in, The fusion weight is the value corresponding to the m-th view. Obtain the information of the top K constituent stocks of the fund, and calculate the constituent stock embedding and the fund's holding aggregation representation, where K is a natural number; The multi-view fusion representation and the holding aggregation representation are hierarchically fused to obtain a joint fund representation for decision-making.

[0010] In some embodiments of this application, the training includes a pre-training phase and an online incremental update phase: In the pre-training phase, a user simulator is introduced. The user simulator generates an overall fund rating and like / dislike labels for constituent stocks based on historical statistics and preference distribution to simulate the user's feedback on the recommended fund. During the online incremental update phase, the strategy network and the dual commentator network are updated online based on real-time user feedback on recommended funds, wherein the real-time feedback includes the overall fund score and the like and dislike labels for the constituent stocks; Specifically, a mask is applied to the constituent stocks with missing labels to filter out the missing labels.

[0011] In some embodiments of this application, the loss function for updating the parameters of the policy network is as follows: , in, The main loss predicted by the fund; This is the risk consistency loss, used to constrain the matching of recommended funds with users' risk tolerance. This is a hierarchical consistency loss used to constrain the consistency between user status and fund holdings structure; Robust loss for missing feedback; For hyperparameter weights.

[0012] In some embodiments of this application, the dynamic calibration of the user state through a multi-dimensional market state representation includes: Constructing a multi-dimensional market state representation The multidimensional market state representation includes market index volatility. Industry rotation signals and style factor momentum ; The multidimensional market state representation is encoded into a market state vector using a multilayer perceptron according to the following formula: ; The market environment change intensity index is calculated using the following formula: , in, For backtracking window; When the market environment change intensity index Exceeding the preset threshold At that time, the historical preference weights are calculated according to the following formula: , And prior preference weights: , in, For sensitivity parameters; The user state is dynamically calibrated according to the following formula: , in, For user states modeled based on historical sequences, For prior preference states based on user profiles, This represents element-wise multiplication. These are the weighting coefficients. This refers to the calibrated user status.

[0013] In some embodiments of this application, the hierarchical fusion of the multi-view fusion representation and the position aggregation representation to obtain a joint fund representation for decision-making further includes: The fund's constituent stock holdings are aggregated hierarchically according to industry classification, and for each industry... The fund's weighting in this sector is calculated using the following formula: , in, For the fund Mid-component stocks The weighting or normalized percentage of holdings; Constructing industry holdings vectors for funds ; Extract the user's industry preference vector from the user's historical interactions, and calculate the user's industry preference vector according to the following formula: , in, For users Funds for secondary interactions Feedback The attenuation coefficient is... For the current time, For the first Interaction time; Based on the user's industry preference vector, the attention score for each industry allocation of the candidate funds is calculated according to the following formula: , in, For industry vector dimensions, For users For candidate funds Attention scores for each industry sector; Generate a refined representation of positions using the following formula: , in, constituent stocks The industry to which it belongs This represents the attention weight corresponding to this industry. For embedding of constituent stocks Learnable mappings For more refined representation of holdings; The refined holdings representation and the multi-view fusion representation are hierarchically fused to obtain a joint fund representation for decision-making.

[0014] In some embodiments of this application, the attention scores for each industry allocation of the candidate funds are calculated according to the following formula: Based on the aforementioned multidimensional market state representation Extracting industry rotation signals ; The user's industry preference vector is modulated according to the following formula: , in, The modulation intensity coefficient, For normalization function, This is the user industry preference vector before modulation. This is the modulated user industry preference vector. This represents element-wise multiplication; Based on the modulated user industry preference vector, the attention score of each industry allocation of the candidate funds is calculated according to the following formula, so as to increase the preference weight of strong market industries and decrease the preference weight of weak market industries: .

[0015] A second aspect of this application provides a multi-round adaptive fund recommendation method, including: A policy network is obtained, which is acquired through the training method described in the first aspect of the embodiments of this application; Obtain the user status and the multidimensional market status representation for the current round; The user state and the multidimensional market state representation are input into the policy network to generate recommended actions; Output the fund recommendation results corresponding to the recommended action; Obtain user feedback signals regarding the fund recommendation results, and update the user status based on the feedback signals to proceed to the next round of recommendations.

[0016] In some embodiments of this application, the step of inputting the user state and the multidimensional market state representation into the policy network to generate a recommendation action includes: Obtain the continuous action vector output by the policy network; Before determining the candidate funds in the pool through similarity search. There are 10 candidate funds, among which It is a natural number; In the preceding We will use a weighted random sampling method based on similarity or distance among the candidate funds to determine the funds recommended in this round. The recommended funds are output as the fund recommendation result.

[0017] In summary, the reinforcement learning model training method and multi-round adaptive fund recommendation method provided in the embodiments of this application obtain a joint fund representation by constructing a hierarchical fusion mechanism of fund multi-view attributes and constituent stock holdings, uniformly capturing the overall fund characteristics and fine-grained features of constituent stocks, thus solving the problem of missing representation fusion. By adopting an incremental reward design, the difference between the current and previous round fund overall scores is used as a reward signal to guide preferences to gradually approach each other, avoiding the strategy network from chasing absolutely high-scoring star funds, thus solving the problem of reward design biased towards absolute scores. By introducing a masked positive / negative dual-head value evaluation structure, the positive and negative values ​​of liked and disliked constituent stocks are evaluated separately and missing labels are filtered out, reducing the training bias caused by selective labeling and sparse feedback, thus solving the problem of poor robustness to noise and partial labels. The above three technical features do not exist in isolation, but rather cooperate and work synergistically: hierarchical fusion representation provides a more accurate state representation for incremental rewards, incremental rewards reduce the disturbance of noise feedback to strategy updates, and masked dual-head value evaluation further filters out the training bias caused by partial labels. Together, these three approaches have overcome the three long-standing challenges of existing solutions in the face of scarce data, heterogeneous features, and dynamically changing preferences, providing a systematic reinforcement learning solution for multi-round adaptive fund recommendation. Attached Figure Description

[0018] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a schematic diagram of an interactive scenario involving multiple rounds of fund recommendations; Figure 2 This is a schematic diagram of the reinforcement learning model architecture applicable to fund recommendation in this application; Figure 3 This is a flowchart illustrating a reinforcement learning model training method according to some embodiments of this application; Figure 4 This is a flowchart illustrating a multi-round adaptive fund recommendation method according to some embodiments of this application; Figure 5 This is an example of a multi-round adaptive fund recommendation process in one embodiment of this application. Detailed Implementation

[0019] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0020] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0021] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0022] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0023] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0024] In reinforcement learning frameworks, Markov Decision Processes (MDPs) are a fundamental paradigm for modeling sequential decision problems. Their core elements include: State, representing all relevant information about the current environment or user; Action, output by the policy network based on the current state to influence the environment; Reward, a feedback signal from the environment used to evaluate the merits of the action; and Policy, defining the mapping from state to action, typically parameterized by the policy network. Optimizing the policy network within the MDP framework requires evaluating the value of its output actions. For this, a Critic Network can be introduced to estimate the value of states or state-action pairs, guiding the policy network's update direction. In interactive recommendation scenarios, the multiple "recommendation-feedback-re-recommendation" processes between the system and the user can be formalized as an MDP: the system outputs a recommendation action based on the current user state, the user returns a rating or preference label as a reward signal, the system updates its state accordingly, and proceeds to the next round of decision-making, iterating in this cycle to optimize long-term cumulative rewards. Figure 1 (Diagram of an interactive scenario for multi-round fund recommendations) As shown in the diagram, in a typical multi-round fund recommendation scenario, the system recommends funds with information on the constituent stock holdings to the user. The user rates the fund as a whole and marks individual constituent stocks as liked or disliked, forming a complete closed loop of "recommendation - feedback - re-recommendation".

[0025] Compared to general product recommendations, fund recommendations face more unique challenges. First, funds, as financial products, exhibit significant multi-source heterogeneity: their multi-view attributes include at least numerical views (returns, volatility, drawdown, beta, etc.), category views (fund type, thematic tags, risk levels, etc.), textual views (fund introduction, investment strategy description, etc.), and hierarchical structure views (fund-constituent stock holdings). Unifying the representation of this heterogeneous information is the primary challenge. Second, users' investment preferences dynamically shift with market conditions, and fund ratings are often a gradual process. Users rarely give the highest score in the initial interaction but gradually clarify their preferences through multiple rounds of comparison. Furthermore, real user feedback is subjective, noisy, and partially labeled. Users typically only provide an overall star rating for the fund, indicating their likes / dislikes for a few constituent stocks, while many constituent stocks remain unreviewed.

[0026] As described in the background section, existing fund recommendation schemes based on Markov decision processes have long been plagued by three technical defects: lack of representation fusion, reward design bias towards absolute scores, and poor robustness to noise and partial annotations.

[0027] To systematically address the aforementioned shortcomings, this application proposes a multi-round adaptive grant recommendation scheme based on reinforcement learning. For example... Figure 2(A schematic diagram of the reinforcement learning model architecture applicable to fund recommendation in this application) shows that this application constructs a complete reinforcement learning model training architecture through the collaborative design of multi-view and constituent stock hierarchical fusion representation, incremental reward and masked dual-head value assessment.

[0028] Specifically, this application constructs a Markov decision process for fund recommendations, with core innovations in three aspects: First, it generates a joint fund representation through hierarchical fusion of multi-view encoding and constituent stock holdings, uniformly capturing the overall characteristics of the fund and the fine-grained characteristics of its constituent stocks; second, it uses the difference in the overall fund score between adjacent rounds as an incremental reward, guiding the strategy network to focus on the gradual changes in preferences over multiple rounds rather than chasing absolute high scores; third, it models user likes and dislikes of constituent stocks through masked positive / negative dual-head constituent stock commentators and filters out missing annotations. The training process includes two stages: offline pre-training using a user simulator and online incremental updates in real-world scenarios.

[0029] Based on the policy network trained above, this application also provides a corresponding multi-round adaptive fund recommendation method: combining the current user state and the market state, the recommended fund is determined from the candidate fund pool through similarity retrieval and weighted random sampling, and the state is updated according to user feedback to enter the next round of interaction, forming a complete recommendation closed loop.

[0030] The following section will provide a detailed description of the specific implementation methods for each step, in conjunction with the accompanying drawings.

[0031] Figure 3 This is a flowchart illustrating a reinforcement learning model training method according to some embodiments of this application. For example... Figure 3 As shown, the specific methods for training reinforcement learning models include: S310, Construct a Markov decision process for fund recommendation, wherein the state includes a user state constructed based on the user's interaction history and its corresponding overall fund score and constituent stock label, and the action includes a continuous action embedding output by the strategy network based on the current user state, which is used to map to a specific recommended fund in the candidate fund pool through similarity retrieval, wherein the user state is dynamically calibrated through a multi-dimensional market state representation.

[0032] This step is used to formalize the multi-round fund recommendation interaction process into a sequential decision problem, clarify the definition of core elements such as state, action, and reward under the MDP framework, and build the basic architecture of state representation and strategy network, including fund multi-view and hierarchical fusion representation, time-aware sequence encoding, hierarchical hybrid expert strategy structure, and market state-aware user state dynamic calibration mechanism.

[0033] Specifically, such as Figure 1As shown, this application constructs a closed-loop decision-making process of "recommendation-feedback-re-recommendation" for fund recommendation tasks in multi-round interaction scenarios, and formalizes it as a Markov Decision Process (MDP): the state in the MDP refers to the user state of the system, denoted as ( (Interaction rounds) are used to represent a user's historical interactions and preferences, and include at least the most recent interactions. The interaction includes the fund sequence, corresponding rating feedback (e.g., 1-5 stars), optional constituent stock like / dislike labels, and the resulting preference statistics; actions are determined by the policy network based on the current user state. Output continuous motion embedding (This can be understood as the "next recommendation intent vector"), and the action embedding is mapped to a specific set of recommended funds through candidate fund pool constraints and similarity retrieval; the reward comes from the user's feedback signal on the recommended funds. In the action mapping process, a candidate set is first obtained based on compliance constraints and business rules (such as fund type, risk level, subscription and redemption restrictions, etc.) and recall retrieval. Then, the similarity with the candidate fund representation vector is calculated to generate the recommendation ranking. At the same time, in order to avoid local optima and insufficient exploration caused by only selecting the nearest neighbor, this application introduces a random sampling / random perturbation mechanism (e.g., sampling according to similarity distribution or...) within the candidate set. (Internal random sampling) to maintain a balance between exploration and exploitation within a limited interaction budget.

[0034] To obtain stable fund statements ,like Figure 2 As shown, this application constructs a representation learning module that integrates multiple views and hierarchical structures: it encodes multi-source heterogeneous attributes such as historical fund return sequences, risk indicators (volatility, maximum drawdown, beta, etc.), investment direction / industry themes, and rating classifications, and obtains a fund-level representation through self-attention / gating fusion. Simultaneously, it introduces the fund's front-end representation... The large constituent stock set and its weight information form the holding layer representation, and the two are hierarchically merged to obtain the final fund embedding.

[0035] Specifically, different views are encoded separately to obtain the basic representation of the fund, and gating fusion is used to suppress redundant information that is weakly correlated with the user's actual preferences. For example, the view encoding of the fund can be represented as: , ,

[0036] in Embed the fund ID in the table. It is a multilayer perceptron. For text encoders, , , These are the encoded representations for the Fund ID view, numerical view, and text view, respectively. The multi-view encodings are input into a gating network to obtain the weights of each view. : , in, The learnable weight matrix of the view gating network. For bias terms Then, a weighted fusion is performed to obtain a multi-view fusion representation of the fund. : , in, This represents the fusion weight corresponding to the m-th view.

[0037] To leverage fund holdings structure and user feedback on partial annotations of constituent stocks, an aggregate modeling of the fund-constituent stock hierarchical relationship is introduced. Let the constituent stocks... The representation of is The aggregated holdings of the fund are then represented as: , in, This is a learnable mapping or graph structure aggregation operator. The fund multi-view fusion representation and the holdings aggregation representation are further fused to obtain the final fund representation: , in It can be a linear mapping after splicing, gated fusion, or attention fusion.

[0038] Meanwhile, for the set of "likes / dislikes" of constituent stocks given by users in the interaction history, a hierarchical preference consistency constraint is introduced to ensure that the user's preference state and the fund's holding structure remain consistent in the representation space, thereby alleviating the preference estimation bias caused by relying solely on the overall fund score and improving training stability.

[0039] To characterize the dynamic shifts in user preferences, explicit time-aware sequence preference encoding is introduced. Specifically, this is based on user interaction timestamp sequences. Construct a time interval sequence, where Mapping time intervals to discrete buckets and looking up tables yields the time embeddings: , , in For scaling parameters, Embed a table for the time bucket.

[0040] To simultaneously characterize both short-term impulses and long-term plans, a multi-scale time encoder is introduced to obtain time representations. ,in This is the window length.

[0041] The fund representation sequence and the time representation are jointly input into the sequence encoder to obtain the user's dynamic preference state: , in It can be used to model networks such as Transformers, state-space models, or other sequences.

[0042] Furthermore, to enhance the adaptive modeling capability for different preference dimensions, this application introduces a hierarchical hybrid expert structure (MoE) in the policy network, where a gating network dynamically allocates expert weights according to the state, enabling the model to dynamically allocate expert capabilities for different preference dimensions, thereby improving the expressiveness and decision adaptability of multi-dimensional feature fusion; and outputting action embeddings after fusion.

[0043] To address the issue of lagging user preferences under drastic market changes, this application further introduces a market-state-aware dynamic calibration mechanism for user states.

[0044] Specifically, constructing a multi-dimensional market state representation This includes: market index volatility Industry rotation signals and style factor momentum These market features are encoded into market state vectors using a multilayer perceptron.

[0045] Based on the magnitude of changes in market conditions, define an indicator of the intensity of changes in the market environment: , in This is a backtracking window.

[0046] when Exceeding the preset threshold At that time, the historical preference weights are calculated according to the following formula: ; And prior preference weights:

[0047] in This is a sensitivity parameter. The user state is dynamically calibrated according to the following formula: , in, For user states modeled based on historical sequences, For prior preference states based on user profiles, This represents element-wise multiplication. These are the weighting coefficients. This refers to the calibrated user status.

[0048] This mechanism allows the system to primarily rely on historical behavior modeling when the market environment is stable; when the market undergoes dramatic changes, the system automatically increases the weight of prior profiles while injecting market state information to achieve dynamic calibration of preferences.

[0049] S320, Construct a dual-commentator network, comprising: a first commentator network for evaluating incremental rewards, wherein the incremental reward is the difference between the overall fund score in the current round and the overall fund score in the previous round; and a second commentator network for evaluating masked bi-head values, wherein the bi-head values ​​include: a positive value output head for evaluating the positive value of constituent stocks favored by the user, and a negative value output head for evaluating the negative value of constituent stocks disliked by the user.

[0050] The purpose of this step is to construct a dual-commentator network to simultaneously evaluate incremental rewards at the fund level and masked double-headed value at the constituent stock level, providing value signals for strategy network updates. This helps to suppress excessive pursuit of high-scoring star funds in multi-round scoring feedback scenarios and improves robustness to noise and partially labeled feedback.

[0051] Specifically, to address the issues of "over-biasing towards high-scoring star funds" and training instability that can easily arise from multiple rounds of rating feedback, this application employs a dual-critic reinforcement learning optimization framework, simultaneously learning overall satisfaction at the fund level and fine-grained preferences at the constituent stock level. Here, the fund-level critics, acting as the first critic network, drive preferences to gradually converge through multiple rounds of incremental reward signals. The reward can be defined as the change in rating or the relative improvement (e.g., ...). This approach, which involves either normalizing the scores (or applying a normalized increment) to reduce the pursuit of absolute high scores and encourage long-term improvement through continuous refinement, thus suppressing premature bias towards star funds, employs a masked positive / negative dual-head structure for the component stock commentator network. This model separately models the polarized feedback of users' "like / dislike" towards component stocks and filters out missing annotations, ensuring that supervisory signals are calculated only for labeled component stocks. This reduces bias caused by selective annotation and sparse feedback, improving training stability and robustness. The evaluation results of the two commentators will serve as the basis for subsequent policy network updates and will participate in joint optimization along with the multi-task loss function.

[0052] S330: Based on user feedback on recommended funds, obtain the overall fund score and the like and dislike labels of the constituent stocks, and input them into the dual critic network for evaluation.

[0053] The purpose of this step is to explain how user feedback is obtained and what content is included during the training process, as well as the rules for handling missing annotations, so as to provide a data foundation for subsequent strategy optimization.

[0054] Specifically, this application introduces a user simulator for offline training under conditions of privacy compliance and scarcity of real-world interaction. This user simulator generates multi-round ratings and localized portfolio annotations based on historical statistics and preference distributions to simulate user feedback on recommended funds, and uses this to pre-train strategies and commentators.

[0055] Subsequently, the strategy was calibrated in real-world business using online incremental updates or batch incremental learning to adapt to changes in market conditions and user preference distributions in different years.

[0056] In the training process described above, user history interactions and multi-round rating feedback are constructed into states, and each round of recommendation decisions is constructed into actions, forming an optimizable closed-loop interaction mechanism. User feedback includes an overall fund rating and a "like" or "dislike" rating for the constituent stocks. Users typically only rate a small number of constituent stocks, leaving a large number without any feedback.

[0057] To further reduce the bias caused by selective labeling and sparse feedback, this application applies a mask to the components with missing labels during training, so that the supervision signal is calculated only for the labeled components. This masking mechanism is applicable to both the offline training and online calibration stages.

[0058] S340, based on the evaluation results of the first critic network and the second critic network, the parameters of the policy network are updated, and the user state is updated based on the user's feedback on the recommended fund to enter the next round of interaction, thereby optimizing the long-term cumulative return of the Markov decision process.

[0059] The purpose of this step is to update the parameters of the policy network based on the evaluation results of the dual-critic network and the multi-task loss function, and to update the user state to enter the next round of interaction, thereby achieving long-term cumulative reward optimization of the Markov decision process.

[0060] Specifically, to adapt to the characteristics of real fund recommendations, such as "sparse feedback, high noise, and partial annotation," this application employs multi-task joint training to supplement the supervision signal and enhance robustness. The total loss function is defined as: in, For the primary task loss used in fund prediction or ranking learning; To constrain the risk consistency loss between recommended funds and users' risk tolerance and fund risk levels; This is used to constrain the consistency between user status and fund holding structure by utilizing the labeling of constituent stocks; To address robust loss against missing feedback and selective labeling bias, methods such as mask learning, confidence weighting, or bias correction can be employed. For hyperparameter weights.

[0061] Based on the aforementioned loss function and the evaluation results of the first and second critic networks, the parameters of the policy network are updated to adjust its output actions towards higher-value directions. The policy optimization objective can be expressed as maximizing the expected discounted return: , in As a discount factor, The system sets an upper limit on the number of interaction rounds and incorporates the consistency of fund-level returns and constituent stock-level preferences into commentator learning and strategy updates.

[0062] Simultaneously, the user's status is updated based on user feedback on the recommended funds to proceed to the next round of interaction. By constructing user history and multi-round rating feedback into status, constructing each round of recommendation decisions into actions, and using user feedback as reward signals, a complete "recommendation-feedback-re-recommendation" closed-loop interaction mechanism is formed, thereby optimizing the long-term cumulative return of the Markov decision-making process.

[0063] One embodiment of this application solves the fine-grained matching problem between constituent stock holding structure and user industry preferences by using an industry-level attention mechanism and time-varying modeling of user industry preferences.

[0064] As described in S310, when constructing a joint fund representation, the fund's multi-view fusion representation... It has been obtained through multi-view encoding and gating fusion. Based on this, in order to further solve the problem of fine-grained matching between the constituent stock holding structure and user industry preferences, the embodiment refines the holding aggregation representation described in S310.

[0065] First, the fund's constituent stock holdings The data is aggregated hierarchically according to industry classification. For the fund Mid-component stocks The weighting or normalized percentage of holdings. Let the industry set be... (e.g., Shenwan Level 1 Industry Classification), for each industry The fund's weighting in this sector is calculated using the following formula: .

[0066] This is used to construct the fund's industry holdings vector. .

[0067] Secondly, extract users' industry preference vectors from their historical interactions. This involves analyzing each fund a user has interacted with. Extract its industry holdings vector Based on user feedback regarding the fund (Purchase, favorite, or ignore) A time decay mechanism is introduced to give higher weight to recent interactions: , in, The attenuation coefficient is... For the current time, For the first Interaction time This represents a vector of user industry preferences.

[0068] Then, based on the user's industry preference vector, the attention score of each industry allocation of the candidate funds is calculated through an attention mechanism: , in For industry vector dimensions, For users For candidate funds The attention scores for each industry allocation reflect the degree to which users' preferences match the fund's industry allocation.

[0069] Finally, by combining industry-level attention weights with the original constituent stock embeddings, a refined portfolio representation that takes into account user industry preferences is generated:

[0070] in, constituent stocks The industry to which it belongs This represents the attention weight corresponding to this industry. For embedding of constituent stocks Learnable mappings This provides a more refined representation of holdings.

[0071] This formula means that when a user currently prefers a certain industry, the constituent stocks within that industry are given a higher weight in the holdings representation, while industries that the user is not interested in are suppressed.

[0072] The refined position representation is fused with the multi-view representation described in S310. Hierarchical fusion is performed to obtain a joint fund representation for decision-making: , in It can be a linear mapping after splicing, gated fusion, or attention fusion.

[0073] A preferred embodiment of this application, based on the above embodiments, further integrates the market state perception user state dynamic calibration mechanism described in S310 with the industry-level attention holding representation, so that the two can be linked through market industry rotation signals to achieve accurate recommendations during market fluctuations.

[0074] The market state perception user state dynamic calibration mechanism and industry-level attention holding representation described in S310 are not isolated entities, but can be deeply coordinated through the market state dimension and industry preference dimension.

[0075] Specifically, the industry rotation signal in the market state representation constructed in S310 This can be further used to guide the calibration of user industry preference vectors in industry-level attention holding representations. When calculating user industry preferences, a modulation factor based on market industry strength is introduced: , in, The modulation intensity coefficient, For normalization function, This is the user industry preference vector before modulation. This is the modulated user industry preference vector. This indicates element-wise multiplication.

[0076] This mechanism allows the system to appropriately increase the preference weight of an industry when it shows strong rotation signals in the market, even if users have little historical interaction with that industry, thus making a trend-following recommendation. Conversely, it reduces the weight of industries that are weakening in the market to avoid recommending users to buy industry allocation funds with a high probability of decline.

[0077] Furthermore, the modulated user industry preference vector is used as the user industry preference vector in the industry-level attention holding representation. The attention scores of each industry allocation of the candidate funds are recalculated to generate a refined holding representation modulated by market state. This representation is then hierarchically fused with the multi-view fusion representation to obtain a collaboratively modulated joint fund representation.

[0078] Through the aforementioned collaborative mechanism, the calibrated user status obtained by S310 is integrated with market status information. The refined holdings representation obtained from the industry-level attention holdings representation reflects the user's fine-grained preferences for industry allocation through industry attention. The matching score of the two can simultaneously capture the user's overall risk appetite adjustment in the current market environment and the degree of matching between the user's fine-grained preferences for specific industry allocations. This enables the model to recommend fund products that both match the user's current risk tolerance and their industry preferences during periods of market volatility. For example, when the market turns bearish, the S310 market status perception calibration mechanism will reduce the weight of the user's aggressive preferences, while the industry-level attention mechanism will simultaneously focus on the user's preferred defensive industries (such as pharmaceuticals and consumer goods). The final recommended fund is both low-risk and heavily invested in defensive sectors, achieving accurate recommendations through a dual match.

[0079] Figure 4 This is a flowchart illustrating a multi-round adaptive fund recommendation method according to some embodiments of this application. Figure 4 As shown, the method specifically includes: S410, Obtain the policy network, which is obtained through the training method described in S310-S340.

[0080] S420, Obtain the user status and the multi-dimensional market status representation for the current round.

[0081] S430, input the user state and the multi-dimensional market state representation into the policy network to generate a recommendation action.

[0082] S440, Output the fund recommendation result corresponding to the recommendation action.

[0083] S450, Obtain the user's feedback signal on the fund recommendation result, and update the user's status based on the feedback signal to proceed to the next round of recommendations.

[0084] One embodiment of this application provides a specific implementation method for multi-round adaptive fund recommendation based on a trained policy network.

[0085] First, the policy network trained using the aforementioned training method is obtained. This policy network is jointly optimized with a masked dual-head value evaluation network during the training phase, and then used independently for online recommendation after training. The policy network includes at least a multi-view encoder, a component-level fusion module, a temporal encoder, a hybrid expert (MoE) module, and a market state awareness calibration module.

[0086] At the start of each round of recommendations, the user status and multi-dimensional market status representation for the current round are obtained. The user status is set based on the user's historical interactions and preference information, and is updated in subsequent rounds based on feedback. The multi-dimensional market status representation includes market index volatility, industry rotation signals, and style factor momentum, which are preprocessed.

[0087] User state and multi-dimensional market state representations are input into the policy network, which outputs continuous action embeddings (which can be understood as "next recommendation intent vectors"). These action embeddings are then mapped to specific recommendation fund sets through candidate fund pool constraints and similarity retrieval. First, a candidate set is obtained based on compliance constraints, business rules, and recall retrieval. Then, the similarity with the candidate fund representation vectors is calculated to generate recommendation rankings. Simultaneously, a random sampling / random perturbation mechanism is introduced within the candidate set (e.g., sampling according to similarity distribution or random sampling within the Top-N) to maintain a balance between exploration and utilization.

[0088] Users can provide feedback on the recommendation results, and the system updates the user's status based on the feedback. The updated user status then enters the next round of recommendations, forming a multi-round adaptive recommendation closed loop.

[0089] Figure 5 This is an example of a multi-round adaptive fund recommendation process in one embodiment of this application.

[0090] Rounds 1 to 3: The first three rounds employed existing reinforcement learning methods for recommendations. Although this method incorporated a Markov decision process framework, it lacked a unified encoding of the fund's multi-view attributes and an effective fusion of the constituent stock hierarchy, resulting in significant fluctuations in recommendation results across different rounds.

[0091] Round 1: Recommended bond funds with a risk level of R3 (medium risk), with a historical return of 7.73%. User rating: 3 stars.

[0092] Round 2: We recommend equity-oriented mixed funds with a risk level adjusted to R4 (medium-high risk), with a historical return of 33.59%. These recommended funds include those representing user-preferred industries or constituent stocks, raising their rating to 4 points.

[0093] Round 3: Recommended bond funds with a risk level of R3 (medium risk) and a historical return of 4.84%. The rating drops back to 3 points.

[0094] The scores in the first three rounds fluctuated from 3 to 4 to 3. Although the recommended funds varied in terms of risk level (R3 / R4), historical returns (4.84% to 33.59%), and fund type (bond / equity-biased mixed fund) in each round, the lack of a unified representation and integration of diverse attribute information in the existing methods made it impossible to accurately capture users' preference trade-offs across multiple dimensions, resulting in poor recommendation stability.

[0095] Rounds 4 through 6: Starting from round 4, the system adopts the method described in this application. Through multi-view encoding and constituent stock hierarchical fusion, the system uniformly represents the multi-source heterogeneous attributes of funds and integrates constituent stock holding information to form a decision representation. Time-aware sequence encoding characterizes the dynamic drift of user preferences; a hierarchical hybrid expert structure adaptively allocates expert capabilities for different preference dimensions; and a dual-commentator network guides preferences to gradually converge through incremental rewards. Building upon this, an industry-level attention mechanism is further introduced, hierarchically aggregating the fund's constituent stock holdings according to industry classification. Combining this with the industry preference trajectory in the user's historical interactions, the system calculates the attention score for each industry allocation, achieving fine-grained alignment between the recommended fund's industry allocation and the user's industry preferences, avoiding the industry allocation mismatch problem caused by simple holding weighting in existing methods. Simultaneously, a market-state-aware dynamic calibration mechanism is incorporated. When market volatility intensifies, the system can simultaneously explore and adjust from two dimensions: risk tolerance and industry allocation, achieving more accurate recommendation matching.

[0096] Rounds 4 and 5: The system explores a new preference space, recommending bond funds with a risk level of R3 and historical returns of 1.15% and 2.17%, respectively. Since the system has not yet converged to the user's preferred allocation during the exploration phase, the score temporarily drops to 2 points.

[0097] Round 6: The system synthesizes all feedback signals from the previous five rounds, confirming that users' true preference is for medium-to-high risk, concentrated equity-oriented mixed funds. This round's recommended risk level is R4 (medium-to-high risk), with a historical return of 37.98%, and the rating rises to 4 points.

[0098] Through the above 6 rounds of interaction, the recommendation score of the proposed method rebounded from 2 points in the 4th and 5th rounds to 4 points in the 6th round. Compared with the score fluctuation of the existing method in the 1st to 3rd rounds (3→4→3), the recommendation stability and user satisfaction have been improved. Figure 5 The results also show that: By employing a random sampling / random perturbation mechanism, the system avoids local optima caused by relying solely on the most recent single-round mapping, achieving a balance between exploration and exploitation within a limited interaction budget. By using incremental reward signals, the system reduces the pursuit of absolute high scores and encourages continuous optimization through continuous learning.

[0099] In summary, the reinforcement learning model training method and multi-round adaptive fund recommendation method provided in the embodiments of this application obtain a joint fund representation by constructing a hierarchical fusion mechanism of fund multi-view attributes and constituent stock holdings, uniformly capturing the overall fund characteristics and fine-grained features of constituent stocks, thus solving the problem of missing representation fusion. By adopting an incremental reward design, the difference between the current and previous round fund overall scores is used as a reward signal to guide preferences to gradually approach each other, avoiding the strategy network from chasing absolutely high-scoring star funds, thus solving the problem of reward design biased towards absolute scores. By introducing a masked positive / negative dual-head value evaluation structure, the positive and negative values ​​of liked and disliked constituent stocks are evaluated separately and missing labels are filtered out, reducing the training bias caused by selective labeling and sparse feedback, thus solving the problem of poor robustness to noise and partial labels. The above three technical features do not exist in isolation, but rather cooperate and work synergistically: hierarchical fusion representation provides a more accurate state representation for incremental rewards, incremental rewards reduce the disturbance of noise feedback to strategy updates, and masked dual-head value evaluation further filters out the training bias caused by partial labels. Together, these three approaches have overcome the three long-standing challenges of existing solutions in the face of scarce data, heterogeneous features, and dynamically changing preferences, providing a systematic reinforcement learning solution for multi-round adaptive fund recommendation.

[0100] Although the foregoing embodiments are described within the general context of computer systems running in conjunction with operating systems and applications, those skilled in the art will recognize that other implementations can also be performed by incorporating other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via communication networks. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0101] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0102] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A reinforcement learning model training method, characterized in that, include: A Markov decision process for fund recommendation is constructed, wherein the state includes the user state constructed based on the user's interaction history and the corresponding overall fund score and constituent stock label, and the action includes the continuous action embedding output by the strategy network based on the current user state, which is used to map to specific recommended funds in the candidate fund pool through similarity retrieval. The user state is dynamically calibrated through multi-dimensional market state representation. Constructing a dual-critic network, the dual-critic network includes: The first-rate critic network is used to evaluate incremental rewards, which are the differences between the overall fund score in the current round and the overall fund score in the previous round. A second commentator network is used to evaluate masked two-head values, which include: a positive value output head for evaluating the positive value of constituent stocks that the user likes, and a negative value output head for evaluating the negative value of constituent stocks that the user dislikes. The overall fund score and the like and dislike labels of the constituent stocks are obtained based on user feedback on the recommended funds, and then input into the dual critic network for evaluation; Based on the evaluation results of the first and second critic networks, the parameters of the policy network are updated, and the user state is updated based on the user's feedback on the recommended funds to enter the next round of interaction, thereby optimizing the long-term cumulative return of the Markov decision process.

2. The method according to claim 1, characterized in that, The policy network includes a hierarchical hybrid expert structure, which comprises: Multiple expert networks, each designed to handle a different preference dimension; A gated routing network is used to dynamically assign weights to the multiple expert networks based on the input state. The outputs of the multiple expert networks are fused according to the weights to output an action embedding, which is used to map to a specific recommended fund in the candidate fund pool through similarity retrieval.

3. The method according to claim 1, characterized in that, The recommended funds in the candidate fund pool are used to generate a joint fund representation for similarity retrieval in the following manner: Encode the multi-view attribute set of the fund separately to obtain the encoded representation of each view; The encoded representations of each view are input into the view gating network, and the fusion weights of each view are obtained according to the following formula: in, The learnable weight matrix of the view gating network. For bias terms, , , These are the encoded representations for the Fund ID view, Numeric view, and Text view, respectively. The multi-view fused representation of the fund is obtained by weighting and fusing the coded representations of each view according to the following formula: ， in, The fusion weight is the value corresponding to the m-th view. Obtain the information of the top K constituent stocks of the fund, and calculate the constituent stock embedding and the fund's holding aggregation representation, where K is a natural number; The multi-view fusion representation and the holding aggregation representation are hierarchically fused to obtain a joint fund representation for decision-making.

4. The method according to claim 1, characterized in that, The training includes a pre-training phase and an online incremental update phase: In the pre-training phase, a user simulator is introduced. The user simulator generates an overall fund rating and like / dislike labels for constituent stocks based on historical statistics and preference distribution to simulate the user's feedback on the recommended fund. During the online incremental update phase, the strategy network and the dual commentator network are updated online based on real-time user feedback on recommended funds, wherein the real-time feedback includes the overall fund score and the like and dislike labels for the constituent stocks; Specifically, a mask is applied to the constituent stocks with missing labels to filter out the missing labels.

5. The method according to claim 1, characterized in that, The loss function for updating the parameters of the policy network is as follows: ， in, The main loss predicted by the fund; This is the risk consistency loss, used to constrain the matching of recommended funds with users' risk tolerance. This is a hierarchical consistency loss used to constrain the consistency between user status and fund holdings structure; Robust loss for missing feedback; For hyperparameter weights.

6. The method according to claim 3, characterized in that, The dynamic calibration of the user state through multi-dimensional market state representation includes: Constructing a multi-dimensional market state representation The multidimensional market state representation includes market index volatility. Industry rotation signals and style factor momentum ; The multidimensional market state representation is encoded into a market state vector using a multilayer perceptron according to the following formula: ； The market environment change intensity index is calculated using the following formula: ， in, For backtracking window; When the market environment change intensity index Exceeding the preset threshold At that time, the historical preference weights are calculated according to the following formula: ， And prior preference weights: ， in, For sensitivity parameters; The user state is dynamically calibrated according to the following formula: ， in, For user states modeled based on historical sequences, For prior preference states based on user profiles, This represents element-wise multiplication. These are the weighting coefficients. This refers to the calibrated user status.

7. The method according to claim 6, characterized in that, The step of hierarchically fusing the multi-view fusion representation with the holding aggregation representation to obtain a joint fund representation for decision-making further includes: The fund's constituent stock holdings are aggregated hierarchically according to industry classification, and for each industry... The fund's weighting in this sector is calculated using the following formula: ， in, For the fund Mid-component stocks The weighting or normalized percentage of holdings; Constructing industry holdings vectors for funds ; Extract the user's industry preference vector from the user's historical interactions, and calculate the user's industry preference vector according to the following formula: ， in, For users Funds for secondary interactions Feedback The attenuation coefficient is... For the current time, For the first Interaction time; Based on the user's industry preference vector, the attention score for each industry allocation of the candidate funds is calculated according to the following formula: ， in, For industry vector dimensions, For users For candidate funds Attention scores for each industry sector; Generate a refined representation of positions using the following formula: ， in, constituent stocks The industry to which it belongs This represents the attention weight corresponding to this industry. For the embedding of constituent stocks Learnable mappings For more refined representation of holdings; The refined holdings representation and the multi-view fusion representation are hierarchically fused to obtain a joint fund representation for decision-making.

8. The method according to claim 7, characterized in that, The attention scores for the industry allocations of the candidate funds are calculated using the following formula: Based on the aforementioned multidimensional market state representation Extracting industry rotation signals ; The user's industry preference vector is modulated according to the following formula: ， in, The modulation intensity coefficient, For normalization function, This is the user industry preference vector before modulation. This is the modulated user industry preference vector. This represents element-wise multiplication; Based on the modulated user industry preference vector, the attention score of each industry allocation of the candidate funds is calculated according to the following formula, so as to increase the preference weight of strong market industries and decrease the preference weight of weak market industries: 。 9. A multi-round adaptive fund recommendation method, characterized in that, include: A policy network is obtained by any of the training methods described in claims 1-8; Obtain the user status and the multidimensional market status representation for the current round; The user state and the multidimensional market state representation are input into the policy network to generate recommended actions; Output the fund recommendation results corresponding to the recommended action; Obtain user feedback signals regarding the fund recommendation results, and update the user status based on the feedback signals to proceed to the next round of recommendations.

10. The method according to claim 9, characterized in that, The step of inputting the user state and the multi-dimensional market state representation into the policy network to generate recommendation actions includes: Obtain the continuous action vector output by the policy network; Before determining the candidate funds in the pool through similarity search. There are 10 candidate funds, among which It is a natural number; In the preceding We will use a weighted random sampling method based on similarity or distance among the candidate funds to determine the funds recommended in this round. The recommended funds are output as the fund recommendation result.

Images