Personalized recommendation method and system based on dynamic heterogeneous graph and reinforcement learning

Abstract

The invention belongs to the technical field of computers, and particularly relates to a personalized recommendation method and system based on a dynamic heterogeneous graph and reinforcement learning. The method comprises the following steps: firstly, constructing a global heterogeneous information graph of multiple types of nodes offline, and learning static embedding of the nodes by using a graph neural network; secondly, dynamically constructing a session history into a session graph in a real-time interaction process of the user, and aggregating by adopting a graph convolutional network to generate a dynamic state vector of the user; inputting the dynamic state vector into an actor and commentator reinforcement learning framework; and finally, using a dominant function calculated by the commentator network as a stable learning signal, and performing end-to-end joint training on the whole model to optimize long-term cumulative return. According to the method, by introducing the session graph volume accumulation device, the accuracy of dynamic state representation is remarkably improved; and an actor commentator framework is adopted, so that the problem of high variance of a traditional strategy gradient method is effectively solved, and the training stability and efficiency are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a personalized recommendation method and system based on dynamic heterogeneous graphs and reinforcement learning. Background Technology

[0002] With the rapid development of information technology, personalized recommendation systems have become a core component of major online platforms (such as e-commerce and content platforms). However, existing recommendation technologies generally suffer from the following shortcomings: Limitations of static representation: Traditional recommendation algorithms (such as collaborative filtering) and some models based on graph neural networks (GNNs) typically construct a static user-item interaction graph. This static representation cannot capture the rapid and dynamic changes in user interests within a specific session, leading to delayed recommendation results and failing to meet users' immediate needs. Short-sighted optimization objectives: Most existing models adopt a supervised learning paradigm, and their optimization objective is usually to maximize the accuracy of the next click (CTR prediction). This "short-sighted" objective often leads to homogenized recommended content, sacrificing user experience, content diversity, and novelty, which is detrimental to improving long-term user retention and platform loyalty. The challenge of state representation in reinforcement learning applications: Although existing research has attempted to apply reinforcement learning (RL) to recommendation systems to optimize long-term rewards, a core challenge is generally faced: how to define a "state" that accurately and comprehensively reflects the user's current intent. If the state representation is too simple (such as using only a static embedding of user IDs), the RL agent makes decisions in an information-scarce environment, resulting in significantly reduced learning efficiency and final performance. Therefore, there is an urgent need in this field for a new technical solution that can simultaneously address the three major challenges of dynamically capturing user interests, optimizing long-term value, and providing high-quality state representation. Summary of the Invention

[0003] This invention addresses the technical problems existing in the background art by proposing an innovative personalized recommendation architecture that deeply couples dynamic heterogeneous graphs with reinforcement learning. It aims to overcome the inherent limitations of traditional recommendation systems in information representation, user intent capture, and optimization goal setting. This invention introduces a graph convolution-based conversation graph aggregator to capture users' short-term, dynamic, structured intents, and fuses these with users' long-term, stable interest representations. Finally, it employs a reinforcement learning framework based on actor / critic for decision optimization.

[0004] To achieve the above objectives, the technical solution adopted by the present invention includes the following steps:

[0005] S1. Offline data preparation and heterogeneous information graph construction: Obtain user behavior data and item relationship data, map user, item and relationship entities in the data to unique integer IDs, construct a global heterogeneous information graph containing user nodes, item nodes and relationship nodes, and initialize learnable initial feature vectors for each type of node.

[0006] S2. Static node embedding encoding based on heterogeneous graph neural network: Using an L-layer heterogeneous graph neural network as the encoder, for each node in the global heterogeneous information graph, the node embedding is updated layer by layer through neighbor message calculation, same-relationship neighbor information aggregation, cross-relationship information fusion and nonlinear activation. The L-layer output is used as the final static embedding vector of the node, and the user static embedding matrix and the item static embedding matrix are stored.

[0007] S3. Real-time dynamic state generation of user interests based on conversation graph convolution: When a user starts a new conversation, their static embedding vector is used as the initial state. At the decision time, a conversation graph containing unique item nodes and temporal edges and semantic edges is dynamically constructed based on the current conversation history. An L-layer lightweight graph neural network is applied to the conversation graph for local graph convolution, and then attention pooling is used to generate a dynamic state vector representing the macro-interest of the conversation.

[0008] S4. Fusion of long-term static interest and short-term dynamic interest and generation of final state: Extract the user's long-term static interest vector and the dynamic state vector, generate a gating vector through a gating network, and perform adaptive weighted fusion of the long-term static interest vector and the short-term dynamic interest vector based on the gating vector to obtain the final state vector.

[0009] S5. Recommendation action generation based on actor-critic framework: Construct structurally independent actor network and critic network, input the final state vector into the two networks in parallel, the actor network outputs the item recommendation probability distribution, the critic network outputs the state value scalar, and the recommendation action is determined by random sampling based on the probability distribution;

[0010] S6. Joint training and parameter update based on the actor-critic framework: At each moment of interaction between the model and the user, collect a quintuple of state, action, immediate reward, next state and current state value, calculate the advantage function, construct a joint loss function including Actor loss and Critic loss, calculate the gradient based on the joint loss function, and perform end-to-end update of the parameters of the heterogeneous graph neural network, Actor network, Critic network and node embedding matrix, iterating until the model converges;

[0011] S7. Online Inference and Recommendation List Generation: Receive the user's online recommendation request and current session history, obtain the final state vector, input the final state vector into the trained Actor network to obtain the item recommendation probability distribution, sort the probability distribution, select the K items with the highest probability to form a recommendation list and return it to the user.

[0012] Preferably, the implementation of constructing a global heterogeneous information graph and initializing learnable initial feature vectors for each type of node in step S1 includes:

[0013] S11. Obtain user behavior data from the data source. Item relationship data This involves uniquely identifying and mapping all entities in the data, including users, items, and relationships, to integer IDs. A one-to-one correspondence between the original identifier of an entity and consecutive integer IDs is established using a bijective function. A bijective function is defined for the user entity. The original set of user identifiers Mapping to a continuous set of integers ; enabling any original user Each ID corresponds to a unique integer ID, and its inverse function can map the integer ID back to the original ID; similarly, similar mapping functions are defined for items and relational entities. and This yields an integer mapping between items and categories;

[0014] S12. Based on the mapped data, construct a user-based... ,thing ,relation Heterogeneous Infographic The node set V includes user nodes, item nodes, and relationship nodes; the edge set E includes the edge set describing the user-item interaction relationship. and the edge set describing the relationship of ownership between items ;

[0015] S13. Initialize node features for each node in the heterogeneous graph. Initialize a d-dimensional learnable initial feature vector. This is achieved through an embedding layer, i.e. ,in It is a trainable embedding matrix; for each type of node in graph G, an independent trainable embedding matrix is ​​defined, namely the user embedding matrix. Item embedding matrix and relation embedding matrix Before training begins, the weights of each embedding matrix are randomly initialized using a Gaussian distribution with a mean of 0 and a variance of 1; then, arbitrary nodes are obtained. initial feature vector At that time, perform an embedded lookup operation based on the node type: if it is a user node, perform the lookup... The row whose index is the integer ID of the node; if it is an item node, search for... The row whose index is the integer ID of the node; if it is a relation node, search for... The row whose index is the integer ID of the node is the row vector found. And the embedding matrix , , These are all part of the model parameters and are updated synchronously with the gradient descent of the loss function during subsequent model training.

[0016] Preferably, the specific implementation method of step S2 based on the static node embedding encoding of heterogeneous graph neural networks includes:

[0017] S21. Define the heterogeneous graph neural network encoder architecture. Use a heterogeneous graph neural network with L layers as the encoder for node embedding to realize the layer-by-layer transmission and aggregation of node information in the global heterogeneous information graph.

[0018] S22, for the encoder's first... layer, Each node in the diagram The embedding vector is updated through the following mechanism:

[0019] Message computation: For nodes Each neighbor node Where r is the relation type, and its value is calculated and passed to... message vector The calculation method is as follows: ;in, Is node u in the th order? Layer embedding vectors, It is specific to relation r. Layer-learnable weight matrix;

[0020] Neighbor information aggregation: Aggregate messages from all neighbor nodes of the same relation type r to obtain a relation-specific aggregation vector. The mean aggregation method is used, and the formula is as follows: ;

[0021] Cross-relation information fusion and update: Summing the information aggregated from all relation types and obtaining the node information through a non-linear activation function ReLU. In the New embedding vector of the layer The formula is: ;

[0022] S23. Input the entire heterogeneous graph G into the L-layer GNN encoder to complete the embedding calculation of all nodes, and finally obtain the L-layer output. As the final static embedding vector of the node;

[0023] S24. Store the static embeddings of all user nodes as a matrix. Each vector represents a user's long-term interest preferences; the static embedding of all item nodes is stored as a matrix. .

[0024] Preferably, the specific implementation of step S3, which generates the user's real-time interest dynamic state based on session graph convolution, includes:

[0025] S31. First, perform session initialization. When user u starts a new session, its initial state is... Set as a pre-computed static embedding vector ;

[0026] S32. At decision time t, based on the current user's session history... Dynamically construct a miniature session graph containing only the current session information within computing resources. ;

[0027] S33, in the dynamic session diagram Above, an L-layer lightweight graph neural network is applied to capture local and non-linear dependencies between items within a session. The node embedding update formula for the layer is: ,in Is node v in the session graph The set of neighbors in;

[0028] S34. Embed all nodes after local graph convolution. Perform an attention pooling operation to generate a d-dimensional vector, which is the user's final dynamic state vector. .

[0029] As a preferred approach, weighted pooling is performed using an attention mechanism to obtain... The specific implementation includes:

[0030] S341. Use the mean of all nodes in the session as the initial query vector: ;

[0031] S342, Regarding the conversation diagram Each node in Calculate its embedding vector Relevance score between the query vector q and the query vector q This includes first using two independent learnable weight matrices. and Embedding nodes respectively The query vector q is linearly transformed, the two transformed vectors are added together, and then fused using a non-linear activation function to obtain a combined hidden representation vector: ;

[0032] S343. Obtain the hidden representation vector , with another learnable attention context vector Perform a dot product operation to obtain the attention score: ;

[0033] S344. Use the Softmax function to score the attention of all nodes. Normalization is performed to obtain the final attention weights. ;

[0034] S345, embed the vector of each node. Its corresponding attention weight Multiply them, then sum all the weighted vectors to obtain the dynamic state vector. : .

[0035] Preferably, the specific implementation of step S4, the fusion of long-term static interest and short-term dynamic interest, and the generation of the final state, includes:

[0036] S41. Obtain long-term and short-term interest vectors; the long-term interest vector... To obtain, for the current user u, from the computed user static embedding matrix The corresponding row vector is extracted directly from the data. The short-term dynamic interest vector Obtain the dynamic state vector representing the user's current session macro-level interests as the short-term dynamic interest vector, i.e. ;

[0037] S42. Design a gating network. Long-term static interest vectors and short-term dynamic interest vectors The concatenated vectors are then input into a gating network. This network uses a fully connected layer and applies a sigmoid activation function to ensure that each element in the output gating vector g is between 0 and 1, as shown in the formula: ,in, and These are the learnable parameters of the gated network. It is the Sigmoid function;

[0038] S43. Using the gated vector g and Perform a weighted fusion operation to obtain ,in For the generated final state representation, This is element-wise multiplication.

[0039] Preferably, step S5, which generates recommended actions based on the actor critic framework, is specifically implemented by, at decision time t, taking the time t... Recorded as As a unified input, it is fed in parallel into the actor network and the critic network for a forward propagation computation to obtain all the information needed for decision-making: The actor network computation formula is: ,in Let A be the probability distribution for choosing action A at time t. For policy function Learnable parameters For A function with parameters, representing a vector of final states. Processing; The formula for calculating the critic network is: ,in, State at time t State value, For parameters A value function estimator with learnable weights; then using probability distributions The final recommended action is determined by random sampling. .

[0040] Preferably, the specific implementation of step S6, which is based on the actor critic framework for joint training and parameter update, includes:

[0041] S61. Single-step temporal difference data collection: at each decision time t, data is collected from the current state. Recommended actions Instant rewards Next state and current state value The quintuple that constitutes the quintuple;

[0042] S62. Calculate the advantage function, and calculate the action at time t. Relative to state Advantage of average value ,in, Discount factor;

[0043] S63. Construction of the joint loss function, total loss ,in For weight parameters, Losses to actors ; Loss to critics , For strategy In state The entropy of the probability distribution of the next action;

[0044] S64, based on Calculate the gradient, update the parameters synchronously, and iterate the training until the model converges.

[0045] Preferably, the system is applicable to the method described in any one of claims 1-8, comprising:

[0046] Offline data preparation and heterogeneous information graph construction: acquire user behavior data and item relationship data, map user, item and relationship entities in the data to unique integer IDs, construct a global heterogeneous information graph containing user nodes, item nodes and relationship nodes, and initialize learnable initial feature vectors for each type of node;

[0047] Static node embedding encoding based on heterogeneous graph neural network: An L-layer heterogeneous graph neural network is used as the encoder. For each node in the global heterogeneous information graph, the node embedding is updated layer by layer through neighbor message calculation, same-relationship neighbor information aggregation, cross-relationship information fusion and nonlinear activation. The L-layer output is used as the final static embedding vector of the node. The user static embedding matrix and the item static embedding matrix are stored.

[0048] User real-time interest dynamic state generation based on conversation graph convolution: When a user starts a new conversation, their static embedding vector is used as the initial state. At the decision time, a conversation graph containing unique item nodes and temporal edges and semantic edges is dynamically constructed based on the current conversation history. The conversation graph is then subjected to local graph convolution by an L-layer lightweight graph neural network, and then a dynamic state vector representing the macro-interest of the conversation is generated through attention pooling.

[0049] Fusion of long-term static interests and short-term dynamic interests and generation of final state: Extract the user's long-term static interest vector and the dynamic state vector, generate a gating vector through a gating network, and perform adaptive weighted fusion of the long-term static interest vector and the short-term dynamic interest vector based on the gating vector to obtain the final state vector;

[0050] Recommendation action generation based on the actor-critic framework: Construct structurally independent actor networks and critic networks, input the final state vector into the two networks in parallel, the actor network outputs the item recommendation probability distribution, and the critic network outputs the state value scalar, and the recommendation action is determined by random sampling based on the probability distribution;

[0051] Joint training and parameter update based on the actor-critic framework: At each moment of interaction between the model and the user, a quintuple of state, action, immediate reward, next state, and current state value is collected. The advantage function is calculated, and a joint loss function including Actor loss and Critic loss is constructed. The gradient is calculated based on the joint loss function, and the parameters of the heterogeneous graph neural network, Actor network, Critic network, and node embedding matrix are updated end-to-end until the model converges.

[0052] Online inference and recommendation list generation: Receive user online recommendation requests and current session history to obtain the final state vector. Input the final state vector into the trained Actor network to obtain the item recommendation probability distribution. Sort the probability distribution and select the K items with the highest probabilities to form a recommendation list and return it to the user.

[0053] Compared with the prior art, the advantages and positive effects of the present invention are as follows:

[0054] 1. At the state representation level, a deep understanding from "linear sequence" to "structured intent" is achieved, which greatly improves the accuracy of recommendations.

[0055] Most existing technologies treat user behavior history as a flat linear sequence, making it difficult to capture the complex non-adjacent relationships between items. This invention innovatively introduces a "graph convolution-based session graph aggregator." This mechanism dynamically constructs a mini-graph from the user's session history at each decision point and performs local graph convolution on it. This allows the model to go beyond simple temporal dependencies, deeply understanding the structured, non-linear interest network formed by the user in the current session, thereby generating an unprecedentedly accurate profile of the user's immediate intent.

[0056] 2. At the information fusion level, intelligent fusion from "rigid splicing" to "adaptive gating" has been achieved, significantly enhancing the robustness of the model.

[0057] This invention addresses the challenge of balancing stable and volatile user interests by introducing a gating fusion mechanism that integrates long-term static interests with short-term dynamic interests. This mechanism, through a learnable gating network, adaptively determines the weights of long-term and short-term interests based on the context of the current session. When user intent is clear, emphasis is placed on short-term signals to ensure immediacy; when user behavior is sparse or chaotic, greater reliance is placed on long-term profiles to ensure stability. This dynamic, context-aware "intelligent ferring" enables the model to exhibit extremely high robustness in various complex scenarios, especially cold-start and short-session scenarios.

[0058] 3. At the decision optimization level, a paradigm upgrade from "high variance" to "stable learning" has been achieved, effectively ensuring the convergence efficiency and performance of the model.

[0059] 4. Traditional reinforcement learning recommendation methods (such as REINFORCE) use Monte Carlo rewards for learning, which suffers from severe gradient variance problems, leading to unstable training and low efficiency. This invention employs an advanced "Actor-Critic" framework. This framework introduces a "critic" network to evaluate state value and uses an advantage function instead of the original reward as the learning signal. The advantage function, as a guide signal with lower variance and richer information, allows the model to learn stably and efficiently at every step of the interaction. This not only accelerates the model's convergence speed but also enables the model to more reliably learn the optimal recommendation strategy that optimizes the long-term cumulative value of users.

[0060] In summary, this invention, through systematic innovation in three core aspects—state representation, information fusion, and decision optimization—constructs a novel recommendation paradigm capable of accurate perception, intelligent fusion, and robust decision-making. It not only achieves breakthroughs in the accuracy and real-time performance of recommendations but also demonstrates significant superiority over existing technologies in model robustness and the ability to optimize long-term user value. Attached Figure Description

[0061] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a flowchart of a personalized recommendation method and system based on dynamic heterogeneous graphs and reinforcement learning.

[0063] Figure 2 This is a module interaction and data flow graph of a personalized recommendation method and system based on dynamic heterogeneous graphs and reinforcement learning. Detailed Implementation

[0064] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0065] Numerous specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways than those described herein, and therefore the invention is not limited to the specific embodiments disclosed in the following specification.

[0066] In this embodiment, to overcome the inherent limitations of traditional recommender systems in information representation, user intent capture, and optimization goal setting, this invention introduces a graph convolution-based conversation graph aggregator to capture users' short-term, dynamic, structured intents, and fuses them with users' long-term, stable interest representations. Finally, an actor-critic reinforcement learning framework is used for decision optimization. The specific implementation is as follows: Figure 1 As shown.

[0067] First, offline data preparation and heterogeneous information graph construction are performed. User behavior data and item relationship data are acquired, and unique integer IDs are mapped to user, item, and relationship entities in the data. A global heterogeneous information graph containing user nodes, item nodes, and relationship nodes is constructed, and learnable initial feature vectors are initialized for each type of node. Specifically, user behavior data is obtained from the data source. Item relationship data This involves uniquely identifying and mapping all entities in the data, including users, items, and relationships, to integer IDs. A one-to-one correspondence between the original identifier of an entity and consecutive integer IDs is established using a bijective function. A bijective function is defined for the user entity. The original set of user identifiers Mapping to a continuous set of integers ; enabling any original user Each ID corresponds to a unique integer ID, and its inverse function can map the integer ID back to the original ID; similarly, similar mapping functions are defined for items and relational entities. and This yields an integer mapping between items and categories; based on the mapped data, a user-based... ,thing ,relation Heterogeneous Infographic The node set V includes user nodes, item nodes, and relationship nodes; the edge set E includes the edge set describing the user-item interaction relationship. and the edge set describing the relationship of ownership between items Perform node feature initialization for each node in the heterogeneous graph. Initialize a d-dimensional learnable initial feature vector. This is achieved through an embedding layer, i.e. ,in It is a trainable embedding matrix; for each type of node in graph G, an independent trainable embedding matrix is ​​defined, namely the user embedding matrix. Item embedding matrix and relation embedding matrix Before training begins, the weights of each embedding matrix are randomly initialized using a Gaussian distribution with a mean of 0 and a variance of 1; then, arbitrary nodes are obtained. initial feature vector At that time, perform an embedded lookup operation based on the node type: if it is a user node, perform the lookup... The row whose index is the integer ID of the node; if it is an item node, search for... The row whose index is the integer ID of the node; if it is a relation node, search for... The row whose index is the integer ID of the node is the row vector found. And the embedding matrix , , All of these are part of the model parameters, user embedding matrix Its shape is V. U ×d, where V U d represents the total number of users, and d is the preset embedding vector dimension. Item embedding matrix. Its shape is V. I ×d, where V I This represents the total number of items. (Category embedding matrix) Its shape is V. C ×d, where V C This represents the total number of relationships. It is updated synchronously with the gradient descent of the loss function during subsequent model training. As training progresses, the embedding layer gradually learns how to map nodes with similar behavioral patterns or attributes to nearby positions in the vector space. For example, the embedding vectors of two items frequently purchased by the same group of users will become increasingly similar. The initially random vectors eventually evolve into high-quality feature representations containing rich semantic information.

[0068] Static node embedding encoding based on heterogeneous graph neural networks: An L-layer heterogeneous graph neural network is used as the encoder. For each node in the global heterogeneous information graph, the node embedding is updated layer by layer through neighbor message calculation, aggregation of information from neighbors with the same relationship, fusion of information across relationships, and nonlinear activation. The output of the L-layer is used as the final static embedding vector of the node, and the user static embedding matrix and the item static embedding matrix are stored. Specifically, a heterogeneous graph neural network encoder architecture is defined, using an L-layer heterogeneous graph neural network as the encoder for node embedding, to realize the layer-by-layer transmission and aggregation of node information in the global heterogeneous information graph; for the encoder's... layer, Each node in the diagram The embedding vector is updated through the following mechanism: message computation: for node Each neighbor node Where r is the relation type, and its value is calculated and passed to... message vector The calculation method is as follows: ;in, Is node u in the th order? Layer embedding vectors, It is specific to relation r. Layer learnable weight matrix; Neighbor information aggregation: Aggregate messages from all neighbor nodes of the same relation type r to obtain a relation-specific aggregation vector. The mean aggregation method is used, and the formula is as follows: Cross-relation information fusion and update: Summing the information aggregated from all relation types and obtaining the node information through a non-linear activation function ReLU. In the New embedding vector of the layer The formula is: The entire heterogeneous graph G is input into the L-layer GNN encoder to complete the embedding calculation of all nodes, and the final output of the L-layer is obtained. The final static embedding vector of the node; the static embeddings of all user nodes are stored as a matrix. Each vector represents a user's long-term interest preferences; the static embedding of all item nodes is stored as a matrix. This step is performed in an offline environment.

[0069] User real-time interest dynamic state generation based on conversation graph convolution: When a user starts a new conversation, their static embedding vector is used as the initial state. At the decision time, a conversation graph containing unique item nodes, temporal edges, and semantic edges within the conversation is dynamically constructed based on the current conversation history. An L-layer lightweight graph neural network is applied to the conversation graph for local graph convolution, and then attention pooling is used to generate a dynamic state vector representing the macro-level interests of the conversation. Specifically, conversation initialization is performed first. When user u starts a new conversation (t=0), their initial state is... Set as a pre-computed static embedding vector At decision time t, based on the current user's session history... Dynamically construct a miniature session graph containing only the current session information within computing resources. The type of conversation diagram The construction begins with node construction, which is the set of nodes in the session graph. Depend on The unique item ID in the data is used to construct edges between nodes. It contains at least two types: temporal edges, connecting adjacent items in a session sequence; and semantic edges, connecting items in the global heterogeneous graph G that share common attributes (such as belonging to the same category or brand); then node feature initialization is performed: Each node initial features The global static embedding vector obtained above can be used directly. In the dynamic session graph Above, an L-layer lightweight graph neural network is applied to capture local and non-linear dependencies between items within a session. The node embedding update formula for the layer is: ,in Is node v in the session graph Neighbor set in the graph; graph-level representation readout: embedding of all nodes after local graph convolution. Perform an attention pooling operation to generate a d-dimensional vector, which is the user's final dynamic state vector. .

[0070] The dynamic state vector Weighted pooling is implemented using an attention mechanism, specifically as follows: The query vector is defined as follows. To calculate the importance of each item in a session, a "reference point" or "query vector" q is needed to measure them. This query vector q aims to represent the user's global or overall interest. An effective strategy is to use the mean of all nodes in the session as the initial query vector: For the conversation graph Each node in Calculate its embedding vector Relevance score between the query vector q and the query vector q This includes first using two independent learnable weight matrices. and Embedding nodes respectively The query vector q is linearly transformed, the two transformed vectors are added together, and then fused using a non-linear activation function to obtain a combined hidden representation vector: The resulting hidden representation vector , with another learnable attention context vector Perform a dot product operation to obtain the attention score: Attention scores for all nodes were obtained using the Softmax function. Normalization is performed to obtain the final attention weights. ; embed the vector of each node Its corresponding attention weight Multiply them, then sum all the weighted vectors to obtain the dynamic state vector. : Compared to simple mean pooling, attention pooling gives the model a dynamic focusing capability. For example, in a session containing "running shoes," "T-shirt," and "headphones," if the user interacts more deeply with "running shoes" (e.g., connects to more nodes in the session graph), the attention mechanism will automatically assign a higher weight α to the "running shoes" node. This allows the resulting dynamic state vector St to more accurately capture the user's core intent in the current session, ignoring secondary or incidental browsing behaviors, thus providing a higher-quality, higher signal-to-noise ratio state representation for subsequent reinforcement learning decisions.

[0071] Then, the long-term static interest and short-term dynamic interest are fused and the final state is generated: the user's long-term static interest vector and the dynamic state vector are extracted, a gating vector is generated through a gating network, and the long-term static interest vector and the short-term dynamic interest vector are adaptively weighted and fused based on the gating vector to obtain the final state vector. Specifically, the purpose is to provide the decision network with a more comprehensive and robust state representation that combines user historical accumulation and immediate intent. In a preferred embodiment of the present invention, a learnable gating mechanism is used for adaptive fusion, specifically: obtaining long-term and short-term interest vectors; the long-term interest vector... To obtain, for the current user u, from the computed user static embedding matrix The corresponding row vector is extracted directly from the data. The short-term dynamic interest vector Obtain the dynamic state vector representing the user's current session macro-level interests as the short-term dynamic interest vector, i.e. Then design a gating network. Long-term static interest vectors and short-term dynamic interest vectors The concatenated vectors are then input into a gating network. This network uses a fully connected layer and applies a sigmoid activation function to ensure that each element in the output gating vector g is between 0 and 1, as shown in the formula: ,in, and These are the learnable parameters of the gated network. It is the Sigmoid function; it uses a gated vector g. and Perform a weighted fusion operation to obtain ,in For the generated final state representation, This is element-wise multiplication. Compared to fixed fusion strategies (such as concatenation or simple addition), gated fusion gives the model a context-aware adaptive capability. When the user's session history is long and the intent is clear, The information contained is of high quality, and the gating network will learn to generate a gating vector g with a value close to 1, thereby allowing... To a greater extent by Dominant. When user session history is short (cold start) or behavior is chaotic, If the signal is weak or unreliable, the gating network will learn to generate a gating vector g with a value close to 0, thereby allowing... It relies more on more stable and reliable long-term interests. This dynamic, learnable "weight transfer" makes the final state representation... It can always combine long-term and short-term information in the optimal way, which greatly improves the robustness and performance of the model in various complex and variable scenarios.

[0072] Next, by introducing an actor-critic architecture, the questions of "what to do" and "how well to do it" are decoupled. The existence of the critic network provides a stable, low-variance baseline for the actor network's learning, enabling the model to obtain timely and effective feedback signals at each step, thereby significantly improving the stability and convergence speed of training and overcoming the shortcomings of high variance and full-round updates in the traditional REINFORCE algorithm. Recommendation action generation based on the actor-critic framework: Construct structurally independent actor networks and critic networks, inputting the final state vector into both networks in parallel. The actor network outputs an item recommendation probability distribution, and the critic network outputs a state value scalar. Recommendation actions are determined by random sampling based on the probability distribution. Specifically, the agent architecture is first defined, consisting of actor and critic networks. The reinforcement learning used in this invention consists of two structurally independent, functionally complementary deep neural networks, whose parameters are optimized during training. The function of the actor network is: decision executor. Its sole responsibility is to decide what action to take next based on the current environmental state. Input: The generated d-dimensional dynamic state vector that fuses short-term and long-term interests. Network structure: Uses an L-layer perceptron (MLP), consisting of an input layer, one or more hidden layers using the ReLU activation function, and an output layer. Output: A dimensional probability distribution vector ,in This represents the total number of items. The output is normalized using the Softmax function to ensure that the sum of the probabilities of all actions is 1. The i-th element Pt[i] represents the state. The probability of selecting recommended item i is given below. The Critic Network functions as a situation assessor. Its role is not to choose actions, but to score the "goodness" or "badness" of the current state, i.e., to assess the expected value of the long-term reward obtainable from the current state. Input: Similar to the Actor Network, it is a d-dimensional dynamic state vector. Network Structure: Typically, it adopts an MLP structure similar to that of the actor network. Output: A scalar, i.e., a single real value. This value represents the state. The value estimation. At decision time t, the value at time t. Recorded as As a unified input, it is fed in parallel into the actor network and the critic network for a forward propagation computation to obtain all the information needed for decision-making: The actor network computation formula is: ,in Let A be the probability distribution for choosing action A at time t. For policy function Learnable parameters For A function with parameters, representing a vector of final states. Processing; The formula for calculating the critic network is: ,in, State at time t State value, For parameters A value function estimator with learnable weights; then using probability distributions The final recommended action is determined by random sampling. This sampling process embodies the "exploration-exploitation" balance of reinforcement learning, meaning that while the model selects what it considers the optimal action with a high probability, it also retains the possibility of exploring suboptimal actions. The sampled item IDs, i.e. As the result of this recommendation, it is prepared to be presented to users. The beneficial effect of this step is that by introducing an actor-critic architecture, the two questions of "what to do" and "how well to do it" are decoupled. The existence of the critic network provides a stable, low-variance baseline for the learning of the actor network, enabling the model to obtain timely and effective feedback signals at each step, thereby significantly improving the stability and convergence speed of training and overcoming the shortcomings of high variance and full-round updates in the traditional REINFORCE algorithm.

[0073] Joint training and parameter update based on the actor-critic framework: At each moment of interaction between the model and the user, a quintuple of state, action, immediate reward, next state, and current state value is collected. An advantage function is calculated, and a joint loss function including Actor loss and Critic loss is constructed. The gradient is calculated based on this joint loss function, and the parameters of the heterogeneous graph neural network, Actor network, Critic network, and node embedding matrix are updated end-to-end until the model converges. Specifically, single-step temporal difference data collection is performed first. At each decision time t, data from the current state is collected. Recommended actions Instant rewards Next state and current state value The quintuple is constructed; the dominance function is calculated, and the action is calculated at time t. Relative to state Advantage of average value ,in, Discount factor; joint loss function construction, total loss ,in For weight parameters, Losses to actors ; Loss to critics , For strategy In state Entropy of the action probability distribution; parameter update and iterative training. Based on the total loss. The gradient is calculated, and all relevant parameters (including the GNN encoder, actor network, and critic network) are updated end-to-end. This process is repeated until the model converges. By recalculating the user's state representation at each step of the session, this invention can capture subtle changes and shifts in user interests in real time, thereby providing highly immediate and relevant recommendations and significantly improving the user experience.

[0074] Finally, online inference and recommendation list generation: The system receives user online recommendation requests and current session history to obtain a final state vector. This final state vector is then input into the trained Actor network to obtain an item recommendation probability distribution. The K items with the highest probabilities are sorted according to this distribution to form a recommendation list, which is then returned to the user. Specifically, when an online user u triggers a recommendation event in the marketplace (e.g., entering the homepage or refreshing the information stream), the system receives a recommendation request. This request contains at least two key pieces of information: User ID: user_id = u; Current session history: a time-sorted sequence of interactive items from the start of the current session to the current time t. Using the trained model, perform the above steps to generate short-term dynamic interest vectors. and compared it with the user's long-term static interest vector. Merge to obtain the final state Subsequently, The input is fed into the policy network to obtain the recommendation probability distribution. Finally, a recommendation list is generated. This involves analyzing the probability distribution. The system sorts the items, selects the K items with the highest probabilities to form the final recommendation list, and returns it to the user. This step does not use random sampling but rather a deterministic Top-K selection. This invention creatively uses the output of a dynamic heterogeneous graph as the state input for RL. The dynamic state vector, rich in context and real-time information, provides the RL agent with an "information-rich" decision-making environment, greatly improving the learning efficiency and final performance of the RL algorithm, forming a virtuous cycle of "high-quality representation" and "long-term decision-making." The interaction relationships and data transmission logic of the above core modules (GNN encoder, gating fusion module, AC network) are as follows: Figure 2 As shown.

[0075] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments for application in other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A personalized recommendation method based on dynamic graph and reinforcement learning, characterized in that, The method comprises the following steps: S1, offline data preparation and heterogeneous information graph construction: obtaining user behavior data and item relationship data, mapping user, item and relationship entities in the data to unique integer IDs, constructing a global heterogeneous information graph containing user nodes, item nodes and relationship nodes, and initializing learnable initial feature vectors for each type of node; S2, static node embedding coding based on heterogeneous graph neural network: using an L-layer heterogeneous graph neural network as an encoder, for each node in the global heterogeneous information graph, updating the node embedding layer by layer through neighbor message calculation, relationship neighbor information aggregation, cross-relationship information fusion and non-linear activation, taking the L-layer output as the final static embedding vector of the node, and storing the user static embedding matrix and the item static embedding matrix; S3, user real-time interest dynamic state generation based on session graph convolution: when a user starts a new session, taking the static embedding vector as the initial state, dynamically constructing a session graph containing unique item nodes and temporal edges and semantic edges in the current session history at the decision time, applying an L-layer lightweight graph neural network to local graph convolution on the session graph, and then generating a dynamic state vector representing the macro interest of the session through attention pooling; S4, fusion of long-term static interest and short-term dynamic interest and generation of final state: extracting the long-term static interest vector and the dynamic state vector of the user, generating a gating vector through a gating network, adaptively weighting and fusing the long-term static interest vector and the short-term dynamic interest vector based on the gating vector to obtain a final state vector; S5, recommendation action generation based on actor-critic framework: constructing an actor network and a critic network that are independent in structure, inputting the final state vector into the two networks in parallel, outputting an item recommendation probability distribution by the actor network, outputting a state value scalar by the critic network, and determining a recommended action by random sampling based on the probability distribution; S6, joint training and parameter updating based on actor-critic framework: collecting a five-tuple of state, action, immediate reward, next state and current state value at each time of model and user interaction, calculating an advantage function, constructing a joint loss function containing Actor loss and Critic loss, calculating the gradient based on the joint loss function, and updating the parameters of the heterogeneous graph neural network, the actor network, the critic network and the node embedding matrix end-to-end until the model converges; S7, online reasoning and recommendation list generation: receiving a user online recommendation request and the current session history, obtaining a final state vector, inputting the final state vector into the trained actor network to obtain an item recommendation probability distribution, sorting the probability distribution to select the top K items to form a recommendation list and return to the user. 2.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 1, wherein, The implementation of constructing a global heterogeneous information graph and initializing learnable initial feature vectors for each type of node in step S1 comprises: S11, obtaining user behavior data from a data source and item relationship data ; all entities in the data including users, items and relationships are uniquely identified and integer ID mapped, and a one-to-one correspondence between the original identifiers of the entities and the continuous integer IDs is established through a bijective function, and a bijective function is defined for the user entity , the original user identifier set is mapped to a continuous integer set ; so that any original user corresponds to a unique integer ID, and its inverse function can map the integer ID back to the original ID; similarly, similar mapping functions are also defined for item and relationship entities and , the integer mapping of items and categories is obtained; S12. Based on the mapped data, construct a user-based... ,thing ,relation Heterogeneous Infographic The node set V includes user nodes, item nodes, and relationship nodes; the edge set E includes the edge set describing the user-item interaction relationship. and the edge set describing the relationship of ownership between items ; S13, performing node feature initialization for each node in the heterogeneous graph Initialize a d-dimensional learnable initial feature vector for each node in the graph , which is realized by an embedding layer, i.e. where is a trainable embedding matrix; for each type of node in the graph G, a separate trainable embedding matrix is defined, respectively, as a user embedding matrix , an item embedding matrix , and a relation embedding matrix ; the weights of each embedding matrix are randomly initialized by a Gaussian distribution with a mean of 0 and a variance of 1 before training begins; when obtaining the initial feature vector of an arbitrary node , an embedding lookup operation is performed according to the node type: if it is a user node, the row in indexed by the integer ID of the node is found; if it is an item node, the row in indexed by the integer ID of the node is found; if it is a relation node, the row in indexed by the integer ID of the node is found, and the found row vector is ; and the embedding matrices , , are part of the model parameters and are updated synchronously with the loss function gradient descent in the subsequent model training process. 3.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 1, wherein, The specific implementation method of step S2 based on static node embedding coding of heterogeneous graph neural network comprises: S21, define a heterogeneous graph neural network encoder architecture, adopt a heterogeneous graph neural network with L layers as a node embedding encoder for realizing layer-by-layer transmission and aggregation of node information in a global heterogeneous information graph; S22, for the first layer of the encoder, each node in the graph is updated by the following mechanism: Message computation: For nodes Each neighbor node Where r is the relation type, calculate its passed to message vector The calculation method is as follows: ;in, Is node u in the th order? Layer embedding vectors, It is specific to relation r. Layer-learnable weight matrix; Neighbor information aggregation: aggregate messages from all neighbor nodes of the same relationship type r to get relationship-specific aggregated vector ; and the formula is as follows: ; Cross-relation information fusion and update: Sum the information aggregated from all relation types and get the node In the first layer, the new embedding vector is given by: ; S23, input the whole heterogeneous graph G into the GNN encoder of L layer, complete the embedding calculation of all nodes, and finally obtain the L layer output as the final static embedding vector of the node S24, store static embeddings of all user nodes as a matrix Each vector represents a user's long-term interest preference; static embeddings of all item nodes are stored as a matrix . 4.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 1, wherein, The specific implementation of the step S3 includes: S31, first, a session initialization is performed, when a user u starts a new session, its initial state is set to the computed static embedding vector ; S32、at the decision moment t, according to the current user's session history dynamically constructing a micro-session graph containing only the current session information in the computing resource ; S33、in the dynamic conversation graph Above, the L-layer lightweight graph neural network is applied to capture local and nonlinear dependency relationships among items within a conversation, and the first The node embedding update formula of the L-layer is as follows: Wherein is the neighbor set of node v in the conversation graph ​ S34, all node embeddings after local graph convolution performing an attention pooling operation to generate a d-dimensional vector, which is the final dynamic state vector of the user . 5.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 4, wherein, The weighted pooling is obtained by using an attention mechanism The specific implementation includes: S341. Use the mean of all nodes in the session as the initial query vector: ; S342、For each node in the session graph , compute its embedding vector , and the relevance score between the embedding vector and the query vector q , including first linearly transforming the node embedding and the query vector q with two independent learnable weight matrices respectively and adding the two transformed vectors and fusing them through a non-linear activation function to get a combined hidden representation vector: ;​ S343、the obtained hidden representation vector with another learnable attention context vector dot product operation is performed to obtain an attention score: ; S344, using a Softmax function on the attention scores for all nodes normalization to get the final attention weights ; S345、the embedding vector of each node is multiplied by the attention weight corresponding to it with its corresponding attention weight are multiplied, and all the weighted vectors are summed to obtain the dynamic state vector : . 6.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 1, wherein, The specific implementation of the step S4 includes: S41, obtaining a long-term interest vector and a short-term interest vector; the long-term interest vector is obtained as the corresponding row vector directly extracted from the computed user static embedding matrix for the current user u, i.e. ; the short-term dynamic interest vector is obtained as the dynamic state vector representing the user's current session macro interest as the short-term dynamic interest vector, i.e. ; S42, designing a gating network concatenating the long-term static interest vector and the short-term dynamic interest vector The concatenated vector is input into the gating network. The network passes through a fully connected layer and applies a Sigmoid activation function to ensure that each element value in the output gating vector g is between 0 and 1, and the formula is: wherein, and are learnable parameters of the gating network, is a Sigmoid function. S43, using the gating vector g on and performing a weighted fusion operation to obtain wherein is the generated final state representation, is an element-wise multiplication.

7. The personalized recommendation method based on dynamic graph and reinforcement learning according to claim 1, characterized in that, The step S5 generates a specific implementation of the recommended action based on the actor-critic framework, which is specifically implemented as, at a decision time t, obtaining the state value is denoted as As a unified input, it is sent into the actor network and critic network in parallel, and a forward propagation calculation is performed to obtain all the information required for decision-making: the actor network calculation formula is: wherein is the probability distribution of selecting action A at time t, is the policy function with learnable parameters, is a function with as a parameter, and the final state representation vector is processed; the critic network calculation formula is: wherein, is the state value of the state at time t, is a value function estimator with the parameter as a learnable weight; then the probability distribution is used to determine the final recommended action in a random sampling manner. 8.The personalized recommendation method based on dynamic graph and reinforcement learning of claim 1, wherein, The specific implementation of the step S6 includes: S61, single step time series difference data collection, at each decision time t, collect a five tuple consisting of the current state , recommended action , immediate reward , next state and current state value ​ S62, computing the advantage function, computing the action at time t with respect to the state average value of the advantage wherein, is a discount factor; S63, joint loss function construction, total loss wherein is a weight parameter, is actor loss ; is critic loss , is a policy entropy of the action probability distribution in state ; S64, based on Compute gradient, update parameters synchronously, iterate training to model convergence. 9.The personalized recommendation system based on dynamic graph and reinforcement learning of claim 1, wherein, The system is applicable to the method of any one of claims 1-8, comprising: Offline data preparation and heterogeneous information graph construction: obtaining user behavior data and item relationship data, mapping user, item, and relationship entities in the data to unique integer IDs, constructing a global heterogeneous information graph containing user nodes, item nodes, and relationship nodes, and initializing learnable initial feature vectors for each type of node; Static node embedding encoding based on a heterogeneous graph neural network: adopting an L-layer heterogeneous graph neural network as an encoder, updating node embedding layer by layer for each node in the global heterogeneous information graph through neighbor message calculation, relationship neighbor information aggregation, cross-relationship information fusion, and nonlinear activation, taking the L-layer output as the final static embedding vector of the node, and storing the user static embedding matrix and the item static embedding matrix; User real-time interest dynamic state generation based on a session graph convolution: when a user starts a new session, taking the static embedding vector as the initial state, dynamically constructing a session graph containing unique item nodes and temporal edges and semantic edges in the current session history at the decision time, applying an L-layer lightweight graph neural network to the session graph for local graph convolution, and then generating a dynamic state vector representing the macro interest of the session through attention pooling; Fusion and final state generation of long-term static interest and short-term dynamic interest: extracting the long-term static interest vector of the user and the dynamic state vector, generating a gating vector through a gating network, adaptively weighting and fusing the long-term static interest vector and the short-term dynamic interest vector based on the gating vector to obtain a final state vector; Recommendation action generation based on an actor-critic framework: constructing an actor network and a critic network that are independent in structure, inputting the final state vector into the two networks in parallel, outputting an item recommendation probability distribution by the actor network, outputting a state value scalar by the critic network, and determining a recommendation action by random sampling based on the probability distribution; Joint training and parameter updating based on an actor-critic framework: collecting a five-tuple of state, action, immediate reward, next state, and current state value at each time of model and user interaction, calculating an advantage function, constructing a joint loss function containing an actor loss and a critic loss, calculating a gradient based on the joint loss function, and updating the parameters of the heterogeneous graph neural network, the actor network, the critic network, and the node embedding matrix end-to-end, and iterating until the model converges. Online inference and recommendation list generation: receiving a user online recommendation request and current session history, obtaining a final state vector, inputting the final state vector into a trained Actor network to obtain an item recommendation probability distribution, sorting the probability distribution to select the K items with the highest probability to form a recommendation list and return to the user.

Images