A reinforcement learning sequence recommendation method and system based on dual state enhancement

Abstract

The present invention relates to the technical field of reinforcement learning and sequence recommendation, and in particular to a reinforcement learning sequence recommendation method and system based on dual state enhancement. The method comprises obtaining the knowledge graph data of the project and sorting the knowledge graph data according to the timestamp to obtain the interaction sequence; constructing a sequence enhancement model according to the interaction sequence and the project distribution, including constructing a sequence enhancement layer, constructing a knowledge perception enhancement layer and constructing a reward layer; using the constructed sequence enhancement model in combination with the dual state enhancement mechanism, inputting the output of the sequence enhancement model into the Markov decision framework, and updating the model state through the state transition probability; training the sequence enhancement model, and making real-time recommendations to users based on the trained sequence enhancement model. The present invention can better capture the global and local context information in the sequence, enrich the state representation, and alleviate the sparsity problem of the state space.

Description

Technical Field

[0001] The present invention relates to the technical field of reinforcement learning and sequence recommendation, and in particular to a reinforcement learning sequence recommendation method and system based on dual-state enhancement. Background Art

[0002] With the widespread use of streaming services such as Spotify, Netflix, and Amazon Prime, sequential recommendation systems have received widespread attention in recent years. Unlike traditional recommendation systems that regard recommendations as prediction tasks, SR sets the recommendation task as a sequential decision process. In this process, the system generates recommendations based on the user's historical interaction data. Each recommendation decision takes into account the user's previous behavior and preferences. The goal is to explore changes in user interests while using the learned preference patterns to provide personalized and efficient recommendations, thereby optimizing the effect of the entire recommendation sequence.

[0003] A classic method for implementing sequential recommendation is based on latent semantic models, such as matrix factorization. These MF-based models assume that user preferences are static, that is, they do not model the time dependency of user preferences. The key advantage of modern sequential recommendation systems is that they can capture the dynamic changes of user preferences and optimize the recommendation results by analyzing the user's behavior sequence. Due to the great potential of reinforcement learning in decision-making and long-term planning in dynamic environments, some recent technologies have introduced reinforcement learning models into sequential recommendation systems.

[0004] Sequential recommendation systems that apply reinforcement learning also face many problems. First, due to the limitations of offline training data, the model can only access limited user behavior patterns, resulting in sparse state space. Moreover, this data solidification problem limits the generalization ability of the model, making it less adaptable when faced with unseen situations. Secondly, the interaction data between users and items is often sparse and limited, resulting in insufficient information for the model to capture user preferences. This data sparsity not only makes it challenging for the model to understand and predict user needs, but also limits the ability of reinforcement learning algorithms to explore and optimize recommendation strategies during training. These two problems combined not only reduce the user experience, but also weaken the overall efficiency of the system. At this stage, a reinforcement learning sequential recommendation method and system based on dual state enhancement is needed. Summary of the invention

[0005] In order to solve the problems of sparse state space, sparse data and insufficient generalization ability in a reinforcement learning sequence recommendation system, the present invention provides a reinforcement learning sequence recommendation method and system based on dual state enhancement.

[0006] In a first aspect, the present invention provides a reinforcement learning sequence recommendation method based on dual state enhancement, which adopts the following technical solution:

[0007] A reinforcement learning sequence recommendation method based on dual state enhancement, comprising:

[0008] Obtain the knowledge graph data of the project and sort the knowledge graph data according to timestamps to obtain the interaction sequence;

[0009] Constructing a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, a knowledge-aware enhancement layer, and a reward layer;

[0010] Utilize the constructed sequence enhancement model combined with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer;

[0011] The output of the sequence enhancement model is input into the Markov decision framework, and the model state is updated through the state transition probability;

[0012] Perform model training on the sequence enhancement model, including model optimization using the Adam optimizer;

[0013] Based on the trained sequence enhancement model, real-time recommendations are made to users.

[0014] Furthermore, the method of obtaining the knowledge graph data of the project and sorting the knowledge graph data according to timestamp includes obtaining data from three e-commerce recommendation datasets with different domains and sparsity levels, namely, the knowledge graph information of the project, sorting the records of each user according to the timestamp to form an interaction sequence, using the last item of each sequence as test data for the next recommendation task, and using the remaining data as a training dataset.

[0015] Furthermore, the construction of the sequence enhancement layer includes adding, deleting and truncating operations based on the acquired interactive sequence, wherein the truncation operation includes given a sequence length and a shearing rate, for the original sequence, selecting a truncation position to generate a new sequence, calculating the standard deviation of the time intervals of all subsequences, and selecting the subsequence with the smallest standard deviation to complete the sequence enhancement operation.

[0016] Furthermore, the construction of the sequence enhancement layer also includes inputting the enhanced sequence into a Transformer encoder for feature extraction, projecting the high-dimensional item representation to a low-dimensional dense vector using an item embedding matrix, capturing the positional features of the sequence through a learnable position embedding, adding the item embedding and the position embedding to obtain the input, capturing the complex dependencies between items in the sequence using a self-attention mechanism and a multi-head self-attention mechanism, improving the quality and expressiveness of the features through a combination of position embedding and item embedding and smoothing processing of a GELU activation function, and finally, using a recurrent neural network aggregation with a gated recurrent unit to obtain a state representation vector of the enhanced sequence to capture the user's preference features based on the enhanced sequence.

[0017] Furthermore, the construction of the knowledge perception enhancement layer includes using the knowledge graph to infer the current user's preferences, obtaining the embedding vector of the project through the TransE method, and using the average pooling method to aggregate the embedding vectors of the projects in the user's historical interactions to obtain a vector that reflects the user's knowledge features in the interaction history. At the same time, the multi-hop behavior of the knowledge graph is utilized, with the current preferences and the historical knowledge graph as input, and the user's preferences for future steps are predicted through a multi-layer perceptron, thereby further enriching the user's state representation.

[0018] Furthermore, the construction of the reward layer includes adopting a dual reward strategy, dividing the reward into sequence reward and knowledge-aware reward. The sequence reward uses the BLUE metric to perform sequence reward recommendation, focusing on the degree of match between the predicted sequence and the user's actual interaction sequence. The knowledge graph reward uses the TransE method to aggregate items to obtain a feature vector representation of the subsequence, and uses cosine similarity to calculate the similarity between the actual subsequence and the predicted subsequence, focusing on the degree of match between the recommendation result and the knowledge characteristics of the user's preferred items.

[0019] Furthermore, the updating of the model state by the state transition probability includes obtaining the current state according to the knowledge graph of the enhanced sequence and the historical interaction items, selecting items from the item candidate set for recommendation according to the current state by the agent, and marking the recommended items as actions, calculating the probability of selecting each item by the softmax function, and calculating the probability of transferring to the next state by the state transition probability function according to the current state and the action, wherein the formula for calculating the probability of selecting each item is:

[0020] ,

[0021] in, is represented by the action taken by the agent at time step t, The state at time step t is Denoted as the embedding vector of the jth item, Expressed as the parameters of a bilinear matrix, Represented as status The embedding vector of .

[0022] In the second aspect, a reinforcement learning sequence recommendation system based on dual state enhancement includes:

[0023] The data acquisition module is configured to: acquire the knowledge graph data of the project and sort the knowledge graph data according to the timestamp to obtain the interaction sequence;

[0024] The preprocessing module is configured to: construct a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, constructing a knowledge-aware enhancement layer, and constructing a reward layer;

[0025] The conversion module is configured to: utilize the constructed sequence enhancement model in combination with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer;

[0026] The model module is configured to: input the output of the sequence enhancement model into the Markov decision framework and update the model state through the state transition probability;

[0027] The feature extraction module is configured to: perform model training on the sequence enhancement model, including model optimization using the Adam optimizer;

[0028] The transformation module is configured to make real-time recommendations to users based on the trained sequence enhancement model.

[0029] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, a reinforcement learning sequence recommendation method based on dual-state enhancement.

[0030] In a fourth aspect, the present invention provides a terminal device comprising a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the reinforcement learning sequence recommendation method based on dual-state enhancement.

[0031] In summary, the present invention has the following beneficial technical effects:

[0032] 1. The present invention constructs a sequence enhancement layer to add, delete and truncate the acquired interaction sequence, thereby avoiding the state solidification problem caused by reliance on offline data sets in traditional reinforcement learning sequence recommendation. The enhanced sequence is input into the Transformer encoder for feature extraction, and the self-attention mechanism and multi-head self-attention mechanism are used to capture the complex dependencies between items in the sequence. Then, the state representation vector of the enhanced sequence is obtained through aggregation through a recurrent neural network with a gated recurrent unit, which can better capture the global and local context information in the sequence, enrich the state representation, and alleviate the sparsity problem of the state space.

[0033] 2. The present invention constructs a knowledge perception enhancement layer and uses the knowledge graph to infer the current user's preferences. Each item is associated with an entity in the knowledge graph. The embedding vector is obtained by the TransE method. The embedding vectors of items in the user's historical interactions are aggregated using the average pooling method to obtain the knowledge features of the current preference. The entity and relationship information in the knowledge graph supplements the user's background knowledge, enriches the data source, and alleviates the data sparsity problem.

[0034] 3. The present invention utilizes the constructed sequence enhancement model combined with a dual state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer. This fusion of multiple information makes the state representation richer and more diverse, which can not only capture the global and local contextual information in the enhanced sequence, but also integrate external knowledge from the knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall process of a reinforcement learning sequence recommendation method based on dual-state enhancement in an embodiment of the present invention.

[0036] Figure 2 It is a model framework diagram of a reinforcement learning sequence recommendation method based on dual-state enhancement in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0038] Example 1

[0039] Reference Figure 1 , a dual-state enhanced reinforcement learning sequence recommendation method of this embodiment includes:

[0040] Obtain the knowledge graph data of the project and sort the knowledge graph data according to timestamps to obtain the interaction sequence;

[0041] Constructing a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, a knowledge-aware enhancement layer, and a reward layer;

[0042] Utilize the constructed sequence enhancement model combined with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer;

[0043] The output of the sequence enhancement model is input into the Markov decision framework, and the model state is updated through the state transition probability;

[0044] Perform model training on the sequence enhancement model, including model optimization using the Adam optimizer;

[0045] Based on the trained sequence enhancement model, real-time recommendations are made to users.

[0046] Specifically, a dual-state enhanced reinforcement learning sequence recommendation method includes the following steps:

[0047] S1. Obtain the knowledge graph data of the project and sort the knowledge graph data according to timestamps to obtain an interaction sequence;

[0048] We obtain data from three e-commerce recommendation datasets (Beauty, Sports, and Yelp) with different domains and sparsity levels, including the knowledge graph information of the items. We sort the records of each user according to the timestamp to form an interaction sequence. Each interaction sequence contains information about the user's interactions with multiple items, and there is also a time interval sequence, which represents the time interval between the interaction items. We use the last item of each sequence as the test data for the next recommendation task, and the remaining data as the training dataset.

[0049] S2. Constructing a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, a knowledge perception enhancement layer, and a reward layer;

[0050] like Figure 2 As shown, the model consists of three main modules: sequence enhancement layer, knowledge-aware enhancement layer and reward layer;

[0051] (1) Sequence enhancement layer: In the recommendation scenario, an overly aggressive sequence enhancement strategy may have a negative impact on the original sequence, bring too much noise, and deviate from the original features. Therefore, the design of the sequence enhancement strategy should focus on local transformation, and avoid destroying the overall sequence structure by making moderate perturbations to the original sequence. The three existing strategies are improved. Instead of performing random operations on the sequence, the item interval (i.e., time difference) and the correlation between items in the sequence are considered, so as to maintain the sequence's temporal order and internal logic while enhancing the sequence. (Add): Given a length of The sequence and rate of addition, Then the number of items to be inserted at this time is , the target position is recorded as At each target location, choose to insert an item that is related to it but does not interact with it.

[0052] First, the time interval of the sequence Sort from largest to smallest, and then keep the top-k positions as the selected target positions to insert the virtual items. The specific formula is as follows:

[0053] ,

[0054] ,

[0055] Among them, descend represents the sorting function, Indicates inserting virtual items at the selected k positions respectively , represents the enhanced sequence obtained after the addition operation, represents the positions of the first k selected targets, To add an operation.

[0056] (Delete): Given a sequence length n and a coverage rate , then the number of items to be deleted is .

[0057] Still for the time interval of the sequence Sort from large to small, then keep the indexes of the first h target positions and delete the remaining items. The formula can be expressed as:

[0058] ,

[0059] ,

[0060] in, It means to keep the first h positions with a larger time interval. It means to perform a delete operation on the original sequence. To delete items, represents the enhanced sequence obtained after addition and deletion operations, (Truncate): Given a sequence length n and a shear rate , for the original sequence , select a truncation position p, the newly generated sequence is , The standard deviation of the time intervals of all possible subsequences will be calculated, and the subsequence with the smallest standard deviation will be selected. It can be described as follows:

[0061] ,

[0062] ,

[0063] in, It means to find the parameter value that minimizes the objective function. It is represented as the final enhanced sequence, that is, the new sequence is obtained by truncation operation. Expressed as calculating the standard deviation of the time interval of the subsequence, Indicates a truncation operation. represents the standard deviation, In this way, the subsequence with the most stable time interval is selected as the result after truncation.

[0064] At the same time, since short sequences are more sensitive to data amplification, the truncation operation is abandoned for short sequences. Specifically, a hyperparameter K is defined as a threshold to determine the length of the subsequence. Sequences with a length greater than the parameter K are defined as long sequences, and three data augmentation strategies are used to operate them respectively. For sequences with a length less than K, only addition and deletion operations are used for processing.

[0065] In addition, compared with traditional recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), Transformer[2] can better capture long-range dependencies in sequences and is not restricted by sequence length, extracting global features of sequences. In addition, Transformer's self-attention mechanism can dynamically assign weights to each element in the sequence, thereby effectively mining important information in the sequence.

[0066] Therefore, the augmented sequence is input into the Transformer for feature extraction. The Transformer encoder projects the high-dimensional one-hot item representation to a low-dimensional dense vector using the item embedding matrix. In order to represent the position information of the sequence, it uses a learnable position embedding to capture this feature of the sequence. Finally, the item embedding and the position embedding are added to obtain the input representation of the item in the user sequence:

[0067] ,

[0068] in, Represents enhanced sequence Medium Project The expression, Represents the position embedding corresponding to time step t. After the embedding layer, a self-attention module is introduced to capture the dependencies between each pair of items in the sequence. In addition, in order to extract information from different subspaces at each position, multi-head self-attention is used instead of a single attention function. It first projects the input representation into h subspaces using different linear projections. Then, it applies the self-attention mechanism to each head and derives the output representation by connecting the intermediate results and projecting them again. The calculation formula is as follows:

[0069] ,

[0070] ,

[0071] ,

[0072] in, represents the input of each head in the multi-head attention mechanism, Represents the splicing operation, represents the output of each attention head in the multi-head attention mechanism, h represents the number of heads, is the function for calculating the attention weight, is the function used to normalize the attention weights, are the linear transformation weight matrices of query, key and value in the self-attention model, respectively. is the output transformation weight matrix, h represents the number of heads. After linear projection, a position feedforward network is introduced to enable the model to better capture nonlinear relationships:

[0073] ,

[0074] ,

[0075] in, Represented as a position feed-forward network, represents a vector sequence, Represented as a feedforward network for the input vector The processing process, is a weight matrix for linear transformation, is the first bias term, added to the linear transformation, is represented as a parameter matrix, Expressed as a bias term, GELU is smooth over the entire interval and combines the nonlinearity of ReLU with the characteristics of Gaussian noise. It smoothes the input with a Gaussian distribution so that values ​​less than 0 are retained with a certain probability instead of being completely set to 0. This processing method can retain more information and is more consistent with the distribution of real data. Finally, a recurrent neural network with a gated recurrent unit is used to aggregate the enhanced user time-perceived historical behavior to capture the user's preference characteristics based on the enhanced sequence:

[0076] ,

[0077] in, Denoted as represents the output state at time step t, Represented as a recurrent neural network with gated recurrent units, represents the hidden state at time step t-1, is the input information processed by GRU at time step t, Represents the parameter set of GRU.

[0078] (2) Knowledge-aware enhancement layer: Although sequence data provide a lot of information, they may introduce noise and fail to capture deeper future relationships between users and items. To address this problem, a knowledge-aware enhancement module is introduced. Knowledge graph embedding technology enhances the representation of users and items by leveraging rich relational data. However, previous methods usually only consider direct connections between entities, which limits the diversity of recommendation results and has a limited scope of use. To address this problem, while using the knowledge information in the knowledge graph to analyze the user's current preferences, multi-hop behavior is introduced to capture more distant path relationships. This method can capture deeper associations, making the recommendation results more diverse and accurate.

[0079] It is relatively easy to infer the current user's preferences based on historical data. Each item i is associated with an entity in the KG. , and an embedding vector can be obtained through the TransE method Then, the average pooling method is used to aggregate the KG embedding vectors of the items in the user's historical interactions to obtain the knowledge features of the current preference. Reflects the knowledge characteristics of users in their interaction history:

[0080] ,

[0081] in, is the average pooling operation, Represents the embedding vector of the item in the user's historical interaction, n represents the number of items in the user's historical interaction, and using the multi-hop behavior of the knowledge graph to capture the evolution of the user's interest in future time steps is an important method to achieve exploration and recommendation diversity in reinforcement learning. Therefore, calculating future knowledge preferences is particularly important in sequential recommendation based on reinforcement learning. Multilayer perceptron (MLP), as a tool for predicting future preference representations, has good nonlinear modeling capabilities, flexibility, and information transfer advantages of fully connected layers. Therefore, multilayer perceptron becomes an ideal choice for capturing the possible evolution of users' interests in future time steps and providing accurate predictions for sequential recommendation tasks. In time step n, the current preference and the historical knowledge graph are used as input to predict the user's preferences for the next k steps:

[0082] ,

[0083] in, Represented as knowledge features of user preferences predicted at future time steps, Represented as a multi-layer perceptron, It is represented by aggregating the KG embedding vectors of items in the user's historical interactions through the average pooling method, which reflects the knowledge characteristics of the user in the interaction history. Represents a knowledge graph.

[0084] (3) Reward layer: The dual reward strategy divides the reward into sequence reward and knowledge-aware reward. In sequence recommendation, consider using sequence reward to measure the overall performance. Sequence reward mainly focuses on the matching degree between the predicted sequence and the user's actual interaction sequence. The BLUE metric is used for sequence reward recommendation. Formally, given an interaction subsequence and recommended subsequences , the reward function is defined as:

[0085] ,

[0086] The formula approximates the overall performance by considering the future k-step rewards, is the M-gram subsequence of a given interaction sequence, M refers to how many M-gram precisions are used, and m is an index from 1 to M, which is used to traverse all M-gram subsequences. is represented as the actual interaction subsequence, is represented as the predicted subsequence, It refers to the number of times the M-gram subsequence appears in the recommended subsequence. It is expressed as the number of times the M-gram subsequence appears in the interaction sequence. Here, the minimum value of the number of times the same M-gram subsequence appears in the actual interaction subsequence and the recommended subsequence is taken as the numerator. The reason is as follows: If the number of occurrences of an M-gram item in the recommended sequence exceeds the number of occurrences in the actual sequence, repeated recommendations do not improve the matching degree, thus avoiding exaggeration of the recommendation performance. In the knowledge graph reward, the focus is on the matching degree between the recommendation result and the knowledge features of the user's preferred items. Given an actual interaction subsequence and the predicted subsequence , use the TransE method to perform project aggregation to obtain the feature vector representation of the subsequence, and use cosine similarity to calculate the similarity between the two as a reward:

[0087] ,

[0088] in, and are the feature vector representations of the actual interaction subsequence and the predicted subsequence respectively, and the total reward is:

[0089] ,

[0090] in, Expressed as a sequence reward, Expressed as a knowledge graph reward, the reward loss function is actually the maximum and minimum reward function, that is, the difference between maximizing the reward of the target item and minimizing the reward of the non-target item:

[0091] ,

[0092] in, is represented as model parameters, It is expressed as the sum of the difference between the maximum reward and the minimum reward for each time step within the time range. Specifically, RLoss will make the model pay more attention to the items that can bring the maximum reward to the user, while reducing the probability of recommending items that may bring the minimum reward to the user.

[0093] S3, using the constructed sequence enhancement model combined with a dual state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge perception enhancement layer, and the reward layer;

[0094] Based on the above discussion, the final enhanced state representation can be given. , which means It is composed of the following three vectors:

[0095] ,

[0096] By combining these three factors, the current augmented state can not only capture the global and local contextual information in the augmented sequence, but also integrate external knowledge from the knowledge graph, making the state representation richer and more diverse. Specifically, It is the state representation vector of the enhanced sequence. By performing moderate enhancement operations on the original sequence, these features not only include the user's behavior patterns and interest preferences, but also introduce more details and variability in the sequence. By improving the quality of feature representation, the expressiveness of state representation is enhanced. The entity and relationship information in the knowledge graph is used to supplement the user's background knowledge. The possible evolution of users’ interests is predicted through multi-hop relationships. This fusion of multiple information not only alleviates the problem of data sparsity, but also improves the diversity and comprehensiveness of state representation, thereby achieving a more effective balance between development and exploration in the Markov decision process (MDP) framework.

[0097] S4, input the output of the sequence enhancement model into the Markov decision framework, and update the model state through the state transition probability;

[0098] First, the symbols used in this embodiment are defined. U represents a group of users, and I represents a group of items. ,For each user, there is an interaction sequence , indicating that the user has interacted with n items. Each item sequence has a time interval sequence ,in For Project and The time interval between timestamps.

[0099] The knowledge graph containing entity set E and relationship set R is defined as , where the triple Represents the header entity With tail entity The relationship is Based on these notations, the sequential recommendation task aims to predict the next item that a user will interact with given their interaction history. In the MDP process, the agent interacts with the environment in discrete time steps. At each time step t, the state Represents the user's preference at the current time, and its characteristics come from the historical sequence of user interactions The enhancement sequence (The specific enhancement strategy will be presented in Section S2), and the knowledge graph of historical interaction items, i.e. . Assume the initial state is ,in, Indicates that the initial state is an empty set. According to the current state, the agent selects an item from the item candidate set I Make a recommendation, recorded as an action , the action behavior is determined by the strategy Modeling, the policy defines a function that transforms the state As input, it outputs the distribution of all possible items. The softmax function is used to calculate the probability of selecting an item:

[0100] ,

[0101] in, represents the embedding vector of the jth item, are the parameters of the bilinear matrix, Representation status The state transition probability is used to update the state and is defined as follows:

[0102] ,

[0103] This means that when the agent takes an action at time step t After that, the system will and actions Transition to next state .

[0104] S5. Perform model training on the sequence enhancement model, including model optimization using the Adam optimizer;

[0105] Based on the trained sequence enhancement model, real-time recommendations are made to users.

[0106] The Adam optimizer is used for model optimization. The Adam optimizer is an optimization algorithm that combines the advantages of adaptive learning rate and momentum, which can effectively accelerate the model training process and improve convergence. When training the sequence enhancement model, the Adam optimizer adjusts the learning rate according to the gradient information of the model parameters, so that the model can be updated faster in the direction of the optimal solution. Specifically, the Adam optimizer maintains the first-order moment estimate (i.e., the exponential moving average of the gradient) and the second-order moment estimate (i.e., the exponential moving average of the square of the gradient) of each parameter, and dynamically adjusts the learning rate based on these estimates.

[0107] The goal of training a sequence enhancement model is to minimize the loss function, which usually measures the difference between the recommendation results predicted by the model and the actual user interaction. During the training process, the model receives a large amount of user interaction data as input, including the user's historical interaction sequence, the knowledge graph information of the project, etc.

[0108] For each time step, the model selects an item for recommendation based on the current state, and calculates the value of the loss function based on the user's feedback on the recommended item (such as clicks, purchases, etc.). Then, the gradient of the loss function is propagated back to the various parameters of the model through the back-propagation algorithm, and the Adam optimizer is used to update the parameter values ​​according to the gradient information.

[0109] When the sequence enhancement model is trained, it is used to make real-time recommendations to users. In actual applications, when a user interacts with the system, the model calculates the probability of each item being recommended based on the user's current state (including the user's historical interaction sequence and knowledge graph information). Specifically, the model first obtains the user's current state representation, that is, it obtains the state representation vector of the enhanced sequence by performing an enhancement operation on the user's historical interaction sequence. , combined with the current preference knowledge features in the knowledge graph and future preference prediction features , and the final state representation is , then, the state is represented as Input into the model and calculate the recommendation probability of each item through the softmax function ), finally, one or more items are selected and recommended to the user according to the recommendation probability.

[0110] Example 2

[0111] The difference between this embodiment and embodiment 1 is that this embodiment provides a reinforcement learning sequence recommendation system based on dual-state enhancement, including:

[0112] The data acquisition module is configured to: acquire the knowledge graph data of the project and sort the knowledge graph data according to the timestamp to obtain the interaction sequence;

[0113] The preprocessing module is configured to: construct a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, constructing a knowledge-aware enhancement layer, and constructing a reward layer;

[0114] The conversion module is configured to: utilize the constructed sequence enhancement model in combination with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer;

[0115] The model module is configured to: input the output of the sequence enhancement model into the Markov decision framework and update the model state through the state transition probability;

[0116] The feature extraction module is configured to: perform model training on the sequence enhancement model, including model optimization using the Adam optimizer;

[0117] The transformation module is configured to make real-time recommendations to users based on the trained sequence enhancement model.

[0118] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device for a reinforcement learning sequence recommendation method based on dual-state enhancement.

[0119] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the reinforcement learning sequence recommendation method based on dual-state enhancement.

[0120] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A reinforcement learning sequence recommendation method based on dual state enhancement, characterized in that: include Obtain the knowledge graph data of the project and sort the knowledge graph data according to timestamps to obtain the interaction sequence; A sequence enhancement model is constructed according to the interaction sequence and the item distribution, including constructing a sequence enhancement layer, constructing a knowledge-aware enhancement layer and constructing a reward layer. The construction of the sequence enhancement layer includes adding, deleting and truncating operations based on the acquired interaction sequence, wherein the truncation operation includes a given sequence length and a shearing rate. For the original sequence, a truncation position is selected to generate a new sequence, the standard deviation of the time intervals of all subsequences is calculated, and the subsequence with the smallest standard deviation is selected to complete the sequence enhancement operation. The enhanced sequence is also input into the Transformer encoder for feature extraction, the high-dimensional item representation is projected to a low-dimensional dense vector using an item embedding matrix, the position feature of the sequence is captured by a learnable position embedding, the item embedding and the position embedding are added to obtain the input, the self-attention mechanism and the multi-head self-attention mechanism are used to capture the complex dependency between items in the sequence, and the quality and expression ability of the feature are improved by combining the position embedding and the item embedding and smoothing the GELU activation function. Finally, a recurrent neural network with a gated recurrent unit is used for aggregation to obtain a state representation vector of the enhanced sequence to capture the user's preference features based on the enhanced sequence; The construction of the knowledge perception enhancement layer includes using the knowledge graph to infer the current user's preferences, obtaining the embedded vector of the project through the TransE method, and using the average pooling method to aggregate the embedded vectors of the projects in the user's historical interactions to obtain a vector reflecting the knowledge characteristics of the user in the interaction history. At the same time, the multi-hop behavior of the knowledge graph is used, and the current preference and the historical knowledge graph are used as input to predict the user's preference in the future step through a multi-layer perceptron, so as to further enrich the user's state representation; Utilize the constructed sequence enhancement model combined with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer; The output of the sequence enhancement model is input into the Markov decision framework, and the model state is updated through the state transition probability; Perform model training on the sequence enhancement model, including model optimization using the Adam optimizer; Based on the trained sequence enhancement model, real-time recommendations are made to users.

2. According to claim 1, a dual-state enhanced reinforcement learning sequence recommendation method is characterized in that: The method of obtaining the knowledge graph data of the project and sorting the knowledge graph data according to timestamp includes obtaining data from three e-commerce recommendation datasets with different domains and sparsity levels, namely, the knowledge graph information of the project, sorting the records of each user according to the timestamp to form an interaction sequence, using the last item of each sequence as the test data for the next recommendation task, and using the remaining data as the training dataset.

3. The method for recommending sequences based on dual-state reinforcement learning according to claim 1, characterized in that: The construction of the reward layer includes adopting a dual reward strategy, dividing the reward into a sequence reward and a knowledge-aware reward. The sequence reward uses the BLUE metric to perform sequence reward recommendation, focusing on the matching degree between the predicted sequence and the user's actual interaction sequence. The knowledge graph reward uses the TransE method to aggregate items to obtain a feature vector representation of the subsequence, and uses cosine similarity to calculate the similarity between the actual subsequence and the predicted subsequence, focusing on the matching degree between the recommendation result and the knowledge characteristics of the user's preferred items.

4. The method for recommending sequences based on dual-state reinforcement learning according to claim 1, characterized in that: The state of the model is updated by the state transition probability, including obtaining the current state according to the knowledge graph of the enhanced sequence and the historical interaction items, selecting items from the item candidate set for recommendation according to the current state by the intelligent agent, and marking the recommended items as actions, and calculating the probability of selecting each item by the softmax function, and calculating the probability of transferring to the next state by the state transition probability function according to the current state and action, wherein the formula for calculating the probability of selecting each item is: ， in, is represented by the action taken by the agent at time step t, The state at time step t is Denoted as the embedding vector of the jth item, Expressed as the parameters of a bilinear matrix, Represented as status The embedding vector of .

5. A reinforcement learning sequence recommendation system based on dual state enhancement, executing the method as claimed in claim 1, characterized in that: include: The data acquisition module is configured to: acquire the knowledge graph data of the project and sort the knowledge graph data according to the timestamp to obtain the interaction sequence; The preprocessing module is configured to: construct a sequence enhancement model according to the interaction sequence and item distribution, including constructing a sequence enhancement layer, constructing a knowledge-aware enhancement layer, and constructing a reward layer; The conversion module is configured to: utilize the constructed sequence enhancement model in combination with a dual-state enhancement mechanism, including combining the output vectors of the sequence enhancement layer, the knowledge-aware enhancement layer, and the reward layer; The model module is configured to: input the output of the sequence enhancement model into the Markov decision framework and update the model state through the state transition probability; The feature extraction module is configured to: perform model training on the sequence enhancement model, including model optimization using the Adam optimizer; The transformation module is configured to make real-time recommendations to users based on the trained sequence enhancement model.

6. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executing a reinforcement learning sequence recommendation method based on dual-state enhancement as described in claim 1.

7. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the method for recommending a sequence of reinforcement learning based on dual-state enhancement as claimed in claim 1.

Images