An individualized test question recommendation method based on a knowledge graph and reinforcement learning

Abstract

The application discloses a kind of personalized test question recommendation method based on knowledge graph and reinforcement learning, by constructing user answering interactive knowledge graph and reinforcement learning model, the learning behavior data of user, knowledge point mastery and preferred item type are input into the good reinforcement learning model, generate recommended question list;Reinforcement learning model includes the multidimensional state representation of time sequence feature, semantic association and learning demand prediction, proposes the triple reward mechanism of fusion instant feedback, long-term effect and exploration incentive, and develops adaptive forgetting model to optimize review strategy.The application adopts knowledge point progressive relationship network to realize scientific advancement, and is suitable for course learning mechanism balance exploration and utilization, and personalized memory curve improves review efficiency.

Description

Technical Field

[0001] This invention belongs to the field of smart education technology, specifically relating to a personalized test question recommendation method based on knowledge graphs and reinforcement learning. Background Technology

[0002] In the field of personalized education, test recommendation technology is one of the core tools for improving learning efficiency. Traditional methods mainly rely on collaborative filtering (CF), knowledge tracing (KT), and deep learning-based sequence recommendation models. However, these technologies have significant shortcomings in dynamic modeling, knowledge correlation capture, and long-term goal optimization. For example, knowledge graph recommendation methods do not consider temporal learning states (CN112487152A), and the proposed reinforcement learning schemes lack knowledge point continuity constraints. Specifically, traditional collaborative filtering methods struggle to capture dynamic learning needs and are sensitive to sparse answer data; rule-based recommendations lack in-depth mining of knowledge point relationships; and reinforcement learning methods suffer from low exploration efficiency and a single reward signal in long-term learning effect optimization. Summary of the Invention

[0003] To overcome the shortcomings of the above technologies, this invention provides a personalized test question recommendation method based on knowledge graphs and reinforcement learning, which is applicable to scenarios such as online learning platforms and adaptive education systems, and solves problems such as uneven knowledge point coverage and insufficient optimization of long-term learning effects in traditional recommendation technologies.

[0004] The present invention is achieved by at least one of the following technical solutions.

[0005] A personalized test question recommendation method based on knowledge graphs and reinforcement learning includes the following steps:

[0006] By inputting user learning behavior data, knowledge mastery, and preferred question types into a reinforcement learning model, a list of recommended questions is generated.

[0007] Furthermore, the state representation s of the reinforcement learning model t Defined as a combination of three representation vectors for a state.

[0008]

[0009] Where h 1:t This represents a Transformer-based sequence-level state used to encode the temporal dependencies of a user's historical answer sequences; the sequence-level state representation h 1:t Calculated using the following formula:

[0010]

[0011] Where Transformer(·) is a gated loop unit, For question i t Knowledge graph embedding vector, Φ transformer This represents all relevant parameters of the Transformer network;

[0012] P 1:t The knowledge-level user preference state based on knowledge graph embedding is calculated using the following formula:

[0013]

[0014] Among them, e u This represents the user's embedding vector, and W is the weight matrix. The knowledge graph embedding vector represents the test question at time t. This represents the embedding of the user interaction question at time j;

[0015] f t+1:t+k This represents a prediction of users' future knowledge preferences, used to predict potential learning needs through a multilayer perceptron.

[0016] f t+1:t+k =MLP(P 1:t ;Φ mlp )

[0017] Among them, f t+1:t+k Φ represents the k-step future preference representation at time t. mlp This indicates the parameters used in the user preference-guided network.

[0018] Furthermore, the user's historical answer sequence is constructed based on user interaction data. First, the user's interaction records are sorted according to timestamps to ensure that each answer is arranged in chronological order. The sorted interaction records are then integrated into a user interaction sequence, forming an ordered historical answer sequence {i1,i2,…,i...}. t This records all of a user's answering behaviors within a specific time period.

[0019] Furthermore, the reward function of the reinforcement learning model is a multi-granularity composite reward function R(s). t ,a t It includes two different reward mechanisms, as shown in the following formula:

[0020]

[0021] in Indicates sequence-level rewards. Indicates knowledge-level reward, i t:t+kThis represents the user interaction subsequence from time t to time t+k. This represents the recommended subsequence from time t to time t+k.

[0022] Furthermore, the sequence-level reward function is:

[0023]

[0024] Where prec m w represents the matching precision of m-grams of m consecutive sequences. m The weights of m consecutive sequences m-grams satisfying Used to control the contribution of different m-grams to the reward; M is the length of the maximum m-gram; correction precision prec m The calculation formula is:

[0025]

[0026] Among them, i m For i t:t+k m-gram subsequences, #(i m i t:t+k ) for i m in i t+1:t+k The number of times something appears in the equation determines how many m-gram precision fractions to use.

[0027] Furthermore, the sequence-level reward function employs cosine similarity:

[0028]

[0029] Among them, P t+1:t+k and These represent the knowledge-level user preference states P based on knowledge graph embedding. 1:t The actual subsequence level and the predicted subsequence level are obtained.

[0030] Furthermore, the reinforcement learning model is trained using policy gradient optimization and pairwise ranking mechanisms, including:

[0031] First, a set of truncated subsequences is sampled using the Monte Carlo algorithm to train the Markov Decision Process (MDP). The Monte Carlo method generates multiple subsequences by random sampling to simulate user interaction behavior.

[0032] Secondly, a pairwise ranking mechanism is used to optimize user preference prediction. Based on the subsequences generated by the Monte Carlo method, future preference subsequences are derived. The reward value of the future preference subsequences is calculated according to the formula. The reward value is used to construct the preference order on multiple subsequences. Then, a pairwise constraint is added to train the user preference guidance network, which is used to rank multiple subsequences sampled by Monte Carlo according to knowledge-level rewards. It is required that the guidance network score of high reward subsequences is higher than that of low reward subsequences.

[0033] Furthermore, the formal definition of pairwise constraints is as follows:

[0034]

[0035] in, This indicates that the network induces user preference predictions for the l-th subsequence. The rating, Indicates the induction network for the lth ′ User preference prediction for each subsequence The rating, It is the knowledge-level reward for the l-th subsequence. It is the lth ′ The knowledge-level reward for each subsequence, i t:t+k This represents the user interaction subsequence from time t to time t+k. This represents the lth time from time t to time t+k. ω Recommended subsequences.

[0036] Furthermore, the loss function of the reinforcement learning model is:

[0037]

[0038] This represents the total loss of the reinforcement learning model. Let λ represent the loss function in the policy gradient method, and let λ represent the loss function used to control...

[0039] In total losses The weights in This represents the loss function used for pairwise comparison learning.

[0040] The present invention provides a computer device comprising: a memory and a processor, and a computer program stored in the memory, wherein when the computer program is executed on the processor, it implements the aforementioned personalized test question recommendation method based on knowledge graphs and reinforcement learning.

[0041] Compared with existing technologies, the beneficial effects of the present invention are as follows:

[0042] This invention combines user learning behavior data, knowledge mastery, and preferred question types, using reinforcement learning algorithms to dynamically optimize the recommendation strategy. This ensures that the recommended questions not only meet students' current learning needs but also guide them towards mastering higher-level knowledge. Unlike traditional test recommendation systems, this method uses a knowledge-guided reinforcement learning mechanism to combine the logical connections between knowledge points with users' personalized needs, constructing a dynamic, accurate, and efficient recommendation model. The system aims to help students learn efficiently, solving problems such as information overload and inaccurate question selection in traditional question banks, while providing students with clear learning path planning and targeted practice. Attached Figure Description

[0043] Figure 1 A framework diagram of a personalized test question recommendation method based on knowledge graphs and reinforcement learning according to an embodiment of the present invention.

[0044] Figure 2 This invention provides a flowchart for constructing a user-question interaction knowledge graph G and a user-question interaction sequence.

[0045] Figure 3 The flowchart illustrates the specific implementation of the personalized test question recommendation method of this invention. Detailed Implementation

[0046] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0047] like Figures 1-3 As shown in this embodiment, a personalized test question recommendation method based on knowledge graphs and reinforcement learning includes the following steps:

[0048] (1) Construct a user-question interaction knowledge graph G and a user-question interaction sequence. The user-question interaction knowledge graph G contains triple relationships of question attributes, knowledge points, and user behavior, wherein: the question entity attributes include knowledge point affiliation, difficulty coefficient, historical error rate, and chapter; the user entity attributes include learning stage, knowledge point mastery, and historical answering behavior characteristics, such as... Figure 2 As shown, the specific steps include the following:

[0049] like Figure 2 (a) shows the alignment of questions with knowledge points: the original question bank data was cleaned, redundant symbols and non-core content were removed, and mathematical formulas were stored in LaTeX standard format. Figure 2(b) Utilizes regular expressions and subject-specific custom dictionaries to optimize word segmentation, filter common stop words, and retain key semantic information. Based on the semantic similarity between the question text and knowledge graph entities (calculated using pre-trained word vectors), knowledge points are dynamically matched, and after threshold determination, association relationships are marked. Unmatched questions are manually labeled based on usage frequency to achieve full labeling of all questions.

[0050] like Figure 2 (c) and (d) are used to construct a user-question interaction knowledge graph G. Interaction data such as user ID, question ID, answer results, and duration are integrated to construct a "user-interaction-question" triple, establishing a multi-dimensional association between user behavior and knowledge points to support personalized recommendations and analysis.

[0051] A user-question interaction sequence is constructed based on user interaction data to facilitate analysis of the user's learning process. First, user interaction records are sorted according to timestamps to ensure that each answer is arranged chronologically. Next, the sorted interaction records are integrated into a user interaction sequence, forming an ordered historical answer sequence {i1, i2, ..., i...}. t This records all of a user's answering behaviors within a specific time period.

[0052] (2) Model the test item recommendation problem as a Markov decision process, and define the state representation for knowledge enhancement, including...

[0053] a. Transformer-based sequence-level state representation h 1:t This method encodes the temporal dependencies of a user's historical answer sequence. Sequence-level state representation aims to capture the temporal dependencies within the historical answer sequence. A Transformer model is used to encode the user's historical answer sequence to capture long-term dependencies within the sequence. Specifically, given a user's historical answer sequence {i1, i2, ..., i...} t}, sequence-level state representation h 1:t Calculated using the following formula:

[0054]

[0055] Where Transformer(·) is a gated loop unit, For question i t Knowledge graph embedding vector, Φ transformer This represents all relevant parameters of the Transformer network.

[0056] b. Based on the knowledge-level user preference state embedded in the knowledge graph, capture the semantic features of the association between knowledge points in the test questions. The goal of knowledge-level user preference representation is to enhance the understanding of user preferences by utilizing the structured information in the knowledge graph. In the user answer knowledge graph G, for each question i...t Corresponding to an entity To capture users' knowledge preferences, an additive attention mechanism is introduced. This involves modeling the relationships between nodes in the knowledge graph and assigning attention weights to the neighbors of each node, thereby aggregating information from neighboring nodes. Specifically, this is based on the knowledge-level user preference state P embedded in the knowledge graph. 1:t Calculated using the following formula:

[0057]

[0058] Among them, e u This represents the user's embedding vector, and W is the weight matrix. Indicates the user interaction question i at time t. t Knowledge graph embedding vectors. This represents the embedded representation of the user interaction question at time j.

[0059] c. User's future knowledge point preference prediction representation t t+1:t+k To predict potential learning needs, a user preference guidance network based on a multilayer perceptron (MLP) is constructed to capture the dynamic evolution of user interests and predict future user preference sequences. At time step t, a knowledge-level user preference state P based on knowledge graph embedding is predicted. 1:t k-step future user preferences as input:

[0060] f t+1:t+k =MLP(P 1:t ;Φ mlp )

[0061] Among them, f t+1:t+k Φ represents the k-step future preference representation at time t. mlp d. The final state representation of the reinforcement learning model. In this embodiment, the state representation s of the reinforcement learning model is... t Ultimately defined as a combination of three representation vectors for a given state.

[0062]

[0063] This state representation method naturally incorporates KG data into the MDP framework for state representation, which not only makes full use of the structured information in the knowledge graph, but also achieves a good balance between exploration and utilization, thereby improving the performance of the recommendation system.

[0064] As one example, a multi-granularity composite reward function is used as the reward function R(s) in the reinforcement learning model. t ,a tTo define the reward function for step k, two different reward mechanisms are integrated, as shown in the following equation.

[0065]

[0066] This multi-granularity reward function design aims to comprehensively evaluate the performance of the recommender system from both sequence-level and knowledge-level dimensions, ensuring that the recommended sequences are not only structurally consistent but also highly relevant in terms of semantics and knowledge features. Through this composite reward mechanism, the performance of the recommender system can be better balanced between exploration and exploitation, thereby achieving superior recommendation results.

[0067] In sequence recommendation tasks, an effective reward function needs to comprehensively consider the global consistency and local matching degree between the recommended sequence and the actual question interaction sequence. To this end, a reward function based on weighted n-gram matching is designed to more comprehensively evaluate the quality of the recommended sequence. Specifically, given the actual question interaction subsequence i... t+1:t+k and recommended subsequences Define the sequence-level reward function as follows:

[0068]

[0069] Among them, prec m w represents the matching precision of m-grams of m consecutive sequences. m The weights of m consecutive sequences m-grams satisfying Used to control the contribution of different m-grams to the reward; M is the length of the maximum m-gram. Precision correction (prec) m The calculation formula is:

[0070]

[0071] Among them, i m For i t:t+k m-gram subsequences, #(i m i t:t+k ) for i m in i t+1:t+k The number of times i appears in the sequence, M determines how many m-gram precision fractions to use, i t:t+k This represents the user interaction subsequence from time t to time t+k. If you want to focus more on local matching (short n-grams), you can set w... m To decrease the weight, such as To balance local and global matching, you can set a uniform weight. This algorithm employs uniform weighting, making the sequence more focused on global matching. This reward function encourages recommendation algorithms to generate more consistent m-grams from the actual sequence. It naturally measures the sequence-level performance of the task. This approach not only encourages recommendation algorithms to generate recommendations highly consistent with the actual sequence but also avoids the bias that can arise from single-granularity evaluation through multi-granularity assessment.

[0072] Knowledge-level reward functions, on the other hand, are not concerned with exact matching of questions, but rather with the quality of features reflecting the knowledge level within the sequence. Given an actual subsequence and a predicted subsequence i... t+1:t+k and Based on the knowledge-level user preference state P embedded in the knowledge graph 1:t The knowledge representations of the actual subsequence level and the predicted subsequence level are obtained, denoted as P respectively. t+1:t+k and These two knowledge representations reflect users' preferences for project attributes or features. To measure the similarity between these two vectors, cosine similarity is used as the knowledge-level reward function:

[0073]

[0074] By leveraging external knowledge graphs to enhance reinforcement learning for sequence recommendation, the key lies in learning effective knowledge-level state representations for sequence recommendation. To this end, a Markov decision process is first trained by sampling a set of truncated subsequences using the Monte Carlo algorithm. Based on these subsequences, a pairwise ranking mechanism is proposed to learn user preferences and guide the network. This approach better combines sequence-level and knowledge-level rewards, thereby improving the overall performance of the recommendation system.

[0075] As one example, a reinforcement learning model is trained using policy gradient optimization and pairwise ranking mechanisms:

[0076] First, a Markov Decision Process (MDP) is trained by sampling a set of truncated subsequences using the Monte Carlo algorithm. The Monte Carlo method generates multiple possible subsequences through random sampling, simulating user interaction behavior. Based on these subsequences, the state transition probabilities and the expected value of the reward function can be better estimated. According to state s... t The intelligent agent in a t An action a is performed at point A. t Select one question from question set I. t+1 For making recommendations, strategy π uses the softmax function to calculate the probability of an item, which is defined as follows:

[0077]

[0078] Where π(a) t |st ) indicates that in state s t Take action a t The probability, The embedded representation of the user interaction entity at time j Let W1 represent the embedding vector of the j-th question, and W1 be the parameter in the bilinear product. It is state s t The embedding vector.

[0079] Specifically, for each state s t Sample a k-step subsequence according to the policy function. This method allows for the generation of multiple subsequences for training and optimizing strategies.

[0080] A pairwise ranking mechanism is used to optimize the user preference prediction representation. Based on state s t L simulated subsequences were obtained Derive based on s t The k-step future preference subsequences are used to obtain the knowledge-level current user preference state based on the knowledge graph, denoted as . Based on the current user preference representation, MLP is used to predict the user preference prediction for the next k steps, denoted as [prediction]. Their reward values ​​are calculated according to a formula. Pairwise comparisons can be formed as an additional constraint to improve learning. Specifically, preference orders on L subsequences are first constructed using their reward values, given the user preference prediction for the l-th subsequence. and the l ′ User preference prediction for each subsequence Then, a pairwise constraint is added to train the user preference guidance network. This constraint sorts the L subsequences sampled in Monte Carlo according to their knowledge-level rewards, forcing the network to score higher for high-reward subsequences than for low-reward subsequences. The pairwise constraint is formally defined as follows:

[0081]

[0082] in, This represents the score given by the induced network to subsequence l. It is a knowledge-level reward for the subsequence, i t:t+k This represents the user interaction subsequence from time t to time t+k. This represents the lth time from time t to time t+k. ′ A recommended subsequence. A loss function is constructed using a pairwise comparison mechanism. as follows:

[0083]

[0084] and Let l represent the user preferences at times t+1 to t+k. ′ The l-th subsequence and the l-th subsequence. L represents the total number of subsequences.

[0085] The network is trained L times to optimize its parameters. This pairwise ranking mechanism is introduced to enhance the effectiveness of the user preference prediction network's learning.

[0086] The goal of policy gradient optimization is to learn a stochastic policy π to maximize the expected cumulative reward J(Θ) across all uses. The derivative of J(Θ) can be given by:

[0087]

[0088] Where n represents the total number of steps from the current time step t to the final time step, and u represents an identifier for the time step used in calculating the cumulative reward. π(a t |s t ;Θ) indicates that in state s t Choose action a t The probability, Let γ denote the gradient (i.e., derivative) of the objective function J(Θ) with respect to the policy parameter Θ. j-t Θ is the discount factor used to balance the importance of current and future rewards, Θ represents all the parameters to be learned, and R... j This represents the reward received at time step j. Represent the policy π with respect to a given state s t Select the next action a t The gradient of the policy is calculated. In this way, the policy parameter Θ can be updated based on the reward value of each subsequence, thereby optimizing the policy π. The key to the policy gradient method lies in calculating the importance weight of each action (i.e., the gradient of the policy parameter Θ). We adjust the parameters to maximize the expected return. Therefore, the loss function for policy gradient optimization can be expressed as:

[0089]

[0090] R represents the loss function in the policy gradient method. j This represents the instant reward received at time step j.

[0091] To further optimize the training process, a loss function is introduced to comprehensively consider the constraints of policy gradient and pairwise ranking. The design of the loss function for the reinforcement learning model is as follows:

[0092]

[0093] This represents the total loss of the reinforcement learning model. Let λ represent the loss function in the policy gradient method, and let λ represent the loss function used to control...

[0094] In total losses The weights in The loss function for pairwise comparison learning.

[0095] The first term is the log-likelihood loss of the policy gradient, used to maximize the expected return; the second term is the pairwise ranking constraint loss, used to optimize the performance of the user preference guidance network. By balancing these two losses, both the policy and user preference representations can be optimized simultaneously. The policy function is optimized using the loss function to update the state representation, and steps (2) to (7) are repeated to obtain the final personalized test recommendation model. A list of recommended questions is generated based on the finally learned personalized test recommendation model, and questions are recommended to the students. After the students complete the questions, the system updates the student state and knowledge graph based on the feedback (correct / incorrect) and enters the next round of recommendation. The students' feedback is used to update the state and knowledge graph, forming a closed-loop optimization that continuously improves the performance of the recommendation system.

[0096] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A personalized test question recommendation method based on knowledge graphs and reinforcement learning, characterized in that, Includes the following steps: The reinforcement learning model is built by inputting users' learning behavior data, knowledge mastery, and preferred question types, and a list of recommended questions is generated. State representation of reinforcement learning models Defined as a combination of three representation vectors for a state. : in This represents a sequence-level state based on Transformer, used to encode the temporal dependencies of a user's historical answer sequence; sequence-level state representation Calculated using the following formula: in, For gated loop unit, For the test questions Knowledge graph embedding vectors, This represents all relevant parameters of the Transformer network; The knowledge-level user preference state based on knowledge graph embedding is calculated using the following formula: in, Represents the user's embedding vector. It is a weight matrix. The knowledge graph embedding vector represents the test question at time t. express Embedding real-time user interaction questions; This represents a prediction of users' future knowledge preferences, used to predict potential learning needs through a multilayer perceptron. in, Indicates time of Future preference indication, These represent parameters used in the user preference-guided network. The reinforcement learning model is trained using policy gradient optimization and pairwise ranking mechanisms, including: First, a set of truncated subsequences is sampled using the Monte Carlo algorithm to train the Markov Decision Process (MDP). The Monte Carlo method generates multiple subsequences through random sampling to simulate user interaction behavior. Secondly, a pairwise ranking mechanism is used to optimize user preference prediction. Based on the subsequences generated by the Monte Carlo method, future preference subsequences are derived. The reward value of the future preference subsequences is calculated according to the formula. The preference order on multiple subsequences is constructed using the reward value. Then, a pairwise constraint is added to train the user preference guidance network, which is used to rank multiple subsequences sampled by Monte Carlo according to knowledge-level rewards. It is required that the guidance network score of high reward subsequences is higher than that of low reward subsequences. The formal definition of pairwise constraints is as follows: in, Indicates the influence of the network on the first User preference prediction for each subsequence The rating, Indicates the influence of the network on the first User preference prediction for each subsequence The rating, It is the first Knowledge-level reward for each subsequence It is the first Knowledge-level reward for each subsequence express Time's up Time-based user interaction subsequence express Time's up Time of the first Recommended subsequences.

2. The personalized test question recommendation method based on knowledge graph and reinforcement learning according to claim 1, characterized in that: The user's historical answer sequence is constructed based on user interaction data. First, the user's interaction records are sorted according to timestamps to ensure that each answer is arranged in chronological order. The sorted interaction records are then integrated into a user interaction sequence, forming an ordered historical answer sequence. It records all of a user's answering behaviors within a specific time period.

3. The personalized test question recommendation method based on knowledge graphs and reinforcement learning according to claim 1, characterized in that: The reward function of a reinforcement learning model is a multi-granularity composite reward function. It includes two different reward mechanisms, as shown in the following formula: in Indicates sequence-level rewards. This indicates a knowledge-level reward. express Time's up Time-based user interaction subsequence express Time's up Recommend subsequences at all times.

4. The personalized test question recommendation method based on knowledge graphs and reinforcement learning according to claim 3, characterized in that: The sequence-level reward function is: in This represents the matching precision of an m-gram of m consecutive sequences. The weights of m consecutive sequences m-grams satisfying This is used to control the contribution of different m-grams to the reward; It is the length of the largest m-gram; Correction accuracy The calculation formula is: in, for m-gram subsequences, for exist The number of times it appears in Determine how many m-gram precision fractions to use.

5. The personalized test question recommendation method based on knowledge graphs and reinforcement learning according to claim 3, characterized in that: The knowledge-level reward function uses cosine similarity: in, and These represent the knowledge-level user preference states based on knowledge graph embedding. The actual subsequence level and the predicted subsequence level are obtained.

6. The personalized test question recommendation method based on knowledge graph and reinforcement learning according to claim 1, characterized in that: The loss function of the reinforcement learning model is: This represents the total loss of the reinforcement learning model. This represents the loss function in the policy gradient method. Indicates use for control In total losses The weights in This represents the loss function used for pairwise comparison learning.

7. A computer device, characterized in that, include: The memory and processor, and the computer program stored in the memory, which, when executed on the processor, implements a personalized test item recommendation method based on knowledge graphs and reinforcement learning as described in any one of claims 1 to 6.

Images