A Personalized Exercise Recommendation Method and Recommendation System Integrating Reinforcement and Contrastive Learning
By integrating reinforcement and contrastive learning methods, this approach addresses the issues of long-term learning needs and data sparsity in exercise recommendation, achieving more accurate exercise recommendations and enhanced learning gains, and is suitable for online education platforms.
Patent Information
- Application Number
- CN202310655470.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-05
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-06-05
AI Technical Summary
Existing exercise recommendation methods struggle to accurately capture students' long-term learning needs, suffer from data sparsity and overfitting issues, and ignore changes in students' knowledge status during the learning process, leading to unreasonable recommendations and dependency.
This study employs a fusion of reinforcement and contrastive learning approaches. By formalizing the exercise recommendation process into a Markov decision process through self-supervised reinforcement learning, it combines Double Q-learning and the DKT model. Data augmentation techniques are used to generate additional historical records, and contrastive learning is introduced to capture changes in students' knowledge states. Exercise recommendations are then performed through a self-supervised head.
It improves the accuracy and learning gain of exercise recommendations, enabling better recommendations of exercises that are suitable for improving students' learning abilities, alleviating the data sparsity problem, and improving recommendation performance and learning outcomes.
Smart Images

Figure CN119089030B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of reinforcement learning and contrastive learning, and relates to a personalized exercise recommendation method and recommendation system that integrates reinforcement and contrastive learning. Background Technology
[0002] With the development of the internet, intelligent education has become a new hot topic. Online intelligent education, as an extension of traditional education, can conveniently provide students with advanced technology to help them consolidate their knowledge. In recent years, online platforms such as MOOCs, Khan Academy, and Online Judging (OJ) have continuously expanded in scale, resulting in a vast amount of test questions, overwhelming students with choices; and teachers, burdened with numerous teaching tasks, finding it difficult to address the needs of every student. Therefore, helping students find suitable test questions from this vast sea of resources is an important task.
[0003] Personalized exercise recommendation refers to a method that uses students' learning history to understand their mastery of exercises. By analyzing information such as the correctness of answers, answering time, the questions answered, and the relevant knowledge points, students are more accurately identified, and exercises are recommended to help them learn more efficiently. In recent years, scholars have applied traditional recommendation algorithms, cognitive diagnostics, knowledge tracking, and reinforcement learning to exercise recommendation. Traditional recommendation treats students as users, exercises as products, and student scores as user ratings of products; therefore, traditional recommendation algorithms can be used to solve the exercise recommendation problem. This method is simple and easy to understand and has achieved good results. However, because there is a fundamental difference between the user-product relationship and the student-exercise relationship, traditional recommendation methods cannot consider other student attributes and have problems capturing changes in knowledge state during the learning process. Cognitive diagnostic methods in educational data mining can also be used for personalized exercise recommendations for students, such as the IRT model [1], which describes students as one-dimensional ability values and combines parameters such as exercise difficulty and discrimination to model students accordingly; the DINA model [2], as a multidimensional discrete cognitive model, introduces the question-knowledge point Q matrix and diagnoses the students' mastery of knowledge points by combining more information unique to the field of education. However, cognitive diagnostic methods can only give the discrete mastery of students' knowledge points and ignore the long-term needs of students in practice. On this basis, knowledge tracking technology has been proposed, the purpose of which is to track the changes in students' knowledge level through historical learning and thus recommend exercises. In addition, some scholars have also proposed exercise recommendation methods based on reinforcement learning in the field of education, such as the hierarchical reinforcement learning method [3] proposed by Zhang, which achieves exercise recommendation by modifying user data and adjusting the recommendation model on the modified data.
[0004] Although some current research has achieved good results in the task of recommending exercises, there are still some shortcomings. (1) Most traditional recommendation methods are based on students' preferences, which may lead to recommending exercises that are too difficult or too easy. They ignore information about exercises and knowledge concepts related to students and lack the rationality of recommendations. (2) Most existing exercise recommendation methods are based on students' current performance in answering questions, while ignoring students' knowledge status throughout the answering process, i.e., long-term needs. (3) When students interact with intelligent education systems, they are likely to rely on the courses provided by the system, resulting in a limited number of exercises processed, which leads to the problem of sparsity in educational data. For example, iFlytek collected only 0.99% of the interactions from their online education recommendation system ZhiXue. Representations learned from sparse datasets using reinforcement learning methods are prone to overfitting and are difficult to accurately capture students' knowledge status. Summary of the Invention
[0005] To address the challenges of capturing students' long-term needs, data sparsity between students and exercises, overfitting of hidden representations that fail to accurately capture students' knowledge states, and a focus on imitating student behavior while neglecting the improvement of students' learning abilities, this invention aims to provide a method that integrates reinforcement and contrastive learning to model students and recommend suitable exercises for different students. This invention uses students' learning history as input, obtaining exercise and interaction information through embedding and encoding layers. It introduces contrastive learning concepts, combining three data augmentation methods to obtain additional historical records, which are then integrated into reinforcement learning as state supplementation to capture long-term needs during the learning process. Then, the exercise and interaction information are concatenated to obtain an effective representation, which is then learned. Finally, a self-supervised head is used to obtain the exercise recommendation results.
[0006] Reinforcement learning is a machine learning technique that primarily guides an agent to take actions and interact with an unfamiliar environment to maximize rewards. This invention uses a self-supervised reinforcement learning method to formalize the problem recommendation process as a Markov Decision Process (MDP). The agent interacts with students in a continuous decision-making process, observing the environment (including a pool of students and a pool of problems) and performing actions to obtain rewards and learn how to make optimal decisions. The basic elements of an MDP include: state space, action space, state transition probabilities, reward function, and discount factor. The state space is the set of all possible states the agent can be in, and the action space is the set of all actions the agent can perform. The state transition probability is the probability distribution of the agent transitioning to the next state after performing an action in the current state. The reward function is the reward value obtained by the agent for performing an action in a given state. The discount factor is a discount coefficient used to calculate future rewards, reflecting the agent's uncertainty about future rewards. In an MDP, the agent needs to make decisions through a policy. A policy is a mapping that maps each state to a probability distribution of performing a certain action. The goal of the agent is to maximize the long-term expected cumulative reward by learning the optimal policy. The basic methods for solving the MDP problem are value iteration or policy iteration. In value iteration, the agent finds the optimal policy by iteratively updating the value function of each state. In policy iteration, the agent finds the optimal policy by iteratively updating the policy. In this invention, by focusing on the student's long-term needs in practice, the maximum long-term cumulative reward is obtained to maximize the student's long-term satisfaction with the system. Specifically, in this invention, the environment includes a student pool and a problem pool. At time step t, the agent receives the student's current state, selects a problem, and the student answers the problem. The agent receives a corresponding reward based on the student's performance (this invention designs a dynamic reward mechanism centered on improving student learning ability). Afterward, the process moves to the next state, and the agent proposes a new recommendation, and so on. The reinforcement learning part adopts the Double Q-learning method, using two reinforcement learning heads (RL heads). These two heads supervise and complement each other during the learning process. Generally, in Double Q-learning, the RL head refers to the estimator of the Q-value function. The Q-value function is a state-action value function used to represent the cumulative reward value. Double Q-learning employs two Q-value functions, Q1 and Q2. These two Q-value functions are used to address the estimation bias problem in the Q-learning algorithm, thereby improving the algorithm's learning performance.The Regression Header (RL) plays a crucial role in Double Q-learning, its primary function being to estimate the Q-value of taking an action in a given state. The RL head typically employs a deep neural network to fit the Q-value function, taking the current state and action as input and outputting an estimated Q-value for that action in the current state. Deep neural networks usually contain multiple fully connected or convolutional layers, with the specific structure chosen based on the nature of the problem and the characteristics of the data. In Double Q-learning, each update to the RL head requires using one Q-value function (Q1 or Q2) as the objective function and the other as the estimation function. Specifically, each update randomly selects one Q-value function as the objective function and uses its estimated value as the reward for taking an action in the current state. Then, the estimated value of the other Q-value function is used to estimate the optimal action in the next state, and the estimated Q-value for the current state is calculated. This avoids estimation bias issues inherent in Q-learning algorithms and improves the learning performance. The RL head also acts as a regularizer, helping to learn additional information and improve recommendation performance. Furthermore, this invention incorporates a dynamic reward mechanism, primarily utilizing the trained Deep Knowledge Tracking (DKT) model during the learning process to capture students' mastery of exercises before and after recommendations, thereby recommending exercises that enhance learning abilities. The reward function defined in this invention can also be used to fine-tune the model for better performance.
[0007] Double Q-learning is a reinforcement learning method that uses two estimators. Double Q-learning uses two Q-functions (corresponding to two estimators), and each Q-function updates the next state using the value of the other Q-function. Both Q-functions are learned from different experience sets. In this invention, it serves as the basic structure for the reinforcement learning portion.
[0008] The DKT (Deep Knowledge Tracing) model is a model that models a student's knowledge based on their historical question-answering sequences. Using this model, it can accurately predict a student's performance in future interactions. Specifically, the DKT model can be used to predict a student's knowledge mastery during the learning process. This model uses a neural network to model the student's knowledge state and continuously updates the model based on the student's answers. In the DKT model, each student is represented as a knowledge vector, where each element represents the student's mastery of a specific knowledge point. This knowledge vector is continuously updated as the student learns, reflecting changes in the student's knowledge state. When predicting the student's next answer, the DKT model employs a sequence modeling method. Specifically, the model treats the student's answer sequence as a time series and uses a recurrent neural network (RNN) to model the sequence. During training, the DKT model updates its parameters by minimizing the error between the predicted and actual values. Simultaneously, the model also employs techniques such as dropout to avoid overfitting. In this invention, the DKT model is mainly used in the design of dynamic reward mechanisms and the definition of learning gain metrics.
[0009] Contrastive learning, a learning method within the self-supervised learning paradigm, focuses on learning the common features among similar instances and distinguishing the differences between dissimilar instances. To address the sparsity between students and exercises, and the problem that hidden representations are prone to overfitting and often fail to accurately capture students' knowledge states, this invention introduces the concept of contrastive learning, integrating the relevant information obtained. The addition of contrastive learning can group similar learning histories together and separate different learning histories, thereby learning effective representations. Simultaneously, additional historical records obtained through domain-specific data augmentation methods within contrastive learning serve as a state supplement for the reinforcement learning component, resulting in more effective representations of students' knowledge states.
[0010] This invention proposes a personalized exercise recommendation method that integrates reinforcement and contrastive learning, the method comprising the following steps:
[0011] Step 1: Select the dataset and preprocess it.
[0012] Step two involves data augmentation of the original student history sequence. The data augmentation used combines three strategies: mask, permute, and replace, reflecting different aspects of the learning history and resulting in two relevant view representations of the original learning history. The augmented learning history uses a question masking method, which enhances the estimation of missing context and mitigates bias caused by sparse educational data. The permute method reorders interactions, generating more information about the student's learning process representation. The replace method replaces questions, treating the resulting learning history as noise in the original history, enhancing the model's error correction capabilities while improving accuracy and information utilization. The proficiency shown in the augmented learning history is similar to that before.
[0013] View representation transforms or processes each sample in the original dataset in different ways, generating multiple new samples. These new samples can be seen as different perspectives or representations of the original dataset, thus providing a more comprehensive description of the dataset's features. Using view representation techniques for data augmentation can improve the model's generalization ability and robustness, thereby better adapting to different application scenarios.
[0014] Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representations of the exercises and student-exercise interaction vectors, respectively;
[0015] Step four: Based on the exercise and interaction vector feature representations obtained in step three, encode the exercises and interaction vectors using existing ensembleable sequence recommendation models such as GRU and / or SASRec and / or NextItNet and / or Bert4Rec to obtain the corresponding hidden representations; In this invention, the above-mentioned existing sequence recommendation models can be integrated, and the above-mentioned sequence recommendation models play an encoding role in this invention to obtain hidden features;
[0016] Step 5: Concatenate the hidden states obtained in Step 4 to obtain the concatenated representation of the original problem and the interaction sequence. And the representation after data augmentation of exercises and interaction sequences. and After data augmentation, two related view representations will be obtained, as one positive pair;
[0017] Step 6: After obtaining the data-augmented hidden representations of the exercises and interactions from Step 4, the data is passed through an average pooling layer to obtain the historical representations of the entire set of exercises and interactions.
[0018] Step 7: Develop the latent representation obtained in Step 5. , and As input, the Q-value for each input is obtained through the Q-value function and concatenated to obtain the final Q-value. The Q-value is the function value in reinforcement learning, representing the expected total reward from the agent's choice of action up to the final state. Q-value = Reward obtained from the current action + Discount rate * Expected (cumulative) {State transition probability * Next state V-value}. The V-value of the next state refers to the expected reward obtained after taking an action in the current state and transitioning to the next state. The V-value is a value estimate for a state, used to assess whether the action taken in the current state is conducive to obtaining a high reward. The V-value can be calculated using the Bellman equation, which multiplies the V-value of the next state by a discount factor and adds it to the immediate reward of the current state. During the interaction between the reinforcement learning agent and the environment, the agent decides what action to take next based on the current state and available actions, then observes the feedback from the environment to obtain reward signals and the next state. In this process, the agent updates the V-value based on the current state's V-value and the reward obtained from the action taken, continuously optimizing the strategy to ultimately achieve the optimal decision. Then, the cumulative reward is maximized through self-supervised reinforcement learning methods;
[0019] Step eight involves training the model using gradient descent, combining self-supervised, reinforcement, and contrastive learning losses. The resulting student representations are then processed through a self-supervised head to obtain the final exercise recommendation results, suggesting more suitable exercises to the students.
[0020] In step one, the selected datasets are ASSIST09[5] and EdNet[6]. ASSIST09 is information collected from the ASSISTments online tutoring system for the 2009-2010 academic year; EdNet is a large-scale hierarchical dataset collected by the intelligent tutoring platform Santa since 2017. The datasets used all contain the required information: student ID, exercise ID, knowledge points contained in the exercise, whether the student answered correctly or incorrectly, the student's study time, and the student's time spent answering the questions. The preprocessing of the datasets includes deleting answer records without knowledge points, deleting records with student answer sequences ≤ 3, and deleting duplicate records caused by an exercise having two or more knowledge points.
[0021] In step two, the student set and the problem set are represented by U and Q, respectively, and the student history sequence is defined as... ={ , ,..., Each interaction consists of tuples. { Composed of, among which Let Q be the t-th exercise in the exercise set, and also include the related knowledge points k∈K involved in the exercise, where K is the knowledge point set, representing all the knowledge points contained in the exercise set Q. The answer is ∈{0,1}, meaning the question was answered correctly. It equals 1, otherwise It is 0.
[0022] The data augmentation method described in step two is a combination of three augmentation strategies: mask, permute, and replace. Inspired by the literature Contrastive Learning for Knowledge Tracing [4], this invention also uses a combination of multiple data augmentation methods. The difference from the literature is that, since the length of the student history sequence set in this invention is relatively small (set to 10), the operation of randomly deleting some exercises is abandoned. The specific description is as follows:
[0023] Random data augmentation was applied to each student's learning history. This involved randomly performing mask, permute, and replace operations on each student's learning history at a certain ratio. This ensured that the proficiency level displayed after data augmentation was similar to that before data augmentation, and that the learning history could be used as additional historical information to supplement the RL part.
[0024] Three data augmentation methods were used in a certain proportion, each employing symbolic representation. , , To indicate, , , The adjustment range is 0.2 to 0.5. The combination of the three data augmentation methods is mainly reflected in the fact that each method has a set proportion, that is, the original student learning sequence will be simultaneously masked, replaced, and reordered; the three augmentation methods are described below:
[0025] Data augmentation using the mask method: This method uses exercise masking, which replaces some exercises in the original learning history with special markers without changing the students' answers. Specifically, for each student sequence, a certain proportion ( A portion of the exercises are randomly masked, and special markers ([masks]) are placed at the corresponding positions of the masked exercises. The learning history of the appropriately masked portions can be seen as an enhanced version of the original learning history, which helps the model learn useful information better. Furthermore, this data augmentation method facilitates the estimation of missing context, which can, to some extent, avoid the bias caused by sparse educational data and learn better representations.
[0026] The permute method performs data augmentation by reordering subsequences of the original student history sequence. This means that the order within the sequence will change to some extent, but the student knowledge states represented by the interaction sequences will remain similar. The reordering is based on... The probability is used to randomly select a subsequence from the original sequence. For each historical sequence, a subsequence of length is selected. Start by randomly shuffling a continuous subsequence, represented as ( , ,..., ),in is the reordering ratio, and t is the time t of the answer sequence. After shuffling, the student history sequence is obtained after the "permute" method enhancement, which can serve as additional information for the reinforcement learning part. Because students do not obtain learning materials from elsewhere during the entire process of recommending exercises, it is assumed that each student's proficiency within a sequence remains consistent. In other words, during the learning process, students have already mastered certain knowledge concepts and can solve exercises related to those concepts, regardless of the order in which the exercises appear. The sequence after using the "permute" method generates more descriptive information about the student's learning process;
[0027] The `replace` method is used for data augmentation: this method replaces the original exercises with simpler or more complex ones based on the students' initial responses. Using replacement introduces noise, enhancing the model's error correction capabilities, improving accuracy, and increasing its ability to utilize information. The resulting learning history still presents a similar knowledge state to the original learning history.
[0028] To implement the replacement method, a knowledge structure is constructed using only simple cognitive relationships. The difficulty of each question is determined by obtaining all questions in the training dataset and the percentage of correct and incorrect answers. Question difficulty is expressed as... Let N(correct) represent the number of correct answers to this exercise. This represents the total number of answers to the exercise. Then, all exercises are sorted from lowest to highest order: {q(1), q(2), ..., q(N)}, where N represents the number of exercises. q(1) represents the simplest exercise, and q(N) represents the most difficult exercise. It is assumed that a student with more complex knowledge is more likely to answer lower-level concepts correctly, and vice versa. Therefore, when replacing knowledge points, the original answer patterns do not need to be changed. For the student's learning sequence, using... The probability is randomly selected for replacement. Given the c-th exercise, if the answer is correct, the original exercise q(c) is replaced with the simpler exercise q(cj); otherwise, it is replaced with the more difficult exercise q(c+j). (This does not change the student's answer to this question.)
[0029] Using data augmentation methods to analyze students' learning history sequences After processing and enhancement, two related view representations of the same history are obtained. , .
[0030] In step three, the embedding layer is divided into two parts: a problem embedding layer and a student-problem interaction embedding layer; the problems and interactions are represented by the two embedding layers respectively to obtain the corresponding embedding representations.
[0031] In step three, the steps of converting the original user history sequence and the enhanced sequence into vector feature representations are as follows:
[0032] (1) Constructing a problem matrix ∈ Student-exercise interaction matrix ∈ (Involving the correctness of student answers), where N represents the number of exercises and d represents the embedding dimension;
[0033] (2) Given an interaction =( , The problem embedding vector at time t is obtained through the embedding layer. = and interactive embedding vectors = ;
[0034] (3) Obtain the exercise embedding vector of the augmented data = and interactive embedding vectors = .
[0035] In step four, the steps for encoding exercises and interaction vectors using an existing ensembleable sequence recommendation model are as follows:
[0036] (1) Transform the task of recommending the next exercise into a multi-classification problem (that is, treat an exercise as a category, and the result of the multi-classification problem is which exercise to select), and encode a series of exercises and interactions using an existing ensembleable sequence recommendation model;
[0037] (2) Process the original student history sequence, for a given exercise embedding sequence Exercise Encoder Learning exercises are represented as follows: = Interactive encoder Embedded sequences through input interactions Obtain the corresponding student-exercise interaction representation = );
[0038] (3) The augmented student sequence is processed to obtain a representation of the learning history through the hidden representations of exercises and interactions: = ( ), = ( );
[0039] In step five, the student history sequence is obtained. and enhanced sequences , This is mainly achieved by piecing together the obtained exercises and interactive information. = ⊕ , = ⊕ , = ⊕ ;
[0040] In step six, the formula for obtaining the entire history representation of exercises and interactions is as follows:
[0041] =pool( ) =pool( ),
[0042] in, and This is the hidden representation of the enhanced exercises and interactions obtained in step four. `pool(·)` is an average pooling layer. The final output... ;
[0043] In step seven, the task of recommending exercises is viewed as a Markov decision process. A Markov decision process can be defined by (S, A, P, R, ...). Defined by a tuple consisting of ,γ):
[0044] In a Markov decision process, the state transition function T(s,a,s') represents the probability of transitioning to the next state s' after taking action a from the current state s. The state transition function can be expressed as T(s,a,s'): S×A×S→[0,1], meaning that the input is the current state s and the action a, and the output is the probability of transitioning to the next state s'. In some cases, the state transition function may depend on time t or the previous state-action sequence. The reward function R(s,a,s') represents the immediate reward obtained by transitioning to state s' after taking action a from state s. The reward function can be expressed as R(s,a,s'): S×A×S→R, meaning that the input is the current state s, the action a, and the next state s', and the output is the immediate reward value. The state transition function and reward function are usually assumed to be properties of the environment and are determined during the problem modeling phase. Ultimately, the goal of a Markov decision process is to find a policy π: S→A that maximizes the expected reward. Expected reward is the cumulative value of the immediate rewards obtained from the state-action sequence, decayed by a discount factor γ, and can be expressed as: ,in This represents the immediate reward obtained at time t.
[0045] (1) State S is used to simulate the student's learning state space. At time t, state S is... Let be defined as the student's prior learning history. The student's state at time t can be represented as... =G( )∈S(t>0) (Student state representation is obtained through recommendation model G);
[0046] (2) Action space A, which is a discrete action space containing all exercises. Based on state... , ∈A is defined as recommending exercises to students. In offline data, the actions at time t can be obtained from the interaction between the student and the exercises;
[0047] (3) State transition probability P, S×A×S → It indicates that a state will be entered after the Agent takes an action. Mapping to a new state The probability of;
[0048] (4) Reward function R. The agent recommends based on state. Exercises (actions) are used to interact with students in the environment. Rewards (R) are given based on student feedback. , );
[0049] (5) Initial state distribution , It has ~ The initial state distribution of (normal distribution);
[0050] (6) The discount factor γ for future rewards. γ∈[0,1], where γ=0 means that only immediate rewards are considered, and γ=1 means that all future rewards are fully considered.
[0051] In step seven, given the representation of students' learning of history, the task of recommending exercises is transformed into a multi-classification problem. Reinforcement learning methods are used to maximize cumulative rewards, primarily seeking the optimal strategy. Recommend practice problems to students to improve their learning abilities. This is achieved by maximizing the expected cumulative reward, as shown in the following formula:
[0052] [R(τ)],
[0053] R(τ)= ,
[0054] θ∈ This represents the parameters of the policy function. τ=( , , , The trajectory is generated by the objective policy and obtained by performing operations based on the objective policy that maximizes cumulative reward. ~ , ~ (·| ), ~P(·| , These correspond to the initial state distribution, the choice of action, and the state transition probability, respectively.
[0055] The specific steps involved in self-supervised learning and reinforcement learning are as follows:
[0056] (1) Obtain the hidden states of the exercises and interactions based on step five;
[0057] (2) The recommendation model accepts the embedding vector as input and generates n-class logs. =[ , ,..., ]∈ n is the number of candidate exercises; the categorized log is composed of... =f( This was obtained using a fully connected layer.
[0058] (3) By using logarithms Optimize cross-entropy loss to train the recommendation model:
[0059] = - , = ,
[0060] in, It is an indicator function. If the student interacts with the i-th exercise at the next timestamp, then =1, otherwise =0. Since the recommender model G has already encoded the input vector into a latent representation... Therefore, it can be used directly. This serves as the current state of the RL head, without requiring the introduction of an additional model. What is needed is to stack an extra final layer to perform the corresponding Q-value calculation.
[0061] ( , ) = δ( + ) = δ(G( ) + ),
[0062] Where z∈{1,2,3}, ∈ , , }, ∈{ , δ is the activation function. These are learnable parameters;
[0063] (4) The obtained , , These are superimposed into a vector Q-value function, where, , , Corresponding to , , The calculated Q-value, a function value in reinforcement learning, represents the expected total reward from the agent's choice of action up to the final state. Q-value = Reward for the current action + Discount rate * Expected (cumulative) {State transition probability * Next state V-value}:
[0064] Q( , ) = [ ( , ), ( , ), ( , )],
[0065] (5) At each time step t, Q-Network obtains a small batch of experience samples (generally, the batch size can be set to 64, 128, 256, or 512 or even larger as needed, and in this invention the batch size is set to 256) from the experience pool D generated from the implicit feedback data. , The temporal difference (TD-error) is used to calculate the loss of the RL component, and its loss function is:
[0066] = ,
[0067] in, It is the immediate reward in the current state. It is the successor state. This is the current state. and Let represent the estimated values of the state-value functions for the current state and the successor state, respectively, and γ be the discount factor, representing the discount ratio for future rewards.
[0068] Using empirical replay improves sampling efficiency, and training a fixed target network can prevent approximation errors. and Rapid propagation between two states;
[0069] in The dynamic reward mechanism set up for this invention serves as an immediate reward for the current state, and the specific steps are as follows:
[0070] I. To improve the basic recommendation model G by recommending exercises to enhance students' learning abilities, a pre-trained DKT model is used. This model uses an LSTM as a neural network layer to simulate the interaction between students and exercises.
[0071] II. Based on the results obtained from the DKT model, a dynamic reward mechanism is designed. The reward mechanism is defined as follows:
[0072] ,
[0073] ,
[0074] Where N is the number of exercises, and β is an adjustment factor, empirically set to 10. This adjustment factor is used to increase the obtained reward value, because the reward value in the formula is the difference in knowledge mastery before and after the recommendation. This value is relatively small. To make the impact of the reward value larger (e.g., 0.001 and -0.001 have very small positive and negative effects, but if multiplied by 10 or 100, the positive and negative values will become larger, and in the final cumulative reward, the reward value may change from 0.18 and -0.18 to 1.8 and -1.8), it needs to be multiplied by a corresponding coefficient, generally 10, 100, etc. Specifically, first, the student's exercise sequence before time t is input to obtain the student's mastery of each exercise before the recommendation, and the correctness information of the recommended exercises can be obtained from it. Then, the interaction sequence at time t is input. Inputting into the DKT model, we obtain the students' mastery level after the recommended exercises, with the mastery level expressed as a probability value. The final reward is calculated as the average difference between the mastery levels of all exercises at time t and time t-1, representing the increment of the student's current knowledge state. This dynamic reward mechanism effectively captures changes in students' knowledge state during the learning process, allowing for corresponding rewards based on these changes to enhance model training.
[0075] (6) RL acts as a regularizer, based on the rewards defined in the reward mechanism. The model G is fine-tuned by evaluating the recommendation quality of the top k exercises;
[0076] (7) To improve the stability of learning, Double Q-learning is used to alternately train the learnable parameters.
[0077] In step eight, the steps for the contrastive learning loss are as follows:
[0078] (1) Obtain a positive pair from the data augmentation in step two. For this pair, it is instantiated as the exercise history. , ) and interaction history ( , );
[0079] (2) Represent other historical augmented samples from the same mini-batch (the same content involved in the aforementioned self-supervised learning and reinforcement learning, the batch size of the mini-batch here is also 256) as negative samples. ∈ , ∈ ;
[0080] The representation of negative samples for the exercises. This represents the interaction of negative samples;
[0081] (3) Define the comparative learning loss using two parts: exercises and interactions:
[0082] = -log ,
[0083] = -log ,
[0084] Here, sim is a cosine similarity function.
[0085] Using a contrastive loss function to learn effective representations, we can bring together similar learning histories and separate different learning histories.
[0086] In step eight, the model is trained using gradient descent, and the final overall objective function is a linear combination of self-supervised, reinforcement, and contrastive losses:
[0087]
[0088] Here, α is a hyperparameter representing the proportion of influence of the contrastive learning (CL) component.
[0089] The present invention also provides a system for implementing the above-described exercise recommendation method, the system comprising: a student historical sequence conversion module, a student knowledge state capture module, and an exercise recommendation module;
[0090] The learning history sequence transformation module processes the student history sequence. First, it obtains the original student sequence, the data-enhanced sequence 1, and the data-enhanced sequence 2. Then, it divides the obtained student sequence (including the data-enhanced sequence) into two parts: exercises (containing only exercise sequences) and interactions (containing exercise information and correct / incorrect information). These parts are then processed through an exercise embedding layer and a student-exercise interaction embedding layer, respectively, and finally through a recommendation model layer (exercise encoder and interaction encoder) to obtain the feature representations of the exercise and interaction vectors.
[0091] The student knowledge state capture module is used to learn student exercise and interaction representations. A self-supervised reinforcement method is used to formalize the exercise recommendation process as a Markov decision process, involving interaction between the student and the recommendation agent. By leveraging the student's long-term needs during practice, the maximum long-term cumulative reward is obtained to maximize student long-term satisfaction with the system. RL heads are used as regularizers, and a Double Q-learning approach is employed, with the two RL heads mutually supervising and complementing each other during the learning process. Reinforcement learning helps learn additional information and improve recommendation performance; contrastive learning is introduced, fusing the information obtained, incorporating relevant knowledge points, and interaction information obtained after encoder processing, to learn effective representations; simultaneously, domain-specific data augmentation methods in contrastive learning are used to obtain enhanced exercise and interaction information, which can serve as a state supplement for the reinforcement learning part. This method alleviates the data sparsity problem to some extent. When students interact with the intelligent system, they easily rely on the courses provided by the system, resulting in a limited number of exercises processed, thus leading to the problem of educational data sparsity. For example, iFlytek collected data from their online education recommendation system ZhiXue (zhixue.com) showing only 0.99% interaction. Representations learned from sparse datasets using reinforcement learning methods are prone to overfitting and struggle to accurately capture students' knowledge states. This invention uses three enhancement methods to obtain enhanced learning representations that are similar in proficiency to the original student response sequences. These enhanced representations can be added to reinforcement learning as supplementary information, providing additional data for the reinforcement learning process.
[0092] The student knowledge status capture module also introduces a custom reward mechanism to promptly obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the goal of improving students' learning abilities.
[0093] The proposed exercise recommendation module integrates the learning representations obtained from the two modules mentioned above and recommends exercises through a self-supervised head. The proposed exercise recommendation method can effectively improve learning gain and recommendation accuracy, enabling it to recommend exercises that can enhance learners' learning abilities.
[0094] The beneficial effects of this invention include:
[0095] 1. Using self-supervised reinforcement learning, the exercise recommendation process is formalized as a Markov decision process, with interactions between the student and the recommendation agent (intelligent agent). By focusing on the long-term needs of learning during practice and obtaining long-term cumulative rewards, the long-term student satisfaction with the system is maximized. The RL head, acting as a regularizer, helps learn additional information and improves recommendation performance.
[0096] 2. A novel dynamic reward mechanism is proposed. Using a pre-trained DKT model, it captures students' mastery of exercises before and after recommendation, and assigns positive or negative rewards based on this mastery to fine-tune the recommendation model. Furthermore, it focuses more on recommending exercises that can improve students' learning abilities.
[0097] This invention introduces the concept of contrastive learning and integrates the information obtained from it. Adding contrastive learning can group similar learning histories together and separate different learning histories, thereby learning effective representations. It can also incorporate relevant knowledge points and interaction information obtained after the encoder. This invention also uses additional historical records obtained through domain-specific data augmentation methods in contrastive learning as a supplement to the state in the reinforcement learning part. Using this method alleviates the data sparsity problem to some extent.
[0098] 3. The method proposed in this invention was integrated with four advanced recommendation models (GRU, SASRec, NextItNet, Bert4Rec), and corresponding experiments were conducted on the ASSIST09 and EdNet datasets. On the ASSIST09 dataset, the performance of HR and NDCG recommendation metrics was improved by 4%-13%. In EdNet, the method of this invention brought an average performance improvement of 6%.
[0099] 4. The method of this invention also shows a significant improvement in the student learning gain index (L_G). This indicates that the present invention can achieve better learning gain and recommend exercises that can improve students' learning abilities. Attached Figure Description
[0100] Figure 1 This is a flowchart of the present invention.
[0101] Figure 2 This is a diagram of the model architecture that integrates reinforcement and contrastive learning in the embodiment.
[0102] Figure 3 This is a system module diagram of the present invention.
[0103] Figure 4 This is a schematic diagram of the student history sequence conversion module of the present invention. Detailed Implementation
[0104] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.
[0105] This invention provides a personalized exercise recommendation method that integrates reinforcement and contrastive learning. It enhances the standard recommendation model through two output layers: self-supervised learning and reinforcement learning, while fusing information obtained from contrastive learning to achieve exercise recommendation for learners. First, a self-supervised reinforcement learning method is used to formalize the exercise recommendation process as a Markov decision process (MDP), facilitating interaction between the student and the recommendation agent. The RL head acts as a regularizer, helping to learn additional information and improve recommendation performance. Then, contrastive thinking is incorporated to group similar learning histories together and separate different ones, thereby learning effective representations. Three data augmentation methods are used to generate additional data, which is then used for contrastive learning and incorporated into the RL as supplementary information. This invention also uses a trained DKT model to capture the student's exercise mastery before and after recommendation, thereby setting a novel reward mechanism. Finally, student and exercise-related information is used as training data to train the proposed framework. Representations of questions and interactions are learned, then concatenated accordingly, and finally, exercise recommendations are performed through a self-supervised head. This invention also provides a system for implementing the above method.
[0106] The personalized exercise recommendation method provided by this invention, such as Figure 1 As shown, the method includes the following steps:
[0107] Step 1: Select the dataset and preprocess it.
[0108] Step two involves data augmentation of the original student sequence. The data augmentation method used combines three data augmentation strategies to reflect different aspects of the learning history, resulting in two relevant view representations of the original learning history. .
[0109] Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representations of the exercises and interaction vectors, respectively.
[0110] Step four: Based on the exercise and interaction vector feature representations obtained in step three, encode the exercises and interaction vectors using existing sequence recommendation models to obtain the corresponding hidden representations.
[0111] Step 5: Concatenate the hidden states obtained in Step 4 to obtain the concatenated representation of the original problem and the interaction sequence. And the representation after data augmentation of exercises and interaction sequences. and .
[0112] Step six: Based on the hidden representations of the exercises and interactions obtained in step four, the entire historical representation of the exercises and interactions is obtained through an average pooling layer.
[0113] Step 7, take the result obtained in Step 5 , and As input, the exercise recommendation is treated as a Markov decision process, and the corresponding Q-values are obtained and concatenated. Then, a self-supervised reinforcement learning method is used to maximize the cumulative reward.
[0114] Step eight involves training the model using gradient descent, combining self-supervised, reinforcement, and contrastive learning losses. The resulting student representations are then processed through a self-supervised head to obtain the final exercise recommendation results, suggesting more suitable exercises to the students.
[0115] The present invention also provides a system for implementing the above-described exercise recommendation method, the system comprising: a student historical sequence conversion module, a student knowledge state capture module, and an exercise recommendation module;
[0116] The learning history sequence transformation module processes the student history sequence. First, the original student sequence is augmented to obtain augmented sequence 1 and augmented sequence 2. Then, the obtained student sequence (including the data-augmented sequence) is divided into two parts: exercises (containing only exercise sequences) and interactions (containing exercise and correct / incorrect information). These parts are then processed through an exercise embedding layer and a student-exercise interaction embedding layer, respectively. Finally, the exercise encoder and interaction encoder are used to obtain the feature representation of the exercise and interaction vectors.
[0117] The student knowledge state capture module is used to learn student exercise and interaction representations. A self-supervised reinforcement method is used to formalize the exercise recommendation process as a Markov decision process, with interaction between the student and the recommendation agent. By leveraging the student's long-term needs during practice, the maximum long-term cumulative reward is obtained to maximize student long-term satisfaction with the system. An RL head is used as a regularizer, and a Double Q-learning method is employed, with the two RL heads mutually supervising and complementing each other during the learning process. Reinforcement learning helps learn additional information and improve recommendation performance; contrastive learning is introduced, fusing the information obtained therein with knowledge point information and interaction information obtained after the encoder, thereby learning effective representations; simultaneously, domain-specific data augmentation methods in contrastive learning are used to obtain enhanced exercise and interaction information, which can serve as a state supplement for the reinforcement learning part. This method alleviates the data sparsity problem to some extent.
[0118] The student knowledge status capture module also introduces a custom reward mechanism to promptly obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the goal of improving students' learning abilities.
[0119] The proposed exercise recommendation module integrates the learning representations obtained from the two modules mentioned above and recommends exercises through a self-supervised head. The proposed exercise recommendation method can effectively improve learning gain and recommendation accuracy, achieving the goal of recommending exercises that can enhance learners' learning abilities.
[0120] Example 1
[0121] The specific process of this embodiment is as follows:
[0122] (1) First, select the datasets ASSIST09 and EdNet. Perform data preprocessing, delete answer records without knowledge points, delete records with student answer sequences ≤ 3, and delete duplicate records caused by a question having two or more knowledge points.
[0123] (2) For the dataset used, the input sequence consists of the last ten questions before the target timestamp. If the sequence length is less than 10, 0 is generally used as the padding item to indicate that there are no questions answered in this part. Each student's historical sequence is stored according to the answer time, and the attributes include questions, knowledge points, and whether the answers are correct or incorrect.
[0124] (3) The preprocessed data is divided into training set, validation set and test set in a ratio of 8:1:1.
[0125] (4) Adopt , , The probability is calculated by using three data augmentation methods to augment the original input student sequence, resulting in the corresponding positive student sequence. , Data augmented using data augmentation techniques can be viewed as noisy points in the original learning history to better learn useful information, while maintaining consistency in the student's knowledge state represented in the sequence.
[0126] (5) Then input the obtained student sequence into the exercise embedding layer and the interaction embedding layer to obtain the embedding representations of the exercises and interactions respectively.
[0127] (6) Then input the obtained embedding representation into the exercise encoder. and interactive encoder In this process, the problem embedding and the student-problem interaction embedding are encoded to obtain the corresponding hidden representations. The hidden representations are then concatenated to form the state representation of the student's history.
[0128] (7) Next, calculate a*= which maximizes the Q value. Q( ( ), that is, to select recommended exercises.
[0129] (8) Next, a* is added to the original sequence to obtain the sequence of questions to be answered after the question is recommended. Then, it is input into the trained DKT model to obtain the students' mastery of the questions after the question is recommended. The corresponding reward is obtained according to the reward mechanism set by this invention. and transition to the next state. This cycle continues.
[0130] (9) Next, use the training data to calculate the TD-error and self-supervised and contrastive losses, update the parameters, and use the validation set to fine-tune the parameters.
[0131] (10) Using the pre-trained parameters, the learned student representations are passed through a self-supervised head to obtain the final exercise recommendation results, and more accurate exercises are recommended to the students.
[0132] The method of this invention can also be used for other educational datasets, and the specific process will not be described in detail here.
[0133] The parameters in the above embodiments of the present invention are determined based on the experimental results of the validation set; that is, different combinations of parameters are tested on the validation set, and a set of parameters with better accuracy is selected. Specifically, for trainable parameters, such as weights... bias The gradient descent algorithm is used for training. Because the method proposed in this invention can extend existing recommendation models and be integrated with them, different recommendation models require different model parameters. For example, the hidden state size is 64 for the GRU model; the number of self-attention heads is set to 1 for the SASREC model, etc. The model used in this invention also has common parameters, such as a mini-batch size of 256, a learning rate of 0.01, a dropout rate of 0.1, and a discount factor γ of 0.5. γmask, γrep, and γper are adjusted within the range of 0.2 to 0.5; in this embodiment, they are set to {0.2, 0.3, 0.5}. In actual testing, adjusting the above parameters appropriately according to requirements can also achieve the purpose of this invention.
[0134] Example 2
[0135] This invention considers four different sequential recommendation models and integrates them into the proposed method to test the effectiveness of the invention:
[0136] (1) GRU[7]: is a pioneering model for learning user behavior sequences.
[0137] (2) NextItNet[8]: It is a simple and effective model whose network structure consists of multiple stacked convolutional layers.
[0138] (3)SASRec[9]: It is a sequence recommendation model built on the transformer structure, which can capture the long-term dependencies between different interactions in the student behavior sequence.
[0139] (4) Bert4Rec
[10] : is a converter-based sequence recommendation model that uses bidirectional self-attention to model student behavior.
[0140] This invention was tested on two datasets:
[0141] (1) ASSIST09[5]: This dataset was collected from data from ASSISTments (an online tutoring system) for the academic years 2009-2019. This dataset is divided into two data files, which were spliced together in the experiment. After removing the "NAN" knowledge concept and repeated sequences containing multiple knowledge concepts, 112 knowledge points were provided, and more than 3,000 students answered 15,000 questions. The dataset contains a total of 279,658 interactions, and the sparsity of the dataset is 99.5%.
[0142] (2) EdNet[6]: This dataset is a large-scale dataset collected by Santa (an AI tutoring platform) since 2018. It contains 131,417,236 interactions from 784,309 students, making it the largest public dataset in terms of both total number of students and number of interactions. There are four datasets of different levels, KT-1, KT-2, KT-3 and KT-4, all of which contain information such as the question time, student ID, exercise ID, and student-submitted answers. In this invention, 4% of the student data was randomly sampled on EdNet KT-1 for experiments. It contains 2,119 students and 277,000 interactions across 142 knowledge points and 11,000 exercises. The sparsity is approximately 98.8%.
[0143] This invention will be evaluated from two aspects: improving students' learning ability and recommendation accuracy. The evaluation indicators are defined as follows:
[0144] (1) Regarding the former aspect (improving students' learning ability), this invention uses students' learning gain as an indicator:
[0145] ,
[0146] Where M represents the number of interactions, and gain(·) indicates whether the student's ability has improved, that is, whether the student's mastery of the exercises is higher after the recommendation than before the recommendation; if so, it is 1, otherwise it is 0. The calculation principle of the exercise mastery level is the same as that of the reward part.
[0147] (2) For recommendation accuracy, the commonly used recommendation performance evaluation metrics, hit rate (HR) and standardized discounted cumulative return (NDCG), are used. Specifically, HR is used to assess whether the actual questions are in the recommendation list. NDCG considers the position of each actual answered question in the recommendation list. This is equivalent to standardization among different students, with higher rankings in the recommendation list being assigned higher scores.
[0148] Using the four integrated basic recommendation models of this invention, both recommendation accuracy and learning gain performance were improved on the ASSIST09 and EdNet datasets. On the ASSIST09 dataset, performance improvements for HR and NDCG recommendation metrics ranged from 4% to 13%. In EdNet, the method of this invention delivered an average performance improvement of 6%. For the L_G evaluation metric, improvements exceeding 10% were achieved on both datasets.
[0149] References
[0150] [1] Fan Xiao. Item response theory and classical test theory: Anempirical comparison of their item / person statistics. Education andPsychological Measurement, 1998, 58(3): 357-381.
[0151] [2] De la Thrre J. DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 2009, 34(1): 115-130.
[0152] [3] J. Zhang, B. Hao, B. Chen, C. Li, H. Chen, and J. Sun. Hierarchical reinforcement learning for course recommendation in moocs. Proceedings of the AAAI conference on artificial intelligence, 2019, 33(1):435-442.
[0153] [4] W. Lee, J. Chun, Y. Lee, K. Park, and S. Park. Contrastivelearning for knowledge tracing. Proceddings of the ACM Web Conference, 2022,2330-2338.
[0154] [5] M. Feng, N. Heffernan, and K. Koedinger. Addressing theassessment challenge with an online system that tutors as it assesses, Usermodeling and user-adapted interaction, 2009,19, 243–266.
[0155] [6] Y. Choi, Y. Lee, D. Shin, J. Cho, S. Park, S. Lee, J. Baek, C.Bae, B. Kim, and J. Heo. Ednet: A large-scale hierarchical dataset ineducation. in Artificial Intelligence in Education: 21st InternationalConference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II21. Springer, 2020, 69–73.
[0156] [7] B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk.Sessionbased recommendations with recurrent neural networks. arXiv preprintarXiv:1511.06939, 2015.
[0157] [8] F. Yuan, A. Karatzoglou, I. Arapakis, JM Jose, and X. He. Asimple convolutional generative network for next item recommendation. inProceedings of the twelfth ACM international conference on web search anddata mining, 2019, 582–590.
[0158] [9] W.-C. Kang and J. McAuley. Self-attentive sequentialrecommendation. 2018 IEEE international conference on data mining (ICDM). IEEE, 2018, 197–206.
[0159]
[10] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang.Bert4rec: Sequential recommendation with bidirectional encoderrepresentations from transformer. Proceedings of the 28th ACM international conference on information and knowledge management, 2019, 1441–1450.
[0160] The scope of protection of this invention is not limited to the above embodiments. The described embodiments are merely some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.
[0161] The term "comprising," and any variations thereof, is intended to cover a non-exclusive inclusion, meaning that a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus. Variations and advantages that may occur to those skilled in the art without departing from the spirit and scope of the invention are included in this invention and are protected by the appended claims.
Claims
1. A personalized exercise recommendation method integrating reinforcement and contrastive learning, characterized in that, The method includes the following steps: Step 1: Select the dataset and preprocess it. Step two involves data augmentation of the original student history sequence. This is achieved by combining three data augmentation methods: question masking, interactive permute, and question replacement, resulting in two relevant view representations of the original learning history. ; In step two, the student set and the problem set are represented by U and Q, respectively, and the student history sequence is defined as... ={ , ,..., Each interaction consists of tuples. { Composed of, among which ∈Q represents the t-th exercise in the exercise set, and also includes the related knowledge points k∈K involved in the exercise, where K is the knowledge point set, representing all the knowledge points contained in the exercise set Q; The answer is ∈{0,1}, meaning the question was answered correctly. It equals 1, otherwise The value is 0; the enhancement method includes masking, permute reordering, and replacement. Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representations of the exercises and student-exercise interaction vectors, respectively; In step three, the embedding layer includes an exercise embedding layer and a student-exercise interaction embedding layer; The steps for converting the original user history sequence and the enhanced sequence into vector feature representations are as follows: (1) Constructing a problem matrix ∈ Student-exercise interaction matrix ∈ Where N represents the number of exercises, d represents the embedding dimension, and the interaction matrix includes the correctness of the student's answers; (2) Given an interaction =( , The problem embedding vector at time t is obtained through the embedding layer. = and interactive embedding vectors = ; (3) Obtain the exercise embedding vector of the augmented data = and interactive embedding vectors = ; Step 4: Based on the exercise and interaction vector feature representations obtained in Step 3, encode the exercises and interaction vectors using ensembled sequence recommendation models, including GRU and / or SASRec and / or NextItNet and / or Bert4Rec, to obtain the corresponding hidden representations. In step four, the steps for encoding exercises and interaction vectors using existing recommendation models are as follows: (1) The task of recommending the next exercise is transformed into a multi-classification problem, and an ensemble sequence recommendation model is used to encode a series of exercises and interactions; (2) Process the original student history sequence, for a given exercise embedding sequence Exercise Encoder Learning exercises are represented as follows: = Interactive encoder Embedded sequences through input interactions Obtain the corresponding interactive representation of the exercises. = ); (3) The augmented student sequence is processed to obtain a representation of the learning history through the hidden representations of exercises and interactions: = ( ), = ( ); Step 5: Concatenate the hidden states obtained in Step 4 to obtain the representation after concatenating the original exercise and the interaction sequence. And the representation after data augmentation of exercises and interaction sequences. and ; In step five, the and stated , The steps are as follows: The representational information of exercises and interactions is pieced together to form the state in reinforcement learning: = ⊕ , = ⊕ , = ⊕ ; Step 6: After obtaining the data-augmented hidden representations of the exercises and interactions from Step 4, the data is processed through an average pooling layer to obtain the entire exercise and interaction history representation. In step six, the formula for obtaining the entire history representation of exercises and interactions is as follows: =pool( ), =pool( ), in, and It is the enhanced hidden representation of the exercises and interactions obtained in step four. pool(·) is an average pooling layer, and the final output is... ; Step 7: Develop the latent representation obtained in Step 5. , and As input, the Q-values for each input are obtained through the Q-value function and concatenated to obtain the final Q-value; then, a self-supervised reinforcement learning method is used to maximize the cumulative reward. Step 8: Combine self-supervised, reinforcement, and contrastive learning losses, and use gradient descent to train the model. The obtained student representations are then passed through a self-supervised head to obtain the final exercise recommendation results, recommending more suitable exercises to the students.
2. The exercise recommendation method according to claim 1, characterized in that, In step one, the preprocessing of the dataset includes deleting answer records without knowledge points, deleting records with student answer sequences ≤ 3, and deleting duplicate records caused by a question having two or more knowledge points.
3. The exercise recommendation method according to claim 1, characterized in that, In step seven, considering the representation of students' learning of history, the task of recommending exercises is transformed into a multi-classification problem. Reinforcement learning methods are used to maximize cumulative rewards, as follows: (1) Obtain the hidden states of the exercises and interactions based on step five; (2) The recommendation model accepts the embedding vector as input and generates n-class logs. =[ , ,..., ]∈ , where n is the number of candidate exercises, and the classification log is... = This was achieved using a fully connected layer. (3) By using logarithms Optimize cross-entropy loss to train the recommendation model: = - , = , in, It is an indicator function that states if a student interacts with the i-th exercise at the next timestamp, then... =1, otherwise =0; Use directly As the current state of the RL head, an additional final layer is stacked to perform the corresponding Q-value calculation: ( , ) = δ( + ) = δ(G( ) + ), Where z∈{1,2,3}, ∈ , , }, ∈{ , }; δ is the activation function, These are learnable parameters; (4) The obtained , , Superimposed into a vector Q-value function: Q( , ) = [ ( , ), ( , ), ( , )], (5) At each time step t, Q-Network obtains an empirical sample with a batch size of 256 from the empirical pool D generated from the implicit feedback data. , To calculate the TD-error, we use it as the loss for the RL part, and its loss function is: = , in, It is the immediate reward in the current state. It is the successor state. This is the current state. and These represent the estimated values of the state-value functions for the current state and the successor state, respectively, and γ is the discount factor, representing the discount ratio for future rewards. Using empirical replay improves sampling efficiency and prevents approximation errors from propagating rapidly between two states when training a fixed target network. (6) RL acts as a regularizer, based on the defined reward mechanism. The model G is fine-tuned by evaluating the recommendation quality of the first k exercises; (7) To improve the stability of learning, Double Q-learning is used to alternately train the learnable parameters.
4. The exercise recommendation method according to claim 3, characterized in that, In step seven, a custom reward function was defined. Based on the results obtained from the DKT model, a dynamic reward mechanism is designed, and the reward mechanism is defined as follows: , , Where N is the number of exercises, and β is an adjustment factor, which is set to 10 based on experience. First, the sequence of students' answers before time t is input to obtain the students' mastery of each exercise before recommendation, and the correctness information of recommended exercises can be obtained from it. Then, the interaction sequence at time t is input into the DKT model to obtain the students' mastery after recommendation. Finally, the average difference between the mastery of all exercises at time t and time t-1, that is, the increment of the current student's knowledge state, is used as the final reward.
5. The exercise recommendation method according to claim 1, characterized in that, Step eight combines self-supervised learning, reinforcement, and contrastive learning loss. The steps of the contrastive learning part are as follows: (1) Obtain a positive pair from the data augmentation in step two. The object described can be instantiated as a problem history. , ) and interaction history ( , ); (2) Represent augmented samples from other historical data in a batch of 256 as negative samples. ∈ , ∈ ; (3) Define the comparative learning loss using two parts: exercises and interactions: = -log , = -log , Here, sim is a cosine similarity function; a contrastive loss function is used to learn effective representations, thereby bringing together similar learning histories and separating different learning histories. (4) The final overall objective function is a linear combination of self-supervised, reinforcement, and contrastive losses: , Here, α is a hyperparameter representing the proportion of influence of the contrastive learning component.
6. A personalized exercise recommendation system integrating reinforcement and contrastive learning, characterized in that, The system implements the exercise recommendation method as described in any one of claims 1-5, and the system includes: a student history sequence conversion module, a student knowledge state capture module, and an exercise recommendation module; The learning history sequence transformation model processes the student's historical sequence, passing the student's answer sequence through an embedding layer and an encoding layer to obtain feature representations of the exercises and interaction vectors, and also performs data augmentation processing. The student knowledge state capture module is used to learn the student's exercises and interaction representations; using a self-supervised reinforcement method, the exercise recommendation process is formalized as a Markov decision process, which interacts between the student and the recommendation agent; by obtaining the maximum long-term cumulative reward through the student's long-term needs in practice, the system can maximize the student's long-term satisfaction with the system. The student knowledge status capture module also introduces a custom reward mechanism to promptly obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the goal of improving students' learning abilities. The exercise recommendation module integrates the learning representations obtained from the two modules mentioned above and recommends exercises through a self-supervised head, thereby recommending exercises that can improve learners' learning abilities.
