Individualized exercise recommendation method and recommendation system integrating reinforcement and comparative learning

By integrating reinforcement learning and contrast learning methods, combined with Double Q-learning and deep knowledge tracking models, it solves the problem of difficult to capture students' long-term needs and data sparsity in the existing technology, and achieves more accurate exercise recommendations and learning gain improvements.

CN119089030APending Publication Date: 2024-12-06EAST CHINA NORMAL UNIV

Patent Information

Application Number
CN202310655470.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing exercise recommendation methods are difficult to capture students' long-term needs, and there are data sparsity and overfitting problems, which ignore the changes in the knowledge state of students during the learning process, resulting in poor recommendation results.

Method used

Using a method of fusion reinforcement learning and contrast learning, the exercise recommendation process is formalized into the Markov decision-making process through self-supervised reinforcement learning, combined with Double Q-learning and deep knowledge tracking models, and used data augmentation technology to generate additional historical records and capture Students' knowledge status changes and exercise recommendations are made through self-supervising heads.

Benefits of technology

It improves the accuracy and learning gain of exercise recommendations, alleviates the problem of data sparsity, enhances the robustness and generalization ability of the recommendation model, and improves students' learning experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089030A_ABST
    Figure CN119089030A_ABST
Patent Text Reader

Abstract

The invention discloses a personalized exercise recommendation method fusing reinforcement and comparative learning, and the method employs a self-supervised reinforcement learning method, enables an exercise recommendation process to be formalized into a Markov decision process, carries out the interaction between a student and an intelligent agent, and obtains a feature representation. Meanwhile, a comparative learning thought is integrated, similar learning histories are drawn together, and different learning histories are separated, so that effective characterization is learned; splicing the hidden states to obtain spliced representations # imgabs0 # and # imgabs1 #, and further obtaining a whole exercise and interaction history representation; # imgabs2 # and # imgabs3 # are used as input, and a final Q value is obtained through calculation and splicing of a Q value function; then, through a self-supervised reinforcement learning method, the accumulated reward is maximized; and in combination with self-supervision, reinforcement and learning loss comparison, model training is performed by using gradient descent, a final exercise recommendation result is obtained by using the model, and more suitable exercises are recommended to students. The invention also discloses a system for realizing the method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning and contrastive learning, and relates to a personalized exercise recommendation method and recommendation system integrating reinforcement and contrastive learning. Background Art

[0002] With the development of the Internet, smart education has become a new hot spot of the times. As an extension of traditional education, online smart education can conveniently provide students with advanced technology and help them consolidate their knowledge. In recent years, the scale of teaching on online platforms such as MOOC, Khan Academy and Online Judging (OJ) has continued to expand, resulting in a huge number of test resources, which makes it difficult for students to choose; teachers have complicated teaching tasks and it is difficult to take into account the problems of every student. Therefore, how to help students find suitable test questions in the vast sea of ​​test resources is an important task.

[0003] Personalized exercise recommendation refers to the method of more accurately locating students based on their learning history, using information such as whether they answered correctly, when they answered, the questions they answered, and the knowledge points involved, and then recommending exercises to help them learn more efficiently. In recent years, scholars have used traditional recommendation algorithms, cognitive diagnosis, knowledge tracking, reinforcement learning and other methods in exercise recommendation. Based on the traditional recommendation that students are regarded as users, test questions as commodities, and students' scores on exercises as users' ratings of commodities, traditional recommendation algorithms can be used to solve the exercise recommendation problem. This method is simple and easy to understand and has achieved good results. However, due to the essential difference between user-commodity and student-exercise, the use of traditional recommendation methods cannot consider other attributes of students, and there are problems in capturing the changes in knowledge status during the learning process. Cognitive diagnosis methods in educational data mining can also be used to recommend personalized exercises for students. For example, the IRT model [1] describes students as one-dimensional ability values ​​and models students in combination with parameters such as exercise difficulty and discrimination. The DINA model [2], as a multidimensional discrete cognitive model, introduces the test question-knowledge point Q matrix to diagnose the degree of mastery of students' knowledge points by combining more information specific to the educational field. However, cognitive diagnosis methods can only give students a discrete degree of mastery of knowledge points, ignoring the long-term needs of students in practice. On this basis, knowledge tracking technology was proposed, which aims to track the changes in students' knowledge levels through historical learning and recommend exercises. In addition, some scholars have proposed exercise recommendation methods based on reinforcement learning in the field of education, such as the hierarchical reinforcement learning method proposed by Zhang [3], which achieves exercise recommendation by modifying user information and adjusting the recommendation model based on the modified information.

[0004] Although some current studies have achieved good results in the exercise recommendation task, there are still some shortcomings. (1) Most traditional recommendation methods are based on students' preferences, which may lead to the recommendation of exercises that are too difficult or too simple. The information about exercises and knowledge concepts related to students is ignored, and the recommendation lacks rationality. (2) Most existing exercise recommendation methods are based on students' current exercise situation, but ignore the students' knowledge state in the entire process of answering questions, that is, long-term needs. (3) When students interact with intelligent education systems, they will easily rely on the courses provided by the system, resulting in a limited number of exercises processed, thus leading to the sparsity problem of educational data. For example, the data collected by iFlytek from their online education recommendation system ZhiXue (zhixue.com) only has 0.99% interaction. Representations learned from sparse data sets using reinforcement learning methods are often prone to overfitting and are difficult to accurately capture students' knowledge state. Summary of the invention

[0005] In order to solve the current problems of difficulty in capturing students' long-term needs; data sparsity between students and exercises; hidden expressions are prone to overfitting and cannot accurately capture students' knowledge status; more attention is paid to imitating students' behaviors while ignoring the improvement of students' learning ability, etc., the purpose of the present invention is to provide a method that integrates reinforcement and contrastive learning, models students, and recommends exercises suitable for different students. The present invention uses students' learning history as input, obtains information about exercises and interactions through embedding and encoding layers, introduces the idea of ​​contrastive learning, and uses three data enhancement methods to combine to obtain additional historical records, which are integrated into reinforcement learning as a state supplement to capture long-term needs in the learning process. Then the exercises and interaction information are spliced ​​to obtain effective representation and learning, and finally the exercise recommendation results are obtained through the self-supervision head.

[0006] Reinforcement learning is a machine learning technique that mainly allows an agent to take actions in an unfamiliar environment and interact with it to maximize the reward. The present invention uses a self-supervised reinforcement learning method to formalize the exercise recommendation process into a Markov decision process (MDP). The agent interacts with the student in a continuous decision-making process, obtains rewards and learns how to make the best decision by observing the environment (including a student pool and an exercise pool) and performing actions. The basic elements of MDP include: state space, action space, state transition probability, reward function and discount factor. Among them, the state space is the set of all states that the agent may be in, and the action space is the set of all actions that the agent can perform. The state transition probability is the probability distribution of the agent transferring to the next state after performing an action in the current state. The reward function is the reward value obtained by the agent performing an action in a certain state. The discount factor is the discount coefficient used to calculate future rewards, which reflects the degree of uncertainty of the agent about future rewards. In MDP, the agent needs to make decisions through strategies. A strategy is a mapping relationship that maps each state to the probability distribution of performing an action. The goal of the agent is to maximize the long-term expected cumulative reward by learning the optimal strategy. The basic method to solve the MDP problem is value iteration or policy iteration. In value iteration, the agent finds the optimal strategy by iteratively updating the value function of each state. In policy iteration, the agent finds the optimal strategy by iteratively updating the strategy. In the present invention, by paying attention to the long-term needs of students in practice, the maximum long-term cumulative reward is obtained to maximize the long-term satisfaction of students with the system; specifically, in the present invention, the environment includes a student pool and an exercise pool. At time step t, the agent receives the current state of the student, selects an exercise, and then the student answers the exercise, and the agent obtains a corresponding reward based on the student's performance (the present invention designs a dynamic reward mechanism with the core of improving students' learning ability). After that, it transfers to the next state, and then the agent makes a new recommendation, and so on. The reinforcement learning part adopts the Double Q-learning method, using two reinforcement learning heads (RL heads, Reinforcement Learning Head), which supervise and complement each other during the learning process. Generally speaking, in Double Q-learning, the RL head refers to the estimator of the Q value function. The Q-value function is a state-action value function used to represent the cumulative reward value. In Double Q-learning, two Q-value functions are used, namely Q1 and Q2. The purpose of these two Q-value functions is to solve the estimation bias problem in the Q-learning algorithm, thereby improving the learning effect of the algorithm.RL head plays an important role in Double Q-learning. Its main function is to estimate the Q value of taking an action in a given state. In the design of RL head, a deep neural network is usually used to fit the Q value function. Its input is the current state and action, and its output is the estimated Q value of taking the action in the current state. Deep neural network usually contains multiple fully connected layers or convolutional layers, and its specific structure can be selected according to the nature of the problem and the characteristics of the data. In Double Q-learning, each time the RL head is updated, one of the Q value functions (Q1 or Q2) needs to be used as the objective function and the other Q value function as the estimation function. Specifically, each time the RL head is updated, one of the Q value functions is randomly selected as the objective function, and the estimated value of the Q value function is used as the reward value of taking an action in the current state. Then the estimated value of the other Q value function is used to estimate the optimal action in the next state, and the estimated value of the Q value in the current state is calculated. In this way, the estimation bias problem in the Q-learning algorithm can be avoided and the learning effect of the algorithm can be improved. As a regularizer, the RL head helps to learn additional information and improve the recommendation performance. In addition, the present invention also sets up a dynamic reward mechanism, which mainly uses the trained deep knowledge tracking (DKT) model in the learning process to timely capture the students' mastery of exercises before and after the recommendation, so as to achieve the purpose of recommending exercises that improve students' learning ability. And the reward function set according to the present invention can be used to fine-tune the model to achieve better results.

[0007] The Double Q-learning is a reinforcement learning method using two estimators. Double Q-learning will use two Q functions (corresponding to two estimators), and each Q function will use the value of the other Q function to update the next state. Both Q functions are learned from different experience sets. In the present invention, it is used as the basic structure of the reinforcement learning part.

[0008] The DKT (Deep Knowledge Tracing) model is a model that models students' knowledge based on their historical question-solving sequences, and the model can be used to accurately predict students' performance in future interactions. Specifically, the DKT (Deep Knowledge Tracing) model can be used to predict students' knowledge mastery in the learning process. The model uses a neural network to model students' knowledge status and continuously updates the model based on students' answers. In the DKT model, each student is represented as a knowledge vector, in which each element represents the student's mastery of a certain knowledge point. The knowledge vector is continuously updated as the student's learning process, thereby reflecting the changes in the student's knowledge status. When predicting the student's next question answering situation, the DKT model adopts a sequence modeling method. Specifically, the model regards the student's answer sequence as a time series and uses a recurrent neural network (RNN) to model the sequence. During the training process, the DKT model updates the model parameters by minimizing the error between the predicted value and the actual value. At the same time, the model also uses dropout and other technologies to avoid overfitting problems. In the present invention, the DKT model is mainly used in the design of a dynamic reward mechanism and the definition of a learning gain indicator.

[0009] Contrastive learning is a type of learning method under the self-supervised learning paradigm, which focuses on learning the common features between similar instances and distinguishing the differences between non-similar instances. In order to make up for the sparsity between students and exercises, the hidden representation is easy to overfit and often cannot accurately capture the problem of students' knowledge status. The present invention introduces the idea of ​​contrastive learning and integrates the relevant information obtained. The addition of contrastive learning can bring similar learning histories together and separate different learning histories to learn effective representations. At the same time, the additional historical records obtained after the data enhancement method in a specific field of contrastive learning are used as a state supplement for the reinforcement learning part to obtain more effective representations of the students' knowledge status.

[0010] The present invention proposes a personalized exercise recommendation method integrating reinforcement and contrastive learning, the method comprising the following steps:

[0011] Step 1: Select a data set and preprocess the data set;

[0012] Step 2: Data augmentation is performed on the original student history sequence. The data augmentation used combines three data augmentation strategies: mask, permute, and replace, which reflect different aspects of the learning history and obtain two related view representations of the original learning history, namely u +1 ,u +2; mask is a question masking method. This enhancement promotes the estimation of missing context and can mitigate the bias caused by sparse educational data; permute is a reordering interaction method. This enhancement generates more information about the representation of the student's learning process; replace is a question replacement method. The learning history obtained by this method can be regarded as the noise of the original history, which can enhance the model's error correction ability and improve the model's accuracy and ability to utilize information. The proficiency shown by the learning history after data enhancement is similar to that before.

[0013] View representation transforms or processes each sample in the original dataset in different ways to generate multiple new samples. These new samples can be seen as different perspectives or expressions of the original dataset, which can more comprehensively describe the characteristics of the dataset. Using view representation technology for data augmentation can improve the generalization and robustness of the model, thereby better adapting to different application scenarios.

[0014] Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representation of the exercises and the student-exercise interaction vector respectively;

[0015] Step 4: According to the feature representation of the exercises and interaction vectors obtained in step 3, an existing sequence recommendation model that can be integrated, such as GRU and / or SASRec and / or NextItNet and / or Bert4Rec, is used to encode the exercises and interaction vectors to obtain corresponding hidden representations; the present invention can integrate the above existing sequence recommendation models, which play an encoding role in the present invention to obtain hidden features;

[0016] Step 5: Concatenate the hidden states obtained in step 4 to obtain the concatenated representation of the original exercise and the interaction sequence. And the representation after data augmentation exercises and interaction sequences splicing and After data augmentation, two related view representations are obtained as a positive pair;

[0017] Step 6: After obtaining the hidden representation of the exercise and interaction after data enhancement according to step 4, the historical representation of the entire exercise and interaction is obtained through an average pooling layer;

[0018] Step 7: The potential representation obtained in step 5 and As input, the Q value corresponding to each input is obtained through the Q value function, and then concatenated to obtain the final Q value; the Q value is the function value in reinforcement learning, which represents the expectation of the total reward of the agent until the final state after selecting this action. Q value = reward obtained by the current action + discount rate * expectation (cumulative) {state transition probability * next state V value}. The V value of the next state refers to the expected reward value that can be obtained after performing an action in the current state and transferring to the next state. The V value is an estimate of the value of the state, which is used to evaluate whether the action taken in the current state is conducive to obtaining a high reward in the end. The V value can be calculated by the Bellman equation, that is, the V value of the next state is multiplied by the discount factor and added to the immediate reward of the current state to obtain the V value of the current state. In the process of interaction between the agent and the environment of reinforcement learning, the agent will decide what action to take next based on the current state and optional actions, and then observe the feedback from the environment to obtain the reward signal and the next state. In this process, the agent will update the V value based on the V value of the current state and the reward obtained by the action taken, so as to continuously optimize the strategy and finally achieve the optimal decision. Then, the cumulative reward is maximized through self-supervised reinforcement learning methods;

[0019] Step 8: Combine self-supervision, reinforcement and contrastive learning loss to train the model using gradient descent. The obtained student representation is passed through a self-supervision head to obtain the final exercise recommendation result and recommend more suitable exercises to students.

[0020] In step 1, the selected datasets are ASSIST09[5] and EdNet[6]. ASSIST09 is information collected from the ASSISTments online tutoring system in the 2009-2010 academic year; EdNet is a large-scale hierarchical dataset collected by the intelligent tutoring platform Santa since 2017. The datasets used all contain the required information: student ID, exercise ID, knowledge points included in the exercise, whether the student's answer is right or wrong, the student's study time, and the student's exercise time. The preprocessing of the dataset includes deleting answer records without knowledge points, deleting records with student answer sequences ≤3, and deleting duplicate records caused by an exercise having two or more knowledge points.

[0021] In step 2, the student set and exercise set are denoted as U and Q respectively, and the student history sequence is defined as u 1:t ={u1,u2,...,u t}. Each interaction is represented by a tuple u t = {q t ,r t}, where q t∈Q is the tth exercise in the exercise set, and also contains the relevant knowledge points k∈K involved in the exercise, where K is the knowledge point set, representing all the knowledge points contained in the exercise set Q. t ∈{0,1} is the answer result: that is, the correct answer to this question r t is equal to 1, otherwise r t is 0.

[0022] The data enhancement method described in step 2 is a combination of three enhancement strategies: mask, permute, and replace. Inspired by the literature Contrastive Learning for Knowledge Tracing [4], the present invention also uses a combination of multiple data enhancement methods. The difference from the literature is that since the length of the student history sequence set in the present invention is small (set to 10), the operation of randomly deleting certain exercises is abandoned. The specific description is as follows:

[0023] Random data augmentation is applied to each student’s learning history, that is, mask, permute and replace operations are randomly performed on each student’s learning history in a certain proportion, so that the proficiency of each student’s learning history after data augmentation is similar to that before data augmentation, and can be used as additional historical information to supplement the RL part;

[0024] Three data enhancement methods are adopted in a certain proportion, using the symbol γ mask , γ per , γ rep To express, γ mask , γ per , γ rep Adjusted within the range of 0.2 to 0.5. The combination of the three data enhancement methods is mainly reflected in the fact that each method sets the corresponding ratio, that is, the original student learning sequence will be masked, replaced and reordered at the same time; the three enhancement methods are described as follows:

[0025] Mask method for data augmentation: Use the exercise mask method, that is, use special markers to replace some exercises in the original learning history, but do not change the student's answer. Specifically, for each student sequence, a certain proportion (γ mask The learning history of the properly masked part can be regarded as an enhanced version of the original learning history, which is conducive to the model to better learn useful information. In addition, this data augmentation method also promotes the estimation of missing context, which can avoid the bias caused by sparse educational data to a certain extent and learn better representations;

[0026] The permute method is used for data augmentation: reordering is performed in the subsequences of the original student history sequence, which means that the order within the sequence will change to a certain extent, but the student knowledge state represented by the interaction sequence will remain similar. The reordering is based on γ per The probability of randomly selecting a subsequence in the original sequence is p =[γ per *t] starts to randomly disrupt the continuous subsequence, and the subsequence is represented by (u r ,u r+1 ,...,u r +L p -1), where γ per is the ratio of reordering, and t is the time t of the answer sequence. After shuffling, the student history sequence enhanced by the "permute" method is obtained, which can be used as additional information for the reinforcement learning part. Because students do not obtain learning materials from other places during the entire process of exercise recommendation, it is assumed that the proficiency of each student in a sequence remains consistent. In other words, during the learning process, students have mastered certain knowledge concepts and are able to solve exercises related to the knowledge concepts, regardless of the order in which the exercises appear. The sequence after using the "permute" method produces more descriptive information about the student's learning process;

[0027] Data augmentation using the replace method: This method replaces the original exercises with simpler or more complex exercises based on the students' original answers. The replacement method can add noise, enhance the model's error correction ability, and improve the model's accuracy and ability to utilize information. The learning history obtained after replacement still presents a knowledge state similar to the original learning history.

[0028] In order to implement the replacement method, only simple cognitive relationships are used to build the knowledge structure. All exercises in the training data set and the proportion of correct and incorrect answers are obtained, that is, the difficulty of each question is obtained. The difficulty of the exercise is expressed as Indicates, where N(correct) represents the number of correct answers to this exercise, N a Indicates the total number of answers to the exercise. Then sort all the exercises in ascending order {q(1), q(2), ... q(N)}, where N is the number of exercises. q(1) is the easiest exercise, and q(N) is the most difficult exercise. Assume that a student who has mastered a more complex knowledge point is more likely to correctly answer a low-level concept, and vice versa. Therefore, when replacing a knowledge point, there is no need to change the original answer. For the student's learning sequence, γ repThe probability of random selection is . Given the cth exercise, if the answer is correct, the original exercise q(c) is replaced by a simpler exercise q(cj), otherwise it is replaced by a more difficult exercise q(c+j)(j∈N + ) without changing the student's answer to the question.

[0029] The data enhancement method is used to enhance the student's learning history sequence u 1:t After processing and enhancement, two related views of the same history are obtained: +1 ,u +2 .

[0030] In step 3, the embedding layer is divided into two parts: an exercise embedding layer and a student-exercise interaction embedding layer; exercises and interactions are respectively passed through two embedding layers to obtain corresponding embedding representations;

[0031] In step 3, the steps of converting the original user history sequence and the enhanced sequence into vector feature representation are as follows:

[0032] (1) Construct the exercise matrix E Q ∈P N×d , the interaction matrix E between students and exercises I ∈P 2N×d (involving the correctness of students’ answers), where N represents the number of exercises and d represents the dimension of embedding;

[0033] (2) Given an interaction u t =(q t ,r t ), and obtain the exercise embedding vector at time t through the embedding layer and the interaction embedding vector

[0034] (3) Obtain the exercise embedding vector of the enhanced data and the interaction embedding vector

[0035] In step 4, the steps of encoding the exercises and interaction vectors using the existing integrable sequence recommendation model are as follows:

[0036] (1) Convert the next exercise recommendation task into a multi-classification problem (i.e., treat an exercise as a category, and the result of the multi-classification problem is which exercise to select), and use the existing integrated sequence recommendation model to encode a series of exercises and interactions;

[0037] (2) Process the original student history sequence and embed the given exercise sequence Exercise Encoder G Q The learning exercises show: Interactive Encoder GI Interactive embedding sequence through input Get the corresponding student-exercise interaction representation

[0038] (3) Process the student sequence after data augmentation and obtain the representation of learning history through the hidden representation of exercises and interactions:

[0039] In step 5, the student history sequence is obtained and the enhanced sequence It is mainly obtained by splicing the obtained exercises and interactive information:

[0040] In step 6, the historical representation formula for the entire exercise and interaction is as follows:

[0041]

[0042] in, and is the hidden representation of the augmented questions and interactions obtained in step 4. pool(·) is an average pooling layer. The final output z Q ∈P d , z I ∈P d ;

[0043] In step 7, the exercise recommendation task is regarded as a Markov decision process, which can be defined by a tuple consisting of (S, A, P, R, ρ0, γ):

[0044] In the Markov decision process, the state transition function T(s,a,s') represents the probability of transitioning to the next state s' after taking action a in the current state s. The state transition function can be expressed as T(s,a,s'): S×A×S→[0,1], that is, the current state s and the action a taken are input, and the probability value of transitioning to the next state s' is output. In some cases, the state transition function may depend on time t or the previous state-action sequence. The reward function R(s,a,s') represents the immediate reward obtained by transitioning to state s' after taking action a in state s. The reward function can be expressed as R(s,a,s'): S×A×S→R, that is, the current state s, the action a taken, and the next state s' transferred to are input, and the immediate reward value is output. The state transition function and the reward function are usually assumed to be properties of the environment and are determined in the problem modeling stage. Ultimately, the goal of the Markov decision process is to find a strategy π: S→A that maximizes the expected return. The expected return is the cumulative value of the immediate rewards obtained from the state-action sequence, attenuated by the discount factor γ, which can be expressed as Where R t Represents the immediate reward obtained at time t.

[0045] (1) State S is used to simulate the student’s learning state space. At time t, state s t ∈S, defined as the student’s previous learning history. The student’s state at time t can be expressed as s t =G(u 1:t )∈S(t>0)(student status representation is obtained through the recommendation model G);

[0046] (2) Action space A, which is a discrete action space containing all exercises. Based on the state s t , a t ∈A is defined as recommending exercises q to students t+1 In offline data, the actions at time t can be obtained from the interaction between students and exercises;

[0047] (3) State transition probability P, S×A×S→P, which means that after the agent takes an action, a state s t Mapped to a new state s t+1 probability;

[0048] (4) Reward function R. The agent recommends based on the state s t Exercises (actions) to interact with students in the environment. Rewards R (s t ,a t );

[0049] (5) initial state distribution ρ0, ρ0 is an initial state distribution with s0~ρ0 (normal distribution);

[0050] (6) Discount factor γ for future rewards. γ∈[0,1], γ=0 means only the immediate reward is considered, and γ=1 means all future rewards are fully considered.

[0051] In step 7, given the representation of the student’s learning history, the exercise recommendation task is transformed into a multi-classification problem, and the reinforcement learning method is used to maximize the cumulative reward, mainly to seek the optimal strategy π θ =(a|s) Recommend exercises to students to improve their learning ability. This is achieved by maximizing the expected cumulative reward, as follows:

[0052]

[0053]

[0054] θ∈P dRepresents the parameters of the policy function. τ=(s0,a0,s1,a1,...) is the trajectory generated by the target strategy. According to the target strategy that maximizes the cumulative reward, operations are performed to obtain s0~ρ0, a t ~π θ (·|s t ), s t+1 ~P(·|s t ,a t ), corresponding to the initial state distribution; the choice of action; and the state transition probability;

[0055] The specific steps involved in the self-supervised learning reinforcement learning content are as follows:

[0056] (1) Obtain the hidden state of the exercises and interactions according to step 5;

[0057] (2) The recommendation model accepts the embedding vector as input and generates n classification log y t+1 =[y1,y2,...,y n ]∈R n , n is the number of candidate exercises; the classification log is composed of y t+1 =f(s t ) is obtained, in which a fully connected layer is used;

[0058] (3) Based on the logarithm y t+1 Optimize cross entropy loss to train the recommendation model:

[0059]

[0060] Among them, Y i is an indicator function. If the student interacts with the i-th exercise at the next time stamp, then Y i =1, otherwise Y i = 0. Since the recommendation model G has encoded the input vector into a potential representation s t , so you can directly use s t As the current state of the RL head, there is no need to introduce additional models. What is needed is to stack an additional final layer to perform the corresponding Q-value calculation:

[0061] Q z (s t ,a t )=δ(s t W z +b z )=δ(G(e 1:t )W z +b z ),

[0062] where z∈{1,2.3}, δ is the activation function, W z , b z are learnable parameters;

[0063] (4) The obtained Q1, Q2, and Q3 are superimposed into a vector Q value function, where Q1, Q2, and Q3 correspond to The corresponding Q value is calculated. The Q value is the function value in reinforcement learning. It represents the expectation of the total reward of the final state after the agent selects this action. Q value = reward obtained by the current action + discount rate * expectation (accumulation) {state transition probability * next state V value}:

[0064]

[0065] (5) At each time step t, Q-Network obtains a small batch of experience samples (s) from the experience pool D generated from the implicit feedback data (generally speaking, the batch size can be set to 64 or 128 or 256 or 512 or even larger according to actual needs. In the present invention, the batch size of the small batch is set to 256). t a t ,r t ,s t+1 ) is used to calculate the temporal difference TD-error as the loss of the RL part, and its loss function is:

[0066]

[0067] Among them, R t is the immediate reward in the current state, s t+1 is the successor state, s t is the current state, Q(s t ,a t ) and Q(s t+1 , a') represent the estimated values ​​of the state value function of the current state and the successor state respectively, γ is the discount factor, which represents the discount ratio of future rewards.

[0068] Experience replay improves sampling efficiency and prevents approximation errors from increasing in s for training of a fixed target network. t and t+1 Rapid propagation between two states;

[0069] Where R t The dynamic reward mechanism set for the present invention is used as an instant reward in the current state, and the specific steps are as follows:

[0070] I. In order to make the basic recommendation model G better recommend exercises to improve students' learning ability, the trained DKT model is used. This model uses LSTM as a neural network layer to simulate the interaction between students and exercises.

[0071] II. Design a dynamic reward mechanism based on the results obtained from the DKT model. The reward mechanism is defined as follows:

[0072]

[0073] p t =DKT(u1,u2,...,u t ),

[0074] Where N is the number of exercises, β is an adjustment factor, which is set to 10 based on experience; this adjustment factor is used to expand the reward value obtained, because the reward value in the formula is obtained by the difference in the degree of knowledge mastered before and after the recommendation, and this value will be relatively small. In order to increase the impact of the reward value (for example, 0.001, -0.001 have very small positive and negative effects. If multiplied by 10. or 100, the positive and negative values ​​will become larger, and the final cumulative reward may change from 0.18, -0.18 to 1.8, -1.8), it is necessary to multiply by the corresponding coefficient. Generally speaking, this coefficient is 10, 100... Specifically, first input the student's question sequence before time t to obtain the student's mastery of each exercise before the recommendation, and obtain the right and wrong information of the recommended exercises from it. Then the interaction sequence at time t (u1, u2, ..., u t ) is input into the DKT model to obtain the students’ mastery level after the recommended exercises. The mastery level is expressed by a probability value p t Finally, the average of the difference between the mastery of all exercises at time t and time t-1, that is, the increment of the current student's knowledge state, is used as the final reward. The designed dynamic reward mechanism can well capture the changes in the student's knowledge state during the learning process, so as to obtain corresponding rewards according to the changes in the knowledge state to strengthen the training of the model.

[0075] (6) RL acts as a regularizer, according to the reward R defined in the reward mechanism. t Set up fine-tuning of the model G by evaluating the recommendation quality of the first k exercises;

[0076] (7) In order to improve the stability of learning, Double Q-learning is used to alternately train the learnable parameters.

[0077] In step eight, the steps of contrastive learning loss are as follows:

[0078] (1) Obtain a positive pair (u) from the data enhancement in step 2+1 ,u +2 ), for which it is instantiated as the problem history (z Q,+1 ,z Q,+2 ) and interaction history (z I , +1 , z I,+2 );

[0079] (2) Represent the enhanced samples from other histories in the same mini-batch (the same as that involved in the self-supervised reinforcement learning above, where the batch size of the mini-batch is also 256) as negative samples, z Q,- ∈Z Q,- , z I,- ∈Z I,- ;

[0080] Z Q,- is the representation of the negative sample of the exercise, Z I,- is the representation of interactive negative samples;

[0081] (3) Define contrastive learning loss using two parts: exercises and interactions:

[0082]

[0083]

[0084] Among them, sim is a cosine similarity function.

[0085] Use contrastive loss function to learn effective representations, pull similar learning histories together, and separate different learning histories.

[0086] In step eight, the model is trained using gradient descent, and its final overall objective function is a linear combination of self-supervision, reinforcement and contrast losses:

[0087] L=L S +L q +αL cl

[0088] Among them, α is a hyperparameter, which represents the influence ratio of the contrastive learning (CL) part.

[0089] The present invention also provides a system for implementing the above-mentioned exercise recommendation method, the system comprising: a student history sequence conversion module, a student knowledge state capture module, and an exercise recommendation module;

[0090] The learning history sequence conversion module processes the student history sequence, first obtaining the original sequence of the student sequence, the sequence 1 after data enhancement, and the sequence 2 after data enhancement; then the obtained student sequence (including the sequence after data enhancement) is divided into two parts: exercises (only including exercise sequences) and interactions (including exercises and right and wrong information), which are respectively passed through the exercise embedding layer and the student-exercise interaction embedding layer, and then passed through the recommendation model layer (exercise encoder and interaction encoder), and finally the feature representation of the exercises and interaction vectors is obtained;

[0091] The student knowledge state capture module is used to learn the student's exercises and interaction representations. Using the self-supervised reinforcement method, the exercise recommendation process is formalized as a Markov decision process, and the interaction is carried out between the student and the recommendation agent. Through the long-term needs of students in practice, the maximum long-term cumulative reward is obtained to maximize the long-term satisfaction of students with the system. The RL head is used as a regularizer, and the Double Q-learning method is adopted. The two RL heads supervise and complement each other during the learning process. The use of reinforcement learning helps to learn additional information and improve the recommendation performance; the contrastive learning idea is introduced to integrate the information obtained, introduce relevant knowledge point information and the interaction information obtained after the encoder, so as to learn effective representations; at the same time, the enhanced exercises and interaction information are obtained by using the data enhancement method in a specific field of contrastive learning, which can be used as the state supplement of the reinforcement learning part. The use of this method alleviates the data sparsity problem to a certain extent; when students interact with the intelligent system, they will easily rely on the courses provided by the system, resulting in a limited number of exercises processed, thus leading to the sparsity problem of educational data. For example, the data collected by iFLYTEK from their online education recommendation system ZhiXue (zhixue.com) has only 0.99% interactions. Representations learned from sparse data sets using reinforcement learning methods are often prone to overfitting and are difficult to accurately capture the student's knowledge state. The enhanced learning representation obtained by the present invention through three enhancement methods is similar to the proficiency of the original student answer sequence, which can be added to RL as additional supplementary information, that is, there is more data information for RL to learn.

[0092] The student knowledge status capture module also introduces a customized reward mechanism to timely obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the purpose of improving students' learning ability.

[0093] The exercise recommendation module integrates the learning representations obtained by the above two modules and recommends exercises through a self-supervised head. The proposed exercise recommendation method can effectively improve learning gain and recommendation accuracy, and recommend exercises that can improve learners' learning ability.

[0094] The beneficial effects of the present invention include:

[0095] 1. Use self-supervised reinforcement learning methods to formalize the exercise recommendation process as a Markov decision process, with interactions between students and recommendation agents (agents). By focusing on the long-term needs of learning in practice, long-term cumulative rewards are obtained to maximize the long-term satisfaction of students with the system. The RL head acts as a regularizer, which helps to learn additional information and improve recommendation performance.

[0096] 2. A new dynamic reward mechanism is proposed. By using the trained DKT model, the students’ mastery of exercises before and after the recommendation is captured, and positive and negative rewards are given according to the mastery to fine-tune the recommendation model. And more attention is paid to recommending exercises that can improve students’ learning ability.

[0097] The idea of ​​contrastive learning is introduced to integrate the information obtained. The addition of contrastive learning can bring similar learning histories together and separate different learning histories to learn effective representations. And relevant knowledge point information and interactive information obtained after passing the encoder can be introduced. The present invention also uses additional historical records obtained by data enhancement methods in specific fields in contrastive learning as a state supplement for the reinforcement learning part. Using this method alleviates the data sparsity problem to a certain extent.

[0098] 3. The proposed method is integrated with four advanced recommendation models (GRU, SASRec, NextItNet, Bert4Rec), and corresponding experiments are conducted on ASSIST09 and EdNet datasets. On the ASSIST09 dataset, the performance of HR and NDCG recommendation indicators is improved by 4%-13%. In EdNet, the proposed method brings an average performance improvement of 6%.

[0099] 4. The method of the present invention also significantly improves the students' learning gain index (L_G). This shows that the present invention can obtain better learning gain and recommend exercises that can improve students' learning ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0100] Figure 1 It is a flow chart of the present invention.

[0101] Figure 2 This is a model architecture diagram that integrates reinforcement and contrastive learning in the implementation example.

[0102] Figure 3 It is a system module diagram of the present invention.

[0103] Figure 4It is a structural schematic diagram of the student history sequence conversion module of the present invention. DETAILED DESCRIPTION

[0104] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0105] The present invention provides a personalized exercise recommendation method that integrates reinforcement and contrastive learning. The standard recommendation model is enhanced by two output layers of self-supervised learning and reinforcement learning, and the information obtained by contrastive learning is integrated to achieve exercise recommendation for learners. First, a self-supervised reinforcement learning method is used to formalize the exercise recommendation process as a Markov decision process (MDP) to interact between students and recommendation agents. The RL head, as a regularizer, helps to learn additional information and improve recommendation performance. Then, the contrast idea is integrated to bring similar learning histories together and separate different learning histories to learn effective representations. Three data enhancement methods are used to generate additional data, which are used for contrastive learning and incorporated into RL as supplementary information. The present invention also uses a trained DKT model to capture the mastery of students' exercises before and after recommendation, so as to set up a novel reward mechanism. Finally, the relevant information of students and exercises is used as training data to train the proposed framework. The representation of questions and interactions is obtained through learning, and then they are spliced ​​accordingly, and finally exercises are recommended through the self-supervised head. The present invention also provides a system for implementing the above method.

[0106] The personalized exercise recommendation method provided by the present invention is as follows: Figure 1 As shown, the method comprises the following steps:

[0107] Step 1: Select a data set and preprocess it.

[0108] Step 2: Data augmentation is performed on the original student sequence. The data augmentation method used combines three data augmentation strategies to reflect different aspects of the learning history and obtains two related views of the original learning history, representing u +1 ,u +2 .

[0109] Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representations of the exercises and interaction vectors respectively.

[0110] Step 4: Based on the feature representation of exercises and interaction vectors obtained in step 3, the existing sequence recommendation model is used to encode the exercises and interaction vectors to obtain the corresponding hidden representations.

[0111] Step 5: Concatenate the hidden states obtained in step 4 to obtain the concatenated representation of the original exercise and the interaction sequence. And the representation after data augmentation exercises and interaction sequences splicing and

[0112] Step 6: Based on the hidden representation of the exercises and interactions of the enhanced data obtained in step 4, the historical representation of the entire exercises and interactions is obtained through an average pooling layer.

[0113] Step 7: Convert the and As input, the exercise recommendation is regarded as a Markov decision process, the corresponding Q value is obtained and concatenated, and then the cumulative reward is maximized through a self-supervised reinforcement learning method.

[0114] Step 8: Combine self-supervision, reinforcement and contrastive learning loss to train the model using gradient descent. The obtained student representation is passed through a self-supervision head to obtain the final exercise recommendation result and recommend more suitable exercises to students.

[0115] The present invention also provides a system for implementing the above-mentioned exercise recommendation method, the system comprising: a student history sequence conversion module, a student knowledge state capture module, and an exercise recommendation module;

[0116] The learning history sequence conversion module processes the student history sequence. First, the original student sequence is subjected to data enhancement to obtain enhanced sequence 1; enhanced sequence 2; then the obtained student sequence (including the sequence after data enhancement) is divided into two parts: exercises (containing only exercise sequences) and interactions (containing exercises and right and wrong information), which are respectively passed through the exercise embedding layer and the student-exercise interaction embedding layer, and then passed through the exercise encoder and the interaction encoder, and finally the feature representation of the exercises and interaction vectors is obtained;

[0117] The student knowledge state capture module is used to learn the student's exercises and interaction representations. Using the self-supervised reinforcement method, the exercise recommendation process is formalized as a Markov decision process, and interactions are performed between students and recommendation agents. Through the long-term needs of students in practice, the maximum long-term cumulative reward is obtained to maximize the student's long-term satisfaction with the system. Using the RL head as a regularizer, and adopting the Double Q-learning method, the two RL heads supervise and complement each other during the learning process. The use of reinforcement learning helps to learn additional information and improve recommendation performance; introduce the idea of ​​contrastive learning, fuse the information obtained, introduce knowledge point information and the interaction information obtained after the encoder, so as to learn effective representations; at the same time, after using the data enhancement method in a specific field of contrastive learning, the enhanced exercises and interaction information are obtained, which can be used as a state supplement for the reinforcement learning part. The use of this method alleviates the data sparsity problem to a certain extent;

[0118] The student knowledge status capture module also introduces a customized reward mechanism to timely obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the purpose of improving students' learning ability.

[0119] The exercise recommendation module integrates the learning representations obtained by the above two modules and recommends exercises through a self-supervised head. The proposed exercise recommendation method can effectively improve learning gain and recommendation accuracy, and achieve the purpose of recommending exercises that can improve learners' learning ability.

[0120] Example 1

[0121] The specific process of this embodiment is as follows:

[0122] (1) First, we select the datasets ASSIST09 and EdNet. We perform data preprocessing, delete the answer records without knowledge points, delete the records with student answer sequences ≤ 3, and delete the duplicate records caused by one exercise having two or more knowledge points.

[0123] (2) For all datasets, the input sequence consists of the last ten exercises before the target timestamp. If the sequence length is less than 10, 0 is generally used as a filler item to indicate that no exercises were answered in this part. Each student's historical sequence is stored according to the answering time, and the attributes include exercises, knowledge points, and correct or incorrect answers.

[0124] (3) The preprocessed data is divided into training set, validation set and test set in a ratio of 8:1:1.

[0125] (4) Using γ mask , γ rep , γper The probability of using three data enhancement methods to enhance the original input student sequence data to obtain the corresponding positive student sequence u +1 ,u +2 The data obtained using data augmentation can be viewed as noisy points in the original learning history to better learn useful information, and the student knowledge state represented in the sequence remains consistent.

[0126] (5) The obtained student sequence is then input into the exercise embedding layer and the interaction embedding layer to obtain the embedding representations of the exercises and interactions, respectively.

[0127] (6) Then input the obtained embedding representation into the exercise encoder G Q and the interactive encoder G I In , the exercise embedding and the student-exercise interaction embedding are encoded to obtain the corresponding hidden representations. The hidden representations are concatenated as the state representation of the student history.

[0128] (7) Then calculate a*=argmax which maximizes the Q value a Q(s t+1 ,a), that is, select the recommended exercises.

[0129] (8) Then, a* is added to the original sequence, that is, the question sequence after the question is recommended is obtained. Then, it is input into the trained DKT model to obtain the student's mastery of the question after the question is recommended. The corresponding reward R is obtained according to the reward mechanism set by the present invention. t , and transition to the next state s t+1 Repeat this cycle.

[0130] (9) Then use the training data to calculate TD-error as well as self-supervision and contrast losses, update the parameters, and use the validation set to tune the parameters.

[0131] (10) Using the pre-trained parameters, the learned student representation is passed through a self-supervised head to obtain the final exercise recommendation result, thereby recommending more accurate exercises to students.

[0132] The method of the present invention can also be used for other educational data sets, and the specific process will not be described in detail.

[0133] The parameters in the above embodiments of the present invention are determined based on the experimental results of the validation set, that is, different parameter combinations are tested on the validation set, and a set of parameters with better accuracy is selected. Specifically, for trainable parameters, such as weight W z , bias b z, and trained using a gradient descent algorithm. Because the method proposed in the present invention can expand the existing recommendation model and be integrated with the recommendation model, different recommendation models are used and different model parameters need to be set. For example, in the GRU model, the hidden state size is 64; the number of self-attention heads in the SASRec model is set to 1, etc. The model used in the present invention also has common parameters, such as a mini-batch of 256, a learning rate of 0.01, a dropout setting of 0.1, a discount factor γ set to 0.5, and γmask, γrep, and γper are adjusted within the range of 0.2 to 0.5. In this embodiment, they are set to {0.2, 0.3, 0.5}. In the actual tests above, the purpose of the present invention can also be achieved by appropriately adjusting the above parameters according to needs.

[0134] Example 2

[0135] The present invention considers four different sequential recommendation models and integrates them into the method proposed in the present invention to test the effectiveness of the present invention:

[0136] (1) GRU[7]: is a pioneering model for learning user behavior sequences.

[0137] (2) NextItNet[8]: is a simple yet effective model whose network structure consists of multiple stacked convolutional layers.

[0138] (3)SASRec[9]: is a sequential recommendation model built on the Transformer structure, which can capture the long-term dependencies between different interactions in the student behavior sequence.

[0139] (4) Bert4Rec

[10] : is a transformer-based sequence recommendation model that uses bidirectional self-attention to model students’ behaviors.

[0140] The present invention conducts experiments on two data sets:

[0141] (1) ASSIST09[5]: This dataset is collected from ASSISTments (an online tutoring system) in the academic year 2009-2019. This dataset is divided into two data files, which are spliced ​​in the experiment. After removing "NAN" knowledge concepts and repeated sequences containing multiple knowledge concepts, 112 knowledge points are provided, and more than 3,000 students answer 15,000 exercises. The dataset contains a total of 279,658 interactions, and the sparsity of the dataset is 99.5%.

[0142] (2) EdNet[6]: This dataset is a large-scale dataset collected by Santa (an AI tutoring platform) since 2018. It contains 131,417,236 interactions from 784,309 students and is the largest public dataset in terms of the total number of students and the number of interactions. There are four datasets of different levels, KT-1, KT-2, KT-3, and KT-4, all of which contain information such as the time of questioning, student ID, exercise ID, and answers submitted by students. This paper randomly samples 4% of student data on EdNet KT-1 for experiments, which contains 2,119 students and 277,000 interactions on 142 knowledge points and 11,000 exercises. The sparsity is about 98.8%.

[0143] The present invention is evaluated from two aspects: improving students' learning ability and recommending accurate performance. The evaluation indicators are defined as follows:

[0144] (1) Regarding the first aspect (improving students’ learning ability), the present invention uses students’ learning gain as an indicator:

[0145]

[0146] Where M represents the number of interactions, and gain(·) indicates whether the student's ability has been improved, that is, whether the length of the exercises mastered by the student after the recommendation is higher than before the recommendation. If so, it is 1, otherwise it is 0. The calculation principle of the exercise mastery degree is the same as that of the reward part.

[0147] (2) For recommendation accuracy, the commonly used recommendation performance evaluation indicators hit rate (HR) and normalized discounted cumulative gain (NDCG) are used to measure it. Specifically, HR is used to evaluate whether the real exercises are in the recommendation list. NDCG considers the position of each real answer exercise in the recommendation list. It is equivalent to standardization between different students, and higher ranking positions in the recommendation list are assigned higher scores.

[0148] The four basic recommendation models integrated by the present invention have achieved certain improvements in the recommendation accuracy index and learning gain performance on the ASSIST09 and EdNet datasets. On the ASSIST09 dataset, the performance of the HR and NDCG recommendation indicators has improved by 4%-13%. In EdNet, the method of the present invention brings an average performance improvement of 6%. For the L_G evaluation indicator, it brings more than 10% improvement in both datasets.

[0149] References

[0150] [1]Fan Xiao.Item response theory and classical test theory:Anempirical comparison of their item / person statistics.EducationandPsychologicalMeasurement,1998,58(3):357-381.

[0151] [2]De la Thrre J.DINA model and parameter estimation:Adidactic.Journal of Educational and Behavioral Statistics,2009,34(1):115-130.

[0152] [3]J.Zhang,B.Hao,B.Chen,C.Li,H.Chen,and J.Sun.Hierarchicalreinforcement learning for course recommendationin moocs.ProceedingsoftheAAAI conference on artificial intelligence,2019,33(1):435-442.

[0153] [4]W.Lee,J.Chun,Y.Lee,K.Park,and S.Park.Contrastive learning forknowledge tracing.Proceddings of the ACMWeb Conference,2022,2330-2338.

[0154] [5]M.Feng,N.Heffernan,and K.Koedinger.Addressing the assessmentchallenge with an online system that tutors as it assesses,User modelinganduser-adapted interaction,2009,19,243–266.

[0155] [6]Y.Choi,Y.Lee,D.Shin,J.Cho,S.Park,S.Lee,J.Baek,C.Bae,B.Kim,andJ.Heo.Ednet:A large-scale hierarchical dataset in education.inArtificialIntelligence in Education:21st International Conference,AIED 2020,Ifrane,Morocco,July 6–10,2020,Proceedings,Part II 21.Springer,2020,69–73.

[0156] [7]B.Hidasi,A.Karatzoglou,L.Baltrunas,and D.Tikk.Sessionbasedrecommendations with recurrent neural networks.arXiv preprint arXiv:1511.06939,2015.

[0157] [8]F.Yuan,A.Karatzoglou,I.Arapakis,J.M.Jose,and X.He.A simpleconvolutional generative network for next item recommendation.in Proceedingsofthe twelfth ACM international conference on web search and data mining,2019,582–590.

[0158] [9]W.-C.Kang and J.McAuley.Self-attentive sequentialrecommendation.2018IEEE international conference on datamining(ICDM).IEEE,2018,197–206.

[0159]

[10] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang. Bert4rec: Sequentialrecommendation with bidirectional encoder representations from transformer. Proceedings of the 28thACM international conference on information and knowledge management, 2019, 1441–1450.

[0160] The protection content of the present invention is not limited to the above embodiments. The described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0161] The term "comprises" and any variation thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the appended claims.

Claims

1. A personalized exercise recommendation method integrating reinforcement and contrastive learning, characterized in that: The method comprises the following steps: Step 1: Select a data set and preprocess the data set; Step 2: Data augmentation is performed on the original student history sequence by combining three data augmentation methods: question mask, reordering interaction permute, and question replacement. Two related view representations of the original learning history are obtained, namely u +1 ,u +2 ; Step 3: Input the original student history sequence and the enhanced sequence into the embedding layer to obtain the feature representation of the exercises and the student-exercise interaction vector respectively; Step 4: According to the feature representation of the exercises and interaction vectors obtained in step 3, an integrable sequence recommendation model including GRU and / or SASRec and / or NextItNet and / or Bert4Rec is used to encode the exercises and interaction vectors to obtain corresponding hidden representations; Step 5: Concatenate the hidden states obtained in step 4 to obtain the concatenated representation of the original question and the interaction sequence. And the representation after data augmentation exercises and interaction sequences splicing and Step 6: After obtaining the hidden representation of the exercise and interaction after data enhancement according to step 4, the entire exercise and interaction history representation is obtained through an average pooling layer; Step 7: The potential representation obtained in step 5 and As input, the Q value corresponding to each input is obtained through the Q value function, and concatenated to obtain the final Q value; then the cumulative reward is maximized through the self-supervised reinforcement learning method; In step eight, we combine self-supervision, reinforcement, and contrastive learning losses, use gradient descent to train the model, pass the obtained student representation through a self-supervision head, obtain the final exercise recommendation result, and recommend more suitable exercises to students.

2. The method for recommending exercises according to claim 1, characterized in that: In step 1, the preprocessing of the data set includes deleting answer records without knowledge points, deleting records with student answer sequences ≤ 3, and deleting duplicate records caused by one exercise having two or more knowledge points.

3. The method for recommending exercises according to claim 1, wherein: In step 2, the student set and exercise set are denoted as U and Q respectively, and the student history sequence is defined as u 1:t ={u1,u2,...,u t }, each interaction is represented by a tuple u t = {q t ,r t }, where q t ∈Q is the tth exercise in the exercise set, and also contains the relevant knowledge points k∈K involved in the exercise, where K is the knowledge point set, representing all the knowledge points contained in the exercise set Q; r t ∈{0,1} is the answer result, that is, the correct answer to this question r t is equal to 1, otherwise r t is 0; the enhancement methods include mask, permute and replace.

4. The method for recommending exercises according to claim 1, wherein: In step 3, the embedding layer includes an exercise embedding layer and a student-exercise interaction embedding layer; The steps of converting the original user history sequence and the enhanced sequence into vector feature representation are as follows: (1) Construct the exercise matrix E Q ∈P N×d , the interaction matrix E between students and exercises I ∈P 2N×d , where N represents the number of exercises, d represents the dimension of embedding, and the interaction matrix contains the correctness of the student's answer; (2) Given an interaction u t =(q t ,r t ), and obtain the exercise embedding vector at time t through the embedding layer and the interaction embedding vector (3) Obtain the exercise embedding vector of the enhanced data and the interaction embedding vector 5. The method for recommending exercises according to claim 1, characterized in that: In step 4, the steps of encoding the exercises and interaction vectors using the existing recommendation model are as follows: (1) Convert the next exercise recommendation task into a multi-classification problem and use an integrated sequence recommendation model to encode a series of exercises and interactions; (2) Process the original student history sequence and embed the given exercise sequence Exercise Encoder G Q The learning exercises show: Interactive Encoder G I Interactive embedding sequence through input Get the corresponding exercise interaction representation (3) Process the student sequence after data augmentation and obtain the representation of learning history through the hidden representation of exercises and interactions:

6. The method for recommending exercises according to claim 1, characterized in that: In step 5, the original historical sequence is obtained and the enhanced sequence Here are the steps: The representation information of exercises and interactions is concatenated to form the state in reinforcement learning: and / or, In step 6, the historical representation formula for the entire exercise and interaction is as follows: in, and is the hidden representation of the enhanced questions and interactions obtained in step 4, pool(·) is an average pooling layer, and the final output z Q ∈P d , z I ∈P d .

7. The method for recommending exercises according to claim 1, characterized in that: In step 7, given the representation of the student’s learning history, the exercise recommendation task is transformed into a multi-classification problem, and the reinforcement learning method is used to maximize the cumulative reward. The steps are as follows: (1) Obtain the hidden states of the exercises and interactions according to step 5; (2) The recommendation model accepts the embedding vector as input and generates n classification log y t+1 =[y1,y2,...,y n ]∈P n , n is the number of candidate exercises, and the classification log is represented by y t+1 =f(s t ) is obtained, in which a fully connected layer is used; (3) Based on the logarithm y t+1 Optimize cross entropy loss to train the recommendation model: Among them, Y i is an indicator function. If the student interacts with the i-th exercise at the next time stamp, then Y i =1, otherwise Y i =0; use s directly t as the current state of the RL head, and stack an additional final layer to do the corresponding Q-value calculation: Q z (s t ,a t )=δ(s t W z +b z )=δ(G(e 1:t )W z +b z ), where z∈{1,2.3}, δ is the activation function, W z , b z are learnable parameters; (4) Superimpose the obtained Q1, Q2, and Q3 into a vector Q value function: (5) At each time step t, Q-Network obtains experience samples (s) with a batch size of 256 based on the experience pool D generated from the implicit feedback data. t a t ,r t ,s t+1 ) is used to calculate TD-error as the loss of the RL part, and its loss function is: Among them, R t is the immediate reward in the current state, s t+1 is the successor state, s t is the current state, Q(s t ,a t ) and Q(s t+1 , a') represent the estimated values ​​of the state value function of the current state and the subsequent state respectively, γ is the discount factor, which represents the discount ratio of future rewards; Experience replay improves sampling efficiency, and training a fixed target network prevents the approximation error from propagating quickly between two states. (6) RL acts as a regularizer, according to the defined reward mechanism R t Evaluate the recommendation quality of the first k exercises to fine-tune the model G; (7) In order to improve the stability of learning, Double Q-learning is used to alternately train the learnable parameters.

8. The method for recommending exercises according to claim 7, characterized in that: In step 7, the reward function R is customized t , based on the results obtained from the DKT model, a dynamic reward mechanism is designed. The reward mechanism is defined as follows: p t =DKT(u1,u2,...,u t ), Where N is the number of exercises, β is an adjustment factor, which is set to 10 based on experience. First, the student's exercise sequence before time t is input to obtain the student's mastery of each exercise before recommendation, and the right and wrong information of the recommended exercises can be obtained from it. Then, the interaction sequence at time t is input into the DKT model to obtain the student's mastery of the exercises after the recommended exercises. Finally, the average of the difference between the mastery of all exercises at time t and time t-1, that is, the increment of the current student's knowledge status, is used as the final reward.

9. The method for recommending exercises according to claim 1, characterized in that: In step 8, we combine self-supervision, reinforcement, and contrastive learning losses. The steps of the contrastive learning part are as follows: (1) Obtain a positive pair (u) from the data enhancement in step 2 +1 ,u +2 ); the pair can be instantiated as a problem history (z Q,+1 ,z Q,+2 ) and interaction history (z I,+1 , z I,+2 ); (2) The enhanced samples from other histories in the batch with a batch size of 256 are represented as negative samples z Q,- ∈Z Q,- , z I,- ∈Z I,- ; (3) Define contrastive learning loss using two parts: exercises and interactions: Among them, sim is a cosine similarity function; the contrast loss function is used to learn effective representations to achieve the goal of pulling together similar learning histories and separating different learning histories; (4) The final overall objective function is a linear combination of self-supervision, reinforcement and contrast losses: L=L S +L q +αL cl , Among them, α is a hyperparameter, which represents the influence ratio of the contrastive learning part.

10. A personalized exercise recommendation system integrating reinforcement and contrastive learning, characterized in that: The system implements the exercise recommendation method as claimed in any one of claims 1 to 9, and the system comprises: a student history sequence conversion module, a student knowledge state capture module, and an exercise recommendation module; The learning history sequence conversion model processes the student history sequence, passes the student answer sequence through the embedding layer and the encoding layer to obtain the feature representation of the exercise and the interaction vector, and also performs data enhancement processing; The student knowledge state capture module is used to learn the student's exercises and interaction representations; the exercise recommendation process is formalized as a Markov decision process using a self-supervised reinforcement method, and the interaction is performed between the student and the recommendation agent; the maximum long-term cumulative reward is obtained through the student's long-term needs in practice, so as to maximize the student's long-term satisfaction with the system; The student knowledge status capture module also introduces a custom reward mechanism to timely obtain changes in students' knowledge mastery during the learning process, so that the recommended exercises can achieve the purpose of improving students' learning ability; The exercise recommendation module integrates the learning representations obtained by the above two modules and recommends exercises through a self-supervision head, thereby recommending exercises that can improve the learning ability of learners.

Citation Information

Patent Citations

  • Dynamic perception test question recommendation method and system fused with deep knowledge tracking

    CN114372137A

  • Multi-relation cognitive diagnosis method based on graph convolutional network

    CN115510286A

  • Reinforcement learning method

    EP3543918A1

Cited By

  • Advertisement recommendation method and system based on deep reinforcement learning

    CN119444326A

  • Relation attribute-based interpretable exercise recommendation method and system

    CN119557343A

  • A method and system for recommending explainable exercises based on relational attributes

    CN119557343B

  • Knowledge tracking method and system based on reinforcement learning and negative sampling optimization

    CN119670867A

  • Personalized recommendation system and method for intelligent terminal

    CN120234471A