A personalized exercise recommendation method based on reinforcement learning

Abstract

The present invention provides a personalized exercise recommendation method based on reinforcement learning, which relates to the field of educational data mining technology. The present invention first obtains the learner's learning record, determines the learner's potential knowledge level through a knowledge tracking model, and uses it as part of the learner's characteristics, making the learner's characteristic modeling more accurate. Then, a reinforcement learning algorithm is used to delete unsatisfactory exercises mistakenly selected by the learner in the exercise record, thereby improving the accuracy of the recommendation. Finally, exercises are recommended to the learner through a personalized recommendation model. The present invention combines personalized recommendation, knowledge tracking, and reinforcement learning algorithms. While taking into account the learner's potential knowledge level, it also eliminates the impact of incorrectly selected exercises during the learning process, and has important theoretical and practical application value.

Description

Technical Field

[0001] The present invention relates to the technical field of educational data mining, and in particular to a personalized exercise recommendation method based on reinforcement learning. Background Art

[0002] The development of emerging information and communication technologies, such as mobile communications, the internet, the Internet of Things, cloud computing, big data, and artificial intelligence, is transforming how we think, work, live, and learn. Today's education is evolving toward a "networked, digital, personalized, ubiquitous, and intelligent" future, with a plethora of new educational models emerging, including mobile learning, generalized learning, intelligent learning, and blended learning.

[0003] In recent years, online learning, as an emerging personalized learning model, has successfully attracted a large number of learners to register and use it due to its convenience, openness, and rich learning resources. In this new generation of internet-based learning environment, learning time is more flexible, learning methods are more diverse, and learning resources are more abundant. Learners can independently arrange their study time, learning methods, and learning resources based on their own learning situation and learning goals.

[0004] However, unlike traditional classrooms, online education platforms cannot provide real-time supervision and guidance to learners, leading to problems such as "information overload" and "knowledge wandering." These issues primarily manifest themselves in learners facing a vast number of learning resources of varying quality. They often spend considerable time finding resources of interest, are unsure how to plan their studies, and sometimes, even after dedicating considerable time to studying, are unable to effectively achieve their learning goals. These issues can lead to decreased learning efficiency, diminished learning quality, reduced motivation, and an increased risk of learning failure. The emergence of these issues has sparked reflection among educators and researchers, and the use of computers to replace teachers in guiding and assisting learners has gradually become a hot research topic.

[0005] Solving the practical problem of online learners having difficulty finding interesting exercises when faced with a vast amount of learning resources, and proposing feasible personalized exercise recommendation algorithms to significantly improve learners' learning efficiency, are urgent issues that need to be addressed at this stage. To this end, the following three issues need to be considered:

[0006] First, how to accurately construct the characteristics of learners.

[0007] Commonly used personalized recommendation models, whether matrix factorization, recurrent neural network, or attention mechanism, all use their practice history to model learner characteristics when solving exercise recommendation problems. They don't consider the learner's performance on the exercises. This can lead to the following problem: Suppose learner i and learner j have essentially the same practice history, but their performance on the exercises differs. If learner i gets the vast majority of the exercises right, while learner j gets the vast majority wrong, then the exercises they choose next are likely to be different.

[0008] It can be seen that constructing learner characteristics based solely on the exercises they have done is not accurate enough. Therefore, how to consider the learner's potential knowledge level when modeling the learner is the primary consideration.

[0009] Second, how to eliminate the impact of incorrectly selected exercises during the learning process.

[0010] When selecting exercises, learners often choose exercises they are dissatisfied with, such as those chosen due to difficulty or category. However, the exercise history does not include the learner's satisfaction level with the exercises. These incorrectly selected exercises can become a distractor when modeling learners' interest characteristics. Although researchers have attempted to use attention mechanisms to assign different attention coefficients to each learner's historical exercises to distinguish their importance, they still cannot completely eliminate the impact of these incorrectly selected exercises. Therefore, how to eliminate the impact of incorrectly selected exercises is a necessary issue that needs to be considered.

[0011] Third, how to accurately recommend exercises.

[0012] After considering the learner's potential knowledge level and eliminating the impact of incorrectly selected exercises, the final goal is to accurately recommend exercises to the learner. Therefore, choosing the right personalized recommendation algorithm is a crucial consideration.

[0013] Integrating reinforcement learning algorithms to address challenges encountered in online education is a current research hotspot in educational data mining. Combining knowledge tracking models, personalized recommendation models, and reinforcement learning models not only takes into account learners' potential knowledge levels but also mitigates the impact of incorrectly selected exercises, effectively addressing the "information overload" problem in online education. Using reinforcement learning for personalized exercise recommendations in online education is a promising approach to improving learner efficiency. Summary of the Invention

[0014] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a personalized exercise recommendation method based on reinforcement learning. The method is a personalized exercise recommendation method based on reinforcement learning and combines a knowledge tracking model and a personalized recommendation model to solve the practical problem that learners in online education find it difficult to find learning resources of interest.

[0015] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0016] A personalized exercise recommendation method based on reinforcement learning includes the following steps:

[0017] Step 1: Use the knowledge tracking model to calculate the learner's potential knowledge level and incorporate it into the feature construction of the personalized recommendation model and the state representation of the exercise record modification model;

[0018] Step 2: Build and train a personalized recommendation model for exercise recommendations;

[0019] Step 3: Design and train an exercise record modification model based on the Deep Q-Learning algorithm based on reinforcement learning to remove exercises that are disliked or unsatisfactory and that were mistakenly selected during the learning process;

[0020] Step 4: Jointly train the personalized recommendation model and the exercise record modification model;

[0021] Step 5: Use the exercise record modification model obtained after the joint training in step 4 to modify the learner's exercise record, and then use the personalized recommendation model obtained after the joint training in step 4 to recommend exercises to the learner to obtain an exercise recommendation list.

[0022] Furthermore, in step 1, the knowledge tracking model used is the deep knowledge tracking model DKT; the DKT model uses the long short-term memory network LSTM to use the temporal relationship to predict the next moment's test score based on the learner's historical learning record; the DKT model first generates a one-hot vector from the learner's historical score through one-hot encoding, inputs the one-hot vector into the LSTM network, extracts features through the LSTM layer, inputs the extracted features into the hidden layer, and then outputs the prediction results from the output layer. The output of the DKT model represents the probability that the learner will correctly answer each exercise, that is, the learner's next test score; the output of the LSTM layer is used as the learner's potential knowledge level, and is added to the feature construction of the personalized recommendation model and the state representation of the exercise record modification model. The input of the DKT model is the learner's practice record. The practice record of learner i at time t is specifically expressed as in represents the exercise selected by learner i at time t, Indicates the answer result of learner i at time t; exercise record Contains only the exercises and practice records that learner i has chosen to study The answer results of learner i are also recorded.

[0023] Furthermore, the personalized recommendation model in step 2 includes three parts, namely, the Embedding layer, the GRU layer and the fully connected layer; the Embedding layer is used to map the one-hot vector of the learner's exercise records to a low-dimensional vector space for encoding; the GRU layer is a gated recurrent unit layer, which is also an improved recurrent neural network model for extracting sequence features of exercise records; the fully connected layer is used to calculate the probability of the learner selecting each exercise based on the learner's characteristics, and recommend exercises to the learner according to the size of the selection probability.

[0024] Furthermore, the specific method of step 2 is:

[0025] Step 2-1: Record the exercises done by learner i through the Embedding layer One-hot vector Mapped to a low-dimensional vector space for encoding, the output is a low-dimensional vector

[0026] Step 2-2: Extract the sequence features of the exercise records through the GRU layer;

[0027] The update gate of GRU determines the amount of state information at the previous moment and the current moment that is passed on to the future. The calculation formula is as follows:

[0028]

[0029] in, The low-dimensional vector representation of the exercises done by learner i at time t, h t-1 represents the hidden state information at time t-1, W z represents the weight coefficient of the update gate, σ(·) is the sigmoid activation function;

[0030] The reset gate of the GRU layer determines the amount of state information at the previous moment to be forgotten. The calculation formula is as follows:

[0031]

[0032] Among them, W r Represents the weight coefficient of the reset gate;

[0033] The calculation formula for the current memory content is as follows:

[0034]

[0035] Among them, W h is another weight coefficient of the reset gate, reset gate r t and hidden state information h t-1 The product of the corresponding elements determines the information to be retained at the previous moment, and * is the operator representing the matrix dot multiplication;

[0036] The calculation formula for the final memory of the current time step is as follows:

[0037]

[0038] Among them, (1-z t )*h t-1 Indicates the amount of information from the previous moment that is retained to the final memory at the current moment. Indicates the amount of current memory content retained to the current moment; the final h t It is the sequence characteristics of learners’ exercise records;

[0039] Step 2-3: Calculate the probability of the learner selecting each exercise based on the learner's characteristics through the fully connected layer, as shown in the following formula:

[0040] y=softmax(W j ·[K i , h t ]+b j )

[0041] Among them, W j is the weight coefficient of the fully connected layer, b j is the bias coefficient of the fully connected layer, is the potential knowledge level of learner i calculated by the DKT model; [K i , h t ] is the sequence feature h of the learner's exercise record obtained by the GRU layer and the potential knowledge level of the learner i. t Splicing is performed; softmax(·) is an activation function that limits the output value to between 0 and 1;

[0042] Step 2-4: The personalized recommendation model uses cross entropy as the loss function to train and update the model. The calculation formula is as follows:

[0043]

[0044] Where M is the number of learners, p i is the true probability distribution of learner i choosing an exercise at the next moment, q iThe predicted probability distribution of the exercise selected by learner i at the next moment is given by the personalized recommendation model;

[0045] The cross entropy loss function is an indicator to measure the difference between the true probability distribution p and the model predicted probability distribution q;

[0046] Step 2-5: Sort the probability of learner i selecting each exercise calculated by the personalized recommendation model in descending order, and take the top K exercises to form an exercise recommendation list and recommend it to learner i.

[0047] Furthermore, the exercise record modification model in step 3 adopts a reinforcement learning-related algorithm, including the model's action representation, state representation, reward function, and reinforcement learning algorithm, as follows:

[0048] In order to delete the exercises that the learners do not like or are not satisfied with during the learning process, each step a t There are only two values, a t =0 means deleting the exercise in the exercise record, a t =1 means to keep the exercise in the exercise record;

[0049] The learner's state is expressed as follows:

[0050] S=[k1,k2,...,k N ,p1,p2,...,p N ]

[0051] Among them, k1, k2, ..., k N Represents the learner's potential knowledge level. Specifically, the representation of the i-th learner is Given by the knowledge tracking model; p1, p2, ..., p N It is a low-dimensional vector representation of the learner's exercise record and position identifier. The role of the position identifier is to record the modified location;

[0052] The reward function of the reinforcement learning module is given by the personalized recommendation model and is shown in the following formula:

[0053]

[0054] Among them, e target is the exercise that the learner actually chooses at the next moment. represents the probability of selecting the target exercise according to the modified exercise record, p(e target |E i) represents the probability of selecting the target exercise based on the original exercise record; the reinforcement learning module adopts a round-up update strategy, and the reward function is obtained only after the modification of a learner's entire learning record is completed. The reward function is 0 at other times;

[0055] The reinforcement learning algorithm used is the deep Q network algorithm DQN, which combines the neural network with the Q-Leaming algorithm in the traditional reinforcement learning algorithm;

[0056] The reinforcement learning module uses the square of the difference between the true value and the predicted value as the loss function to train and update the parameters of the DQN model. The specific formula of the loss function is shown below:

[0057]

[0058] Among them, Q θ (s t , a t ) means in state s t Next select action a t The predicted value of the reward to be obtained is calculated by the prediction Q network, and the network parameter of the prediction Q network is θ; Indicates state s t Next select action a t The actual value of the reward that can be obtained; Calculated by the target Q network, it represents the next state s t+1 The maximum reward value that can be obtained, the network parameters of the target Q network are r t is the current reward value, given by the reward function;

[0059] The gradient of the loss function is shown below:

[0060]

[0061] Update the network parameters according to gradient descent.

[0062] Furthermore, the specific process of modifying the learner's exercise record in step 3 is as follows:

[0063] Step 3-1: Model initialization, including initializing the parameters of the prediction Q network and the target Q network; initializing the experience replay pool with a capacity of N; initializing the learner's modified exercise record set Learner number i = 1, time t = 0;

[0064] Step 3-2: Get the exercise record of learner i and initial state s0;

[0065] Step 3-3: Change the state s tEigenvector φ(s t ) as the input of the prediction Q network to obtain the Q value corresponding to the action in the current state;

[0066] Step 3-4: Use the ε-greedy strategy to select action a in the current Q value t ;

[0067] Step 3-5: If a t =0, then delete in

[0068] Step 3-6: In state s t Execute the current action a t , get the next state s t+1 and reward r t ;

[0069] Step 3-7: Replace {s t , a t , r t , s t+1}This quadruple is stored in the experience replay pool;

[0070] Step 3-8: Update status s t =s t+1 ;

[0071] Step 3-9: Sample m samples from the experience replay pool {s j , a j , r j , s j+1}, j = 1, 2, ..., m, calculate the current target Q value y j :

[0072]

[0073] Step 3-10: Use mean square error loss function To update the parameters of the prediction Q network;

[0074] Step 3-11: Update the parameters of the target Q network after every C steps, and the parameter values are the parameter values of the current predicted Q network;

[0075] Step 3-12: Determine whether the time reaches the set value T; if not, return to step 3-3; if it reaches, execute the next step;

[0076] Step 3-13: Record the modified exercises i Recorded as Will Add to the learner's revised exercise record

[0077] Step 3-14: Determine whether all learners' exercise records have been modified. If not, return to step 3-2 and continue to modify the next learner's exercise record. If yes, end the step.

[0078] Furthermore, the process of the joint training in step 4 is specifically as follows:

[0079] Step 4-1: Initialize the parameters α=α0 of the personalized recommendation model, β=β0 of the knowledge tracking model, and θ=θ0 of the reinforcement learning module;

[0080] Step 4-2: Utilize learner practice records Train the knowledge tracking model;

[0081] Step 4-3: Utilize learner exercise records and knowledge tracking models to train personalized recommendation models;

[0082] Step 4-4: Fix the parameters α=α1 of the personalized recommendation model and β=β1 of the knowledge tracking model, and pre-train the reinforcement learning module. The specific method is:

[0083] Step 4-4-1: Reinforcement learning algorithm in exercise record Select an action on the screen;

[0084] Step 4-4-2: Calculate the reward function Reward based on the selected action;

[0085] Step 4-4-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm;

[0086] Step 4-4-4: Loop through steps 4-4-1 to 4-4-3 until all exercise records are completed. The cycle is complete;

[0087] Step 4-4-5: Repeat steps 4-4-1 to 4-4-4 until the parameters of the reinforcement learning module are optimized;

[0088] Step 4-5: Fix the knowledge tracking parameter β = β1 and jointly train the personalized recommendation model and reinforcement learning module. The specific method is:

[0089] Step 4-5-1: Reinforcement learning algorithm in exercise record Select an action on the screen;

[0090] Step 4-5-2: Calculate the reward function Reward based on the selected action;

[0091] Step 4-5-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm;

[0092] Step 4-5-4: Repeat steps 4-5-1 to 4-5-3 until all The cycle is complete;

[0093] Step 4-5-5: Update the parameters of the recommendation model according to the loss function of the recommendation model;

[0094] Step 4-5-6: Repeat steps 4-5-1 to 4-5-5 until the personalized recommendation model and reinforcement learning module parameters are optimized.

[0095] The beneficial effects of adopting the above technical solution are as follows: the personalized exercise recommendation method based on reinforcement learning provided by the present invention first obtains the learner's learning record, determines the learner's potential knowledge level through the knowledge tracking model, and uses it as part of the learner's characteristics, making the learner's characteristic modeling more accurate. This chapter then attempts to use the reinforcement learning algorithm to delete unsatisfactory exercises that the learner mistakenly selected in the exercise record, thereby improving the accuracy of the recommendation. Finally, exercises are recommended to the learner through the personalized recommendation model. The method of the present invention combines personalized recommendation, knowledge tracking, and reinforcement learning algorithms. While taking into account the learner's potential knowledge level, it also eliminates the impact of mistakenly selected exercises during the learning process, and has important theoretical and practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 A structural diagram of a personalized exercise recommendation model provided by an embodiment of the present invention;

[0097] Figure 2 A flowchart of a personalized exercise recommendation method based on reinforcement learning provided by an embodiment of the present invention;

[0098] Figure 3 A structural diagram of the knowledge tracking model DKT provided in an embodiment of the present invention;

[0099] Figure 4 A structural diagram of a long short-term memory (LSTM) network provided in an embodiment of the present invention;

[0100] Figure 5 A structural diagram of a personalized recommendation model provided by an embodiment of the present invention;

[0101] Figure 6 This is a structural diagram of the Deep Q Network (DQN) provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0102] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0103] This embodiment provides a personalized exercise recommendation method based on reinforcement learning, such as Figure 1 As shown, the model constructed by the method of this embodiment consists of three parts, namely, a knowledge tracking model, a personalized recommendation model, and an exercise record modification model. The knowledge tracking model can calculate the learner's potential knowledge level and add it to the feature construction of the personalized recommendation model and the state representation of the exercise record modification model. The personalized recommendation model provides a reward function for the exercise record modification model and recommends exercises to learners. The exercise record modification model modifies the learner's historical exercise records, and updates them based on the reward function provided by the personalized learning model to determine the quality of the modification, thereby improving the accuracy of exercise recommendations. The process of this method is as follows Figure 2 The specific method is as follows.

[0104] Step 1: Use the knowledge tracking model to calculate the learner's potential knowledge level and add it to the feature construction of the personalized recommendation model and the state representation of the exercise record modification model.

[0105] The knowledge tracing model adopted in this embodiment is the Deep Knowledge Tracing (DKT) model. The DKT model uses a recurrent neural network or a long short-term memory network LSTM to use temporal relationships to predict the next moment's test scores based on the learner's historical learning records. The recurrent neural network in this embodiment uses a long short-term memory network LSTM. The DKT model first generates a one-hot vector for the learner's historical scores through one-hot encoding, inputs it into the LSTM network, extracts features through the LSTM layer, inputs this feature into the hidden layer, and then outputs the prediction results from the output layer. The output of DKT represents the probability of the learner correctly answering each exercise, that is, the score of the learner's next answer.

[0106] The structure of the DKT model is as follows Figure 3 As shown in the figure, the model shown in the figure is a knowledge tracking model based on the Long Short-Term Memory (LSTM) network, which can judge the learner's potential knowledge level through the learner's performance in the learning record. The input of the DKT model is the practice record of learner i The practice record of learner i at time t is specifically expressed as in represents the exercise number selected by learner i at time t, represents the performance of learner i on the exercise at time t, 1 means the exercise is correct, and 0 means the exercise is wrong. Converted into a one-hot vector through one-hot encoding Input to the LSTM network.

[0107] LSTM network is an improved recurrent neural network that can solve the problem that RNN cannot handle long-distance dependencies. The structure of LSTM is as follows: Figure 4 shown.

[0108] Unlike recurrent neural networks, long short-term memory neural networks introduce a memory state and control the storage of information through three gating units of neurons, so that the memory state of neurons always saves the information of the entire long sequence.

[0109] The forget gate in the LSTM network is responsible for controlling the state retention of the previous moment. The calculation formula is shown as follows:

[0110]

[0111] Among them, W f is the weight matrix of the forget gate, is the input of the forget gate at time t, and here is the practice record of learner i at time t, Represents the concatenation of two vectors, h t-1 represents the output at time t-1, b f represents the bias term of the forget gate, and σ(·) is the sigmoid activation function.

[0112] The input gate in the LSTM network is responsible for controlling the input of the current state into the long-term state. The calculation formula is shown as follows:

[0113]

[0114] Among them, W I is the weight matrix of the input gate, b I is the bias term of the input gate.

[0115] The current input unit state is expressed as follows:

[0116]

[0117] Among them, W c is the weight matrix of the unit state, b c is the bias term of the cell state, and tanh is the activation function.

[0118] Through the above three formulas and the cell state C at the previous moment t-1 Get the current unit state, as shown below:

[0119]

[0120] Here, * is an operator representing matrix dot product.

[0121] The output gate in the LSTM network is responsible for controlling whether the long-term state is used as the current output, which is expressed as follows:

[0122]

[0123] Among them, W o is the weight matrix of the output gate, b c is the bias term of the output gate.

[0124] Finally, the output state is obtained by the following formula:

[0125] h t =o t *tanh(C t )

[0126] The DKT model can comprehensively consider the learner's long-term practice performance and recent practice performance to determine the learner's potential knowledge level. The design of the forget gate is in line with the fact that learners will gradually lose their mastery of previously learned knowledge over time. In this embodiment, the output of the LSTM layer is recorded as the learner i's potential knowledge level on N knowledge points. It is used as part of the learner's features to improve the recommendation performance.

[0127] Step 2: Build and train a personalized recommendation model, which includes three parts: Embedding layer, GRU layer and fully connected layer. The role of the Embedding layer is to map the one-hot vector of the exercise record done by learner i to a low-dimensional vector space for encoding; the GRU layer is a gated recurrent unit layer, which is also an improved recurrent neural network model, and its role is to extract the sequence features of the exercise record; the role of the fully connected layer is to calculate the probability of learner i choosing each exercise through his characteristics, and recommend exercises to learners according to the size of the selection probability. This personalized recommendation model has two functions: one is to provide a reward function for the exercise record modification model, and the other is to recommend exercises to learners. The structure of the personalized recommendation model is as follows Figure 5 The specific method is as follows.

[0128] Step 2-1: Record the exercises done by learner i through the Embedding layer One-hot vector Mapped to a low-dimensional vector space for encoding, the output is a low-dimensional vector

[0129] Step 2-2: Extract the sequence features of the exercise records through the GRU layer.

[0130] The GRU layer has only two operations: the update gate and the reset gate. The GRU layer calculates the outputs of the reset and update gates based on the current input and the network hidden state at the previous moment. It then calculates the candidate hidden state based on the current input and the output of the reset gate. Finally, it calculates the final hidden state based on the candidate hidden state and the output of the update gate, and then calculates the current output based on the hidden state.

[0131] The update gate of GRU determines the amount of state information at the previous moment and the current moment that is passed on to the future. The calculation formula is as follows:

[0132]

[0133] in, The low-dimensional vector representation of the exercises done by learner i at time t, h t-1 represents the hidden state information at time t-1, W z represents the weight coefficient of the update gate, and σ(·) is the sigmoid activation function.

[0134] The reset gate of the GRU layer determines the amount of state information at the previous moment to be forgotten. The calculation formula is as follows:

[0135]

[0136] Among them, W r Represents resetting the weight coefficient of the gate.

[0137] The calculation formula for the current memory content is as follows:

[0138]

[0139] Among them, W h is the reset gate r t Another weight coefficient, reset gate r t and hidden state information h t-1 The product of the corresponding elements determines the information to be retained at the previous moment, and * is the operator representing the matrix dot multiplication.

[0140] The calculation formula for the final memory of the current time step is as follows:

[0141]

[0142] Among them, (1-z t )*h t-1Indicates the amount of information from the previous moment that is retained to the final memory at the current moment. Indicates the amount of current memory content retained until the current moment. The final h t It is the sequence feature of the learner's exercise records.

[0143] Step 2-3: Calculate the probability of the learner selecting each exercise based on the learner's characteristics through the fully connected layer, as shown in the following formula:

[0144] y=softmax(W j ·[K i , h t ]+b j )

[0145] Among them, W j is the weight coefficient of the fully connected layer, b j is the bias coefficient of the fully connected layer, is the potential knowledge level of learner i calculated by the DKT model; [K i , h t ] is the sequence feature h of the learner's exercise record obtained by the GRU layer and the potential knowledge level of the learner i. t Splicing is performed; softmax(·) is an activation function that limits the output value to between 0 and 1;

[0146] Step 2-4: The personalized recommendation model uses cross entropy as the loss function to train and update the model. The calculation formula is as follows:

[0147]

[0148] Where M is the number of learners, p i is the true probability distribution of learner i choosing an exercise at the next moment, q i The personalized recommendation model gives the predicted probability distribution of the exercise that learner i will choose at the next moment. The cross-entropy loss function is an indicator that measures the difference between the true probability distribution p and the model-predicted probability distribution q.

[0149] Step 2-5: Sort the probability of learner i choosing each exercise calculated by the personalized recommendation model in descending order, and take the top K exercises to form an exercise recommendation list and recommend it to learner i.

[0150] Step 3: Build and train an exercise history modification model to remove exercises that learners mistakenly select during the learning process that they dislike or are dissatisfied with, thereby providing more accurate exercise recommendations. Because the exercise history modification model uses reinforcement learning algorithms, we will provide a detailed introduction to the model's action representation, state representation, reward function, and reinforcement learning algorithm, following the general reinforcement learning development process.

[0151] (1) Action indication

[0152] The exercise record modification model is to delete exercises that learners do not like or are not satisfied with, so each step of the action a t There are only two values, a t =0 means deleting the exercise in the exercise record, a t =1 means to keep the exercise in the exercise record.

[0153] (2) Status indication

[0154] The learner's state is expressed as follows:

[0155] S=[k1,k2,...,k N ,p1,p2,...,p N ]

[0156] Among them, k1, k2, ..., k N represents the learner's potential knowledge level, given by the knowledge tracking model; p1, p2, ..., p N It is a low-dimensional vector representation of the learner's exercise record and position identifier. The role of the position identifier is to record the modified location.

[0157] (3) Reward Function

[0158] The reward function of the reinforcement learning module is given by the personalized recommendation model and is shown in the following formula:

[0159]

[0160] Among them, e target is the exercise that the learner actually chooses at the next moment. represents the probability of selecting the target exercise according to the modified exercise record, p(e target |E i ) represents the probability of selecting the target exercise based on the original exercise record. The reinforcement learning module adopts a round-up update strategy. The reward function is obtained only after the entire learning record of a learner is modified. The reward function is 0 at all other times.

[0161] (4) Reinforcement Learning Algorithms

[0162] This embodiment adopts the Deep Q Network (DQN) algorithm, which combines the neural network with the Q-Leaming algorithm in the traditional reinforcement learning algorithm. The structure of DQN is as follows Figure 6 shown.

[0163] The reinforcement learning module uses the square of the difference between the true value and the predicted value as the loss function to train and update the parameters of the DQN model. The specific formula of the loss function is shown below:

[0164]

[0165] Among them, Q θ (s t , a t ) means in state s t Next select action a t The predicted value of the reward to be obtained is calculated by the prediction Q network, and the network parameter of the prediction Q network is θ. Indicates state s t Next select action a t The actual value of the reward that can be obtained. Calculated by the target Q network, it represents the next state s t+1 The maximum reward value that can be obtained, the network parameters of the target Q network are r t is the current reward value, given by the reward function.

[0166] The gradient of the loss function is shown below, and the network parameters are updated according to gradient descent.

[0167]

[0168] The specific process of modifying a learner's exercise record is as follows:

[0169] Step 3-1: Model initialization, including initializing the parameters of the prediction Q network and the target Q network; initializing the experience replay pool with a capacity of N; initializing the learner's modified exercise record set Learner number i = 1, time t = 0;

[0170] Step 3-2: Get the exercise record of learner i and initial state s0;

[0171] Step 3-3: Change the state s t Eigenvector φ(s t ) as the input of the prediction Q network to obtain the Q value corresponding to the action in the current state;

[0172] Step 3-4: Use the ε-greedy strategy to select action a in the current Q value t ;

[0173] Step 3-5: If a t =0, then delete in

[0174] Step 3-6: In state s t Execute the current action a t , get the next state s t+1 and reward r t ;

[0175] Step 3-7: Replace {s t , a t , r t , s t+1}This quadruple is stored in the experience replay pool;

[0176] Step 3-8: Update status s t =s t+1 ;

[0177] Step 3-9: Sample m samples from the experience replay pool {s j , a j , r j , s j+1}, j = 1, 2, ..., m, calculate the current target Q value y j :

[0178]

[0179] Step 3-10: Use mean square error loss function To update the parameters of the prediction Q network;

[0180] Step 3-11: Update the parameters of the target Q network after every C steps, and the parameter values are the parameter values of the current predicted Q network;

[0181] Step 3-12: Determine whether the time reaches the set value T; if not, return to step 3-3; if it reaches, execute the next step;

[0182] Step 3-13: Record the modified exercises i Recorded as Will Add to the learner's revised exercise record

[0183] Step 3-14: Determine whether all learners' exercise records have been modified. If not, return to step 3-2 and continue to modify the next learner's exercise record. If yes, end the step.

[0184] Step 4: Jointly train the personalized recommendation model and the exercise record modification model to obtain optimal model parameters and improve the accuracy of exercise recommendations. The joint training process of the personalized exercise recommendation model based on the reinforcement learning algorithm proposed in this embodiment is described in detail as follows.

[0185] Step 4-1: Initialize the parameters α=α0 of the personalized recommendation model, β=β0 of the knowledge tracking model, and θ=θ0 of the reinforcement learning module;

[0186] Step 4-2: Utilize learner practice records Record the training of the knowledge tracking model;

[0187] Step 4-3: Use Exercise Record Record and knowledge tracking models are used to train personalized recommendation models;

[0188] Step 4-4: Fix the parameters α=α1 of the personalized recommendation model and β=β1 of the knowledge tracking model, and pre-train the reinforcement learning module. The specific method is:

[0189] Step 4-4-1: Reinforcement learning algorithm in exercise record Select an action on the screen;

[0190] Step 4-4-2: Calculate the reward function Reward based on the selected action;

[0191] Step 4-4-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm;

[0192] Step 4-4-4: Loop through steps 4-4-1 to 4-4-3 until all exercise records are completed. The cycle is complete;

[0193] Step 4-4-5: Repeat steps 4-4-1 to 4-4-4 until the parameters of the reinforcement learning module are optimized;

[0194] Step 4-5: Fix the knowledge tracking parameter β = β1 and jointly train the personalized recommendation model and reinforcement learning module. The specific method is:

[0195] Step 4-5-1: Reinforcement learning algorithm in exercise record Select an action on the screen;

[0196] Step 4-5-2: Calculate the reward function Reward based on the selected action;

[0197] Step 4-5-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm;

[0198] Step 4-5-4: Repeat steps 4-5-1 to 4-5-3 until all The cycle is complete;

[0199] Step 4-5-5: Update the parameters of the recommendation model according to the loss function of the recommendation model;

[0200] Step 4-5-6: Repeat steps 4-5-1 to 4-5-5 until the personalized recommendation model and reinforcement learning module parameters are optimized.

[0201] Step 5: Use the exercise record modification model obtained after the joint training in step 4 to modify the learner's exercise record, and then use the personalized recommendation model obtained after the joint training in step 4 to recommend exercises to the learner to obtain an exercise recommendation list.

[0202] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A personalized exercise recommendation method based on reinforcement learning, characterized by: The following steps are involved: Step 1: Use the knowledge tracking model to calculate the learner's potential knowledge level and incorporate it into the feature construction of the personalized recommendation model and the state representation of the exercise record modification model; Step 2: Build and train a personalized recommendation model for exercise recommendations; Step 3: Design and train an exercise record modification model based on the Deep Q-Learning algorithm based on reinforcement learning to remove exercises that are disliked or unsatisfactory and that were mistakenly selected during the learning process; The exercise record modification model adopts reinforcement learning related algorithms, including the model's action representation, state representation, reward function, and reinforcement learning algorithm, as follows: In order to delete the exercises that the learners do not like or are not satisfied with during the learning process, each step a t There are only two values, a t =0 means deleting the exercise in the exercise record, a t =1 means to keep the exercise in the exercise record; The learner's state is expressed as follows: S=[k1,k2,…,k N ,p1,p2,…,p N ] Among them, k1, k2, …, k N Represents the learner's potential knowledge level. Specifically, the representation of the i-th learner is Given by the knowledge tracking model; p1, p2, …, p N It is a low-dimensional vector representation of the learner's exercise record and position identifier. The role of the position identifier is to record the modified location; The reward function of the reinforcement learning module is given by the personalized recommendation model and is shown in the following formula: Among them, e target is the exercise that the learner actually chooses at the next moment. represents the probability of selecting the target exercise according to the modified exercise record, p(e target |E i ) represents the probability of selecting the target exercise based on the original exercise record; the reinforcement learning module adopts a round-up update strategy, and the reward function is obtained only after the modification of a learner's entire learning record is completed. The reward function is 0 at other times; The reinforcement learning algorithm uses the deep Q network algorithm DQN, which combines the neural network with the Q-Learning algorithm in the traditional reinforcement learning algorithm; The reinforcement learning module uses the square of the difference between the true value and the predicted value as the loss function to train and update the parameters of the DQN model. The specific formula of the loss function is shown below: Among them, Q θ (s t ,a t ) means in state s t Next select action a t The predicted value of the reward to be obtained is calculated by the prediction Q network, and the network parameter of the prediction Q network is θ; Indicates state s t Next select action a t The actual value of the reward that can be obtained; Calculated by the target Q network, it represents the next state s t+1 The maximum reward value that can be obtained, the network parameters of the target Q network are r t is the current reward value, given by the reward function; The gradient of the loss function is shown below: Update network parameters according to gradient descent; Step 4: Jointly train the personalized recommendation model and the exercise record modification model; Step 5: Use the exercise record modification model obtained after the joint training in step 4 to modify the learner's exercise record, and then use the personalized recommendation model obtained after the joint training in step 4 to recommend exercises to the learner to obtain an exercise recommendation list.

2. The personalized exercise recommendation method based on reinforcement learning according to claim 1, characterized in that: In step 1, the knowledge tracking model used is the deep knowledge tracking model DKT; the DKT model uses a long short-term memory network (LSTM) to use temporal relationships to predict the next moment's test score based on the learner's historical learning records; the DKT model first generates a one-hot vector from the learner's historical scores through one-hot encoding, inputs the one-hot vector into the LSTM network, extracts features through the LSTM layer, inputs the extracted features into the hidden layer, and then outputs the prediction results from the output layer. The output of the DKT model represents the probability of the learner correctly answering each exercise, that is, the learner's next test score; the output of the LSTM layer is used as the learner's potential knowledge level and is added to the feature construction of the personalized recommendation model and the state representation of the exercise record modification model; The input of the DKT model is the learner's practice record The practice record of learner i at time t is specifically expressed as in represents the exercise selected by learner i at time t, Indicates the answer result of learner i at time t; exercise record Contains only the exercises and practice records that learner i has chosen to study The answer results of learner i are also recorded.

3. The personalized exercise recommendation method based on reinforcement learning according to claim 1, characterized in that: The personalized recommendation model in step 2 includes three parts: the Embedding layer, the GRU layer, and the fully connected layer. The Embedding layer is used to map the one-hot vector of the learner's exercise records to a low-dimensional vector space for encoding. The GRU layer is a gated recurrent unit layer, which is also an improved recurrent neural network model for extracting sequence features of exercise records. The fully connected layer is used to calculate the probability of the learner selecting each exercise based on the learner's characteristics, and recommend exercises to the learner according to the size of the selection probability.

4. The personalized exercise recommendation method based on reinforcement learning according to claim 3, characterized in that: The specific method of step 2 is: Step 2-1: Record the exercises done by learner i through the Embedding layer One-hot vector Mapped to a low-dimensional vector space for encoding, the output is a low-dimensional vector Step 2-2: Extract the sequence features of the exercise records through the GRU layer; The update gate of GRU determines the amount of state information at the previous moment and the current moment that is passed on to the future. The calculation formula is as follows: in, The low-dimensional vector representation of the exercises done by learner i at time t, h t-1 represents the hidden state information at time t-1, W z represents the weight coefficient of the update gate, σ(·) is the sigmoid activation function; The reset gate of the GRU layer determines the amount of state information at the previous moment to be forgotten. The calculation formula is as follows: Among them, W r Represents the weight coefficient of the reset gate; The calculation formula for the current memory content is as follows: Among them, W h is another weight coefficient of the reset gate, reset gate r t and hidden state information h t-1 The product of the corresponding elements determines the information to be retained at the previous moment, and * is the operator representing the matrix dot multiplication; The calculation formula for the final memory of the current time step is as follows: Among them, (1-z t )*h t-1 Indicates the amount of information from the previous moment that is retained to the final memory at the current moment. Indicates the amount of current memory content retained to the current moment; the final h t It is the sequence characteristics of learners’ exercise records; Step 2-3: Calculate the probability of the learner selecting each exercise based on the learner's characteristics through the fully connected layer, as shown in the following formula: y =softmax(W j ·[K i ,h t ]+b j ) Among them, W j is the weight coefficient of the fully connected layer, b j is the bias coefficient of the fully connected layer, is the potential knowledge level of learner i calculated by the DKT model; [K i ,h t ] is the sequence feature h of the learner's exercise record obtained by the GRU layer and the potential knowledge level of the learner i. t Splicing is performed; softmax(·) is an activation function that limits the output value to between 0 and 1; Step 2-4: The personalized recommendation model uses cross entropy as the loss function to train and update the model. The calculation formula is as follows: Where M is the number of learners, p i is the true probability distribution of learner i choosing an exercise at the next moment, q i The predicted probability distribution of the exercise selected by learner i at the next moment is given by the personalized recommendation model; The cross entropy loss function is an indicator to measure the difference between the true probability distribution p and the model predicted probability distribution q; Step 2-5: Sort the probability of learner i selecting each exercise calculated by the personalized recommendation model in descending order, and take the top K exercises to form an exercise recommendation list and recommend it to learner i.

5. The personalized exercise recommendation method based on reinforcement learning according to claim 4, characterized in that: The specific process of modifying the learner's exercise record in step 3 is as follows: Step 3-1: Model initialization, including initializing the parameters of the prediction Q network and the target Q network; initializing the experience replay pool with a capacity of N; initializing the learner's modified exercise record set Learner number i = 1, time t = 0; Step 3-2: Get the exercise record of learner i and initial state s0; Step 3-3: Change the state s t Eigenvector φ(s t ) as the input of the prediction Q network to obtain the Q value corresponding to the action in the current state; Step 3-4: Use the ε-greedy strategy to select action a in the current Q value t ; Step 3-5: If a t =0, then delete in Step 3-6: In state s t Execute the current action a t , get the next state s t+1 and reward r t ; Step 3-7: Replace {s t ,a t ,r t ,s t+1 }This quadruple is stored in the experience replay pool; Step 3-8: Update status s t =s t+1 ; Step 3-9: Sample m samples from the experience replay pool {s j ,a j ,r j ,s j+1 }, j = 1, 2, ..., m, calculate the current target Q value y j : Step 3-10: Use mean square error loss function To update the parameters of the prediction Q network; Step 3-11: Update the parameters of the target Q network after every C steps, and the parameter values are the parameter values of the current predicted Q network; Step 3-12: Determine whether the time reaches the set value T; if not, return to step 3-3; if it reaches, execute the next step; Step 3-13: Record the modified exercises i Recorded as Will Add to the learner's revised exercise record Step 3-14: Determine whether all learners' exercise records have been modified. If not, return to step 3-2 and continue to modify the next learner's exercise record. If yes, end the step.

6. The personalized exercise recommendation method based on reinforcement learning according to claim 5, characterized in that: The process of the joint training in step 4 is specifically as follows: Step 4-1: Initialize the parameters α=α0 of the personalized recommendation model, β=β0 of the knowledge tracking model, and θ=θ0 of the reinforcement learning module; Step 4-2: Utilize learner practice records Train the knowledge tracking model; Step 4-3: Utilize learner exercise records and knowledge tracking models to train personalized recommendation models; Step 4-4: Fix the parameters α=α1 of the personalized recommendation model and β=β1 of the knowledge tracking model, and pre-train the reinforcement learning module. The specific method is: Step 4-4-1: Reinforcement learning algorithm in exercise record Select an action on the screen; Step 4-4-2: Calculate the reward function Reward based on the selected action; Step 4-4-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm; Step 4-4-4: Loop through steps 4-4-1 to 4-4-3 until all exercise records are completed. The cycle is complete; Step 4-4-5: Repeat steps 4-4-1 to 4-4-4 until the parameters of the reinforcement learning module are optimized; Step 4-5: Fix the knowledge tracking parameter β = β1 and jointly train the personalized recommendation model and reinforcement learning module. The specific method is: Step 4-5-1: Reinforcement learning algorithm in exercise record Select an action on the screen; Step 4-5-2: Calculate the reward function Reward based on the selected action; Step 4-5-3: Update the parameters of the reinforcement learning module according to the loss function of the Deep Q-Learning algorithm; Step 4-5-4: Repeat steps 4-5-1 to 4-5-3 until all The cycle is complete; Step 4-5-5: Update the parameters of the recommendation model according to the loss function of the recommendation model; Step 4-5-6: Repeat steps 4-5-1 to 4-5-5 until the personalized recommendation model and reinforcement learning module parameters are optimized.

Images