A personalized learning path recommendation method based on reinforcement learning

Abstract

The application provides a kind of personalized learning path recommendation method based on reinforcement learning, it is related to educational data mining technical field.The method first constructs learner simulator according to the learning record of scholar, the simulator can judge the learning level of learner;Then the knowledge relationship graph between the knowledge points contained in the exercise is automatically constructed by the concept map automatic construction model based on text classification and association rule mining;Based on the knowledge relationship graph and the cognitive diagnosis model, an exercise navigation module is designed to select potential candidate exercises;After the reinforcement learning agent selects the action in the action space, the state transition is determined in the state space, the model parameters are updated according to the loss function and optimization strategy, and the reinforcement learning model is optimized;Finally, the designed reinforcement learning model is used to recommend exercises for learners, and the reinforcement learning model parameters are updated according to the learning situation of learners.The method can recommend efficient and reasonable learning path for learners.

Description

Technical Field

[0001] This invention relates to the field of educational data mining technology, and in particular to a personalized learning path recommendation method based on reinforcement learning. Background Technology

[0002] The development of emerging information and communication technologies (ICT) such as mobile communications, the Internet, the Internet of Things, cloud computing, big data, and artificial intelligence is profoundly changing the way people think, produce, live, and learn. Therefore, combining education with emerging technologies such as artificial intelligence and big data is an inevitable trend.

[0003] In recent years, the rapid development of internet technology has driven profound changes in education methods. Online learning, as a new learning model, has provided the possibility of personalized learning, and massive open online courses (MOOCs) have attracted a large number of learners due to their free access, openness, and high-quality learning resources. Compared with the construction of traditional teaching environments, the new education system places higher demands on the construction of the next-generation teaching environment. This is mainly reflected in the ubiquitous nature of teaching requiring an integrated intelligent teaching space, the complexity of cognition requiring more effective teaching interaction, and the differentiation of learning requiring personalized teaching supply. In the new generation of internet-based learning environments, there are more opportunities for interaction between learners and teachers, more flexible learning methods, and richer learning resources. In the new education system, learners have the opportunity to do only the exercises they need to complete, instead of doing monotonous assignments, and educators have the opportunity to truly teach according to aptitude and individual needs.

[0004] However, while granting learners a high degree of freedom, the platform also reduces guidance, leading to a lack of reasonable learning paths and resulting in problems such as "information overload" and "knowledge disorientation." This manifests primarily in learners blindly browsing and studying resources, a lack of logical structure in the curriculum, and an increased risk of learning failure. The problems of aimless learning, declining learning quality, and reduced participation that gradually emerge during the course have prompted reflection among educators and researchers.

[0005] A feasible learning path recommendation algorithm can significantly improve learners' learning efficiency. To achieve this, the following three major issues need to be considered:

[0006] First, how to assess a learner's knowledge level?

[0007] In traditional teaching models, learners' knowledge levels are typically assessed by professionally trained educators who then provide appropriate guidance. However, online learning platforms lack expert assessment of learners' knowledge levels. Therefore, determining a learner's knowledge level through their learning records is our primary concern.

[0008] Second, how to uncover the relationships between knowledge points.

[0009] The main difference between learning path recommendation and practice question recommendation is that learning path recommendation recommends a sequence of learning activities for learners. In educational theory, researchers generally believe that the learning sequence has a significant impact on learners' learning efficiency; this learning sequence refers to the order between knowledge points. For example, recommending a learner with two linear equations to study a linear equation in two variables when the learner is not yet familiar with linear equations in one variable will clearly not yield any learning results. Therefore, studying the relationships between knowledge points is fundamental to ensuring the effectiveness of learning paths. Thus, how to accurately extract the relationships between knowledge points from learning materials and learners' practice sequences is a necessary issue we need to consider.

[0010] Third, how to construct a scientific and efficient personalized learning path recommendation algorithm.

[0011] Existing personalized learning path recommendation methods, such as treating personalized learning path recommendation as a sequence recommendation problem and using traditional sequence recommendation algorithms, fail to consider the learner's own knowledge mastery and cannot effectively integrate with the learner's individual situation. While the recommended results may offer some help, they cannot guarantee maximum improvement. Other studies, although incorporating the learner's learning situation into personalized learning paths, neglect the knowledge relevance between recommended exercises, significantly reducing the scientific validity and effectiveness of the recommended paths. Therefore, designing a personalized learning path recommendation algorithm that considers both the learner's individual situation and the relationships between knowledge points is a crucial issue we need to address.

[0012] Combining reinforcement learning algorithms to address problems encountered in online education is a current research hotspot in educational data mining. Reinforcement learning-based learning path recommendation models, which integrate learner learning levels and the relationships between knowledge points, can effectively provide each learner with the most suitable learning sequence, offering guidance and effectively solving the problems of "information overload" and "knowledge disorientation" in online education. Utilizing reinforcement learning for path recommendation in online education is a good way to effectively improve learners' learning efficiency. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a personalized learning path recommendation method based on reinforcement learning, which recommends efficient and reasonable learning paths for learners, in order to solve the problem of poor learning outcomes in online education due to a lack of effective guidance.

[0014] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a personalized learning path recommendation method based on reinforcement learning, comprising the following steps:

[0015] Step 1: Construct a learner simulator based on the learners' learning records. This simulator can determine the learners' learning level, with the aim of simulating dynamic students from a static dataset.

[0016] Step 1.1: Train the DKT model using the learner's learning records. This model is a knowledge tracking model based on deep learning. It can determine whether the student can do a certain exercise correctly in the next moment by using the learner's learning records. Use the model to judge the student's performance on the recommended learning path.

[0017] Step 1.2: Train the KPT model using the knowledge points contained in the learner's learning records and exercises. This model is a cognitive diagnostic model that combines the forgetting curve and the learning curve. The model judges the learner's mastery of each knowledge point and takes into account the student's learning and forgetting factors, making the judgment of the learner's knowledge level more accurate. Use the model to judge the learner's mastery of knowledge points before and after learning, and use the difference in the mastery of knowledge points before and after learning to judge the quality of the learning path.

[0018] Step 2: Automatically construct a knowledge relationship graph between the knowledge points contained in the practice questions by using a concept graph construction model based on text classification and association rule mining;

[0019] Step 2.1: Analyze and classify the practice question texts to obtain a matrix of knowledge points for the practice questions;

[0020] The exercise text is segmented into words. After segmentation and stop word filtering, the exercise text is represented as Q = (Q1, Q2, ..., Q...). j Q m ), where m represents the number of practice questions, Q j This represents the j-th practice question;

[0021] The TF-IDF method is chosen to extract the text features of the practice questions. The text features corresponding to practice question Q after TF-IDF extraction are represented as W = (W1, W2, ..., W...). j ,…,W m ), where W jW represents the textual features of the j-th exercise. j =(W j1 W j2 ,…,W jk , ..., W jr ), W jr Let r represent the weight of the k-th feature term of the j-th exercise question, where r is the dimension of the exercise question text features;

[0022] The formula for calculating the weights of text feature terms in the practice questions is as follows:

[0023] W jk =TF j，k ×IDF ik

[0024] Among them, TF j，k IDF represents the word frequency of the feature term of the k-th exercise text within the j-th exercise text. k The frequency of the k-th exercise text feature term is also called the inverse document frequency (IVF). The word frequency is directly proportional to the weight of the feature term, while the IVF is inversely proportional to the weight of the feature term.

[0025] After obtaining the text features W of the practice questions, the Adaboosting classification model is used to classify the text features, and each category is regarded as a knowledge point, resulting in the knowledge point matrix QC of the practice questions, as shown in the following formula:

[0026]

[0027] Among them, qc mk A value of 1 indicates that exercise m contains knowledge point k, qc mk A value of 0 indicates that exercise m does not contain knowledge point k;

[0028] Step 2.2: A knowledge relationship graph between the knowledge points contained in the exercises is constructed using an association rule mining method based on the Apriori algorithm. Learner learning record data is used to mine association rules between exercises. Combined with the QC matrix of exercise knowledge points obtained in the text classification stage, the association rules between exercises are mapped to association rules between concepts based on the relevant rules between exercises and the knowledge points contained in the exercises, thus achieving automatic construction of the concept graph. Step 2.2.1: Learner learning records are digitized into a performance matrix G, as shown in the following formula:

[0029]

[0030] Where m is the total number of practice questions, n is the total number of students, and when s... i Answer the practice questions correctly. j At that time, g ijThe value is 1 when student s i Answering the wrong practice question e j At that time, g ij The value is 0;

[0031] Calculate the consistency between two exercises; consistency between exercises refers to the number of times a learner answers both exercises correctly or incorrectly at the same time, as shown in the following formula:

[0032]

[0033] Where Count(Q) a Q b ) represents exercise Q a Q b Consistency between learners, where n is the total number of learners, and ⊙ represents the XOR operation, g is valid only if learner i answers both questions correctly or incorrectly at the same time. ai ⊙g bi The value is 1 only if it is true, otherwise it is 0;

[0034] When Count(Q) a Q b If n < n × 40%, it indicates that the relationship between the two questions is weak, and the relationship between the two questions will not be considered in the following steps;

[0035] Step 2.2.2: Combine the QC matrix of practice questions and the G matrix of scores to mine the association rules between practice questions and between knowledge points;

[0036] (1) Discovering association rules among practice questions

[0037] Consider the following four scenarios for the association rules of practice questions: The learner correctly answered practice question Q. a Then, at the same time, they correctly answered practice question Q. b The learner answered exercise Q correctly. b Then, at the same time, they correctly answered practice question Q. a The learner answered exercise question Q incorrectly. a Then, at the same time, they answered exercise question Q incorrectly. b The learner answered exercise question Q incorrectly. b Then, at the same time, he answered exercise question Q incorrectly. a The association rules for these four types of practice questions are summarized into two cases: from correct answer to correct answer, and from incorrect answer to incorrect answer; then the confidence of the association rules between practice questions is calculated for each of these two cases.

[0038] The confidence score of the association rule between practice questions is calculated using the following formula:

[0039]

[0040] Among them, Conf(Q a →Q b ) represents Q a →Q b The confidence level, Sup(Q) a Q b Sup(Q) represents the support between practice questions. a ) represents practice question Q a The support level; when calculating the confidence level from a correct answer to a correct answer, Sup(Q) a ) represents exercise Q a The number of times Sup(Q) is answered correctly a Q b ) represents exercise Q a And Exercise Q b The number of times both answers are correct; when calculating the confidence level from incorrect to incorrect answer, Sup(Q) a ) represents exercise Q a The number of times Sup(Q) was answered incorrectly a Q b ) represents exercise Q a And Exercise Q b The number of times it was done incorrectly at the same time;

[0041] (2) Construct a new matrix of knowledge points for practice questions

[0042] By inputting the exercise knowledge point matrix QC and the learner's learning record into the cognitive diagnostic model KPT of the student simulator, a new exercise knowledge point matrix QC' is obtained.

[0043] (3) Mining association rules between knowledge points

[0044] Combining the new practice question knowledge point matrix QC', the association rules between practice questions are mapped to association rules between knowledge points using the following formula:

[0045]

[0046] Among them, K i Exercise Q a Includes knowledge points, K j Exercise Q b The knowledge points included; q ai The knowledge point matrix Q, annotated by experts, is given; v bj The knowledge point matrix V, calculated by the learner simulator, is given; to remove unnecessary correlations between knowledge points, a threshold Min is set for the relevance of knowledge points. Rev ,when Therefore, it is assumed that there is no connection between these two knowledge points;

[0047] Step 3: Design a practice question navigation module based on the knowledge relationship graph and cognitive diagnostic model, and select potential candidate practice questions in the navigation module; the practice question navigation module includes a knowledge point selection pool and a practice question selection pool;

[0048] The process of selecting potential candidate practice questions in the navigation module is as follows:

[0049] Step S1: When a learner wants to learn a certain target knowledge point, according to the knowledge relationship diagram, the preorder node of the target knowledge point is put into the knowledge point selection pool. Then, the preorder nodes of the knowledge points in the knowledge point selection pool are put into the knowledge point selection pool. This process is repeated until there are no more preorder nodes of the knowledge points in the knowledge point selection pool.

[0050] Step S2: Randomly select a knowledge point from the knowledge point selection pool, remove it from the knowledge point selection pool, and put the practice questions containing that knowledge point into the practice question selection pool;

[0051] Step S3: Use a reinforcement learning model to select practice questions to learn from the practice question selection pool. After the learner completes the practice question, the learner's mastery of the knowledge point is judged based on the learner simulator. If the mastery level is greater than the set threshold, it is determined that the learner has mastered the knowledge point and does not need to continue learning the knowledge point. Practice questions containing the knowledge point are deleted from the practice question pool.

[0052] Step S4: Repeat steps S1-S3 until one of the following two termination conditions is met: termination condition 1 is that the learner has learned the practice questions containing the target knowledge points, and termination condition 2 is that the length of the learning path has reached the specified length limit.

[0053] Step 4: After the reinforcement learning agent selects an action in the action space, it determines the state transition in the state space, updates the model parameters according to the loss function and optimization strategy, and optimizes the reinforcement learning model.

[0054] Step 4.1: Design the action space for reinforcement learning; the action taken by the reinforcement learning agent is represented as the next exercise to be selected by the learner; the action space is generated by the exercise navigation module in Step 3 and is defined by a single binary vector of length M;

[0055] The action is represented as follows:

[0056] A i = [a1, ..., a i , ..., a M ]

[0057] When selecting action A i At that time, a i The value is 1, and the rest are 0;

[0058] At the same time, in order to prevent students from repeatedly learning the same practice question, a restriction was set on the actions taken by the reinforcement learning agent, namely, the same action is only selected once.

[0059] Step 4.2: Design the state space for reinforcement learning; before recommending learning paths to learners, the KPT model is used to diagnose the learners' mastery of knowledge points, represented by a vector of length M with values ​​between 0 and 1; ignoring the learning order of recommended practice questions, a compact state space is constructed; the state representation is as follows:

[0060] S = [k1, k2, ..., k M e1, e2, ..., e N g1, g2, ..., g N ]

[0061] Where k1, k2, ..., k M These represent the learner's level of mastery of each knowledge point before learning, where M is the number of knowledge points; e1, e2, ..., e N Let g1, g2, ..., g be the number of practice problems, and N be the number of practice problems. N This indicates whether the learner answered the questions correctly or incorrectly; when the learner selects question i, e... i The value is 1 when the trainee answers e correctly. i At that time, g i The value is 1;

[0062] Step 4.3: Design the reward function for reinforcement learning; Based on the design goals of the reinforcement learning model, construct a reward function with multiple elements. This function prioritizes changes in the learner's mastery of knowledge points and includes the following factors that influence the learning path:

[0063] (1) Changes in learners' mastery of knowledge points:

[0064]

[0065] Where R1 represents the change in the degree of mastery of the knowledge point. k represents the learner's level of mastery of knowledge point i before learning. i This indicates the learner's level of mastery of knowledge point i after learning; the level of mastery of knowledge point i is given by the cognitive diagnostic model.

[0066] (2) Smoothness of practice questions

[0067] A smoothness factor is added to the loss function. There are many ways to design this factor. In this embodiment, the smoothness R2 of the exercise is represented by the squared difference in difficulty between two consecutive exercises.

[0068]

[0069] Where, d t+1 d indicates the difficulty of the next exercise in the learning path. t (3) Length of the learning path

[0070] Add a factor R3 related to the learning path length to the reward function:

[0071] R3 = -L

[0072] (4) Scholar participation

[0073] By incorporating an engagement factor R4 into the reward function, we ensure that, from the perspective of the entire learning path, learners neither find it too easy nor too difficult.

[0074]

[0075] in, This is the difficulty threshold, ranging from 0 to 1; considering the above four factors, the reward function is expressed as follows:

[0076] R = R1 + αR2 + βR3 + γR4

[0077] Where α, β, and γ are all penalty parameters, with values ​​ranging from 0 to 1;

[0078] Step 4.4: Use a proximal strategy to optimize the selection of the learning path;

[0079] Step 5: Use the reinforcement learning model designed in Step 4 to recommend exercises to learners, and update the reinforcement learning model parameters based on the learners' learning progress;

[0080] The reinforcement learning model designed in step 4 is used to recommend exercises to the learner. The recommended exercises are submitted to the knowledge tracking model in the learner simulator. The knowledge tracking model determines whether the learner can answer the exercise correctly. The reinforcement learning model updates the state of the RL agent based on the judgment of the cognitive diagnostic model and continues to recommend exercises to the learner until the termination condition is met. The generated learning path is submitted to the cognitive diagnostic model in the learner simulator. The cognitive diagnostic model determines how much the learner's knowledge level has improved before and after learning and passes this information to the reinforcement learning model. The reinforcement learning model updates its model parameters based on this information.

[0081] The beneficial effects of adopting the above technical solution are as follows: The present invention provides a personalized learning path recommendation method based on reinforcement learning. This method recommends personalized learning paths based on reinforcement learning and combined with cognitive diagnostic models and knowledge tracing models. It considers the learner's learning goals, knowledge level, and the relationship between knowledge points, as well as the smoothness of the learning path and the learner's participation. It recommends efficient and reasonable learning paths for learners to solve the problem of poor learning outcomes in online education due to a lack of effective guidance. Attached Figure Description

[0082] Figure 1 A model structure diagram of a personalized learning path recommendation method based on reinforcement learning provided in this embodiment of the invention;

[0083] Figure 2 A schematic diagram of the structure of the knowledge tracking model provided in this embodiment of the invention;

[0084] Figure 3 A schematic diagram of the structure of the cognitive diagnostic model provided in this embodiment of the invention;

[0085] Figure 4 The present invention provides a structural diagram for automatically constructing a concept map model.

[0086] Figure 5 A schematic diagram of the PPO model structure provided in this embodiment of the invention. Detailed Implementation

[0087] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0088] In this embodiment, a personalized learning path recommendation method based on reinforcement learning is described, such as... Figure 1 As shown, it includes the following steps:

[0089] Step 1: Construct a learner simulator based on the learners' learning records. This simulator can determine the learners' learning level, with the aim of simulating dynamic students from a static dataset.

[0090] Step 1.1: Train the DKT (deep knowledge model) model using the learner's learning records. This model is a knowledge tracking model based on deep learning. It can determine whether the student can do a certain exercise correctly in the next moment by using the learner's learning records. Use the model to judge the student's performance on the recommended learning path.

[0091] The structure of the DKT model is as follows: Figure 2As shown, the DKT model first generates a one-hot vector by one-hot encoding the learner's historical scores, inputs the one-hot vector into the LSTM network, extracts features through the input layer, inputs these features into the hidden layer, and then outputs the prediction result from the output layer, which is the learner's score for the next question.

[0092] The formulas for the input gate, forget gate, forget gate memory, output gate, long memory, and short memory in an LSTM network are as follows:

[0093] i t =σ(W i ·[h t-1 x t ]+b i )

[0094] f t =σ(W f ·[h t-1 x t ]+b f )

[0095] o t =σ(W o ·[h t-1 x t ]+b o )

[0096]

[0097]

[0098] h t =o t *tanh(C t )

[0099] Step 1.2: Train the KPT (Knowledge Proficiency Tracing) model using the knowledge points contained in the learners' learning records and exercises. This model is a cognitive diagnostic model that combines the forgetting curve and the learning curve. This model judges the learner's mastery of each knowledge point and takes into account the student's learning and forgetting factors, making the judgment of the learner's knowledge level more accurate. Use this model to judge the learner's mastery of knowledge points before and after learning, and use the difference in the mastery of knowledge points before and after learning to judge the quality of the learning path.

[0100] The structure of the KPT model is as follows: Figure 3As shown, the KPT model tracks learners' knowledge proficiency using educational priors, making it an interpretable probabilistic knowledge tracking model. KPT first associates each exercise with a knowledge vector, where each element represents an explicit knowledge point. The Q-matrix, describing the relationship between exercises and knowledge points, is labeled by educational experts. To track learners' knowledge proficiency, relevant information for each learner is embedded into the same knowledge space. KPT also considers the changes in learners' knowledge proficiency over time, incorporating the memory curve and forgetting curve from traditional educational theory into its modeling.

[0101] The KPT model employs a phased feedback user interaction modeling approach. Given learners' practice records and expert-annotated Q-matrices, KPT uses the Q-matrices to map each learner's latent vectors into the knowledge space and combines learning and forgetting curves to determine the learner's mastery of knowledge points. In the prediction phase, the KPT model predicts the learner's performance R on exercises within a T+1 time window. T+1 and mastery of knowledge points U T+1 .

[0102] The KPT model represents the practice representation tensor R in the following form:

[0103]

[0104] Where N(·) represents a normal distribution, This represents the variance of the normal distribution that R follows. V represents the degree of knowledge mastery of learner i within the time window t. j This indicates the knowledge points included in exercise j. express and V j dot product, Indicates whether student i answered question j correctly within time window t (1 indicates correct answer, 0 indicates incorrect answer).

[0105] The V matrix represents the knowledge points included in the exercises, and is calculated using the following formula:

[0106]

[0107] Among them, D T It is a set obtained through the Q matrix. Each element (j, q, p) in the set represents a problem j that contains knowledge point q but not knowledge point p. D T The calculation formula is shown below:

[0108] Satisfying Q jq =1 and Q jp =1

[0109] In pedagogy, there are two classic theories regarding the relationship between learners' knowledge acquisition and time: the forgetting curve and the retention curve. The forgetting curve shows that the more a learner studies a particular knowledge point, the more firmly they grasp it. The retention curve shows that a learner's knowledge acquisition declines over time. The KPT model considers both theories, and the calculation formulas are shown below:

[0110]

[0111]

[0112] Among them, in l t (*) Used to capture how well students improve their knowledge through continuous practice. t (*) indicates how knowledge proficiency decays over time. This represents the frequency with which students practice knowledge point k within time window t. r and D are two hyperparameters used to control the scale of knowledge mastery improvement. Δt represents the time interval between the current moment and the t-1 time window, and S is a hyperparameter representing memory intensity.

[0113] KPT models the U matrix as shown in the following formula:

[0114] in

[0115] Satisfying 0≤α i ≤1

[0116] Where, α i This is used to balance forgetting and memory factors, while also capturing students' learning abilities.

[0117] Since the learner's initial knowledge level is unknown during the t=1 time window, the KPT model assumes that it follows a zero-mean Gaussian distribution. The tensor U, which ultimately represents the learner's mastery of knowledge points, can be calculated by the following formula.

[0118]

[0119] The optimization objective of the KPT model is shown in the following equation:

[0120]

[0121] in, The learner performance matrix R indicates that it follows a normal distribution.

[0122] The variance of the cloth, The variance of the learner's knowledge mastery matrix U represents the normality of the distribution. This represents the variance of U1 following a normal distribution. The variance of the knowledge point matrix V represents the normal distribution. The learner i's knowledge point mastery level within the time window t is updated based on the objective function. Exercise j contains knowledge point V j .

[0123] Based on learner i's mastery of knowledge point k at time t, learner i's mastery of knowledge point k at the next time step can be calculated, as shown in the following formula:

[0124]

[0125] Learner i's performance on exercise j at time t+1 is shown in the following formula:

[0126]

[0127] Step 2: By means of Figure 4 The concept graph automatic construction model shown here, based on text classification and association rule mining, automatically constructs a knowledge relationship graph between the knowledge points contained in the exercise questions.

[0128] Knowledge relationship graphs, composed of knowledge points and edges describing the relationships between them, play a crucial role in providing personalized learning guidance and assessing students' knowledge levels of various concepts. This paper presents an automatic concept graph construction model based on text classification and association rule mining. This model uses text classification technology to replace expert experience, automatically matching concept labels to practice questions. Combined with association rule methods, it mines the connections between knowledge points, ultimately achieving automatic construction of knowledge relationship graphs.

[0129] Step 2.1: Analyze and classify the practice question texts to obtain a matrix of knowledge points for the practice questions;

[0130] The exercise text is segmented into words. After segmentation and stop word filtering, the exercise text is represented as Q = (Q1, Q2, ..., Q...). j Q m ), where m represents the number of practice questions, Q j This represents the j-th practice problem;

[0131] The TF-IDF method is chosen to extract the text features of the practice questions. The text features corresponding to practice question Q after TF-IDF extraction are represented as W = (W1, W2, ..., W...). j ,…,W m ), where W j W represents the textual features of the j-th exercise. j =(Wj1 W j2 ,…,W jk , ..., W jr ), W jr Let r represent the weight of the k-th feature term of the j-th exercise question, where r is the dimension of the exercise question text features;

[0132] The formula for calculating the weights of text feature terms in the practice questions is as follows:

[0133] W jk =TF j，k ×IDF ik

[0134] Among them, TF j，k IDF represents the word frequency of the feature term of the k-th exercise text within the j-th exercise text. k The frequency of the k-th exercise text feature term is also called the inverse document frequency (IVF). The word frequency is directly proportional to the weight of the feature term, while the IVF is inversely proportional to the weight of the feature term.

[0135] After obtaining the text features W of the practice questions, the Adaboosting classification model is used to classify the text features, and each category is regarded as a knowledge point, resulting in the knowledge point matrix QC of the practice questions, as shown in the following formula:

[0136]

[0137] Among them, qc mk A value of 1 indicates that exercise m contains knowledge point k, qc mk A value of 0 indicates that exercise m does not contain knowledge point k;

[0138] Step 2.2: After obtaining the QC (Knowledge Point Matrix) of the practice questions, an association rule mining method based on the Apriori algorithm is used to construct a knowledge relationship graph between the knowledge points contained in the practice questions. Learner learning record data is used to mine the association rules between practice questions. Combined with the QC obtained in the text classification stage, the association rules between practice questions are mapped to the association rules between concepts according to the relevant rules between the practice questions and the knowledge points contained in the practice questions, thereby realizing the automatic construction of the concept graph.

[0139] Step 2.2.1: Before mining association rules, learners' learning records need to be digitized into a performance matrix G, as shown in the following formula:

[0140]

[0141] Where m is the total number of practice questions, n is the total number of students, and when s... i Answer the practice questions correctly. jAt that time, g ij The value is 1 when student s i Answering the wrong practice question e j At that time, g ij The value is 0;

[0142] Calculate the consistency between two exercises; consistency between exercises refers to the number of times a learner answers both exercises correctly or incorrectly at the same time, as shown in the following formula:

[0143]

[0144] Where Count(Q) a Q b ) represents exercise Q a Q b Consistency between learners, where n is the total number of learners, and ⊙ represents the XOR operation, g is valid only if learner i answers both questions correctly or incorrectly at the same time. ai ⊙g bi The value is 1 only if it is true, otherwise it is 0;

[0145] When Count(Q) a Q b If n < n × 40%, it indicates that the relationship between the two questions is weak, and the relationship between the two questions will not be considered in the following steps;

[0146] Step 2.2.2: Combine the QC matrix of practice questions and the G matrix of scores to mine the association rules between practice questions and between knowledge points;

[0147] (1) Discovering association rules among practice questions

[0148] Consider the following four scenarios for the association rules of practice questions: The learner correctly answered practice question Q. a Then, at the same time, they correctly answered practice question Q. b The learner answered exercise Q correctly. b Then, they correctly answered practice question Q at the same time. a The learner answered exercise question Q incorrectly. a Then, at the same time, they answered exercise question Q incorrectly. b The learner answered exercise question Q incorrectly. b Then, at the same time, he answered exercise question Q incorrectly. a That is, from correct to correct and from incorrect to incorrect; therefore, the association rules for these four types of exercises are summarized into two cases: from correct answer to correct answer and from incorrect answer to incorrect answer; then the confidence of the association rules between exercises in these two cases is calculated respectively.

[0149] The confidence score of the association rule between practice questions is calculated using the following formula:

[0150]

[0151] Among them, Conf(Q) a →Q b ) represents Q a →Q b The confidence level, Sup(Q) a Q b Sup(Q) represents the support between practice questions. a ) represents practice question Q a The support level; when calculating the confidence level from a correct answer to a correct answer, Sup(Q) a ) represents exercise Q a The number of times Sup(Q) is answered correctly a Q b ) represents exercise Q a And Exercise Q b The number of times both answers are correct; when calculating the confidence level from incorrect to incorrect answer, Sup(Q) a ) represents exercise Q a The number of times Sup(Q) was answered incorrectly a Q b ) represents exercise Q a And Exercise Q b The number of times it was done incorrectly at the same time;

[0152] (2) Construct a new matrix of knowledge points for practice questions

[0153] The exercise knowledge point matrix QC obtained in the exercise text analysis stage and the learner's learning record are put into the cognitive diagnostic model KPT of the student simulator to obtain a new exercise knowledge point matrix QC′. In the following steps, the new exercise-concept matrix QC′ will replace the original exercise-concept matrix QC in the calculation.

[0154] (3) Mining the association rules between knowledge points

[0155] The degree of relevance between knowledge points indicates the strength of the connection between two concepts, which is reflected in the association rules between the two knowledge points. Using the new practice question knowledge point matrix QC′, the association rules between practice questions are mapped to the association rules between knowledge points using the following formula:

[0156]

[0157] Among them, K i Exercise Q a Includes knowledge points, K j Representing Exercise Q b The knowledge points included; q aiThe knowledge point matrix Q, annotated by experts, is given; v bj The knowledge point matrix V, calculated by the learner simulator, is given; to remove unnecessary correlations between knowledge points, a threshold Min is set for the relevance of knowledge points. Rev ,when If no relationship exists between the two knowledge points, then it is assumed that there is no connection between them. In this embodiment, the algorithm for constructing the knowledge relationship graph is shown in Table 1.

[0158]

[0159] Step 3: Design a practice question navigation module based on the knowledge relationship graph and cognitive diagnostic model, and select potential candidate practice questions in the navigation module; the practice question navigation module includes a knowledge point selection pool and a practice question selection pool; selecting actions according to the navigation module can not only ensure the logical rationality of the learning path, but also reduce the action space for the reinforcement learning algorithm, thereby enabling the reinforcement learning algorithm to converge faster and better.

[0160] The process of selecting potential candidate practice questions in the navigation module is as follows:

[0161] Step S1: When a learner wants to learn a certain target knowledge point, according to the knowledge relationship diagram, the preorder node of the target knowledge point is put into the knowledge point selection pool. Then, the preorder nodes of the knowledge points in the knowledge point selection pool are put into the knowledge point selection pool. This process is repeated until there are no more preorder nodes of the knowledge points in the knowledge point selection pool.

[0162] Step S2: Randomly select a knowledge point from the knowledge point selection pool, remove it from the knowledge point selection pool, and put the practice questions containing that knowledge point into the practice question selection pool;

[0163] Step S3: Use the reinforcement learning model (PPO model) to select the practice questions to be learned from the practice question selection pool. After the learner completes the practice question, the learner's mastery of the knowledge point is judged according to the learner simulator. If the mastery level is greater than the set threshold of 0.6, it is determined that the learner has mastered the knowledge point and does not need to continue learning the knowledge point. The practice questions containing the knowledge point are deleted from the practice question pool.

[0164] Step S4: Repeat steps S1-S3 until one of the following two termination conditions is met: termination condition 1 is that the learner has learned the practice questions containing the target knowledge points, and termination condition 2 is that the length of the learning path has reached the specified length limit.

[0165] Step 4: After selecting an action in the action space, the reinforcement learning agent determines the state transition in the state space, updates the model parameters according to the loss function and optimization strategy, and optimizes the reinforcement learning model; the interaction between the agent and the environment in reinforcement learning is as follows: Figure 5 As shown, the interaction process can be summarized as follows: the agent selects an action in the current state, the environment calculates the agent's state at the next moment based on the action selected by the agent, and gives the agent a reward value. The agent determines the quality of the selection based on the reward value and continues to select actions in the state at the next moment until the termination condition is met.

[0166] This method employs reinforcement learning algorithms and, following standard practices in developing reinforcement models, focuses on the four main components of reinforcement learning: action space, state space, loss function, and optimization strategy.

[0167] Step 4.1: Design the action space for reinforcement learning; Under our navigation strategy, the action taken by the reinforcement learning agent (referred to as RL agent) is represented as the next exercise selected by the learner; The action space is generated by the exercise navigation module in Step 3 and is defined by a single binary vector of length M (M is the number of exercises);

[0168] Actions are represented as follows:

[0169] A i =[a1,…,a i ,…,a M ]

[0170] When selecting action A i At that time, a i The value is 1, and the rest are 0;

[0171] To ensure the rationality of the learning path, we require actions to be selected from the practice question navigation module. At the same time, to prevent students from repeatedly learning the same practice question, we have imposed a restriction on the actions taken by the reinforcement learning agent: the same action can only be selected once.

[0172] Step 4.2: Designing the State Space for Reinforcement Learning; The most intuitive way to represent the state of a reinforcement learning agent (RL agent) during training is the learner's level of knowledge mastery. When the RL agent selects an action, the state transitions to the learner's level of knowledge mastery after learning that action. However, this requires diagnosis at every step in each round, which consumes a lot of time. Furthermore, the cognitive diagnosis model itself has errors, which slows down the RL agent's updates. Considering that the learner's level of knowledge mastery before learning, the selected practice questions, and the scores on those questions also help the RL agent infer the quality of the learning outcome, we use the learner's level of knowledge mastery before learning, the selected practice questions, and the scores on those questions as the core of our state space.

[0173] Before recommending learning paths to learners, we use the KPT model to diagnose learners' mastery of knowledge points, representing it as a vector of length M with values ​​between 0 and 1. Inspired by DKT, we attempted to combine the learned exercises and scores into a joint one-hot encoding, adding it to the state in the order of learning, hoping that the state could represent the order of practice. However, our simulations show that such a large state space makes RL inefficient during training. We also evaluated some other state representation methods, but they were ultimately abandoned due to a lack of practicality. Researcher Jonathan Bassen found that even ignoring the order of previous activities, the RL agent could still make effective decisions when constructing the state. Therefore, we adopted the idea of ​​this research, ignoring the learning order of recommended exercises, and constructed a compact state space. Experimental results show that this state space reduces training time while ensuring the effectiveness of the model.

[0174] The state is represented as follows:

[0175] S = [k1, k2, ..., k M e1, e2, ..., e N g1, g2, ..., g N ]

[0176] Where k1, k2, ..., k M These represent the learner's level of mastery of each knowledge point before learning, where M is the number of knowledge points; e1, e2, ..., e N Let g1, g2, ..., g be the number of practice problems, and N be the number of practice problems. N This indicates whether the learner answered the questions correctly or incorrectly; when the learner selects question i, e... i The value is 1 when the trainee answers e correctly. i At that time, g i The value is 1;

[0177] Step 4.3: Design the reward function for reinforcement learning; Based on the design goals of the reinforcement learning model, construct a reward function with multiple elements. This function prioritizes changes in the learner's mastery of knowledge points and includes the following factors that influence the learning path:

[0178] (1) Changes in learners' mastery of knowledge points:

[0179]

[0180] Where R1 represents the change in the degree of mastery of the knowledge point. k represents the learner's level of mastery of knowledge point i before learning. iThis indicates the learner's level of mastery of knowledge point i after learning; the level of mastery of knowledge point i is given by the cognitive diagnostic model.

[0181] (2) Smoothness of practice questions

[0182] Research in educational psychology shows that learning is a continuous process, and the difficulty of consecutively recommended practice questions should not vary significantly. When the difference is too large, student performance will deteriorate. Therefore, we add a smoothness factor to the loss function. There are many ways to design this factor; in this embodiment, the smoothness R² of the practice questions is represented by the squared difference in difficulty between two consecutive questions.

[0183]

[0184] Where, d t+1 d indicates the difficulty of the next exercise in the learning path. t R represents the difficulty of the previous exercise, and L represents the length of the learning path; since we want to maximize the reward function, we negative R2.

[0185] (3) Length of the learning path

[0186] Previous researchers have found that the length of the learning path affects student learning efficiency; when the recommended learning path is too long, learners tend to give up. Furthermore, because we use a reinforcement learning algorithm, without certain constraints, the RL agent may recommend an excessively long practice sequence to maximize learners' mastery of knowledge points, thus increasing their learning burden. This contradicts our model's design intent. Therefore, we incorporate a factor R3 related to the learning path length into the reward function.

[0187] R3 = -L

[0188] (4) Scholar participation

[0189] In educational psychology research, learner engagement significantly impacts learning efficiency. Therefore, ensuring high student participation during the learning process is crucial; however, modeling this factor presents a significant challenge. We have identified two major factors influencing learner engagement from educational psychology research: exercises that are too easy and exercises that are too difficult. When exercises are too easy, learners often find them unchallenging and are reluctant to spend time studying them. Conversely, when exercises are too difficult, learners often experience negative emotions such as giving up due to their inability to learn. Therefore, we incorporate an engagement factor R4 into the reward function to ensure that, throughout the learning path, learners neither perceive the learning as too easy nor too difficult.

[0190]

[0191] in, This is the difficulty threshold, which ranges from 0 to 1; we also make R4 negative to maximize the reward function.

[0192] Considering the four factors above, the reward function is expressed as follows:

[0193] R = R1 + αR2 + βR3 + γR4

[0194] Among them, α, β, and γ are all penalty parameters, with values ​​ranging from 0 to 1. If we want to increase the influence of a certain factor, we can increase the corresponding parameter.

[0195] Step 4.4: Use a proximal strategy to optimize the selection of the learning path;

[0196] Reinforcement learning algorithms, which can learn actively, make decisions in real time based on interactions with the environment, but often take a long time to converge. This is because the training and sample efficiency of reinforcement learning algorithms is often very low; typically, it requires thousands of samples and hundreds of iterations to learn a simple policy, or hundreds of thousands of samples and tens of thousands of iterations to learn a more complex policy. We overcome this challenge faced by Reinforcement Learning (RS) by using Proximal Policy Optimization (PPO), a policy gradient method that leverages deep neural networks to learn scheduling policies more efficiently. The use of neural networks reduces the dimensionality of the state / action space, thus reducing the number of samples the algorithm needs to converge.

[0197] PPO is an improved actor-critic algorithm, where the actor determines the action to take based on the environment state, and the critic estimates the expected reward from a given state. The actor and critic are represented by different neural networks and converge to a good policy using stochastic gradient descent (SGD). The RL agent can choose the optimal action based on past information or try other actions that might yield higher rewards. In PPO, the actor network outputs the probability of taking each action and samples recommended actions from this distribution; as training progresses, the actor network increases the probability of actions that yield higher rewards.

[0198] The actor network selects actions based on the current policy within the action space constructed based on the knowledge relationship graph, and updates the network parameters according to the advantage value provided by the critic. The calculation formula is shown in the following formula:

[0199] L actor (θ)=E t [min(r t (θ)A t ,clip(r t (θ), 1-ε, 1+ε)At )]

[0200] Where θ is a parameter in the actor network, E t L represents the expectation at time t. actor (θ) represents the loss function of the actor network, r t (θ) represents the ratio of the current policy to the previous policy, ε is a hyperparameter used to limit the magnitude of policy updates, typically taking values ​​of 0.1 or 0.2. The function clip(·) adjusts r. t The value of (θ) is restricted to the interval [1-ε, 1+ε];

[0201] Using clip(r) t The policy update magnitude constraint (θ), 1-ε, 1+ε) not only reduces the algorithm complexity but also improves its flexibility and stability. t The reward obtainable under the current action is given by the critic network; r t (θ) and A t The calculation formula is shown below:

[0202]

[0203] A t =γV φ (s t+1 )+R(a t s t )-V φ (s t )

[0204] Where, π θ (a t |s t ) indicates that under the current policy, in state s t Select action a at time t The probability, This indicates that under the policy in the previous time step, in state s t Select action a at time t The probability of V; φ (s t ) indicates that in state s t The potential profit value, R(a) t s t ) indicates that in state s t Select action a at time t The reward value that can be obtained is given by the reward function mentioned above, V. φ (s t+1 ) indicates the next state s t+1The potential future returns are determined by γ, which is a decay factor that reduces the potential future returns to encourage the algorithm to accumulate returns quickly in the current moment.

[0205] In reinforcement learning recommendation models, the critic network is used to determine whether a selected action is good or bad. The loss function of the critic network is L. critic (φ) is shown in the following formula:

[0206]

[0207] Among them, R(a) t ,s t It is calculated from the reward function;

[0208] Step 5: Use the reinforcement learning model designed in Step 4 to recommend exercises to learners, and update the reinforcement learning model parameters according to the learners' learning progress;

[0209] The reinforcement learning model designed in step 4 is used to recommend exercises to the learner. The recommended exercises are submitted to the knowledge tracking model in the learner simulator. The knowledge tracking model determines whether the learner can answer the exercise correctly. The reinforcement learning model updates the state of the RL agent based on the judgment of the cognitive diagnostic model and continues to recommend exercises to the learner until the termination condition is met. The generated learning path is submitted to the cognitive diagnostic model in the learner simulator. The cognitive diagnostic model determines how much the learner's knowledge level has improved before and after learning and passes this information to the reinforcement learning model. The reinforcement learning model updates its model parameters based on this information.

[0210] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A personalized learning path recommendation method based on reinforcement learning, characterized in that: Includes the following steps: Step 1: Construct a learner simulator based on the learners' learning records. This simulator can determine the learners' learning level, with the aim of simulating dynamic students from a static dataset. Step 2: Automatically construct a knowledge relationship graph between the knowledge points contained in the practice questions by using a concept graph construction model based on text classification and association rule mining; Step 3: Design a practice question navigation module based on the knowledge relationship graph and cognitive diagnostic model, and select potential candidate practice questions in the navigation module; the practice question navigation module includes a knowledge point selection pool and a practice question selection pool; Step 4: After the reinforcement learning agent selects an action in the action space, it determines the state transition in the state space, updates the model parameters according to the loss function and optimization strategy, and optimizes the reinforcement learning model. Step 4.1: Design the action space for reinforcement learning; the action taken by the reinforcement learning agent is represented as the next exercise to be selected by the learner; the action space is generated by the exercise navigation module in Step 3 and is defined by a single binary vector of length M; The action is represented as follows: ； When selecting an action hour, The value is 1, and the rest are 0; At the same time, in order to prevent students from repeatedly learning the same practice question, a restriction was set on the actions taken by the reinforcement learning agent, namely, the same action is only selected once. Step 4.2: Design the state space for reinforcement learning; before recommending learning paths to learners, the KPT model is used to diagnose the learners' mastery of knowledge points, represented by a vector of length M with values ​​between 0 and 1; ignoring the learning order of recommended practice questions, a compact state space is constructed; the state representation is as follows: ； in, These represent the learner's level of mastery of each knowledge point before learning, where M is the number of knowledge points. Let N represent the number of practice problems. This indicates the learner's correct or incorrect answers on the practice questions; when the learner selects practice questions... hour, The value is 1 when the trainee does it correctly. hour, The value is 1; Step 4.3: Design the reward function for reinforcement learning; Based on the design goals of the reinforcement learning model, construct a reward function with multiple elements. This function prioritizes changes in the learner's mastery of knowledge points and includes multiple factors influencing the learning path: Step 4.4: Use a proximal strategy to optimize the selection of the learning path; Step 5: Use the reinforcement learning model designed in Step 4 to recommend exercises to learners, and update the reinforcement learning model parameters based on the learners' learning progress.

2. The personalized learning path recommendation method based on reinforcement learning according to claim 1, characterized in that: Includes the following steps: Step 1.1: Train the DKT model using the learner's learning records. This model is a knowledge tracking model based on deep learning. It can determine whether the student can do a certain exercise correctly in the next moment by using the learner's learning records. Use the model to judge the student's performance on the recommended learning path. Step 1.2: Train the KPT model using the knowledge points contained in the learner's learning records and exercises. This model is a cognitive diagnostic model that combines the forgetting curve and the learning curve. The model judges the learner's mastery of each knowledge point and takes into account the student's learning and forgetting factors, making the judgment of the learner's knowledge level more accurate. Use the model to judge the learner's mastery of knowledge points before and after learning, and use the difference in the mastery of knowledge points before and after learning to judge the quality of the learning path.

3. The personalized learning path recommendation method based on reinforcement learning according to claim 2, characterized in that: Step 2 includes the following steps: Step 2.1: Analyze and classify the practice question texts to obtain a matrix of knowledge points for the practice questions; Step 2.2: The association rule mining method based on the Apriori algorithm is used to construct a knowledge relationship graph between the knowledge points contained in the exercises; the association rules between the exercises are mined using learner learning record data, and combined with the QC matrix of exercise knowledge points obtained in the text classification stage, the association rules between the exercises are mapped to the association rules between concepts according to the relevant rules between the exercises and the knowledge points contained in the exercises, so as to realize the automatic construction of the concept graph.

4. The personalized learning path recommendation method based on reinforcement learning according to claim 3, characterized in that: The specific method for step 2.1 is as follows: The exercise question text is segmented into words, and the resulting text after segmentation and stop word filtering is represented as follows: Where m represents the number of practice questions. Indicates the first One practice question; The TF-IDF method was chosen to extract the text features of the practice questions. The text features corresponding to practice question Q after TF-IDF extraction are represented as follows: ,in, Indicates the first Textual features of each exercise question , Indicates the first The first exercise question The weights of each feature term, where r is the dimension of the text features in the exercise questions; The formula for calculating the weights of text feature terms in the practice questions is as follows: ； in, Indicates the first The first exercise text in the text The word frequency of text features in each exercise question. Indicates the first The number of times a feature term appears in the entire exercise text dataset is also called the inverse document frequency. The term frequency is directly proportional to the weight of the feature term, while the inverse document frequency is inversely proportional to the weight of the feature term. Obtain the text features of the practice questions Then, the Adaboosting classification model was used to classify the text features, and each category was treated as a knowledge point, resulting in a knowledge point matrix for the practice questions. As shown in the formula below: ； in, A value of 1 indicates that the exercise... Includes knowledge points , A value of 0 indicates that the exercise... Does not include knowledge points .

5. The personalized learning path recommendation method based on reinforcement learning according to claim 4, characterized in that: The specific method for step 2.2 is as follows: Step 2.2.1: Digitize the learner's learning records into a performance matrix G, as shown in the following formula: ； Where m is the total number of practice questions, and n is the total number of students. Do the practice questions correctly hour, The value is 1 when the student Doing practice problems wrong hour, The value is 0; Calculate the consistency between two exercises; consistency between exercises refers to the number of times a learner answers both exercises correctly or incorrectly at the same time, as shown in the following formula: ； in, Representing exercises Consistency between them The total number of learners The XOR operation is only valid when the learner... You answered both questions correctly or incorrectly at the same time. The value is 1 only if it is true, otherwise it is 0; when If the relationship between the two questions is weak, then the following steps will not consider the relationship between the two questions. Step 2.2.2: Combine the knowledge point matrix QC and the score matrix G to mine the association rules between practice questions and between knowledge points; (1) Discovering association rules among practice questions; Consider the following four scenarios for the association rules of practice questions: The learner answered the practice questions correctly. Then they answered the practice questions correctly at the same time. The learner answered the practice questions correctly. Then they answered the practice questions correctly at the same time. The learner answered the practice questions incorrectly. Then, they answered the practice questions incorrectly at the same time. The learner answered the practice questions incorrectly. Then, at the same time, they answered the practice questions incorrectly. The association rules for these four types of practice questions are summarized into two cases: from correct answer to correct answer, and from incorrect answer to incorrect answer; then the confidence of the association rules between practice questions is calculated for each of these two cases. The confidence score of the association rule between practice questions is calculated using the following formula: ； in, express confidence level This indicates the level of support between practice questions. Exercises The support level; when calculating the confidence level from the correct answer to the correct answer, Representing exercises The number of times a question was answered correctly. Representing exercises and exercises The number of times a response is answered correctly; when calculating the confidence level from incorrect to incorrect response, Representing exercises The number of times a question was answered incorrectly. Representing exercises and exercises The number of times it was done incorrectly at the same time; (2) Construct a new matrix of knowledge points for practice questions; By inputting the exercise knowledge point matrix QC and the learner's learning record into the cognitive diagnostic model KPT of the student simulator, a new exercise knowledge point matrix QC' is obtained. (3) Mining association rules between knowledge points; Combining the new practice question knowledge point matrix QC', the association rules between practice questions are mapped to association rules between knowledge points using the following formula: ； in, Representing exercises The knowledge points included Representing exercises The knowledge points included; Knowledge point matrix annotated by experts Give; Knowledge point matrix calculated by learner simulator Provide a threshold for the relevance of knowledge points. ,when If so, then it is considered that there is no relationship between these two knowledge points.

6. The personalized learning path recommendation method based on reinforcement learning according to claim 5, characterized in that: Step 3 describes the process of selecting potential candidate practice questions in the navigation module as follows: Step S1: When a learner wants to learn a certain target knowledge point, according to the knowledge relationship diagram, the preorder node of the target knowledge point is put into the knowledge point selection pool. Then, the preorder nodes of the knowledge points in the knowledge point selection pool are put into the knowledge point selection pool. This process is repeated until there are no more preorder nodes of the knowledge points in the knowledge point selection pool. Step S2: Randomly select a knowledge point from the knowledge point selection pool, remove it from the knowledge point selection pool, and put the practice questions containing that knowledge point into the practice question selection pool; Step S3: Use a reinforcement learning model to select practice questions to learn from the practice question selection pool. After the learner completes the practice question, the learner's mastery of the knowledge point is judged based on the learner simulator. If the mastery level is greater than the set threshold, it is determined that the learner has mastered the knowledge point and does not need to continue learning the knowledge point. Practice questions containing the knowledge point are deleted from the practice question pool. Step S4: Repeat steps S1-S3 until one of the following two termination conditions is met: termination condition 1 is that the learner has learned the practice questions containing the target knowledge points, and termination condition 2 is that the length of the learning path has reached the specified length limit.

7. The personalized learning path recommendation method based on reinforcement learning according to claim 6, characterized in that: The factors influencing the learning path mentioned in step 4.3 include: (1) Changes in learners' mastery of knowledge points: ； in, This represents the change in the degree of mastery of knowledge points. This indicates that the learner had prior knowledge of the knowledge points before learning. The degree of mastery of the above, This indicates the learner's understanding of the knowledge points after learning. The degree of mastery of the knowledge points; the degree of mastery of the knowledge points is given by the cognitive diagnostic model; (2) The smoothness of the practice questions; A smoothness factor is added to the loss function, and the smoothness of the exercise is represented by the squared difference in difficulty between two consecutive exercises. : ； in, This indicates the difficulty level of the next practice question in the learning path. L indicates the difficulty of the previous exercise, and L represents the length of the learning path; (3) The length of the learning path; Add a factor related to the learning path length to the reward function. : ； (4) The degree of scholarly participation; Add an engagement factor to the reward function This ensures that, from the perspective of the entire learning path, learners will neither find it too easy nor too difficult: ； in, This is the difficulty threshold, with a value between 0 and 1; Considering four factors—changes in learners' mastery of knowledge points, the smoothness of practice questions, the length of the learning path, and learner engagement—the reward function is expressed as follows: ； in, , , These are all penalty parameters, with values ​​ranging from 0 to 1.

8. The personalized learning path recommendation method based on reinforcement learning according to claim 7, characterized in that: The specific method for step 5 is as follows: The reinforcement learning model designed in step 4 is used to recommend exercises to the learner. The recommended exercises are submitted to the knowledge tracking model in the learner simulator. The knowledge tracking model determines whether the learner can answer the exercise correctly. The reinforcement learning model updates the state of the RL agent based on the judgment of the cognitive diagnostic model and continues to recommend exercises to the learner until the termination condition is met. The generated learning path is submitted to the cognitive diagnostic model in the learner simulator. The cognitive diagnostic model determines how much the learner's knowledge level has improved before and after learning and passes this information to the reinforcement learning model. The reinforcement learning model updates its model parameters based on this information.

Images