An intelligent recommendation learning method based on reinforcement learning and related devices

Abstract

The application discloses a kind of intelligent recommendation learning method and related device based on reinforcement learning, it is related to education informatization and artificial intelligence technical field, which comprises obtaining student learning data;Initialize student historical answer record and state-action value function table;Based on student historical answer record and hidden Markov model, get continuous mastery state, get discrete mastery state after discretization;Then adopt action selection strategy to recommend candidate exercises in question set, update exercise recommendation sequence;Then update student historical answer record and continuous mastery state, determine total reward, and update state-action value function table, repeatedly iterate to determine the continuous mastery state of student to current knowledge point and exercise recommendation sequence, until satisfying termination condition, obtain optimal exercise recommendation sequence, the application can improve the accuracy of exercise recommendation, to improve the learning efficiency and knowledge mastery level of student.

Description

Technical Field

[0001] This application relates to the fields of educational informatization and artificial intelligence technology, and in particular to an intelligent recommendation learning method and related apparatus based on reinforcement learning. Background Technology

[0002] Existing smart education platforms generally have low accuracy in judging the degree of knowledge mastery, resulting in a significant amount of learning time being wasted on content that learners have already mastered. Existing knowledge tracing models are mainly divided into two categories: Bayesian models and deep learning models. Bayesian knowledge tracing models have strong interpretability but do not fully consider the difficulty parameter of exercises, assuming that once students master a knowledge point, they will remain in that state indefinitely, ignoring the forgetting curve that is common in student learning. Deep knowledge tracing models have higher prediction accuracy but weak interpretability and are highly dependent on massive amounts of training data, resulting in a significant performance drop in small-sample teaching scenarios.

[0003] In the field of personalized recommendation technology, collaborative filtering algorithms often ignore the dynamic changes in students' knowledge mastery levels, easily leading to recommendations that are either too difficult or too easy. Meanwhile, recommendation methods based solely on knowledge tracking struggle to fully utilize the similarity information among student groups. Therefore, existing knowledge tracking models generally suffer from problems such as difficulty in balancing interpretability and data efficiency, insufficient modeling of forgetting patterns and question difficulty factors, lack of quantitative assessment of long-term learning benefits, and failure to comprehensively consider multiple factors such as question difficulty, student ability, and practice time. Ultimately, these issues result in inaccurate and unreliable question recommendations, leading to a decline in students' learning efficiency and knowledge mastery.

[0004] Therefore, how to provide an intelligent learning solution with high accuracy in recommending exercises, and recommend more accurate and optimal exercise results to students, thereby improving their learning efficiency and knowledge mastery, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this application is to provide an intelligent recommendation learning method and related device based on reinforcement learning, which can improve the accuracy of exercise recommendations, thereby improving students' learning efficiency and knowledge mastery.

[0006] To achieve the above objectives, this application provides the following solution: Firstly, this application provides an intelligent recommendation learning method based on reinforcement learning, including: Acquire student learning data; the student learning data includes a set of knowledge points, a set of questions, student historical answer records, and a state-action value function table.

[0007] Initialize the student's historical answer records and the state-action value function table.

[0008] Based on the student's historical answer records and the Hidden Markov Model, the student's continuous mastery status of the current knowledge point is obtained, and the continuous mastery status of the current knowledge point is discretized to obtain the discrete mastery status of the current knowledge point.

[0009] Based on the current discrete mastery status of the knowledge points, a reinforcement learning action selection strategy is used to recommend candidate exercises from the question set and add them to the exercise recommendation sequence to obtain an updated exercise recommendation sequence.

[0010] Based on the updated exercise recommendation sequence, the student's historical answer records and the current continuous mastery status of the knowledge points are updated, and the total reward is calculated in combination with the reward function; the total reward includes the answer accuracy reward, the knowledge progress reward, and the personalized adaptation reward.

[0011] Based on the total reward, update the state-action value function table and return "Based on the student's historical answer records and hidden Markov model, obtain the student's continuous mastery state of the current knowledge point, and discretize the continuous mastery state of the current knowledge point to obtain the discrete mastery state of the current knowledge point" in a loop until the termination condition is met to obtain the optimal exercise recommendation sequence; the termination condition is reaching the maximum number of recommended questions or mastering all knowledge points.

[0012] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the reinforcement learning-based intelligent recommendation learning method described above.

[0013] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned intelligent recommendation learning method based on reinforcement learning.

[0014] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned intelligent recommendation learning method based on reinforcement learning.

[0015] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides an intelligent recommendation learning method and related apparatus based on reinforcement learning. By discretizing the continuous mastery state based on students' historical answer records and a Hidden Markov Model, it addresses the problems of poor interpretability, low data efficiency, and lack of modeling of forgetting patterns and exercise difficulty in existing knowledge tracking models. This achieves high-precision and interpretable assessment of knowledge mastery in small-sample scenarios. By recommending exercises based on discrete mastery states and employing a reinforcement learning action selection strategy, it solves the problems of mismatched difficulty levels and neglect of students' dynamic abilities and group similarities in traditional recommendations, achieving a more personalized and difficulty-adapted exercise recommendation sequence. By updating answer records and mastery states and combining them with a reward function to calculate the total reward, it addresses the lack of quantification of long-term learning benefits and insufficient integration of multi-dimensional factors in existing methods, achieving a comprehensive reward calculation integrating knowledge progress, difficulty adaptation, and answer results. By updating the state-action value function table based on the total reward and iteratively updating it, it addresses the lack of long-term planning and low learning efficiency in exercise recommendation, achieving an optimal recommendation strategy that obtains the maximum learning benefit with the fewest exercises. This improves the accuracy of exercise recommendation, thereby enhancing students' learning efficiency and knowledge mastery. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of an intelligent recommendation learning method based on reinforcement learning in one embodiment of this application; Figure 2 A flowchart illustrating an intelligent recommendation learning method based on reinforcement learning, provided as an embodiment of this application; Figure 3 A flowchart illustrating an intelligent recommendation learning method based on reinforcement learning, provided for another embodiment of this application; Figure 4 A flowchart illustrating a complete knowledge tracing logic provided for another embodiment of this application; Figure 5 A flowchart illustrating a problem recommendation decision-making process provided in one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The intelligent recommendation learning method based on reinforcement learning provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send student learning data to server 104. After receiving the student learning data, server 104 initializes the student's historical answer records and state-action value function table (Q-table); based on the student's historical answer records and Hidden Markov Model (HMM), it obtains the student's continuous mastery state of the current knowledge point, and discretizes the continuous mastery state of the current knowledge point to obtain the discrete mastery state of the current knowledge point; based on the discrete mastery state of the current knowledge point, it uses a reinforcement learning action selection strategy to recommend candidate questions from the question set. The system generates practice questions and adds them to a practice question recommendation sequence, resulting in an updated practice question recommendation sequence. Based on the updated practice question recommendation sequence, the system updates the student's historical answer records and the continuous mastery status of the current knowledge point, and calculates the total reward using a reward function. According to the total reward, the system updates the state-action value function table and returns a loop that iterates through the process of "obtaining the student's continuous mastery status of the current knowledge point based on the student's historical answer records and a hidden Markov model, and discretizing the continuous mastery status of the current knowledge point to obtain the discrete mastery status of the current knowledge point" until a termination condition is met, resulting in the optimal practice question recommendation sequence. The server 104 can feed back the obtained optimal practice question recommendation sequence to the terminal 102. In addition, in some embodiments, the intelligent recommendation learning method based on reinforcement learning can also be implemented by the server 104 or the terminal 102 alone. For example, the terminal 102 can directly perform optimal practice question recommendation processing on the student's learning data, or the server 104 can obtain the student's learning data from the data storage system and perform optimal practice question recommendation processing on the student's learning data.

[0021] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0022] In one exemplary embodiment, such as Figure 2 As shown, a reinforcement learning-based intelligent recommendation learning method is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 206.

[0023] Step 201: Obtain student learning data; the student learning data includes a set of knowledge points, a set of questions, student historical answer records, and a state-action value function table. The state-action value function table is a table showing the relationship between the discrete mastery status of knowledge points and question IDs.

[0024] Step 202: Initialize the student's historical answer record and state-action value function table.

[0025] Step 203: Based on the student's historical answer records and the hidden Markov model, obtain the student's continuous mastery status of the current knowledge point, and discretize the continuous mastery status of the current knowledge point to obtain the discrete mastery status of the current knowledge point.

[0026] Step 204: Based on the discrete mastery status of the current knowledge points, a reinforcement learning action selection strategy is used to recommend candidate exercises from the question set and add them to the exercise recommendation sequence to obtain the updated exercise recommendation sequence.

[0027] Step 205: Based on the updated exercise recommendation sequence, update the student's historical answer records and the current continuous mastery status of the knowledge points, and calculate the total reward in combination with the reward function; the total reward includes the answer accuracy reward, the knowledge progress reward, and the personalized adaptation reward.

[0028] Step 206: Based on the total reward, update the state-action value function table and return "Based on the student's historical answer records and hidden Markov model, obtain the student's continuous mastery state of the current knowledge point, and discretize the continuous mastery state of the current knowledge point to obtain the discrete mastery state of the current knowledge point" in a loop until the termination condition is met, and obtain the optimal exercise recommendation sequence as the final exercise recommendation result; the termination condition is reaching the maximum number of recommended questions or mastering all knowledge points.

[0029] Implementing steps 201 to 206 above enables personalized exercise recommendations with good interpretability and data efficiency, improving students' learning efficiency and knowledge mastery. Furthermore, this application effectively addresses issues such as insufficient interpretability, low data efficiency, and lack of long-term planning in existing technologies, enabling quantitative assessment of long-term learning benefits and providing a precise and efficient personalized recommendation solution for smart education platforms.

[0030] In another exemplary embodiment of this application, such as Figure 3 As shown, another intelligent recommendation learning method based on reinforcement learning is presented. First, after a student logs into the system, the student's true knowledge state and belief state are initialized. The current knowledge level is then assessed using a Hidden Markov Model combined with the student's historical answer sequence (historical answer record), and discretized into three levels: "not mastered," "partially mastered," and "mastered." Subsequently, the system selects recommended questions based on the ε-greedy strategy of Q-learning and the difficulty of the questions. After the student answers the questions and obtains the correct or incorrect results, the system updates the true knowledge state by considering the forgetting factor and difficulty matching, and updates the belief state based on the observation results and difficulty. The reward value is calculated based on knowledge growth, difficulty matching, and answer results, and then the Q-table is updated to optimize subsequent recommendations. This process is repeated until the preset number of recommendations or the target is reached.

[0031] In another exemplary embodiment of this application, step 203 is replaced by steps 301 to 304: Step 301: Extract knowledge points from the student's historical answer records to obtain the answer sequence related to the knowledge points.

[0032] Step 302: Based on the answer sequence, the Baum-Welch algorithm is used to train the Hidden Markov Model, optimize the model parameters of the Hidden Markov Model, and obtain the trained Hidden Markov Model; the model parameters include belief states, state transition matrix and observation matrix.

[0033] Step 303: Based on the trained Hidden Markov Model, the Forward-Backward algorithm is used to obtain the student's continuous mastery status of the current knowledge point.

[0034] Step 304: Discretize the student's continuous mastery status of the current knowledge point to obtain the discrete mastery status of the current knowledge point.

[0035] It should be noted that, in one exemplary embodiment of this application, an assumption needs to be made regarding the student's knowledge state: the evolution of the student's proficiency in a certain knowledge point is a stochastic process defined on a probability space. Its state space is... .in, Indicates that the student is in It is always in a state of uncertainty. Indicates that the student is in Always in a state of control The time interval is unknown. Therefore, it needs to be estimated from the interaction between the student and the system. Specifically, this interaction occurs at a fixed system period. The following questions will be released, with a maximum of [number] questions. Wheel, among which To estimate the maximum number of rounds students can master, students must complete the required period. Complete the questions within [timeframe] and you will receive feedback on your score. Among them, [the following is unclear and likely incomplete: "among them, "] (in, (This represents the time complexity), ensuring that students have enough time to improve their understanding of the knowledge points based on feedback.

[0036] Regarding the student's problem-solving process, this application introduces the following additional assumptions: (1) Students’ knowledge status is constantly updated with each round of question-solving. That is, after each round of question-solving, students may improve their knowledge level by going through the process of error correction and reflection; or they may experience a decline in their knowledge level due to forgetting or fuzzy memory.

[0037] (2) If no relevant questions are recommended in this round, the state transition matrix will change according to the student’s forgetting coefficient.

[0038] (3) The probability of a student transitioning from a “not mastered” state to a “mastered” state is estimated using the maximum likelihood estimation method.

[0039] (4) Students have similar learning abilities, but their learning efficiency is affected by the degree of matching between the difficulty of the questions and the degree of mastery of the prerequisite knowledge points. Specifically, learning efficiency is jointly determined by the degree of matching between the difficulty of the questions and the students' current knowledge level and the degree of mastery of the prerequisite knowledge points, and adjusts the learning probability in the state transition matrix. The learning efficiency is highest when the difficulty of the questions is slightly higher than the students' current level; the higher the degree of mastery of the prerequisite knowledge points, the higher the efficiency of learning the current knowledge points.

[0040] Regarding the aforementioned knowledge evolution process, this application provides an exemplary embodiment that, assuming Markov properties, uses Hidden Markov Models (HMMs) to characterize the dynamic evolution of students' knowledge points. Its core is determining the quintuples of the HMM. ;in, For the parameters of the Hidden Markov Model; The time range of the Hidden Markov Model; For the first k The students' grasp of the knowledge points at the beginning of the course; For the first k The student's score in the second round of questions, i.e., the score of the first round of questions. k The student's performance in answering the questions; This is the state transition matrix; This is the observation matrix.

[0041] Here, the time horizon represents the time span for estimating students' mastery level, and is a set of discrete time steps. This is despite the evolution of the degree of knowledge mastery. Defined in continuous time, where, Indicates that the student is in The knowledge mastery status at any given moment; however, the embodiments of this application only apply to each round of problem-solving. Only then can the performance be estimated. Therefore, embodiments of this application focus on discrete event grid points. To simplify the expression, we will use the term "token of speech" consistently. describe.

[0042] State space (State): Definition That is, the student's knowledge state space is defined as The state transition probability does not change over time, specifically: .in, This is the state transition matrix; The state transition matrix is ​​the first... Line 1 The probability of a column; The number of rows in the matrix; is the number of columns in the matrix. The probability of learning after one practice session (transitioning from a state of not mastery to a state of mastery). For the probability of forgetting (from mastery to non-mastery), correspondingly, This represents the probability that a student still hasn't learned the material after one practice session (remaining in a state of not having mastered it). The probability that a student has not forgotten (remains in a state of mastery).

[0043] The state transition matrix includes learning probability and forgetting probability; the learning probability is determined based on the matching degree between the difficulty of the question and the student's current knowledge level (i.e., the difficulty matching model) and the probability of mastering the prerequisite knowledge points; the forgetting probability is the probability that the student will transition from a mastered state to a non-mastered state.

[0044] For learning probability Learning efficiency is determined by the match between the difficulty of the questions and the student's current knowledge level. and the probability of mastering prerequisite knowledge points The decision is made jointly. The formula for calculating the match between the difficulty level of the questions and the students' current knowledge level is as follows: (1); in, The degree to which the difficulty of the questions matches the students' current knowledge level; The difficulty level of the question. For students to understand the current knowledge points The probability of mastery ; This is a parameter estimate for the expected value of the difficulty matching model; Parameter estimation for the variance of the difficulty matching model; It is an exponential function with the natural constant e as its base.

[0045] The formula for calculating the comprehensive learning factor is: (2); in, For comprehensive learning factors; For balancing matching degree and The weighting coefficient for the percentage weight; Prerequisite knowledge points Mastering probability, that is ; For the current knowledge point Prerequisite knowledge points.

[0046] Effective learning probability The calculation formula is: (3); in, Based on the probability of learning.

[0047] Setting up the observation space: Definition That is, the observation state space of the student's answer is defined as ; where 0 indicates an incorrect answer and 1 indicates a correct answer. Indicates the first The student answered the question correctly. Indicates the first The student answered the question incorrectly. The relationship between them can be described by the observation matrix. For the first The observation matrix of students' answer status. For the first The observation matrix of the next student's answer status Line 1 The possible values ​​of the column; where, For students in the The probability of guessing correctly even when the answer is unknown; For students in the The probability of making a mistake even when the knowledge is already acquired; and so on. , Considering that student guessing can significantly interfere with the accuracy of observations, this application embodiment further calibrates the observation matrix. The observation matrix includes a guess probability and an error probability; the guess probability is the probability that a student answers correctly even without mastering the knowledge; the error probability is the probability that a student answers incorrectly even with mastery of the knowledge.

[0048] In this embodiment of the application, observation The value of is not only related to the current state of knowledge mastery, but also to the th . The difficulty of the wheel problem Relevant. Taking the second problem-solving exercise as an example, the performance in answering the questions... Based on the difficulty of the exercises and knowledge status The difficulty level of the exercises will be decided jointly. The probability of guessing and the probability of error in the observation matrix The value of ... The impact, among which, For the first The difficulty level of the sub-problems. The formula for calculating the observation matrix is ​​set as follows: (4); in, To provide students with a basic probability estimate; Based on the students' basic error probability, it can be found that as the difficulty of the exercises increases, the basic guess probability has a smaller impact on the actual correct guess probability, which is consistent with the phenomenon that the more difficult the questions are, the harder it is to guess correctly. Similarly, the basic error probability has a greater impact on the actual error probability, which is consistent with the phenomenon that the more difficult the questions are, the easier it is for students to make mistakes. This is the student's individual level coefficient, derived from the student's initial test scores.

[0049] The formula for calculating the student's individual level coefficient is as follows: (5); in, A These are the students' initial test scores.

[0050] Figure 4 To establish a complete knowledge tracking logic, the system dynamically estimates students' mastery of each knowledge point based on their historical answer performance and predicts the probability of correct answers in the future. Figure 4 The three columns represent the sequence of problem difficulty. That is, different difficulty levels of practice questions, state sequence That is, the potential mastery of knowledge points by students, observation sequence. This refers to the results observed after students answer the questions. Firstly, students' understanding of the knowledge points... There is an initial mastery probability (initial belief state). (Obtained through the initial test, where,) For students to understand the knowledge points (Initial belief state), after the student's first practice, the student's mastery state changes from... Transferred to The students' answers generated observations The probability of observation depends on the current state. and the difficulty of the questions According to observations Update the belief state for the next time step. Then proceed to the next time step and continue recommending questions, repeating until the next time step (number of question recommendations). Finish.

[0051] The knowledge tracing task based on Hidden Markov Models (HMMs) consists of two phases: a learning phase and a prediction phase. The learning phase uses the Baum-Welch (unsupervised learning) algorithm to learn all the parameters of the HMM. Then, in the prediction phase, the system recursively estimates the hidden states using the student's real-time answer observation sequence.

[0052] First, the expression for the initial belief state is: (6); in, This represents the initial belief state, i.e., the initial probability of mastery. This represents the probability that a student's initial state is one of not having mastered the material; This represents the probability that the initial state is known.

[0053] With the number of questions k The advancement, according to the first k The observed answer results The belief state is recursively updated using a state update function. The formula for calculating the belief state is: (7); in, For the first This is a state of belief; For state update functions; For the first The observed answer results; No. -1 belief state; For conditional probability; For the first The student's performance in answering the questions; This represents the current number of questions submitted.

[0054] According to Bayes' criterion and the law of total probability, the matrix analytical form of the state update function is as follows: (8); in, For diagonalization operators; Indicates the answer result All rows of the observation matrix, It is a column vector; Vector The elements on the main diagonal form a vector matrix.

[0055] In another exemplary embodiment of this application, the process of continuously mastering state discretization is as follows, that is, assuming a common Each knowledge point The probability of mastering a state of continuous mastery is ,in, i For the first i One key point; the discretization threshold is... Here, in order to connect with the first i The continuous mastery of each knowledge point The distinction is made by using the discrete mastery status of the knowledge points after further discretization. The formula for calculating the discrete mastery status of the knowledge points is as follows: (9); in, For the first i The discrete mastery status of each knowledge point; the discrete mastery status of students' knowledge points in the recommendation system is ( There are a total of There are several possible states.

[0056] The exercise recommendation system uses Hidden Models (HMMs) to model the implicit changes in students' knowledge states, and uses the continuous mastery states output by the HMM as the state input to the Markov Decision Process (MDP). The MDP is responsible for deciding which exercise to recommend and updates the strategy based on the student's answer feedback (correct / incorrect) to maximize long-term rewards (i.e., learning outcomes).

[0057] This application discretizes the continuous mastery probabilities of multiple knowledge points estimated from the Hidden Markov Model (HMM) into a composite state and learns a Q-function to guide exercise recommendation. The exercise recommendation decision process is as follows: Figure 5 As shown, firstly, the continuous knowledge state of students is output through the HMM knowledge tracing model and transformed into a continuous mastery probability vector (i.e., a continuous mastery state vector). Then, using discretization thresholds (0.3 and 0.7), the state is discretized, and the continuous state (i.e., the continuous mastery state vector) is discretized into three levels: not mastered, partially mastered, and proficiently mastered, resulting in a discrete state vector (i.e., a discrete mastery state vector). This is then used to construct the MDP state, i.e., the MDP composite state space (with...). (There are several possible states). Then, based on the ε-greedy strategy, Q-learning decision is executed, recommending exercises to students based on the knowledge points and difficulty of the questions. After the students answer the questions, the correctness observation value is generated. The reward is calculated by combining three dimensions: the reward for answering correctly, the reward for knowledge progress, and the reward for personalized adaptation. Then, the recommendation strategy is updated and optimized through the Q value. Finally, the student's state is updated and the cycle begins, realizing the dynamic tracking of the student's learning state and the continuous optimization of the exercise recommendation strategy.

[0058] In another exemplary embodiment of this application, step 204 is replaced by steps 401 to 404: Step 401: Determine a random number.

[0059] Step 402: Determine the relationship between the random number and the exploration rate to obtain the determination result.

[0060] It should be noted that the exploration rate described in this embodiment is 0.05.

[0061] Step 403: If the judgment result is that the random number is less than the exploration rate, then randomly select a question from the candidate questions in the question set as a recommended question and add it to the question recommendation sequence to obtain the updated question recommendation sequence.

[0062] Step 404: If the judgment result is that the random number is greater than or equal to the exploration rate, then the score of each candidate exercise in the question set is calculated using the scoring function, and the candidate exercise with the highest score is selected as the recommended exercise and added to the exercise recommendation sequence to obtain the updated exercise recommendation sequence.

[0063] It should be noted that the expression for the scoring function is: (10); in, For scoring functions; For the current moment t State-action value function value; For the current moment t The discrete mastery status of knowledge points; The difficulty level of the current question; For students to answer the questions The degree of mastery of all related knowledge points; The title ID; This represents the maximum level of mastery of the relevant knowledge points for the question. The recommended questions are those that are more difficult than the students' level. This is the adjustment coefficient.

[0064] In another exemplary embodiment of this application, in the MDP, the action is the recommended exercise ID. Each exercise is associated with one or more knowledge points and a difficulty value. The total reward for each exercise... r It consists of the following parts: The calculation method for the correct answer reward is as follows: when the difficulty of the question is less than the pre-screening threshold, if the student answers the current question correctly, the correct answer reward is the first reward value; if the student answers the current question incorrectly, the correct answer reward is the second reward value.

[0065] When the difficulty of a question is greater than or equal to the pre-screening threshold, if a student answers the question correctly, the correct answer reward is the third reward value; if a student answers the question incorrectly, the correct answer reward is the fourth reward value.

[0066] In one embodiment of this application, the pre-screening threshold is set to 0.7, that is, when the difficulty of the question... The formula for calculating the reward for correct answers is as follows: (11); in, Rewards are given for correct answers. When the difficulty of a question is less than 0.7, the first reward value is +0.5 for a correct answer and -0.3 for an incorrect answer.

[0067] When the difficulty of the question is hour: (12); When the difficulty of a question is greater than or equal to 0.7, the third reward value is +0.7 if the student answers the question correctly; the fourth reward value is -0.2 if the student answers the question incorrectly.

[0068] It is evident that when the questions are easy, the reward for a student to answer the question correctly is smaller, and the penalty for a student to answer the question incorrectly is larger (i.e., the reward is a larger negative number); when the questions are difficult, the reward for a student to answer the question correctly is larger, and the penalty for a student to answer the question incorrectly is smaller (i.e., the reward is a smaller negative number).

[0069] The knowledge progress reward is calculated as the increase in the degree of mastery of the knowledge point predicted by the knowledge tracking model for answering the question correctly. The formula for calculating the knowledge progress reward is as follows: (13); in, Rewards for advancements in knowledge; To determine the average probability of mastering the current question before answering it correctly; This represents the average probability of mastering the current question after answering it correctly.

[0070] It should be noted that, ;in, This represents the average probability that students have mastered all the knowledge points covered in this question. It refers to all the knowledge points related to the question; It is the students' opinion on the first i The probability of mastering a knowledge point in a continuous state.

[0071] Personalized matching rewards are based on the degree to which the difficulty of the questions matches the student's current level.

[0072] The formula for calculating the personalized adaptation reward is as follows: (14); in, Personalized rewards for adaptation; d The difficulty level is denoted by "probability"; "target" represents the target mastery probability set by matching the student's current difficulty level with the difficulty of the question.

[0073] It should be noted that, Here, `target` represents the target mastery probability set for the student to match the difficulty of the current question. Since we hope that the questions students actually answer are slightly more challenging than their current level of mastery, the target value should be 0.1 greater than the actual mastery probability.

[0074] The formula for calculating the total reward is as follows: (15); in, , , Let be the gain coefficient, satisfying ,in, jThis is an index variable, with values ​​of 1, 2, and 3.

[0075] In another exemplary embodiment of this application, a personalized question recommendation method based on a combination of Q-learning algorithm and Hidden Markov Model can be constructed based on the knowledge tracking process and reward function settings. This method tracks students' mastery of knowledge points and dynamically adjusts recommended questions using reinforcement learning strategies to achieve optimal learning path planning.

[0076] First, initialize the system parameters and data structures, as shown in the following steps: First, define the set of knowledge points. and the set of questions Each question ID Associate a subset of knowledge points It also has a difficulty attribute. The set of questions participating in this round of recommendations will be marked as candidate exercises. Then initialize the student's historical answer records. ,in This is the current question ID, which can also be considered as an action in the action space. Indicate whether the student's answer is correct. The difficulty level of the problem is set. Next, the algorithm hyperparameters are defined: Set as the learning rate (default 0.1); Discount factor (default 0.9); For the exploration rate (used in ε-greedy strategies); These are the strategy preference weighting coefficients; and This is the discretization threshold; This represents the maximum number of iterations. Finally, initialize the state-action value function table. Wherein, Let be a zero matrix, where For discrete mastery states of knowledge points, the Q value is related to the state and action, and is initialized to zero.

[0077] Next, based on the student knowledge status tracking of the Hidden Markov Model (HMM), for each knowledge point... Perform the following operations to estimate the student's continuous mastery of the current knowledge point: First, extract the knowledge points from the students' history answer records. Related answer sequence Answer sequence The input to the Hidden Markov Model (HMM) is used for HMM training. The Baum-Welch algorithm is then used to train the HMM, with model parameters including belief states, state transition matrices, and observation matrices. Next, the Forward-Backward algorithm is run to calculate the student's state at the current time step. The probability of continuous mastery of this knowledge point (continuous mastery status). The mastery probabilities of all knowledge points are summarized to form the current continuous mastery status of this knowledge point. .

[0078] Then, in order to adapt to the Q-learning algorithm, the continuous mastery state of the current knowledge point is mapped to the discrete mastery state of the current knowledge point. For each knowledge point: if The current discrete mastery status of knowledge points This indicates that the status is not yet known; if The current discrete mastery status of knowledge points This indicates a state of partial control; if The current discrete mastery status of knowledge points This indicates a state of complete mastery. Ultimately, we obtain the discrete mastery state of the current knowledge point. .

[0079] Next, action selection (i.e., question recommendation) is performed based on the ε-greedy strategy. The next recommended question is chosen according to the current discrete mastery status of the knowledge point: First, a random number is determined, and its relationship with the exploration rate is assessed. If the random number is less than the exploration rate, a question is randomly selected from the candidate question set (this process is called exploration). If the random number is greater than or equal to the exploration rate, a score is calculated for each candidate question using a scoring function, and the question with the highest score is selected (this process is called utilization). Then, the optimal action is determined. : This refers to the most suitable question for recommendation. Next, the ID of the most suitable question for recommendation is... Add to the recommended sequence.

[0080] Then, students wait to answer all the questions in the recommended question sequence and receive their answers. Where 1 indicates a correct answer and 0 indicates an incorrect answer. The system also retrieves the set of knowledge points associated with the question. This is to update students' history answer records. Add it to the sequence of the corresponding knowledge points. Then, target the set of affected knowledge points. Rerun the Forward-Backward algorithm of the Hidden Markov Model to update the mastery probability and obtain a new continuous mastery state vector. The total reward at the current moment is calculated based on the current state, actions, answer results, and state changes.

[0081] Next, the new state of continuous mastery. Discretization is performed to obtain the discrete mastery status of the knowledge points at the next time step. After setting initial values, iteratively calculate using the standard Q-learning iterative formula. The expression for the Q-value of the next state is: (16); in, The state-action value (Q-function value) for the next state; The state-action value (Q-function value) of the current state; The total reward for the next moment; The maximum predicted state-action value (Q-function value) for the next state; For learning rate, This is the discount factor.

[0082] Finally, update the continuous mastery status of the current knowledge point ( Determine whether the termination condition is met (such as reaching the maximum number of recommended questions). (Or, all knowledge points have been mastered). If this condition is not met, return to the step of discretizing the continuous mastery status of the current knowledge points and continue to the next round of recommendations; if the condition is met, output the recommendation sequence and end the process.

[0083] This application proposes an intelligent recommendation learning method based on reinforcement learning, effectively addressing the problems of insufficient interpretability, low data efficiency, and lack of long-term planning in existing technologies. By incorporating forgetting patterns and question difficulty factors into a Hidden Markov Model (HMM), an improved HMM knowledge tracking model with good interpretability is constructed, maintaining high prediction accuracy even in small-sample teaching scenarios. Furthermore, question recommendation is implemented as a Markov decision process, and a multi-objective reward function that comprehensively considers knowledge progress, difficulty fit, and answer results is designed, enabling a quantitative evaluation of long-term learning benefits and providing a precise and efficient personalized recommendation solution for smart education platforms.

[0084] This application significantly improves the accuracy of knowledge state assessment by introducing a forgetting factor based on the number of times the student has not practiced and a question difficulty factor, reducing the prediction error by approximately 25% compared to the traditional BKT model. The reward function simultaneously considers multiple dimensions, including the matching degree between question difficulty and student level, and the degree of mastery of prior knowledge points, resulting in a higher degree of difficulty fit for recommended questions compared to traditional methods. Employing the Q-learning algorithm to learn the optimal recommendation strategy quantifies the long-term practice value of each question for a specific student, and the long-term cumulative total reward of the question recommendation sequence is also improved compared to immediate recommendation strategies, achieving the goal of obtaining greater practice effects with fewer questions.

[0085] In practical circuit course teaching applications, this application improves the accuracy of judging the mastery of knowledge points, reduces wasted learning time, and results in greater improvement in knowledge mastery when students practice the same number of exercises. Furthermore, the system applying this method achieved an interpretability score of 4.2 (out of 5.0) in a satisfaction survey, effectively overcoming the recommendation accuracy problem during the cold start phase for new users. This provides reliable technical support for personalized exercise recommendations in small-sample teaching scenarios, significantly improving the precision teaching capabilities of the smart education platform and student learning efficiency.

[0086] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores student learning data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an intelligent recommendation learning method based on reinforcement learning.

[0087] Those skilled in the art will understand that Figure 6The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0088] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0089] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0090] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations and be authorized by the owner of the corresponding device.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0092] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An intelligent recommendation learning method based on reinforcement learning, characterized in that, The intelligent recommendation learning method based on reinforcement learning includes: Acquire student learning data; the student learning data includes a set of knowledge points, a set of questions, student historical answer records, and a state-action value function table; Initialize the student historical answer records and state-action value function table; Based on the student's historical answer records and the hidden Markov model, the student's continuous mastery status of the current knowledge point is obtained, and the continuous mastery status of the current knowledge point is discretized to obtain the discrete mastery status of the current knowledge point. Based on the current discrete mastery status of the knowledge points, a reinforcement learning action selection strategy is used to recommend candidate exercises from the question set and add them to the exercise recommendation sequence to obtain an updated exercise recommendation sequence. Based on the updated exercise recommendation sequence, the student's historical answer records and the current continuous mastery status of the knowledge points are updated, and the total reward is calculated in combination with the reward function; the total reward includes the answer accuracy reward, the knowledge progress reward, and the personalized adaptation reward. Based on the total reward, update the state-action value function table and return "Based on the student's historical answer records and hidden Markov model, obtain the student's continuous mastery state of the current knowledge point, and discretize the continuous mastery state of the current knowledge point to obtain the discrete mastery state of the current knowledge point" in a loop until the termination condition is met to obtain the optimal exercise recommendation sequence; the termination condition is reaching the maximum number of recommended questions or mastering all knowledge points.

2. The intelligent recommendation learning method based on reinforcement learning according to claim 1, characterized in that, Based on the student's historical answer records and the Hidden Markov Model, the continuous mastery state of the student on the current knowledge point is obtained, and the continuous mastery state of the current knowledge point is discretized to obtain the discrete mastery state of the current knowledge point, specifically including: Knowledge points are extracted from the students' historical answer records to obtain answer sequences related to the knowledge points; Based on the answer sequence, the Baum-Welch algorithm is used to train the Hidden Markov Model (HMM) and optimize its parameters to obtain a trained HMM. The model parameters include belief states, state transition matrices, and observation matrices. Based on the trained Hidden Markov Model, the Forward-Backward algorithm is used to obtain the student's continuous mastery status of the current knowledge point. The continuous mastery status of the student's current knowledge point is discretized to obtain the discrete mastery status of the current knowledge point.

3. The intelligent recommendation learning method based on reinforcement learning according to claim 2, characterized in that, The formula for calculating the belief state is: ； in, For the first k This is a state of belief; For state update functions; For the first k The observed answer results; No. k -1 belief state; For conditional probability; For the first k The students' grasp of the knowledge points at the beginning of the course; For the first k The student's performance in answering the questions; k This represents the current number of questions asked. The state transition matrix includes learning probability and forgetting probability; the learning probability is determined based on the matching degree between the difficulty of the question and the student's current knowledge level and the probability of mastering the prerequisite knowledge points; the forgetting probability is the probability that the student will transition from a mastered state to a non-mastered state. The observation matrix includes a guessing probability and an error probability; the guessing probability is the probability that a student answers correctly even without mastering the knowledge; the error probability is the probability that a student answers incorrectly even with mastery of the knowledge.

4. The intelligent recommendation learning method based on reinforcement learning according to claim 3, characterized in that, The formula for calculating the match between the difficulty of the questions and the students' current knowledge level is as follows: ； in, The degree to which the difficulty of the questions matches the students' current knowledge level; The difficulty level of the question; For students to understand the current knowledge points The probability of mastering; For the parameter estimation of the expected value of the difficulty matching model, Parameter estimation for the variance of the difficulty matching model; It is an exponential function with the natural constant e as its base; The formula for calculating the learning probability is: ； in, For learning probability; Based on learning probability; It is a comprehensive learning factor.

5. The intelligent recommendation learning method based on reinforcement learning according to claim 1, characterized in that, Based on the current discrete mastery status of the knowledge points, a reinforcement learning action selection strategy is used to recommend candidate exercises from the question set and add them to the exercise recommendation sequence to obtain an updated exercise recommendation sequence, specifically including: Determine a random number; Determine the relationship between the random number and the exploration rate to obtain the determination result; If the judgment result is that the random number is less than the exploration rate, a question is randomly selected from the candidate questions in the question set as a recommended question and added to the question recommendation sequence to obtain an updated question recommendation sequence; If the judgment result is that the random number is greater than or equal to the exploration rate, the score of each candidate exercise in the question set is calculated using the scoring function, and the candidate exercise with the highest score is selected as the recommended exercise and added to the exercise recommendation sequence to obtain the updated exercise recommendation sequence.

6. The intelligent recommendation learning method based on reinforcement learning according to claim 5, characterized in that, The expression for the scoring function is: ； in, For scoring functions; For the current moment t State-action value function value; For the current moment t The discrete mastery status of knowledge points; The difficulty level of the question; For students to answer the questions The degree of mastery of all related knowledge points; The title ID; This represents the maximum level of mastery of the relevant knowledge points for the question. The recommended questions are those that are more difficult than the students' level. This is the adjustment coefficient.

7. The intelligent recommendation learning method based on reinforcement learning according to claim 1, characterized in that, The method for calculating the reward for correct answers is as follows: When the difficulty of the question is less than the pre-screening threshold, if the student answers the question correctly, the correct answer reward is the first reward value; if the student answers the question incorrectly, the correct answer reward is the second reward value. When the difficulty of a question is greater than or equal to the pre-screening threshold, if a student answers the question correctly, the correct answer reward is the third reward value; if a student answers the question incorrectly, the correct answer reward is the fourth reward value. The formula for calculating the knowledge advancement reward is as follows: ； in, Rewards for advancements in knowledge; To determine the average probability of mastering the current question before answering it correctly; The average probability of mastering the question after answering the current question correctly. The formula for calculating the personalized adaptation reward is as follows: ； in, Personalized rewards for adaptation; d The difficulty level is set as the question's level; the target is the student's target mastery probability set to match the current question's difficulty. The formula for calculating the total reward is as follows: ； in, , , This is the gain coefficient; Rewards are given for correct answers.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the intelligent recommendation learning method based on reinforcement learning as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the intelligent recommendation learning method based on reinforcement learning as described in any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the intelligent recommendation learning method based on reinforcement learning as described in any one of claims 1-7.

Images