Student autonomous learning ability improvement method based on reinforcement learning and Bayesian optimization

Abstract

The invention discloses a reinforcement learning and Bayesian optimization-based student autonomous learning ability improvement method. The method comprises the following steps of S1, collecting a student behavior data set in a student learning process; s2, forming a learning feature set of student individuals; s3, establishing a learning path optimization model based on a hierarchical reinforcement learning algorithm; s4, key parameters in the high-layer reinforcement learning model and the low-layer reinforcement learning model are adjusted through Bayesian optimization; s5, a reward mechanism based on a reinforcement learning algorithm; s6, updating the learning behavior model of the student at the end of each learning cycle; and S7, forming a cross-stage task connection mechanism based on the long-term learning target of the high-level reinforcement learning model and the short-term learning task of the low-level reinforcement learning model, and automatically formulating a learning strategy after each learning stage for the student. According to the invention, the self-adaptability of the system and the learning efficiency of students are obviously improved.

Description

Technical Field

[0001] The present invention relates to the technical field of student autonomous learning, and in particular to a method for improving student autonomous learning ability based on reinforcement learning and Bayesian optimization. Background Art

[0002] In the existing technology, the improvement of students' autonomous learning ability mainly relies on online learning systems and adaptive learning systems. Online learning systems and adaptive learning systems support the learning process by recommending personalized learning content and paths to students. However, most of the current online learning systems adopt a static recommendation mechanism, that is, the system generates learning paths and recommends learning resources based on students' initial learning data or fixed rules. This method can meet students' basic learning needs to a certain extent, but it is difficult to cope with dynamic changes in students' learning process, such as changes in students' learning status, generation of real-time feedback and fluctuations in cognitive levels. Therefore, the improvement of students' autonomous learning ability in long-term learning is often limited and cannot be effectively motivated and guided.

[0003] In addition, although existing adaptive learning systems attempt to adjust learning paths through certain feedback mechanisms, they rely on a large amount of historical data and fixed parameter models for optimization, resulting in inefficient adjustment of learning strategies and difficulty in real-time adaptation to complex learning environments and individual differences among students. In the process of dealing with individual students' short-term learning goals and long-term learning plans, existing technologies find it difficult to coordinate the formulation of long-term learning goals with the flexible adjustment of short-term learning tasks. There is a lack of effective use of students' real-time feedback, and thus the inability to achieve dynamic optimization of learning paths.

[0004] Another shortcoming of the existing technology is that the efficiency of strategy search and learning resource recommendation in existing systems is low. Existing adaptive learning systems usually rely on traditional optimization methods, such as grid search or random search. Although grid search or random search can find suitable learning strategies to a certain extent, due to the high computational complexity and large demand for feedback data, the system often requires a lot of computing resources and time in the strategy search process, resulting in lag and inefficiency in strategy updates.

[0005] To sum up, the existing technologies have the following main shortcomings: First, online learning systems and adaptive learning systems have limitations in real-time adjustment of learning paths and resource recommendations, and are unable to adjust learning strategies in a timely manner based on students' dynamic feedback; second, the existing systems lack effective mechanisms for coordinating long-term and short-term learning tasks, making it difficult to achieve personalized dynamic learning optimization; finally, the strategy search efficiency of the existing systems is low, resulting in insufficient adaptability of the systems in complex learning environments, making it difficult to quickly converge to the optimal learning strategy. Summary of the invention

[0006] One purpose of the present invention is to propose a method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization. The present invention significantly improves the adaptability of the system and the learning efficiency of students.

[0007] A method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to an embodiment of the present invention includes the following steps:

[0008] S1. Collect student behavior data sets during the learning process, including study time, test scores, and course participation;

[0009] S2. Preprocess the student behavior data set, extract the student learning status, learning interest and cognitive level features in the student behavior data set, and form a learning feature set of individual students;

[0010] S3. Establish a learning path optimization model based on a hierarchical reinforcement learning algorithm. The learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model.

[0011] S4. Use Bayesian optimization to adjust key parameters in high-level reinforcement learning models and low-level reinforcement learning models;

[0012] S5. Based on the reward mechanism in the reinforcement learning algorithm, the high-level reinforcement learning model gives long-term rewards according to the achievement of long-term learning goals, and the low-level reinforcement learning model gives short-term rewards according to the completion of short-term learning tasks;

[0013] S6. At the end of each learning cycle, the student's learning behavior model is updated. The high-level reinforcement learning model adjusts the student's long-term learning goal according to the updated learning behavior model and the long-term learning features in the student behavior dataset. The low-level reinforcement learning model adjusts the short-term learning task according to the updated learning behavior model and the short-term learning features in the student behavior dataset. The updated learning path and strategy are globally optimized through Bayesian optimization.

[0014] S7. Based on the long-term learning goals of the high-level reinforcement learning model and the short-term learning tasks of the low-level reinforcement learning model, a cross-stage task connection mechanism is formed to automatically formulate learning strategies for students after each learning stage.

[0015] Optionally, the S1 specifically includes:

[0016] S11. Collect student behavior data during the learning process, including study time, test scores, and course participation:

[0017] Study time T i , records the total time each student spends studying in a specific period of time;

[0018] Test score S i , collect students' scores in each stage of the test, the test scores are expressed as scores, and each student's test score is S i ∈[0,100];

[0019] Course participation P i , records students’ participation in each course, which is defined as the ratio of students’ actual participation time to the total course time;

[0020] S12, the learning time T in the student behavior data i , test score S i and course participation P i Record and store data to form student behavior data sets:

[0021]

[0022] Optionally, the S2 specifically includes:

[0023] S21, preprocess the student behavior dataset D to remove abnormal data and redundant information;

[0024] S22. Extract student learning status features L from student behavior dataset i , the student learning status feature evaluates the student's overall learning status by balancing study time and test scores, capturing the relationship between students' time investment and academic performance during the learning process:

[0025]

[0026] Among them, L i is the learning status characteristic of student i, T i is the study time of student i, S i is the test score of student i, α and β are weight coefficients, indicating the relative importance of learning time and test scores in the learning state. and are the sum of study time and test scores of all students respectively;

[0027] S23. Extracting students’ learning interest characteristics I i The learning interest feature evaluates students’ interest in participating in course learning by combining learning time and students’ course participation. When students’ course participation deviates greatly from the overall level, the interest value is adjusted through an exponential function to reflect students’ learning enthusiasm and interest changes:

[0028]

[0029] Among them, I i is the learning interest characteristics of student i, Ti For students’ study time, T max is the maximum learning time among all students, λ· is the penalty factor for participation deviation, P i For students' course participation, is the average course participation of all students;

[0030] S24. Extracting students’ cognitive level features C i The cognitive level feature evaluates the student's cognitive ability improvement by comparing the current test score with the changes in historical test scores, and quantifies the student's relative cognitive level by the ratio of the student's current test score to the highest test score in the group:

[0031]

[0032] Among them, C i is the cognitive level characteristic of student i, S i is the current test score, S prev is the score of the previous test, S max is the highest score among all students, δ and ∈ are adjustment parameters;

[0033] S25, extract the learning state feature L i , Learning interest characteristics I i and cognitive level characteristics C i Combined to form the learning feature set F of individual students i ={L i ,I i ,C i}:

[0034]

[0035] Optionally, the S3 specifically includes:

[0036] S31. Establishing a learning path optimization model using the learning feature set of individual students, wherein the learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model;

[0037] S32, the high-level reinforcement learning model uses the student's long-term learning characteristics as the state space S H , state space S H From the learning feature set F of individual students i The action space A is extracted from the long-term learning goal, including the cumulative learning time, overall test scores and long-term learning interests. H , including curriculum planning and knowledge point mastery, through the strategy function π H (a|s) determines the state s∈S H The action a∈A is chosen H ;

[0038] S33. Use Bellman's optimal equation to calculate the value function V of the high-level reinforcement learning model based on long-term learning characteristics. H (s) to optimize:

[0039]

[0040] Among them, V H (s) is the value function of state s, R H (s,a) is the long-term reward for taking action a in state s, γ H is the high-level discount factor, P H (s ′ |s,a) is the state transition probability from state s to the next state s' through action a;

[0041] S34, the low-level reinforcement learning model uses students' short-term learning characteristics and real-time feedback data as the state space S L , state space S L Contains the learning feature set F of individual students i ={L i ,I i ,C i The learning state feature L in i , Learning interest characteristics I i and cognitive level characteristics C i And real-time feedback data; take short-term learning tasks as action space A L , including daily learning content arrangement and specific knowledge point exercises, through the strategy function π L (a|s) determines the state s∈S L The action a∈A is chosen L ;

[0042] S35. The low-level reinforcement learning model plans short-term learning tasks according to the long-term goals set by the high-level reinforcement learning model. It uses the short-term learning features and real-time feedback data in the student behavior dataset to optimize the short-term learning tasks so that the short-term learning tasks are consistent with the students' current learning progress. The deep Q network algorithm is used to update the low-level value function Q. L (s,a):

[0043]

[0044] Among them, Q L (s,a) is the value function of taking action a in state s, α is the learning rate, r is the immediate reward, γ L is the low-level discount factor, s' is the next state after executing action a;

[0045] S36. Long-term learning goal A through high-level reinforcement learning model H and short-term learning tasks A of low-level reinforcement learning models L , combined with the individual learning feature set F i , forming a learning path optimization model.

[0046] Optionally, the S4 specifically includes:

[0047] S41. In the high-level reinforcement learning model, Bayesian optimization is used to adjust the weight parameter θ of the learning path H , the learning path weight parameters affect students' long-term learning goals A H The choice of the optimal weight is selected through Bayesian optimization Maximize the high-level value function V H (s):

[0048]

[0049] Among them, θ H is the weight parameter of the high-level learning path, Represents the feature space S learned from the long term H The state s sampled in;

[0050] S42, Bayesian optimization through Gaussian process regression model GP (μ, σ) modeling learning path weight parameter θ H With the high-level value function V H (s), where μ is the mean function, σ is the covariance function, and the next sampling point θ is selected by the expected improvement criterion H :

[0051]

[0052] S43. In the low-level reinforcement learning model, use Bayesian optimization to optimize the key parameter θ of the learning resource recommendation strategy L Make adjustments, including content relevance, difficulty level, and time allocation for learning resource recommendations, and determine the optimal parameters through Bayesian optimization Let the low-level value function Q L (s,a)maximize;

[0053] S44. Bayesian optimization models the parameters θ of the learning resource recommendation strategy through the Gaussian process regression model L With the low-level value function Q L (s, a) relationship, and select the optimal parameter θ using the expected improvement criterion L .

[0054] Optionally, the S5 specifically includes:

[0055] S51. The high-level reinforcement learning model gives long-term rewards based on the achievement of long-term learning goals. The long-term reward R H (s,a) consists of the following two parts:

[0056] Course Completion Rewards H,course (s,a), rewards are given by calculating the overall completion of the student's courses, defined as the proportion of courses completed by the student to the total number of courses:

[0057]

[0058] Among them, C completed The number of courses completed by the student, C total is the total number of courses;

[0059] Test score improvement reward R H,exam (s,a), rewards are calculated based on the student's improvement in each exam as the difference between the current exam score and the previous exam score:

[0060] R H,exam (s,a)=max(0,S i -S prev );

[0061] Among them, S i is the student’s current test score, S prev The student’s previous test scores;

[0062] S52. Add the course completion reward and the test score improvement reward to calculate the total long-term reward R in the high-level reinforcement learning model H (s,a):

[0063] R H (s,a)=α course ·R H,course (s,a)+α exam ·R H,exam (s,a);

[0064] Among them, α course and α exam They are the weight coefficients of course completion and test score improvement in long-term rewards;

[0065] S53, the low-level reinforcement learning model gives short-term rewards R according to the completion of short-term learning tasks L (s,a), the short-term reward includes the following three parts:

[0066] Test score improvement reward R L,score (s,a), rewards students based on their improvement in scores on short tests:

[0067] R L,score (s,a)=max(0,S i -S prev,test );

[0068] Among them, S prev,test The result of the last test;

[0069] Learning time increase reward R L,time (s,a), rewards students based on the amount of time they spend studying each day:

[0070] R L,time (s,a)=log(1+T i );

[0071] Among them, T i for the amount of learning time students devote to their studies during the day;

[0072] Mission Completion Efficiency Reward R L,efficiency (s,a), rewards students based on their efficiency in completing daily tasks, defined as the ratio of the time taken to complete the task to the estimated time of the task:

[0073]

[0074] Among them, T estimated is the estimated completion time of the task, T actual The time actually spent on completing the task;

[0075] S54. Calculate the total short-term reward R in the low-level reinforcement learning model by adding the test score improvement reward, learning time increase reward and task completion efficiency reward. L (s,a):

[0076] R L (s,a)=β score ·R L,score (s,a)+β time ·R L,time (s,a)+β efficiency ·R L,efficiency (s,a);

[0077] Among them, β score , β time and β efficiency are the weight coefficients of test score improvement, study time increase and task completion efficiency in short-term rewards;

[0078] S55. Through the reward mechanism of S51-S54, the learning path optimization model adjusts students' learning paths and learning tasks in real time, so that students can continuously gain motivation and improve their autonomous learning ability during the learning process.

[0079] Optionally, the S6 specifically includes:

[0080] S61. After each learning cycle, the student learning behavior model is rebuilt and updated based on the latest student behavior data to reflect the changes in the student's learning status, interests and cognitive level in each learning cycle;

[0081] S62. Based on the updated learning behavior model, the high-level reinforcement learning model adjusts the long-term learning goals according to the students’ long-term learning characteristics, and dynamically updates the long-term learning goals based on the students’ historical learning behaviors and current learning progress, so that the learning path is consistent with the students’ cognitive development;

[0082] S63, the low-level reinforcement learning model adjusts the short-term learning tasks according to the short-term learning characteristics reflected in the latest learning behavior model and the real-time feedback data within the cycle, so that the short-term tasks are adapted to the students' current learning status and real-time feedback;

[0083] S64. By globally adjusting the strategies of the high-level reinforcement learning model and the low-level reinforcement learning model, and optimizing the learning path and strategy in combination with Bayesian optimization, the high-level reinforcement learning model and the low-level reinforcement learning model can dynamically optimize the learning path and resource recommendation strategy in each cycle.

[0084] The beneficial effects of the present invention are:

[0085] (1) The present invention introduces high-level and low-level reinforcement learning models based on reinforcement learning to hierarchically manage and dynamically optimize students' long-term learning goals and short-term learning tasks. The high-level reinforcement learning model plans students' long-term learning goals and makes real-time adjustments based on students' historical learning behavior models and the latest learning status; the low-level reinforcement learning model adjusts learning tasks based on real-time feedback from short-term learning tasks, so that the learning path can dynamically adapt to students' current learning status and feedback information. Different from the static learning path recommendation in the prior art, the present invention continuously updates the learning strategy through the reinforcement learning algorithm, and can continuously optimize the learning path according to students' learning performance and real-time feedback, thereby improving the flexibility and adaptability of the learning path.

[0086] (2) The present invention combines the Bayesian optimization algorithm to efficiently adjust the key parameters in the high-level reinforcement learning model and the low-level reinforcement learning model. The Bayesian optimization can efficiently find the optimal learning strategy through the Gaussian process regression model with less feedback data, which solves the problems of low strategy search efficiency and excessive consumption of computing resources in the existing system. By globally optimizing the learning path and learning resource recommendation strategy, the present invention effectively shortens the strategy search time and improves the system's adaptability to complex learning scenarios, making personalized learning resource recommendations more accurate and responsive.

[0087] (3) The present invention designs a continuous feedback mechanism based on long-term and short-term reward mechanisms in the reinforcement learning algorithm. The high-level reinforcement learning model gives long-term rewards based on the achievement of students' long-term learning goals, covering the dimensions of course completion and test score improvement. The low-level reinforcement learning model gives immediate feedback based on the completion of short-term learning tasks, including test score improvement, increase in learning time and improvement in task completion efficiency. Through the optimization of the reward mechanism, students can continuously obtain feedback and incentives during the learning process and gradually improve their autonomous learning ability. Compared with the existing system that cannot fully utilize student feedback information, the present invention effectively improves the dynamic adjustment ability of learning paths and task planning, and enhances students' learning enthusiasm and the effect of improving their autonomous learning ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0089] Figure 1 A flowchart of a method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization proposed by the present invention;

[0090] Figure 2 This is a schematic diagram of the application of the reward mechanism of the high-level reinforcement learning model and the low-level reinforcement learning model in the method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization proposed by the present invention. DETAILED DESCRIPTION

[0091] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0092] refer to Figure 1-2 , a method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization, comprising the following steps:

[0093] S1. Collect student behavior data sets during the learning process, including study time, test scores, and course participation;

[0094] S2. Preprocess the student behavior data set, extract the student learning status, learning interest and cognitive level features in the student behavior data set, and form a learning feature set of individual students;

[0095] S3. Establish a learning path optimization model based on a hierarchical reinforcement learning algorithm. The learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model.

[0096] S4. Use Bayesian optimization to adjust key parameters in high-level reinforcement learning models and low-level reinforcement learning models;

[0097] S5. Based on the reward mechanism in the reinforcement learning algorithm, the high-level reinforcement learning model gives long-term rewards based on the achievement of long-term learning goals, and the low-level reinforcement learning model gives short-term rewards based on the completion of short-term learning tasks;

[0098] S6. At the end of each learning cycle, the student's learning behavior model is updated. The high-level reinforcement learning model adjusts the student's long-term learning goal according to the updated learning behavior model and the long-term learning features in the student behavior dataset. The low-level reinforcement learning model adjusts the short-term learning task according to the updated learning behavior model and the short-term learning features in the student behavior dataset. The updated learning path and strategy are globally optimized through Bayesian optimization.

[0099] S7. Based on the long-term learning goals of the high-level reinforcement learning model and the short-term learning tasks of the low-level reinforcement learning model, a cross-stage task connection mechanism is formed to automatically formulate learning strategies for students after each learning stage.

[0100] In this implementation, S1 specifically includes:

[0101] S11. Collect student behavior data during the learning process, including study time, test scores, and course participation:

[0102] Study time T i , records the total time each student spends studying in a specific period of time;

[0103] Test score S i , collect students' scores in each stage of the test, the test scores are expressed as scores, and each student's test score is S i ∈[0,100];

[0104] Course participation P i , records students’ participation in each course, which is defined as the ratio of students’ actual participation time to the total course time;

[0105] S12. Learning time T in student behavior data i , test score S i and course participation P i Record and store data to form student behavior data sets:

[0106]

[0107] In this implementation, S2 specifically includes:

[0108] S21, preprocess the student behavior dataset D to remove abnormal data and redundant information;

[0109] S22. Extract student learning status features L from student behavior dataset i , the student learning status feature evaluates the student's overall learning status by balancing study time and test scores, capturing the relationship between students' time investment and academic performance during the learning process:

[0110]

[0111] Among them, L i is the learning status characteristic of student i, T i is the study time of student i, S i is the test score of student i, α and β are weight coefficients, indicating the relative importance of learning time and test scores in the learning state. and are the sum of study time and test scores of all students respectively;

[0112] S23. Extracting students’ learning interest characteristics I i The learning interest feature evaluates students’ interest in participating in course learning by combining learning time and students’ course participation. When students’ course participation deviates greatly from the overall level, the interest value is adjusted through an exponential function to reflect students’ learning enthusiasm and interest changes:

[0113]

[0114] Among them, I i is the learning interest characteristics of student i, T i For students’ study time, T max is the maximum learning time among all students, λ· is the penalty factor for participation deviation, P i is the course participation of students, and P is the average course participation of all students;

[0115] S24. Extracting students’ cognitive level features C i The cognitive level feature evaluates the student's cognitive ability improvement by comparing the current test score with the changes in historical test scores, and quantifies the student's relative cognitive level by the ratio of the student's current test score to the highest test score in the group:

[0116]

[0117] Among them, C i is the cognitive level characteristic of student i, S i is the current test score, S prev is the score of the previous test, S maxis the highest score among all students, δ and ∈ are adjustment parameters;

[0118] S25, extract the learning state feature L i , Learning interest characteristics I i and cognitive level characteristics C i Combine to form the learning feature set F of individual students i ={L i ,I i ,C i}:

[0119]

[0120] In this implementation, S3 specifically includes:

[0121] S31. Establish a learning path optimization model using the learning feature set of individual students. The learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model.

[0122] S32, the high-level reinforcement learning model uses the student's long-term learning characteristics as the state space S H , state space S H From the learning feature set F of individual students i The action space A is extracted from the long-term learning goal, including the cumulative learning time, overall test scores and long-term learning interests. H , including curriculum planning and knowledge point mastery, through the strategy function π H (a|s) determines the state s∈S H The action a∈A is chosen H ;

[0123] S33. Use Bellman's optimal equation to calculate the value function V of the high-level reinforcement learning model based on long-term learning characteristics. H (s) to optimize:

[0124]

[0125] Among them, V H (s) is the value function of state s, R H (s,a) is the long-term reward for taking action a in state s, γ H is the high-level discount factor, P H (s ′ |s,a) is the state transition probability from state s to the next state s' through action a;

[0126] S34, the low-level reinforcement learning model uses students' short-term learning characteristics and real-time feedback data as the state space S L , state space S LContains the learning feature set F of individual students i ={L i ,I i ,C i The learning state feature L in i , Learning interest characteristics I i and cognitive level characteristics C i And real-time feedback data; take short-term learning tasks as action space A L , including daily learning content arrangement and specific knowledge point exercises, through the strategy function π L (a|s) determines the state s∈S L The action a∈A is chosen L ;

[0127] S35. The low-level reinforcement learning model plans short-term learning tasks according to the long-term goals set by the high-level reinforcement learning model. It uses the short-term learning features and real-time feedback data in the student behavior dataset to optimize the short-term learning tasks so that the short-term learning tasks are consistent with the students' current learning progress. The deep Q network algorithm is used to update the low-level value function Q. L (s,a):

[0128]

[0129] Among them, Q L (s,a) is the value function of taking action a in state s, α is the learning rate, r is the immediate reward, γ L is the low-level discount factor, s' is the next state after executing action a;

[0130] S36. Long-term learning goal A through high-level reinforcement learning model H and short-term learning tasks A of low-level reinforcement learning models L , combined with the individual learning feature set F i , forming a learning path optimization model.

[0131] In this implementation, S4 specifically includes:

[0132] S41. In the high-level reinforcement learning model, Bayesian optimization is used to adjust the weight parameter θ of the learning path H , the learning path weight parameter affects students’ long-term learning goals A H The choice of the optimal weight is selected through Bayesian optimization Maximize the high-level value function V H (s):

[0133]

[0134] Among them, θ His the weight parameter of the high-level learning path, Represents the feature space S learned from the long term H The state s sampled in;

[0135] S42, Bayesian optimization through Gaussian process regression model GP (μ, σ) modeling learning path weight parameter θ H With the high-level value function V H (s), where μ is the mean function, σ is the covariance function, and the next sampling point θ is selected by the expected improvement criterion H :

[0136]

[0137] S43. In the low-level reinforcement learning model, use Bayesian optimization to optimize the key parameter θ of the learning resource recommendation strategy L Make adjustments, including content relevance, difficulty level, and time allocation for learning resource recommendations, and determine the optimal parameters through Bayesian optimization Let the low-level value function Q L (s,a)maximize;

[0138] S44. Bayesian optimization models the parameters θ of the learning resource recommendation strategy through the Gaussian process regression model L With the low-level value function Q L (s, a) relationship, and select the optimal parameter θ using the expected improvement criterion L .

[0139] In this implementation, S5 specifically includes:

[0140] S51. The high-level reinforcement learning model gives long-term rewards based on the achievement of long-term learning goals. The long-term reward R H (s,a) consists of the following two parts:

[0141] Course Completion Reward R H,course (s,a), rewards are given by calculating the overall completion of the student's courses, defined as the proportion of courses completed by the student to the total number of courses:

[0142]

[0143] Among them, C completed The number of courses completed by the student, C total is the total number of courses;

[0144] Test score improvement reward R H,exam (s,a), rewards are calculated based on the student's improvement in each exam as the difference between the current exam score and the previous exam score:

[0145] RH,exam (s,a)=max(0,S i -S prev );

[0146] Among them, S i is the student’s current test score, S prev The student’s previous test scores;

[0147] S52. Add the course completion reward and the test score improvement reward to calculate the total long-term reward R in the high-level reinforcement learning model H (s,a):

[0148] R H (s,a)=α course ·R H,course (s,a)+α exam ·R H,exam (s,a);

[0149] Among them, α course and α exam They are the weight coefficients of course completion and test score improvement in long-term rewards;

[0150] S53, the low-level reinforcement learning model gives short-term rewards R according to the completion of short-term learning tasks L (s,a), short-term rewards include the following three parts:

[0151] Test score improvement reward R L,score (s,a), rewards students based on their improvement in scores on short tests:

[0152] R L,score (s,a)=max(0,S i -S prev,test );

[0153] Among them, S prev,test The result of the last test;

[0154] Learning time increase reward R L,time (s,a), rewards students based on the amount of time they spend studying each day:

[0155] R L,time (s,a)=log(1+T i );

[0156] Among them, T i for the amount of learning time students devote to their studies during the day;

[0157] Mission Completion Efficiency Reward R L,efficiency(s,a), rewards students based on their efficiency in completing daily tasks, defined as the ratio of the time taken to complete the task to the estimated time of the task:

[0158]

[0159] Among them, T estimated is the estimated completion time of the task, T actual The time actually spent on completing the task;

[0160] S54. Calculate the total short-term reward R in the low-level reinforcement learning model by adding the test score improvement reward, learning time increase reward and task completion efficiency reward. L (s,a):

[0161] R L (s,a)=β score ·R L,score (s,a)+β time ·R L,time (s,a)+β efficiency ·R L,efficiency (s,a);

[0162] Among them, β score , β time and β efficiency are the weight coefficients of test score improvement, study time increase and task completion efficiency in short-term rewards;

[0163] S55. Through the reward mechanism of S51-S54, the learning path optimization model adjusts students' learning paths and learning tasks in real time, so that students can continuously gain motivation and improve their autonomous learning ability during the learning process.

[0164] In this implementation, S6 specifically includes:

[0165] S61. After each learning cycle, the student learning behavior model is rebuilt and updated based on the latest student behavior data to reflect the changes in the student's learning status, interests and cognitive level in each learning cycle;

[0166] S62. Based on the updated learning behavior model, the high-level reinforcement learning model adjusts the long-term learning goals according to the students’ long-term learning characteristics, and dynamically updates the long-term learning goals based on the students’ historical learning behaviors and current learning progress, so that the learning path is consistent with the students’ cognitive development;

[0167] S63, the low-level reinforcement learning model adjusts the short-term learning tasks according to the short-term learning characteristics reflected in the latest learning behavior model and the real-time feedback data within the cycle, so that the short-term tasks are adapted to the students' current learning status and real-time feedback;

[0168] S64. By globally adjusting the strategies of the high-level reinforcement learning model and the low-level reinforcement learning model, and optimizing the learning path and strategy in combination with Bayesian optimization, the high-level reinforcement learning model and the low-level reinforcement learning model can dynamically optimize the learning path and resource recommendation strategy in each cycle.

[0169] Embodiment 1:

[0170] In Example 1, a key high school carried out a three-month student autonomous learning ability improvement plan from March to June 2023, using the "student autonomous learning ability improvement method based on reinforcement learning and Bayesian optimization" of the present invention. The test experimental group consisted of 100 first-year high school students. The system analyzed each student's learning behavior data, including study time, test scores, and course participation, and optimized their learning path in real time and recommended learning resources.

[0171] In the first month of the experiment, the system monitored the learning behavior data of student Zhang. In the first week of March, Zhang studied for 1.5 hours a day. The system extracted features from his learning data set, showing that his participation in mathematics courses was 60% and his test score was 65 points, indicating that his interest in learning was low. The system designed a personalized learning path optimization plan for Zhang based on the reinforcement learning model. The high-level reinforcement learning model set Zhang's long-term learning goal to improve his mathematics score to 75 points, while the low-level reinforcement learning model planned daily learning tasks for him, initially arranging him to study algebra for 1 hour and geometry for 30 minutes every day.

[0172] In the second week of March, through real-time feedback from the learning behavior model, the system monitored that Zhang's efficiency in completing daily learning tasks was only 75%, and that he had a lot of procrastination in the learning process. The system optimized the learning tasks based on the feedback data, increased the daily geometry practice time, and recommended more interactive learning resources through the Bayesian optimization algorithm, such as a geometry problem-solving platform with instant feedback.

[0173] On March 12, the system found that Zhang's completion rate was 80% when completing the geometry learning task, which was 5% higher than the previous week. By optimizing Zhang's learning path, the system adjusted the recommendation strategy of his learning resources to appropriately reduce the learning difficulty and combine the review content of basic concepts. At the same time, the low-level reinforcement learning model gradually shortened the duration of each learning task of Zhang based on real-time feedback data, and allocated more practice time to Zhang's weaker algebra part.

[0174] By March 24, Zhang's test score improved to 72 points. The system recorded that his study time increased to 2 hours a day and his task completion efficiency also increased to 90%. At this time, the system's reward mechanism came into play. By rewarding Zhang for his improved performance in algebra learning, he was further motivated to continue his daily learning tasks. The high-level reinforcement learning model updated his long-term learning goal to 80 points and adjusted the time allocation of learning tasks accordingly, encouraging Zhang to increase his learning intensity.

[0175] To further demonstrate the effectiveness of the present invention, the system recorded the performance of students in the experimental and control groups in real time throughout the three-month study program and conducted detailed data analysis.

[0176] Table 1 Data of Zhang, a student in the experimental group

[0177]

[0178] Li, a student in the control group, used the traditional online learning system to study. His learning tasks were not adjusted in real time during the entire experiment. Li studied for 1 hour in the first week and scored 66 points on the test. The system generated a fixed learning plan based on his initial learning performance and failed to adjust the learning path based on real-time feedback. During the entire learning process, Li's study time and task completion efficiency were relatively stable, but because the system did not dynamically optimize his learning interest and learning status, his learning effect was not ideal.

[0179] Table 2 Data of Li, a student in the control group

[0180]

[0181]

[0182] In this experiment, the study time, learning efficiency and test scores of Zhang in the experimental group were significantly higher than those of Li in the control group. In terms of learning path optimization and resource recommendation, the method of the present invention helped Zhang improve his math scores in a short period of time through dynamic feedback adjustment, and significantly improved his autonomous learning ability, while Li failed to achieve significant learning effects due to the lack of real-time feedback and optimization mechanism and the fixed learning path.

[0183] It can be seen from Example 1 that the dynamic learning path optimization and real-time feedback mechanism of the present invention can effectively solve the problems in the prior art of being unable to dynamically adjust the learning path and inefficient strategy search. The system continuously optimizes the learning strategy according to the individual learning characteristics of students, real-time feedback and Bayesian optimization algorithm, thereby improving learning efficiency and grades, and verifying the feasibility and effectiveness of the method of the present invention.

[0184] The present invention introduces high-level and low-level reinforcement learning models based on reinforcement learning to perform hierarchical management and dynamic optimization of students' long-term learning goals and short-term learning tasks. The high-level reinforcement learning model plans for students' long-term learning goals and makes real-time adjustments based on students' historical learning behavior models and the latest learning status; the low-level reinforcement learning model adjusts learning tasks based on real-time feedback from short-term learning tasks, so that the learning path can dynamically adapt to students' current learning status and feedback information. Different from the static learning path recommendation in the prior art, the present invention continuously updates the learning strategy through the reinforcement learning algorithm, and can continuously optimize the learning path according to students' learning performance and real-time feedback, thereby improving the flexibility and adaptability of the learning path.

[0185] The present invention combines the Bayesian optimization algorithm to efficiently adjust the key parameters in the high-level reinforcement learning model and the low-level reinforcement learning model. The Bayesian optimization can efficiently find the optimal learning strategy through the Gaussian process regression model with less feedback data, which solves the problems of low strategy search efficiency and excessive consumption of computing resources in the existing system. Through the global optimization of the learning path and the learning resource recommendation strategy, the present invention effectively shortens the strategy search time and improves the system's adaptability to complex learning scenarios, making personalized learning resource recommendations more accurate and responsive.

[0186] The present invention designs a continuous feedback mechanism based on long-term and short-term reward mechanisms in the reinforcement learning algorithm. The high-level reinforcement learning model gives long-term rewards according to the achievement of students' long-term learning goals, covering the dimensions of course completion and test score improvement. The low-level reinforcement learning model gives immediate feedback according to the completion of short-term learning tasks, including test score improvement, increase in learning time and improvement in task completion efficiency. Through the optimization of the reward mechanism, students can continuously obtain feedback and incentives in the learning process and gradually improve their autonomous learning ability. Compared with the existing system that cannot fully utilize student feedback information, the present invention effectively improves the dynamic adjustment capability of learning paths and task planning, and enhances students' learning enthusiasm and the effect of improving their autonomous learning ability.

[0187] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization, characterized in that: The steps include: S1. Collect student behavior data sets during the learning process, including study time, test scores, and course participation; S2. Preprocess the student behavior data set, extract the student learning status, learning interest and cognitive level features in the student behavior data set, and form a learning feature set of individual students; S3. Establish a learning path optimization model based on a hierarchical reinforcement learning algorithm. The learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model. S4. Use Bayesian optimization to adjust key parameters in high-level reinforcement learning models and low-level reinforcement learning models; S5. Based on the reward mechanism in the reinforcement learning algorithm, the high-level reinforcement learning model gives long-term rewards according to the achievement of long-term learning goals, and the low-level reinforcement learning model gives short-term rewards according to the completion of short-term learning tasks; S6. At the end of each learning cycle, the student's learning behavior model is updated. The high-level reinforcement learning model adjusts the student's long-term learning goal according to the updated learning behavior model and the long-term learning features in the student behavior dataset. The low-level reinforcement learning model adjusts the short-term learning task according to the updated learning behavior model and the short-term learning features in the student behavior dataset. The updated learning path and strategy are globally optimized through Bayesian optimization. S7. Based on the long-term learning goals of the high-level reinforcement learning model and the short-term learning tasks of the low-level reinforcement learning model, a cross-stage task connection mechanism is formed to automatically formulate learning strategies for students after each learning stage.

2. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S1 specifically includes: S11. Collect student behavior data during the learning process, including study time, test scores, and course participation: Study time T i , records the total time each student spends studying in a specific period of time; Test score S i , collect students' scores in each stage of the test, the test scores are expressed as scores, and each student's test score is S i ∈[0,100]; Course participation P i , records students’ participation in each course, which is defined as the ratio of students’ actual participation time to the total course time; S12, the learning time T in the student behavior data i , test score S i and course participation P i Record and store data to form student behavior data sets:

3. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S2 specifically includes: S21, preprocess the student behavior dataset D to remove abnormal data and redundant information; S22. Extract student learning status features L from student behavior dataset i , the student learning status feature evaluates the student's overall learning status by balancing study time and test scores, capturing the relationship between students' time investment and academic performance during the learning process: Among them, L i is the learning status characteristic of student i, T i is the study time of student i, S i is the test score of student i, α and β are weight coefficients, indicating the relative importance of learning time and test scores in the learning state. and are the sum of study time and test scores of all students respectively; S23. Extracting students’ learning interest characteristics I i The learning interest feature evaluates students’ interest in participating in course learning by combining learning time and students’ course participation. When students’ course participation deviates greatly from the overall level, the interest value is adjusted through an exponential function to reflect students’ learning enthusiasm and interest changes: Among them, I i is the learning interest characteristics of student i, T i For students’ study time, T max is the maximum learning time among all students, λ· is the penalty factor for participation deviation, P i For students' course participation, is the average course participation of all students; S24. Extracting students’ cognitive level features C i The cognitive level feature evaluates the student's cognitive ability improvement by comparing the current test score with the changes in historical test scores, and quantifies the student's relative cognitive level by the ratio of the student's current test score to the highest test score in the group: Among them, C i is the cognitive level characteristic of student i, S i is the current test score, S prev is the score of the previous test, S max is the highest score among all students, δ and ∈ are adjustment parameters; S25, extract the learning state feature L i , Learning interest characteristics I i and cognitive level characteristics C i Combined to form the learning feature set F of individual students i ={L i ,I i ,C i }:

4. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S3 specifically includes: S31. Establishing a learning path optimization model using the learning feature set of individual students, wherein the learning path optimization model includes a high-level reinforcement learning model and a low-level reinforcement learning model; S32, the high-level reinforcement learning model uses the student's long-term learning characteristics as the state space S H , state space S H From the learning feature set F of individual students i The action space A is extracted from the long-term learning goal, including the cumulative learning time, overall test scores and long-term learning interests. H , including curriculum planning and knowledge point mastery, through the strategy function π H (a|s) determines the state s∈S H The action a∈A is chosen H ; S33. Use Bellman's optimal equation to calculate the value function V of the high-level reinforcement learning model based on long-term learning characteristics. H (s) to optimize: Among them, V H (s) is the value function of state s, R H (s,a) is the long-term reward for taking action a in state s, γ H is the high-level discount factor, P H (s ′ |s,a) is the state transition probability from state s to the next state s' through action a; S34, the low-level reinforcement learning model uses students' short-term learning characteristics and real-time feedback data as the state space S L , state space S L Contains the learning feature set F of individual students i ={L i ,I i ,C i The learning state feature L in i , Learning interest characteristics I i and cognitive level characteristics C i And real-time feedback data; take short-term learning tasks as action space A L , including daily learning content arrangement and specific knowledge point exercises, through the strategy function π L (a|s) determines the state s∈S L The action a∈A is chosen L ; S35. The low-level reinforcement learning model plans short-term learning tasks according to the long-term goals set by the high-level reinforcement learning model. It uses the short-term learning features and real-time feedback data in the student behavior dataset to optimize the short-term learning tasks so that the short-term learning tasks are consistent with the students' current learning progress. The deep Q network algorithm is used to update the low-level value function Q. L (s,a): Among them, Q L (s,a) is the value function of taking action a in state s, α is the learning rate, r is the immediate reward, γ L is the low-level discount factor, s' is the next state after executing action a; S36. Long-term learning goal A through high-level reinforcement learning model H and short-term learning tasks A of low-level reinforcement learning models L , combined with the individual learning feature set F i , forming a learning path optimization model.

5. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S4 specifically includes: S41. In the high-level reinforcement learning model, Bayesian optimization is used to adjust the weight parameter θ of the learning path H , the learning path weight parameters affect students' long-term learning goals A H The choice of the optimal weight is selected through Bayesian optimization Maximize the high-level value function V H (s): Among them, θ H is the weight parameter of the high-level learning path, Represents the long-term learning feature space S H The state s sampled in; S42, Bayesian optimization through Gaussian process regression model GP (μ, σ) modeling learning path weight parameter θ H With the high-level value function V H (s), where μ is the mean function, σ is the covariance function, and the next sampling point θ is selected by the expected improvement criterion H : S43. In the low-level reinforcement learning model, use Bayesian optimization to optimize the key parameter θ of the learning resource recommendation strategy L Make adjustments, including content relevance, difficulty level, and time allocation for learning resource recommendations, and determine the optimal parameters through Bayesian optimization Let the low-level value function Q L (s,a)maximize; S44. Bayesian optimization models the parameters θ of the learning resource recommendation strategy through the Gaussian process regression model L With the low-level value function Q L (s, a) relationship, and select the optimal parameter θ using the expected improvement criterion L .

6. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S5 specifically includes: S51. The high-level reinforcement learning model gives long-term rewards based on the achievement of long-term learning goals. The long-term reward R H (s,a) consists of the following two parts: Course Completion Reward R H,course (s,a), rewards are given by calculating the overall completion of the student's courses, defined as the proportion of courses completed by the student to the total number of courses: Among them, C completed The number of courses completed by the student, C total is the total number of courses; Test score improvement reward R H,exam (s,a), rewards are calculated based on the student's improvement in each exam as the difference between the current exam score and the previous exam score: R H,exam (s,a)=max(0,S i -S prev ); Among them, S i is the student’s current test score, S prev The student’s previous test scores; S52. Add the course completion reward and the test score improvement reward to calculate the total long-term reward R in the high-level reinforcement learning model H (s,a): R H (s,a)=a course ·R H,course (s,a)+a exam ·R H,exam (s,a); Among them, α course and α exam They are the weight coefficients of course completion and test score improvement in long-term rewards; S53, the low-level reinforcement learning model gives short-term rewards R according to the completion of short-term learning tasks L (s,a), the short-term reward includes the following three parts: Test score improvement reward R L,score (s,a), rewards students based on their improvement in scores on short tests: R L,score (s,a)=max(0,S i -S prev,test ); Among them, S prev,test The result of the last test; Learning time increase reward R L,time (s,a), rewards students based on the amount of time they spend studying each day: R L,time (s,a)=log(1+T i ); Among them, T i for the amount of learning time students devote to the day; Mission Completion Efficiency Reward R L,efficiency (s,a), rewards students based on their efficiency in completing daily tasks, defined as the ratio of the time taken to complete the task to the estimated time of the task: Among them, T estimated is the estimated completion time of the task, T actual The time actually spent on completing the task; S54. Calculate the total short-term reward R in the low-level reinforcement learning model by adding the test score improvement reward, learning time increase reward and task completion efficiency reward. L (s,a): R L (s,a)=β score ·R L,score (s,a)+β time ·R L,time (s,a)+β efficiency ·R L,efficiency (s,a); Among them, β score , β time and β efficiency are the weight coefficients of test score improvement, study time increase and task completion efficiency in short-term rewards; S55. Through the reward mechanism of S51-S54, the learning path optimization model adjusts students' learning paths and learning tasks in real time, so that students can continuously gain motivation and improve their autonomous learning ability during the learning process.

7. The method for improving students' autonomous learning ability based on reinforcement learning and Bayesian optimization according to claim 1 is characterized in that: The S6 specifically includes: S61. After each learning cycle, the student learning behavior model is rebuilt and updated based on the latest student behavior data to reflect the changes in the student's learning status, interests and cognitive level in each learning cycle; S62. Based on the updated learning behavior model, the high-level reinforcement learning model adjusts the long-term learning goals according to the students’ long-term learning characteristics, and dynamically updates the long-term learning goals based on the students’ historical learning behaviors and current learning progress, so that the learning path is consistent with the students’ cognitive development; S63, the low-level reinforcement learning model adjusts the short-term learning tasks according to the short-term learning characteristics reflected in the latest learning behavior model and the real-time feedback data within the cycle, so that the short-term tasks are adapted to the students' current learning status and real-time feedback; S64. By globally adjusting the strategies of the high-level reinforcement learning model and the low-level reinforcement learning model, and optimizing the learning path and strategy in combination with Bayesian optimization, the high-level reinforcement learning model and the low-level reinforcement learning model can dynamically optimize the learning path and resource recommendation strategy in each cycle.

Images