Intelligent learning path planning method and system based on reinforcement learning

Abstract

The application relates to the technical field of intelligent learning path planning, and discloses an intelligent learning path planning method and system based on reinforcement learning. The method comprises the following steps: collecting user learning behavior data, and extracting a multi-dimensional feature vector containing a knowledge mastery index, a learning time length distribution curve and error type clustering results; according to the knowledge mastery index, a prior learning path set with similar feature vectors is matched in a historical learning database; based on the learning time length distribution curve of the set, a reinforcement learning model state transition probability matrix reflecting the migration law of different learning stages is constructed; the current learning task completion degree and the error type clustering result change trend of the user are monitored in real time, and the posterior distribution parameters of the matrix are updated; a dynamic learning sequence is generated according to the updated matrix, and is output to a learning path execution terminal. The method combines multi-dimensional user data, historical experience and real-time state adjustment paths, and adapts to the learning characteristics and dynamic needs of the user.

Description

Technical Field

[0001] This invention relates to the field of intelligent learning path planning technology, specifically to an intelligent learning path planning method and system based on reinforcement learning. Background Technology

[0002] In current educational settings, learning path planning often relies on fixed templates or teacher experience, making it difficult to adapt to the diverse learning characteristics of different users. Traditional models employ a uniform order and pace of learning content regardless of differences in users' knowledge base or learning rhythm. This leads to some users repeatedly investing time in knowledge points they have already mastered, while neglecting targeted learning for their weaker areas. For example, in subject-based learning, students in the same class may have significantly different levels of mastery of basic concepts, but current planning methods still require all students to complete the same exercises and lessons simultaneously, failing to adjust the depth and order of content based on individual circumstances.

[0003] As the concept of intelligent education advances, some solutions attempt to introduce data-driven path planning. However, most focus only on single-dimensional data, such as judging knowledge mastery solely based on quiz accuracy or allocating learning tasks based solely on study time. These solutions fail to integrate multi-dimensional information to construct a comprehensive user learning profile. The learning paths generated by such solutions are often one-sided. For example, they might recommend advanced content based solely on accuracy, ignoring the user's insufficient study time or lack of depth of understanding on that knowledge point, leading to frequent knowledge gaps in subsequent learning.

[0004] Existing intelligent learning planning solutions lack effective utilization of historical learning experience. While the vast amount of user learning data accumulated in the education field contains rich information on the correlation between path selection and learning outcomes, most solutions lack a systematic historical database and an effective mechanism for matching similar users. Each planning attempt requires starting from scratch, increasing computational costs and making it difficult to learn from previously validated learning path patterns. Furthermore, although some solutions introduce dynamic adjustment concepts, the adjustments are often based on static thresholds, such as setting "return to review after three consecutive errors." This fails to capture the dynamic trends in the user's learning process in real time. For example, when a user's error type shifts from conceptual confusion to calculation errors, the focus of the learning content cannot be adjusted accordingly, resulting in path adjustments lagging behind the user's actual learning needs.

[0005] Reinforcement learning has demonstrated its advantages in sequential decision-making, continuously optimizing strategies through interaction with the environment. However, its application in learning path planning remains insufficient. Existing solutions combining reinforcement learning often fail to construct state transition models tailored to the learning scenario, simply applying general algorithm frameworks. This fails to accurately reflect the transfer patterns between different learning stages, such as the transition conditions from basic concept learning to comprehensive application practice, and the connection logic between different knowledge modules. Consequently, the paths output by reinforcement learning models become disconnected from actual learning patterns, making it difficult to effectively support the user's continuous learning process. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent learning path planning method and system based on reinforcement learning to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, this invention provides an intelligent learning path planning method based on reinforcement learning, the method comprising: Collect user learning behavior data and extract multidimensional feature vectors, which include knowledge mastery index, learning duration distribution curve and error type clustering results; Based on the knowledge mastery index in the multidimensional feature vector, a set of prior learning paths with similar feature vectors is matched in the historical learning database; Based on the learning duration distribution curve in the prior learning path set, a state transition probability matrix of the reinforcement learning model is constructed, which reflects the transfer pattern between different learning stages. Real-time monitoring of the changing trends of the user's current learning task completion rate and error type clustering results, and updating the posterior distribution parameters in the state transition probability matrix; A dynamic learning sequence is generated based on the updated state transition probability matrix and output to the learning path execution terminal.

[0008] Preferably, the step of collecting user learning behavior data and extracting multidimensional feature vectors includes: The system collects timestamp sequences of user interactions using embedded sensors to generate raw behavior logs. Semantic parsing is performed on the operation types in the original behavior logs to divide them into knowledge unit tags and calculate the cumulative learning time under each tag; Extract the incorrect question numbers and error reason codes from the user's answer records, and use a density clustering algorithm to generate error type clustering results; The clustering results of the knowledge unit labels, cumulative learning time, and error types are aligned according to the time window and fused into the multidimensional feature vector.

[0009] Preferably, the set of prior learning paths with similar feature vectors in the matching historical learning database includes: From the historical learning database, a set of candidate paths whose knowledge unit tag overlap exceeds a preset threshold is selected; Calculate the dynamic time warping distance between the learning duration distribution curve of each path in the candidate path set and the user's current learning duration distribution curve; The candidate path set is sorted according to the dynamic time warping distance, and the top N paths with the smallest distance are selected as the prior learning path set.

[0010] Preferably, the state transition probability matrix for constructing the reinforcement learning model includes: The stage division nodes of each path in the prior learning path set are analyzed to generate a stage transfer relationship topology graph; Statistically count the migration frequency between adjacent stages in the topology graph, and initialize the baseline parameters of the state transition probability matrix; The abnormal migration probability weights in the baseline parameters are adjusted based on the distribution of interruption locations in the user's historical learning behavior data.

[0011] Preferably, updating the posterior distribution parameters in the state transition probability matrix includes: Detect the deviation between the user's current learning task completion rate and the expected progress, and calculate the gradient influence coefficient of the deviation on the state transition probability matrix; When a high-frequency error category is added to the error type clustering result, the associated state node in the state transition probability matrix is ​​marked. The transition weights of the state transition probability matrix are dynamically adjusted based on the gradient influence coefficient and the distribution density of the associated state nodes.

[0012] Preferably, the generation of dynamic learning sequences includes: Monte Carlo tree search is performed based on the updated state transition probability matrix to generate candidate learning action sequences; For each action node in the candidate learning action sequence, calculate the ratio of knowledge coverage gain to time cost; The action node with the highest ratio is selected as the target task for the next learning stage, and the dynamic learning sequence is generated by prioritizing it.

[0013] Preferably, the method further includes a feedback calibration phase after executing the dynamic learning sequence, comprising: Record the user's actual knowledge test score and time consumption after completing the dynamic learning sequence; The actual knowledge test score is compared with the expected score range to generate a knowledge coverage error signal. Based on the deviation ratio between the time consumption data and the expected time consumption, a time allocation error signal is generated; The knowledge weight parameters in the state transition probability matrix are adjusted using the knowledge coverage error signal. The time allocation error signal is used to optimize the task duration allocation strategy in the dynamic learning sequence.

[0014] Preferably, the adjustment of the knowledge weight parameters in the state transition probability matrix includes: Identify the systematic deviation components in the knowledge coverage error signal and calculate the knowledge weight compensation amount using the moving average method; The migration probability value of the corresponding knowledge unit in the state transition probability matrix is ​​scaled proportionally according to the compensation amount.

[0015] Preferably, the optimized task duration allocation strategy in the dynamic learning sequence includes: Extract the timeout task identifier from the time allocation error signal and trace its context node in the dynamic learning sequence; Shorten the standard time threshold of adjacent tasks in the context node that have a dependency on the timed-out task.

[0016] Preferably, the present invention also includes an intelligent learning path planning system based on reinforcement learning, the system including a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the program to implement the steps in the method described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This method collects user learning behavior data and extracts multi-dimensional feature vectors containing knowledge mastery indicators, learning time distribution curves, and error type clustering results. It no longer relies on a single data dimension to judge a user's learning status, but comprehensively considers the user's actual mastery of knowledge, the pattern of learning time allocation, and the characteristics of common error types to form a comprehensive user learning profile. For example, for users with moderate knowledge mastery, concentrated learning time in the evening, and whose errors are mostly logical deduction mistakes, the planned learning path will focus on logic training content and match key learning periods to the user's most efficient learning window in the evening, ensuring the path is highly adapted to the user's learning foundation, time habits, and weaknesses.

[0018] This method leverages prior learning paths with similar feature vectors in a historical learning database, effectively drawing upon past successful learning path experiences and reducing the randomness of path planning. The historical database stores a wealth of information on the correlation between user learning paths and learning outcomes, providing mature models for current user path planning, eliminating the need to start from the initial state each time. For example, when a new user's knowledge mastery indicators and learning duration distribution are highly similar to a certain type of user in the historical database, the effective prior path set validated by that type of user can be directly referenced. Adjustments can then be made based on the current user's specific circumstances, shortening the path planning cycle and improving the rationality of the initial path.

[0019] The state transition probability matrix of the reinforcement learning model is constructed based on the learning duration distribution curve in the prior learning path set. This enables the model to accurately reflect the transfer patterns between different learning stages, ensuring that path planning conforms to the inherent logic of the learning process. The learning duration distribution curve contains the characteristics of users' time investment and progress connection patterns at different learning stages. The state transition probability matrix built on this basis can quantify the reasonable conditions and probabilities of processes such as "transfer from the basic knowledge learning stage to the comprehensive application stage" and "transfer from the concept understanding stage to the exercise training stage." This ensures that the paths generated by the reinforcement learning model follow the progressive laws of learning cognition, avoiding knowledge jumps or chaotic stage connections, and helping users gradually build a coherent knowledge system.

[0020] This method monitors the changes in the user's current learning task completion rate and error type clustering results in real time and updates the posterior distribution parameters in the state transition probability matrix, enabling the learning path to dynamically adapt to changes in the user's learning state. During the learning process, the user's mastery of knowledge gradually improves, and error types may also change as learning deepens, for example, from initial conceptual confusion to later detail oversight. This method captures these changes in real time and adjusts the parameters of the state transition probability matrix accordingly, thereby adjusting the direction of the subsequent learning path. For example, when it detects that the user's task completion rate for a certain knowledge point reaches the expected level and the error type changes from high-frequency to occasional, the matrix parameters will be adjusted accordingly, driving the path from the consolidation stage of that knowledge point to the learning stage of new knowledge points; if it detects an increase in the frequency of a certain type of error, the parameters will be adjusted to add specialized learning content for that type of error, ensuring that the path always remains consistent with the user's real-time learning needs.

[0021] Dynamic learning sequences are generated based on the updated state transition probability matrix and output to the learning path execution terminal, allowing users to access tailored learning content in real time and improving the coherence and relevance of the learning process. The execution terminal transforms the dynamic learning sequences into actionable content such as specific course schedules and practice tasks. Users can progress step-by-step by following the content pushed by the system without manually adjusting their learning plans. Furthermore, each pushed content is generated based on the latest learning status analysis, ensuring that users are always within an appropriate range of learning difficulty and pace. This avoids problems such as low learning efficiency or excessive learning pressure caused by a fixed learning path, making the learning process more adaptive and flexible. Attached Figure Description

[0022] Figure 1 This is a schematic diagram illustrating the working principle of the intelligent learning path planning method based on reinforcement learning described in this invention. Figure 2 A flowchart for collecting user learning behavior data and extracting multidimensional feature vectors; Figure 3 A flowchart for updating the posterior distribution parameters in the state transition probability matrix; Figure 4 This is a comprehensive analysis diagram of learning path effectiveness and knowledge state transition. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figure 1This invention provides an intelligent learning path planning method based on reinforcement learning. The method includes: continuously capturing user interaction events through a data acquisition module deployed on a learning terminal. These events include, but are not limited to, the start and end times of video viewing, the correctness and time spent answering exercises, page dwell time, and note-taking frequency. After cleaning and preprocessing, the raw data is transformed into a structured multidimensional feature vector. This vector contains three core components: a knowledge mastery index calculated by evaluating the user's answer accuracy and response speed in specific knowledge units; a learning time distribution curve recording the statistical patterns of time spent by the user on different learning content; and error type clustering results, which use an unsupervised learning algorithm to classify error patterns. The system accesses a historical learning database, which stores a large number of anonymized historical learning path records. By calculating the overlap of knowledge unit labels and the similarity of learning time curves between the current user's multidimensional feature vector and the historical path feature vectors, the system selects the set of prior learning paths with the highest matching degree. Building upon this foundation, the reinforcement learning model is initialized. Its core component, the state transition probability matrix, is constructed by analyzing the implicit learning stage transition patterns within the prior path set. This matrix quantifies the tendency to transition from the current learning state to the next possible state. During the user's actual learning process, the system monitors the dynamic changes in task completion progress and error types in real time. Using this observational data, it continuously adjusts the parameters in the state transition probability matrix using a Bayesian update method to better reflect the user's current situation. Based on the updated probability model, the path planning engine uses a search algorithm to generate a dynamic and personalized sequence of learning actions. This sequence is encapsulated as an instruction set and sent to the user's learning interface to drive their next learning activity.

[0025] Example 1: See Figure 2The initial stage of data collection relies on a variety of sensing modules deeply integrated into the application layer of the learning terminal. These modules are not simple loggers, but listening programs that can continuously capture low-level interactive events. They record the movement speed of each mouse trajectory, the frequency of pause and fast forward operations during video playback, the gesture swipe angle on the touch screen, and the keystroke interval sequence when typing on the keyboard with millisecond precision. All these raw event streams are timestamped synchronously with a high-precision clock to form a raw behavior log arranged in strict chronological order. This log is like a digital microscope, recording the micro-behavior of the user's interaction with the learning content in all aspects. The subsequent semantic parsing process of the raw behavior logs requires the assistance of a pre-built knowledge graph mapping system. This knowledge graph anchors all learning resources within the platform to a standardized knowledge unit system. The parsing engine scans every event in the logs. For example, when the event type is "video player status change" and the parameter is "play", the engine queries the knowledge point identifier marked in the video metadata and maps it to a structured label such as "middle school mathematics - algebra - quadratic equation root-finding formula". For exercise answering events, the engine not only records the question ID, but also obtains multiple knowledge unit tags tested by the question by querying the question knowledge base and assigns weights. The core task of semantic parsing is to elevate these scattered, low-level behavior events into knowledge unit access records with clear pedagogical significance and accurately count their duration within a specific time window.

[0026] In terms of error data processing, the system receives structured answer records from the practice and assessment module. Each record contains a unique question identifier, the user-submitted answer options or text, the standard answer, and an error classification coding system predefined by the system or teacher. This coding system may include categories such as "C01 - Conceptual Error", "C02 - Calculation Error", and "C03 - Oversight in Reading the Question". Density clustering algorithm is applied to these error record sets with multi-dimensional labels. This algorithm does not require pre-specifying the number of categories. It automatically identifies dense areas by calculating the density distribution of data points in the feature space. Records that are similar in dimensions such as error cause coding, related knowledge points, and error occurrence timestamps are grouped into the same cluster. Each cluster generates a vector descriptor representing its central features, thus forming the error type clustering result. This process can adaptively discover potential systemic knowledge misunderstandings or habitual error patterns that users may have. The fusion operation of multidimensional feature vectors is performed synchronously within a sliding time window, which is usually set to the duration of a complete learning session, such as 30 minutes or a continuous learning period. The fusion module summarizes all knowledge unit labels generated within the window and their corresponding cumulative learning durations, and associates the clustering results of the most significant error types within the window (usually based on cluster size or frequency of occurrence). Finally, a fixed-dimensional feature vector is generated. This vector slides and updates continuously in the time dimension, thus forming a dynamically evolving multidimensional feature trajectory, providing a snapshot sequence of changes over time for subsequent similarity matching.

[0027] In the process of matching similar paths in the historical learning database, the system first performs a coarse screening based on the knowledge scope. It calculates the Jaccard similarity coefficient between the set of knowledge unit labels in the current user's feature vector and the set of knowledge unit labels covered by each historical path in the database. This coefficient measures the ratio of the intersection to the union of the two sets. Only when this ratio exceeds a configurable threshold (e.g., 0.7) is the historical path included in the candidate set. This step ensures that subsequent similarity comparisons are conducted between paths with a considerable degree of overlap in teaching content, avoiding meaningless comparisons between paths for learning trigonometric functions and paths for learning ancient literature. For the candidate path set that passes the coarse screening, the system initiates a more refined learning pattern similarity measurement. Here, the dynamic time warping algorithm is used to compare the user's current learning duration distribution curve with the complete duration distribution curve of the candidate historical path. The learning duration distribution curve is a type of time series data that reflects the intensity distribution of learning activities on the time axis. The dynamic time warping algorithm can effectively handle the nonlinear scaling and local deformation that may exist between the two curves on the time axis. It aligns the two sequences by finding an optimal curved path to minimize the sum of the distances between corresponding points in the sequences. This minimum cumulative distance is the similarity measure between the two. The smaller the distance value, the more similar the learning rhythm and time allocation pattern represented by the two curves are.

[0028] After calculating the similarity of all candidate paths, the system sorts the candidate paths in ascending order of dynamic time regularization distance and selects the top N paths to form a prior learning path set. This set contains the successful experiences of learners who are most similar to the current user in terms of knowledge structure, learning pace, and time allocation patterns. It provides a valuable and highly referenceable template library for the subsequent construction of reinforcement learning models and state transition probability matrices. The entire matching process, through multi-level screening and precise measurement, ensures that the recommended prior paths are not only relevant in content but also highly consistent in learning behavior patterns.

[0029] Example 2: Constructing the core component of the reinforcement learning model, the state transition probability matrix. The construction of this matrix is ​​not a simple statistical induction, but a complex process integrating historical path pattern analysis and abnormal behavior identification. The starting point for construction is a deep analysis of the set of prior learning paths obtained in the above examples. Each prior path is a time series composed of continuous learning activities, each bound to a specific knowledge unit objective. The system needs to identify meaningful "stage" dividing nodes from these seemingly continuous sequences. Stage division is not simply based on equal time division, but on various heuristic rules. For example, when a significant jump occurs in the knowledge unit involved in the learning activity, when a formal unit test or evaluation point appears in the path record, or when a long rest interval is detected, these nodes are considered the boundaries of the learning stages. By analyzing the stage nodes of all prior paths, the system can construct a stage transition relationship topology graph. This topology graph is a directed graph structure, where nodes represent different learning stages, and directed edges between nodes represent historical instances where a student directly enters the next stage from the previous stage.

[0030] After establishing the stage transition relationship topology graph, the system begins large-scale frequency statistics to initialize the baseline parameters of the state transition probability matrix. The statistical process targets each node in the topology graph (i.e., each learning stage). ), traverse all prior paths, and accurately calculate from Starting from there, the next stage involves migrating to every other possible node in the graph. The frequency of (including itself, i.e., remaining at the current stage for consolidation), for example, assuming there are 100 historical paths that have gone through a stage. The next stage for 60 of these paths is... 20 items are 10 are There are also 10 records showing that the user's learning activities were interrupted after this stage, so from arrive The migration frequency is 60, to It's 20, to It is 10, the interrupt is 10, and the state transition probability matrix is ​​from... arrive Baseline probability Then initialized to Similarly, Initialized to 0.2. Initializing to 0.1, this maximum likelihood estimation-based initialization method ensures that the initial state of the matrix reflects the most common path selection tendencies in historical data. However, relying solely on transition frequency is insufficient to build a robust model; the system also needs to consider anomalies during the learning process, particularly abnormal interruptions. Therefore, a baseline parameter adjustment mechanism based on the distribution of interruption locations is introduced. The system separately analyzes unplanned interruption events recorded in historical learning behavior data (such as users actively exiting before completing a learning phase, or not logging in for an extended period after performing extremely poorly in a phase evaluation), locating the user's learning phase before these interruption events occurred. Assuming that in the above example, from... Of the 10 interruptions recorded from the start of the phase, 7 occurred when attempting to start from... Migrate to Shortly thereafter, this high proportion of interrupted correlations indicated that from arrive This transition path may pose a high learning risk or cognitive overload problem. To mitigate such risks during path planning, the system penalizes the corresponding transition weights in the state transition probability matrix by lowering them. For example, based on the initial probabilities, the system adjusts the weights accordingly. A decay factor is applied to reduce its probability value. Simultaneously, to prevent the sum of probabilities from being non-zero, the reduced probability value is proportionally redistributed to other migration options (such as...). and Alternatively, a transition probability pointing to the "consolidation and review stage" can be added. In this way, the state transition probability matrix not only encodes the information of "how to usually go", but also internalizes the lessons learned of "which paths are easy to fail", making it more robust in path recommendation.

[0031] The construction of the state transition probability matrix also needs to consider the abstraction level and granularity of the learning stage itself. The system maintains a multi-layered learning stage system, from macroscopic knowledge modules to microscopic knowledge points. The state transition probability matrix can also be constructed at multiple granularities and work together. For example, a macroscopic matrix describes the transition probability from the "algebra module" to the "geometry module," while another microscopic matrix describes the transition probability from the "factorization" stage to the "completion" stage within the "algebra module." In specific planning, the system may first use the macroscopic matrix to determine the general direction, and then use the microscopic matrix to plan the specific steps. The final generated state transition probability matrix is ​​a sparse matrix because there is not a direct transition relationship between any two learning stages. Each element of the matrix carries the statistical regularity and risk adjustment information of the historical path. As an environmental model of the reinforcement learning agent, it provides a probabilistic basis for subsequent real-time monitoring, dynamic updates, and optimal sequence search, making the planning of the learning path no longer static and preset, but a dynamic decision-making process based on probabilistic reasoning.

[0032] Example 3: See Figure 3 The system initiates a high-frequency monitoring loop during the user's learning process. This loop samples user interaction data with the learning content at second-level intervals, including page dwell time, video completion rate, and the correctness and reaction time of answers to exercises. These raw data streams are cleaned and aggregated by the real-time processing module, transforming them into quantifiable progress indicators. The system internally pre-defines an ideal learning trajectory based on a cognitive science model. This trajectory not only specifies the recommended completion time for each knowledge unit but also defines the threshold for the level of mastery that should be achieved. The real-time monitoring module calculates a multi-dimensional progress deviation vector by comparing the user's current actual state with the state of the corresponding point on the ideal trajectory. Each component of this vector reflects the degree of advancement or lag in a specific knowledge dimension or skill element. Error pattern identification runs continuously as an independent asynchronous process. The system maintains an error event buffer within a sliding time window. Whenever a new error event is recorded, it triggers an incremental clustering analysis. This analysis uses a density-based algorithm to discover clustering phenomena of error events in the feature space. Error features include the knowledge point to which the error belongs, the error type encoding, and even the length of hesitation time when answering. When the event density of a local area exceeds a dynamic threshold within a short period of time, the system determines that a new error cluster is forming and sends a signal to the core decision engine. This signal contains the feature center vector of this error cluster and its density growth rate. This information indicates that a new learning obstacle that needs attention may have emerged.

[0033] The update process of the state transition probability matrix is ​​essentially a Bayesian learning process, where the system updates the observed progress deviation vector. Incorrect clustering signals are considered as new evidence, which is used to update beliefs about the user's hidden cognitive states, thereby revising the estimates of transition probabilities between states, for each element in the matrix. That is, from the state Transferred to The system assesses the conditional probability of new evidence supporting the likelihood of this transition occurring. This assessment is based on a probabilistic model that describes the probability of generating various types of observational data in a specific cognitive state. For example, if continuous observations are made in the relevant state... If there is a lag in progress on a knowledge point and it is accompanied by a specific error pattern, then all knowledge points directly dependent on... As solid prior knowledge, subsequent state transition probabilities, such as The probability of transitions may be lowered because evidence suggests that users may not be ready to make such transitions. Correspondingly, the probability of transitions to state transitions that point to consolidation, review, or targeted remedies will be increased. The entire update process must ensure the normalization of the probability distribution, that is, the sum of all transition probabilities from any state remains 1. The generation of dynamic learning sequences is modeled as a sequence decision problem and solved using the Monte Carlo Tree Search algorithm. This algorithm evaluates the long-term value of different decisions by simulating a large number of possible learning paths. The root node of the search tree represents the current learning state, and each branch represents an optional learning action. The algorithm iteratively executes four steps: selection, expansion, simulation, and backtracking. In the selection phase, the algorithm starts from the root node and selects a path using a strategy that balances exploration and utilization (such as the UCT algorithm) based on the number of visits to each child node and the average utility value, until it reaches a node that has not yet been fully explored. In the expansion phase, one or more child nodes representing new possible actions are added to the node. In the simulation phase, starting from the newly expanded node, a complete learning path is quickly simulated according to a simple default strategy (such as randomly selecting actions) until the endpoint. In the backtracking phase, the cumulative utility value obtained from this simulation is propagated back along the visited node path, updating the statistical information of each node on the path.

[0034] To quantify the merits of a simulated path, a utility function needs to be defined. To evaluate this, the function needs to comprehensively consider the knowledge value gained from the path and the time cost incurred, and its calculation result should be a dimensionless scalar value that can be directly compared. The utility function is defined as follows:

[0035] in: The total utility of this simulated learning path is represented by a dimensionless pure number. This represents the total number of learning stages included in the simulation path; it is a count. It is a discount factor between 0 and 1, dimensionless, that discounts the value of learning gains further away from the present. Representative at the The value of knowledge gained in each learning stage is a dimensionless value calculated based on the global importance weight of knowledge points and the expected improvement in mastery at that stage. It is the estimated total time required to complete the entire simulated path, and has the dimension of time. It is a pre-defined reference time unit (e.g., 1 standard study hour), which also has a time dimension, therefore the denominator is... This represents the total time as a multiple of the reference time, and is a dimensionless value. Therefore, both the numerator and denominator of the entire formula are dimensionless, resulting in the calculated utility. It is also dimensionless, and its physical meaning is "the discounted value of knowledge obtained per unit relative time".

[0036] After numerous simulations and iterations, the Monte Carlo tree search algorithm gradually focuses on action sequences that exhibit high expected utility. Ultimately, it selects the action corresponding to the child node with the highest confidence upper bound from the root node as the optimal decision for the next step. This decision, together with the high-value actions in the following steps, constitutes a recent dynamic learning sequence recommended to the user. This sequence is not fixed; it is constantly replanned as learning progresses and new monitoring data arrives, thereby achieving true personalization and adaptability.

[0037] Example 4: After the user completes a dynamic learning sequence generated by the above examples, the system immediately initiates the evaluation process. This sequence may include a series of continuous learning activities, such as "watching a video explaining the graph of a quadratic function", "completing 10 basic discriminant exercise questions", and "solving 2 practical application problems". At the end of the sequence, the system provides the user with a comprehensive knowledge test. The scope of the test questions strictly corresponds to the core knowledge points covered by the sequence plan, such as the definition of quadratic functions, graph properties, vertex coordinate solving, and the application of discriminants. After the test is completed, the system automatically grades and generates the actual knowledge test score. At the same time, the learning terminal backend accurately records the time spent by the user in completing each task in the sequence and summarizes the actual total time spent. The feedback signal generation module meticulously compares the actual output with the expected target during sequence generation. The expected target is determined during sequence generation, including an expected score range based on historical data and the difficulty of knowledge points (e.g., for this sequence, the expected score range is 80-90 points) and an expected total time based on the standard learning speed (e.g., it is expected to take 45 minutes). If the user's actual test score is 75 points, which is lower than the lower limit of the expected range of 80 points, the system generates a positive knowledge coverage error signal. The magnitude of this signal is related to the degree of deviation (e.g., a deviation of -5 points). This signal indicates that the effectiveness of the sequence in knowledge transfer or consolidation has not met the design expectations. Conversely, if the actual score reaches 95 points, which exceeds the upper limit of the expected range, a negative error signal is generated, which may indicate that the sequence difficulty is too low for the user, and there is room for optimization of learning efficiency.

[0038] In terms of time, if the actual total time recorded is 55 minutes, while the expected time is 45 minutes, the system calculates the time deviation ratio as (55-45) / 45≈22.2%, and generates a positive time allocation error signal. This signal indicates that the time planning of the sequence is too tight for the user's current learning pace, or that there are unforeseen difficulties in the sequence that cause time delays. The magnitude of the deviation ratio determines the signal strength, and a high-intensity signal will trigger a more significant adjustment to the time allocation strategy. The knowledge coverage error signal is transmitted to the adjustment subsystem of the state transition probability matrix. This error signal is used as the basis for correcting the knowledge weight parameters in the matrix. Assuming that the current erroneous sequence mainly involves the "quadratic function" knowledge module, the system will focus on examining all transition paths related to the "quadratic function" node in the matrix. A positive error signal (insufficient actual mastery) may trigger the following adjustments: reduce the probability of jumping directly from the "basic concepts of quadratic functions" node to the "complex applications of quadratic functions" node, because actual results show that such a jump may be too aggressive; at the same time, correspondingly increase the probability of transferring from the "basic concepts of quadratic functions" node to consolidation nodes such as "deep understanding of quadratic function graphs" or "discriminative consolidation exercises". By increasing the weight of review and consolidation, the system compensates for the insufficient knowledge transmission effect. The magnitude of the adjustment is proportional to the strength of the error signal.

[0039] Referring to Table 1, the time allocation error signal is sent to the duration allocation strategy optimizer of the dynamic learning sequence generator. This optimizer analyzes the information contained in the error signal, especially in combination with the task identifier to determine which specific steps caused the time to exceed the limit. It may backtrack the sequence execution log to see the actual time distribution of each subtask in the total 55-minute timeout, with the aim of finding the "time bottleneck".

[0040] Table 1: Comparison of Actual and Expected Time for Each Task in the Learning Sequence

[0041] Based on the analysis in the table above, the system identified T002 (basic practice questions) and T004 (comprehensive application questions) as the main time-consuming tasks, especially T002, which exceeded the time limit by 50%. The optimizer will not simply and equally compress the time of all tasks in the subsequent sequence. Instead, it will perform contextual analysis. It may determine that the timeout in T002 is due to an insufficient understanding of related concepts (such as the discriminant), leading to slow problem-solving speed. This insufficient understanding, in turn, affects the efficiency of completing the T004 task, which requires this foundation. Therefore, the optimization strategy might be: when planning the next learning sequence for the user, a brief "quick review of prior knowledge" segment will be automatically inserted before the task nodes strongly related to the "quadratic function discriminant," or more sufficient baseline time will be allocated to practice questions like T002. Simultaneously, the baseline time for well-mastered parts (such as the exploration of image properties in T003, which actually took less time than expected) may be slightly reduced. Through this structured and targeted adjustment, the overall time allocation will better match the user's actual learning pace.

[0042] See Figure 4 The first image comprehensively evaluates the user's learning process from two dimensions: learning effectiveness and time efficiency. The histogram of knowledge coverage error distribution reflects the differences in mastery of learning content among different users, while the box plot of test scores visually shows the comparison between the overall learning effect and the expected goals. Furthermore, task time deviation analysis identifies time bottlenecks in the learning process. The heatmap of the knowledge state transition probability matrix in the second image reveals the learning path relationships between different knowledge modules. Dark areas represent high-probability transition paths, and light areas represent low-probability transition paths. This matrix provides an important basis for dynamically adjusting the learning sequence.

[0043] Example 5: The process of adjusting knowledge weight parameters begins with a deep analysis of the continuously input knowledge coverage error signal. This system error signal is not an isolated numerical point, but a signal sequence formed over time, reflecting the deviation in knowledge mastery after multiple recent learning sequences. The system uses a moving average method to process this time series signal. The core of the moving average method is to smooth random fluctuations to extract the systematic trend components. For example, the system sets an observation window containing feedback data from the last 5 learning sequences and calculates the average value of the knowledge coverage error within this window, which is used as the knowledge weight compensation amount. Assuming that the actual score of the last 5 sequences in the knowledge module of "trigonometric function graph transformation" is consistently lower than the lower limit of the expected score range, and the error value sequence is [-4, -5, -3, -6, -4] points, then the compensation amount calculated by the moving average is approximately -4.4 points. This stable negative compensation amount indicates that the user has a continuous problem of insufficient knowledge mastery in this module, rather than an accidental performance failure. After obtaining the knowledge weight compensation amount, the system transforms it into a specific adjustment operation on the state transition probability matrix. The adjustment follows the principle of proportional scaling and targets the state nodes and their migration paths corresponding to the knowledge units associated with the error signal. Taking the aforementioned "trigonometric function image transformation" module as an example, this module corresponds to several state nodes in the state transition probability matrix, such as "sine function image translation", "cosine function periodic transformation", and "phase transformation comprehensive application". The system will identify all migration paths that directly point to these weak state nodes, as well as migration paths that start from these weak nodes. For each such migration path, its corresponding conditional probability value will be scaled according to the magnitude of the compensation amount. The scaling factor is proportional to the absolute value of the compensation amount, and the direction is determined by the positive or negative sign of the compensation amount. A negative compensation amount (indicating insufficient mastery) will cause the probability of the "direct in-depth" path pointing to the weak node to be reduced, while it may increase the probability of the path pointing to the "pre-knowledge review" or "basic concept consolidation" node. This adjustment is not a simple addition or subtraction, but a proportional adjustment that maintains the normality of the probability distribution (the sum of all migration probabilities is 1).

[0044] In optimizing the task time allocation strategy in the dynamic learning sequence, the system processes the time allocation error signal, which contains the specific task identifier that caused the timeout. The key to the optimization strategy lies in retrospective analysis rather than simply penalizing timed-out tasks. The system locates the position of the timed-out task in the executed sequence and deeply analyzes its contextual dependencies. For example, suppose a comprehensive application problem about "solving triangles" is marked as severely timed out. The system will backtrack its predecessor tasks in the original learning sequence and may find that its immediate predecessor tasks are "sine theorem application practice" and "cosine theorem basic training". By analyzing the user's accuracy and efficiency in completing these predecessor tasks, the system may infer that the root cause of the timeout is not that the final application problem itself is too difficult, but rather that the user is not proficient enough in understanding and applying the sine or cosine theorem, which leads to frequent backtracking of basic knowledge when solving comprehensive problems, thus consuming a lot of time.

[0045] Based on this contextual backtracking analysis, the optimizer's optimization measures are forward-looking and relevant. It doesn't simply compress the estimated time for individual integrated application problems in the next sequence, as that might prevent the task from being completed. Instead, the optimizer shortens the standard time threshold for those preceding context nodes that are strongly dependent on timed-out tasks. The logic is to reserve more time for subsequent difficult tasks by optimizing the learning efficiency of the preceding steps. Specifically, the system may adjust the sequence planning algorithm to make the proficiency testing standards for key preceding tasks such as "sine theorem application practice" more stringent before arranging complex tasks like "solving triangles," or automatically embed a short, quick test of preceding knowledge to ensure that users have the necessary proficiency before entering complex tasks. This improves the overall rationality of time allocation and avoids bottleneck effects. This optimization method reflects that the system not only focuses on surface phenomena but also strives to fundamentally improve the smoothness of the learning path.

[0046] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0047] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent learning path planning based on reinforcement learning, characterized in that, include: Collect user learning behavior data and extract multidimensional feature vectors, which include knowledge mastery index, learning duration distribution curve and error type clustering results; Based on the knowledge mastery index in the multidimensional feature vector, a set of prior learning paths with similar feature vectors is matched in the historical learning database; Based on the learning duration distribution curve in the prior learning path set, a state transition probability matrix of the reinforcement learning model is constructed, which reflects the transfer pattern between different learning stages. Real-time monitoring of the changing trends of the user's current learning task completion rate and error type clustering results, and updating the posterior distribution parameters in the state transition probability matrix; A dynamic learning sequence is generated based on the updated state transition probability matrix and output to the learning path execution terminal.

2. The intelligent learning path planning method based on reinforcement learning as described in claim 1, characterized in that, The process of collecting user learning behavior data and extracting multidimensional feature vectors includes: The system collects timestamp sequences of user interactions using embedded sensors to generate raw behavior logs. Semantic parsing is performed on the operation types in the original behavior logs to divide them into knowledge unit tags and calculate the cumulative learning time under each tag; Extract the incorrect question numbers and error reason codes from the user's answer records, and use a density clustering algorithm to generate error type clustering results; The clustering results of the knowledge unit labels, cumulative learning time, and error types are aligned according to the time window and fused into the multidimensional feature vector.

3. The intelligent learning path planning method based on reinforcement learning as described in claim 2, characterized in that, The set of prior learning paths with similar feature vectors in the matching historical learning database includes: From the historical learning database, a set of candidate paths whose knowledge unit tag overlap exceeds a preset threshold is selected; Calculate the dynamic time warping distance between the learning duration distribution curve of each path in the candidate path set and the user's current learning duration distribution curve; The candidate path set is sorted according to the dynamic time warping distance, and the top N paths with the smallest distance are selected as the prior learning path set.

4. The intelligent learning path planning method based on reinforcement learning as described in claim 3, characterized in that, The state transition probability matrix for constructing the reinforcement learning model includes: The stage division nodes of each path in the prior learning path set are analyzed to generate a stage transfer relationship topology graph; Statistically count the migration frequency between adjacent stages in the topology graph, and initialize the baseline parameters of the state transition probability matrix; The abnormal migration probability weights in the baseline parameters are adjusted based on the distribution of interruption locations in the user's historical learning behavior data.

5. The intelligent learning path planning method based on reinforcement learning as described in claim 4, characterized in that, Updating the posterior distribution parameters in the state transition probability matrix includes: Detect the deviation between the user's current learning task completion rate and the expected progress, and calculate the gradient influence coefficient of the deviation on the state transition probability matrix; When a high-frequency error category is added to the error type clustering result, the associated state node in the state transition probability matrix is ​​marked. The transition weights of the state transition probability matrix are dynamically adjusted based on the gradient influence coefficient and the distribution density of the associated state nodes.

6. The intelligent learning path planning method based on reinforcement learning as described in claim 5, characterized in that, The generation of dynamic learning sequences includes: Monte Carlo tree search is performed based on the updated state transition probability matrix to generate candidate learning action sequences; For each action node in the candidate learning action sequence, calculate the ratio of knowledge coverage gain to time cost; The action node with the highest ratio is selected as the target task for the next learning stage, and the dynamic learning sequence is generated by prioritizing it.

7. The intelligent learning path planning method based on reinforcement learning as described in claim 6, characterized in that, The method further includes a feedback calibration phase after executing the dynamic learning sequence, comprising: Record the user's actual knowledge test score and time consumption after completing the dynamic learning sequence; The actual knowledge test score is compared with the expected score range to generate a knowledge coverage error signal. Based on the deviation ratio between the time consumption data and the expected time consumption, a time allocation error signal is generated; The knowledge weight parameters in the state transition probability matrix are adjusted using the knowledge coverage error signal. The time allocation error signal is used to optimize the task duration allocation strategy in the dynamic learning sequence.

8. The intelligent learning path planning method based on reinforcement learning according to claim 7, characterized in that, The knowledge weight parameters in the adjusted state transition probability matrix include: Identify the systematic deviation components in the knowledge coverage error signal and calculate the knowledge weight compensation amount using the moving average method; The migration probability value of the corresponding knowledge unit in the state transition probability matrix is ​​scaled proportionally according to the compensation amount.

9. The intelligent learning path planning method based on reinforcement learning according to claim 8, characterized in that, The optimized task duration allocation strategy in the dynamic learning sequence includes: Extract the timeout task identifier from the time allocation error signal and trace its context node in the dynamic learning sequence; Shorten the standard time threshold of adjacent tasks in the context node that have a dependency on the timed-out task.

10. A reinforcement learning-based intelligent learning path planning system, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 9.

Images