A method, device, electronic device and storage medium for recommending an exercise path

Abstract

The present application discloses a method, device, electronic device and storage medium for recommending practice paths, which relates to the field of online education technology. The method includes: obtaining the learner's historical learning records; using a high-level intelligent agent to determine the learner's first learning goal based on the historical learning records; using a middle-level intelligent agent to determine the first practice path based on the prior relationship between the first learning goal and the knowledge point; using a low-level intelligent agent to obtain recommended exercises corresponding to the first practice path for the learner to learn; wherein the high-level intelligent agent, the middle-level intelligent agent and the low-level intelligent agent are pre-built algorithms. The decision-making of the high-level, middle-level and low-level intelligent agents fully balances the prior relationship between the learner's historical learning records and knowledge points, which can avoid over-reliance on historical learning records and avoid a single learning goal, so that the obtained recommended exercises are in line with the learner's knowledge status in terms of difficulty, novelty and diversity, and are suitable for learners to practice.

Description

Technical Field

[0001] The present application relates to the field of online education technology, and in particular to a practice path recommendation method, device, electronic device and storage medium. Background Art

[0002] The field of online education includes practice path recommendation, but existing practice path recommendation schemes still have some problems: (1) Over-reliance on historical learning records and lack of reasonable arrangement of learning goals. The knowledge level of learners is a constantly changing process. Existing recommendation schemes over-rely on learners' historical data for recommendation, ignoring changes in learners' knowledge status and new learning needs. In addition, learners may have multiple learning goals during the learning process, and existing recommendation schemes do not make reasonable plans for different learning goals. (2) Emphasis on a single learning goal, ignoring learners' personalized characteristics, learning participation, etc. Existing recommendation schemes try to explore learners' weaknesses and continuously recommend knowledge points that learners have not mastered based on a single strategy. They lack in-depth research on individual differences among learners and do not consider the impact of exercise difficulty, diversity, and novelty on learners' participation. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide a practice path recommendation method, device, electronic device and storage medium to determine a practice path suitable for a learner.

[0004] To achieve the above purpose, an embodiment of the present application provides a method for recommending a practice path, the method comprising:

[0005] Obtain learners' historical learning records;

[0006] Determining the first learning goal of the learner according to the historical learning record using a high-level intelligent agent;

[0007] Determine a first practice path using a middle-level agent according to a priori relationships between the first learning goal and the knowledge points;

[0008] Using a low-level intelligent agent to obtain recommended exercises corresponding to the first practice path for the learner to learn;

[0009] Among them, the high-level agent, the middle-level agent and the low-level agent are pre-constructed algorithms.

[0010] In some embodiments, the method further comprises:

[0011] A knowledge tracking environment model is constructed using a dynamic key-value memory network;

[0012] The history exercises learned by the learner and the history answer results corresponding to the history exercises are added to the knowledge tracking environment model.

[0013] In some embodiments, obtaining the learner's historical learning record includes:

[0014] The historical learning record of the learner is obtained from the knowledge tracking environment model; wherein the historical learning record includes the historical exercises and the historical answer results.

[0015] In some embodiments, the using a high-level agent to determine the first learning goal of the learner according to the historical learning record includes:

[0016] Obtaining a high-level input vector of the high-level agent; wherein the input vector includes the historical learning record and a plurality of second learning objectives;

[0017] Encoding the high-level input vector using a gated recurrent unit to obtain a high-level hidden state representation vector of the high-level input vector;

[0018] A proximal strategy optimization algorithm is used as the high-level intelligent agent; wherein the proximal strategy optimization algorithm includes a strategy network and a value function network;

[0019] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0020] The objective function is:

[0021]

[0022] Among them, θ actor are the parameters of the policy network, A t is the advantage function, g t The second learning goal is is the high-level hidden state representation vector, ε is a preset threshold;

[0023] The expression of the gradient ascent method is:

[0024]

[0025] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0026] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0027] The loss function is:

[0028]

[0029] Among them, θ critic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0030] The expression of the gradient descent method is:

[0031]

[0032] Wherein, β is the second preset weight parameter;

[0033] The high-level agent after parameter update selects the first learning objective from multiple second learning objectives according to the high-level hidden state representation vector.

[0034] In some embodiments, the determining the first practice path according to the prior relationship between the first learning goal and the knowledge point by using the middle-level agent includes:

[0035] Acquire multiple second practice paths from the directed graph corresponding to the prior relationship of the knowledge points;

[0036] Constructing a middle-level input vector according to the historical learning record, the first learning goal, the sub-goals of the first learning goal, and each second practice path;

[0037] Encoding the middle-level input vector using a gated recurrent unit to obtain a middle-level hidden state representation vector of the middle-level input vector;

[0038] An actor-critic algorithm is used as the middle-level agent; wherein the actor-critic algorithm includes a policy network and a value function network;

[0039] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0040] The objective function is:

[0041]

[0042] Among them, θ actor are the parameters of the policy network, A t is the advantage function, o t is the first practice path, is the middle-layer hidden state representation vector;

[0043] The expression of the gradient ascent method is:

[0044]

[0045] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0046] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0047] The loss function is:

[0048]

[0049] Among them, θ critic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0050] The expression of the gradient descent method is:

[0051]

[0052] Wherein, β is the second preset weight parameter;

[0053] The middle-level agent after parameter update selects the first training path from multiple second training paths according to the middle-level hidden state representation vector.

[0054] In some embodiments, the using a low-level agent to obtain recommended exercises corresponding to the first practice path for the learner to learn includes:

[0055] Select multiple candidate questions from a pre-built question bank;

[0056] Constructing a low-level input vector according to the historical learning record, the first learning goal, the sub-goal of the first learning goal, a plurality of second practice paths, the sub-paths of the second practice path, and each of the candidate exercises;

[0057] Encoding the low-level input vector using a gated recurrent unit to obtain a low-level hidden state representation vector of the low-level input vector;

[0058] An actor-critic algorithm is used as the low-level agent; wherein the actor-critic algorithm includes a policy network and a value function network;

[0059] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0060] The objective function is:

[0061]

[0062] Among them, θ actor are the parameters of the policy network, At is the advantage function, q t is the recommended exercise, is the low-level hidden state representation vector;

[0063] The expression of the gradient ascent method is:

[0064]

[0065] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0066] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0067] The loss function is:

[0068]

[0069] Among them, θ critic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0070] The expression of the gradient descent method is:

[0071]

[0072] Wherein, β is the second preset weight parameter;

[0073] The recommended exercises are selected from a plurality of the candidate exercises according to the low-level hidden state representation vector using the low-level agent after parameter update.

[0074] In some embodiments, the method further comprises:

[0075] Obtaining the test answer results of the learner on the recommended exercises;

[0076] The recommended exercises and the test answer results are added to a pre-constructed knowledge tracking environment model; wherein the knowledge tracking environment model is constructed using a dynamic key-value memory network.

[0077] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a practice path recommendation device, the device comprising:

[0078] A learning record acquisition unit is used to acquire the learner's historical learning record;

[0079] A learning goal determination unit, configured to determine the first learning goal of the learner according to the historical learning record by using a high-level intelligent agent;

[0080] A practice path determination unit, configured to determine a first practice path using a middle-level agent according to a priori relationships between the first learning goal and the knowledge points;

[0081] A recommended exercise acquisition unit, configured to use a low-level agent to acquire recommended exercises corresponding to the first practice path for the learner to learn;

[0082] Among them, the high-level agent, the middle-level agent and the low-level agent are pre-constructed algorithms.

[0083] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0084] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0085] The embodiments of the present application include at least the following beneficial effects:

[0086] This application can obtain the learner's historical learning records; use a high-level agent to determine the learner's first learning goal based on the historical learning records; use a middle-level agent to determine the first practice path based on the prior relationship between the first learning goal and the knowledge point; use a low-level agent to obtain recommended exercises corresponding to the first practice path for the learner to learn; wherein the high-level agent, the middle-level agent, and the low-level agent are pre-built algorithms. This application combines high, middle, and low-level agents to respectively determine the learner's first learning goal, first practice path, and recommended exercises, fully balancing the prior relationship between the learner's historical learning records and knowledge points, avoiding over-reliance on historical learning records and avoiding a single learning goal, so that the obtained recommended exercises are in line with the learner's knowledge status in terms of difficulty, novelty, and diversity, and are suitable for learners to practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0088] Figure 1 A flowchart of a method for recommending a practice path provided in an embodiment of the present application;

[0089] Figure 2 An example flow chart of a multi-step exercise path recommendation method provided in an embodiment of the present application;

[0090] Figure 3 An exemplary flow chart of another method for recommending a practice path provided in an embodiment of the present application;

[0091] Figure 4 A schematic diagram of the structure of a practice path recommendation device provided in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0094] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0095] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0096] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0097] Before describing the embodiments of the present application in detail, some terms and related technologies involved in the embodiments of the present application are first described. The terms and related technologies involved in the embodiments of the present application are subject to the following explanations:

[0098] Knowledge tracking: Model the learner's knowledge mastery based on the learner's historical answer records, and then obtain the learner's current knowledge status to predict the learner's future answering performance.

[0099] Three-layer reinforcement learning: Introducing a three-layer agent mechanism into the reinforcement learning framework, abstractly separating actions and states, decomposing and simplifying the problem scale, reducing storage requirements and computational workload, and speeding up learning.

[0100] Practice path recommendation: Generate a sequence of recommended exercises based on the learner’s personalized parameters and the characteristics of the exercises, forming a practice path that guides learners to achieve their learning goals.

[0101] Exercise difficulty: the error rate of a single knowledge point in the learner’s historical learning records.

[0102] Novelty: knowledge points that learners have not mastered so far, or knowledge points that have never appeared in the learners' historical answer records.

[0103] Diversity: Measures the degree of difference between knowledge points and reflects the richness of recommended exercises.

[0104] Multi-objective: Simultaneously consider the impact of three indicators, namely, difficulty, novelty, and diversity, on the generated practice paths.

[0105] In the era of intelligent education, practice path recommendation can help learners achieve their learning goals faster, reduce the impact of information overload, and provide a suitable sequence of learning resources to a large number of online learners. Therefore, practice path recommendation plays an important role in improving learners' learning ability and knowledge level. As users increasingly rely on the information and decisions provided by recommendation systems, the novelty and diversity of recommended content become increasingly important. Considering the balance between novelty and diversity in personalized learning recommendation systems has a certain impact on the applicability of learning, that is, it is necessary to maintain a steady improvement in learning effects while maintaining learning enthusiasm.

[0106] In recent years, researchers have continuously applied various algorithms to practice path recommendation. Genetic algorithm is applied to practice path recommendation, and reasonable learning resources are recommended to learners based on the best fitness value obtained by genetic algorithm. The characteristics of learners and the association between knowledge points are constructed into knowledge graph, and graph theory is applied to recommendation. With the development of deep learning and reinforcement learning, deep learning technology has emerged in recommendation systems and has become a new frontier of recommendation algorithms. The related technology first uses the long-short term memory network (LSTM) to track the learner's dynamically changing knowledge level, and then uses the actor-critic algorithm to determine the order of recommended exercises. On this basis, a more complete framework (GEHRL) is designed. GEHRL divides practice path recommendation into two parts, namely the high-level agent for sub-goal selection and the low-level agent for exercise recommendation, which are implemented by the proximal policy optimization (PPO) algorithm and the actor-critic algorithm. Although practice path recommendation has received much attention in recent years, existing technologies still have some problems: (1) Over-reliance on historical data and lack of reasonable arrangement of learning goals. (2) Emphasis on single goal orientation and neglect of learners' personalized characteristics, learning participation, etc.

[0107] Therefore, the embodiments of the present application provide a method, device, electronic device and storage medium for recommending a practice path. The technical solution of the present application includes: obtaining the learner's historical learning records; using a high-level intelligent agent to determine the learner's first learning goal based on the historical learning records; using a middle-level intelligent agent to determine the first practice path based on the prior relationship between the first learning goal and the knowledge point; using a low-level intelligent agent to obtain the recommended exercises corresponding to the first practice path for the learner to learn; wherein the high-level intelligent agent, the middle-level intelligent agent and the low-level intelligent agent are pre-built algorithms. The present application combines high, middle and low-level intelligent agents to respectively determine the learner's first learning goal, the first practice path and the recommended exercises, which fully balances the prior relationship between the learner's historical learning records and knowledge points, avoids over-reliance on historical learning records and avoids a single learning goal, so that the obtained recommended exercises are in line with the learner's knowledge status in terms of difficulty, novelty and diversity, and are suitable for learners to practice.

[0108] The embodiment of the present application provides a method for recommending a practice path, and relates to the field of online education technology. The method for recommending a practice path provided in the embodiment of the present application can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the practice path recommendation method, etc., but is not limited to the above forms.

[0109] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0110] Reference Figure 1 The present application embodiment provides a method for recommending a practice path, which may include but is not limited to S100 to S130, wherein the high-level agent, the middle-level agent, and the low-level agent in the method are pre-built algorithms, which are as follows:

[0111] S100: Obtain the learner's historical learning records.

[0112] Specifically, the learning record may include the exercises that the learner has practiced and his / her answer results, answering time and answering time, etc.

[0113] As an optional implementation, the embodiment of the present application may further include:

[0114] A knowledge tracking environment model is constructed using a dynamic key-value memory network;

[0115] The history exercises learned by the learner and the history answer results corresponding to the history exercises are added to the knowledge tracking environment model.

[0116] Specifically, the knowledge tracing model (KTM) can track the learner's learning progress according to the learner's historical records, obtain the learner's learning level, and then feed back the learner's current learning status to the three-layer reinforcement learning framework (TRF) as part of the three-layer reinforcement learning environment variable; the three-layer reinforcement learning model includes high, medium and low-level agents. In this embodiment, the original dynamic key-value memory network (DKVMN) model can be used as the knowledge tracing environment model.

[0117] Therefore, as a further implementation, S110 may include:

[0118] The historical learning record of the learner is obtained from the knowledge tracking environment model; wherein the historical learning record includes the historical exercises and the historical answer results.

[0119] In order to record the learner's learning record, after obtaining the recommended exercises, the embodiment of the present application may further include:

[0120] Obtaining the test answer results of the learner on the recommended exercises;

[0121] The recommended exercises and the test answer results are added to a pre-constructed knowledge tracking environment model; wherein the knowledge tracking environment model is constructed using a dynamic key-value memory network.

[0122] Specifically, after the learner obtains the recommended exercises and takes the test, the corresponding learning record will be fed back to the knowledge tracking model KTM together with the previous historical records. The knowledge tracking model KTM updates the learner's mastery of the knowledge points based on the latest data. Given a learner's historical learning record u = {(q1, s1), (q2, s2), (q3, s3), ..., (q L ,s L )}, q t ∈Q comes from any exercise in the exercise bank, s t ∈[0,1] is the learner’s answer result. So far, the knowledge tracking model KTM is used to retrieve the learner’s implicit knowledge level from the previous learning records.

[0123] S110: Determine the first learning goal of the learner according to the historical learning record using a high-level intelligent agent.

[0124] Specifically, the first learning goal is a learning goal selected for the learner from a plurality of candidate learning goals; the candidate learning goal can be used as the second learning goal.

[0125] Furthermore, S110 may include:

[0126] Obtaining a high-level input vector of the high-level agent; wherein the input vector includes the historical learning record and a plurality of second learning objectives;

[0127] Encoding the high-level input vector using a gated recurrent unit to obtain a high-level hidden state representation vector of the high-level input vector;

[0128] A proximal strategy optimization algorithm is used as the high-level intelligent agent; wherein the proximal strategy optimization algorithm includes a strategy network and a value function network;

[0129] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0130] The objective function is:

[0131]

[0132] Among them, θ actor are the parameters of the policy network, A t is the advantage function, g t The second learning goal is is the high-level hidden state representation vector, ε is a preset threshold;

[0133] The expression of the gradient ascent method is:

[0134]

[0135] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0136] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0137] The loss function is:

[0138]

[0139] Among them, θ critic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0140] The expression of the gradient descent method is:

[0141]

[0142] Wherein, β is the second preset weight parameter;

[0143] The high-level agent after parameter update selects the first learning objective from multiple second learning objectives according to the high-level hidden state representation vector.

[0144] Specifically, according to the learner's historical learning records or the output results of the knowledge tracking model KTM, this embodiment can define the knowledge points that the learner has not yet mastered as the first learning goal (i.e., the incorrect knowledge points in the learner's answer results), the first learning objectives can be sorted according to the chronological order of appearance, and the high-level agent (HRA) can be used to select the mixed first learning objectives one by one according to the sorting and pass them to the lower-level agents.

[0145] State encoder of the high-level agent: high-level input vector The historical learning records of the learners t ={(q1,s1),(q2,s2),(q3,s3),…,(q L ,s L )} and learning objectives As a variant of the long-short term memory network LSTM (Long-Short Term Memory, LSTM), the gated recurrent unit GRU can spend less computing resources to achieve similar experimental results as the long-short term memory network LSTM. Therefore, this embodiment can use the gated recurrent unit GRU to Encode. The input of the gated recurrent unit GRU is the historical learning record u t The output is u t The hidden state representation of Learning objectives By multi-hot encoding Indicates that the corresponding learning goal (i.e., the first learning goal) has its subscript set to 1, and the rest are 0. Concatenate and get the high-level hidden state representation vector in the following form:

[0146] Among them, [] represents the vector concatenation operation.

[0147] The high-level agent (HRA) can select appropriate learning goals and feed them back to the next-level agent based on the learner's historical learning records. The proximal policy optimization (PPO) algorithm is a reinforcement learning algorithm based on policy gradient. The main idea is to improve learning efficiency and stability by approximate optimization of the strategy. The advantage of the PPO algorithm is that it can perform efficient learning in a large-scale environment, has good stability and convergence, and can achieve good performance in different tasks. Therefore, this embodiment can use PPO as the model of the high-level agent HRA.

[0148] The PPO algorithm consists of a policy network (Actor) and a value function network (Critic).

[0149] The update strategy of the Actor network is implemented by maximizing the objective function. It is based on the policy gradient method and updates the parameters of the Actor network through gradient ascent.

[0150] The PPO algorithm needs to repeatedly train the policy network and value function network, use the collected experience data to update, and gradually optimize the policy network to achieve the purpose of improving the policy effect.

[0151] S120: Using the middle-level agent to determine a first practice path according to the prior relationship between the first learning goal and the knowledge point.

[0152] Specifically, the first practice path is a practice path selected for the learner from a plurality of candidate practice paths, and the candidate practice path can serve as the second practice path.

[0153] Each exercise contains different knowledge points, and there are prior relationships between the knowledge points.

[0154] Further, S120 may include:

[0155] Acquire multiple second practice paths from the directed graph corresponding to the prior relationship of the knowledge points;

[0156] Constructing a middle-level input vector according to the historical learning record, the first learning goal, the sub-goals of the first learning goal, and each second practice path;

[0157] Encoding the middle-level input vector using a gated recurrent unit to obtain a middle-level hidden state representation vector of the middle-level input vector;

[0158] An actor-critic algorithm is used as the middle-level agent; wherein the actor-critic algorithm includes a policy network and a value function network;

[0159] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0160] The objective function is:

[0161]

[0162] Among them, θ actor are the parameters of the policy network, A t is the advantage function, o t is the first practice path, is the middle-layer hidden state representation vector;

[0163] The expression of the gradient ascent method is:

[0164]

[0165] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0166] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0167] The loss function is:

[0168]

[0169] Among them, θ critic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0170] The expression of the gradient descent method is:

[0171]

[0172] Wherein, β is the second preset weight parameter;

[0173] The middle-level agent after parameter update selects the first training path from multiple second training paths according to the middle-level hidden state representation vector.

[0174] The prior relationship of knowledge points can be saved in the concept graph (KCG). In real life, there may be multiple candidate practice paths to achieve a certain learning goal, that is, there are multiple candidate practice paths in the concept graph KCG that can achieve the first learning goal g selected by the high-level agent HRA. t The starting node of a candidate practice path can be any exercise, but the ending node is the learning goal g t, the candidate practice paths may contain other exercises with a priori relationships. Optionally, in this embodiment, the length of the practice path may be defined as 20, that is, the number of exercises contained in the practice path does not exceed 20. The middle-level agent ORA may select a target practice path (i.e., the first practice path) from the multiple candidate practice paths provided by the concept graph KCG. The first practice path may comprehensively consider the impact of the difficulty, novelty, and diversity of the exercises on the learner's learning enthusiasm.

[0175] State encoder of the middle-level agent: middle-level input vector The same gated recurrent unit GRU is used for encoding. Contains learners' learning records t ={(q1,s1),(q2,s2),(q3,s3),…,(q L ,s L )}, Learning objectives Sub-goal g and practice path LP = {o1, o2, ...}, the above variables are encoded by the gated recurrent unit GRU to obtain their respective hidden state representations, and then they are concatenated to obtain the middle-level agent ORA input vector The final middle-level hidden state representation vector is:

[0176]

[0177] The mid-level agent (ORA) is part of the high-level agent (HRA) environment. The actions taken by the mid-level agent ORA will affect the state transition of the high-level agent HRA. The goal of the mid-level agent ORA is to select a target practice path from many paths that can achieve the learning goal. The length of the path is appropriate, and the exercises included can balance the requirements of exercise difficulty, novelty and diversity. The Actor-Critic algorithm combines the policy network (Actor) and the value function network (Critic), and has the advantages of reducing variance and faster convergence speed. Therefore, this embodiment can use the Actor-Critic algorithm as a model of ORA. When training the Critic network, the Actor-Critic algorithm can use the same mean square error as the PPO algorithm as the loss function, and the parameter update is achieved by minimizing the loss function of the value function, but the calculation of the value function uses the reward corresponding to the mid-level agent ORA. The main difference lies in the objective function and policy update. The PPO algorithm constructs the objective function by comparing the ratio term and the advantage function, and uses the clipping operation to limit the amplitude of the policy update to ensure the stability of the policy update. The Actor-Critic algorithm uses the advantage function based on the value function to construct the objective function and updates the parameters based on the gradient direction without involving additional clipping operations.

[0178] S130: Utilize a low-level intelligent agent to obtain recommended exercises corresponding to the first practice path for the learner to learn.

[0179] Specifically, the first practice path is a target practice path selected for the learner from a plurality of candidate practice paths, and the candidate practice paths can serve as the second practice path.

[0180] Further, S130 may include:

[0181] Select multiple candidate questions from a pre-built question bank;

[0182] Constructing a low-level input vector according to the historical learning record, the first learning goal, the sub-goal of the first learning goal, a plurality of second practice paths, the sub-paths of the second practice path, and each of the candidate exercises;

[0183] Encoding the low-level input vector using a gated recurrent unit to obtain a low-level hidden state representation vector of the low-level input vector;

[0184] An actor-critic algorithm is used as the low-level agent; wherein the actor-critic algorithm includes a policy network and a value function network;

[0185] Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network;

[0186] The objective function is:

[0187]

[0188] Among them, θ actor are the parameters of the policy network, A t is the advantage function, q t is the recommended exercise, is the low-level hidden state representation vector;

[0189] The expression of the gradient ascent method is:

[0190]

[0191] Wherein, α is the first preset weight parameter, and t is the learning step size;

[0192] Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network;

[0193] The loss function is:

[0194]

[0195] Among them, θcritic are the parameters of the value function network, is the value function of the value function network, y t is the target value;

[0196] The expression of the gradient descent method is:

[0197]

[0198] Wherein, β is the second preset weight parameter;

[0199] The recommended exercises are selected from a plurality of the candidate exercises according to the low-level hidden state representation vector using the low-level agent after parameter update.

[0200] Specifically, the low-level agent (LRA) can be used to select corresponding exercises from the exercise bank and recommend them to learners according to the first exercise path selected by the middle-level agent ORA.

[0201] State Encoder of Low-Level Agent: The Low-Level Agent (LRA) can use the same Gated Recurrent Unit (GRU) to encode data. The input vector of the Low-Level Agent LRA is Contains more abundant information. On the basis of , it also includes the target practice path o, inputs all variables into GRU, and obtains the hidden state representation corresponding to the input variables. The low-level hidden state representation vector is as follows:

[0202]

[0203] The actions of the low-level agent (LRA) depend on the decisions made by the previous two agents. The low-level agent can select appropriate exercises to recommend to the learner. The learner receives the recommended exercises and takes the test, and then feeds the results back to the low-level agent LRA. In this embodiment, a variable can be used to record the number of exercises currently recommended. The initial value of the variable is 0. Each time an exercise is recommended, the value of the variable increases by 1. Once the variable value reaches the set threshold, the current recommendation process is terminated. The low-level agent LRA can also use the Actor-Critic algorithm. The basic process is similar to that of the middle-level agent ORA. The only difference is in the representation of the objective function, which uses the state and action of the low-level agent.

[0204] Faced with massive learning resources and diverse learning needs, how to quickly achieve learning goals, reduce learning costs, and improve knowledge levels is a major issue in learner development. Practice path recommendation comprehensively considers differences in learners' knowledge status, learning preferences, educational background, etc., and develops personalized practice paths for specific learners to better meet learners' learning needs. Existing practice path recommendation schemes use a combination of reinforcement learning and knowledge tracking to recommend more reasonable and effective practice paths for learners. However, existing practice path recommendation schemes ignore the prior relationship between knowledge points, or ignore the impact of factors such as multiple goals on learning enthusiasm.

[0205] In response to the above problems, this application adopts a three-layer intelligent agent to correspond to and reconcile the relationship between knowledge relevance, multi-goal achievement, and path formation. The three-layer reinforcement learning framework of this application not only considers the prior relationship between knowledge points, but also considers the balance of exercise difficulty, novelty and diversity in the recommendation process, making the recommended content more reasonable and effective.

[0206] Next, the solution of the embodiment of the present application will be introduced and explained in conjunction with specific application examples:

[0207] Reference Figure 2 , this embodiment provides an example flow chart of a multi-step exercise path recommendation method. Figure 3 This is an example flow chart of another exercise path recommendation method provided in this embodiment.

[0208] The solution to implement this embodiment can use the Knowledge Tracing Model (KTM) and the Three-Layer Reinforcement Learning Framework (TRF). The Three-Layer Reinforcement Learning Framework TRF can include four sub-units, namely, the High-Level Agent (HRA), the Mid-Level Agent (ORA), the Low-Level Agent (LRA) and the Knowledge Concept Graph (KCG). The high-level agent obtains the learner's knowledge state based on the feedback of the knowledge tracing model KTM, and then selects a learning goal for it. t There may be multiple goal-based learning paths (GLPs) to achieve a specific learning goal. After receiving the learning goal, the middle-level agent ORA selects a goal path that comprehensively considers the difficulty, novelty, and diversity of the exercises under the limitations of the concept graph KCG. tThe lower-level agent LRA selects relevant learning items q from the exercise bank based on the decision made by the upper-level agent. t Recommended to learners.

[0209] This implementation also conducts verification experiments:

[0210] The data sets used include: junyi, Assist09.

[0211] Simulator: In order to verify the recommendation effects of different methods, this embodiment provides two different simulators to compare the degree of improvement of learners' knowledge level, namely, the simulator based on knowledge structure (KSS) and the simulator based on knowledge evolution (KES). KSS is a rule-based simulator that does not require additional data for initialization, while KES is a data-driven simulator, which is divided into KES-junyi and KES-Assist09 according to different data.

[0212] Baseline models: CB, DQN, SAC, RLTutor, GEHRL.

[0213] The above method recommends different learning items to learners according to the learners' knowledge status. This embodiment uses the degree of improvement of the learners' knowledge level in the learning goal as an evaluation index to evaluate the performance of different methods.

[0214] Practice path recommendation: |U| learners, the exercise bank contains |Q| exercises, and the historical learning process of each learner is recorded as u={(q1,s1),(q2,s2),(q3,s3),…,(q L ,s L )}, u∈U, q t ∈Q represents the exercises answered by the learner at time t, s t is the learner's answer result. If the learner answers correctly, s t The value of is 1, otherwise it is 0. In the learning process, each learner has his or her own learning goal. is a subset of the question bank. The task of this embodiment is to recommend a practice path for learners that can achieve the above goals. Initially, the learner is tested and obtains an initial score E s , the knowledge points that the learner has not mastered are set as learning goals, and the high-level agent selects a sub-goal. At this time, the possible practice path LP = {o1, o2, ...}, after a recommendation process is completed, the practice path o = {q1, q2, ..., q K}, the learner is tested again and obtains a new learning record u t ={(q1,s1),(q2,s2),…,(qK ,s K )}, ending score E e The new learning record is incorporated into the learner's historical learning record, and the learner's knowledge state changes, that is, u t =u t ∪u t-1 It is not difficult to find that the effectiveness of the evaluation practice path It can be obtained by the following formula:

[0215]

[0216] E sup Represents the full score of the test question, E s is the score obtained by the learner on the initial test, and E e is the test score of the learner on the recommended path, then the learner's knowledge improvement can be expressed as express, The larger the value of is, the more reasonable the recommended path is and the more it meets the learner's requirements.

[0217] Experimental comparison: This embodiment sets the length of the training path to 10, 20, and 30, and uses a simulator to verify the overall performance of all methods on these three path lengths. The experiment shows that the method of this embodiment achieves better results than all the comparison methods and achieves the best performance results.

[0218] Reference Figure 4 The embodiment of the present application also provides a practice path recommendation device, which can implement the above practice path recommendation method, and the device includes:

[0219] A learning record acquisition unit is used to acquire the learner's historical learning record;

[0220] A learning goal determination unit, configured to determine the first learning goal of the learner according to the historical learning record by using a high-level intelligent agent;

[0221] A practice path determination unit, configured to determine a first practice path using a middle-level agent according to a priori relationships between the first learning goal and the knowledge points;

[0222] A recommended exercise acquisition unit, configured to use a low-level agent to acquire recommended exercises corresponding to the first practice path for the learner to learn;

[0223] Among them, the high-level agent, the middle-level agent and the low-level agent are pre-constructed algorithms.

[0224] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0225] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned exercise path recommendation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0226] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0227] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0228] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0229] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the exercise path recommendation method of the embodiment of this application;

[0230] Input / output interface 503, used to implement information input and output;

[0231] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0232] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0233] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0234] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned exercise path recommendation method is implemented.

[0235] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0236] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0237] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0238] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0239] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0240] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0241] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0242] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0243] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0244] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0245] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0246] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0247] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for recommending a practice path, characterized in that: The method comprises: Obtain learners' historical learning records; Determining the first learning goal of the learner according to the historical learning record using a high-level intelligent agent; Determine a first practice path using a middle-level agent according to a priori relationships between the first learning goal and the knowledge points; Using a low-level intelligent agent to obtain recommended exercises corresponding to the first practice path for the learner to learn; Wherein, the high-level agent, the middle-level agent and the low-level agent are pre-constructed algorithms; The step of using a high-level agent to determine the first learning goal of the learner according to the historical learning record includes: Obtaining a high-level input vector of the high-level agent; wherein the input vector includes the historical learning record and a plurality of second learning objectives; Encoding the high-level input vector using a gated recurrent unit to obtain a high-level hidden state representation vector of the high-level input vector; A proximal strategy optimization algorithm is used as the high-level intelligent agent; wherein the proximal strategy optimization algorithm includes a strategy network and a value function network; Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network; The objective function is: J( )= [my( , clip( , 1- ε , 1+ ε ) )]; in, are the parameters of the policy network, is the advantage function, The second learning goal is is the high-level hidden state representation vector, ε is the preset threshold; The expression of the gradient ascent method is: = + α J( )； in, α is the first preset weight parameter, t is the learning step size; Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network; The loss function is: L( )= [ ]； in, are the parameters of the value function network, is the value function of the value function network, is the target value; The expression of the gradient descent method is: = - β L( )； in, β is the second preset weight parameter; The high-level agent after parameter update selects the first learning goal from multiple second learning goals according to the high-level hidden state representation vector.

2. A method for recommending a practice path according to claim 1, characterized in that: The method further comprises: A knowledge tracking environment model is constructed using a dynamic key-value memory network; The history exercises learned by the learner and the history answer results corresponding to the history exercises are added to the knowledge tracking environment model.

3. A method for recommending a practice path according to claim 2, characterized in that: The acquiring of the learner's historical learning records includes: The historical learning record of the learner is obtained from the knowledge tracking environment model; wherein the historical learning record includes the historical exercises and the historical answer results.

4. A method for recommending a practice path according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtaining the test answer results of the learner on the recommended exercises; The recommended exercises and the test answer results are added to a pre-constructed knowledge tracking environment model; wherein the knowledge tracking environment model is constructed using a dynamic key-value memory network.

5. A training path recommendation device, characterized in that: The device comprises: A learning record acquisition unit is used to acquire the learner's historical learning record; A learning goal determination unit, configured to determine the first learning goal of the learner according to the historical learning record by using a high-level intelligent agent; A practice path determination unit, configured to determine a first practice path using a middle-level agent according to a priori relationships between the first learning goal and the knowledge points; A recommended exercise acquisition unit, configured to use a low-level agent to acquire recommended exercises corresponding to the first practice path for the learner to learn; Wherein, the high-level agent, the middle-level agent and the low-level agent are pre-constructed algorithms; The step of using a high-level agent to determine the first learning goal of the learner according to the historical learning record includes: Obtaining a high-level input vector of the high-level agent; wherein the input vector includes the historical learning record and a plurality of second learning objectives; Encoding the high-level input vector using a gated recurrent unit to obtain a high-level hidden state representation vector of the high-level input vector; A proximal strategy optimization algorithm is used as the high-level intelligent agent; wherein the proximal strategy optimization algorithm includes a strategy network and a value function network; Using a gradient ascent method to maximize the objective function of the policy network to update the parameters of the policy network; The objective function is: J( )= [my( , clip( , 1- ε , 1+ ε ) )]; in, are the parameters of the policy network, is the advantage function, The second learning goal is is the high-level hidden state representation vector, ε is the preset threshold; The expression of the gradient ascent method is: = + α J( )； in, α is the first preset weight parameter, t is the learning step size; Minimizing the loss function of the value function network by using a gradient descent method to update the parameters of the value function network; The loss function is: L( )= [ ]； in, are the parameters of the value function network, is the value function of the value function network, is the target value; The expression of the gradient descent method is: = - β L( )； in, β is the second preset weight parameter; The high-level agent after parameter update selects the first learning goal from multiple second learning goals according to the high-level hidden state representation vector.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images