Method, system, device and medium for learner behavior simulation based on imitation learning

Abstract

The application discloses a learner behavior simulation method and system based on imitation learning, equipment and medium, the interactive process of students is regarded as a sequential decision task, and is formalized as a Markov decision process, the long-term return of students in the interactive process is optimized, and the generation process of the training and generation stages is unified to maintain consistency, and a learner behavior simulator framework including two-stage imitation learning is established, in the direct imitation learning stage, the simulator is encouraged to restore the existing mode in the student interaction, in the adversarial imitation learning stage, in order to ensure the rationality and diversity of imitation, the interactive sequence that has never appeared in the collected student data is generated, and the adversarial reward is given according to the quality of the generated interaction, so that the simulator generates diversified and reasonable interaction. After training, the learner behavior can be better simulated, and practical support is provided for the training and verification of an online education platform.

Description

Technical Field

[0001] This invention relates to the field of learner behavior simulation technology, and in particular to a method, system, device and medium for simulating learner behavior based on imitation learning. Background Technology

[0002] Online education systems have attracted many learners worldwide. They can provide various intelligent services through interaction with students, such as personalized recommendations and computerized adaptive testing, based on the students' individual needs. Typically, this learning interaction process goes like this: the system agent first recommends a learning project (e.g., an exercise), and after receiving feedback from the student (e.g., correct or incorrect), the system provides a new learning project based on the student's individual needs.

[0003] To support these services, online education systems primarily rely on machine learning algorithms to optimize agent models by analyzing extensive student interactions. However, in practical applications, the optimization of agent models (such as recommendation strategies and selection algorithms) can be divided into two paradigms: offline learning and online learning. Both paradigms encounter the problem of a lack of high-quality interactive data. Typically, offline learning is based on supervised learning. In contrast, online reinforcement learning attempts to interact with real students in an online environment, but this can lead to a poor experience when the agent model is not well-trained. Specifically, providing students with inappropriate learning items through an poorly trained agent model incurs significant costs, as students must sacrifice time to learn meaningless tasks. Furthermore, it can lead to serious problems such as unfairness and high cognitive load.

[0004] To address the various problems encountered in the practical application of online education systems, a feasible approach is to establish a learner simulator to mimic and replace real students in interacting with a proxy model. Therefore, the proxy model can be initially optimized before the arrival of real students to provide a better learning experience.

[0005] In this field, scholars have been dedicated to designing ideal student simulators, ranging from early rule-based simulators designed by experts to recent deep learning-based simulators. Rule-based learner simulators rely on artificially constructed rules or patterns to simulate students' feedback to learning resources and the evolution of their abilities through predefined behavioral patterns and evolutionary strategies. Examples include simulators based on memory curves. Deep learning-based learner simulators utilize neural networks to model student learning, memory, and other behaviors, achieving good results. Typical deep learning simulators include EERNN and KES. However, the design of such simulators is limited, mainly in the following aspects: (1) Existing simulators define the task of simulating student behavior as a single-step optimization task, ignoring the long-term goals of student learning behavior; (2) Existing simulators often affect the performance of simulated behavior due to differences between training and actual use.

[0006] Therefore, designing a learner simulation scheme that can overcome the above limitations and provide practical support for the training and verification of online education platforms is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a method, system, device, and medium for simulating learner behavior based on imitation learning, which helps to improve the fidelity and diversity of learner behavior generation data and provides strong support for the verification and deployment of online education platforms.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A learner behavior simulation method based on imitation learning, comprising:

[0010] Collect real interaction sequences from multiple students. Each student's real interaction sequence records the exercises completed and the answers given by the student at each moment.

[0011] A learner behavior simulator is constructed and trained, comprising a simulator and a discriminator. The training consists of two alternating phases: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming the first simulated interaction sequence. The real interaction sequence of the corresponding student is used as a supervision signal, and a reinforcement learning paradigm is adopted to train the simulator. In the adversarial imitation phase, for each student p, a student q with the closest knowledge level to student p is found. By simulating the interaction behavior of student q with student p's exercise list, a second simulated interaction sequence is generated. Student p's real interaction sequence is used as the real sample, and the second simulated interaction sequence is used as the generated sample to train the simulator and discriminator.

[0012] The simulator in the trained learner behavior simulator simulates student behavior and generates interaction sequences for each student.

[0013] A learner behavior simulation system based on imitation learning, comprising:

[0014] The data collection unit is used to collect real interaction sequences of multiple students. Each student's real interaction sequence records the exercises completed and the answers given by the student at each moment.

[0015] The learner behavior simulator construction and training unit is used to construct and train a learner behavior simulator, which includes a simulator and a discriminator. The training consists of two phases, which are performed alternately: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming a corresponding first simulated interaction sequence. The real interaction sequence of the corresponding student is used as a supervision signal, and a reinforcement learning paradigm is used to train the simulator. In the adversarial imitation phase, for each student p, the student q with the closest knowledge mastery level to student p is found. By simulating the interaction behavior of student q with student p's exercise list, a second simulated interaction sequence is generated. Student p's real interaction sequence is used as the real sample, and the second simulated interaction sequence is used as the generated sample to train the simulator and the discriminator.

[0016] The learner behavior simulation unit is used to simulate student behavior through the simulator in the trained learner behavior simulator and generate interaction sequences for each student.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by this invention, treating the student's interaction process as a sequential decision-making task and formalizing it as a Markov Decision Process (MDP) optimizes the student's long-term rewards during the interaction process and unifies the generation process in the training and generation phases to maintain consistency. Furthermore, a learner behavior simulator framework incorporating two stages of imitation learning is established. In the direct imitation learning stage, the simulator is encouraged to reproduce existing patterns in student interactions. In the adversarial imitation learning stage, to ensure the rationality and diversity of imitation, interaction sequences never before seen in the collected student data are generated, and adversarial rewards are given based on the quality of the generated interactions, thereby encouraging the simulator to generate diverse and reasonable interactions. After training, learner behavior can be better simulated, providing practical support for the training and validation of online education platforms. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a learner behavior simulation method based on imitation learning, provided as an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the learner behavior simulator provided in an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a learner behavior simulation system based on imitation learning provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0027] First, the following explanations are provided for the terms that may be used in this article:

[0028] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0029] The following is a detailed description of a learner behavior simulation method, system, device, and medium based on imitation learning provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of reagents or instruments used in the embodiments of this invention are not specified, they are all conventional products that can be purchased commercially.

[0030] Example 1

[0031] This invention provides a learner behavior simulation method based on imitation learning, such as... Figure 1 As shown, it mainly includes the following steps:

[0032] Step 1: Collect real interaction sequences from multiple students.

[0033] In this embodiment of the invention, the real interaction sequence of each student records the exercises and answers completed by the student at each moment.

[0034] Step 2: Build and train a learner behavior simulator.

[0035] In this embodiment of the invention, the learner behavior simulator includes a simulator and a discriminator; training includes two phases, which are performed alternately: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming a corresponding first simulated interaction sequence, and uses the corresponding student's real interaction sequence as a supervision signal, employing a reinforcement learning paradigm to train the simulator; In the adversarial imitation phase, for each student p, a student q with the closest knowledge mastery level to student p is found, and a second simulated interaction sequence is generated by simulating the interaction behavior of student q with student p's exercise list, using student p's real interaction sequence as the real sample and the second simulated interaction sequence as the generated sample to train the simulator and discriminator. Specifically:

[0036] Considering the interaction characteristics between real students and online education platforms, real students typically enter online education platforms to achieve long-term learning goals. Therefore, this invention defines the learner's interaction process as a Markov decision process, meaning that the learner's behavior 'a' at each moment is only related to the corresponding state 's'. Reinforcement learning methods are applied to optimize long-term rewards, thereby enabling the student to learn strategy 'π'. In this invention, as... Figure 2 As shown in the simulator design section, based on the characteristics of the practice resources, the features of the exercises are represented using embedding and linear networks. A simple but effective behavioral module (i.e., student policy) is designed based on a multilayer perceptron (MLP), which is based on the student policy π and the exercise e. t Features x t and state s t Determines student behavior a t , represented as π θ (a t |s t ,e t ), t represents time, π θ This represents a behavioral module with parameter θ; simultaneously, GRU is used to model the student's state transitions s during the learning process. t+1 ~g(s t ,e t ,a t ), where g is an abbreviation for GRU, representing the state transition function. By training the above modules, the simulation of student behavior can be achieved.

[0037] However, reinforcement learning requires a pre-specified reward and then learns a behavioral module π that maximizes the expected average of that reward. But student interactions are complex, and the logic behind student interactions remains an unexplored problem; manually defining the reward function is extremely impractical. Therefore, for simulators that cannot be optimized within the reinforcement learning paradigm, the remaining problem to be solved is how to construct the reward. To this end, this invention turns to imitation learning, a machine learning method inspired by expert demonstrations (i.e., collected sequences of real student interactions). Specifically, it employs a direct-adversarial imitation learning learner simulator. Student policies π are learned from expert demonstrations, the reward function driving student behavior in the expert demonstrations is reconstructed, and this reward function is optimized through reinforcement learning to maximize long-term rewards, achieving accurate imitation of student behavior. In the direct imitation learning phase, a reward is defined to encourage the simulator to reproduce existing patterns in student interactions. In the adversarial imitation learning phase, to ensure the rationality and diversity of the imitation, the learner simulator proposed in this invention generates interactions that have never appeared in the collected student data. A parameterized discriminator is then applied to evaluate the generated interactions, and adversarial rewards are given based on the quality of the generated interactions, thereby encouraging the simulator to generate diverse and rational interactions. Furthermore, the discriminator in this invention is optimized pairwise rather than by classification, in order to further improve stability.

[0038] In the direct imitation stage, the process of generating the first simulated interaction sequence is as follows:

[0039] 1) The proxy model provides an exercise e1.

[0040] 2) The simulator extracts the features x1 of the exercises.

[0041] 3) The behavior module in the simulator determines the student's behavior based on the feature x1 of the exercise and the given initial state s1. This decision-making process is denoted as Represented as:

[0042]

[0043] Where σ represents the Sigmoid function, the initial state s1 can be extracted from the interaction sequence of the corresponding student in the dataset, or it can come from a new student (e.g., a randomly initialized state); Linear represents a linear network, and student behavior is the student's feedback on the exercises; θ is the parameter of the behavior module.

[0044] 4) Update the student status to s2, and continue executing steps 1) to 3) until the end.

[0045] In this way, it is possible to interact with the agent model and help train the agent model (e.g., recommendation strategies, selection algorithms) and generate diverse high-fidelity interaction sequences by generating high-fidelity interaction sequences.

[0046] Step 3: Simulate student behavior using the simulator in the trained learner behavior simulator to generate interaction sequences for each student.

[0047] After training in step 2 above, student behavior can be simulated using a simulator, generating interaction sequences for each student in accordance with steps 1) to 4) above.

[0048] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0049] I. Problem Definition and Formalization.

[0050] In an online education platform, suppose there are |U| students and |E| exercises, where U is the set of students and E is the set of exercises. Specifically, the collected student interaction sequence (also known as expert demonstration) is denoted as D = {u1, u2, ..., u...}. |U|}. The interaction sequence for a specific student is denoted as u = {(e1,a1),(e2,a2),…}, where e t ∈E represents the exercises that the student completed at time t, a t This indicates that the student's understanding of exercise e t Feedback. If he answers correctly... t Then a t The value is 1 if the condition is met, and 0 otherwise. For an exercise e∈E, it is denoted as e={i,k}, where i is the id of the exercise, k∈K is the knowledge concept corresponding to the exercise (e.g., function, addition), and K is the set of knowledge concepts. A practical learner simulator can generate the corresponding 'a' for any student given an exercise e, ensuring high fidelity and diversity in the generated results.

[0051] To accurately simulate complex student interactions, this invention treats the interaction between students and online education platforms as a sequential decision-making process and formulates it as a Markov Decision Process (MDP). The corresponding MDP elements are defined as follows:

[0052] State S: S is the state space simulating the students' knowledge proficiency. t Let S represent the student's level of knowledge mastery at time t. The initial state s1 can be initialized based on the accuracy of each knowledge concept in the interaction sequence, or it can be initialized randomly.

[0053] Action A: At time t, It is a simulator that imitates students practicing e t The simulation determines whether a student can correctly answer a question using the behavior module. Decide.

[0054] Transition T (State Transition T): Next state s t+1 It is determined by the current state s t Practice e t and the answer results Decide. Where g represents the state transition function.

[0055] Reward R: In the two imitation processes, there are two reward functions (r) 1 ,r 2 These two reward functions are defined to guide the simulator to update parameters and ensure high fidelity and diversity. These two reward functions will be described in detail later.

[0056] II. Data Collection and Preprocessing.

[0057] 1. Data collection.

[0058] In this embodiment of the invention, widely used learner interaction data records are used as the input dataset. The interaction data includes exercises and students' answering behaviors. Examples of such data include the open-source learning answering behavior dataset released by Junyi-Academy (Junyi) and interaction records of students' English learning on the Santa platform (EdNet). In addition, some commonly used datasets such as Assistment, as well as student data collected offline from web scraping and offline sources, can also be used to supplement the data.

[0059] 2. Data preprocessing.

[0060] Before building a learner behavior simulator, the collected data needs to be preprocessed to ensure the simulator's effectiveness. Preprocessing mainly includes the following:

[0061] (1) Data filtering.

[0062] This invention primarily focuses on simulating learner interaction behavior, aiming to recover student behavior sequences from a distributional perspective. Therefore, it is necessary to filter out excessively short learner interaction sequences.

[0063] (2) Sampling.

[0064] During the training process, due to the complexity of student interaction behaviors, each real student interaction was applied multiple times to improve the effectiveness of sampling and thus enhance data utilization during training.

[0065] 3. Construct and train a learner behavior simulator.

[0066] The learner behavior simulator based on imitation learning constructed in this invention includes a simulator and a discriminator, and features a two-stage imitation process: direct imitation and adversarial imitation. The simulator simulates student behavior during both imitation stages; the discriminator evaluates and rewards the interactive behavior generated by the simulator during the adversarial imitation stage. Figure 2 As shown, the main principles of the learner behavior simulator are illustrated. The left side shows the two-stage imitation process, the middle part shows the principle of the simulator, and the right side shows the principle of the discriminator.

[0067] 1. Emulator.

[0068] The simulator includes: a first embedding layer, a first linear network, and a first gated recurrent unit.

[0069] The first embedding layer is responsible for mapping exercises to vector space to obtain corresponding features; whereby an exercise includes: exercise ID and knowledge concept. The exercise ID and knowledge concept are mapped to vector space respectively and then concatenated together as the feature of the exercise.

[0070] In this embodiment of the invention, a general architecture is proposed to extract feature information. First, for exercise e at time t... t ={i, k}, and the first embedding layer maps i and k to a low-dimensional vector. Where R is the symbol for the set of real numbers, and d represents the dimension. Exercise e t Features x t It can be represented by its ID and knowledge concept vector. The connection is obtained, that is in This represents the vector concatenation operation.

[0071] The first linear network is responsible for combining the state at the corresponding time step with the characteristics of the exercise to generate the corresponding answer. This is equivalent to the decision-making process of the behavior module, and can be represented as follows: Where σ represents the Sigmoid function, Linear1 represents the first linear network, and e t Let x be the problem corresponding to time t. t Exercises representing time t t Features For the generated exercise e t The corresponding answer results Let π represent the state of student p at time t. θ This indicates a behavior module with parameter θ.

[0072] The first gated loop unit is responsible for tracking the student's knowledge status and updating it, represented as:

[0073]

[0074]

[0075] in, For Exercise e t The vector obtained after mapping knowledge concepts to a vector space is the knowledge concept representation vector; 0 k For a k-dimensional zero vector, This represents the behavior of answering questions by combining different parts; The symbol is a concatenation symbol; GRU1 represents the first gated loop unit, and g1 is an abbreviation for the first gated loop unit. Let represent the state of student p at time t+1.

[0076] 2. Discriminator.

[0077] The discriminator includes: a second embedded layer, a second linear network, and a second gated recurrent unit.

[0078] The second embedding layer is responsible for mapping exercises to a vector space to obtain corresponding features. Each exercise includes an exercise ID and a knowledge concept. The exercise ID and knowledge concept are mapped to the vector space separately and then concatenated together to form the exercise's features. The principle here is the same as the embedding layer in the aforementioned simulator, so it will not be elaborated further.

[0079] The second gated recurrent unit is responsible for judging the features combined with the exercises and calculating the score in conjunction with the second linear network; among them, the second simulated interaction sequence Information corresponding to time t The process of making the judgment is represented as follows:

[0080]

[0081]

[0082] in, This is a simulated interaction record for student p. x t Exercises representing time t t Features Exercise e t The corresponding answer result, z t As the intermediate vector, GRU2 represents the second gated recurrent unit, and Linear2 represents the second linear network. Calculated for the discriminator The score.

[0083] Similarly, the discriminator uses the same method to analyze the student's actual interaction sequence u. p Perform a discrimination test to obtain D. ω (u p ).

[0084] 3. Two-stage imitation learning.

[0085] In this embodiment of the invention, a learner behavior simulator is trained through a two-stage imitation learning process.

[0086] (1) Direct imitation.

[0087] The simulator generates corresponding answer results in chronological order based on the questions in each student's real interaction sequence, forming the first simulated interaction sequence. A reward is awarded based on whether the student's real interaction sequence matches the answer results in the first simulated interaction sequence. The Proximal Policy Optimization (PPO) algorithm is applied to update the simulator based on the reward. Specifically:

[0088] To ensure the high fidelity of the generated learner interaction sequences, this invention introduces a reinforcement-based imitation learning paradigm, learning directly from collected real student interaction sequences. Specifically, a simple yet effective reward function is designed to guide the imitation process. For u = {(e1,a1),(e2,a2),…}∈D, this invention uses real behavior as the monitoring signal. The reward is determined based on whether the student's real interaction sequence matches the answer results in the first simulated interaction sequence. When the simulator generates student behavior from the first simulated interaction sequence… The reward is set to 1 if the result matches the 'a' in the real student interaction sequence; otherwise, it is 0, as shown below:

[0089]

[0090] Where, r t 1 The behavior module π represents the reward obtained at time t during the direct imitation phase. θ It will tend to directly mimic students' real-world reactions to ensure the authenticity of the generated learning interactions. t Exercises representing time t, The generated answer results are represented by D, which represents the set of real interaction sequences formed by multiple students. Exercises representing time t t The generated answer results Exercise e in set D t The corresponding actual answer results are consistent.

[0091] In the direct imitation stage, N real interaction sequences τ = {u1, u2, ..., u} are typically sampled from set D. N For student p, their actual interaction sequence is denoted as u. p = {(e1,a1),(e2,a2),…}, where e represents a question, a represents the answer (a=1 if correct, a=0 otherwise); the subscript numbers represent the question numbers, each question number corresponds to a time point. The simulator starts from the real interaction sequence u. p Extract the knowledge level of student p as the initial state. Generate the answer results for each exercise in the order of the exercises to form the first simulated interactive sequence. At each time t, the simulator extracts exercise e. t Features x t And combine the state at the corresponding moment Generate corresponding answer results And utilize the state at the corresponding moment Exercise e t and the generated answer results Perform a state update to obtain the state at the next moment.

[0092] Then, the proximal policy optimization algorithm is applied to train the simulator based on the reward, and the simulator parameters are updated, as shown below:

[0093]

[0094] Where, r 1 This represents the reward obtained during the direct imitation phase. Let H be the mathematical expectation, and let H represent the entropy. Represents the simulator behavior module π θ The decision-making process, where θ is the parameter of the behavioral module. This refers to the behavioral module π θ Student behavior is determined based on the characteristics x and state s of exercise e.

[0095] (2) Counter-imitation.

[0096] Imitating existing behavioral patterns in a given dataset can hinder performance due to potential sparsity or insufficient data. A good simulator needs to generate more diverse and reasonable interaction sequences. Therefore, this invention employs Generative Adversarial Imitation Learning (GAIL) to learn behavioral patterns that satisfy the real distribution of student behavior but are not present in the given dataset. To enrich the collected data while ensuring consistency with the real distribution, real learning scenarios are considered, where students with similar knowledge levels are often recommended to similar exercises. Based on this assumption, the simulator can generate learner interaction sequences beyond the real dataset. The main process is as follows:

[0097] Similar to the direct imitation stage, after sampling N real interaction sequences, the simulator extracts each student's knowledge mastery level based on their real interaction sequences. For student p, it finds a student q whose knowledge mastery level is closest to p's using cosine similarity. By simulating the interaction between student q and student p's exercise list, a second simulated interaction sequence is generated. That is, select all the unanswered exercises from student p's exercise list, and simulate student q answering all the selected unanswered exercises to generate a second simulated interaction sequence. For student p, its actual interaction sequence u p As a real sample, the second simulated interaction sequence These samples were used as generated samples (fake samples) to train the discriminator in the simulator and GAIL.

[0098] Here, the knowledge mastery level of students p and q refers to the information extracted from the actual interaction sequence of the corresponding students, denoted as p and q respectively.

[0099] It is worth noting that GAIL-based methods typically use the cross-entropy loss function to optimize the discriminator:

[0100]

[0101] in, For mathematical expectation, D ω Let ω be the discriminator, and ω be the parameters of the discriminator. Let represent the set formed by multiple second simulated interaction sequences, D represent the set formed by multiple students' real interaction sequences, (s,a) represent a real interaction record, where s is the knowledge mastery level in the real interaction record and a is the corresponding student behavior; (s',a') represent a simulated interaction record, where s is the knowledge mastery level in the simulated interaction record and a' is the corresponding student behavior.

[0102] Using this loss function, the simulator can be viewed as optimizing the JS divergence between the real distribution (i.e., the real student interaction sequence) and the generated distribution:

[0103]

[0104] Where H represents entropy, D JS Represents the JS divergence; where It is a normalized occupancy measure that reflects the behavior module π of the simulator. θ and expert strategy π E The distribution of interactions is considered. However, using a traditional discriminator here is not feasible due to the following challenges: First, a traditional discriminator only uses the collected student interaction data once in each optimization step, resulting in low data utilization. This problem is often even more unacceptable in the construction of student simulators, as the scale of collected real student interaction sequences is very limited. Second, student behavior is often complex due to guessing and sliding factors, meaning that students may answer incorrectly on exercises they shouldn't due to guessing factors, and vice versa. A traditional discriminator may converge too quickly, giving negative rewards before the simulator has a good fit to the distribution. Therefore, it cannot help the simulator learn complex distributions. Third, a traditional discriminator may stagnate during training. This is because when the discriminator reaches its optimum, it ignores real samples in the gradient, stops learning why real samples are real, and makes the generated samples not become more realistic.

[0105] To overcome these challenges and train a more stable student simulator suitable for more complex distributions, this invention proposes a pairwise optimization strategy. This strategy enables the discriminator to estimate the probability that real samples are more realistic than generated samples. Specifically, it uses a discriminator D... ω The real interaction sequence and the corresponding second simulated interaction sequence are distinguished, and the difference Δ between the distinction results is calculated, expressed as: in, This represents the second simulated interaction sequence calculated by the discriminator. The score, D ω (u p ) represents the true interaction sequence u of student p calculated by the discriminator. p The score.

[0106] The discriminant is trained using the difference in the discrimination results, and the parameters of the discriminant are updated as follows:

[0107]

[0108] Where ω represents the discriminator D ωThe parameter is f(.); f(.) is the transformation function, for example, the log-sigmoid function can be used (f(x)=log(σ(x))). The expression represents the expected value. Since the operations are the same at each step when calculating the expected value, the subscript symbol is omitted.

[0109] Furthermore, the reward obtained during the adversarial imitation phase is determined based on the score of the second simulated interaction sequence calculated by the discriminator; wherein, the reward obtained at time t during the adversarial imitation phase is expressed as: in, Refers to the second simulated interaction sequence The information corresponding to time t, i.e. x t Exercises representing time t, This indicates the corresponding answer result.

[0110] The simulator is trained using the proximal policy optimization algorithm, and the simulator parameters are updated as follows:

[0111]

[0112] Where, r 2 This represents the reward obtained during the counter-imitation phase. Let H be the mathematical expectation, and let H represent the entropy. Represents the simulator behavior module π θ The decision-making process, where θ is the parameter of the behavioral module. This refers to using θ as the behavioral module π θ Student behavior is determined based on the characteristics x and state s of exercise e. The solutions provided in these embodiments of the invention help improve the fidelity and diversity of learner behavior generation, providing strong support for the verification and deployment of online education platforms. Furthermore, they can serve as a general framework, integrating different machine learning tasks (e.g., knowledge tracing, computer adaptive testing, etc.) to learn under the same paradigm, providing a foundation for a better understanding of human behavior and the realization of strong artificial intelligence. Taking knowledge tracing as an example, it relies on a large number of high-quality learner interaction sequences to improve its ability to track student abilities. Through high-quality learning interaction sequences generated by learner simulators, the knowledge tracing model can be better trained.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0114] Example 2

[0115] This invention also provides a learner behavior simulation system based on imitation learning, which is mainly implemented based on the method provided in the foregoing embodiments, such as... Figure 3 As shown, the system mainly includes:

[0116] The data collection unit is used to collect real interaction sequences of multiple students. Each student's real interaction sequence records the exercises completed and the answers given by the student at each moment.

[0117] The learner behavior simulator construction and training unit is used to construct and train a learner behavior simulator, which includes a simulator and a discriminator. The training consists of two phases, which are performed alternately: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming a corresponding first simulated interaction sequence. The real interaction sequence of the corresponding student is used as a supervision signal, and a reinforcement learning paradigm is used to train the simulator. In the adversarial imitation phase, for each student p, the student q with the closest knowledge mastery level to student p is found. By simulating the interaction behavior of student q with student p's exercise list, a second simulated interaction sequence is generated. Student p's real interaction sequence is used as the real sample, and the second simulated interaction sequence is used as the generated sample to train the simulator and the discriminator.

[0118] The learner behavior simulation unit is used to simulate student behavior through the simulator in the trained learner behavior simulator and generate interaction sequences for each student.

[0119] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0120] Example 3

[0121] The present invention also provides a processing device, such as Figure 4As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0122] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0123] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0124] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0125] The output device can be a display terminal;

[0126] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0127] Example 4

[0128] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0129] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0130] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A learner behavior simulation method based on imitation learning, characterized in that, include: Collect real interaction sequences from multiple students. Each student's real interaction sequence records the exercises completed and the answers given by the student at each moment. Construct and train a learner behavior simulator, which includes a simulator and a discriminator; The training consists of two phases, which are conducted alternately: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming the first simulated interaction sequence. The real interaction sequence of the corresponding student is used as a supervision signal, and a reinforcement learning paradigm is adopted to train the simulator. In the adversarial imitation phase, for each student p, the student q with the closest knowledge level to student p is found. By simulating the interaction behavior of student q with student p's exercise list, a second simulated interaction sequence is generated. Student p's real interaction sequence is used as the real sample, and the second simulated interaction sequence is used as the generated sample to train the simulator and the discriminator. The simulator in the trained learner behavior simulator simulates student behavior and generates interaction sequences for each student.

2. The learner behavior simulation method based on imitation learning according to claim 1, characterized in that, The step of generating corresponding answer results through a simulator based on the order of exercises in each student's real interaction sequence, forming the corresponding first simulated interaction sequence, includes: For a student p, its real interaction sequence is denoted as u p = {(e1, a1), (e2, a2),...}, where e represents an exercise, a represents an answer result, a = 1 if the answer is correct, otherwise a = 0; the subscript number represents the exercise number, and each exercise number corresponds to a time point; The simulator is based on a real interaction sequence. p Extract the knowledge level of student p as the initial state. Generate the answer results for each exercise in the order of the exercises to form the first simulated interactive sequence. At each time t, the simulator extracts exercise e. t Feature x t And combine the state at the corresponding moment Generate corresponding answer results And utilize the state at the corresponding moment Exercise e t and the generated answer results Perform a state update to obtain the state at the next moment.

3. The learner behavior simulation method based on imitation learning according to claim 1, characterized in that, The step of using the real interaction sequences of corresponding students as supervision signals and employing a reinforcement learning paradigm to train the simulator includes: Whether to give a reward is determined by whether the student's actual interaction sequence matches the answer results in the first simulated interaction sequence, as shown below: Where, r t 1 e represents the reward obtained at time t during the direct imitation phase. t Exercises representing time t, The generated answer results are represented by D, which represents the set of real interaction sequences formed by multiple students. The problem e represents the time interval t. t The generated answer results The answers were consistent; The near-field policy optimization algorithm is applied, and the simulator is trained based on the reward. The simulator parameters are then updated, as follows: Where, r 1 This represents the reward obtained during the direct imitation phase. Let H be the mathematical expectation, and let H represent the entropy. Represents the simulator behavior module π θ The decision-making process, where θ is the parameter of the behavioral module. This refers to the behavioral module π θ Student behavior is determined based on the characteristics x and state s of exercise e.

4. A learner behavior simulation method based on imitation learning according to claim 2 or 3, characterized in that, The simulator includes: a first embedding layer, a first linear network, and a first gated recurrent unit; wherein: The first embedding layer is responsible for mapping exercises to vector space to obtain corresponding features; whereby, an exercise includes: exercise ID and knowledge concept. The exercise ID and knowledge concept are mapped to vector space respectively and then concatenated together as the feature of the exercise. The first linear network is responsible for combining the state at the corresponding time with the characteristics of the problem to generate the corresponding answer; The first gated loop unit is responsible for tracking the student's knowledge status and updating it, represented as: Among them, e t For the exercises corresponding to time t, For the generated exercise e t The corresponding answer results For Exercise e t The vector obtained after mapping knowledge concepts to a vector space; 0 k For a k-dimensional zero vector, This represents the concatenated answer behavior; ⊕ is the concatenation symbol, GRU1 represents the first gated loop unit, and g1 is an abbreviation for the first gated loop unit. Let represent the state of student p at time t+1.

5. The learner behavior simulation method based on imitation learning according to claim 1, characterized in that, For each student p, find the student q whose knowledge level is closest to that of student p. Generate a second simulated interaction sequence by simulating the interaction between student q and student p's exercise list. Use student p's real interaction sequence as the real sample and the second simulated sequence as the generated sample. Train the simulator and discriminator by: The simulator extracts each student's knowledge mastery level based on their actual interaction sequences. For student p, it finds a student q whose knowledge mastery level is closest to p's using cosine similarity. By simulating the interaction between student q and student p's exercise list, a second simulated interaction sequence is generated. That is, select all the unanswered exercises from student p's exercise list, and simulate student q answering all the selected unanswered exercises to generate a second simulated interaction sequence. Using discriminator D ω The real interaction sequence and the corresponding second simulated interaction sequence are distinguished, and the difference Δ between the distinction results is calculated, expressed as: in, This represents the second simulated interaction sequence calculated by the discriminator. The score, D ω (u p ) represents the true interaction sequence u of student p calculated by the discriminator. p The score; The discriminant is trained using the difference in the discrimination results, and the parameters of the discriminant are updated as follows: Where ω represents the discriminator D ω The parameter; f(.) is the transformation function, Represents the mathematical expectation; The simulator is trained using the proximal policy optimization algorithm, and the simulator parameters are updated as follows: Where, r 2 This represents the reward obtained during the counter-imitation phase. Let H be the mathematical expectation, and let H represent the entropy. Represents the simulator behavior module π θ The decision-making process, where θ is the parameter of the behavioral module. This refers to using θ as the behavioral module π θ Student behavior is determined based on the characteristics x and state s of exercise e.

6. The learner behavior simulation method based on imitation learning according to claim 5, characterized in that, The reward obtained during the adversarial imitation phase is determined based on the score of the second simulated interaction sequence calculated by the discriminator; The reward obtained at time t during the anti-imitation phase is represented as: in, Refers to the second simulated interaction sequence The information corresponding to time t, i.e. e t Exercises representing time t, This indicates the corresponding answer result.

7. A learner behavior simulation method based on imitation learning according to claim 5 or 6, characterized in that, The discriminator includes: a second embedding layer, a second linear network, and a second gated recurrent unit; wherein: The second embedding layer is responsible for mapping exercises to vector space to obtain corresponding features; whereby, an exercise includes: exercise ID and knowledge concept. The exercise ID and knowledge concept are mapped to vector space respectively and then concatenated together as the feature of the exercise. The second gated recurrent unit is responsible for judging the features combined with the exercises and calculating the score in conjunction with the second linear network; among them, for the simulated interaction sequence Information corresponding to time t The process of making the judgment is represented as follows: in, x t Exercises representing time t t Features Exercise e t The corresponding answer result, z t GRU2 represents the second gated recurrent unit, and Linear2 represents the second linear network.

8. A learner behavior simulation system based on imitation learning, characterized in that, include: The data collection unit is used to collect real interaction sequences of multiple students. Each student's real interaction sequence records the exercises completed and the answers given by the student at each moment. The learner behavior simulator construction and training unit is used to construct and train the learner behavior simulator, which includes a simulator and a discriminator. The training consists of two phases, which are conducted alternately: In the direct imitation phase, the simulator generates corresponding answer results based on the order of exercises in each student's real interaction sequence, forming the first simulated interaction sequence. The real interaction sequence of the corresponding student is used as a supervision signal, and a reinforcement learning paradigm is adopted to train the simulator. In the adversarial imitation phase, for each student p, the student q with the closest knowledge level to student p is found. By simulating the interaction behavior of student q with student p's exercise list, a second simulated interaction sequence is generated. Student p's real interaction sequence is used as the real sample, and the second simulated interaction sequence is used as the generated sample to train the simulator and the discriminator. The learner behavior simulation unit is used to simulate student behavior through the simulator in the trained learner behavior simulator and generate interaction sequences for each student.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images