Knowledge tracking method and system based on reinforcement learning and negative sampling optimization

Abstract

The invention provides a knowledge tracking method based on reinforcement learning and negative sampling optimization, and relates to the technical field of knowledge tracking, and the method comprises the steps: obtaining a knowledge point set of a historical answer sequence of a learner, and obtaining the knowledge point representation of an exercise; generating an initial exercise map according to the knowledge point representation, and dynamically adjusting the edge weight of the initial exercise map to obtain an adjusted exercise map; using the exercise map to obtain a historical positive sample set and a historical negative sample set associated with the current exercise; similarity is calculated based on the historical positive sample set and the historical negative sample set, and knowledge point vectors of the current exercises are obtained through sample loss function optimization; and dynamically updating the knowledge state of the learner according to the knowledge point vector, and outputting the future answering condition of the learner based on the updated knowledge state. According to the method, the potential relevance between exercises is effectively captured, the prediction reasoning capability is enhanced, and the prediction precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of knowledge tracking technology, and in particular to a knowledge tracking method and system based on reinforcement learning and negative sampling optimization. Background Art

[0002] The development of personalized education has made knowledge tracing (KT) gradually become an important research field in intelligent online education. The core task of knowledge tracing is to predict the learner's future answer situation based on the exercise representation and the learner's historical answer situation.

[0003] At present, existing knowledge tracking usually relies on multimodal feature fusion to represent exercises. The advantage of this method is that the overall knowledge state is generated by the fusion of the states of each knowledge point. The representation of the overall knowledge state contains richer information, and the prediction effect is better overall.

[0004] However, the relationship between the states of various knowledge points will affect the aggregate representation of the knowledge point states and the representation of the overall knowledge state, resulting in certain limitations in capturing the correlation between positive and negative samples in the current answer exercises and historical answer exercises, which leads to a decrease in the predictive performance of the knowledge tracking method and low prediction precision and accuracy. Summary of the invention

[0005] The present invention provides a knowledge tracking method and system based on reinforcement learning and negative sampling optimization, which are used to solve the problem of low precision and accuracy of prediction of the knowledge tracking method.

[0006] In one aspect, the present invention provides a knowledge tracking method based on reinforcement learning and negative sampling optimization, the method comprising: Obtain the knowledge point set of the learner's historical answer sequence and obtain the knowledge point representation of the exercise; Generating an initial exercise graph according to the knowledge point representation, and dynamically adjusting the edge weights of the initial exercise graph to obtain an adjusted exercise graph; Using the problem graph, obtaining a set of historical positive samples and a set of historical negative samples associated with the current problem; Calculate the similarity based on the historical positive sample set and the historical negative sample set, and optimize the sample loss function to obtain the knowledge point vector of the current exercise; The learner's knowledge state is dynamically updated according to the knowledge point vector, and the learner's future answer status is output based on the updated knowledge state.

[0007] Furthermore, the knowledge point representation is obtained by: A set of knowledge points of the exercise is obtained, and each knowledge point is one-hot encoded to obtain a one-hot encoded vector of each knowledge point; meanwhile, a knowledge point weight of each knowledge point in the exercise is obtained; and a knowledge point representation of the exercise is obtained based on the one-hot encoded vector and the knowledge point weight.

[0008] Furthermore, the knowledge point represents It is obtained by weighted summation of knowledge points, and the calculation formula is: ; in, Indicates exercises; Indicates knowledge points, Representation Exercises The corresponding knowledge point set, ; Represents the weight of knowledge points, satisfying ,and ; represents a one-hot encoded vector, , , and knowledge points The corresponding position is marked as 1, and the rest of the positions are marked as 0; Indicates the size of the knowledge point set.

[0009] Furthermore, the edge weights may be obtained by: Each edge The edge weight is initialized to 1, and the two nodes connected by the edge and The features are linearly transformed and activated, and two new nodes are obtained accordingly. ; Calculate new nodes The attention between them is calculated, and the edge weight is obtained according to the attention.

[0010] Furthermore, the method for obtaining the historical negative sample set includes: S301: traverse the exercise graph based on edge weights to obtain an exercise node sequence; S302: randomly selecting a central node in the exercise node sequence as the initial node of the Markov chain; S303: For each central node, calculate the current state space corresponding to the current central node, and select the next central node according to the reinforcement learning algorithm function; S304: updating the corresponding state space according to the next central node, calculating the reward function, and updating the reinforcement learning algorithm function according to the reward function; S305: Repeat steps S303 to S304 until a node is sampled from the exercise node sequence. negative samples, and thus obtain the historical negative sample set.

[0011] Furthermore, the reward function is: ; in, Indicates the current center node, Indicates the next central node; Indicates the current center node With the next central node The edge weights between ; express The set of neighbor nodes of Represents the adjustment factor, which is used to balance the rewards of positive and negative samples; Represents the weighted shortest path length between the current node and the next node.

[0012] Furthermore, the update formula of the reinforcement learning algorithm function is: ; in, Represents the current state space; represents the next state space; represents the learning rate; represents the discount factor; Indicates that in the next state space The action below; Indicates execution of an action Rewards received afterwards.

[0013] Furthermore, the calculation formula of the similarity is: ; in, Representation vector With vector The dot product of and Represents vectors and vector The L2 norm of ; Indicates the current question The knowledge point vector of Represents the negative sample set The node vector of the history exercise.

[0014] Furthermore, the learner's future answer status is output based on the updated knowledge state vector, including:

[0015] in, represents the weight matrix; Indicates the updated knowledge status; Represents the bias term.

[0016] In a second aspect, the present invention provides a knowledge tracking system based on reinforcement learning and negative sampling optimization, comprising a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of any of the above methods.

[0017] In general, the present invention provides a knowledge tracking method and system based on reinforcement learning and negative sampling optimization. The technical solution conceived by the present invention can achieve the following beneficial effects compared with the prior art: (1) The present invention adjusts the exercise graph and samples the positive and negative sample sets based on the added exercise graph. It adopts the negative sampling strategy of weighted probability priority traversal to sample in the exercise graph, and optimizes the knowledge point vector represented by the exercise based on the positive and negative sample sets, so that the sample set is close to the target distribution, improves the clustering effect of the positive sample exercise nodes, and effectively distinguishes the negative sample exercise nodes, effectively mines the potential relationship between exercises, captures the correlation between positive and negative samples, enhances the prediction performance, and improves the precision and accuracy of the prediction.

[0018] (2) The present invention dynamically adjusts the edge weights between nodes in the exercise graph through the graph attention network, further exploring the similarities between historical exercises and currently answered exercises, thereby enabling the selection of historical exercise samples with a strong correlation with the central node in the exercise graph, enhancing the clustering effect of positive sample exercises, and effectively distinguishing negative sample exercises, thereby optimizing the exercise representation and improving the prediction performance.

[0019] (3) The present invention is based on Markov Chain Monte Carlo and negatively samples the state space of the exercise graph. This method can not only approximate the target distribution and further optimize the exercise representation, but also simulate the cognitive process of learners relying on historical exercise reasoning when solving new problems, thereby improving their performance in complex learning situations. It can also effectively capture the potential correlation between exercises and improve the prediction accuracy in knowledge tracking tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a method flow diagram of a knowledge tracking method and system based on reinforcement learning and negative sampling optimization provided by the present invention; Figure 2It is a logical architecture diagram of a knowledge tracking method and system based on reinforcement learning and negative sampling optimization provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings and embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0023] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method, step or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such method, step or device. In the absence of further restrictions, the elements defined by the sentence "include a ..." do not exclude the existence of other identical elements in the method, step or device including the elements.

[0024] In order to improve the predictive performance of the knowledge tracking method and enhance the precision and accuracy of the prediction, the present invention provides a knowledge tracking method and system based on reinforcement learning and negative sampling optimization, which inputs the learner's historical answer sequence into the established knowledge tracking model to obtain the learner's future answer situation. That is, the input of the knowledge tracking model is the learner's historical answer sequence, and the output is the learner's future answer situation.

[0025] On the one hand, if Figure 1 and 2 As shown, the method includes: S101: Construct a knowledge point representation module for the exercise, that is, obtain a knowledge point set of the learner's historical answer sequence and obtain the knowledge point representation of the exercise.

[0026] The knowledge point representation of an exercise is obtained through the knowledge points contained in the exercise and the relationship between the exercises. It should be noted that an exercise includes multiple knowledge points, and each knowledge point plays a different role in the exercise. In the knowledge tracking model, the implicit features of the exercise are captured by extracting the feature information of the knowledge points contained in the exercise. The knowledge point representation of the exercise includes the historical knowledge point representation of the historical exercises and the current knowledge point representation of the current exercise.

[0027] Define the learner's historical answer sequence as ;in, Indicates the learner's current answer record. Indicates the answer record at the last moment.

[0028] Define the learner's current answer record as ;in, Indicates the exercise that the learner is currently answering; Indicates the learner's answer to the current exercise, 0 indicates an incorrect answer, and 1 indicates a correct answer; Indicates all the knowledge points included in the current exercise. The number of knowledge points included is , then the current exercise The corresponding knowledge point set is .

[0029] By constructing the knowledge point encoding of the exercise and combining it with the attention mechanism to calculate the knowledge point weights of each knowledge point, we can finally get the knowledge point representation of the exercise.

[0030] As an embodiment, obtaining the knowledge point representation includes: obtaining a set of knowledge points of an exercise, performing one-hot encoding on each knowledge point to obtain a one-hot encoding vector for each knowledge point; simultaneously obtaining the knowledge point weight of each knowledge point in the exercise; and obtaining the knowledge point representation of the exercise based on the one-hot encoding vector and the knowledge point weight.

[0031] It should be noted that after obtaining the one-hot encoding vector, in order to reflect the importance of each knowledge point in the exercise, it is also necessary to assign a knowledge point weight to each knowledge point. These knowledge point weights are gradually optimized through the back propagation algorithm during the model training process.

[0032] More specifically, knowledge points represent It is obtained by weighted summation of knowledge points, and its calculation formula is: ;in, Indicates exercises; Indicates knowledge points, Representation Exercises The corresponding knowledge point set, ; Represents the weight of knowledge points, satisfying ,and ; represents a one-hot encoded vector, , , and knowledge points The corresponding position is marked as 1, and the rest of the positions are marked as 0; Indicates the size of the knowledge point set.

[0033] S102: Constructing an exercise graph module. That is, generating an initial exercise graph according to the knowledge point representation, and dynamically adjusting the edge weights of the initial exercise graph to obtain an adjusted exercise graph.

[0034] It should be noted that in the process of learners answering exercises, analogies of historical exercises are used to assist in answering. This process reveals the characteristic associations between exercises and reflects the association of exercises in learners' answering situations. Therefore, the exercise graph is constructed by representing historical knowledge points and current knowledge points, and the edge weights between nodes are calculated using the graph attention network, and the exercise graph is adjusted by updating the edge weights.

[0035] The exercise graph consists of nodes and edges, where nodes represent exercises and edges represent the relationship between exercises. Constructing an exercise graph can more accurately and comprehensively capture and express the complex relationship between exercises. It not only retains the knowledge point information of the exercises, but also reflects the potential connection between exercises through edge weight adjustment.

[0036] As an embodiment, the construction of the initial exercise graph includes: And the current knowledge point representation , with all exercises As nodes, the relationships between different exercises are used as edges to construct the initial exercise graph .in, ; Represents a collection of nodes. ; represents the edge set, , and Respectively represent exercises and Exercises A collection of knowledge points, Representation Exercises and Exercises The shared knowledge point set. If the shared knowledge point set is a non-empty set, then in the exercise and Exercises Add an edge between the two corresponding nodes .

[0037] In order to further enhance the expressive power of the exercise graph, the present invention introduces a graph attention mechanism to dynamically adjust the edge weights of the initial exercise graph to obtain the adjusted exercise graph ,in, represents the edge set in the adjusted exercise graph, ; Represents edge weight.

[0038] Specifically, initially, since these edges are established based on shared knowledge points, the edge weights can be obtained by firstly The edge weight is initialized to 1, and the two nodes connected by the edge and The features are linearly transformed and activated, and two new nodes are obtained accordingly. ; Then, calculate the new node The attention between them is calculated, and the edge weight is obtained according to the attention.

[0039] use The function normalizes the attention of all neighbor nodes to obtain edge weights. As an embodiment, the edge weight calculation formula is: ; in, Represents a new node attention between; Represents a new node and neighbor nodes attention between; Representation Node The set of neighbor nodes.

[0040] Furthermore, the calculation formula of attention is: ; in, represents the parameter vector of the attention mechanism, represents the concatenation operation of feature vectors, Represents the activation function.

[0041] Step 103: construct a graph sampling module, that is, use the exercise graph to obtain a set of historical positive samples and a set of historical negative samples associated with the current exercise.

[0042] The graph sampling module adopts the neighbor node sampling strategy to extract the historical positive sample set, that is, by sampling the neighbor nodes of the current exercise, the historical exercise samples with a high correlation with the current exercise are selected.

[0043] As an embodiment, obtaining the historical positive sample set includes: Find the problem that matches the current problem in the problem graph Directly connected historical exercises, get the set of neighbor nodes , and then get the edge set in the adjusted exercise graph , and based on the edge set Get the adjusted edge weights ; Multiple edge weights Sort from high to low, choose The historical exercises of the neighbor nodes corresponding to the edge weights are taken as positive samples to obtain the positive samples related to the current exercises. A set of historical positive samples with high correlation: . At the same time, the historical positive sample set Knowledge point representation Extract it.

[0044] In the exercise graph, the edge weights between exercise nodes reflect the similarity between exercises. Similar exercise nodes have similar sampling distributions. Therefore, in the negative sample sampling process, nodes with greater similarity can be selected, which is conducive to accelerating the convergence speed of the knowledge tracking model.

[0045] Exercises The nodes in Perform weighted probability-first traversal. The weighted probability-first traversal can be based on the traditional depth-first search (DFS) strategy and introduce edge weights between nodes.

[0046] As an embodiment, at each traversal, the traversal order is determined according to the edge weights between nodes, thereby ensuring that nodes with greater similarity are traversed first. Specifically, for the current node, all nodes connected to the current node are calculated and recorded as a neighbor node set. In the neighbor node set, the node that is most similar to the current node is selected as a positive sample and added to the historical positive sample set.

[0047] However, as the traversal proceeds, the similarity between nodes gradually decreases, resulting in differences in the sampling distribution of samples, affecting the model training effect.

[0048] In order to improve the model training effect, the present invention introduces reinforcement learning to optimize the negative sample sampling strategy, and proposes a negative sample sampling strategy that combines the Markov chain Monte Carlo method (MCMC) with reinforcement learning. The sampling strategy of the Markov chain is optimized by reinforcement learning, thereby improving the sampling efficiency of negative samples.

[0049] After obtaining the exercise graph, the exercise graph is traversed based on the edge weights to obtain the exercise node sequence, in which each node is a central node. The last node of the previous central node in the Markov chain is used as the initial node of the Markov chain of historical exercises similar to the current exercise, and reinforcement learning sampling is performed to obtain a set of historical negative samples with a high correlation with the current exercise.

[0050] In the negative sampling process, a Markov chain is used to generate negative samples. Each time, an action is selected through the reinforcement learning algorithm function according to the current state space to determine the next node.

[0051] As an embodiment, a method for obtaining a historical negative sample set includes: S301: Traverse the exercise graph based on edge weights to obtain an exercise node sequence.

[0052] It should be noted that the central node is obtained by traversing the node of the current exercise and the last node of the historical exercise in the historical answer sequence. Among them, the last node of the historical exercise is the starting node, the node of the current exercise is the end node, and the exercise node sequence is obtained by traversing the edge weights in the middle. Each node in the exercise node sequence is a central node.

[0053] S302: Randomly select a central node in the exercise node sequence as the initial node of the Markov chain.

[0054] S303: For each central node, calculate the current state space corresponding to the current central node, and select the next central node according to the reinforcement learning algorithm function.

[0055] It should be noted that at the first sampling, the initial node Can be used as the initial state space , so that the next central node is selected The process is a reinforcement learning problem.

[0056] The reinforcement learning algorithm function is the Q-learning algorithm function. Q-learning is a reinforcement learning algorithm based on value iteration, which is updated by maximizing the expected return of the current strategy.

[0057] As an embodiment, the update formula of the reinforcement learning algorithm function is: ; in, Represents the current state space; represents the next state space; represents the learning rate; represents the discount factor; In the next state space The action below; Indicates execution of an action Rewards received afterwards.

[0058] S304: updating the corresponding state space according to the next central node, calculating the reward function, and updating the reinforcement learning algorithm function according to the reward function; The reward function is a function that evaluates the sampling quality based on the similarity between the central nodes and the edge weights. Nodes with higher correlation will be given higher rewards; Irrelevant nodes are given negative rewards.

[0059] As an embodiment, the reward function is calculated based on the edge weight between the current central node and the next central node and the weighted shortest path length between the two central nodes.

[0060] Specifically, the reward function can be expressed as: ; in, Indicates the current center node, Indicates the next central node; Indicates the current center node With the next central node The edge weights between express The set of neighbor nodes of Represents the adjustment factor, which is used to balance the rewards of positive and negative samples; Represents the weighted shortest path length between the current node and the next node.

[0061] S305: Repeat the above steps until a node is sampled from the exercise node sequence. negative samples, and thus obtain the historical negative sample set.

[0062] S104: Constructing a contrastive learning module. That is, calculating similarity based on a historical positive sample set and a historical negative sample set, and obtaining a knowledge point vector of the current exercise through sample loss function optimization.

[0063] Through the graph sampling module, a set of positive and negative samples associated with the current exercise is obtained. The contrastive learning module is used to cluster the positive sample exercises and distinguish the negative sample exercises, so as to generate a knowledge point vector representation that can fully reveal the internal structure and pattern of the current exercise.

[0064] Therefore, the present invention first calculates the positive similarity between the current knowledge point representation and the historical positive sample set, as well as the negative similarity between the current knowledge point representation and the historical negative sample set; then compares the positive similarity and the negative similarity, optimizes the exercise sample through the sample loss function, obtains the exercise representation of the current exercise, and reduces the historical exercise samples that are irrelevant to the current exercise.

[0065] It should be noted that similarity calculation is used to evaluate the similarity between the positive and negative sample sets of the current exercise and the historical exercises, so that the knowledge tracking model can capture the characteristics of the exercises and their correlation, thereby enhancing the ability to distinguish; at the same time, by analyzing a large amount of historical data, the generalization ability and prediction performance of the knowledge tracking model are improved.

[0066] In order to calculate the similarity between the positive and negative sample sets of the current exercise and the historical exercises, the cosine similarity measurement method is adopted. As a specific embodiment, the similarity calculation formula is: ; in, Representation vector With vector The dot product of and Represents vectors and vector The L2 norm of . Indicates the current question The knowledge point vector of Represents the negative sample set The node vector of the history exercise.

[0067] In order to calculate the sample loss function, the similarity results of all positive and negative sample sets of the current exercise and the historical exercises are stored in the similarity matrix.

[0068] Specifically, the similarity matrix is ​​calculated as: ; in, Indicates the current question The knowledge point vector of Indicates the first The nodes of the history exercises are represented by nodes in the exercise graph. ; Represents the negative sample set The nodes of the history exercises are represented by nodes in the exercise graph. .

[0069] The sample loss function plays a key role in optimizing the sample set. By reducing the similarity between the positive samples of the current question and the historical questions and increasing the similarity between the negative samples of the current question and the historical questions, the knowledge tracking model can more accurately capture and distinguish the characteristics of different questions.

[0070] As an embodiment, the sample loss function is obtained by summing the losses of all exercises, specifically: ; in, ; Indicates the current question The knowledge point vector of Indicates the first The nodes of the history exercises are represented by nodes in the exercise graph. ; Represents the negative sample set The nodes of the history exercises are represented by nodes in the exercise graph. ; Represents the set of negative samples.

[0071] S105: Constructing a prediction module. That is, dynamically updating the learner's knowledge state according to the knowledge point vector and outputting the learner's future answer status.

[0072] In the process of predicting the learner's answer, the core task is to reasonably predict the learner's future answer based on the learner's knowledge state. Through the optimized knowledge point vector, the learner's knowledge state changes are dynamically updated to obtain the learner's future answer.

[0073] The present invention introduces a gated recurrent unit (GRU) network structure to help the knowledge tracking model capture the learner's current memory of historical answer exercises and the degree of dependence on previously learned content, so as to update the learner's knowledge state and predict the learner's answer at the next moment based on the learner's knowledge state. The learner's knowledge mastery is represented by capturing the hidden state and mapped to the probability of answering the next question.

[0074] In the learner's knowledge state updating stage, the goal is to gradually update the learner's knowledge state by inputting exercise knowledge points. Therefore, the present invention introduces a gated recurrent unit network structure to capture the learner's knowledge state changes in the historical answer sequence. Further, the learner's knowledge state changes in the historical answer sequence are captured by obtaining the learner's memory of the historical answer exercises and the learner's dependence on the previous learning content.

[0075] In order to obtain the learner's memory of historical answer exercises, the present invention controls the retention degree of the previous time step information in the current time step through the update gate, reflecting the learner's reliance on historical knowledge when processing new knowledge.

[0076] As an example, the update gate The calculation method is: ; in, , Represents the Sigmoid activation function; and represents the weight matrix; Indicates the current time step Input; Represents the hidden state of the current time step, which is also the final hidden state; Represents the bias term.

[0077] In order to adjust the learner's reliance on previously learned content, so that when dealing with new exercises, the learner's knowledge state can be effectively updated without being disturbed by historical exercise factors; the present invention uses a reset gate to control the degree of ignoring the hidden state of the previous time step.

[0078] As an example, the reset gate The calculation method is: ; in, , Represents the Sigmoid activation function; and represents the weight matrix; Indicates the current time step Input; Represents the hidden state of the current time step, which is also the final hidden state; Represents the bias term.

[0079] It should be noted that in order to better represent the current time step Input , the knowledge point vector of the current exercise is taken as the current time step Input , that is: ;in, , indicating the The knowledge point vector of the exercise, Represents the vector dimension.

[0080] As an example, the final hidden state The hidden state of the previous time step can be and candidate hidden states Weighted fusion.

[0081] Furthermore, the final hidden state The calculation method can be: ; in, represents the update gate; represents the hidden state of the previous time step; represents a candidate hidden state.

[0082] As an embodiment, the candidate hidden state can be calculated based on the reset gate, and the specific calculation formula is: ; in, represents the candidate hidden state, which is the combination of the hidden state of the current time step and the adjusted hidden state of the previous time step; represents element-wise multiplication; Represents the Sigmoid activation function; and represents the weight matrix; Indicates the current time step Input; Reset gate. represents the hidden state of the previous time step; Represents the bias term.

[0083] It should be noted that after the learner's knowledge state is dynamically updated, the learner's answer to the next exercise is predicted based on the updated knowledge state. The prediction module of the present invention only depends on the final hidden state of the current time step. , it can avoid using the learner’s historical answer sequence, thereby improving the objectivity of the prediction.

[0084] The prediction of the learner's answer to the next exercise can be calculated through the output of the fully connected layer and the Sigmoid activation function.

[0085] In order to optimize the performance of the prediction module, the present invention adopts the cross-entropy loss function to measure the difference between the predicted results and the actual answer results. The minimum cross-entropy loss function makes it possible to more accurately fit the actual answer situation when predicting the learner's answer situation, thereby improving the accuracy of the prediction.

[0086] As an example, the cross entropy loss function is: ; in, represents the total number of samples; , represents the actual answer result, , which represents the predicted probability that the learner will answer correctly.

[0087] It should be noted that the predicted probability It can be calculated by the fully connected layer and the Sigmoid activation function. Further, the predicted probability The calculation formula is: ; in, Represents the Sigmoid activation function; Represents the weight matrix of the fully connected layer; represents the final hidden state; represents the bias term; The prediction module predicts the learner's future answers through optimized knowledge point vectors.

[0088] Considering that different examination weights of exercises for knowledge points will lead to different difficulties of exercises, the present invention will The knowledge point vector of the distributed exercises to be answered Concatenate and obtain the updated knowledge state through a fully connected layer with an activation function , which is used to represent the future state of knowledge points and the relationship between the states of each future knowledge point, as well as information such as learning behavior.

[0089] Further, the updated knowledge status The calculation formula is: ; in, Represents the Sigmoid activation function; represents the weight matrix; Represents the knowledge point vector to be answered; Represents the current state of the knowledge point, which is equal to the final hidden state; Represents the bias term.

[0090] Output the learner's future answer status based on the updated knowledge state vector, including:

[0091] in, represents the weight matrix; Indicates the updated knowledge status; Represents the bias term.

[0092] In addition, in order to verify that the prediction of the present invention is more accurate, the present invention verifies the performance of the method through experiments, and compares the prediction accuracy of the present invention with that of other existing methods. The comparison results are shown in Table 1: Table 1 Comparison of prediction results between the present invention and other existing methods

[0093] It can be seen that the knowledge tracking method based on negative sampling using the improved Markov Chain Monte Carlo method proposed in the present invention has a higher prediction accuracy, which proves the superiority of this method in the knowledge tracking task.

[0094] In a second aspect, the present invention provides a knowledge tracking system based on reinforcement learning and negative sampling optimization, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods. The technical features of the system are consistent with the technical features of the above methods, and are not described one by one here.

[0095] In summary, the present invention adjusts the exercise graph and samples the positive and negative sample sets based on the added exercise graph. It adopts the negative sampling strategy of weighted probability priority traversal to optimize the knowledge point vector represented by the exercise, so that the sample set is close to the target distribution, improves the clustering effect of the positive sample exercise nodes, and effectively distinguishes the negative sample exercise nodes, effectively captures the potential correlation between exercises, enhances the predictive reasoning ability, and improves the prediction accuracy.

[0096] It should be noted that, for the above-mentioned various embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0097] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed method or system can be implemented in other ways. For example, the above-described embodiments are only schematic, and the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0099] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0100] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0101] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0102] Those skilled in the art will appreciate that all or part of the various circuits in the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0103] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0104] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge tracking method based on reinforcement learning and negative sampling optimization, characterized in that: The method comprises: Obtain the knowledge point set of the learner's historical answer sequence and obtain the knowledge point representation of the exercise; Generating an initial exercise graph according to the knowledge point representation, and dynamically adjusting the edge weights of the initial exercise graph to obtain an adjusted exercise graph; Using the problem graph, obtaining a set of historical positive samples and a set of historical negative samples associated with the current problem; Calculate the similarity based on the historical positive sample set and the historical negative sample set, and optimize the sample loss function to obtain the knowledge point vector of the current exercise; The learner's knowledge state is dynamically updated according to the knowledge point vector, and the learner's future answer status is output based on the updated knowledge state.

2. According to claim 1, a knowledge tracking method based on reinforcement learning and negative sampling optimization is characterized in that: The knowledge point representation is obtained by: A set of knowledge points of the exercise is obtained, and each knowledge point is one-hot encoded to obtain a one-hot encoded vector of each knowledge point; meanwhile, a knowledge point weight of each knowledge point in the exercise is obtained; and a knowledge point representation of the exercise is obtained based on the one-hot encoded vector and the knowledge point weight.

3. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 2 is characterized in that: The knowledge point represents It is obtained by weighted summation of knowledge points, and the calculation formula is: ； in, Indicates exercises; Indicates knowledge points, Representation Exercises The corresponding knowledge point set, ; Represents the weight of knowledge points, satisfying ,and ; represents a one-hot encoded vector, , , and knowledge points The corresponding position is marked as 1, and the rest of the positions are marked as 0; Indicates the size of the knowledge point set.

4. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 1 is characterized in that: The edge weights can be obtained by: Each edge The edge weight is initialized to 1, and the two nodes connected by the edge and The features are linearly transformed and activated, and two new nodes are obtained accordingly. ; Calculate new nodes The attention between them is calculated, and the edge weight is obtained according to the attention.

5. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 1 is characterized in that: The method for obtaining the historical negative sample set includes: S301: traverse the exercise graph based on edge weights to obtain an exercise node sequence; S302: randomly selecting a central node in the exercise node sequence as the initial node of the Markov chain; S303: For each central node, calculate the current state space corresponding to the current central node, and select the next central node according to the reinforcement learning algorithm function; S304: updating the corresponding state space according to the next central node, calculating the reward function, and updating the reinforcement learning algorithm function according to the reward function; S305: Repeat steps S303 to S304 until a node is sampled from the exercise node sequence. negative samples, and thus obtain the historical negative sample set.

6. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 5, characterized in that: The reward function is: ； in, Indicates the current center node, Indicates the next central node; Indicates the current center node With the next central node The edge weights between ; express The set of neighbor nodes of Represents the adjustment factor, which is used to balance the rewards of positive and negative samples; Represents the weighted shortest path length between the current node and the next node.

7. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 5, characterized in that: The update formula of the reinforcement learning algorithm function is: ； in, Represents the current state space; represents the next state space; represents the learning rate; represents the discount factor; Indicates that in the next state space The action below; Indicates execution of an action Rewards received afterwards.

8. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 1, characterized in that: The calculation formula of the similarity is: ； in, Representation vector With vector The dot product of and Represents vectors and vector The L2 norm of ; Indicates the current question The knowledge point vector of Represents the negative sample set The node vector of the history exercise.

9. The knowledge tracking method based on reinforcement learning and negative sampling optimization according to claim 1, characterized in that: Output the learner's future answer status based on the updated knowledge state vector, including: in, represents the weight matrix; Indicates the updated knowledge status; Represents the bias term.

10. A knowledge tracking system based on reinforcement learning and negative sampling optimization, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 9.

Images