A cognitive diagnosis method based on reliable student ability representation

Abstract

The application discloses a cognitive diagnosis method for student ability representation based on reliability, and steps of the method comprise the following steps: firstly, modeling the student ability representation as a distribution based on a Bayesian method, and using variance in the distribution to represent the reliability of the diagnosed ability representation; secondly, considering potential differences of each student, and using a pre-training model to establish individual prior distribution of latent variables of different ability concepts; thirdly, using uncertainty regularization technology to make the variance in the distribution neither too volatile nor too stable; and finally, using a calibration loss function to ensure the reliability of the diagnosed ability representation. The application integrates the Bayesian method and the calibration loss function, and can perform cognitive diagnosis on the student ability representation and evaluate the reliability of the ability representation.

Description

TECHNICAL FIELD

[0001] The present application relates to student personalized cognitive diagnosis in the field of smart education. The main technical idea is to model students as a probability distribution using Bayesian models, providing feedback on the reliability of the diagnostic results while diagnosing their ability levels. This method aims to provide more accurate and targeted learning recommendations for students, helping them better understand and master the knowledge they learn. BACKGROUND

[0002] In recent years, smart education has attracted increasing attention. This field uses student answer data for analysis to diagnose their mastery of specific knowledge concepts. These diagnostic results can be used by teachers in teaching, course recommendation, and test selection.

[0003] Cognitive diagnosis, as a basic but important technology in smart education, originated from educational psychology. They model various factors that affect student answers from multiple aspects, such as student ability vectors, exercise difficulty, discrimination, guessing, and errors. These diagnostic factors are combined through different diagnostic functions designed by hand to model the final student answer. With the increasing application of deep neural networks, more and more researchers have begun to combine neural networks and cognitive diagnosis, using text information or knowledge concept relationships to enrich student and exercise-related representations, or using neural networks to fit complex interactions between students and exercises.

[0004] Despite the significant progress made in cognitive diagnosis, there are still many challenges. One of the main problems is that existing methods often ignore the reliability feedback of diagnostic results. This unreliability may be due to factors such as model errors, data noise, etc. If the diagnostic results are unreliable, they cannot accurately reflect the student's ability status and cannot be used as a reliable standard for downstream tasks. Therefore, more attention needs to be paid to this issue and measures need to be taken to improve the reliability of diagnosis. SUMMARY

[0005] To solve the above problems, the present application proposes a cognitive diagnosis method based on reliable student ability representation, which can accurately obtain the student's ability representation and evaluate the reliability of the student's ability representation, thereby helping to better understand and master the knowledge learned.

[0006] To achieve the above application purposes, the following technical solutions are adopted:

[0007] The cognitive diagnosis method based on reliable student ability representation of the present application is characterized in that it is performed according to the following steps:

[0008] Step 1, define a student set S = {s1, s2, …, s i..., s N}, wherein s i represents the i-th student, and N represents the number of students;

[0009] Define an exercise set E = {e1, e2,..., e j ..., e M}, wherein e j represents the j-th exercise, and M represents the number of exercises;

[0010] Define a knowledge concept set C = {c1, c2,..., c k ..., c K}, wherein c k represents the k-th knowledge concept, and K represents the number of knowledge concepts;

[0011] Let the i-th student s i 's answer log for the j-th exercise e j be R ij = {s i , e j , r ij}, wherein r ij is the i-th student s i 's true answer result for the j-th exercise e j ;

[0012] Step two, the processing of the ability vector calculation module:

[0013] Step 2.1, obtain the i-th student s i 's ability representation distribution

[0014]

[0015] In formula (1), | represents the conditional probability, represents the one-hot encoding of the i-th student s i ; z i represents a vector randomly sampled from the normal distribution ; μ i , respectively represent the mean and variance of the i-th student s i , and are obtained by formula (2), and d is the dimension of the hidden vector;

[0016]

[0017] In formula (2), W μ , W σ ∈ R N×d are the mean transfer matrix and the variance transfer matrix to be trained; T represents transposition;

[0018] Step 2.2, use formula (3) to get the i-th student s i The variance of the i-th student s k on the k-th knowledge concept c Constrained, get the constrained variance

[0019]

[0020] In formula (3), denotes the i-th student s i on the k-th knowledge concept c k ; α is a pre-set parameter; will be defined as the i-th student s i on the k-th knowledge concept c k diagnostic ability characterization reliability;

[0021] Step 2.3, use formula (4) to map the sampled vector z i , get the ability vector θ i of the i-th student s i ∈R d :

[0022]

[0023] In formula (2), ~ represents sampling, denotes the Sigmoid activation function;

[0024] Step three, the processing of the cognitive diagnosis module:

[0025] Step 3.1, use formula (4) to get the difficulty degree j and the discrimination degree of the j-th exercise e

[0026]

[0027] In formula (5), denotes the one-hot encoding of the j-th exercise e j ; W diff ∈R M×d , W disc ∈R M×1 respectively denote the difficulty matrix and the discrimination matrix to be trained;

[0028] Step 3.2, use formula (6) to get the prediction probability ρ(y i =1) of the i-th student s j answering the j-th exercise e ij correctly:

[0029]

[0030] In equation (6), y ij s represents the i-th student i For the j-th exercise e j The predicted response result, f cd It is a diagnostic paradigm that includes diagnostic factors, such as the student's ability vector, the difficulty of the exercise, and the discrimination factor.

[0031] Step 4: Construct a reliability cognitive diagnosis model using the capability vector calculation module and the cognitive diagnosis module, and define the training objective for the reliability cognitive diagnosis model:

[0032] Step 4.1: Optimize the parameters of the reliability cognitive diagnostic model using equation (7):

[0033]

[0034] In equation (7), p φ (z i ) is the i-th student s i The prior distribution of the ability representation; s represents the i-th student i The posterior distribution of the ability representation; p(r ij |z i ) is the i-th student s i Ability vector θ i For the j-th exercise e j The correct answer is r. ij The predicted probability; E represents the expected value; It is to find the distribution and distribution p φ (z i The KL divergence of ) is calculated; and log p(r) is calculated using equations (7) and (8) respectively. ij |z i )and

[0035] L pred =log p(r ij |z i ) = r ij logy ij +(1-r ij log(1-y) ij (8)

[0036]

[0037] In equations (8) and (9), L pred and LKL respectively denote the prediction loss and the KL loss, μ ik and denote the mean and variance of the ith student s i on the kth knowledge concept c k ; N(μ mean , 1) denotes the prior consensus of all students on all knowledge concepts and is derived from equation (10); μ mean，k denotes the mean of all students on the knowledge concept c k :

[0038]

[0039] In equation (10), μ i ∈ R K denotes the prior knowledge of the ith student s i on all knowledge concepts;

[0040] Step 4.2, construct the partial order relation of the confidence and variance of the ith student s i and the u-th student s u on the kth knowledge concept c k and the v-th knowledge concept c v using equation (11):

[0041]

[0042] In equation (11), denotes equivalent to; θ ik denotes the mastery level of the ith student s i on the kth knowledge concept c k ; θ uv denotes the mastery level of the u-th student s u on the v-th knowledge concept c v ; p(y ij = r ij | θ ik ) denotes the probability of correctly answering the jth exercise e i when the mastery level of the ith student s k on the kth knowledge concept c ik is θ j ; denotes the confidence of the ith student s i on the kth knowledge concept c k ; denotes the confidence of the u-th student s u on the v-th knowledge concept c v ;

[0043] Step 4.3, constructing the loss function L by using formula (12) RL :

[0044]

[0045] In formula (12), o ik represents the proportion of the number of times that the i-th student s i predicts correctly when answering the k-th knowledge concept c k to the total number of times that the i-th student s i answers the k-th knowledge concept c k ; o uv represents the proportion of the number of times that the u-th student s u predicts correctly when answering the v-th knowledge concept c v to the total number of times that the u-th student s u answers the v-th knowledge concept c v ; g() represents a comparison function, and has:

[0046]

[0047] Step 5, constructing the total loss function L by using formula (14):

[0048] L = L pred + γ × L' KL + β × L RL (14)

[0049] In formula (14), γ and β are different hyperparameters;

[0050] Step 7, training the reliability cognitive diagnosis model by using the gradient descent method, and calculating the total loss function L to update the model parameters until the total loss function L converges, so as to obtain the trained student ability representation reliability model, which is used for predicting the student ability representation and its reliability.

[0051] The student ability representation reliability modeling method based on Bayes provided by the application also has the following characteristics:

[0052] If the diagnostic paradigm f cd containing the diagnostic factor is adopted, the IRT model of the project response theory is adopted, the loss function of the adaptive IRT model is obtained by using formula (15) and replacing L RL in the total loss function L:

[0053]

[0054] In formula (15), o i represents the proportion of the number of times that the i-th student s iThe number of times of correct prediction in all answered exercises accounts for the ith student s i The proportion of the total number of answers in all answered exercises u The u-th student s u The number of times of correct prediction in all answered exercises accounts for the u-th student s u The proportion of the total number of answers in all answered exercises.

[0055] The electronic device of the present application comprises a memory and a processor, characterized in that the memory is used to store programs supporting the processor to execute the cognitive diagnosis method, and the processor is configured to execute the programs stored in the memory.

[0056] The computer-readable storage medium of the present application stores a computer program, characterized in that the computer program is executed by the processor to perform the steps of the cognitive diagnosis method.

[0057] Compared with the prior art, the present application has the following beneficial effects:

[0058] The present application aims to obtain the diagnostic reliability of student ability representation by introducing a reliable cognitive diagnosis model. First, the student's ability representation is modeled using the Bayesian method, which is represented as a distribution, and the variance of the distribution is used to represent the reliability of the diagnostic ability. At the same time, by using the calibration loss function, the obtained reliability evaluation is further ensured to be more accurate and reasonable. By using this method, not only the student's ability representation can be obtained, but also the reliability evaluation of the ability representation can be obtained, thereby improving the overall diagnostic quality. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 The present application is a method for the overall flowchart. DETAILED DESCRIPTION

[0060] In this embodiment, a reliable student ability representation cognitive diagnosis method is used to model the student's ability representation as a distribution based on the Bayesian method, and the variance in the distribution is used to represent the reliability of the diagnostic ability representation. At the same time, the calibration loss function is used to ensure the reliability of the diagnostic ability representation. Specifically, as shown in Figure 1 The following steps are taken:

[0061] Step one, define the student set S = {s1, s2,..., s i ,..., s N}, wherein s i represents the ith student, and N is the number of students;

[0062] Define the exercise set E = {e1, e2,..., ej ..., e M}, where e j represents the jth exercise, and M is the number of exercises;

[0063] define a set of knowledge concepts C = {c k ..., c K}, where c k represents the kth knowledge concept, and K is the number of knowledge concepts;

[0064] let the answer log of the ith student s i to the jth exercise e j be denoted as R ij = {s i , e j , r ij}, where r ij is the true answer result of the ith student s i to the jth exercise e j ;

[0065] Step two, processing of the ability vector calculation module:

[0066] Step 2.1, obtain the ability representation distribution of the ith student s i using formula (1)

[0067]

[0068] In formula (1), | represents the conditional probability, represents the one-hot encoding of the ith student s i ; z i represents a vector randomly sampled from the normal distribution ; μ i , respectively represent the mean and variance of the ith student s i , and are obtained by formula (2), and d is the dimension of the hidden vector;

[0069]

[0070] In formula (2), W μ , W σ ∈ R N×d are the mean transfer matrix and variance transfer matrix to be trained; T represents transposition;

[0071] Step 2.2, constrain the variance i of the ith student s k on the kth knowledge concept c to variance

[0072]

[0073] In formula (3), denotes the ith student s i on the kth knowledge concept c k ; α is a pre-set parameter; and is defined as the ability vector θ i of the ith student s k on the kth knowledge concept c i ;

[0074] Step 2.3, mapping the sampled vector z i using formula (4) to obtain the ability vector θ i of the ith student s d :

[0075]

[0076] In formula (2), ~ denotes sampling, denotes a Sigmoid activation function;

[0077] Step three, processing of the cognitive diagnosis module:

[0078] Step 3.1, obtaining the difficulty degree j and the discrimination degree of the jth exercise e using formula (4):

[0079]

[0080] In formula (5), denotes one-hot encoding of the jth exercise e j ; W diff ∈R M×d , W disc ∈R M×1 denote the difficulty matrix and the discrimination matrix to be trained, respectively;

[0081] Step 3.2, obtaining the prediction probability ρ(y i = 1) of the ith student s j answering the jth exercise e ij correctly using formula (6):

[0082]

[0083] In formula (6), y ij denotes the ith student si predicted answer result of the jth exercise e j cd is a diagnostic paradigm including diagnostic factors, which include the ability vector of a student, the difficulty degree and the discrimination degree of an exercise;

[0084] Step four, the reliability cognitive diagnosis model is constituted by the ability vector calculation module and the cognitive diagnosis module, and the training target of the reliability cognitive diagnosis model is constructed:

[0085] Step 4.1: the parameters of the reliability cognitive diagnosis model are optimized by using formula (7):

[0086]

[0087] In formula (7), p φ (z i ) is the prior distribution of the ability representation of the ith student s i ; represents the posterior distribution of the ability representation of the ith student s i ; p(r ij |z i ) is the predicted probability that the ability vector θ i of the ith student s i answers the jth exercise e j as the correct answer r ij ; E represents expectation; is the KL divergence of the distribution and the distribution p φ (z i ); and log p(r ij |z i ) and

[0088] L pred = log p(r ij |z i ) = r ij logy ij + (1-r ij ) log (1-y ij ) (8)

[0089]

[0090] In formula (8) and formula (9), L pred and L KL respectively represent the predicted loss and the KL loss, μ ik and respectively represent the ability representation of the ith student s i in the kth knowledge concept c k ​Mean and variance of N(μ) mean ,1) represents the prior consensus of all students on all knowledge concepts, which can be derived from equation (10); μ mean，k This indicates that all students are in the knowledge concept c k The mean of the above:

[0091]

[0092] In equation (10), μ i ∈R K s represents the i-th student i Prior knowledge of all concepts of knowledge;

[0093] Step 4.2: Construct the i-th student s using equation (11). i And the uth student s u The k-th knowledge concept c k and the vth knowledge concept c v The partial order relationship between confidence level and variance:

[0094]

[0095] In equation (11), This means it is equivalent to; θ ik s represents the i-th student i The k-th knowledge concept c k The degree of mastery on θ uv s represents the u-th student u In the vth knowledge concept c v The degree of mastery of the above; p(y ij =r ij |θ ik ) represents the i-th student s i The k-th knowledge concept c k The level of mastery is θ ik At that time, for the j-th exercise e j The probability of answering correctly; s represents the i-th student i The k-th knowledge concept c k Confidence level; s represents the u-th student u In the vth knowledge concept c v Confidence level;

[0096] Step 4.3: Construct the loss function L using equation (12). RL :

[0097]

[0098] In equation (12), oik represents the ith student s i predicts correctly when answering the kth knowledge concept c k of the ith student s i predicts correctly when answering the kth knowledge concept c k of the ith student s uv represents the uth student s u predicts correctly when answering the vth knowledge concept c v of the uth student s u predicts correctly when answering the vth knowledge concept c v of the uth student s

[0099]

[0100] Step five, construct the total loss function L using formula (14):

[0101] L = L pred + γ × L' KL + β × L RL (14)

[0102] In formula (14), γ and β are different hyperparameters; if the diagnostic paradigm f cd contains a diagnostic factor, then the loss function of the adaptive IRT model is obtained using formula (15) and replacing L in the total loss function L RL :

[0103]

[0104] In formula (15), o i represents the ith student s i predicts correctly when answering all exercises of the ith student s j predicts correctly when answering all exercises of the ith student s u represents the uth student s u predicts correctly when answering all exercises of the uth student s u predicts correctly when answering all exercises of the uth student s

[0105] Step seven, train the reliability cognitive diagnosis model using the gradient descent method, and calculate the total loss function L to update the model parameters until the total loss function L converges, thereby obtaining the trained student ability representation reliability model. Using the trained reliability model, not only can the student ability representation be obtained, but also the reliability of the model in diagnosing the student ability representation can be obtained.

[0106] In this embodiment, an electronic device includes a memory and a processor, the memory is configured to store a program supporting the processor to execute the method, and the processor is configured to execute the program stored in the memory.

[0107] In this embodiment, a computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the method.

[0108] Embodiment:

[0109] In order to verify the effectiveness of the method, the application uses three public datasets and one private dataset commonly used in intelligent education research: Assistments2009, Junyi, ENEM and e-Math. The answer records of students with less than 15 answer logs are deleted for each dataset. For the processed dataset Assistments2009, 2493 students, 17671 exercises and 123 knowledge concepts can be obtained; for the processed dataset Junyi, 1000 students, 712 exercises and 39 knowledge concepts can be obtained; for the processed dataset ENEM, 10000 students, 185 exercises and 4 knowledge concepts can be obtained; for the processed dataset e-Math, 517 students, 1582 exercises and 61 knowledge concepts can be obtained.

[0110] For the student performance prediction task, the application uses two types of indicators, one is the commonly used ACC (Accuracy), RMSE (Root Mean Square Error) and AUC (Area Under an ROC Curve), and the other is ECE (Expected Calibration Error) and MCE (Maximum Calibration Error). The smaller the values of ECE and MCE, the better the confidence estimation.

[0111] In the application, three different diagnostic functions IRT, MIRT and NCD are used as baseline, and at the same time, it is also proved that the method of the application can be compatible with different diagnostic functions, so that Reli-IRT, Reli-MIRT and Reli-NCD in the table are obtained. As shown in Table 1:

[0112] Table 1 Experimental results

[0113]

[0114]

[0115] Specifically, firstly, Reli-IRT, Reli-MIRT and Reli-NCD have good effect improvement compared with baseline on all data sets, which shows that the present application can greatly improve the confidence of the result; secondly, Reli-IRT, Reli-MIRT and Reli-NCD are also superior to baseline in ACC, RMSE and AUC, which shows that the present application also has good effect improvement in the prediction task.

Claims

1. A cognitive diagnostic method based on reliable student ability representation, characterized in that, The procedure is as follows: Step 1: Define the student set ,in, Indicates the first There are 1 student, and N is the number of students; Define the exercise set ,in, This represents the j-th exercise, and M is the number of exercises. Define knowledge concept set ,in, Let K represent the k-th knowledge concept, where K is the number of knowledge concepts. Order No. students For the j-th exercise The answer log is recorded as ,in, For the first students For the j-th exercise The actual answer result; Step 2: Processing of the capability vector calculation module: Step 2.1: Use equation (1) to obtain the i-th student. Ability representation distribution : (1) In equation (1), Represents conditional probability. Represents the i-th student One-hot encoding; Indicates from the normal distribution A vector randomly sampled from the data; , They represent the i-th student. The mean and variance of the vector are obtained from equation (2), where d is the dimension of the latent vector; (2) In equation (2), , These are the mean transition matrix and variance transition matrix to be trained; Indicates transpose; Step 2.2: Use equation (3) to select the i-th student. The kth knowledge concept variance Apply constraints to obtain the variance after constraints. : (3) In equation (3), This represents the i-th student derived from the standard Bernoulli distribution. The kth knowledge concept An independent random variable on; These are pre-set parameters; Defined as the i-th student The kth knowledge concept The reliability of the diagnostic capability characterization; Step 2.3: Use equation (4) to process the sampled vector. Perform mapping to obtain the i-th student. Ability Vector : (4) In equation (4), Indicates sampling, This represents the Sigmoid activation function; Step 3: Processing of the cognitive diagnosis module: Step 3.1: Use equation (5) to obtain the j-th exercise. Difficulty and discrimination : (5) In equation (5), This indicates the j-th exercise. One-hot encoding; , These represent the difficulty matrix and discrimination matrix to be trained, respectively. Step 3.2: Use equation (6) to obtain the i-th student. For the j-th exercise Predicted probability of correct answer : (6) In equation (6), Represents the i-th student For the j-th exercise The predicted answer results It is a diagnostic paradigm that includes diagnostic factors, such as the student's ability vector, the difficulty of the exercise, and the discrimination factor. Step 4: Construct a reliability cognitive diagnosis model using the capability vector calculation module and the cognitive diagnosis module, and define the training objective for the reliability cognitive diagnosis model: Step 4.1: Optimize the parameters of the reliability cognitive diagnostic model using equation (7): (7) In equation (7), It is the i-th student The prior distribution of the ability representation; Represents the i-th student The posterior distribution of the ability representation; It is the i-th student Ability Vector For the j-th exercise Answer is the correct answer. The predicted probability; Expressing expectations; It is to find the distribution and distribution The KL divergence; and calculate using equations (7) and (8) respectively. and : (8) (9) In equations (8) and (9), and These represent the prediction loss and the KL loss, respectively. and They represent the i-th student. The kth knowledge concept The mean and variance of the vector; This represents the prior consensus of all students on all knowledge concepts, and is derived from equation (10); This indicates that all students have a grasp of the concept of knowledge. The mean of the above: (10) In equation (10), Represents the i-th student Prior knowledge of all concepts of knowledge; Step 4.2: Construct the i-th student using equation (11) and the uth student The kth knowledge concept and the vth knowledge concept The partial order relationship between confidence level and variance: (11) In equation (11), This means equivalent to; Represents the i-th student The kth knowledge concept The degree of mastery of the above, Represents the u-th student The vth knowledge concept The degree of mastery over time; Indicates the i-th student The kth knowledge concept The level of mastery of the above is At that time, for the j-th exercise The probability of answering correctly; Represents the i-th student The kth knowledge concept Confidence level; Represents the u-th student The vth knowledge concept Confidence level; Step 4.3: Construct the loss function using equation (12) : (12) In equation (12), Represents the i-th student The kth knowledge concept The number of times the i-th student correctly predicted the answer is 1 / 2. The kth knowledge concept The proportion of the total number of responses. Represents the u-th student The vth knowledge concept The number of times the student correctly predicted the answer was the same as the number of times the u-th student answered correctly. The vth knowledge concept The proportion of the total number of responses; g() represents the comparison function, and we have: (13) Step 5: Construct the total loss function using equation (14) : (14) In equation (14), and These are different hyperparameters; Step 7: Train the reliability cognitive diagnostic model using gradient descent and calculate the total loss function. To update the model parameters until the total loss function is reached. The training continues until convergence, thus obtaining a reliable model of the student's ability representation after training, which is used to predict the student's ability representation and its reliability.

2. The cognitive diagnostic method based on reliable student ability representation according to claim 1, characterized in that: If the diagnostic paradigm includes diagnostic factors Using the Item Response Theory (IRT) model, the loss function adapted to the IRT model is obtained using equation (15). and replace the total loss function In : (15) In equation (15), Represents the i-th student The percentage of correct predictions made by the i-th student out of all completed exercises. The proportion of the total number of answers given out of all exercises that have already been answered; Represents the u-th student For all exercises that have been answered, the percentage of correct predictions made by the u-th student is [percentage missing]. The proportion of the total number of times a question has been answered out of all the exercises that have already been answered.

3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store programs that support the processor in executing the cognitive diagnostic method of claim 1 or 2, the processor being configured to execute the programs stored in the memory.

4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the cognitive diagnostic method of claim 1 or 2.

Images