A method for evaluating test attributes and students' knowledge level
By adopting a hierarchical Bayesian IRT model based on 2PL-IRT in educational assessment, the problem of the inability of the existing technology to deal with the differences in exam difficulty and changes in student abilities is solved, and more accurate assessment of students' knowledge level is achieved and higher assessment accuracy and stability.
Patent Information
- Application Number
- CN202411212167.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-08-30
AI Technical Summary
Existing evaluation methods, such as IRT models, cannot effectively deal with the differences in difficulty and changes in students' abilities in different exams, resulting in inaccurate evaluation results and difficulty adaptively assessing students' knowledge levels at different learning stages.
The hierarchical Bayesian IRT model based on 2PL-IRT is used to dynamically adjust the evaluation results by setting the prior distribution of students, test questions and exam attributes, and estimating hidden parameters using Stan programming language and advanced sampling algorithm.
It realizes an accurate assessment of individual differences and test questions for different students, reduces the deviation in knowledge level estimation, provides more accurate evaluation results, and improves the accuracy and stability of the evaluation.
Smart Images

Figure CN118734021B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence and educational assessment technology, and in particular to a method for assessing test attributes and student knowledge levels. Background Art
[0002] In K-12 education scenarios, regular tests (such as monthly tests) are widely used to assess students' knowledge mastery at different stages of learning. Students take regular tests, and test scores change over time. These tests reflect students' learning progress and changes in their ability levels, but are also affected by the attributes of the test questions used in each test. Since each test is designed independently, this leads to differences in the content and difficulty of the test questions between tests. However, existing assessment methods, such as some latent growth item response theory (IRT) models used for diagnosis, usually require test questions across multiple measurement time points and are not suitable for a single test. At the same time, they also ignore the differences between different students and the multi-level relationships between test questions, resulting in inaccurate assessment results.
[0003] Especially in practical applications, the limitations of traditional IRT models are more obvious. When evaluating, for example, when encountering differences in exam difficulty, differences in student abilities, and changes in student abilities in different exams, traditional IRT methods seem to be unable to cope. They cannot adaptively evaluate students' knowledge levels at different stages of learning, and it is difficult to handle the dynamic changes in students' abilities over time. This limitation may affect the effectiveness of teaching and learning to a certain extent, and fail to fully capture the subtle changes in students' learning process. Therefore, there is an urgent need for an adaptive evaluation method that can handle the differences in difficulty of different exams and changes in students' abilities. Summary of the invention
[0004] The purpose of the present invention is to provide a method for evaluating test attributes and student knowledge levels, which can dynamically respond to individual differences among students, accurately evaluate the difficulty of test questions, reduce the deviation in estimating students' knowledge levels, and provide more accurate evaluation results.
[0005] The objective of the present invention is achieved through the following technical solutions:
[0006] A method for evaluating test attributes and student knowledge levels, the method comprising:
[0007] Step 1: First, construct a hierarchical Bayesian IRT model based on 2PL-IRT and set the prior distribution of parameters; the prior distribution of student attributes, test question attributes and test attribute parameters follows Gaussian distribution;
[0008] Step 2: Evaluate and verify the established hierarchical Bayesian IRT model to ensure that all parameters are suitable for the task and the model fits well;
[0009] Step 3: Based on the established hierarchical Bayesian IRT model, the calculation process of the hierarchical Bayesian IRT model is converted into a model representation in the Stan programming language, and the latent parameters are estimated through an advanced sampling algorithm.
[0010] It can be seen from the technical solution provided by the present invention that the above method can dynamically respond to individual differences among different students, accurately evaluate the difficulty of test questions, reduce the deviation in estimating students' knowledge level, and provide more accurate evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0012] Figure 1 A schematic diagram of a flow chart of a method for evaluating test attributes and student knowledge levels provided by an embodiment of the present invention;
[0013] Figure 2 This is a schematic diagram of modeling the hierarchical Bayesian IRT model based on 2PL-IRT described in an embodiment of the present invention. DETAILED DESCRIPTION
[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0015] like Figure 1 The figure is a flow chart of a method for evaluating test attributes and student knowledge levels provided by an embodiment of the present invention, the method comprising:
[0016] Step 1: First, construct a hierarchical Bayesian IRT model based on 2PL-IRT and set the prior distribution of parameters; the prior distribution of student attributes, test question attributes and test attribute parameters follows Gaussian distribution;
[0017] In this step, we first need to specify the prior distribution of the parameters. In order to evaluate the knowledge level of the same student group at different time points, we assume that the knowledge level of the same student at different time points follows the same prior distribution, but there may be fluctuations. Secondly, we use the difficulty variable to model the test question attributes, which is also the prior expectation of the test question difficulty. The difficulty of an exam can also be regarded as an indicator of the difficulty of the test questions.
[0018] like Figure 2 FIG. 1 is a schematic diagram of the modeling of the hierarchical Bayesian IRT model based on 2PL-IRT according to an embodiment of the present invention. The hierarchical Bayesian IRT model constructed The definition is as follows:
[0019] ; ; ; ; ; Among them, use Indicates student id; use Indicates the test id; use Represents the topic id; each tuple Uniquely identifies a question, representing the kth question in the jth exam; then among the various parameters, represents the binary score of student i on question k in the jth test; represents the knowledge level, that is, the knowledge level of student i in the jth test; and They represent the discrimination and difficulty of the test questions, respectively, and represent the test question attributes; It represents the difficulty of the jth test. The test difficulty is a summary of the difficulty of the questions and represents the test attribute. That is, the greater the test difficulty, the greater the average difficulty of the questions included. , , is a flat prior parameter, which is set in advance without prior information and does not affect the shape of the posterior distribution.
[0020] Step 2: Evaluate and verify the established hierarchical Bayesian IRT model to ensure that all parameters are suitable for the task and the model fits well;
[0021] In this step, based on the established hierarchical Bayesian IRT model, the relationship between the assessed test difficulty and the average test score is verified, and the average test score and the test difficulty parameter are obtained. The correlation coefficient between them is negative, and the reliability and validity verification results are valid, indicating that the harder the exam is, the harder it is for the students to get the scores they want. In terms of students’ knowledge level, it is assumed that the ranking of students’ knowledge level should be consistent with their ranking of total scores in the exam. The indicator for verifying the hypothesis is the consistency ratio of each exam, and the formula is as follows:
[0022] ;in, is the consistency ratio of each test, ranging from [0,1]; , represents the student id; j represents the test id; is an indicator function, returning 1 or 0, Denotes a binary score Is it higher than ; The product of the two indicator functions represents the number of pairs of items for which all dichotomous scores and knowledge levels agree; It means to count the number of pairs of items with different dichotomous scores;
[0023] The higher the value of , the more accurate the model evaluation is, based on which the model prediction result data is obtained; the consistency ratio data of most exams It is close to 1.0 (the lowest is 94.89%), indicating the validity of the estimation of student ability. In addition, most student pairs are included in the calculation of the evaluation index, which proves the validity of the evaluation index itself.
[0024] In the posterior prediction model test, in order to evaluate the fit of the model and avoid drawing wrong conclusions, the observation data With copy data The posterior predictive distribution is compared with It is expressed by the following formula:
[0025] ;in, represents the model parameters; Given the observation data The posterior distribution of the model parameters; Indicates the model parameters Given a situation where you copy data The probability distribution of The integral is expressed over all model parameters Add up the above;
[0026] Using the Markov Chain Monte Carlo (MCMC) algorithm, we can get the posterior distribution of the parameters. Extract samples from ; then generate replication data based on these samples ;
[0027] Then, the posterior predictive model check (PPMC) is used. The core of the posterior predictive model check PPMC is to obtain the difference value of the observed data. and the difference between the replicated data Any significant difference between them may mean that there is something wrong with the model. and , you can draw Histogram of and marked The posterior prediction p-value (PPP-value) is calculated at the same time. The posterior prediction p-value indicates the Under the condition of Greater than or equal to the observed data difference value Probability , the formula is as follows:
[0028] ;in, Represents a given observation When copying data difference value Greater than or equal to the observed data difference value probability; As an indicator function, only the part that meets the conditions is considered. When , its value is 1, otherwise it is 0; is calculated by integrating all possible replication data Contribution
[0029] Observe the data With copy data The posterior predictive distribution of the χ2 matrix is compared to estimate the model fit.
[0030] In addition, when selecting the difference metric, this application considers multiple aspects to comprehensively evaluate the performance of the model, including:
[0031] 1) Item Percent Correctness Score: measures the accuracy of the model in predicting the correctness of items;
[0032] 2) Observed score distribution: The overall fit of the model was assessed by comparing the distribution of observed scores and replication scores;
[0033] 3) Biserial correlation coefficient: used to quantify the linear correlation between the observed sequence and the replicate sequence;
[0034] 4) Interaction / association between projects:
[0035] The proportion of individuals who answered paired items correctly: This evaluates the model's ability to capture interactions between items;
[0036] Measuring ratios of associations between pairs of items: further quantifies the extent of associations between items and the accuracy with which the model captures these associations.
[0037] In the specific implementation, in order to test the goodness of fit of the established hierarchical Bayesian IRT model, under the assumption that the difficulty of the test questions is the same, the inter-class correlation coefficient (ICC) of the hierarchical IRT model proposed in this study and the benchmark model 2PL-IRT model are compared. The higher the ICC value, the more reliable the sampling result. The definition of ICC is as follows:
[0038]
[0039] in is the variance of test difficulty; Represents the variance of question difficulty.
[0040] Step 3: Based on the established hierarchical Bayesian IRT model, the calculation process of the hierarchical Bayesian IRT model is converted into a model representation in the Stan programming language, and the latent parameters are estimated through an advanced sampling algorithm.
[0041] In this step, the calculation process of the established hierarchical Bayesian IRT model is converted into a model representation in the Stan programming language, and then the HMC algorithm is used in the Stan model to automatically sample and estimate the latent parameter distribution;
[0042] Set up the computational process of the hierarchical Bayesian IRT model, including the prior distribution and the calculation method of the answer score from the parameter estimation; run the computational process through the Stan model, which is a programming language used to specify statistical models, especially for Bayesian inference; and finally use the Hamiltonian Monte Carlo (HMC) algorithm to automatically sample and estimate the latent parameter distribution;
[0043] Among them, when running the Stan model, the observed data and Stan code are used to estimate the parameters of the hierarchical Bayesian IRT model. The specific process is as follows:
[0044] 1) Define data: Input observation data (such as test scores) into the Stan model;
[0045] 2) Compile model: Stan model compiles the corresponding code and generates the corresponding sampler;
[0046] 3) Sampling: Use the Hamiltonian Monte Carlo HMC algorithm to automatically sample from the posterior distribution of the parameters and estimate the latent parameters of the hierarchical Bayesian IRT model;
[0047] Through the above operations, the posterior distribution of the parameter θ representing the knowledge level is obtained, and automatic sampling from the posterior distribution is achieved to estimate the distribution of the hidden parameters.
[0048] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field.
[0049] In summary, the method described in the embodiment of the present invention not only has high accuracy and stability, but also has good robustness and generalization ability by using a hierarchical Bayesian IRT model and Bayesian reasoning to perform parameter estimation and posterior inference optimization. It can effectively evaluate under different test difficulties, different student groups and different test question attributes, reducing the bias and subjectivity of artificial settings, which makes the method have broad application prospects and potential in actual education evaluation, and can provide more accurate tools for education evaluation.
[0050] In addition, a person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be implemented by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a disk or an optical disk, etc.
[0051] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.
Claims
1. A method for evaluating test attributes and student knowledge levels, characterized in that: The method comprises: Step 1: First, construct a hierarchical Bayesian IRT model based on 2PL-IRT and set the prior distribution of parameters; the prior distribution of student attributes, test question attributes and test attribute parameters follows Gaussian distribution; In step 1, the hierarchical Bayesian IRT model constructed The definition is as follows: ; ; ; ; ; Among them, use Indicates student id; use Indicates the test id; use Represents the topic id; each tuple Uniquely identifies a question, representing the kth question in the jth exam; then among the various parameters, represents the binary score of student i on question k in the jth test; represents the knowledge level, that is, the knowledge level of student i in the jth test; and They represent the discrimination and difficulty of the test questions, respectively, and represent the test question attributes; It represents the difficulty of the jth test. The test difficulty is a summary of the difficulty of the questions and represents the test attribute. That is, the greater the test difficulty, the greater the average difficulty of the questions included. , , It is a flat prior parameter, which is set in advance without prior information and does not affect the shape of the posterior distribution; Step 2: Evaluate and verify the established hierarchical Bayesian IRT model to ensure that all parameters are suitable for the task and the model fits well; In step 2, based on the established hierarchical Bayesian IRT model, the relationship between the assessed test difficulty and the average test score is verified, and the average test score and the test difficulty parameter are obtained. The correlation coefficient between them is negative, and the reliability and validity verification results are valid, indicating that the harder the exam is, the harder it is for the students to get the scores they want. In terms of students’ knowledge level, it is assumed that the ranking of students’ knowledge level should be consistent with their ranking of total scores in the exam. The indicator for verifying the hypothesis is the consistency ratio of each exam, and the formula is as follows: ;in, is the consistency ratio of each test, ranging from [0,1]; , represents the student id; j represents the test id; is an indicator function, returning 1 or 0, Denotes a binary score Is it higher than ; The product of the two indicator functions represents the number of pairs of items for which all dichotomous scores and knowledge levels agree; It means to count the number of pairs of items with different dichotomous scores; The higher the value of , the more accurate the model evaluation is, based on which the model prediction result data is obtained; In the posterior prediction model test, in order to evaluate the fit of the model and avoid drawing wrong conclusions, the observation data With copy data The posterior predictive distribution is compared with It is expressed by the following formula: ;in, represents the model parameters; Given the observation data The posterior distribution of the model parameters; Indicates the model parameters Given a situation where you copy data The probability distribution of The integral is expressed over all model parameters Add up the above; Using the Markov Chain Monte Carlo MCMC algorithm, the posterior distribution of the parameters Extract samples from ; then generate replication data based on these samples ; Then, the posterior prediction model is used to test PPMC. The core of the posterior prediction model test PPMC is to obtain the difference value of the observed data. and the difference between the replicated data For comparison; at the same time, the posterior prediction p value is calculated, which indicates that given the observed data Under the condition of Greater than or equal to the observed data difference value Probability , the formula is as follows: ;in, Represents a given observation When copying data difference value Greater than or equal to the observed data difference value probability; As an indicator function, only the part that meets the conditions is considered. When , its value is 1, otherwise it is 0; is calculated by integrating all possible replication data Contribution Observe the data With copy data The posterior predictive distribution of is compared to estimate the model fit; Step 3: Based on the established hierarchical Bayesian IRT model, the calculation process of the hierarchical Bayesian IRT model is converted into a model representation in the Stan programming language, and the latent parameters are estimated through an advanced sampling algorithm.
2. The method for evaluating test attributes and student knowledge levels according to claim 1, characterized in that: In step 3, the calculation process of the established hierarchical Bayesian IRT model is converted into a model representation in the Stan programming language, and then the HMC algorithm is used in the Stan model to automatically sample and estimate the latent parameter distribution, where: First, the calculation process of the hierarchical Bayesian IRT model is set, including the prior distribution and the calculation method of the answer score estimated by the parameters; the calculation process is run through the Stan model; finally, the Hamiltonian Monte Carlo HMC algorithm is used to automatically sample and estimate the latent parameter distribution; Among them, when running the Stan model, the observed data and Stan code are used to estimate the parameters of the hierarchical Bayesian IRT model. The specific process is as follows: 1) Define data: input observation data into the Stan model; 2) Compile model: Stan model compiles the corresponding code and generates the corresponding sampler; 3) Sampling: Use the Hamiltonian Monte Carlo HMC algorithm to automatically sample from the posterior distribution of the parameters and estimate the latent parameters of the hierarchical Bayesian IRT model; Through the above operations, the posterior distribution of the parameter θ representing the knowledge level is obtained, and automatic sampling from the posterior distribution is achieved to estimate the distribution of the hidden parameters.
