English listening and reading ability grade evaluation system based on cognitive diagnosis technology
Through the English listening and reading ability level assessment system based on cognitive diagnosis technology, the problem that the existing system cannot fully evaluate the effectiveness of English listening and reading ability and feedback report is solved, and accurate assessment of English listening and reading ability and personalized learning support are achieved. , improves teaching quality and learning effectiveness.
Patent Information
- Application Number
- CN202410967170.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-15
AI Technical Summary
The existing English proficiency assessment system lacks a large-scale cognitive diagnostic testing system for Chinese English learners, and cannot fully and accurately reflect the candidates' English listening and reading ability. The effectiveness of feedback reports is limited, making it difficult to provide personalized learning guidance.
A English listening and reading ability level assessment system based on cognitive diagnosis technology is designed, including question bank construction, pre-packing and testing, paper grouping and formal testing, candidate ability evaluation and diagnostic feedback reporting modules, adopting advanced saturation type Cognitive diagnostic model and improved Angoff method are combined with question group factors to build a complete English listening and reading ability level assessment system.
It realizes accurate and comprehensive assessment of English listening and reading ability, provides personalized learning support, enhances the interpretability and transparency of digital scores, helps teachers and students better understand the concepts and essence of language ability, and promotes teaching and learning strategies. Accurate customization.
Smart Images

Figure CN118968830A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of English assessment devices, and in particular to an English listening and reading ability level assessment system based on cognitive diagnosis technology. Background Art
[0002] The development of an English proficiency level assessment system that meets the needs of Chinese English learners and users has become a new demand of the times. How to accurately determine students' English proficiency levels in language tests, whether students have or have achieved the specific ability performance characteristics corresponding to each level, and in-depth analysis and clear pointing out the English micro-skills that students need to improve have become the key to the development of English proficiency level assessment systems.
[0003] Despite the growing demand for diagnostic assessment in the education sector, there are many problems in the actual application of diagnostic assessment. First, there is a lack of large-scale cognitive diagnostic test systems for Chinese English learners on the market. On the one hand, most of the current diagnostic-related applications rely mainly on the renovation method, that is, extracting diagnostic information from non-diagnostic tests. Although this method has its convenience, it also has limitations. For example, it cannot solve the problem of inconsistent measurement dimensions between diagnostic technology (model) and non-diagnostic tests. The core of diagnostic technology lies in the multidimensionality of measurement. It divides language ability into multiple micro-skills (i.e., cognitive attributes) to reveal students' learning status and needs at the micro-skill level. However, non-diagnostic tests are often constructed based on unidimensional item response theory (IRT). In this process, in order to maximize the reliability of the test, test developers tend to exclude items that may introduce multidimensionality. This practice is obviously contrary to the premise of multidimensional measurement required by diagnostic technology, resulting in excessive correlation between attributes, which in turn affects the accuracy of diagnostic information. On the other hand, although diagnostic test systems already exist, their number is extremely small, and these diagnostic tests are limited to the assessment of a single cognitive attribute in terms of question design, and fail to cover questions that measure multiple attributes and allow these attributes to interact. This is contrary to the reality that candidates use multiple skills to answer questions in actual exam scenarios. Therefore, these test systems cannot fully and accurately reflect the actual answering process of candidates, resulting in one-sided diagnostic results. Therefore, it is necessary to develop diagnostic tests that truly meet the requirements of multidimensional measurement of diagnostic technology.
[0004] From the perspective of model selection, past diagnostic tests often only consider attributes as discrete variables when building models, and lack in-depth consideration of the continuous changes in students' abilities or knowledge status during the actual learning process. At the same time, past diagnostic tests widely use question sets (i.e., a group of questions based on the same input material) to improve test efficiency, but the introduction of question sets is contrary to the local independence assumption in cognitive diagnostic models. If the question set effect is ignored, it will not only lead to deviations in the estimation of item parameters, but also reduce the accuracy of the target attributes, thereby affecting the accurate assessment of the examinee's cognitive structure and ability. Therefore, in the process of model selection and construction, it is necessary to fully consider the continuity of students' abilities and the impact of the question set effect.
[0005] Finally, from the perspective of diagnostic process and feedback, the effectiveness of common diagnostic feedback reports is relatively limited. In general, the preparation of score reports can be divided into two categories. One is based on scale anchoring and standard setting, which presents the assessment results through continuous scores or grades based on unidimensional scales; the other decomposes the construct into multidimensional micro-skills, focusing on providing qualitative and personalized feedback, and presenting the ability of the test-taker in the form of attribute mastery. Although the former can connect language tests to external language proficiency frameworks (such as the Common European Framework of Reference for Languages, CEFR) and give meaning to numerical scores in large-scale language assessments, the information provided is relatively general and it is difficult to provide specific guidance for learners' subsequent learning. Although the latter is a commonly used feedback reporting method in diagnostic assessments, its fine-grained characteristics sometimes make it difficult to formulate effective remedial plans and it is difficult to directly help teachers and students improve teaching and learning. Therefore, it is particularly important to find an effective feedback reporting method that takes into account both macro-level ability judgments and micro-granular cognitive attributes. Summary of the invention
[0006] In view of the above problems, the present invention provides an English listening and reading ability level assessment system based on cognitive diagnosis technology. From the question setting stage to the feedback report, the present invention constructs a complete English listening and reading ability level assessment system, so as to accurately, comprehensively and deeply assess the learners' English listening and reading abilities, and provide personalized learning support for them.
[0007] The specific technical solutions are as follows:
[0008] An English listening and reading ability level assessment system based on cognitive diagnosis technology, including:
[0009] The question bank construction module is used to construct a number of questions for each cognitive attribute of the two language sub-abilities of listening and reading, and obtain a question bank for the two parts of listening and reading; each question in the question bank includes time length, topic parameters, and cognitive attributes;
[0010] The pre-test and test module is used to pre-test all questions in the question bank and test them in batches. During the test, the two-parameter logistic model of the item response theory is used to calibrate all questions in different test papers and the true score of the item response theory, and to eliminate questions with too high or too low difficulty values; and each question is tested for item function difference, and questions with item function difference tests are eliminated to optimize the question bank;
[0011] The test paper composition and formal test administration module is used to compose the test paper according to the optimized question bank, build the Q matrix according to the test paper composition results, conduct the formal test and collect test data;
[0012] The candidate ability assessment module uses a high-order saturated cognitive diagnostic model to analyze test data, calculate the candidate's cognitive attribute mastery mode, the probability of the candidate in each cognitive attribute mastery mode, and the candidate's overall ability; the high-order saturated cognitive diagnostic model needs to determine whether it needs to be supplemented with a question group factor to control the potential question group effect based on the fitting index value between the high-order saturated cognitive diagnostic model and the test data;
[0013] The diagnostic feedback report module generates a diagnostic feedback report based on the candidate's ability assessment results. The content of the diagnostic feedback report includes the candidate's English ability level, overall ability, cognitive attribute mastery mode, and probability under each cognitive attribute mastery mode.
[0014] Furthermore, the cognitive attributes of the language sub-ability of listening include understanding details, understanding the main idea, and making inferences; the cognitive attributes of the language sub-ability of reading include understanding vocabulary and syntactic structure, understanding details, understanding the main idea, making inferences, understanding contextual relationships, and article structure and organization.
[0015] Furthermore, the test paper pre-composition and test module includes:
[0016] The pre-test unit, which comprehensively considers the content to be examined, cognitive attributes, and the duration of the entire test, determines the questions contained in each test paper based on the anchor test design, and forms several test papers, all of which should contain all the questions in the question bank; the anchor test design mentioned above means that there are common questions in all test papers;
[0017] The test unit is used to test all pre-group test papers in batches. During the test, the two-parameter logistic model of the item response theory is first used to calibrate all the questions in different test papers. During the calibration process, the Stocking-Lord method of the item response theory is used to make the values equivalent, so that the test scores of different test papers and different time periods are comparable, and the difficulty parameters of all the questions in the question bank are obtained; then the item response theory likelihood ratio test method is used to detect the consistency and inconsistency of the questions. The questions in the question bank are deleted according to the difficulty parameters and the test results of the item function difference, and the questions with too high or too low difficulty values or with item function differences are eliminated.
[0018] Furthermore, the test paper composition and formal test administration module includes:
[0019] The test-setting unit sets the test paper based on the optimized question bank, taking into account the duration of the entire test, the difficulty of the questions, and the diversity of topics corresponding to the questions. After the test paper is set, a Q matrix is constructed based on the correspondence between the questions and cognitive attributes included in each sub-ability.
[0020] The formal testing unit is used to test the test papers generated by the test paper assembling unit and collect test data.
[0021] Furthermore, the high-order saturated cognitive diagnosis model includes two levels of diagnosis:
[0022] First-level diagnosis: Taking the score of each question in the test paper and the Q matrix describing the correspondence between the question and the cognitive attribute as input, the cognitive attribute mastery mode of the examinee and the proportion of examinees classified into each cognitive attribute mastery mode are obtained, and the probability of the examinee being in each cognitive attribute mastery mode is calculated based on the proportion;
[0023] Second-level diagnosis: The joint distribution of cognitive attribute mastery patterns is obtained based on the probability of the examinee in each cognitive attribute mastery pattern, the overall ability is calculated, and the fitting value of the model and the question score is output, using two fitting indicators.
[0024] Furthermore, in the first level diagnosis, the relationship between the examinee's cognitive attribute mastery pattern and the examinee's answer to the question is as follows:
[0025]
[0026] Among them, i represents the examinee, j represents the question, and α i represents the cognitive attribute mastery pattern of examinee i, α i ={α ik}, k = 1, ..., K, K represents the number of cognitive attributes, α ik represents the mastery of cognitive attribute k by examinee i, α ik =1 represents mastery, αik =0 means not mastered; X ij =1 represents that examinee i correctly answers question j, P represents the probability, P(X ij =1|α i ) represents the mastery mode α with cognitive attribute i The probability that candidate i can correctly answer question j; q j is a vector containing all Q matrix entries related to question j; j Represents the weight vector of question j, the upper corner table T represents the transposition, h(α i ,q j ) represents α i and q j A set of linear combinations of j,0 It represents the logarithmic probability of correctly answering question j when the examinee does not master all cognitive attributes.
[0027] Furthermore, in the second-level diagnosis, the relationship between cognitive attributes and examinee ability is described by a two-parameter logistic model, which is expressed as follows:
[0028]
[0029] Among them, θ represents the overall ability of the examinee, P(α i |θ) represents the examinees with overall ability θ being classified into cognitive attribute mastery mode α i The probability, P(α ik |θ) represents the probability that a candidate with overall ability θ masters cognitive attribute k, λ 0k and λ 1k They are the intercept parameter and slope parameter in the high-order saturated cognitive diagnosis model, respectively.
[0030] Furthermore, the question group factor is introduced by adopting a random effect model; the absolute fit index and the relative fit index are used as evaluation criteria to calculate the fit index between the high-order saturated cognitive diagnostic model and the test data before and after the introduction of the question group factor, and whether the question group factor needs to be introduced is determined according to the fit index;
[0031] In the first level diagnosis of the high-order saturated cognitive diagnosis model after the introduction of the question set factor, the relationship between the examinee's cognitive attribute mastery pattern and the examinee's answer to the question is as follows:
[0032]
[0033] Where t represents a question group, which refers to a set of questions with common stimuli, ξ it is the question group factor corresponding to candidate i. The number of question group factors depends on the actual number of question groups in the test paper. Each question group factor is independent of each other and obeys the standard normal distribution; βjt is the regression coefficient of question j in question group t. If question j does not belong to any question group, β jt ξ it =0.
[0034] Furthermore, the recommended critical scores for English proficiency levels are determined using the modified Angoff method. First, the minimum ability for each level is determined, and the correct answer rate for each question by candidates with the minimum ability is estimated based on the difficulty of the test questions. The correct answer rate is selected from a value among "0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100", and the average of the correct answer rates is the recommended critical score.
[0035] Furthermore, the content of the diagnostic feedback report also includes level descriptors for English proficiency levels, qualitative descriptors for cognitive attribute mastery patterns, and learning suggestions.
[0036] The beneficial effects of the present invention are:
[0037] The present invention relies on the "Scale" and "Guidelines" to develop an English proficiency level assessment system based on cognitive diagnosis technology. At the same time, combined with the standard method of cognitive diagnosis assessment and examination docking, a personalized feedback report that takes into account both macro-level ability level judgment and micro-granular cognitive attribute diagnosis is developed, starting from the proposition link to the score report, forming a complete English cognitive diagnosis level assessment system. On the one hand, the English proficiency level examination system based on cognitive diagnosis technology provides data support for personalized teaching and learning. While promoting the precise customization of teaching and learning strategies, it effectively assists teachers in optimizing teaching methods and improving teaching quality, conforming to the trend of the test field from the traditional "selection / screening-oriented test" to a more "learning-oriented assessment". On the other hand, the personalized feedback report system that combines cognitive diagnosis with standard setting not only enhances the interpretability and transparency of digital scores, but also provides the possibility of realizing personalized qualitative feedback, so that schools, teachers, and students can further understand the concept and essence of language ability while more comprehensively and accurately grasping the status of students' English proficiency.
[0038] In addition, the English proficiency level assessment system based on cognitive diagnosis technology involved in the present invention also has broad market value and application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 This is a schematic diagram of the cognitive diagnosis model for high-level question sets;
[0040] Figure 2 It is a specific flow chart of the modified Angoff method. DETAILED DESCRIPTION
[0041] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless otherwise specified, all technical and scientific terms used in this application have the same meanings as those commonly understood by those skilled in the art to which this application belongs.
[0042] An English listening and reading ability level assessment system based on cognitive diagnostic technology can be widely used in scenarios such as educational institution training, employer recruitment and selection, and personal self-study, providing customized English ability assessment services for different users and promoting the prosperity of the online education market. With the continuous development of technologies such as artificial intelligence and big data, more intelligent and personalized learning tools and products can be developed in combination with this assessment system to meet the diverse learning and assessment needs of English learners.
[0043] The system mainly includes question bank construction module, pre-paper compilation and trial test module, paper compilation and formal testing module, candidate ability assessment module, and diagnostic feedback report module. The functions of each module are introduced below.
[0044] (1) A question bank construction module is used to construct a number of questions for each cognitive attribute of the two language sub-skills of listening and reading, thereby obtaining a question bank for the two parts of listening and reading; each question in the question bank includes duration, topic parameters, and cognitive attributes.
[0045] In a specific implementation of the present invention, the content of the question bank construction module includes: (1-1) Cognitive attribute definition: dividing the English proficiency level assessment into two language sub-abilities, listening and reading, and determining the cognitive attributes that each language sub-ability may contain.
[0046] Cognitive attributes refer to the knowledge, abilities, strategies, etc. used by the subjects when completing tasks or answering questions correctly. The present invention adopts the following steps when defining cognitive attributes: combining the contents of the Scale and the Guide, reviewing the descriptions of listening and reading sub-abilities in previous literature, and respectively extracting the most representative cognitive attributes of listening and reading abilities that the learning groups corresponding to Levels 1 to 9 of the Scale (Levels 1 and 2 roughly correspond to primary school level, Level 3 corresponds to junior high school, Level 4 corresponds to high school, Levels 5 and 6 correspond to university, Level 7 corresponds to English majors, and Levels 8 and 9 correspond to high-end foreign language talents) need to master.
[0047] Among them, listening includes three cognitive attributes, namely understanding details, understanding the main idea, and making inferences.
[0048] Understanding details means that the candidates can understand the important information and details of the text, such as the time, place, location and character relationships of the events, the characteristics, quantity, nature of the events, etc.
[0049] Understanding the main idea means that the examinee can understand the general meaning of the text and summarize the main content and central idea of the text.
[0050] Making inferences means that the examinee can infer the speaker's motives, purposes, reasons and other communicative functions and implicit meanings through context or background knowledge; and can infer the speaker's attitudes and opinions.
[0051] Reading includes five cognitive attributes, namely understanding vocabulary and syntactic structure, understanding details, understanding the main idea, making inferences, understanding contextual relationships, and article structure and organization.
[0052] Understanding vocabulary and syntactic structure means that the candidates can understand the meaning of words and use vocabulary and grammar knowledge to understand sentence structure and the meaning it conveys.
[0053] Understanding details means that the candidate can locate the information mentioned in the question and scan the full text to find relevant details.
[0054] Understanding the main idea means that the candidates can integrate the information in the text and summarize the main content, core ideas and central idea of the paragraph or chapter.
[0055] Making inferences means that the candidate can guess the meaning of a word or phrase based on the context; and can make inferences and extrapolate based on factual information.
[0056] Understanding the context, article structure and organization means that the candidates can understand the logical relationship between the context and grasp the organizational structure of the article.
[0057] (1-2) Question bank construction: Based on the cognitive attributes defined in the listening and reading parts, the inductive method in diagnostic technology is used to develop diagnostic questions to measure each attribute and construct the question bank for the listening and reading parts.
[0058] First, it is necessary to develop a diagnostic question bank based on the already defined cognitive attributes. In the question setting process, a team of experts or other existing technologies can be used to design questions for a single cognitive attribute to achieve a more accurate measurement of each cognitive attribute; at the same time, it is necessary to fully consider the inherent correlation between different cognitive attributes and develop questions that integrate multiple cognitive attributes accordingly, with each question including its corresponding duration, topic parameters, cognitive attributes, etc.
[0059] (2) The pre-testing and test module is used to pre-test all questions in the question bank and test them in batches. During the test, the two-parameter logistic model of the item response theory is used to calibrate all questions in different test papers and equate the true scores of the item response theory, eliminating questions with too high or too low difficulty values; and each question is tested for item function differences, and questions that have item function difference tests are eliminated to optimize the question bank.
[0060] In a specific implementation of the present invention, the pre-test paper and test module is composed of a pre-test paper unit and a test unit, wherein:
[0061] The pre-test unit comprehensively examines the content, cognitive attributes, and duration of the entire test. The questions included in each test paper are determined based on the anchor test design to form several test papers. All test papers should include all the questions in the question bank; the anchor test design refers to the existence of common questions in all test papers.
[0062] Specifically, after the question bank is developed, it is necessary to comprehensively consider the project content, the cognitive attributes of the project test, and the duration of the entire test to determine the approximate questions included in each test paper, in preparation for subsequent batch tests. In the pre-test paper compilation process, several test papers need to be formed, and all test papers should include all the questions in the question bank; the test duration should be considered when compiling the papers. In addition, in order to ensure the comparability of scores between different test papers and between exams in different periods, it is necessary to compile the papers based on the anchor test design in the pre-test paper compilation stage. Specifically, in the anchor test design, each sample candidate only takes one test paper, but all test papers contain two parts, namely, each test paper has its own, independent questions, and all test papers have common questions. These common questions are also called "anchor questions." It is on the basis of anchor questions that equivalence is achieved.
[0063] The test unit is used to test all pre-grouped test papers in batches. During the test, the two-parameter logistic model of the item response theory is first used to calibrate all the questions in different test papers. During the calibration process, the Stocking-Lord method of the item response theory is used to make the equivalent value to ensure that the scores between different test papers and between exams in different periods are comparable, and finally the difficulty parameters of all the questions in the question bank are obtained. Subsequently, the item response theory likelihood ratio test method is used to detect the functional differences of the consistent and inconsistent items of the questions. According to the test results of the question parameters and the functional differences of the items, the questions in the question bank are deleted, and the questions with too high or too low difficulty values or with differences in item functions are eliminated.
[0064] Specifically, all questions in the question bank are tested in batches using big data. During the test, the two-parameter logistic model of the item response theory is used to calibrate all questions in different test papers and equalize the true scores of the item response theory, eliminating questions with too high or too low difficulty values while ensuring that the scores of candidates in different test papers are comparable. In addition, it is necessary to conduct a project function difference test on each question to prevent questions with project function difference tests from entering the question bank, so as to ensure the fairness of the questions in the question bank.
[0065] When all questions are tested in batches, in this embodiment, the total number of subjects selected for the test is more than 5,500, all of whom are selected from the target group at each level of the test to ensure that the language ability of the subjects is equivalent to that of the target candidate group.
[0066] (3) A test paper generation and formal test administration module, which is used to generate test papers based on the optimized question bank, construct a Q matrix based on the test paper generation results, and formally administer the test and collect test data. In a specific implementation of the present invention, the test paper generation and formal test administration module is composed of a pre-test paper generation unit and a test test unit, wherein:
[0067] The test-setting unit sets the test paper based on the optimized question bank, taking into account the duration of the entire test, the difficulty of the questions, and the diversity of topics corresponding to the questions. After the test paper is set, a Q matrix is constructed based on the correspondence between the questions contained in each sub-ability and the cognitive attributes.
[0068] After processing by the pre-paper assembly and test modules, the question bank is optimized to form multiple sets of English diagnostic papers with reasonable difficulty, including listening and reading parts, and construct a Q matrix based on the correspondence between questions and cognitive attributes.
[0069] In a specific implementation of the present invention, each set of tests includes two parts: listening and reading. When compiling the test papers, the duration of the entire test, the difficulty of the questions, and the diversity of topics corresponding to the questions are comprehensively considered, and topics familiar to learners should be covered, such as daily life, education, and other general topics. In this embodiment, the listening part contains 30 multiple-choice questions, questions 1-10 are based on 10 short conversations, questions 11-15 are based on a long conversation, and questions 16-20 are based on three monologues. There are 30 questions in the reading part, questions 1-10 are multiple-choice questions, based on two chapters, and each chapter has 5 items. Questions 11-20 are library-style cloze questions, with a total of 10 blanks, and learners are required to choose appropriate words from 15 words to fill in the blanks. Questions 21-25 are matching questions, requiring candidates to select the correct answer from 7 options and fill in 5 blanks, of which 2 options are interference items.
[0070] After the test paper is compiled, a Q matrix will be constructed based on the correspondence between the questions and cognitive attributes contained in each sub-ability. The Q matrix is an important element in cognitive diagnostic assessment and specifies the relationship between questions and cognitive attributes. Table 1 gives an example of a Q matrix, in which the cognitive attributes contained in each sub-ability horizontally, such as reading ability, can be subdivided into information extraction, reasoning, and generalization. The vertical question number column corresponds to each question. If the question involves one or several attributes, then the corresponding attribute is marked as "1", and if it is not involved, it is marked as "0".
[0071] Table 1. Q matrix example
[0072]
[0073] The formal testing unit is used to test the test papers generated by the test paper assembling unit and collect test data.
[0074] In a specific implementation of the present invention, the formal diagnostic test is connected to the intelligent system as a user interface, realizing the full electronicization of the test process. Candidates can easily complete all test contents with the help of a computer, thus getting rid of the constraints of time and place in traditional tests, greatly improving the flexibility and convenience of the test. The system randomly selects a set of test papers for testing. The formal test is to diagnose the learner's English proficiency. The formal diagnostic test will be conducted online throughout the process. The test consists of two parts, including listening and reading, and the test time is about 70 minutes.
[0075] (4) A candidate ability assessment module, which uses a high-order saturated cognitive diagnostic model to analyze test data, calculate the candidate's cognitive attribute mastery pattern, the probability of the candidate being in each cognitive attribute mastery pattern, and the candidate's overall ability; the high-order saturated cognitive diagnostic model needs to determine whether it needs to be supplemented by a question set factor to control the potential question set effect based on the fitting index value between the high-order saturated cognitive diagnostic model and the test data.
[0076] The present invention adopts a high-order saturated cognitive diagnosis model to perform diagnostic data analysis based on the characteristics of cognitive attributes and the relationship between attributes, question type characteristics, number of items and number of subjects, and uses question group factors in the model to control potential question group effects and ensure the accuracy of skill diagnosis.
[0077] In a specific implementation of the present invention, the content of the candidate ability assessment module includes: (5-1) Model selection: Different from the models used in most cognitive diagnosis applications in the past, the present invention uses a high-order LCDM model and introduces a question set factor to effectively deal with the potential question set effect of the test. The LCDM model is a saturated model. Compared with the commonly used simplified models such as the DINA model that only includes interaction effects and the C-RUM model that only includes main effects, the LCDM model covers all possible main effects and interaction effects between latent variables. This saturated characteristic matches the complex cognitive process of language use. Second language learners may use a single cognitive attribute or a combination of multiple cognitive attributes when completing language tasks.
[0078] Compared with the LCDM model, the high-order LCDM model adds a high-order ability factor (i.e., the overall ability of the examinee). The high-order LCDM model is a two-level model. The first level is a conventional cognitive diagnosis model (i.e., LCDM model). At this level, the model assumes that cognitive attributes are discrete variables. The second level is an item response theory model (commonly used single-parameter or two-parameter logistic model). At this level, the model assumes that this newly added high-order ability factor is a continuous variable. The high-order LCDM model describes the macro-level and micro-level cognitive attribute mastery status by determining the joint distribution of high-order ability factors and low-order cognitive attribute factors. Therefore, compared with the traditional cognitive diagnosis model, it has the dual functions of extracting attribute diagnosis information and estimating the overall ability of examinees.
[0079] Introducing the complete analysis process of the high-order LCDM model:
[0080] Level 1: Input the score of each question and the Q matrix that describes the correspondence between cognitive attributes and questions, and obtain the attribute mastery pattern of each candidate (the combination of the candidate's mastery of all cognitive attributes. For example, if there are three cognitive attributes in total, the attribute mastery pattern of candidate A is 101, which means that candidate A has mastered the first and third attributes, but not the second attribute) and the proportion of candidates who are divided into each attribute mastery pattern. Based on this proportion, the probability of each cognitive attribute being mastered by the candidate as a whole can also be calculated. The relationship between the candidate's attribute mastery pattern and the candidate's answer to the question can be written as:
[0081]
[0082] Where i represents the examinee, j represents the question, and α i is a vector that contains the mastery of all attributes of candidate i, which can also be called the attribute mastery mode of candidate i. i ={α ik}, k = 1, ..., K (where K is the number of cognitive attributes and k is a specific cognitive attribute); P represents probability, P(X ij =1|α i ) represents the probability that examinee i with a certain attribute mastery pattern can correctly answer question j; q j is a vector containing all the Q matrix entries related to question j; j Represents a (2 K -1)×1 weight vector; h(α i ,q j ) is a (2 K -1)×1 vector, representing α i and q j A set of linear combinations ofj,0 is the intercept parameter, which represents the log odds of correctly answering question j when the examinee does not master all cognitive attributes.
[0083] This formula is usually transformed logically so that the probability P ranges from 0 to 1. If a test paper is assumed to test three cognitive attributes, the formula after logical transformation can be written as:
[0084]
[0085] The converted formula contains three parameters in total, including the intercept parameter λ j,0 , the main effect λ of attribute k j,1,(k) , and all possible interactions between attributes, for example, the interaction effect λ between attribute k and attribute k' j,2,(k,k’) α ik represents the correspondence between candidate i and attribute k, that is, whether candidate i has mastered attribute k; q jk Indicates the correspondence between question j and attribute k, that is, whether question j tests attribute k.
[0086] Level 2: The joint distribution of attribute mastery modes is obtained based on the probability of the examinee in each attribute mastery mode, the overall ability is calculated, and the fitting value of the model and the question score is output, using two fitting indicators.
[0087] In the second level of the model, if the two-parameter logistic model is used as an example to describe the relationship between low-level cognitive attributes and high-level examinee abilities, the probability model can be written as:
[0088]
[0089] Among them, α i is a vector that contains the mastery of all attributes of candidate i, which can also be called the attribute mastery mode of candidate i. i ={α ik}, k = 1, ..., K; K is the number of cognitive attributes; θ is the overall ability of the examinee; λ 0k and λ 1k are the intercept parameter and slope parameter in the high-order LCDM model; P(α i |θ) indicates that the examinee with ability value θ is classified into attribute mastery mode α i probability.
[0090] In addition, previous cognitive diagnosis applications have ignored the question group effect caused by question group design. The present invention introduces a question group factor in the high-order model to control the estimation bias caused by the question group effect. A question group refers to a collection of questions with common stimuli. For example, several questions under the same reading comprehension belong to a question group. Question group design can greatly improve test efficiency, so it is extremely common in language tests (especially listening and reading tests). However, when there are question groups in the test, since the questions embedded in the question group share the same question group stimulus, there is a certain dependence between the responses of the question group items, which violates a core assumption of the cognitive diagnosis model-the local independence assumption. At this time, if the conventional cognitive diagnosis model is used directly to fit the test data, it may affect the estimation of item parameters and the diagnosis of cognitive attribute status.
[0091] The high-order model of the question set used in the present invention belongs to the random effect model (the question set factors are regarded as random variables). The model assumes that the attribute mastery pattern and the random question set effects are independent, and all question set effects are also assumed to be independent and identically distributed, that is, the probability distribution of each question set factor is the same, and these random variables are independent of each other. In addition, all question set factors follow the standard normal distribution.
[0092] The formula of the high-order model of the question set after logical transformation is as follows:
[0093]
[0094] The difference between this formula and the non-question group high-order model is the last added question group term, β jt ξ it , where ξ it is the question group factor corresponding to candidate i, and t represents the question group. The number of specific question group factors depends on the actual number of question groups in the test paper. Each question group factor is independent of each other and obeys the same standard normal distribution, that is, ξ it ~N(0,1).
[0095] β jt is the regression coefficient of question j in question group t. If question j does not belong to any question group, then this term β jt ξ it Then it is 0.
[0096] Figure 1 Taking a test paper containing two groups of 8 questions as an example, the structure of the cognitive diagnosis model of the high-level question group is specifically explained through graphical display. 1 represents the high-level ability factor, A1-A3 represent three discrete cognitive attributes (i.e., the attribute states have only two types: mastered / not mastered), ITM1-ITM8 represent eight secondary scoring (0-1 scoring) questions, ξ 1 and 2Represents two question group factors, each factor corresponds to a question group, that is, questions 1 to 4 are question group 1, and questions 5 to 8 are question group 2.
[0097] (5-2) Model fitting: After the cognitive diagnostic model is determined, it is fitted to the data. Given that the existence of a set of questions in the test does not necessarily lead to a set of questions effect, the system will first fit both the high-order LCDM model and the high-order set of questions LCDM model to the data. Subsequently, the system will evaluate the fit of the two models to the data and the impact of the set of questions effect on parameter estimation, and select the optimal model based on the fit and effect size for subsequent in-depth cognitive diagnostic analysis.
[0098] Fit indices can be divided into two categories, including absolute fit indices and relative fit indices. Commonly used absolute fit indices are RMSEA and SRMSR. If the values of these two indicators are less than 0.05, it means that the model fits the data. Commonly used relative fit indices are -2log-likelihood (-2LL), Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC). The lower the values of these indicators, the better the fit between the model and the data.
[0099] When comparing, the absolute fit index was judged, and when they were all less than 0.05, the relative fit index was judged.
[0100] (5) A diagnostic feedback report module, which generates a diagnostic feedback report based on the candidate's ability assessment results. The content of the diagnostic feedback report includes the candidate's English proficiency level, overall ability, cognitive attribute mastery mode, and probability under each cognitive attribute mastery mode.
[0101] The present invention combines standard setting and cognitive diagnostic assessment methods to provide examinees with a classification of level based on standard setting and a classification of micro-skill mastery patterns based on cognitive diagnostic assessment.
[0102] In a specific implementation of the present invention, the diagnostic feedback reporting module combines the standard setting and the cognitive diagnostic assessment method to generate a personalized diagnostic feedback report, which includes three stages:
[0103] (6-1) Standard setting. The content of the generated report: the English proficiency level of the examinee. Standard setting is the core link in connecting the test with external language standards. It mainly refers to the process of grading the test based on external language standards and setting the critical score between each two adjacent levels.
[0104] The process of standard setting includes: (1) selecting the standard setting method and completing the preparatory work; (2) describing different performance levels; (3) conducting a round of evaluation; (4) providing feedback on relevant information or providing auxiliary information; (5) conducting a second round of evaluation; and (6) calculating and recommending critical scores.
[0105] In the present invention, the standard setting method applied to the listening and reading parts is the modified Angoff method, which requires the definition of "just qualified candidates" for each level, and estimates their correct answer rate for each question in combination with the difficulty value of the question (the correct answer rate can only be selected from 11 numbers "0, 10, 20, ... 100"), and finally the average value of the correct answer probability of the "just qualified candidates" of this level is the critical score. The specific process of this method is as follows Figure 2 , which includes one round of judgement contact and two rounds of formal judgement. After the first round of formal judgement, the overall mean, frequency, range of judgement results for each question and all question difficulty value information of the first round of judgement are added, and then the second round of judgement is carried out. The results of the second round of judgement will be used to estimate the critical score of each level. Compared with the traditional Angoff method, the modified Angoff method adds question difficulty information, which can reduce cognitive pressure and improve the consistency of judgment.
[0106] (6-2) Cognitive attribute diagnosis. The report generated includes: the total ability score of the examinee, the examinee's cognitive attribute mastery pattern and the probability of cognitive attribute mastery. This content is obtained through cognitive diagnostic analysis using the selected model. The analysis includes: the probability of cognitive attribute mastery at the level of the total examinee group, the total ability score of the examinee at the individual level, the examinee's attribute mastery pattern and attribute mastery probability, and the classification accuracy at the attribute level, attribute mastery pattern level and test level.
[0107] (6-3) Generate a descriptive statement based on the level of (6-1) and the attribute diagnosis results of (6-2). The contents of the generated report include: level descriptive statement, qualitative descriptive statement of cognitive attributes, and learning suggestions.
[0108] The generation of level descriptors requires comparing the candidate's total score with the critical scores of each level. After the level is determined, a preset level descriptor is given, for example: "Your score in this listening test is 20 points, which is approximately level 5 on the scale."
[0109] The qualitative description of cognitive attributes includes the probability of mastering each cognitive attribute, the mastery mode, and related qualitative descriptions of each candidate. For example, "Your mastery of micro-skills in this listening test is as follows: the probability of mastering details is 0.97, the probability of mastering the main idea is 0.99, the probability of mastering inference is 0.99, and your attribute mastery mode is 111. You can understand oral expressions of general topics at a normal speed, obtain key points and details, clarify the logical relationships such as cause and effect, transitions, and progression, and understand the basic cultural connotations of the discourse. When you listen to and watch radio and television programs on general topics, you can understand the main content."
[0110] Learning suggestions need to be generated based on qualitative descriptions of cognitive attributes, for example: "Your next stage of learning goals are: 1) To be able to understand oral expressions with a large amount of information and related to one's professional field, such as lectures, reports, discussions, etc., summarize the main content, and grasp the speaker's way of organizing information, such as the overall framework, cohesion, etc. 2) To be able to understand workplace conversations at a normal speed, such as business negotiations, work exchanges, job interviews, etc., and understand the speaker's views and intentions."
[0111] Specifically, the description should be worded in a way that (1) is easy to interpret, (2) is motivating to students, and (3) provides useful information to students.
[0112] The final feedback report includes both quantitative information, such as the candidate's total test score and performance in each part (i.e. listening, reading), the corresponding scale level, cognitive skill mastery pattern, etc., and qualitative information, such as student English proficiency description, learning suggestions, etc.
[0113] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.
Claims
1. An English listening and reading ability level assessment system based on cognitive diagnosis technology, characterized in that: include: The question bank construction module is used to construct a number of questions for each cognitive attribute of the two language sub-abilities of listening and reading, and obtain a question bank for the two parts of listening and reading; each question in the question bank includes time length, topic parameters, and cognitive attributes; The pre-test and test module is used to pre-test all questions in the question bank and test them in batches. During the test, the two-parameter logistic model of the item response theory is used to calibrate all questions in different test papers and the true score of the item response theory, and to eliminate questions with too high or too low difficulty values; and each question is tested for item function difference, and questions with item function difference tests are eliminated to optimize the question bank; The test paper composition and formal test administration module is used to compose the test paper according to the optimized question bank, build the Q matrix according to the test paper composition results, conduct the formal test and collect test data; The candidate ability assessment module uses a high-order saturated cognitive diagnostic model to analyze test data, calculate the candidate's cognitive attribute mastery mode, the probability of the candidate in each cognitive attribute mastery mode, and the candidate's overall ability; the high-order saturated cognitive diagnostic model needs to determine whether it needs to be supplemented with a question group factor to control the potential question group effect based on the fitting index value between the high-order saturated cognitive diagnostic model and the test data; The diagnostic feedback report module generates a diagnostic feedback report based on the candidate's ability assessment results. The content of the diagnostic feedback report includes the candidate's English ability level, overall ability, cognitive attribute mastery mode, and probability under each cognitive attribute mastery mode.
2. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 1 is characterized in that: The cognitive attributes of the language sub-ability in listening include understanding details, understanding the main idea, and making inferences; the cognitive attributes of the language sub-ability in reading include understanding vocabulary and syntactic structure, understanding details, understanding the main idea, making inferences, understanding contextual relationships, and article structure and organization.
3. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 1 is characterized in that: The pre-test paper and test module includes: The pre-test unit, which comprehensively considers the content to be examined, cognitive attributes, and the duration of the entire test, determines the questions contained in each test paper based on the anchor test design, and forms several test papers, all of which should contain all the questions in the question bank; the anchor test design mentioned above means that there are common questions in all test papers; The test unit is used to test all pre-group test papers in batches. During the test, the two-parameter logistic model of the item response theory is first used to calibrate all the questions in different test papers. During the calibration process, the Stocking-Lord method of the item response theory is used to make the values equivalent, so that the test scores of different test papers and different time periods are comparable, and the difficulty parameters of all the questions in the question bank are obtained; then the item response theory likelihood ratio test method is used to detect the consistency and inconsistency of the questions. The questions in the question bank are deleted according to the difficulty parameters and the test results of the item function difference, and the questions with too high or too low difficulty values or with item function differences are eliminated.
4. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 3 is characterized in that: The test paper composition and formal test administration module includes: The test-setting unit sets the test paper based on the optimized question bank, taking into account the duration of the entire test, the difficulty of the questions, and the diversity of topics corresponding to the questions. After the test paper is set, a Q matrix is constructed based on the correspondence between the questions and cognitive attributes included in each sub-ability. The formal testing unit is used to test the test papers generated by the test paper assembling unit and collect test data.
5. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 1 is characterized in that: The high-order saturated cognitive diagnosis model includes two levels of diagnosis: First-level diagnosis: Taking the score of each question in the test paper and the Q matrix describing the correspondence between the question and the cognitive attribute as input, the cognitive attribute mastery mode of the examinee and the proportion of examinees classified into each cognitive attribute mastery mode are obtained, and the probability of the examinee being in each cognitive attribute mastery mode is calculated based on the proportion; Second-level diagnosis: The joint distribution of cognitive attribute mastery patterns is obtained based on the probability of the examinee in each cognitive attribute mastery pattern, the overall ability is calculated, and the fitting value of the model and the question score is output, using two fitting indicators.
6. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 5 is characterized in that: In the first level of diagnosis, the relationship between the examinee's cognitive attribute mastery pattern and the examinee's question answers is as follows: Among them, i represents the examinee, j represents the question, and α i represents the cognitive attribute mastery pattern of examinee i, α i ={α ik }, k = 1, ..., K, K represents the number of cognitive attributes, α ik represents the mastery of cognitive attribute k by examinee i, α ik =1 represents mastery, α ik =0 means not mastered; X ij =1 represents that examinee i correctly answers question j, P represents the probability, P(X ij =1|α i ) represents the cognitive attribute mastery mode α i The probability that candidate i can correctly answer question j; q j is a vector containing all the Q matrix entries related to question j; j Represents the weight vector of question j, the upper corner table T represents the transposition, h(α i ,q j ) represents α i and q j A set of linear combinations of j,0 It represents the logarithmic probability of correctly answering question j when the examinee does not master all cognitive attributes.
7. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 6 is characterized in that: In the second level of diagnosis, the relationship between cognitive attributes and examinee ability is described by a two-parameter logistic model, which is expressed as follows: Among them, θ represents the overall ability of the examinee, P(α i |θ) represents the examinees with overall ability θ being classified into cognitive attribute mastery mode α i The probability, P(α ik |θ) represents the probability that a candidate with overall ability θ masters cognitive attribute k, λ 0k and λ 1k They are the intercept parameter and slope parameter in the high-order saturated cognitive diagnosis model, respectively.
8. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 6 is characterized in that: The question group factor is introduced by adopting a random effect model; the absolute fit index and the relative fit index are used as evaluation criteria to calculate the fit index between the high-order saturated cognitive diagnostic model and the test data before and after the introduction of the question group factor, and whether the question group factor needs to be introduced is determined according to the fit index; In the first level diagnosis of the high-order saturated cognitive diagnosis model after the introduction of the question set factor, the relationship between the examinee's cognitive attribute mastery pattern and the examinee's answer to the question is as follows: Where t represents a question group, which refers to a set of questions with common stimuli, ξ it is the question group factor corresponding to candidate i. The number of question group factors depends on the actual number of question groups in the test paper. Each question group factor is independent of each other and obeys the standard normal distribution; β jt is the regression coefficient of question j in question group t. If question j does not belong to any question group, β jt ξ it =0.
9. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 1 is characterized in that: The recommended critical scores for English proficiency levels are determined using the modified Angoff method. First, the minimum ability for each level is determined, and the correct answer rate for each question by candidates with the minimum ability is estimated based on the difficulty of the test questions. The correct answer rate is selected from a value among "0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100", and the average of the correct answer rates is the recommended critical score.
10. The English listening and reading ability level assessment system based on cognitive diagnosis technology according to claim 1 is characterized in that: The content of the diagnostic feedback report also includes level descriptors for English proficiency levels, qualitative descriptors for cognitive attribute mastery patterns, and learning suggestions.
Citation Information
Patent Citations
Method for presenting cognitive diagnosis report of English reading of college students
CN110930033A
Personalized test paper composition method and system fusing cognitive characteristics and test question text information
CN112508334A
Cited By
Multi-group-oriented cognitive diagnosis deviation correction method with covariants
CN120495036A
