A method and system for evaluating the difficulty of test questions based on a large model

Abstract

The present invention discloses a method and system for assessing the difficulty of propositional test questions based on a large model, the method comprising the following steps: S1: obtaining propositional test questions and corresponding reference answers; S2: parsing the propositional test questions based on a pre-trained large language model and generating parsing results; S3: comparing the consistency between the parsing results and the reference answers to determine the difficulty level of the propositional test questions; the method and system for assessing the difficulty of propositional test questions based on a large model solves the problem in the prior art that the difficulty of test questions is affected by multiple factors, including but not limited to the complexity of knowledge points, the requirements for problem-solving skills, and the clarity of language expression, and these factors may be difficult to accurately capture and quantify in the existing large model, and therefore the large model cannot be used to accurately assess the difficulty of propositional test questions.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for assessing the difficulty of test questions based on a large model. Background Art

[0002] In talent testing and assessment systems, large-scale model-based methods for assessing test difficulty have garnered widespread attention. By applying advanced machine learning techniques, particularly large-scale pre-trained models like GPT, automated assessment of test difficulty is possible. The core of this approach is to train the model using a large amount of labeled test data, enabling it to identify and learn features related to test difficulty, thereby enabling predictions for new test questions.

[0003] However, this method faces the following key problems in practical applications: Multi-factor difficulty influence: The difficulty of test questions depends not only on the complexity of knowledge points, but also on many factors such as the requirements of problem-solving skills, the logical structure of test questions, and the clarity of language expression. These factors are often not fully quantified and expressed in the training data of large models, resulting in the model being unable to accurately capture these subtle differences when evaluating test questions with higher complexity; Consistency and representativeness of data annotation: Model training relies on a large amount of labeled test question data, and the quality and consistency of this data are crucial to the model's evaluation capabilities.

[0004] In reality, the process of labeling difficulty levels is often subjective and inconsistent. In particular, the definition of difficulty may vary across different cultural contexts, which can lead to instability and lack of representativeness in model evaluation results. Although large models perform well when handling common test question types, their evaluation capabilities may be limited when faced with test questions with innovative or unconventional structures. Whether the model can maintain consistent evaluation results across test questions in different fields or disciplines is a major challenge in current research. Large-scale pre-trained models typically require high computing resources for training and inference. This can lead to performance and cost bottlenecks for real-time difficulty assessment of large-scale test question banks. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for assessing the difficulty of test questions based on a large model, so as to solve the problem in the existing technology that the difficulty of test questions is affected by multiple factors, including but not limited to the complexity of knowledge points, the requirements of problem-solving skills, and the clarity of language expression, and these factors may be difficult to accurately capture and quantify in the existing large model, so the large model cannot be used to accurately assess the difficulty of test questions.

[0006] To achieve the above-mentioned object, the present invention provides the following technical solution: a method for assessing the difficulty of test questions based on a large model, the method comprising the following steps:

[0007] S1: Obtain the test questions and their corresponding reference answers;

[0008] S2: Parse the test questions based on the pre-trained large language model and generate parsing results;

[0009] S3: Compare the consistency between the analysis results and the reference answers to determine the difficulty level of the test questions;

[0010] Wherein, the step S3 includes:

[0011] Perform word segmentation on the parsing results and reference answers, and convert them into sequences respectively;

[0012] By comparing the elements in the two sequences, the distance matrix between the two sequences is calculated. The calculation formula of the distance matrix is:

[0013] ,

[0014] in, represents the cumulative minimum alignment distance from the i-th word in the parsing result sequence to the j-th word in the reference answer sequence, represents the distance between the i-th word in the parsing result sequence and the j-th word in the reference answer sequence, where i and j represent the indexes of the parsing result sequence and the reference answer sequence respectively;

[0015] Through the above calculations, we can obtain the minimum alignment path from the upper left corner of the matrix D(1,1) to the lower right corner D(m,n), where m and n represent the lengths of the parsing result sequence and the reference answer sequence, respectively.

[0016] The difficulty of the test questions is divided according to the size of the distance D(m,n).

[0017] Preferably, obtaining the test questions and their corresponding reference answers in step S1 specifically includes:

[0018] S11: Retrieve test questions from the database;

[0019] S12: extracting the corresponding reference answer based on the identifier of the test question;

[0020] S13: Format the test questions to ensure they meet the requirements of the large language model input;

[0021] S14: The formatted test questions and their corresponding reference answers are stored in a temporary buffer area.

[0022] Preferably, the step S12 of extracting the corresponding reference answer based on the identifier of the test question specifically includes:

[0023] Construct a mapping relationship table between the test question identifiers and the reference answers;

[0024] Query the mapping table based on the identifier of the test question to obtain the corresponding reference answer;

[0025] If the query result is empty, it is considered that the question has no corresponding reference answer;

[0026] If the query result is not empty, the reference answer found will be used as the object for subsequent processing;

[0027] The mapping relationship table between the test question identifier and the reference answer is constructed as follows:

[0028] Collect test questions and their corresponding reference answers, and assign a unique identifier to each test question;

[0029] Establish an association with the reference answer based on the unique identifier of the test question;

[0030] Determine whether the unique identifier of the test question already exists in the mapping relationship table. If not, add the mapping relationship;

[0031] If it exists, update the reference answer in the mapping table.

[0032] Preferably, the step of determining whether the unique identifier of the test question already exists in the mapping relationship table specifically includes:

[0033] Traverse all records in the mapping relationship table;

[0034] If a record with the same unique identifier as the current test question is found, it is considered that the identifier already exists;

[0035] If no identical record is found after the traversal, the identifier is considered to not exist;

[0036] If the identifier does not exist, perform the add operation; if the identifier exists, perform the update operation;

[0037] The performing of the adding operation specifically includes:

[0038] Create a new record containing the unique identifier of the test question and the corresponding reference answer;

[0039] Add the newly created record to the mapping relationship table;

[0040] Update the version number of the mapping table to reflect the latest changes;

[0041] Perform a consistency check to ensure that the add operation was successful.

[0042] Preferably, the specific steps of parsing the test questions based on the pre-trained large language model and generating parsing results in S2 include:

[0043] S21: Inputting the test question into the pre-trained large language model;

[0044] S22: extracting key information of the test questions based on the internal language structure analysis mechanism of the large language model;

[0045] S23: Calculate the relevance score of key information using a formulaic method, where the relevance score R is determined by the frequency F of the keyword in the test question and the position weight W, i.e., R = F × W;

[0046] S24: Generate preliminary analysis results based on the key information.

[0047] Preferably, the step of extracting key information of the test questions based on the internal language structure analysis mechanism of the large language model includes:

[0048] Perform word segmentation on the test questions to obtain the basic vocabulary units of the test questions;

[0049] Identify entities and concepts in test questions based on the semantic understanding capability of the large language model;

[0050] Use a formulaic method to judge the importance of an entity or concept, where the importance I is determined by the frequency F of the entity or concept in the domain knowledge base and the position P in the proposition test question, that is, I=F+P;

[0051] Screen out the key entities and concepts in the exam questions.

[0052] Preferably, the importance of an entity or concept is determined by a formulaic method, wherein the importance I is determined by the frequency F of the entity or concept in the domain knowledge base and the position P in the test question, i.e., I = F + P. Specifically, it includes:

[0053] Query the domain knowledge base to obtain the occurrence frequency F of entities or concepts;

[0054] According to the position of the entity or concept in the test question, different position weights P are assigned;

[0055] Calculate the importance I based on the occurrence frequency F and the position weight P;

[0056] If the importance I is greater than a predetermined threshold T, the entity or concept is considered to be key information.

[0057] Preferably, if the importance I is greater than a predetermined threshold T, then the entity or concept is considered to be key information specifically includes:

[0058] Comparing the calculated importance I with a predetermined threshold T;

[0059] If I>T, the entity or concept is marked as key information;

[0060] If I≤T, ignore the entity or concept;

[0061] Collect all entities or concepts marked as key information;

[0062] The collection of all entities or concepts marked as key information specifically includes:

[0063] Summarize all entities or concepts marked as key information;

[0064] Use a formulaic method to count the number of key information N;

[0065] If the number N exceeds the preset benchmark number B, the difficulty level of the test questions will be increased;

[0066] If the number N does not exceed the preset benchmark number B, the difficulty level of the test questions remains unchanged.

[0067] Preferably, the step S3 specifically includes:

[0068] For the obtained test questions and reference answers, we first use the pre-trained large language model to generate the semantic vector representation of each test question and reference answer. Let the test question be Q, the corresponding reference answer be A, and the parsing result be R. The generated semantic vectors are represented as vectors V Q 、V A and V R ;

[0069] Based on the generated semantic vector, the similarity between the parsing result R and the reference answer A is calculated using the cosine similarity formula. The formula for calculating cosine similarity is:

[0070] ,

[0071] in Represents vector V R and V A The dot product of and are the moduli of vectors VR and VA respectively;

[0072] Evaluate the consistency between the parsing results and the reference answers in multiple dimensions. Combine the above multiple dimensions and use the weighted average method or other comprehensive scoring methods to calculate the comprehensive consistency score. The calculation formula for the comprehensive consistency score St is:

[0073] St = Ws × Ss + Wg × Sg + Wl × Sl + Wc × Sc, where Ss is the semantic similarity score, Sg is the semantic consistency score, Sl is the logical consistency score, Sc is the content consistency score, Ws, Wg, Wl and Wc are the weight coefficients of the scores of each dimension, satisfying Ws + Wg + Wl + Wc = 1;

[0074] Finally, based on the comprehensive consistency score St, the difficulty level of the test questions is evaluated and the difficulty level is set as "easy", "medium" and "difficult", and the corresponding score range is set for each level.

[0075] If St ≥ 0.85, the assessment proposition test question is “easy”;

[0076] If 0.70≤St<0.85, the assessment is “moderate”;

[0077] If St<0.70, the assessment is “difficult”.

[0078] A system for assessing the difficulty of test questions based on a large model, using the method for assessing the difficulty of test questions based on a large model, comprising:

[0079] An input module, which is used to obtain test questions and their corresponding reference answers;

[0080] A large language model parsing module, which is used to parse the test questions based on the pre-trained large language model and generate parsing results;

[0081] The comparison module is used to compare the consistency between the analysis results and the reference answers and determine the difficulty level of the test questions.

[0082] It can be seen from the above technical solution that the present invention has the following beneficial effects:

[0083] This large-model-based test question difficulty assessment method and system obtains test questions and their corresponding reference answers, analyzes the test questions based on a pre-trained large language model and generates analysis results, compares the consistency of the analysis results with the reference answers, and determines the difficulty level of the test questions. It can more accurately capture the various difficulty influencing factors in the test questions, improve the accuracy and applicability of the assessment, and achieve a balance between automation, accuracy and practicality by optimizing the model's training data annotation, enhancing the ability to learn multi-dimensional difficulty features, and improving the model's generalization ability. It achieves efficient difficulty assessment of various types of test questions and solves the problem of existing technologies that the difficulty of test questions cannot be accurately assessed using large models because the difficulty of test questions is affected by multiple factors, including but not limited to the complexity of knowledge points, the requirements of problem-solving skills, and the clarity of language expression. These factors may be difficult to accurately capture and quantify in existing large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Flow chart of the method of the present invention;

[0085] Figure 2 A flowchart of the steps for obtaining test questions and their corresponding reference answers for the present invention;

[0086] Figure 3 The present invention is a flow chart of a method for parsing the test questions based on a pre-trained large language model and generating parsing results. DETAILED DESCRIPTION

[0087] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0088] like Figure 1 and Figure 3 As shown, a method for assessing the difficulty of test questions based on a large model includes the following steps:

[0089] S1: The entire evaluation process begins by obtaining the test questions and their corresponding reference answers. First, we need to collect a series of test questions and their standard answers or scoring standards. These test questions can be multiple-choice questions, fill-in-the-blank questions, short-answer questions, or essay questions from different subject areas. For example, in the field of mathematics, we can choose a typical algebraic equation solving question: "Solve the equation 2x+ 3=7." For this question, the reference answer is "x=2". After collecting these test questions and answers, we input them into the system as the basic data for evaluation. The goal of this stage is to ensure that we have a clear and clear evaluation benchmark, that is, the test questions and their correct answers.

[0090] S2: Parse the test questions based on the pre-trained large language model and generate parsing results. At this stage, we use the pre-trained large language model to analyze each test question. For example, for the math problem mentioned above, "Solve the equation 2x+3=7", the large model will try to understand the requirements of the question, recognize that this is a problem of solving a linear equation, and give the corresponding parsing process. This process may include identifying unknown numbers in the equation, performing necessary mathematical operations, etc. The large model will generate detailed parsing steps, such as "Subtract 3 from both sides of the equation to get 2x=4, and then divide it by 2 to get x=2". In this way, we get the parsing results generated by the large model, which provides a basis for the next step of comparison.

[0091] S3: Determine the difficulty level of the test question by comparing the degree of consistency between the parsing results and the reference answers. At this stage, we compare the parsing results generated by the large model with the standard answers to the test questions. Continuing with the above math question as an example, if the large model can correctly parse the solution of the equation as "x=2", then we can assume that the model has a good understanding of the question requirements and has given the correct answer. Next, we will evaluate the difficulty of the test question based on the degree of match between the parsing results and the standard answers. For example, if the large model can parse the answer quickly and accurately, it means that the question is relatively simple; conversely, if errors occur during the parsing process or complex reasoning steps are required to obtain the correct answer, it means that the question is relatively complex. In this way, we can assign a difficulty level to each test question, such as "easy", "medium" or "difficult".

[0092] Wherein, the step S3 includes:

[0093] Perform word segmentation on the parsing results and reference answers, and convert them into sequences respectively;

[0094] By comparing the elements in the two sequences, the distance matrix between the two sequences is calculated. The calculation formula of the distance matrix is:

[0095] ,

[0096] in, represents the cumulative minimum alignment distance from the i-th word in the parsing result sequence to the j-th word in the reference answer sequence, represents the distance between the i-th word in the parsing result sequence and the j-th word in the reference answer sequence, where i and j represent the indexes of the parsing result sequence and the reference answer sequence respectively;

[0097] Through the above calculations, we can obtain the minimum alignment path from the upper left corner of the matrix D(1,1) to the lower right corner D(m,n), where m and n represent the lengths of the parsing result sequence and the reference answer sequence, respectively.

[0098] The difficulty of the test questions is divided according to the size of the distance D(m,n).

[0099] The above steps address the issue of how to accurately assess the difficulty of test questions using a large model. This approach not only provides an objective difficulty assessment but also helps question setters better understand the challenges candidates may face and adjust talent assessment strategies or test design accordingly.

[0100] Next, we describe the specific steps of a method for assessing the difficulty of test questions based on a large model:

[0101] S11: Retrieving exam questions from the database. This process involves using a database query language or a specific API to locate and extract the required exam questions. For example, in a SQL environment, you can write a SELECT statement that specifies the fields and conditions to be retrieved, such as "Select question from exam_questions where subject = 'Mathematics' and level = 'High School'." This step ensures that the exam questions that match the specific subject and difficulty level are accurately retrieved from the database.

[0102] S12: Extract the corresponding reference answers based on the exam question's identifier. After extracting the exam questions, the system retrieves the corresponding reference answers based on each exam question's unique identifier, such as its ID number. This is typically accomplished by executing an additional query in the database, such as "Select answer from exam_answers where question_id = 'Q123456789'." This ensures that the retrieved reference answers exactly match the exam question.

[0103] S13: The test questions are formatted to ensure that they meet the requirements of the large language model input. In order for the test questions to be correctly parsed and processed by the large language model, they need to be formatted. This may include but is not limited to removing special characters, converting to plain text format, standardizing punctuation, etc. For example, if the original test questions contain HTML tags, they need to be converted to plain text format using regular expressions or other text processing tools, such as " What is the capital of France? ” is converted to “What is the capital of France?”

[0104] S14: Store the formatted test questions and their corresponding reference answers in a temporary cache. Finally, the formatted test questions and their corresponding reference answers will be stored in a temporary cache for subsequent use as input to the large model. This can be achieved by writing data into an in-memory data structure such as an array or list, or using a dedicated cache management system such as Redis. For example, a dictionary structure can be created in which the key is the ID of the test question and the value is a tuple containing the test question text and the reference answer text, such as `{'Q123456789': 'What is the capital of France?', 'Paris'}`. Such a caching mechanism helps improve the system's response speed and reduce frequent access to the database.

[0105] Next, the specific steps of step S12 are described for extracting the corresponding reference answer according to the identifier of the test question:

[0106] Construct a mapping table between question identifiers and reference answers: First, establish a database or data structure to store the question identifiers and their corresponding reference answers. This mapping table can be a hash table, database table, or other data structure that enables fast search. In this table, each question identifier is used as the key, and the corresponding reference answer is stored as the value. This way, when searching for the reference answer for a specific question, you can directly locate the corresponding record using its identifier.

[0107] Based on the question's identifier, the system queries the mapping table to obtain the corresponding reference answer: When a question's identifier is received, the system uses it as a key to query the mapping table. The query process typically involves fast access operations on the storage structure, such as hash lookups, to ensure efficient retrieval of the corresponding data. If the query is successful, meaning a record matching the identifier is found, the reference answer in that record is returned. If the query fails, meaning no match is found, then the question has no corresponding reference answer.

[0108] Processing query results: Different actions are taken based on the query results. If the query result is empty, meaning no records matching the given identifier are found, the system reports that there are no corresponding reference answers for the question. Conversely, if the query result is not empty, meaning a match is found, the found reference answers are used for subsequent processing, such as further analysis of the question's difficulty or other related processing.

[0109] Through the above steps, we can effectively manage and utilize the mapping relationship between propositional examination questions and their reference answers, thereby supporting the implementation of the propositional examination question difficulty assessment method based on the big model.

[0110] Next, we describe the specific steps for constructing a mapping table between question identifiers and reference answers:

[0111] This process begins by collecting the test questions and their corresponding reference answers and assigning a unique identifier to each. This step ensures that each test question has a unique label, facilitating subsequent processing and retrieval. For example, in practice, a database table or data structure can be designed to store these test questions and their identifiers. Each identifier is guaranteed to be unique, either through auto-incrementing or using a universally unique identifier (UUID).

[0112] This process continues by associating the question's unique identifier with the corresponding answer. This means adding information about the corresponding answer to the data structure that stores the question's identifier. For example, you could create a database field to store the answer, or add a key-value pair to the data structure, where the key is the question's unique identifier and the value is the corresponding answer.

[0113] Next, determine whether the unique identifier of the test question already exists in the mapping relationship table. If not, add the mapping relationship; if so, update the reference answer in the mapping relationship table. This step can be implemented by querying the database or data structure to check whether a given identifier already exists. For example, you can write an SQL query statement to find a record with a specific identifier. If it is not found, insert a new record containing the identifier and its corresponding reference answer; if it is found, update the reference answer information in the record. This ensures that the mapping relationship table is always up to date and that each identifier accurately corresponds to the latest reference answer.

[0114] Furthermore, the step of determining whether the unique identifier of the test question already exists in the mapping relationship table specifically includes:

[0115] By traversing all records in the mapping table, check whether the unique identifier of the current test question already exists in the mapping table. This process includes accessing each record in the mapping table in turn and comparing its unique identifier with the unique identifier of the current test question. The comparison method can be adjusted according to the data type of the unique identifier (such as string or number).

[0116] During the traversal process, if a record's unique identifier is found to match the current question's unique identifier, that is, they are identical, then it is determined that the unique identifier already exists in the mapping table. This conclusion is achieved through a string or number equality check, depending on the data type of the unique identifier.

[0117] If no matching identical record is found after the traversal is completed, it is determined that the unique identifier does not exist in the mapping table, and the identifier is considered to be non-existent, which indicates that the test question is new and has not yet been evaluated or recorded.

[0118] If it is determined that the unique identifier does not exist in the mapping table, an add operation will be performed to insert a new record containing the unique identifier and related difficulty assessment information. This insertion process can be implemented through database commands (such as SQL INSERT statements) or equivalent operations in other database management systems.

[0119] If the identifier exists, that is, the unique identifier already exists in the mapping table, an update operation will be performed to modify the existing record to reflect any changes in the difficulty assessment or other relevant attributes of the test question. The update is usually performed through a database command (such as an SQL UPDATE statement) or similar operation.

[0120] Next, the adding operation is described in detail:

[0121] Create a new record containing the unique identifier of the exam question and its corresponding reference answer. This step ensures that each question is uniquely identified and associated with the correct reference answer, which is critical to maintaining the integrity of the database.

[0122] After the new record is created, it is added to the mapping table. Through this step, the association between the unique identifier of the exam question and its reference answer is established, and the database is updated to include the new question and its related data.

[0123] After adding a new record, the mapping table's version number is updated to reflect the latest changes. This version control mechanism helps track modifications and ensures that the system can be restored to a previous state if necessary.

[0124] Finally, a consistency check is performed after the add operation to verify that the new record was successfully added and that the database remains consistent. This check is critical to ensuring the reliability and accuracy of the data stored in the mapping table.

[0125] Next, we will further describe in detail how to parse the test questions based on the pre-trained large language model and generate parsing results:

[0126] First, the test questions are fed into the pre-trained large language model. This step is fundamental to the entire evaluation process, ensuring that the large language model receives the test questions and is ready for subsequent processing.

[0127] Next, the large language model's internal language structure analysis mechanism is used to extract key information from the test questions. This mechanism leverages the language structure knowledge learned during the large language model's training process to effectively identify important words and phrases in the test questions and the relationships between them, thereby helping to determine which information is crucial for understanding the test questions.

[0128] Then, a formulaic method is used to calculate the relevance score of the key information. The relevance score R is determined by the frequency F of the keyword in the question and the position weight W, that is, R = F × W. The frequency of the keyword reflects its importance in the question, while the position weight considers the keyword's position in the sentence or paragraph. Generally, keywords at the beginning or end of a sentence may have a higher weight because they tend to better reflect the core content of the question.

[0129] Finally, a preliminary analysis result is generated based on the key information. This step integrates the information from all previous steps. By analyzing the key information and its relevance scores, a preliminary understanding and interpretation of the test questions can be obtained, providing a basis for further assessment of their difficulty.

[0130] For example, in terms of technical implementation, let's assume we have a question: "Find the roots of the quadratic equation ax² + bx + c = 0." First, we input this question into a pre-trained large language model. Next, the model automatically identifies keywords such as "quadratic equation" and "root" and analyzes their position in the sentence. It then calculates the keyword frequency and position weights to determine their relevance scores. Finally, based on this key information, it generates a preliminary analysis of the question, indicating, for example, that this is a question about solving a quadratic equation and that a specific mathematical formula is required to solve it.

[0131] Next, we will further describe the specific steps for extracting key information from test questions based on the internal language structure analysis mechanism of the large language model:

[0132] The test questions are segmented to obtain their basic vocabulary units. In this step, the test questions are first segmented, breaking them down into a series of basic vocabulary units. This process utilizes the word segmentation algorithm in natural language processing technology to ensure that the test questions can be accurately broken down into the basic elements that constitute their meaning.

[0133] The semantic understanding capabilities of the large language model are leveraged to identify entities and concepts within the test questions. After word segmentation, the powerful semantic understanding capabilities of the large language model are leveraged to identify the entities and concepts represented by these basic lexical units. This step relies on the model's deep understanding of language structure and effectively identifies the key entities and concepts involved in the test questions.

[0134] The importance of an entity or concept is determined using a formulaic approach, where importance I is determined by its frequency of occurrence F in the domain knowledge base and its position P in the test question, i.e., I = F + P. Based on the frequency of occurrence F of the entity or concept in the domain knowledge base and its position P in the test question, the importance I of each entity or concept is calculated using the formula I = F + P. Here, frequency reflects the prevalence and relevance of the entity or concept in a specific domain, while position considers its relative importance in the test question. For example, concepts appearing at the beginning of a sentence are often more important than those appearing at the end.

[0135] Finally, based on the calculated importance value I, the key entities and concepts in the propositional test questions are screened out. This step helps determine which entities and concepts are crucial for understanding the propositional test questions and provides a basis for further difficulty assessment.

[0136] Through the above steps, key information can be effectively extracted from the test questions to provide support for subsequent difficulty assessment.

[0137] Next, we describe the specific steps of using a formulaic method to determine the importance of an entity or concept, where the importance I is determined by the frequency F of the entity or concept in the domain knowledge base and its position P in the test question, that is, I = F + P:

[0138] Query the domain knowledge base to obtain the frequency of occurrence F of entities or concepts, and obtain the frequency data of entities or concepts related to the domain by accessing a structured database or related information base. This process involves querying the statistical information and related data stored in the domain knowledge base to retrieve the frequency of occurrence of each entity or concept under consideration.

[0139] Different position weights P are assigned to entities or concepts based on their position within the question. The system analyzes the structure of the question and assigns weights based on the entity or concept's position within the question. Generally, the system assigns higher weights to components that are more important in determining the question's difficulty, such as the question's main body or key verbs.

[0140] The importance I is calculated based on the occurrence frequency F and the position weight P, and the overall importance I of the entity or concept in the proposition question is calculated using the formula I = F + P. Wherein, I represents the overall importance of the entity or concept in the context of the proposition question.

[0141] If the importance I is greater than a predetermined threshold T, the entity or concept is considered to be key information, and the calculated importance I is compared with the preset threshold T. If the importance I exceeds the threshold T, the system identifies the entity or concept as key information for evaluating the difficulty of the proposition question.

[0142] For example, consider a propositional question in the field of computer science, "What is the main function of the CPU in a computer system?" The system queries the domain knowledge base to obtain the frequency of "CPU," assuming its frequency is 500. The system then analyzes the position of "CPU" in the question and assigns a higher weight, such as 0.8, because it is the main subject of the question. Using the formula I = F + P, the system calculates the importance of "CPU" as I = 500 + 0.8. If the predetermined threshold T is set to 500.5, "CPU" is identified as key information, indicating that understanding the concept of "CPU" is crucial to correctly answering the question.

[0143] Further describing the specific steps of considering the entity or concept as key information if the importance I is greater than the predetermined threshold T:

[0144] The calculated importance of the entity or concept, I, is compared with a pre-set threshold, T. This process can be implemented using a software algorithm, where importance, I, is a quantitative indicator derived from analyzing and evaluating the entities or concepts in the test questions, while threshold, T, is a value set based on prior empirical data or specific standards to determine whether the entity or concept is important enough to affect the difficulty of the test question.

[0145] If the calculated importance I is greater than a predetermined threshold T, the entity or concept is marked as key information. This step can be implemented by setting conditional statements in a computer program, such as using an if-else structure. When I>T is satisfied, the marking operation is executed. Typically, a flag or label is added to the data structure to indicate that the entity or concept is key information.

[0146] If the calculated importance I is less than or equal to the predetermined threshold T, the entity or concept is ignored. Similarly, this step can be implemented by setting a conditional statement in the program, that is, when I ≤ T, no action is performed or it is explicitly excluded from subsequent processing.

[0147] Collect all entities or concepts marked as key information. This step can be accomplished by traversing the processed data structure and extracting the marked key information. In practical applications, arrays, lists, or other data sets can be used to store this key information for further analysis or processing.

[0148] Next, we describe the specific steps to collect all entities or concepts marked as key information:

[0149] All entities or concepts marked as key information are summarized. Through in-depth analysis of the text content of the test questions, key information, including but not limited to entities, concepts, and other important knowledge points, is identified and marked. This key information can be identified using named entity recognition (NER) and keyword extraction algorithms in natural language processing. Once this key information is identified, the system will summarize it and form a list or database containing all key information.

[0150] After summarizing all key information, use a formulaic method to count the number of key information N. For example, this process can be implemented using a simple counting function to calculate the total number of key information N. This step aims to quantify the number of knowledge points involved in the test questions, thereby providing data support for further difficulty assessment.

[0151] If the number N exceeds the preset benchmark number B, the difficulty level of the test question is increased. If the number N does not exceed the preset benchmark number B, the difficulty level of the test question remains unchanged. Next, the number of key information N obtained by statistics is compared with the preset benchmark number B. If N exceeds the preset benchmark number B, it indicates that the test question contains a lot of key information, and therefore its difficulty level needs to be increased; conversely, if N does not exceed the preset benchmark number B, the test question is considered to be of moderate difficulty and no adjustment is required. This comparison can be implemented through programming, for example, by setting a conditional statement in the software to automatically complete this judgment process.

[0152] The step S3 specifically includes:

[0153] For the obtained test questions and reference answers, we first use the pre-trained large language model to generate the semantic vector representation of each test question and reference answer. Let the test question be Q, the corresponding reference answer be A, and the parsing result be R. The generated semantic vectors are represented as vectors V Q 、V A and V R ;

[0154] Based on the generated semantic vector, the similarity between the parsing result R and the reference answer A is calculated using the cosine similarity formula. The formula for calculating cosine similarity is:

[0155] ,

[0156] in represents the dot product of vectors VR and VA, and are the moduli of vectors VR and VA respectively;

[0157] Evaluate the consistency between the parsing results and the reference answers in multiple dimensions. Combine the above multiple dimensions and use the weighted average method or other comprehensive scoring methods to calculate the comprehensive consistency score. The calculation formula for the comprehensive consistency score St is:

[0158] St = Ws × Ss + Wg × Sg + Wl × Sl + Wc × Sc, where Ss is the semantic similarity score, Sg is the semantic consistency score, Sl is the logical consistency score, Sc is the content consistency score, Ws, Wg, Wl and Wc are the weight coefficients of the scores of each dimension, satisfying Ws + Wg + Wl + Wc = 1;

[0159] Finally, based on the comprehensive consistency score St, the difficulty level of the test questions is evaluated and the difficulty level is set as "easy", "medium" and "difficult", and the corresponding score range is set for each level.

[0160] If St ≥ 0.85, the assessment proposition test question is “easy”;

[0161] If 0.70≤St<0.85, the assessment is “moderate”;

[0162] If St<0.70, the assessment is “difficult”.

[0163] In this implementation, the system uses a pre-trained large language model to deeply parse a test question Q and its reference answer A, generating semantic vector representations for each question and answer. The parsed result R is also processed through the same model to generate its own semantic vector VR. These vectors represent the position of the question and answer in semantic space. By calculating the cosine similarity between these vectors, the system can quantify the semantic proximity between the parsed result R and the reference answer A.

[0164] Cosine similarity is calculated by taking the dot product of vectors VR and VA and dividing it by the product of their modulo lengths. A higher similarity score indicates a greater semantic consistency between the parsed result and the reference answer. The system not only calculates semantic similarity but also evaluates grammatical, logical, and content consistency. Taking these dimensions into account, it calculates an overall consistency score, St, using a weighted average. Ultimately, based on this combined score, the system categorizes test questions into three levels: "easy," "medium," or "difficult."

[0165] This implementation effectively combines analysis across multiple dimensions, providing a multi-layered approach to difficulty assessment. By quantifying the degree of semantic, grammatical, logical, and content compatibility, the system can more comprehensively assess the difficulty of questions, ensuring accurate assessments. This multi-dimensional analysis enables the system to adapt to different types of questions, providing a more scientific difficulty categorization, thereby enhancing the fairness and effectiveness of the examination system.

[0166] In this embodiment, the weight coefficients Ws, Wg, Wl, and Wc can be adjusted based on specific application scenarios to accommodate different types of test questions. For example, for questions that emphasize logical reasoning, the weight of Wl can be increased, while for language-based questions, the weights of Wg and Ws can be increased. Furthermore, the difficulty level thresholds can be dynamically adjusted based on the actual situation of the test question bank to better adapt to the difficulty distribution of the test questions in the question bank.

[0167] Through the above steps, the difficulty of the test questions can be effectively evaluated, and the difficulty level can be dynamically adjusted according to the amount of key information to ensure the rationality and fairness of the test questions.

[0168] In actual operation, the system first receives the test questions and their corresponding reference answers through the input interface. At this stage, the user or system enters the test questions and answers into the evaluation system, which serves as the basic data for subsequent analysis. Next, the system uses a pre-trained large language model to perform in-depth analysis of these test questions. Leveraging its powerful natural language processing capabilities, the large language model conducts multi-dimensional analysis of the test questions, including semantic understanding and logical reasoning. Based on these results, it generates a detailed analysis report, known as the analysis result. The system then compares the analysis result with the pre-provided reference answers and uses an algorithm to calculate the degree of consistency between the two. This consistency can be measured using various metrics, such as precision, recall, and F1 score. Based on the degree of consistency, the system automatically assigns each test question a difficulty level, such as easy, medium, or hard. Throughout this process, various components work closely together: the input interface collects raw data; the large language model provides the core analysis capabilities; and the comparison module ensures accurate evaluation of the analysis results, ultimately forming a complete process for evaluating the difficulty of test questions. This design not only improves the accuracy of the assessment, but also greatly enhances the efficiency, bringing revolutionary changes to the field of educational assessment.

[0169] A system for assessing the difficulty of test questions based on a large model is also provided. The system adopts the method for assessing the difficulty of test questions based on a large model, and includes:

[0170] An input module, which is used to obtain test questions and their corresponding reference answers;

[0171] A large language model parsing module, which is used to parse the test questions based on the pre-trained large language model and generate parsing results;

[0172] The comparison module is used to compare the consistency between the analysis results and the reference answers and determine the difficulty level of the test questions.

[0173] In the above solution, the input module's primary function is to receive and store test questions and their corresponding reference answers. This module can collect test questions and answers through user interface input, file upload, or API access. The input module ensures data integrity and accuracy, providing a reliable data foundation for subsequent analysis and difficulty assessment.

[0174] The large language model parsing module is the core of the system. This module parses the input test questions based on a pre-trained large language model (such as GPT-4) and generates parsed results. This module leverages the powerful natural language understanding capabilities of the large language model to perform semantic analysis and logical reasoning on the test questions, resulting in in-depth analysis of each question. The parsing process goes beyond simple text processing and includes a deep understanding of the underlying characteristics of the test question difficulty, such as the complexity of the question stem, the diversity of the answers, and the complexity of the reasoning paths.

[0175] The comparison module is responsible for comparing the parsing results generated by the large language model parsing module with the reference answers. By designing a well-designed comparison algorithm, this module can assess the degree of consistency between the parsing results and the reference answers. The comparison criteria can be based on various dimensions, such as semantic similarity and structural matching. When the degree of consistency is high, the difficulty of the test question is assessed as low; conversely, if the parsing results differ significantly from the reference answers, the difficulty of the test question is assessed as high. This difficulty assessment mechanism helps accurately measure the challenge of the test questions and provides valuable reference advice for test setters.

[0176] In alternative implementations, the performance of the parsing module can be optimized by adjusting the size of the large language model or the diversity of the training data. Furthermore, the algorithm of the comparison module can be customized to meet different exam requirements, such as increasing the ability to handle multiple-answer scenarios or introducing a machine learning-based model to further improve the accuracy of difficulty assessment.

[0177] Through the collaborative work of the above modules, the system can effectively and accurately assess the difficulty of exam questions and provide exam experts with automated and intelligent exam question analysis tools, thereby improving exam quality and exam question setting efficiency.

[0178] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for assessing the difficulty of test questions based on a large model, characterized in that: The method comprises the following steps: S1: Obtain the test questions and their corresponding reference answers; S2: Parse the test questions based on the pre-trained large language model and generate parsing results; S3: Compare the consistency between the analysis results and the reference answers to determine the difficulty level of the test questions; Wherein, the step S3 includes: Perform word segmentation on the parsing results and reference answers, and convert them into sequences respectively; By comparing the elements in the two sequences, the distance matrix between the two sequences is calculated. The calculation formula of the distance matrix is: ， in, represents the cumulative minimum alignment distance from the i-th word in the parsing result sequence to the j-th word in the reference answer sequence, represents the distance between the i-th word in the parsing result sequence and the j-th word in the reference answer sequence, where i and j represent the indexes of the parsing result sequence and the reference answer sequence respectively; Through the above calculations, we can obtain the minimum alignment path from the upper left corner of the matrix D(1,1) to the lower right corner D(m,n), where m and n represent the lengths of the parsing result sequence and the reference answer sequence, respectively. Divide the difficulty of the test questions according to the size of the distance D(m,n); The specific steps of parsing the test questions based on the pre-trained large language model and generating parsing results in S2 include: S21: Inputting the test question into the pre-trained large language model; S22: extracting key information of the test questions based on the internal language structure analysis mechanism of the large language model; S23: Calculate the relevance score of key information using a formulaic method, where the relevance score R is determined by the frequency F of the keyword in the test question and the position weight W, i.e., R = F × W; S24: Generate preliminary analysis results based on the key information.

2. A method for assessing the difficulty of test questions based on a large model according to claim 1, characterized in that: The step S1 of obtaining the test questions and their corresponding reference answers specifically includes: S11: Retrieve test questions from the database; S12: extracting the corresponding reference answer based on the identifier of the test question; S13: Format the test questions to ensure they meet the requirements of the large language model input; S14: The formatted test questions and their corresponding reference answers are stored in a temporary buffer area.

3. A method for assessing the difficulty of test questions based on a large model according to claim 2, characterized in that: The step S12 of extracting the corresponding reference answer based on the identifier of the test question specifically includes: Construct a mapping relationship table between the test question identifiers and the reference answers; Query the mapping table based on the identifier of the test question to obtain the corresponding reference answer; If the query result is empty, it is considered that the question has no corresponding reference answer; If the query result is not empty, the reference answer found will be used as the object for subsequent processing; The mapping relationship table between the test question identifier and the reference answer is constructed as follows: Collect test questions and their corresponding reference answers, and assign a unique identifier to each test question; Establish an association with the reference answer based on the unique identifier of the test question; Determine whether the unique identifier of the test question already exists in the mapping relationship table. If not, add the mapping relationship; If it exists, update the reference answer in the mapping table.

4. A method for assessing the difficulty of test questions based on a large model according to claim 3, characterized in that: The step of determining whether the unique identifier of the test question already exists in the mapping relationship table specifically includes: Traverse all records in the mapping relationship table; If a record with the same unique identifier as the current test question is found, it is considered that the identifier already exists; If no identical record is found after the traversal, the identifier is considered to not exist; If the identifier does not exist, perform the add operation; if the identifier exists, perform the update operation; The performing of the adding operation specifically includes: Create a new record containing the unique identifier of the test question and the corresponding reference answer; Add the newly created record to the mapping relationship table; Update the version number of the mapping table to reflect the latest changes; Perform a consistency check to ensure that the add operation was successful.

5. The method for assessing the difficulty of test questions based on a large model according to claim 1, characterized in that: The step of extracting key information of the test questions based on the internal language structure analysis mechanism of the large language model includes: Perform word segmentation on the test questions to obtain the basic vocabulary units of the test questions; Identify entities and concepts in test questions based on the semantic understanding capability of the large language model; Use a formulaic method to judge the importance of an entity or concept, where the importance I is determined by the frequency F of the entity or concept in the domain knowledge base and the position P in the proposition test question, that is, I=F+P; Screen out the key entities and concepts in the exam questions.

6. A method for assessing the difficulty of test questions based on a large model according to claim 5, characterized in that: The importance of an entity or concept is determined by a formulaic method, where the importance I is determined by the frequency F of the entity or concept in the domain knowledge base and the position P in the test question, i.e., I = F + P. Specifically, it includes: Query the domain knowledge base to obtain the occurrence frequency F of entities or concepts; According to the position of the entity or concept in the test question, different position weights P are assigned; Calculate the importance I based on the occurrence frequency F and the position weight P; If the importance I is greater than a predetermined threshold T, the entity or concept is considered to be key information.

7. A method for assessing the difficulty of test questions based on a large model according to claim 6, characterized in that: If the importance I is greater than a predetermined threshold T, the entity or concept is considered to be key information, specifically including: Comparing the calculated importance I with a predetermined threshold T; If I>T, the entity or concept is marked as key information; If I≤T, ignore the entity or concept; Collect all entities or concepts marked as key information; The collection of all entities or concepts marked as key information specifically includes: Summarize all entities or concepts marked as key information; Use a formulaic method to count the number of key information N; If the number N exceeds the preset benchmark number B, the difficulty level of the test questions will be increased; If the number N does not exceed the preset benchmark number B, the difficulty level of the test questions remains unchanged.

8. The method for assessing the difficulty of test questions based on a large model according to claim 1, characterized in that: The step S3 specifically includes: For the obtained test questions and reference answers, we first use the pre-trained large language model to generate semantic vector representations for each test question and reference answer. Let the test question be Q, the corresponding reference answer be A, and the parsing result be R. The generated semantic vectors are represented as vectors VQ, VA, and VR respectively. Based on the generated semantic vector, the similarity between the parsing result R and the reference answer A is calculated using the cosine similarity formula. The formula for calculating cosine similarity is: ， in Represents vector V R and V A The dot product of and They are vectors V R and V A The model; Evaluate the consistency between the parsing results and the reference answers in multiple dimensions. Combine the above multiple dimensions and use the weighted average method or other comprehensive scoring methods to calculate the comprehensive consistency score. The calculation formula for the comprehensive consistency score St is: St = Ws × Ss + Wg × Sg + Wl × Sl + Wc × Sc, where Ss is the semantic similarity score, Sg is the semantic consistency score, Sl is the logical consistency score, Sc is the content consistency score, Ws, Wg, Wl and Wc are the weight coefficients of the scores of each dimension, satisfying Ws + Wg + Wl + Wc = 1; Finally, based on the comprehensive consistency score St, the difficulty level of the test questions is evaluated and the difficulty level is set as "easy", "medium", and "difficult", and the corresponding score range is set for each level. If St≥0.85, the assessment question is "easy"; If 0.70≤St<0.85, the assessment is "medium"; If St<0.70, the assessment is "difficult".

9. A large-scale model-based examination question difficulty assessment system, characterized by: The method for assessing the difficulty of test questions based on a large model according to any one of claims 1 to 8 is adopted, wherein the system comprises: An input module, which is used to obtain test questions and their corresponding reference answers; A large language model parsing module, which is used to parse the test questions based on the pre-trained large language model and generate parsing results; The comparison module is used to compare the consistency between the analysis results and the reference answers and determine the difficulty level of the test questions.

Images